One-Step Generalization Ratio Guided Optimization for Domain Generalization
Authors: Sumin Cho
Dongwon Kim
Kwangsu Kim
Venue: ICML 2025 oral (2025)
Keywords: Domain Generalization
Optimization
Preconditioning
One-Step Generalization Ratio (OSGR)
Out-of-Distribution
Primary area: general_machine_learning
TL;DR: We propose GENIE, a novel optimizer that leverages the One-Step Generalization Ratio to dynamically balance parameter contributions, mitigating source-domain overfitting and achieving superior generalization performance in Domain Generalization.

Abstract: Domain Generalization (DG) aims to train models that generalize to unseen target domains but often overfit to domain-specific features, known as undesired correlations. Gradient-based DG methods typically guide gradients in a dominant direction but often inadvertently reinforce spurious correlations. Recent work has employed dropout to regularize overconfident parameters, but has not explicitly adjusted gradient alignment or ensured balanced parameter updates. We propose GENIE (Generalization-ENhancing Iterative Equalizer), a novel optimizer that leverages the One-Step Generalization Ratio (OSGR) to quantify each parameter's contribution to loss reduction and assess gradient alignment. By dynamically equalizing OSGR via a preconditioning factor, GENIE prevents a small subset of parameters from dominating optimization, thereby promoting domain-invariant feature learning. Theoretically, GENIE balances convergence contribution and gradient alignment among parameters, achieving higher OSGR while retaining SGD's convergence rate. Empirically, it outperforms existing optimizers and enhances performance when integrated with various DG and single-DG methods.

One-Step Generalization Ratio Guided Optimization for Domain Generalization
Sumin Cho Dongwon Kim
Kwangsu Kim
Abstract
Domain Generalization (DG) aims to train models that generalize to unseen target domains but often overfit to domain-specific features, known as un- desired correlations. Gradient-based DG methods typically guide gradients in dominant direction but often inadvertently reinforce spurious corre- lations. Recent work has employed dropout to regularize overconfident parameters, but has not explicitly adjusted gradient alignment or ensured balanced parameter updates. We propose GENIE (Generalization- Eaairaaliattoo-ENhannnnng Iterative Equalizer), a novel optimizer that leverages the One-Step Gen- eralization Ratio (OSGR) to quantify each param- eter's contribution to loss reduction and assess gradient alignment. By dynamically equalizing OSGR via a preconditioning factor, GENIE pre- vents a small subset of parameters from domi- nating optimization, thereby promoting domain- invariant feature learning. Theoretically, GENIE balances convergence contribution and gradient alignment among parameters, achieving higher OSGR while retaining SGD's convergence rate. Empirically, it outperforms existing optimizers and enhances performance when integrated with various DG and single-DG methods.
C1
VOC2007
SGD
3.5
2.0
Adam
2.5
2.0
1.5
SAM
2.5
2.0
0.
GENIE
60
05
1.0
15
2.0
0.0
1.
2.0
Paramidta
Peremdiir Indox
Figure 1. Hcatmaps visualizing normalized parameter update mag- nitudes by parameter for different optimizers throughout train- ing on the VLCS dataset in the DG. Previous optimizers (SGD, Adam, SAM) exhibit an imbalanced parameter update distribution, where subset of parameters dominates the optimization process, In contrast, GENIE uniformly adjusts parameter- wise OSGR, mit- igating overfitting to specific parameters and promoting a more balanced optimization across the entire parameter space.
to domain-specific features- as spurious correla- tions-whil learning invariant features and causal relation- ships that generalize across diverse domains(Shi et al., 2022; Hemati et al., 2023; Shah et al., 2020a; Ye et al., 2024).
1. Introduction
Several DG methods have attempted to guide the gradient toward dominant direction during training (Parascandolo et al., 2021; Shahtalebi et al., 2021; Shi et al., 2022; Rame et al., 2022). However, this dominant direction often itself driven by spurious features, inadvertently reinforcing un- desired correlations. This suggests that aligning gradients toward a single dominant direction is insufficient to fully solve the problem, highlighting the need for other perspec- tives.
Deep neural networks (DNNs) achieve high accuracy when training and test data share a similar distribution. How- ever, in real- wolld applications. data distributions often shift, causing performance degradation(Muandee et al., 2013). Do- main Generalization (DG) addresses this issue by training models to generalize to distriiittiiitionn data from un- seen domains. The main challenge is to prevent overfitting
'Equal contribution 'Department of Computer Science and Engineering, University of Sungkyunkwan. Suwon. Korea. Correspondence to: Sumin Cho <jsn0707@skku.eduu @sk.uu.ukuuu. Dongwon Kim "kdwaha@skku.edu <kim.kwangsu skku.edu>.
A recent approach (Michalkiewicz al., 2023) introduced parameter wise dropout mechanism based Gradient Signal-to-Noiss Ratios (GSNR) to suppress overly predic- tive parameters and reduce their influence on optimization. While this strategy mitigates parameter updates driven by spurious correlations, it does not adjust the magnitudes of updates based on their individual contributions to general-
Kwangsu Kim
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
ization. This raises the open question of how design opti- mizers that explicitly halance parameter updates according to their principled contributions to generalization, thereby mitigating the influence of spurious correlations.
including methods such domain-invariant feature learning (Sun & Saenko, 2016; Arjovsky et al., 2019; Krueger et al., 2021), data augmentation (Xu et al., 2020; Yan et al., 2020; Wang et al., 2020), and feature disentanglement (Nam et al., 2021; Mahajan et al., 2021). (2) Gradient A Alignment, which focuses on aligning gradients across domains to ensure sta- ble learning dynamics. Representative approaches include minimizing gradient differences (Koyama & Yamaguchi, 2020), increasing gradient inner products (Shi et al., 2022), updating weights only when gradient directions align (Paras- candolo et al., 2021; Shahtalebi et al., 2021), and reducing inter-domain gradient variance (Rame et al., 2022). Re- cently, Sharpness Aware Minima (FAM) et al., 2021) has improved in-distribution generalization, inspiring the development of optimizers specifically designed for OOD tasks (Zhang et al., 2024; Wang et al., 2023). However, most DG studies overlook imbalanced parameter updates caused by differences in convergence speed or generalization ca- pacity during optimization.
Motivated by this perspective, we propose Generalization- ENhancing Iterative Equalizer (GENIE), a novel optimizer for addressing parameter imbalance. Recent work (Liu al., 2020) introduced the One-Step Generalization Ratio (OSGR) that measures how effectively single gradient update reduces test loss compared to training loss. provid- ing insight into a model's 's generalization potential. OSGR reflects the contributions of individual parameters to gener- alization, based on their convergence speed and degree of gradient alignment. To leverage this insight, GENIE inte- grates a preconditioning factor that dynamically balances parameter-wiss wise OSGR throughout training. This prevents small subset of parameters from dominating the optimiza- tion, thereby promoting more robust and invariatn feature learning.
Our theoretical analysis shows that existing optimizers typically focus on either convergence speed or gradient alignment, often resulting in suboptimal generalization. In contrast, GENIE explicitly balances both, achieving higher OSGR while maintaining the convergence rate of SGD(Robbins & Monro, 1951) in non-convex settings. We empirically validated GENIE on five standard DG datasets(Li et al., 2017: Fang et al., 2013; Venkateswara et al., 2017; Beery et al., 2018; Peng et al., 2019) where it consistently outperformed established optimizers, even with extended iterations. Furthermore, using our optimizer in isting DG and Single-DG (SDG) algorithms enhances their performance. We summarize our contributions as follows:
2.2. Preconditioning
Preconditioning improves the efficiency of optimization al- gorithms by incorporating curvature information of the loss function or adjusting the magnitude and direction of pa- rameter updates. It accelerates convergence and enhances stability during training and can be categorized into three main types (Ye, 2024; Amari et al., 2021) (1) Hessian Based Preconditioning: utilizes the inverse approximations of the Hessian matrix to capture curvature information. (Mon- tavon et al., 2012; Dennis & Moré, 1977) (2) Adaptive Learning Rate Based Preconditioning: dynamically adjusts learning rates based on gradient magnitudes, as seen in opti- mizers like AdaGrad (Duchi et al., 2011), 1), RMSProp (Hinton et al., 2012), and Adam (Kingma, 2014). (3) Normalization- Based Preconditioning: normalizes inputs and activations, as exemplified by Batch Normalization(Ioffc & Szegedy, 2015), to improve the Hessian's 's condition number and en- hance training stability. Previous preconditioning methods aim to optimize speed and stability. The application of preconditioning to improve model generalization remains underexplored.
We propose GENTE, a novel optimizer that addresses the overlooked issue of parameter imbalance in DG. It suppresses over-predictive parameters while promoting balanced parameter updates.
We incorporate OSGR, previously used as general- ization metric, into the optimizer's 's core principle. This provides an efficient and novel perspective on general- ization for addressing DG.
3. Method
GENTE is domain-agnosti optimizer. It is vali- dated across multiple DG benchmarks and SDG tasks, demonstrating its broad applicability and scalability.
3.1. Preliminary
To address the challenge of generalization in unseen target domains, a recent study(LLiu et al., 2020) introduced the concept of OSGR R(Z,n) n). OSGR quantifies how well model updates contribute to generalization by measuring the ratio of loss reduction between test D' and training data D after a single optimization step:
2. Related Work
2.1. Domain Generalization
Existing DG methods address domain shift through two main strategies:
(1) Feature Alignment, which aims to align features across domains to ensure consistent optimization,
ED,D'~z ALD
R(Z,n)
(I)
Epag. ALD
2

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
where ALD and ALD represent the loss changes on test and training data, respectively. OSGR is influenced by two key factors: (1) the contribution of each parameter to loss reduction, characterized by the gradient magnitude, and (2) the alignment of parameter gradients across the data distribution. Higher OSGR indicates better generalization, reflecting consistent and balanced parameter updates.
troduce a preconditioning term that more accurately adjusts the OSGR. Next, we inject noise into all parameters to en- courage exploration toward better optima. Finally, we apply random dropout to stabilize parameter updates and reduce overfitting.
3.2.1. PRECONDITIONING
To better understand these dynamics, the following theorem links OSGR to parameter- -iiee statistics:
We propose preconditioning factor Pj to ensure balanced contributions of each parameter the OSGR, thus enhanc- ing generalization. The key idea is to maintain equitable parameter influence on the overall generalization perfor- mance throughout the optimization process. We propose the following corollary for this purpose.
Theorem 3.1 (From Paper(Liu et al., 2020)). The relation-
ship between gradient updates and generalization can be expressed as follows:
E 2D [ofl JEJ eye E En Pggd
R(Z,n) = (( -1- Σ
(2) Tj n
Corollary 3.2 (Preconditioning and OSGR). If each pa- rameter j applies a preconditioner pj. the OSGR can be expressed as:
where J denotes the set of parameter index, 93 is the squared gradient magnitude, 03 the noise variance, and n the number of samples. Parameters with higher Gradient Signal-to-Noise Ratios (GSNR), defined as Tj _ yield oi
Σ
PrED~ [9
1
(3)
j'eJ Pj Dm 19
+
1
JEJ
mery
or equivalently:
higher OSGR, contributing more significantly to generaliza- tion.
Pj P.
-1 n2(
Σ jind Egerpy En-zo (7
A recent study (Michalkiewicz et al., 2023) leveraged GSNR to suppress overly predictive parameters during training, aiming to prioritize robust features and reduce noisy up- dates. However, this approach overlooks parameter-wise imbalances in OSGR, which limits overall generalization performance.
rr n
(4)
From Corollary 3.2, to maintain a balanced influence of parameter j the overall OSGR, we propose:
rj 1
In this context, we propose a preconditioning based ap- proach that dynamically balances OSGR across parameters. By incorporating parameter-specifet preconditioning fac- tors, our method ensures that updates are aligned with both gradient magniude and noise characteristics, preventing overfitting to noisy well-learned features. This strategy not only enhances generalization but also ensures stable convergence in diverse DG settings.
Pj E E
[
(5)
This leads to the OSGR:
R'(Z, n) 1 n JEJ Eyes (ry + +1 nBjeJ (r) + h)
(6)
where BjeJ (ry + + represents the average GSNR con- tribution across parameters. Without preconditioning, pa- rameters with large O but low GSNR may receive higher weights in the OSGR expression, inflating the subtraction term. Our preconditioning alleviates this issue and improves the OSGR. This dynamic adjustment with preconditioning mitigates parameter-we imbalances, ensuring that well- generalized features are not overwhelmed by noisy or overly dominant parameters.
3.2. Proposed Method
Based on Theorem 3.1, Michalkiewicz et al. (2023) intro- duced
approach that prioritizes updates for parameters with low GSNR, aiming to enhance their contribution to generalization. They argue that boosting updates to low-GSNR parameters can increase the overall GSNR and thus improve the optimization signal-to-gradient ratio (OSGR). Inspired by this perspective, we hypothesize the following relationship:
In implementation, we ignore the 1 term as n sufficiently large, and clipping variance by tanh( (222 for stability. More detailed analysis on influence of variance is described in Section 3.3.3. This preconditioner p. is straightforward to compute and requires only the gradient statistics m, and variance de, which can be estimated during training. This efficiency makes it suitable for a wide range of DG tasks.
π.
Conjecture Uniformly distributed OSGR across parameters indicate better generalization performance.
This conjecture guides the design of our method. Rather than modifying the dropout ratio across parameters, we in-
3

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
3.2.2. NOISE INJECTION
Algorithm Algorithm for GENIE
Input: Mini-batches {B, Hi Learning Rate a, Total
To enhance exploration during optimization, we introduce noise injection. where noise term scaled by the variance is added to the gradient. Specifically, the noise scale is deter- mined by tanh( 2= ), reducing noise for high variance parameters while increasing it for low variance parameters. Motivated by (Mansilla et al., 2021), this injection boosts updates to parameters with low preconditioning value.
Steps T.
Hyperparameters: B E [0, 1]. Dropout Probability p
Initialize: Parameters fus ma k- 0, vo +- 0. for to 7 do Compute Gradient:
9t VL(((;
3.2.3. RANDOM MASK
Update Moving Averages:
To further stabilize updates and mitigate overfitting, we ap- ply random dropoul mask. This mask, sampled from Bernoulli distribution, selectively zeroes out gradient com- ponents. By applying random masking after the precondi- tioning step, all parameters are equally considered to ensure robust updates.
my t 3mm-1 (1- (1-B)gr,
Ut + +1
Calculate GSNR and Preconditioning:
o Ut
m
Tj
taah(
@ m
3.3. Analysis
m4
1
90
1
g
Ut
We provide a comprehensive theoretical analysis of our method from three perspectives. First, we examine gen- eralization through the OSGR, which highlights how our effectively balances OSGR value across parameters. Second, we formalize approach under the PAC-Bayes framework, showing that our method explicitly minimizes tighter gen- cralization bound. Finally, we establish that our optimizer retains the convergence rate of standard SGD while en- abling more robust generalization. Proofs are provided in Appendix C.
Noise Injection:
Noise; + Ec tanh( 1
&e ~V(,00))
0,
Random Mask:
Mj ~ Bernoulli(p)
gr + (g. Noises) @ M
Update Parameters:
3.3.1. GENERALIZATION ANALYSIS WITH OSGR
lc+1 + 0: aggt
We obtain the following corollary regarding the OSGR of these optimizers:
end for Output: Final parameters 0r+1-
Corollary 3.3 (OSGR of Optimizers). The OSGR of our proposed optimizer is:
Rowss nEjeJ (ri + +)
thus contributing to faster convergence. It includes terms such as Ep-z" [g]] or Ep [9y]-
Comparing the resulting OSGR across different optimizers, we have:
Alignment Term: adjusts gradients toward stable di- rections. It includes the GSNR term ry.
Rours 2 RSGD Re RAdam-
(8)
Table summarizes the convergence term, alignment term and their resulting OSGR, including SGD, Adam, and our method.
This corollary demonstrates that our proposed precondition- ing achieves better generalization by attaining a higher over- all OSGR. The following remarks provide further context and analysis:
Remark 3.5 (Optimizer-Specific Analysis). SGD maintains baseline OSGR value with no explicit adjustment. Adam introduces a convergence component combined with par- tial alignment factor. In contrast, our method effectively integrates both aspects in balanced manner.
Remark 3.4 (Conceptual Components of Optimizers). The preconditioning applied by common optimizers can viewed the element-wise product of two conceptual com- ponents:
Overall, this analysis highlights how each optimizer's 's de- sign affects generalization through gradient alignment and
Convergence Term: controls the effective step size,

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
Table Comparison of optimizers with preconditioning split into Convergence and Alignment, and OSGR as separate column.Egy means weighted averaging with weight 2; W
OPT.
PRECONDITIONING
OSGR
WEIGHT
CONVERGENCE
ALIGNMENT
SGD
1-
1 Σ
W
W Σ, DDN nn
E Ep~
T3 +
JEJ
v B.nzzn [03]
ADAM
Σw,
W
W =
RD (gs)
+1
+1
Σ, /Benzn [831
m.yy
JEJ
1 n
GENIE
TJ +
W
W; = J
ED (g})
71
Bre + #
convergence gpeed.Incorporating both perspectives, Our method leads higher OSGR and thus improves gen- eralization performance. Furthermore, we demonstrate that the alignment term in our preconditioning achieves higher OSGR value than those of existing preconditioning methods. Detailed justifications are provided in the Appendix C.
update direction:
E E
Ve, Ve.KL(pp
Elg
Pj
19..
GENIE
which matches the preconditioning rule of our optimizer.
3.3.2. GENERALIZATION ANALYSIS WITH PAC-BAYES BOUND
Remark 3.7 (Sharpness and Generalization via KL). This re- sult shows that our method not only improves sharpness-ar done in SAM-but also directly enhances generalization by minimizing both terms in the PAC-Bayes bound. Specifi- cally, the variance adaptation factor qj reduces the variabil- ity of scaled gradients, thereby tightening the empirical loss term Ti through more stable updates. Simultaneously, the 5 term minimizes the KL divergence term T2. This re- sult shows our the generalization property of GENIE comes from correlation with Pac-Bayes theory.
While the previous analysis is based alignment and con- vergence dynamics using OSGR, we now adopt comple- mentary perspective grounded in the PAC-Bayes framework. We formulate the generalization analysis under one-step update setting, where the KL divergence between successive parameter distributions reveals the connection between our preconditioning and a tighter generalization bound.
Theorem 3.6 (PAC-Bayes Interpretation of Precondition- ing). R(0) is the population risk and ((0) is empirical risk. Assume that the loss function L(0) is bounded in [0, Cl. For any X O, with probability at least 1-8 over the draw of and for any data-dependent distribution over parameters 0, the following PAC- PAC-Bayes bound holds:
3.3.3. CONVERGENCE ANALYSIS
This section analyzes the convergence properties of GENIE under non-convex settings. Specifically, adopt three widely used assumptions in the optimization literature:
Assumption 3.8. (Bounded Gradient) There exists con- stant G > such that
AC
+ (L(pI(T) + log
Eg [R())] - Eg [())
| G for all
8n
Assumption 3.9. (L-smooth) The loss function L is L- smooth, meaning there exists a constant L > 0 such that for all 0,, 0z:
Assume that 5 N( Ex) and (Or,, 2nx ), where E/o}² be Σg diag(g; (3)) and Σx- diag(p?). Let g3 Elg?1 variance adaptation factor from SVAG aptimizer(Balie, & Hennig, 2018) that minimizes the variance to reduce max Ti term. (0.-1 = 8, - q ① g)
VV((0)) - < LI 02|1-
(10)
Assumption 3.10. (Lower bounded variance) The variance of the stochastic gradients have lower bound by a constant 1/S.:
Then, minimizing the T2 term via gradient descent yields an
Bllgg C(.)) 2 1/5u, VL.
(11)
5

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
Under these assumptions, we establish the following result regarding the convergence rate:
ing confidence intervals of all experiments are provided in Appendix E.
Theorem 3.11. Under Assumption 3.8 Assumption 3.9, and Assumption 3.10 the average gradient norm over 'T' itera- tions can be expressed as:
4.1. Comparison of Optimizers DG
Experiment Setup. We examined the impact of various optimization methods on generalization performance under domain shifts using Bascline ERM (Vapnik, 1999). The evaluated methods included: Standard optimizers (SGD (Robbins & Monro, 1951)), Adaptive optimizers (Adam (Kingma, 2014), Adam W (Loshchilov & Hutter, 2019), Ad- aBelief (Zhuang et al., 2020), AdaHessian (Yao et al., 2021), YOGI (Zaheer et al., 2018)), Sharpness-aware optimizers (SAM (Foret et al., 2021), GAM (Zhang al., 2023b), FAD (Zhang et al., 2023a)) and our proposed GENIE.
(1+ G. P
1/1
S
B[[ 0o
(12)
VT
where Pt is lower bound of preconditioning value.
Remark 3.12 (Convergence Rate and Intuition). Theo- rem 3.11 shows that the average gradient norm converges at O(T-1/)), the standard rate for stochastic gradient meth- ods in non-convex optimization. This implies that GENIE retains the fundamental convergence properties of SGD. Remark 3.13 (Influence of G-S, and S.). The term G-S? represents trade-off associated with the GSNR. A higher GSNR upper bound( G S,)) indicates a stronger gradient signal, which enhances generalization performance. How- ever, it also acts as multiplicative factor in the gradient norm, potentially slowing down convergence and thereby creating a trade-off. Furthermore, the variance term(S.)) has significant impact on the bound, further influencing the overall convergence behavior. To address this issue, we regulate the variance term using the tanh function, which effectively balances the interplay between generalization and convergence dynamics.
Results. As shown in Table 2, our optimizer achieved supe- rior performance across most datasets, surpassing existing methods. GENIE outperformed Adam, the default optimizer in most DG algorithms/Zhang et al., 2023a), by 5.69%. Ad- ditionally, it achieved improvements of 6.36% over SGD and 4.37% over SAM. In particular, it achieved remark- able performance VLCS, which is prone to early con- vergence and Ceefiitiin((Matuurt & Harada, 2020), and on Terralncognita, a wildlife image dataset with significant challenges such as lighting variations, motion blur, occlu- sions, and severe class MaaallnneeBBerry et al., 2018). These results suggest that GENIE effectively prevents overfitting and enhances the learning of causal relationships by bal- ancing parameter contributions during training. Optimizers designed for generalization, such SAM, GAM and FAD, outperform standard optimizers, underscoring the signifi- cant role of optimization in generalization. These results emphasize the need for developing optimizers specifically tailored for DG.
4. Experiment
Dataset. We followed the standardized protocols of Do- mainBed (Gulrajani & Lopez-Paz, 2021), which include dataset splits, hyperparameter searches, and model selection using validation sets. Our approach was evaluated on five DG benchmark datasets: PACS (Li et al., 2017), VLCS (Fang et al., 2013), OfficeHome (Venkateswara et al., 2017), Terralncognita (Beery et al., 2018), and DomainNet (Peng al., 2019).
Table 2. Comparison of optimizers on DG datascts. Results de- noted by * are reproduced from (Zhang et al., 2023a) using the same protocol our paper. The best results for each dataset are highlighted in bold.
Evaluation. In accordance with DomainBed protocols, models were trained for 15,000 iterations on DomainNet and 5,000 iterations on the other datasets. For all DG and SDG experiments, we employed the Training-domain Val- idation Set approach, partitioning the source domain into training and validation subsets. The optimal model was selected based on validation performance. We followed pre- vious DG methods by constructing 20 train-validation splits, with each split repeated 3 times.
OPT.
PACS
VLCS
OFFICE
TERRA
DOMAIN
AVG.
NET
ADAM
84.2
77.3
67.6
44.4
43.0
63.3
ADAMW
83.6
77.4
68.8
45.2
43.4
63.7
SGD*
79.9
78.1
68.5
44.9
43.2
62.9
YOGLS
81.2
77.6
68.3
45.4
43.5
63.2
ADABELIEF
84.6
78.4
68.0
45,2
43.5
63.9
ADAHESSIANS
84.5
78.6
68.4
44.4
44.4
64.1
SAM"
85.3
78.2
68.0
45.7
43.4
64.1
GAM
86.
78.5
68.2
45.2
43.8
64.4
FAD=
88.2
78.9
69.2
45.7
44.4
65.3
GENIE
87.8
80.7
69.7
52.0
44.1
66.9
We used ResNet-50 (He et al., 201 6b) pre-trained on ImageNet (He et al., 2016a) as back- bone architectures. Detailed implementation details are pre- sented in Appendix D. The detailed results and correspond-
Implementation Details.
Experiment Setup. The computational overhead of an optimizer is a critical factor in its practical applicability. To evaluate this, we trained models on the PACS and VLCCS
6

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
datasets for 5,000, 10,000, and 15,000 iterations, measuring average performance and training time per iteration.
4.3. Single Domain Generalization
Experiment Setup. We evaluated performance in Single Domain Generalization (SDG), which is more constrained but better reflects real-world applications. The flexibilityy to operate in SDG without structural modifications is an advantage of our method over certain existing methods that are limited to multi-source settings, In SDG, the model is trained and validated on a single domain and tested on the others, with results averaged across all source domains. We compared GENIE with Adam, SGD, and SAM, and applied it to existing DG methods.
Results, As reported in Table 3, GENIE consistently outper- formed other optimizers, even at 5,000 iterations, while in- curring lower computational overhead than SGD and Adam. Additionally, GENIE achieved an average of 1.3x faster training compared to SAM, as SAM's update rule requires two sequential moon-aaallleizzalee) gradient computations per step, which doubles the training time. These results perimentally validate the theoretical convergence analysis in Section 3.3.3, confirming GENIE's ability in computational efficiency and convergence speed.
Results. The SDG performance results are presented in Table 4. As in previous DG settings, our optimizer out- performed existing optimizers. When applied to DG meth- ods, conventional optimizers reduced performance, whereas GENIE achieved the highest performance as a standalone model and also improved DG methods when used as an optimizer. These results show that our method enhances DG performance without requiring architectural modifications or multiple source domains, and performs well even as a standalone method.
Table Training time (sec) and average accuracy different iter-
ation levels.
OPT.
ITER.
TRAINING
AVG.
TIME
PACS
VLCS
OFFICE HOMB
(18)
SGD
5000
5,273
69.8
76.7
51.1
10,546
62.5
15000
15,783
75.8
77.7
63.9
ADAM
5000
5,443
84.2
63.6
Table 4. Experimental results GENIE under the SDG setting.
10000
10,934
86,1
77
65.2
16,531
65.2
SAM
5000
5,775
82.4
79.4
69.4
1 10000
11,500
83.5
80.3
69.6
ALGORITHM
PACS
VLCS
OFFICE
TERRA
AVG.
15000
17,191
84.1
80.4
70
GENIE
5000
4,292
88.4
81.3
70
SGD
49.5
60.4
45.9
22.8
44.7
(OURS)
10000
8,582
69.2
SAM
57.7
66.7
59.2
26.8
52.6
15000
12.876
86.9
81.3
69.1
GENIE (OURS)
69.5
69.9
58.6
36.0
58.5
RSC+ADAM
35.5
RSC+SGD
22.2
39.8
1.7
17.6
20,3
RSC-GRNIE(OURS)
68.7
54.4
33.2
56.1
4.2. Integration with Current DG Algorithms
CORAL+ADAM
64.3
56.2
50.7
33.5
51,2
CORAL+SGD
49.5
60.4
45.9
22.8
44.7
CORAL+GENIE(OORRS
70.9
69.2
$6.4
36.7
58.3
Experiment Setup. GENIE is versatile optimizer that integrates seamlessly with various DG algorithms without requiring changes to the training procedure or model archi- tecture. To validate its compatibility, we combined GENIE with several well well-perfore perfformrng DG algorithms- CORAL(Sun & Saenko, 2016) and RSC((uang al., 2020) using ResNet- 50 as the backbone- and compared its performance against other optimization techniques.
4.4. Model Analysis
Ablation. We conducted an ablation study using the PACS dataset in DG setting to evaluate the effects of Precon- ditioning, Noise Injection, and Random Mask (Table 6). The version without all three components corresponds to ERM trained with Adam, while the version incorporating all three represents our proposed GENIE optimizer. Ex- perimental results show that GENIE achieved the highest performance, improving accuracy by 4.9% compared to ERM. Even when using only Preconditioning, performance improved by 3.8%, indicating that simple preconditioning technique enhance generalization. Additionally, in the Cartoon and Sketch domains, where objects are placed on white background, models trained with Noise Injection and Random Mask performed better. Here, we conclude that preconditioning alone is enough for DG, but you can optionally utilize Noise Injection and Random Mask for
Results. The performance evaluation results for DG are summarized in Table 5. GENIE consistently outperforms existing optimization methods, demonstrating robustness and broad applicability. These results validate GENIE's scalability and compatibility with various DG algorithms. Unlike other DG methods, which often require multiple source domains or architecture modifications, GENIE seam- lessly integrates with existing training pipelines, providing consistent performance gains without additional complexity. This establishes GENIE as an algorith--annostic and highly adaptable optimization framework for DG tasks.

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
Table Integration with DG methods, Results obtained from the original literature and DomainBed (Gulrajani & Lopez-Paz, 2021) denoted with f. while results taken from (Zhang et al., 2023a) are denoted with *
ALGORITHM
PACS
VLCS
OFFICEHOME
TERRAINC
AVG.
ERMF(VAPNIK, 1999)
85.5
77.5
66.5
46.1
68.9
IRMP(ARJOVKYY ET AL., 2019)
83.5
78.6
64.3
47.6
68.5
GROUPDROT(SAGAWA AL.,, 2020)
84.4
76.7
66.0
43.2
67.6
I-MIXUPI(XU ET AL., 2020)
84.6
77.4
68.1
47.9
69.5
MLDG*(LI AL.., 2018A)
84.9
77.2
66.8
47.8
69.2
MMD+(LI ET AL., 2018B)
84.7
77.5
66.4
42.2
67.7
DANNI AL., 2016)
83.7
78.6
65.9
46.7
68.7
CDANN ET AL., 2018C)
82.6
77.5
65.7
45.8
67.9
MTLi(BLANCHARD AL.. 2021)
84.6
77.2
45.6
68.5
SAGNETYNNMM ET AI.,, 2021)
86.3
77.8
68.1
48.6
70.2
AZMA(ZHANG ET AI.,, 2021)
85.1
77.6
64.8
45.5
68.3
(KEEEER ET AL., 2021)
84.9
78.3
66.4
46.4
69
MIXSTYLE" (ZHOU ET AL., 2021)
85.2
77.9
60.4
66,9
MIRO" ET AL., 2022)
85.4
78.9
69.5
45.4
69.8
GENIE (OURS)
87.8
80.7
69.7
52.0
72.6
RSC(HUANG ET AL., 2020)+ADAM*
84.5
77.9
65.7
44.5
68.2 2
RSC+ADAMW*
83.4
77.5
66.3
45.1
68.1
RSC+SGD#
82.6
78.1
67
43.9
67.9
(SC+GENIE
87.3
80.6
68.1
49.5
71.4
CORAL(SUN & SAENKO, 2016) + ADAM
86
78.9
68.7
69.3
CORAL+ADAMWs
86.4
79.5
69.8
45,0
70.2
CORAL+SGD*
85.6
78.2
69.5
69.8
CORAI-GENTE(OUR))
87.9
80.7
70.6
48.4
71.9
additional robustness.
Table Ablation study on PACS dataset. Results are reported for evaluations four domains: Art, Cartoon, Photo, and Sketch.
Accuracy
Accuracy
:22
20
PRE
NOISE
MASK
PACS
AvG.
18
40
CONDITION
C
0.0
0.2
0.95
0.59
0.995
0999
Drapout Probatillty(P)
Moving Avg(B)
88.0
79.7
96.7
72.7
84.2
89.5
82.3
98.4
79.4
87.4
GEMEleun
X
85.4
77.4
98.6
78.7
85.0
Figure 2. Performance sensitivity of GENIE dropout probability P and coefficient B.
88.4
Sensitivity Analysis. GENIE employs two key hyperpa- rameters: the dropout probability P and the cocfficient B which is used to compute the moving average and variance of gradients. To analyze sensitivity of these hyperparam- eters, we conducted grid search while keeping all other training settings fixed. As shown Figure GENIE consis- tently outperformed SGD, Adam, and SAM across wide range of Pand B values, demonstrating strong robustness to hyperparameter variation. Notably, while this experi- ment involved hyperparameter tuning via grid search, all other experiments followed the DomainBed protocol, using validation performance for hyperparameter selection.
OSGR of Network Parameters Over Time. To assess whether our approach enhances the overall OSGR of net- work parameters during training, we tracked the average OSGR of all parameters throughout the training process. As shown in Figure 4, the OSGR measurements on the VLCS dataset show that GENIE achieves an OSGR closer to I than prior optimizers. This means superior generalization performance. These findings align with the theoretical Gen- eralization analysis in Section 3.3.1. confirming that GENIE ensures more stable and balanced parameter updates during training, which ultimately leads improved generalization.
8

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
(A) Before Training
(B) SGD
(C) Adam
(D) GENIE
Figure 3. UMAP visualization of learned features on the PACS dataset with Sketch as the unseen target domain. (A) Before training.
(B)-D) After training with SGD, Adam, and GENIE.
0.430
CTNE
6.45-
GENE
SCD
SCD
Adam
Adam
COME
6.990
0.015
0.010
0.022
0.900
6440
0.835
1000
1960 32083 14030
1.20 20.00 30003 J.000 19000
Iteration
Figure 4. OSGR measurements over training iterations for different optimizers on the VLCS dataset. The OSGR values were calculated for all iterations and averaged every 200 iterations for clarity and case of comparison.
Figure 5. Optimization trajectories on a simulated loss landscape.
tively reducing sharpness, thereby improving generaliza- tion(Foret et al., 2021).
Interestingly, while SAM is designed for better generaliza- tion performance, it exhibits inferior OSGR values. This suggests that the sharpness-awe aware regime alone is insuffi cient for generalization, and that the OSGR regime should also be considered when addressing generalization in DG tasks. This observation is consistent with our PAC- ACC-Bayesian analysis in Section 3.3.2, which reveals that inducing bal- anced OSGR values leads to tighter generalization bounds, reinforcing the role of OSGR as a necessary complement to sharpness-awee aware optimization.
Feature Visualization. To examine how the GENIE opti- mizer operates at the feature level, we performed UMAP visualizationssMelnnes al., 2018) on the PACS dataset, with the Sketch domain held out the unseen target. Each color represents different class. The results Figure 3 show that GENIE leads to clear class separation across domains, suggesting effective domain-invariant feature learning.
5. Conclusion
Loss Landscape. We analyzed the convergence paths of SGD, Adam, and GENIE in the loss landscape using the FashionMNIST dataset(Xiao et al., 2017). As shown in Fig- ure 5, each corner represents the local minima of a specific source domain. All optimizers started at (-1,3) and were updated for 30 steps under the same conditions. SGD and Adam follow steep direction and converge quickly. How- ever, fast convergence often causes overfitting to specific source domains in OOD scenarios. Generalizable features are learned later in traini et al., 2019; Shah et al., 2020b; Nakkiran et al., 2019), so rapid convergence can pre- vent the model from acquiring them sufficiently. In contrast, as demonstrated in the theoretical analysis in Section 3.3.2, GENIE leads optimization toward flatter minima by effec-
We introduce GENIE, an optimizer that leverages OSGR guide gradients in effective directions, preventing overly pre- dictive parameters from dominating while ensuring all pa- rameters contribute equitably to learning. GENIE achieves a higher OSGR with improved generalization and ensures fast convergence rate comparable SGD, Empirically, it out- performs slate-ofttheuu optimizers across five DG bench- marks, demonstrating robust performance under significant domain shifts and limited data. Seamlessly integrating with existing DG and SDG methods, GENIE consistently achieves performance improvements. This work highlights the potential of OSGR as a guiding principle, paving the way for its use few-shot learning, meta-learning, and other tasks requiring solutions to source-domain overfitting.

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
Acknowledgements
images for softening bias. In Proceedings of the IEEE International Conference Computer Vision, pp. 1657- 1664, 2013.
This work was supported by Korea Internet & Secu- rity Agency(KISA) grant funded by the Korea govern- ment(PIPC) NN..RS-2023-0033 12000000, Development of per- sonal video information privacy protection technology capa- ble of Al learning in an autonomous driving environment)
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-aware minimization for efficiently improving generalization. International Conference on Learning Representations, 2021.
Impact Statement
Ganin, Y. Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F,, March, M., and Lempitsky, V. Domain- adversarial training of neural networks. Journal of ma- chine learning research, 17(59):1-33 2016.
Our work proposes GENIE, an optimization method that enhances domain generalization by ensuring stable and bal- anced updates. GENIE mitigates overfitting, promotes flat- ter minima, and improves OOD performance, contributing to more robust and generalizable models.
Gulrajani, I. and Lopez-Paz, D. In search of lost domain generalization. International Conference on Learning Represemtations, 2021.
References
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016a.
Amari, S.-i., Ba, J., Grosse, R. B., Li, X., Nitanda, A., Suzuki, T., Wu, D., and J. J. When does preconditioning help or hurt generalization? In International Conference on Learning Representations, 2021.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016b.
Arjovsky, M., Bottou, L., Guirajani, I., and Lopez- Paz, D. Invariant risk minimization. arXiv preprint arXiv: (0009990970999 2019.
Balles, L. and Hennig, Dissecting adam: The sign, magni- tude and variance of stochastic gradients. International Conference on Machine Learning, pp. 404 413. PMLR, 2018.
Hemati, S., Zhang, G., Estiri, A., and Chen, X. Under- standing hessian alignment for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19004-19999 19014, 2023.
Beery, S., Van Horn, G., and Perona, P. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pp. 473, 2018.
Hinton, G., Srivastava, N., and Swersky, K. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Cited on, 14(8):2, 2012.
Blanchard, G., Deshmukh, A. A., Dogan, U., Lee, G., and Scott, C. Domain generalization by marginal transfer learning. Journal of machine learning research, 22(2): 1-55, 2021.
Huang, Z., Wang, H., Xing, P., and Huang, D. Self- challenging improves cross-domain generalization. In Vedaldi, A., Bischof, H., Brox, T., and Frahm, J.-M. (eds.), Computer Vision ECCV 2020, pp. 124-140, Cham, 2020. Springer International Publishing.
Cha, J., Lee, K., Park, S., and Chun, S. Domain gener- alization by mutual-iiffrrmaioon regularization with pre- trained models. In Avidan, Brostow, G., Cissé, M., Farinella, G. M., and Hassner, T. (eds.), Computer Vision ECCV 2022, pp. 440-457, Cham, 2022. Springer Nature Switzerland.
loffe, S. and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448- 456. pmlr, 2015.
Dennis, Jr. J. and Moré, J. J. Quasi-newton methods, motivation and theory. SIAM review, 1911):6-89, 1977.
Kingma, D. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6980. 2014.
Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Koyama, M. and Yamaguchi, S. When is invariance useful in an out-of-distrrbution generalization problem? arXiv preprint arXiv:2008.00888 2020.
Fang, C., Xu, Y., and Rockmore, D. N. Unbiased metric learning: On the utilization of multiple datasets and web
Krueger, D., Caballero, E., Jacobsen, J.H.., Zhang, A., Binas, J. Zhang, D., Priol, R. L.,, and Courville,
10

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
Out-of-distribution generalization via risk extrapola- tion (rex). In Meila, M. and Zhang, T. (eds.), Pro- ceedings of the 38th International Conference chine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 5815-5826. PMLR, 18-24 Jul 2021. URL htwps: http:////ooccddggeeeeeee www.m....p... ptrss/ v139 htt////reeeeeaaa.. htm.lml..........m.mmmmmmm.m....
Michalkiewicz, M., Faraki, M., Yu, X., Chandraker, M., and Baktashmotlagh, M. Domain generalization guided by gradient signal to noise ratio of parameters. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6177-6188, 2023.
Ma-
Montavon, G., Orr, G., and Müller, K.-R. Neural networks: tricks of the trade, volume 7700. springer, 2012.
Li, D., Yang, Y., Song, Y.ZZ., and Hospedales, T. M. Deeper, broader and artier domain generalization. In Proceedings of the IEEE international conference on computer vision, pp. 5542-5550, 2017.
Muandet, K., Balduzzi, D., and Schölkopf, B. Domain generalization via invariant feature representation. In International conference on machine learning, pp. 10-18. PMLR, 2013.
Li, D., Yang, Y., Song, Y.-Z., and Hospedales, T. Learning to generalize: Meta-learning for domain generalization. In Proceedings of the AAAI conference on artificial intel- ligence, volume 32, 2018a.
Nakkiran, P., Kaplun, G., Kalimeris, D., Yang, T., Edelman, B. L.., Zhang, F., and Barak, B. Sgd on neural networks learns functions of increasing complexity. In Proceedings of the 33rd International Conference on Neural Informa- tion Processing Systems, pp. 3496-3506, 2019.
Li, H., Pan, S. J., Wang. S., and Kot, A. C. Domain general- ization with adversarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5400-5409 2018b.
Nam, H., Lee, H., Park, J., Yoon, W., and Yoo, D. Reducing domain gap by reducing style bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8690-8699, June 2021.
Li, Y., Gong, M., Tian, X., Liu, T., and Tao, D. Domain generalization via conditional invariant representations. In Proceedings of the AAAI conference on artificial intel- ligence, volume 32, 2018c.
Parascandolo, G., Neitz, A., Orvieto, A., Gresele, L., and Schölkopf, B. Learning explanations that are hard to vary. In International Conference Learning Representations, 2021.
Liu, J., Bai, Y., Jiang, G., Chen, T. and Wang, H. Understanding why neural networks generalize well through gsnr of parameters. In 8th International Con- ference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https://////.... ooooreeevvvv ew, n forum? Hd-HyevIJStwH.
Peng, X., Bai, Q. Xia, X., Huang, Z., Saenko, K., and Wang, B. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1406-1415, 2019.
Pérez, G. V., Louis, A. A., and Camargo, C. C. QeDepp
Loshchilov, and Hutter, F. Decoupled weight decay reg- ularization. In International Conference on Learning Representations, 2019.
learning generalizes because the parameter-functin map is biased towards simple functions. In 7th International Conference Learning Representations, ICLR 2019, 2019.
Mahajan, D.,
Tople, S., and Sharma, A. Domain generaliza- tion using causal matching. In International conference on machine learning, pp. 7313-7324. PMLR, 2021.
Rame, A., Dancette, C., and Cord, M. Fishr: Invari- ant gradient variances for out distriibbnn generaliza- tion. In Chaudhuri, K., Jegelka, S., Song, L., Szepes- vari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learn- ing, volume 162 of Proceedings of Machine Learn- ing Research, pp- 18347-18377. PMIR, 17-23 Jul 2022. URL https: http:/ppoeeenng..cee/ mlr ..r.ss/ v1 62 / 22a. html.
Mansilla, L., Echeveste, R., Milone, D. H., and Ferrante, E. Domain generalization via gradient surgery. In Pro- ceedings of the IEEE/CVF international conference on computer vision, pp. 6630-6638, 6638, 2021.
Matsuura, T. and Harada, T. Domain generalization using
mixture of multiple latent domains. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 11749-11755 1156, 2020.
Robbins, H. and Monro, S. A Stochastic Approxima- tion Method. The Annals of Mathematical Statis- tics, 22(3):400 - 407, 1951. doi: 10.1214/aoms/ 1177729586. URL wttp:////oo...g/...12222222 a aoms coms/117772958
McInnes, L., Healy, J., and Melville, J. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint / 2018.

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
Sagawa, S., Koh, P. W. Hashimoto, T. B., and Liang. P. Distributionally robust neural networks. In International Conference on Learning Representations, 2020.
Yan, S., Song. H., Li, N., Zou, L.,, and Ren, L. Improve un- supervised domain adaptation with mixup training. arXiv preprint arXiv:2001.0000 2020.
Shah, H., Tamuly, K.,, Raghumathan, A., Jain, P., and Netra- palli, P. The pitfalls of simplicity bias in neural networks. Advances in Neural Information Processing Systems, 33: 9573-9585, 2020a.
Yao, Z., Gholami, A., Shen, S., Mustafa, M., Keutzer, K., and Mahoney, M. Adahessian: An adaptive second order optimizer for machine learning. In proceedings of the AAAI conference artificial intelligence, volume 35, pp. 2021.
Shah, H., Tamuly, K., Raghunathan, A., Jain, P., and Netra- palli, P. The pitfalls of simplicity hias in neural networks. Advances in Neural Information Processing Systems, 33: 9783-9585,220 2020b.
Ye, Q. Preconditioning for accelerated gradient de- scent optimization and regularization. arXiv preprint arXiv:2410.00232, 2024.
Shahtalebi, S., Gagnon-Audet, J.-C., Laleh, T., Faramarzi, M., Ahuja, K., and Rish, I. Sand-mask: An enhanced gradient masking strategy for the discovery of invari-
Ye, W., Zheng, G., Cao, X., Ma, Y., and Zhang, A. Spurious correlations in machine learning: A survey, arXiv preprint urXiv:2402.12715. 2024.
ances in domain generalization, 2021. URL https: //arxiv. org/ abs/ 21066022665
Zaheer, M., Reddi, S., Sachan, D., Kale, S., and Kumar, S. Adaptive methods for nonconvex optimization. Advances in neural information processing systems, 31, 2018.
Shi, Y., Seely, J., Torr, P., Siddharth, N., Hannun, A.,
Usunier, N., and Synnaeve, G. Gradient matching for domain generalization. In International Conference on Learning Representations, 2022.
Zhang, M., Marklund, H., Dhawan, N., Gupta, A., Levine, S., and Finn, C. Adaptive risk minimization: learning to adapt to domain shift. In Proceedings of the 35th Inter- national Conference on Neural Information Processing Systems, pp. 23664-23678, 2021.
Sun, B. and Saenko, K. Deep coral: Correlation alignment for deep domain adaptation. In Computer Vision-ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14, pp. 443- 450. Springer, 2016.
Zhang, R., Fan, Z., Yao, J., Zhang, Y., and Wang, Y. Domain- inspired sharpness- aware minimization under domain shifts. In The Twelfth International Conference on Learn- ing Representations, 2024.
Vapnik, V. N. An overview of statistical learning theory. IEEE transactions neural networks, 10(5):988-99)) 1999.
Zhang, X., Xu, R., Yu, H., Dong, Y., Tian, P., and Cui, P. Flatness- aware minimization for domain generalization. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision (ICCV), pp. 5189-5202, Octo- ber 2023a.
Venkateswara, H., Eusebio, J., Chakraborty, S., and Pan- chanathan, S. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE con- ference on computer vision and pallern recognition, pp. 5018-5027, 2017
Zhang. X., Xu, R., Yu, H., Zou, H., and Cui, P. Gradient norm aware minimization seeks first-order flatness and improves generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20247-20257, 2023b.
Wang, P., Zhang, Z., Lei, Z., and Zhang, L. Sharpness- aware gradient matching for domain generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 3769-3778, 2023.
Wang, Y., Li, H., and Kot, A. C. Heterogencous domain generalization via domain mixup. In ICASSP 2020-2020
Zhou, K., Yang, Y., Qiao, Y., and Xiang, T. Domain gener- alization with mixstyle. In International Conference on Learning Representations, 2021.
IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1622-3626 IEEE, 2020.
Zhuang, J., Tang, T., Ding, Y., Tatikonda, S. C.,, Dvomek, N., Papademetris, X., and Duncan, J. Adabelief optimizer: Adapting stepsizes by the belief in observed gradients. Advances neural information processing systems, 33: 18795-18806, 2020.
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms, 2017.
Xu, M.. Zhang, J.. Ni, Li, T., Wang, C., Tian, Q., and Zhang, W. Adversarial domain adaptation with domain mixup. In Proceedings of the AAAI conference on artifi- cial intelligence, volume 34, pp.
2020.
12

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
A. Notation
Table 7. Final Revised Notation Table
Symbol
Description
neural network
L(0)
loss function
data distribution defined over A' x y
number of data samples
D'
training/test dataset drawn from model parameters, parameter j
0,0,
O..
parameter index j optimization step
gradient of parameter j averaged over training set gradient at step t
squared gradient for parameter
P3
variance of parameter j's gradient
of
variance gradient averaged over training set
ry
gradient signal-to-noise ratio (GSNR), ry
proposed preconditioning factor for parameter j
R(Z,n)
one-step generalization ratio (OSGR)
& & (~((00,0²))
Gaussian noise for noise injection
of parameter index bound of gradient la norm
G
1/5.
lower bound gradient variance Lipschitz constant
P.
lower bound of preconditioning value weighting factor showing in optimizers, probability measure of postcrior and prior covariance matrix Gaussian distribution random error terms in gradients KL divergence between distributions
3D (ob.))
W3
E, Bowz" (o in SGD
p. π
Ep, En
e,c'
(L())(
B. Details of Table
We start out from reinterpretation of the widely-used ADAM optimizer, which maintains moving averages of stochastic gradients and their element-wise square,
m7
m, = 31m4-1 + (1-111))) 1 g
me
(13)
ū B20 (1- B2)g
(14)
3 1
with B1,B2 (0, 1) and updates with learning rate a,
m
0,+1
0,
(15)
v 0x + E
13

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
with small constant preventing division by zero. Ignoring and assuming |me,s the update direction
for the moment, we rewrite
(
sign(m)
(16)
11
m
Vy
Ti
Here, divide the preconditioning into two terms. The convergnece T2 which modulates update size is
En (g,)
sign(m::)=
(17)
2~ (gi)
The alignment term T which includes GSNR is
(18)
Re Pe-m
The Equation (18) can be justified by definition of GSNR using Equation (19) and Equation (20). The variance of gradient average is:
02
mn
(19)
and
m 9j -
(20)
C. Proof of Theorems
C.1. Convergence Analysis
We provide the detailed derivation and proof for Theorem 3.11. From Assumption 3.9, we start with:
L
L(0++1) S +(..))(VVLI)) Oct1 - 02) +
10.41 - 0.11.
(21)
2
T2
where the first term, Ti, is given by:
Ti = (VL(0)), -0))-
(22)
Using our preconditioning, which we defined as:
g²
n
(23)
g²
0
6
p
g
which satysifies, given Assumption 3.10:
1,
<
Sn
(24)
1
9"
For T2, we have:
L x T2
L
gell
<
(25)
2
2
and its expectation satisfies, using Assumption 3.8:
L
E
E
<
(26)
2
14

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
For T2, decompose:
T1 (VI(Oe), 0c+1 -0+)
gt),
(27)
19t.1 [(sign[[VL sign(g.)))
Ti S (L gi) + Ae Σ [[V(())1
(28)
Pi.
73
74
Now, considering 33. we evaluate its expectation:
T3 - Xe -VV()))))))
(29)
Pi is lower bound of our preconditioning value. Considering T₃:
B[Z+] Ae Σ
E
[VL(0)
[[sign[[VL(O.)]>>> + sign[o))))
(30)
P
Thus, we obtain:
Loc Pi
B[TA| = A Σ B
[[V(()))ss
P(sign[[VLL0.)L] # sign(on.)))
(31)
Next, we analyze the probability term:
(V(0)))) sign(gr.)]
(32)
and bound as follows:
P(sig)(VL((.)), * signler.)) s P([[L(())), 9n 2 (V)L).
(33)
Using Chebyshev's inequality:
Var (V)))ll), -9t.)
a²
p//n
(([[V)(0)), - gcsl 2 [VL(()))))
(34)
[VL()))),]
[VL()))) 2
n is the number of gradient samples.
Replacing the n to step T, we bound the expectation:
99.5 P
pTT
| T
T < A
> E
[VL())
(35)
[V(())))]
[J] is the number of parameters indicated by the size parameter index set. Now, summing the inequalities until step T:
(1](00++)) - P A | + X.S..O
I
(36)
Rearranging:
[[L(O++1) < L(0b) - X.P, Σ |L())) +T >
(IJ
L
d s
(37)
T
This results in:
T
VL(O.)) < L(O)) ((L(1
J
+
X
S2
(38)
A.P: T
P.
T
2
VL.00)1 < L(O)) [(L()
J
4
x2
+
(39)
AeP. T
P
T
2
15

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
Taking T + oo, the convergence rate is:
BI]VL())) L(00) -(0))-(00, L(0.)
P + T +
++
(40)
P
From the final steps of our derivation, analyze the convergence rate of the algorithm.
Taking Ar as:
L(06) 1(06)- L(0.)
AT
(41)
T
we have:
1
((0)) - L(0.)
G- St
L(0b) - L(0.)
|J
L[ <
(42)
Pj
T
2 P
T
P1 T
Denoting
V as:
1
L((0) L(O.).2
(43)
VT
T
Finally, we rewrite the bound, concluding that:
(
G
S2
||| [[]))(()))) 50
1
(44)
Pt
2
T
C.2. Proof of Corollary 3.2
We utilized (Liu et al., 2020) for proving Corollary 3.2. In one gradient descent step, the model parameter is updated by A6 l 0₂ -Xp gn (f), where À is the learning rate and is preconditioning If X is small enough, the one-step training and test loss decrease can be approximated by:
OL(D]
AL[D] ~ -
+ 0(22) = Ap (000)-9n(0))) (0(0)+000)))
(45)
00
aL|D'
ALW' ~ A
+ 0(A²) = o gD (0 gb (0)
(46)
00
Usually, there are some differences between the directions of gD (0) and gn (0), so statistically AL tends to be larger than AL[D", and the generalization gap would increase during training. When + 0. one single training step, the empirical generalization gap increases by [ [AD''. For simplicity, denote this quantity as:
V ALDD ALDD") ~ (g) gD) (0) (D),
(47)
which can be further simplified as:
= A(p o x()p - -o0
(48)
-(ā(0) + ()(e-e)) e').
(49)
Here, we replaced the random variables by (0) = ī(0) + and OD (0) = (() + c', where e and e' are random variables with zero mean and variance (²).. Since EEle'] E[e] and e' are independent. The expectation of is:
Ep. (V) = E(Ap @ e e') + +00) X Σ ps -(0)
(50)
where d0,,, is the variance of the average gradient of the parameter 0j. For simplicity, when it involves a single model parameter 0j. we will use only subscript j instead of the full notation. For example, o3, Tý. and gn denote Q0,)), r(,),, and (0,)), respectively.
16

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
Expectation Analysis
Consider the expectation of AL[D] and AL[D'] when À 0:
B==zz((L(D) x Ap gn(()) = E
pj PD
(51)
]
Ep, (0. = Ep.
V) ~ 1 Σ Pj ( D (gb.) -o})
(52)
which simplifies further as:
(DLD"))AA((D)) = x Σ Pj 2 - p2//n).
(53)
Simplification of R(Z,n) n)
Substituting Equation (53) and Equation (51) into R(2,n) n), we have:
E,
R(2,n) =1 n En
(54)
Ep-zn ( We can rewrite Equation (53) as:
When r3
Pj D,ED (gD
1
R(2,n) =1
(55)
n
Eÿ' Py DT 2- 19
D3 + 1'
or equivalently:
Dj DD 2" (973z)
R(2,n) = Σ
(56)
Ex Pj P,DED ( 1 [+++,
3
C.3. Generalization Analysis
C.3.1. PAC BAYES BOUND AND PRECONDITIONING
Theorem C.1 (PAC-Bayes bound). Let D ~ 2n be dataset sampled i.i.d. from a data distribution Z.R(O) is the population risk and L(0) is empirical risk. Assume that the loss function L(0) is bounded in [0, C) for some constant C > 0. For any > 0, with probability at least - over the draw of D, and any data datta-dppenden distribution P over parameters the following PAC-Bayes bound holds:
X
(L( + log
E R(0) S En [(()
8n
Ti
T2
Here, R(0) denotes the expected loss over the true data distribution. The SAM algorithm primarily focuses on minimizing the T; term Theorem C.I, following the incquality:
(D) E (co(te)) < max (0 +e)]
(57)
Iellasp
In contrast, our method simultaneously minimizes both the Ti and T2 terms. First, we determine a preconditioning vector q to reduce [LL)) To minimize variance-ing crrianee-nnuued error, we adopt the variance adaptation factor introduced in the SVAG optimizer(Balles & Hennig, 2018), and solve:
[lq Og E
Σ Elgii 24, Egg Elg
(58)
17

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
Minimizing this expression yields the optimal preconditioning:
|
=
L[9]]
Next, assume (00+11, Ez) and π = (o,,, E,), with both covariances defined as:
Ep diag(q1 (1 pl.92 P2. .2.....qq.J Mn1 PÍu)). En = diag(p = P(u)).
Here, the prior can be treated data-driven prior which is approximated with stochastic gradient descent using all data excluding the current mini-batch. Assuming the variances do not significantly change between steps. Then the KL divergence can be written follows:
(0.+1.s - 02,i2
p q:
KL x)
Σ
|
+
log
(59)
Pi
P
p
SEJ
iEJ
The gradient of this KL with respect to 0g is:
Ve, KL(llT)) = (0t+1s - 0rg)
E[g,1
Oit
E
Dj Elgj
E
Elg
19
(60)
GENIE
This shows that minimizing the KI. divergence in the PAC-Bayes bound via gradient descent naturally leads to the same preconditioning structure. Therefore, our method considers both sharpness (through variance adaptation) and generalization (via KL divergence minimization), whereas SAM focuses solely on sharpness.
important note that our gradient computation is taken with respect the prior mean 0r, rather than the posterior mean 8441- Due to the asymmetric nature of the forward KL divergence (L(pll this choice induces mode-covering behavior rather than mode-seeking. As optimization proceeds, the prior distribution is iteratively adapted to cover broader region of the risk landscape, effectively reducing over-comcentration around a single mode. This mechanism aligns with the argument in Risk Extrapolation (Krueger et al., 2021), where covering a wider set of hypotheses can improve cutt-f--distibutionn generalization. Consequently, although the PAC- PAC-Bayes bound is originally derived under an i..dd assumption, this mode-covering property enables the prior to capture more diverse risk regions, thereby enhancing robustness under distribution shift.
C.3.2. OSGR BASED ANALYSIS
From Section 3.2.1, the OSGR of our method is given by:
Rours = 1 - Σ E, (ry
(61)
nEju (ry + !)
Similarly, from Theorem 3.1, the OSGR SGD is given by:
Regd 1 1 Σ En zm [ n
1
(62)
E, E En 2" [9
rj + n
Ep-za zm |g
Replacing the term E, E, B [3y11
WW,
which represents a weighted average, we rewrite as:
Ragi
Σ
W
(63)
7
+ n
JEJ
18

---

One-Step Gencralization Ratio Guided Optimization for Domain Generalization
If we assume uniform weight W + by Jensen's inequality:
1 nEjes (ry + h)
0 M
W
ri + S
S 1
(64)
JEJ
Thus, we conclude:
S Regd S Roars S
(65)
In the same way, with any preconditioning,
R(2,n) =
Σ
Pj D-
1
(66)
Eg Pj D (9 +
n
Replacing the term E,
Pj Pr (g Σx P3 Bn (
to W with W3 1 1
which represents average, rewrite as:
Rarecondition
Σ jEJ
W,
(67)
+ I
11
Also by Jensen's inequality, we obtain:
< Rprecondition
S Rours S
(68)
Consequently, this result establishes that our method achieves the highest OSGR value among preconditioning methods such as Adam, RMSprop, and SVAG(Balles & Hennig, 2018).
However, the assumption of uniform weights W, |J can be overly restrictive. In practice, preconditioning methods aim to balance the contribution of parameter updates, which becomes especially important when there exists a strong imbalance in the GSNR distribution. We consider the case where a dominant coordinate exists, denoted as Jmax = arg maXjeJ Tj and define the remaining coordinates as = \ {imax], with Jmax arg maxjeJ' Tys such that T' Jmss
To demonstrate that our method still yields a higher OSGR under such imbalance, consider the difference:
nl (Ppurs Rpreccndition)
=
=Σ
W
1
(69)
+
EjeJ (r,+ + 4
JEJ
For sufficiently large n, the terms can be neglected:
1
Roors
Rprecondition)
Σ
W
(70)
Ere)(ry)
JEJ
Separating the contribution the dominant coordinate Jmaxs we have:
1
1
n( (Rouss Rerecondition)
Σ W
+
Wian Jun
(71)
rt
T/mm
E
(r,)
JEJ
Since ETer(Ty) 2 Tlp., this difference is dower-bounded by:
1
J
n( Rours Repecondition)
Σ
W,
+
W Jmss
(72)
T Jmme
jeJ
Further bounding the terms using ry S Tyjm for all j E J'. we obtain:
|
|
Σ JEJ
W
W Winn
2 (1 W Winn)
Wimm
jim
(73)
r3
Fjnm
Jnn
Fin
19

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
Therefore, if the following condition holds:
(1 W Jma)) Jnn
(74)
then Reurs 2 Preecomittoon and our method guarantees superior OSGR performance compared other preconditioning strategies. This result highlights the robustness of our formulation, particularly under skewed GSNR distributions.
D. Implementation Details
D.1. Training details
As introduced in the experimental section, we follow the standard training, hyperparameter search methods, and evaluation protocol proposed by DomainBed (Gulrajani & Lopez-Paz, 2021) to ensure a fair comparison. For each dataset, the models were trained for 15,000 iterations on DomainNet and 5,000 iterations on the other datasets. The search space of hyperparameters is provided in Table All experiments were conducted on an NVIDIA GeForce RTX 4090 under the environment of Python 3.8. 10, PyTorch 1.13.1, Torchvision 0. 1, and CUDA 11.7.
Table 8. The scarch space of hyperparameters.
PARAMETER
DEFAULT VALUE
SEARCH DISTRIBUTION
BATCH
32
UNI.0IM(3.5.0)
LEARNING RATE
0.015
10 10UNPORM(-33--
RESNET DROPOUT
0.0
0.1.
WEIGHT DECAY
0.0
UN 6
20

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
D.2. Pseudo code
Algorithm 2 Unified Algorithm for RMSProp, Adam, and GENIE
Input: Mini-batches {B. learning rate a, total steps T
Preconditioning Parameters: B, f1, 32 [0, 1], noise scale a, mo t- 0, 0o
Initialize: Parameters 0o
for /. = to 7 do
Compute Gradient: Sample mini-batch B, and compute
9t VL(0;;
Compute Preconditioned Gradient:
RMSProp:
1t - BU (1- -39
Jt
ge
Vor +
Adam:
m. t 31m4-1 + (1- -11)9
+ 82 + - -2)g9/
m + m1
Up
mh
9t
B
32
v U0 +
GENIE:
mh < 8m--1 + (1 - -B)9t,
12 x B + (1 - -B)gi
1
07 U4 m
It tanh
my
1
gt. -
It.
1 - Bi Ut
1 )1, )], & ~(0,00)
Noise, + Ee tanh(
o,
g: Å. Noise,
9t f 9: 2 M, Mj Bernoulli())
Update Parameters:
1c+1 t 0g - ağt.
end for
Output: Final parameters 0r+1 for RMSProp, Adam, and GENIE.
D.3. Code
[Pyyhion=Byythen implementation of preconditioning updates. label= (1oel==[sttin spreooodttt
2
def _initialize _inttiaaliaapprrcodtitiiioooioii (self, current_state):
pelttppre_sstae = current_state
self. s (k: corch.zeros_iie (param) for k, param in
)))
5.
self.ge2 = (k: torch. wethheeeoss_ll
torch. wirroos__kkkeee (param) for k, param in
self.networs named named parameters
self.scale 0.0
def _upctte_prccondttiinnnng..ee
1r, moving_avg):
grad_agd = ()
pgrad = (}
pGsnr = (}
13
Upoate scale factors
21

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
self. scale (moving_avg
self.scale 1.0)
15
scalel (1 moving_avg) self.scale
16
scale2 2.0 scalel
rho (1.0 moving_avg) scale2 (11.0 moving_avg) scalel)
19
with torch.no_gret n)
20
Update gradiente and variance
21
for k, paran in self.nctwork nemed__arammterss)) () delta
grad. (atacchhahah))))
self geeang [k] self. gmean[k] moving_avg + delta * (1.0 mmviigg_vv]
self.ge2[k] [k] seif.ge?[k] moving_avg + (delta ** 2) * (1.0 moving_avg)
25
gm self f. s gmean [k] / scalel
ge2 self. gelf.ge211 [k] scalel
28
var ge2 gm. square
29
var /= (1.0 rho)
var = torch. where (var > 0.0, var, aerch (var) 1c-8)
invvar torch.clamp (1 var, min=0.0, max-10.0) mvar rho var mvar = torch. where (mvar > 0.0, mvar, torch. zeros_ likes___kkkeee (mvarr 1e-8)
34
Preconditioned gradient scaling tanh invvar torch. tanh tana
37
38
pGsnrik] = (1.0 / (1.0 + mvar (gm.square 1e-8))) + Lanh_iavva
39
Add noise for stochastic gradient adjustment ncise_scale torch. sumch.sum/ce (tanh_invvar torch. abs (gm)
42
(1.0 (1.0 mvar (gm. (1--))))) /
torch. satttnhh_tttttstttt invvar)
noise torch. normal (torch. zeros like (delta), forch.ones. like (delta))
zeros_like
noise_scale [gdd_s
tanb_invvar) noise
45
46
Compute preconditi tione gradients for k, param in pgrad[k] self.gmean(k] scalel pGanr(k] viee_as (paran)
nawed parreed_prammmerss
49
50
Apply gradients with dropout-based masking for k, param in newwwkk naned parametezs
mask (torch. rand_likee (param) > self.hparama [' p) ffoat () (1
self.hparams 1'p(l) self.prev_state [k] -- (pgrad(k) + grad_agd(k]) (k]) * mask ir
53
E. Experimental Details and Results
E.1. Dataset
This section introduces five representative DG datasets utilized in this paper.
PACS (Li et al., 2017): This dataset includes 4 different domain styles Photo, Art Painting, Cartoon, and Sketch. Each domain contains categories and consists of 9,991 images. It is well-suited for evaluating generalization performance across style variations.
OfficeHome (Venkateswara et al., 2017): This dataset consists of 4 different domain styles Art, Clipart, Product, and Real-world. Each domain includes 65 categories, with total of 15,588 samples.
VLCS (Fang al., 2013): Derived from four distinct datasets Caltech101, LabelMe. VOC2007, and SUN09 this dataset includes 5 shared classes across domains, containing total of 10,729 images. It is ideal for evaluating distributional differences between datasets.
Terra Incognita (Beery et al., 2018): Comprising photographs of wildlife, this dataset collected from different
22

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
locations-LL000 L38, L43, and L46. It includes 10 categories and 24,788 samples and commonly used to measure model generalization performance in real-world scenarios.
DomainNet (Peng et al., 2019): This large-scale dataset includes 6 domains Clipart, Infograph, Painting, Quickdraw, Real, and Sketch. It comprises 345 categories and total of 586.575 samples.
E.2. Detailed Results: Comparison with Existing Optimizers
We compare the performance of existing optimizers with our proposed GENIE optimizer on five datasets from the same DG benchmark. The dataset-specific experimental results are presented Table 9,Table 10, Table 11, Table 12, and Table 13. Additionally, we analyze the performance and training time as the number iterations increases, as shown Table 14, Table 15, Table 16.
Table Comparison Results on the PACS Dataset.
OPTIMIZER
ART
CARTOON
PHOTO
SKETCH
AVG.
ADAM
88.0+1.2
79.720.5
96. 200.4
72.7=0.9
84.3
4
84. 11..5
30.7
96.9 26.4
72.820.6
83.6
SGD*
85.1
76.0+0.3
98.3=0.4
60.3+6.1
79.9
YOGI
84.4+1.7
79.7+0.6
95.8=0.3
65.1
81.2
ADABELIEE
85.4+2.2
80.4+1.1
97.4=0.7
75.1+1.4
84.6
ADAHESSIAN
88.420.6
80.020.9
97.720.4
71.724.1
84.5
SAM
85.721.2
81.021.4
97.120.2
77.421.8
85.3
GAM
85.920.9
81.321.6
98.220.4
79.0a2.1
86.1
FAD*
88.5+0.5
83.0+0.8
98.4=0.2
82.8#0.9
88.2
GENIE (OURS)
88.720.7
82.821.3
98.5.0.1
81.3
87.8
Table Comparison Results on VLCS Dataset.
OPTIMIZER
CALTECH
LABELME
SUN
VOC
AvG.
ADAM"
98.9+0.4
65.921.5
71.021.6
74.5+2.0
77.3
ADAMW
98.3+0.1
65.1
70.9
75.2+1.5
77.4
SGD*
98.4+0.2
64.7+0.7
72.5+0.8
76.6+0.8
78.1
YOGI*
98.1
63.9+1.2
72.5+1.6
75.7+1.2
77.6
ADA ADABELIEF*
98.0+0.1
63.9+0.4
73.4+1.0
78.2+1.8
78.4
ADAHESSIAN
19.1
65.021.7
72.721.3
77.721.0
78.6
SAM*
98.521.0
66.2a1.6
72.0a1.0
76.121.0
78.2
GAM
98.8
65. 1a1.2
72.9a1.0
77.2+1.9
78.5
FAD*
99.1+0.5
66. 86.8
73.6=1.0
76.1
78.9
GENIE (OURS)
99.320.3
67.2+1.5
76.620.3
79.720.8
80.7
Table II. Comparison Results the OfficeHomme Dataset.
OPTIMIZER
ART
CLIPART
PRODUCT
REAL-WORLD
AvG.
ADAM*
63. 940.8
48. 10.6
77 77.020.9
81.821.6
67.6
ADAMW*
66.1 1
48. 40.7
76.6x0.8
83.6+0.4
68.8
SGD*
65.3+0.8
48.8+1.4
76.7+0.3
83.0+0.
68.5
YOGI+
63.5+1.0
49.2+1.2
76.2+0.5
84.5+0.6
68.3
ADABELIEF
65.6+2.0
48.9
74.810..
83.620.9
68
ADAHESSIAN
63.0a2.9
50.021.4
77.720.8
83. $3.020.5
68.4
SAM#
63.5a1.2
48.610.9
77.020.8
82.921.3
68
GAM
63.0+1.2
49.8+0.5
77.610.6
82.4+1.0
68.2
F
63.5+1.
50.3+0.8
78.0+0.4
85.0+0.6
69.2
GENIE (OURS)
66.240.5
55.020.4
77.540.4
80.0+0.5
69.7
23

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
Table 12. Comparison Results the Terralncognita Dataset.
OPTIMIZER
L100
L38
L43
L46
AVG.
ADAM
42.2+3.4
40. #1.2
19.920.2
35.0+2.8
44.4
44.2+6.8
39.8=1.9
60.3a2.0
36.6a1.8
45.2
SGD*
41.8a5.8
39.823.9
60.5+2.2
37.5a1.1
44.9
YOGI
43.9+2.2
42.5+2.6
60.5a1.1
34.8
45.4
ADABELIEF*
42.626.7
43.0=2.0
60.2+1.3
35.1
45.2
ADAHESSIAN"
42.5+4.8
39.5=1.0
58,442.6
37.320.8
44.4
SAM
42.943.5
43.0=2.2
60.5+1.6
36.4+1.2
45.7
GAM
42.242.6
42,9+1.7
60.2+1.8
35.5+0.7
45.2
FAD*
44.3+2.2
43.5+1.7
60.9a2.0
34.1+0.5
45.7
GENIE (OURS)
4.8
2.1
0..2+
45.9= 1.0
52.0
Table 13. Comparison Results the DomainNet Dataset.
OPTIMIZER
CLIP
INFO
PAINT
QUICK
REAL
SKETCH
AVG.
ADAM*
63.0=0.3
20.2+0.4
49.1+0.1
13.0+0.3
62.0+0.4
50.7+0.1
43.0
A
63.020.6
20.620.2
49. 49.620.0
13. 13.019.2
63.620.
50.420.1
43.4
SGD*
61.320.2
20.420.2
49.420.2
12.610.1
65.720.0
49.620.2
43.2
YOGI
63.320.1
20.620.1
10.1
13.240.3
62.820.1
51.020.2
43.5
ADABELIEF*
63.5#0.2
10.5+0.1
50.0+0.3
13.2+0.3
63. 1+0.1
50.7+0.1
43.5
DESSIANNN*
63.3#0.2
21.4=0.1
50.8+0.3
13.6+0.1
65.7*0.1
51.4=0.2
44.4
SAM"
63.3=0.1
20.3+0.3
50.0+0.3
13.6+0.2
63.6#0.3
49.6+0.4
43.4
GAM
63.0.0.5
20.2a0.2
50.3+0.1
13.2+0.3
64.5.0.2
51.6#0.5
43.8
FAD*
64.1.0.3
21.920.2
50.620.3
14.2s0.4
63.6a0.1
52.2+0.2
44.4
GENIE (OURS)
62.5.0.5
21.3a0.4
50.0s0.4
14.0+0.4
64.0+0.7
52.6.0.8
44.1
Table 14. Comparison of Optimizers on PACS Dataset Across Iterations.
OPTIMIZER
ITERATION
TRAINING TIME (/S)
ACCURACY
AVG.
[0]
[2]
[3]
[0]
[1]
[2]
[3]
TIME
ACC
SGD
5000
1785
1819
1823
1848
73.4
61.2
96
48.4
1819
69.8
10000
3570
3643
3646
3749
76.8
65.1
97.4
56.
3652
73,9
150000
5371
5478
5465
5609
77.6
67.1
97.8
60.9
5481
75.9
ADAM
5000
1672
1668
1707
1714
86.2
78.2
95.7
76.6
1690
84.2
10000
3321
3348
3392
3408
86.2
81.7
95.7
80.8
3367
86.
150000
4989
5030
5067
5092
80.2
81.7
95.5
80.8
5045
84.6
SAM
5000
2661
2717
2729
2689
84.9
74
98.2
72.6
2699
82.4
10000
5378
5425
5479
5392
87.1
75.7
98.1
73.3 2
S419
83.5
150000
8113
8125
8228
8073
87.2
77.1
98.4
73.8
8135
84.
GENIE(OURS)
5000
1862
2017
1532
1419
89.3
84.1
98.7
81.6
1708
88.4
10000
3732
4054
3065
2836
87.6
80.9
98.4
81.6
3422
87.1
150000
5607
6077
4586
4256
88.2
80.
98.4
80.9
5132
86.9
24

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
Table 15. Comparison of Optimizers VL.CS Dataset Across Iterations.
OPTIMIZER
ITERATION
TRAINING TIME (/s)
ACCURACY
AVG.
[0]
[1]
[2]
[3]
[0]
[1]
[2]
[3]
TIME
ACC
SGD
5000
9154
4947
9315
9119
97
61.9
73.3
74.6
8134
76.7
10000
18311
10054
18641
18312
97.6
60.3
72.7
77.2
16330
77
150000
27732
14956
27918
27419
98.1
62.4
72.6
77.8
24506
77.7
ADAM
5000
9087
4900
9154
9210
98.1
63
73
73.8
8088
77
10000
18148
10024
18404
18312
98.1
63
73
73.8
16222
77
150000
27532
14963
27615
27387
98.1
63
73.8
24374
SAM
5000
9544
5611
9523
9555
99
63.5
74.6
80.5
8558
79.4
10000
19068
11203
18984
18749
98.9
64.1
76.
81.9
17001
80.3
150000
28510
16817
28772
27860
99
64.6
75.3
82.6
25490
80.4
GENIE(OURS)
5000
8726
4631
7118
5485
99.5
68.6
76.9
80.4
6490
81.4
10000
17452
9273
14235
10885
99.5
68.6
76.9
80.4
12961
81.4
150000
26164
13917
21396
16314
99.5
68.6
76.9
80.4
19448
81.4
Table 16. Comparison of Optimizers Office Dataset Across Iterations.
OPTIMIZER
ITERATION
Training TIME (/s)
ACCURACY
AVG.
[0]
[1]
[2]
[3]
[0]
[1]
[2]
[3]
TIME
ACC
SGD
5000
6,634
6,436
5,842
4,554
46.2
40.1
56.8
61.9
5,867
51.3
10000
13.334
12.665
11,722
8,899
56.8
46.4
70.6
76.2
11,655
62.5
150000
20,031
18.565
17,676
13,175
58.5
47.1
73.4
76.6
17.362
63.9
ADAM
5000
8.000
8,147
5,799
4.255
57.8
49.4
73.8
73.4
6,550
63.6
10000
16,260
16,602
11,605
8.379
59.8
$2.2
73.8
74.8
13,212
65.2
150000
24,776
25,466
17,387
13,062
59.8
52.2
73.8
74.8
20,173
65.2
SAM
5000
6,639
6,598
5,884
5,151
65.4
$4.2
77.1
81.1
6,068
69.4
10000
13,359
12,964
11,813
10,189
54.5
77.9
80.9
12,081
69.6
150000
20.140
18,983
17,722
14,946
65.5
55.1
78.6
80.9
17,948
70
GENTE(OURS)
5000
4,777
5,846
4,025
4,061
66.5
55.4
77.8
80.3
4,677
10000
9,553
11,624
8,062
8,211
65.8
53.3
77.8
79.9
9,363
69.2
150000
14.261
17,333
12,229
12.369
65.8
53.3
77_
80.3
14,048
69.1
E.3. Integration with DG methods
The detailed performance of previous DG methods employed with our optimizer presented Table 17, Table 18, Table 19,
Table 20.
25

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
Table 17. Integration with existing DG algorithms the PACS dataset.
ALGORITHM
ART
CARTOON
PHоTO
SKETCH
AVG.
ERMY(VAPNIKS 1999)
44.4
80.8
97.2+0.3
79.3+1.0
85.5
IRMM(ARIOVSKY 2019)
84.821.3
76.421.1
96. 6
76.121.
83.5
(SAUP ET AL., 2020)
83.5+0.9
19.
96.720.3
78.3+2.0
84.4
I-MIXUP((XU AL.,, 2020)
86. +0.55
78.9
97.6=0.1
75.8+1.8
84.6
MLDG ET AL.., 2018A)
85.5+1.4
80.1
97.4+0.3
76.6+1.1
84.9
MMD*(LI AL.,, 2018B)
86.1+1.4
79.420.9
96.620.2
76.5.0.5
84.7
DANNI(GANIN ET AL., 2016)
86.4+0.8
77.4+0.8
97.3+0.4
73.5+2.3
83.7
CDANN ET AL.,, 2018C)
84.6+1.8
75.5=0.9
96. 36.3
73.5+0.6
82.6
MTLCABLANCHARD ET AL.. 2021)
87.520.8
17.1
96.420.8
77.3a1.8
84.6
SAGNETÍ(NAM ET AL., 2021)
87.421.0
80.720.6
97.1
80.010.4
86.3
ARMY(ZHANG ET AL.,, 2021)
86.8+0.6
76.8+0.5
97,4+0.3
79.3+1.2
85.1
VREXG(KR AL., 2021)
86.0#1.6
79. 6
96.940.5
77.7+1.7
84.9
MIXSTYLE(ZHOU 2021)
86.8+0.5
79.0x1.4
96.610.1
78.5+2.3
85.2
RSC+GENTE(OURS)
87.8+0
82.4+0.8
97.4+0.9
81.4+2.0
87.3
CORAL=GENIE(OURS)
88.8+0.2
82.5.0.5
98.2+001
82.2+0.2
87.9
Table 18. Integration with existing DG algorithms on the VLCS dataset.
ÅLGORITHM
CALTECH
LABELME
SUN
VOC
AvG.
ERM*(VAPNIK, 1999)
98.0s0.3
64.7
71.4
75.2+1.6
77.3
IRM ET AL., 2019)
98.6.0.1
64.920.9
73.4
77.3.0.9
78.6
GROUPDROR(SAGANVA
2020)
97.320.3
63.4a0.9
69.5.0.8
76.7=0.7
76.7
I-MIXUPI(XU ET AL., 2020)
98.320.6
64.8+1.0
72.1
74.3=0.8
77.4
MLDG AL., 2018A)
97,4=0.2
65.2+0.7
71.0+1.4
75.3+1.0
77.2
MMD+(LI ET AL., 2018B)
97.720.1
64.0a1.1
72.820.2
75.323.3
77.5
DANN* ET AL.. 2016)
99.020.1
65.121.4
23.1
77.220.6
78.6
CDANNY(LI ET A1.,, 2018c)
97.1
65.1+1.2
70.7+0.8
77.1+1.5
77.5
MTL. AL.,, 2021)
97.8+0.4
64.3an.3
71.5.0.7
75.3+1.7
77.2
SAGNETi(NAM ET AL., 2021)
97.920.4
64.520.5
71,421.
77.5=0.5
77.8
ARMY ET AL., 2021)
98.7+0.2
63.6#0.7
71.321.2
76.7=0.6
77.6
VREX+(KRUEGER ET AL., 2021)
98,420.3
64.4+1.4
24.1
76.2+1.3
78.3
MIXSTYLE(ZHOU ET AL., 2021)
98.6a0.3
64.5a1.1
72.620.5
75.741.7
77.9
RSC+GENIE(OURS) CORAL+GENIE(OURS
99.120.3
68.3+1.3
76.020.4
79.1:0.6
80.6
99.020.3
68.3+1.0
76.6s1.3
78.9=0.9
80.7
26

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
Table 19. Integration with existing DG algorithms on the OfficeHome dataset.
ALGORITHM
ART
CLIPART
PRODUCT
REAL-WoOD WORLD
AVG.
ERMP(VAPNIK,, 1999)
61.3+0.7
52.4+0.3
75.8+0.1
76.6=0.3
66.5
IRMP(ARJOVSYY ET AL., 2019)
58.922.3
52.221.6
72. 122.9
74.0.2.5
64.3
GROUPDROY(SAGAWA ET AL., 2020)
60.420.7
52.721.0
75.020.7
76.020.7
66
I-MIXUP((XU AL., 2020)
62.4
54.8+0.6
76.9+0.3
78.3
68.1
MLDGG(LI ET AL., 2018A)
61.5=0.9
53.2+0.6
75.0s1.2
77.5=0.4
66.8
MMD ET AL.,, 2018B)
60.4
53.320.3
74.3+0.1
77.4.0.6
66.4
(GNNN ET AL., 2016)
59.9=1.3
53.0+0.3
73.6+0,7
76. 76.920.5
65.9
CDANN AL., 2018C)
61.5=1.4
50.4+2.4
74.4+0.9
76,6=0.
65.7
MTL'iBLANCHARD ET AL., 2021)
61.520.7
52.420.6
74.920.4
76.820.4
66.4
SAGNETF(NAMM AL., 2021)
63.420.2
54.820.4
75.820.4
78.320.3
68.1
(ZHANG ET AI.,, 2021)
58.9
51.0+0.5
74.1+0.1
75.2+0.3
64.8
VKEXG(RR ET AL.., 2021)
60.7
53.0+0.9
75.3+0.1
76.6=0.5
66.4
MIXSTYLE(ZHOU ET AL., 2021)
51.1
53.220.4
08.2.0.7
69.220.6
60.4
RSC+GENTT(OORS) CORAL+GENTF(OURS)
63.2
34.8+0.3
76.0+0.7
78.3+1.9
68.1
66.5+n.2
56.7=0.3
78.820.1
80.4
70.6
Table 20. Integration with existing DG algorithms the Terralncognita datasct.
ALGORITHM
L,100
L38
L43
L46
AVG.
ERMN(VAPNIK, 1999)
54.3+0.4
42.5+0.7
55. 35.620.3
38. 8+2.5
47.8
[RMI(ARJYVSKY ET AL., 2019)
54.6#1.3
39.8+1
56. 2
39.6=0.8
47.6
GROUFDRO=(AAGAWA ET AL., 2020)
41.220.7
38.642.1
56.7
36.422.1
43.2
I-MIXUPf(XU
2020)
59.612.0
42.2
55.920.8
33.921.4
47.9
MLDGA ET AL., 2018A)
54.2+3.0
44.3+1.1
55.6+0.3
36.9+2.2
47.8
MMD*(LI AI., 20188)
41.9+3.0
34.8=1.0
57.0=1.9
35.2+1.8
42.2
DANNI ET AL., 2016)
51.123.5
40.6#0.6
57.4=0.5
37.7=1.8
46.7
CDANN AL., 2018C)
47.0=1.9
41.324.8
54.9=1.7
39.8=2.3
45.8
(TLNCALAN ET AL., 2021)
49.3+1.2
39.6#6.3
55.6#1.1
37.8=0.8
45.6
SAGNET?(NAM AL., 2021)
53.0w2.9
43.0+2.5
57.940.6
40.4+1.3
48.6
ARM*(ZHANG ET AL., 2021)
49.329.7
38.322.4
55.820.8
38.721.3
45.5
VREXE ET AL.,, 2021)
48.2+4.3
41.7+1.3
56.8+0.8
38.7+3.1
46.4
MIXSTYI.R(ZHOU ET AI., 2021)
54.3+1.1
34.1+1.1
55.9+1.1
31.7+2.1
44
RSC+GENIIEOORSS) CORAL+GENIE(OURS)
56.5a3.2
44.5a3.7
55.9=1.0
40.9
49.5
57.0s1.2
42,9+0.7
54.1
39.4+1.3
48.4
E.4. Detailed SDG performance.
This section reports a detailed comparison of our optimizer with existing optimizers across four datasets in the SDG setting. also provides the performance previous methods when employed with our optimizer. Table 21, Table 22 Table 23, Table 24
E.5. Model Analysis
Figure visualizes the behavior of SGD, Adam, and GENIE on various loss landscapes, with the number of iterations fixed 30 for consistency. The results show that GENIE does not directly converge the minima of the training data, whereas Adam and SGD quickly converge the minima. While this behavior might be advantageous IID scenarios, it can lead overfitting the source domain in OOD settings. Figure 7 provides claborates on the parameter update patterns observed across optimizer.
detailed analysis of the experiment. It further
27

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
Table 21. Detailed performance on the PACS datasct in the SDG setting.
ALGORITHM
ART
CARTOON
PHOTO
SKETCH
AVG.
ERM+ADAM
77.5
72.1
54.4
53.3
64.3
ERM+SGD
64.0
66.0
40.8
27.1
49.5
ERM+SAM
70.1
75.6
42.5
42.7
57.7
ERM-GENIE(OURS)
78.620.6
81.920.9
53.8+2.2
63.5+5.2
69.5
RSC+GENIE(OURS)
79.8=2.0
81.0+1.2
56.7+3.3
65.9+4.6
70.9
CORAL-GENIE(OURS)
78.920.6
78.6x2.2
53.921.2
61.3+3.4
68.2
Table 22. Detailed performance on the VLCS dataset in the SDG setting.
ALGORITHM
CALTECH
LABELME
SUN
VOC
AVG.
ERM+ADAM
33.7
53.5
62.3
75.4
56.2
ERM+SGD
46.3
52.4
65.6
77.3
60.4
ERM+SAM
54.4
68.1
65.8
78.7
66.7
ERM+GENTE(OURS)
56.624.3
75.421.1
67.2a1.4
80.520.3
69.9
RSC+GENTE(OURS)
56.3+2.5
72.0+1.5
68.8s1.7
79.6.1.1
69.2
CORAL-GENTE(OURR))
55.9+1.3
71.7*1.8
67.2+1.8
79.9+1.4
68.7
Table 23. Detailed performance on the OfficeHome dataset the SDG setting.
ALGORITHM
ART
CLIPART
PRODUCT
REAL- WORLD
AVG.
ERM+ADAM
52.6
46.5
45.6
50.7
ERM+SGD
47.7
41.4
43.2
51.5
45.9
ERM+SAM
60.1
57.8
55.8
63.1
59.2
ERM+GENIB(OURS)
59.4 =
$8.7
54. 0.5
62.0 0.3
58.6
RSC+GENIE(OURS) CORAL+GENIE(OURE
55.8 2.6
52.5 2.0
49.7 +30
59.7 + 2.0
54.4
28.0
54.8
51.3 1..3
61.4 u.7
56.4
Table 24. Detailed performance on the TerraIncognita dataset in the SDG setting.
ÅLGORITHM
L100
L38
L43
L46
AVG.
ARAM
27.0
25.5
42.9
38.6
33.5
ERM+SGD
22.1
17.9
22.3
29
22.8
ERM+SAM
21.1
21.6
30.3
34.2
26.8
ERM+GENIE(OURS)
28.5 0.5
29.4
41.7
44.5
36.0
RSC+GENIE(OURS)
27.1 115
22.6 6.611.7
39.7
4
33.2
CORAL+GENIE(OURI
29.3 1.0
29.8 4..8
43.
43.6 1.6
36.7
28

---

One-Step Generalization Ratio Guided Optimization for Domain Generalization
030
175
4..0
33
14
-15
-23
-13
06
16
Alrs
12-
-05
-18
-13
Figure 6. Optimization trajectories simulated loss landscape.
Caltech101
LabelMe
SUN09
VOC2007
SGD
0.5
3.0
2.0
1.0
2.8
0.0
3.0
0.0
2.
Parmmeeer Indle
3N77
Porameter Index
307
Parameter Index
1a/7
Paranicter index
Adam
0.5
2.0
2.0
0.0
1.0
15
2.5
0.0
0.5
1.0
2.0
60
1.0
3.5
tndio.
Irden
SAM
Partmeter
GENIE
0.0
1.0
1.5
Parameter MNOO
1n7
Porameter Vndeo
367
Parometerr Indox
1w7
Paramater Indea
267
Figure 7. Heatmaps visualizing normalized parameter update magnitudes by parameter ID for different optimizers throughout training on VLCS dataset DG.
29


Reviews:
[{"review_id": "9i3HbcA15g", "reviewer": "Reviewer_V3wF", "summary": "This paper proposes a novel optimizer that leverages the one-step generalization ratio to assess each parameter’s contribution to loss reduction, aiming to promote domain-invariant feature learning.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. Below, we address the points raised under \"Experimental Designs or Analyses\":\n\n# A1\nBecause of limited time and computational resources, we were unable to include experiments on additional tasks such as object detection or segmentation within the submission period. We fully agree that evaluating GENIE on a wider range of tasks (e.g., detection, segmentation, or face anti-spoofing) would further validate its generalization capabilities. We consider this an important direction for future work and plan to release additional results and code for these tasks moving forward.\n\n# A2\nGENIE uses two key hyperparameters: the dropout probability $p$ and a moving average coefficient $\\beta$ (used for computing the running mean and variance of gradients). We conducted an additional hyperparameter sensitivity analysis on the OfficeHome dataset to examine the impact of these parameters. As shown by our new experimental results, GENIE consistently outperforms SGD, Adam, and SAM across a range of $p$ and $\\beta$ values, demonstrating the method’s robustness to these hyperparameters. (All other training settings were held constant in this analysis.)\n\nWe note that we performed a grid search to tune hyperparameters in this new experiment; in contrast, for all other experiments we followed the DomainBed protocol and selected hyperparameters based on validation performance. We hope these clarifications address your concerns. Thank you again for your thoughtful and constructive feedback. \n\nsee: [**URL**](https://imgur.com/a/puvgaCx). (The link is fully anonymized.)\n\n---\nWe hope these clarifications address your concerns. Thank you again for your thoughtful and constructive feedback.", "claims_and_evidence": "The paper’s claims are clearly stated, and the experiments presented provide convincing evidence in support of these claims.", "methods_and_evaluation_criteria": "Both the proposed method and the chosen evaluation metrics align well with the stated research problem.", "theoretical_claims": "I’ve checked the proofs and no issues are found.", "experimental_designs_or_analyses": "The experimental setup is generally solid. However, to further validate the effectiveness of the proposed optimizer, I recommend extending the experiments to segmentation or detection tasks in domain generalization settings (e.g., from GTAV to Cityscapes, BDD100K, and Mapillary). In addition, exploring how different hyperparameters (such as β1 and β2) influence the optimizer’s performance would provide valuable insights into its sensitivity to hyperparameter choices.", "relation_to_broader_scientific_literature": "The paper references and conducts experiments on domain generalization tasks and includes comparisons with other existing optimizers.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "Please address my concerns raised under “Experimental Designs or Analyses.”", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DJhFoCVs5s", "reviewer": "Reviewer_Z37r", "summary": "This paper proposes GENIE, a novel stochastic optimizer designed for Domain Generalization (DG) tasks. Unlike standard optimizers (SGD, Adam, etc.) that can over-emphasize certain “spurious” features, GENIE uses a metric called One-Step Generalization Ratio (OSGR) to guide parameter updates. The key idea is to balance the contribution of each model parameter to the on-step generalization ability, thereby preventing a small subset of parameters from dominating the learning. \n\n GENIE introduces 3 main components in its update rule: 1) a preconditioning factor per parameter to equalize OSGR contributions, 2) a noise injection term to encourage exploration of flatter minima, and 3) a dropout mask on gradients to reduce overfitting and stabilize updates. By incorporating these, GENIE aims to promote domain-invariant features learning and avoid reliance on domain-specific correlations. \n\nThe authors show theoretically that GENIE achieves a higher OSGR than conventional optimizers while retaining the same convergence rate as SGD. Empirically, GENIE consistently outperforms baseline optimizers like SGD, Adam, Yogi, AdaBelief, AdaHessian, and SAM on average accuracy across several standard DG benchmarks (PACS, VLCS, OfficeHome, TerraIncognita, DomainNet) following DomainBed protocols. \n\nOverall, the paper’s conceptual contribution is introducing OSGD-guided optimization to balance gradient contributions, and its main finding is that this leads to more robust models that generalize better to unseen domains.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your detailed and constructive review. \n# A1\nAs you pointed out, our initial submission reported only the best single trial due to computational constraints. Following your suggestion, we have now conducted additional experiments with three independent trials per optimizer using random seeds {0, 1, 2}, and report the mean accuracy and 95% confidence intervals below. GENIE consistently outperforms the baselines across datasets with low variance. The full set of results will be included in the appendix of the final version.\n|PACS|Art|Cartoon|Photo|Sketch|Avg|\n|----|---|-------|-----|------|----|\n|Adam|88.0±1.2|79.7±0.5|96.7±0.4|72.7±0.9|84.3|\n|SGD|85.1±0.4|76.0±0.3|98.3±0.4|60.3±6.1|79.9|\n|SAM|85.7±1.2|81.0±1.4|97.1±0.2|77.4±1.8|85.3|\n|GENIE(our)|88.7±0.7|82.8±1.3|98.5±0.1|81.3±0.4|**87.8**|\n\n|VLCS|Caltech|LabelMe|SUN|VOC|Avg|\n|----|-------|--------|---|---|----|\n|Adam|98.9±0.4|65.9±1.5|71.0±1.6|74.5±2.0|77.3|\n|SGD|98.4±0.2|64.7±0.7|72.5±0.8|76.6±0.8|78.1|\n|SAM|98.5±1.0|66.2±1.6|72.0±1.0|76.1±1.0|78.2|\n|GENIE(our)|99.3±0.3|67.2±1.5|76.6±0.3|79.7±0.8|**80.7**|\n\n|OfficeHome|Art|Clipart|Product|Real-World|Avg|\n|----------|---|-------|-------|-----------|----|\n|Adam|63.9±0.8|48.1±0.6|77.0±0.9|81.8±1.6|67.6|\n|SGD|65.3±0.8|48.8±1.4|76.7±0.3|83.0±0.7|68.5|\n|SAM|63.5±1.2|48.6±0.9|77.0±0.8|82.9±1.3|68.0|\n|GENIE(our)|66.2±0.5|55.0±0.4|77.5±0.4|80.0±0.5|**69.7**|\n\n|TerraInc|L100|L38|L43|L46|Avg|\n|--------------|----|---|---|---|----|\n|Adam|42.2±3.4|40.7±1.2|59.9±0.2|35.0±2.8|44.4|\n|SGD|41.8±5.8|39.8±3.9|60.5±2.2|37.5±1.1|44.9|\n|SAM|42.9±3.5|43.0±2.2|60.5±1.6|36.4±1.2|45.7|\n|GENIE(our)|55.2±4.8|47.5±2.1|59.2±0.4|45.9±1.0|**52.0**|\n# A2\nFor baseline optimizers, we used results from Zhang et al. (ICCV 2023) obtained under the DomainBed framework. For GENIE, two hyperparameters—dropout probability $p$ and the moving average coefficient $\\beta$—were tuned via the standard DomainBed hyperparameter selection procedure (hparams_seed ∈ {0, ..., 19}), selecting configurations with the highest validation accuracy. We further discuss hyperparameter sensitivity in our response to **Reviewer V3wF (A2)**, and will publicly release optimized hyperparameters for reproducibility.\n# A3\nThe preconditioning value includes the term  $\\tanh(1/\\sigma_t^2)$, which tends to suppress the updates of gradients with high variance. While this helps avoid overfitting to noisy gradients, it can undesirably reduce updates to certain parameters. To counterbalance this suppression, we introduce a reciprocal noise scaling factor $1 - \\tanh(1/\\sigma_t^2)$ which amplifies the noise component where variance is high, encouraging exploration of alternative solutions. This form was chosen to maintain a complementary relationship with the preconditioning factor and encourage escape from sharp or suboptimal minima.\n# A4\nWe treated the dropout probability $p$ as a global hyperparameter, applying it uniformly across all layers and parameters for simplicity and consistency. This hyperparameter was tuned individually for each dataset.\n\n---\nWhile not explicitly raised as formal questions, we would also like to respectfully address several concerns implied in your review:\n# Feature Invariance\nYou correctly noted the lack of feature-level analysis supporting our claim of domain-invariant features. To address this, we have now included t-SNE and UMAP visualizations (fully anonymized): [**URL**](https://imgur.com/a/Pu3xrOi). These results clearly demonstrate improved cross-domain feature alignment and support our claims about domain-invariant feature learning.\n# OSGR Conjecture\nWe agree that the conjecture about uniform OSGR leading to better generalization lacked formal proof. However, Section 3.3.1 shows analytically that our proposed preconditioning yields more uniform parameter-wise OSGR, thereby achieving higher total OSGR across all parameters as implied by Jensen's inequality. Additionally, Liu et al. (ICLR 2020) established that higher total OSGR correlates with improved generalization. We recognize that this point is a critical part of our contribution. To further address this concern, we refer to our response to **Reviewer MLrB**, which includes a PAC-Bayesian analysis comparing GENIE with SAM, showing our method explicitly minimizes the KL term for better generalization.\n\n# Sharpness\nWe acknowledge your point that we claimed GENIE naturally leads to flatter minima without explicitly measuring curvature. While the SAM method is grounded in PAC-Bayesian theory and focuses on sharpness—which can be interpreted as a bound on empirical risk with respect to the posterior parameter distribution—we would like to highlight that, Our preconditioning strategy provide an additional benefit: it also contributes to tightening the generalization bound by explicitly minimizing the KL divergence term as our answer to **Reviewer MLrB**.\n\nWe hope our clarifications adequately address your concerns and demonstrate the rigor and value of our proposed method.", "claims_and_evidence": "Some claims stated in the paper are as follows: \na. Standard optimizers allow a few parameters to dominate updates. \nb. The proposed GENIE optimizer increases OSGR and leads to better domain generalization performance than existing optimizers. \nc. GENIE maintains the convergence speed of SGD despite its modifications. \nd. Integrating GENIE with known DG methods yields additional improvements without altering those methods’ architectures. \ne. Uniformly distributed OSGR across parameters indicates better generalization. \n\nOverall, claims a – d are substantiated by either theoretical derivations or thorough experiments. Claim e, however, is not explicitly proven. This is stated as a conjecture rather than a theorem, and while intuitively supported (and aligned with their results), it’s not rigorously demonstrated beyond the heuristic argument.  \n\nAdditionally, the authors claim that GENIE “naturally” leads to flatter minima despite not explicitly optimizing sharpness. While this is plausible, the paper doesn’t directly measure curvature or sharpness of minima, so this particular point remains a bit informal argument.", "methods_and_evaluation_criteria": "The proposed methods are well-aligned with the DG problem setting. GENIE’s optimizer design is appropriate because it directly tackles a known challenge in DG: avoiding overfitting to source-specific features by modulating how each parameter learns.  \n\nThe approach is conceptually sound – using preconditioning to scale parameter-wise gradients based on their signal-to-noise ratio (GSNR) ensures that no parameter with high variance or spurious signal gets overly large updates. This is analogous to adaptive optimizers like Adam adjusting for gradient variance, but here it’s done with a generalization-focused objective.  \n\nIntroducing noise injection and a random mask (dropout) on gradients is also sensible for DG. These add stochasticity and regularization to escape narrow minima and reduce reliance on any single feature.  \n\nThe evaluation criteria and settings are appropriate and rigorous. The authors adhere to the standard DomainBed evaluation protocol, which is a well-accepted framework for DG comparisons. Furthermore, baselines include both generic optimizers (SGD, Adam variants, SAM) and DG-tailored optimizers like FAD and GAM, as well as DG algorithms like IRM, CORAL, RSC integrated with standard optimizers. This comprehensive evaluation is appropriate for demonstrating GENIE’s effectiveness. \n\nThe only slight critique in methodology is the treatment of statistical significance and variability. The paper reports mean accuracies but does not mention standard deviations or confidence intervals. DG results can have high variance across training runs, so typically multiple runs are averaged. It’s not explicitly stated if results are averaged over several trials or a single run (though DomainBed usually averages over 3 seeds). Aside from that, the chosen benchmarks, baselines, and protocols are the gold standard for this problem, making the valuation criteria appropriate and convincing.", "theoretical_claims": "The paper provides theoretical analysis to support GENIE’s design, including proofs and derivations in Appendix. I think that the main theoretical contributions are Corollary 3.2, Corollary 3.3, and Theorem 3.9. \n\nCorollary 3.2 modifies the original OSGR formulation from Liu et al. 2020 (Theorem 3.1) by adding a per-parameter preconditioner. By setting $p_j = \\frac{1}{E[g_j^2],(r_j + 1/n)}$, it equalizes each parameter’s influence. Corollary 3.3 shows that GENIE’s OSGR is higher than or equal to that of SGD and ADAM. Theorem 3.9 addresses convergence, stating that under standard assumptions (bounded gradients, Lipschitz smoothness, non-zero gradient noise), the average gradient norm under GENIE’s updates decays on the order $O(T^{-1/2})$. \n\nOverall, the theoretical claims seem sound, and I did not find any obvious errors in the proofs. It is important to note that the assertion “uniform OSGR leads to better generalization” is presented as a conjecture—it guides the algorithm design but is not rigorously proven. This is acceptable as a motivational idea rather than a strict claim.", "experimental_designs_or_analyses": "The experimental design is comprehensive and well thought-out. The authors run experiments on multiple datasets (5 benchmarks) and settings to cover different aspects of domain generalization. On average, GENIE is empirically proven to provide better generalization than other optimizers (Tables 2 and 3).  \n\nThe ablation on the PACS dataset (Table 6) is sound, systematically toggles Preconditioning, Noise, and Mask components. The results show that Preconditioning alone yields most of the gain. \n\nThe experimental design could be further strengthened by reporting variability (e.g., results over 3 independent runs as DomainBed usually does). It appears they might have used fewer hyperparameters tuning trials than DomainBed defaults due to computational limits, but I could not find any report if each result is an average of multiple runs or a single best run.", "relation_to_broader_scientific_literature": "The paper’s contributions fit well into the evolution of DG methods: early work focused on invariances and domain adversarial training (MTAE, DANN), then came gradient-based regularization (IRM, VREx, RSC), and now we see optimization-level interventions (SAM, FAD, GENIE). In my opinion, the idea of using one-step generalization ability as a guiding principes is novel in DG.  \n\nOSGR as a metric came from prior work, but applying it to actively adjust training (as an optimizer) is a fresh contribution that pushes the literature toward thinking about “how” the model learns, not just “what” it learns.", "essential_references_not_discussed": "I did not find major omissions of essential references.", "other_strengths_and_weaknesses": "Strengths\n\nOriginality: The paper takes a fresh approach by introducing a new optimizer specifically for domain generalization. Optimizer-level solutions in DG are still relatively rare, so GENIE offers a fresh perspective. \n\nSignificance: The empirical gains, while moderate on average (~2-3% over strong baselines), are consistent and achieved on challenging benchmarks. Hitting a new state-of-the-art on DomainBed’s suite (with a simple plug-in optimizer) is significant for the DG community. Moreover, the method’s success in single-domain generalization (which many algorithms can’t handle) is a notable achievement – it suggests GENIE is capturing something fundamental about robust learning. If these results hold, GENIE could become a go-to optimizer for any training models for unknown target domains. \n\nWeaknesses\n\nComplexity and Practicality: One potential weakness is the added complexity of the optimizer. GENIE introduces several hyperparameters (preconditioning factors via momentum/variance decay, noise scale, dropout probability) and requires tracking second moment estimates per parameter (like Adam). Practitioners might need to tune the noise scale or dropout probability for different tasks.  \n\nLack of Direct Analysis or Feature Invariance: The paper claims GENIE promotes domain-invariant features, but it doesn’t directly evaluate this claim by examining feature representations. For example, some DG papers use metrics like center divergence between domain features or visualization of learned features.", "other_comments_or_suggestions": "Typos and Wording: \n- optionaly --> optionally \n- randam mask --> random mask \n- what’s meant by “... enough for suppress enough ...” at P8?", "questions_for_authors": "1. Did you run multiple trials with different random seeds for training, and if so, how consistent were the results for GENIE and other optimizers? \n2. Could you clarify how hyperparameter search was done for GENIE vs other baselines? \n3. The noise injection uses a factor $1 - \\tanh(1/\\sigma^2_t)$ to scale Gaussian noise. Why was this specific form chosen? \n4. What dropout probability $p$ do you use for the random mask on gradients? Is it the same across all experiments (and all layers/parameters)? And did you need to tune this probability?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aidtOzt8ni", "reviewer": "Reviewer_MLrB", "summary": "The paper presents GENIE (Generalization-ENhancing Iterative Equalizer), an optimizer aimed at improving domain generalization (DG) by using the One-Step Generalization Ratio (OSGR). GENIE dynamically equalizes the contribution of each parameter to loss reduction, preventing overfitting to domain-specific features. The optimizer incorporates preconditioning, noise injection, and random dropout to stabilize updates and promote domain-invariant feature learning. Experimental results show that GENIE outperforms existing optimizers like SGD, Adam, and SAM on DG benchmarks, offering improved generalization without requiring changes to model architecture. GENIE is applicable to various tasks such as DG and single-domain generalization.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful and constructive review. We greatly appreciate your insightful comments and the opportunity to clarify several important points you raised.\n\n# Proof Clarity\nYou suggested additional clarity in the proof derivations, particularly regarding Theorem 3.1. Theorem 3.1 originates from Liu et al. (ICLR 2020), as mentioned in our paper. To maintain brevity, we omitted detailed derivations in the main text. However, in Appendix B.2, we provided a rigorous derivation of Corollary 3.2, which generalizes and clarifies Theorem 3.1. We will ensure this is clearly highlighted in the revised manuscript.\n\n# Impact of Preconditioning on OSGR\nRegarding your point on clarifying the impact of preconditioning, Section 3.3.1 (Generalization Analysis) theoretically demonstrates that preconditioning increases OSGR values based on Jensen's inequality. Empirically, Figure 2 shows that GENIE’s preconditioning consistently achieves higher OSGR compared to other optimizers.\n\nAdditionally, Table 6's ablation study demonstrates how preconditioning, when combined with dropout and noise injection, enhances generalization performance. Figure 1 further illustrates that GENIE yields a more uniformly distributed gradient update across parameters, clearly supporting our claim that preconditioning effectively balances parameter contributions and improves generalization.\n\n# Comparison with Other Domain Generalization Techniques\nIn our generalization analysis, we show that the proposed preconditioning—designed to uniformize the OSGR value—leads to improved generalization performance, thereby mathematically supporting our conjecture. To further address the reviewer’s concern regarding comparisons with other methods, we provide a theoretical comparison with Sharpness-Aware Minimization (SAM).\n\nWe leverage PAC-Bayesian theory as follows:\n$$\nE_{\\mathcal{S}}\\, E_{\\theta \\sim \\widetilde{p}} \\left[ R(\\theta) \\right]\n\\le\nE_{\\mathcal{S}}\\, E_{\\theta \\sim \\widetilde{p}} \\left[\nL(\\theta) + \\frac{\\lambda C^2}{8n} + \\frac{\\mathrm{KL}(\\widetilde{p} \\| \\pi)}{\\lambda}\n\\right]\n$$\n\nPAC-Bayesian theory bounds the expected generalization risk by the sum of the empirical risk, a complexity term, and the KL divergence between the posterior $\\tilde{p}$ and prior $\\pi$.  The SAM optimizer focuses on reducing the empirical risk under perturbed parameters (i.e., minimizing local sharpness), while ignoring the KL divergence term. In contrast, our method explicitly improves the generalization bound by minimizing the KL divergence term in a one-step formulation.\n\nIn our setting, we define the posterior (the updated parameter distribution) as $\\tilde{p} = \\mathcal{N}(\\theta_t, \\Sigma)$ and the prior (previous parameter distribution) as $\\pi = \\mathcal{N}(\\theta_{t+1}, \\Sigma)$. Here, the prior can be treated as a data-driven prior which is approximated using all data excluding the current mini-batch. Parameter distributions depend on the distribution of the gradient update with the effective learning rate $\\frac{1}{\\mathrm{E}[g_j^2]}$.\n\nTaking the derivative of the KL divergence with respect to the gradient-based update, we obtain:\n$$\nKL(\\tilde{p} \\| \\pi) = \\frac{1}{2} \\left[ \\sum_{i=1}^{J} \\frac{\\sigma_i^2}{\\sigma_i^2} + \\sum_{i=1}^{k} \\frac{(\\theta_{t+1,i} - \\theta_{t,i})^2}{\\sigma_i^2} - J + \\sum_{i=1}^{J} \\log\\left( \\frac{\\sigma_i^2}{\\sigma_i^2} \\right) \\right] = \\frac{1}{2} \\sum_{j=1}^{J} \\frac{(\\theta_{t+1,j} - \\theta_{t,j})^2}{\\sigma_j^2}\n$$\n$$\n\\left[ \\nabla_j \\mathrm{KL}(\\tilde{p} \\| \\pi) \\right]\n= \\frac{(\\theta_{t+1,j} - \\theta_{t,j})}{\\sigma_j^2} \n= \\frac{1}{\\mathbb{E}[g_j^2]} \\cdot \\frac{g_{j,t}}{\\sigma_j^2}\n= \\left( \\underset{\\mathrm{GENIE}}{ \\frac{1}{\\mathbb{E}[g_j^2]} \\cdot \\frac{g_{j,t}^2}{\\sigma_j^2} } \\right) \\cdot \\mathrm{sign}(g_{j,t})\n$$\nThis formulation shows that our preconditioning term directly reduces the KL divergence term, thereby contributing to a tighter PAC-Bayesian generalization bound—a benefit not provided by the SAM optimizer.\n\nWe sincerely thank you for your detailed and insightful comments, which have significantly helped us strengthen our paper. We hope these clarifications fully address your concerns and further highlight the robustness and contributions of GENIE.", "claims_and_evidence": "Yes, the claims made in the submission are generally supported by clear and convincing evidence.\n\nGENIE's Effectiveness: The paper claims that GENIE outperforms existing optimizers like SGD, Adam, and SAM in domain generalization tasks. This claim is supported by extensive empirical evidence from experiments on several widely-used domain generalization benchmarks (PACS, VLCS, OfficeHome, etc.). The results show that GENIE consistently achieves higher performance than these optimizers, which provides strong evidence for its effectiveness.\n\nOSGR as a Valid Metric: The paper claims that the One-Step Generalization Ratio (OSGR) effectively measures the generalization capacity of the optimizer. The authors support this claim with both theoretical analysis and empirical validation, demonstrating that GENIE's preconditioning leads to a higher OSGR, which correlates with better generalization.\n\nPreconditioning and Parameter Balance: The claim that GENIE's preconditioning strategy promotes balanced parameter updates and mitigates overfitting is backed by a detailed theoretical framework and experimental results. The authors present a clear rationale for how the preconditioning factor adjusts OSGR dynamically, leading to more stable and robust learning.\n\nComputational Efficiency: The paper claims that GENIE is computationally efficient, offering faster training times compared to SAM. The evidence provided, in the form of training time comparisons, supports this claim by showing that GENIE achieves better performance in less time, especially in comparison to optimizers like SAM, which require more computational overhead.\n\nPotential Issues:\n\nGeneralization to Other Datasets: While GENIE performs well on several benchmarks, the paper does not address its performance on datasets with more extreme or non-standard domain shifts. Testing GENIE on additional challenging datasets might provide more comprehensive evidence of its generalizability.\n\nImpact of Preconditioning: The effectiveness of the preconditioning term is theoretically explained but could benefit from further clarity or empirical validation on how it compares to other domain generalization techniques. Additional experiments could clarify the impact of preconditioning in different scenarios.\n\nOverall, the claims are well-supported, with clear experimental results and solid theoretical backing. However, some additional tests and more detailed comparisons with other methods could further strengthen the evidence.", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluation criteria make sense for the problem at hand.\n\nProposed Method (GENIE): The proposed method, GENIE, is well-suited for addressing the challenges of domain generalization (DG). By incorporating the One-Step Generalization Ratio (OSGR) and using preconditioning, noise injection, and random dropout masks, the method tackles the problem of domain-specific overfitting and promotes robust, domain-invariant feature learning. This approach is aligned with the core issue in DG, which is ensuring that models generalize well to unseen domains without overfitting to spurious correlations.\n\nEvaluation Criteria (OSGR and Benchmark Datasets): The use of OSGR as an evaluation metric is appropriate, as it directly quantifies the contribution of model updates to generalization. The paper demonstrates that higher OSGR correlates with better generalization performance, making it a valuable tool for assessing domain generalization methods. Additionally, the use of widely accepted benchmark datasets like PACS, VLCS, and OfficeHome is a sensible choice, as these datasets are commonly used in DG research and provide a solid basis for comparison with other methods.\n\nOverall, the proposed methods and evaluation criteria are both relevant and effective for tackling the problem of domain generalization and ensuring that the results are comparable with existing approaches in the field.", "theoretical_claims": "Theoretical claims in the paper are based on sound reasoning, and the authors provide a thorough mathematical foundation for the proposed GENIE optimizer. However, as a reviewer, I have not verified the correctness of the proofs in detail. The paper includes several theoretical claims, such as the connection between OSGR and parameter-wise statistics (Theorem 3.1), convergence analysis (Theorem 3.9), and the impact of preconditioning on OSGR (Corollary 3.2).\n\nTheorem 3.1 establishes the relationship between gradient updates and generalization, introducing the concept of Gradient Signal-to-Noise Ratio (GSNR). This is a critical part of the theoretical framework, but it would be beneficial to further break down the derivations to ensure all steps in the proof are clear and rigorous.\n\nTheorem 3.9 provides a convergence rate analysis under certain assumptions. While the theorem is mathematically sound, the assumptions (such as bounded gradients and smooth loss functions) are typical for optimization in deep learning. However, some of these assumptions may not always hold in all practical scenarios, and this could potentially limit the generalizability of the theoretical results.\n\nCorollary 3.2 discusses how preconditioning affects OSGR. The logic behind this corollary is solid, but again, further clarification and empirical validation of how preconditioning impacts OSGR in various scenarios would strengthen the argument.\n\nOverall, the theoretical claims are well-founded and supported by proofs, but a deeper verification and potential clarification of the proof steps would be helpful for ensuring their correctness.", "experimental_designs_or_analyses": "Yes, the experimental designs and analyses presented in the paper generally appear sound and valid. The authors conduct several key experiments to evaluate the effectiveness of the GENIE optimizer, and the results are based on widely accepted benchmark datasets, which ensures that the findings are relevant and comparable to existing work in the domain.\n\nEvaluation on Benchmark Datasets: The authors evaluate GENIE on well-established datasets such as PACS, VLCS, and OfficeHome. These datasets are commonly used in domain generalization (DG) research, which makes the experimental setup reliable. The results consistently show that GENIE outperforms other optimizers like SGD, Adam, and SAM. This approach provides a robust comparison of GENIE's performance across multiple standard datasets, ensuring the validity of the results.\n\nComparison with Other Optimizers: The paper compares GENIE with various baseline optimizers, including SGD, Adam, AdamW, and SAM. This is a solid experimental design, as it allows the authors to clearly show how GENIE improves performance over existing methods. The statistical significance of these comparisons could be strengthened if the authors provided more detailed metrics, such as p-values, to further support the claim that GENIE outperforms these optimizers.\n\nTraining Time and Computational Efficiency: The authors also conduct experiments to compare training times across different optimizers. GENIE consistently performs better than SAM in terms of training time, which supports the claim that it is computationally more efficient. However, additional experiments on more complex models or larger datasets could provide further insight into the scalability and efficiency of GENIE in real-world applications.\n\nAblation Study: The paper includes an ablation study to examine the impact of GENIE's individual components, such as preconditioning, noise injection, and random dropout masks. This is a useful and rigorous analysis that helps to isolate the effects of each component. The ablation study design is valid, and it clearly shows that combining all components leads to the best performance.\n\nPotential Issues:\n\nLimited Number of Datasets: While the paper evaluates GENIE on several standard datasets, testing on a broader range of real-world datasets with more diverse domain shifts could further validate the generalizability of the method.\nLack of Statistical Analysis: The experimental results show that GENIE outperforms other optimizers, but including more statistical analysis (e.g., significance testing) would strengthen the argument for its superiority.\nOverall, the experimental designs and analyses are robust and well-structured, but additional experiments and statistical analysis could provide further confidence in the generalizability of the findings.", "relation_to_broader_scientific_literature": "The key contributions of this paper build on and extend previous work in the field of domain generalization (DG) and optimization methods. Specifically, the introduction of the GENIE optimizer addresses long-standing challenges in DG, particularly the issue of domain-specific overfitting and generalization across unseen domains.\n\nConnection to Domain Generalization (DG): Previous studies in DG (e.g., Carlucci et al. (2019), Motiian et al. (2017)) have primarily focused on designing models that can generalize well across different domains. However, many of these methods have struggled with issues such as spurious correlations or domain-specific biases. This paper's novel use of the One-Step Generalization Ratio (OSGR) as a metric and optimization strategy represents a step forward by offering a more targeted approach to balance parameter updates, which helps mitigate overfitting to domain-specific features. This builds upon the concept of meta-learning and adversarial training for generalization but introduces an optimization-focused solution.\n\nRelation to Optimization Methods: GENIE also relates to advancements in optimization techniques. Prior works on optimizers like Adam and SAM (e.g., Kingma and Ba (2014), Zhang et al. (2020)) have shown success in improving convergence rates and robustness. However, these methods still face limitations when applied to DG tasks, particularly in dealing with domain shifts. The preconditioning strategy used in GENIE aligns with concepts from adaptive optimizers (like AdaGrad and RMSProp) but offers a more sophisticated approach to achieving balanced gradient updates, making it an important advancement in the optimization landscape.\n\nAblation and Empirical Results: The paper also contributes to the ongoing research on ablation studies by isolating the effects of specific components such as noise injection, preconditioning, and dropout. Previous works on DG often combine techniques without isolating their individual impacts, making this paper’s clear ablation study a valuable contribution to understanding how each factor influences model performance.\n\nIn conclusion, the paper makes significant strides in improving domain generalization through its novel optimizer, GENIE, while also contributing to the broader scientific literature on optimization strategies and domain generalization techniques. It provides a clearer understanding of how optimization approaches can be tailored to address specific challenges in DG and offers insights that can benefit future work in both optimization theory and practical applications in machine learning.", "essential_references_not_discussed": "While the paper does an excellent job of citing relevant literature, there are a few related works that could strengthen the context for the key contributions, particularly in the areas of domain generalization (DG) and optimization methods. These works are essential for providing a more comprehensive background and for highlighting how the proposed GENIE optimizer fits within the broader research landscape.\n\nDomain Generalization (DG) Approaches:\nThe paper primarily cites Carlucci et al. (2019) and Motiian et al. (2017), which are foundational in DG, but it misses recent advancements that directly relate to optimizing domain generalization through optimization techniques. For example, Li et al. (2020) introduced Meta-Regularization, which adapts optimization methods for generalization across domains, and this work could provide additional context for the proposed approach. Citing such works would emphasize how GENIE contributes to a broader trend of improving DG through more robust optimization methods.\nOptimization for Generalization:\nThe work on Stochastic Gradient Descent (SGD) and Adam is well covered, but the paper could mention the more recent work on SAM (Sharpness-Aware Minimization) (e.g., Foret et al., 2021), which directly relates to balancing generalization and optimization. SAM has been shown to improve the robustness of models by minimizing sharp minima, which is related to the objectives GENIE aims to achieve with preconditioning. Additionally, there is growing interest in Adaptive Optimization techniques, such as AdaBelief (Zhang et al., 2020), which adaptively correct the update rule based on gradient estimates, and could complement the discussion on GENIE’s preconditioning strategy.\nAblation Studies in Optimization:\nThe ablation study in the paper is an important contribution, but there is no direct mention of ablation studies in optimization, which have been extensively used in understanding the impact of various components in optimizers. For example, the work by Liu et al. (2021) on the effectiveness of gradient clipping and noise injection in improving model generalization could be cited as it deals with similar experimental setups that focus on mitigating overfitting through different optimization techniques.", "other_strengths_and_weaknesses": "None.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The three reviewers acknowledge the importance of this work and recommend acceptance. The reviewers nonetheless raised some questions, most of which were satisfyingly addressed by the authors' feedback. The remaining point to address would be the application of the proposed optimization strategy to other tasks, but the AC acknowledges that, as mentioned by the authors, this may require too much time to do so in the limited timeframe. Altogether, this work is acknowledged as a solid contribution, and the authors are encouraged to incorporate elements of their feedback in the final version.
Decision: Accept (oral)