Exploring Mode Connectivity in Krylov Subspace for Domain Generalization
Authors: Aodi Li
Liansheng Zhuang
Xiao Long
Houqiang Li
Shafei Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Loss landscape
mode connectivity
Krylov space
domain generalization
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: This paper explores the geometric characteristics of loss landscapes to enhance domain generalization (DG) in deep neural networks. 
Existing methods mainly leverage the local flatness around minima for improved generalization. However, recent theoretical studies indicate that flatness does not universally guarantee better generalization. Instead, this paper investigates a global geometrical property for domain generalization, i.e., \emph{mode connectivity}, the phenomenon where distinct local minima are connected by continuous low-loss pathways. Different from flatness, mode connectivity enables transitions from poor to superior generalization models without leaving low-loss regions. 
To navigate these connected pathways effectively, this paper proposes a novel Billiard Optimization Algorithm (BOA), which discovers superior models by mimicking billiard dynamics. 
During this process, BOA operates within a low-dimensional Krylov subspace, aiming to alleviate the curse of dimensionality caused by the high-dimensional parameter space of deep models. 
Furthermore, this paper reveals that oracle test gradients strongly align with the Krylov subspace constructed from training gradients across diverse datasets and architectures. 
This alignment offers a powerful tool to bridge training and test domains, enabling the efficient discovery of superior models with limited training domains.
Experiments on DomainBed demonstrate that BOA consistently outperforms existing sharpness-aware and DG methods across diverse datasets and architectures.
Impressively, BOA even surpasses the sharpness-aware minimization by 3.6\% on VLCS when using a ViT-B/16 backbone.

Published as a conference paper at ICLR 2026
EXPLORING MODE CONNECTIVITY IN KRYLOV SUBSPACE FOR DOMAIN GENERALIZATION
Aodi Li', Liansheng Zhuang!.200 (2 Xiao Long', Houqiang Li & Shafei Wang?
School of Cyber Science and Technology, University of Science and Technology of China 2 National Engineering Laboratory for Brain-Inspired Intelligence Technology and Applications, University of Science and Technology of China *Pngg Cheng Laboratory, Shenzhen, China aodi 1 @maa 1. ustc. edu. cn, 1szhuang@ustc. edu. cn
ABSTRACT
This paper explores the geometric characteristics of loss landscapes enhance domain generalization (DG) in deep neural networks. Existing methods mainly leverage the local flatness around minima for improved generalization. However, recent theoretical studies indicate that flatness does not universally guarantee bet- ter generalization. Instead, this paper investigates global geometrical property for domain generalization, i.e., mode connectivity, the phenomenon where distinct local minima are connected by continuous low-loss pathways. Different from flat- ness, mode connectivity enables transitions from poor to superior generalization models without leaving low-loss regions. navigate these connected pathways effectively, this paper proposes novel Billiard Optimization Algorithm (BOA), which discovers superior models by mimicking billiard dynamics. During this pro- cess, BOA operates within low-dinensional Krylov subspace, aiming allevi- ate the curse of dimensionality caused by the high-dimensional parameter space of deep models. Furthermore, this paper reveals that oracle test gradients strongly align with the Krylov subspace constructed from training gradients across diverse datasets and architectures. This alignment offers a powerful tool to bridge training and test domains, enabling the efficient discovery of superior models with lim- ited training domains. Experiments on DomainBed demonstrate that BOA con- sistently outperforms existing sharpness-awes and DG methods across diverse datasets and architectures. Impressively, BOA even surpasses the sharpness-awan minimization by 3.6% on VLCS when using ViT-B/16 backbone.
INTRODUCTION
Understanding the geometry of loss landscapes (Li at. al. 2018a Xu al. 2024) in deep neural networks has emerged
0.55
144
40
40
30
0.65
30
0.96
powerful and insight- ful approach for interpreting the gen- eralization behavior of these mod- els (Wu et al. [0117] Rangamani et al. 2020). Recent empirical and theoreti- cal studies (Keskar et al. 2016 Dziu- gaatt & Roy| 2017 Jingg
0.35
0.48
10
10
0
10
20
30
60
0.05
10
20
30
40
0.00
(a) Training Loss Landscape
(b) Test Loss Landscape
2019)
Figure 1: Ideal and non-ideal models in the training and test loss landscapes. (a) In the training loss landscape, the ideal (*) and non-ideal (*) models are connected via a low loss path. (b) The ideal model (*) located in the basin of the test loss landscape, while the non-ideal model (*) is not.
have demonstrated a strong connec- tion between generalization perfor- mance and the geometric properties
of the loss surface, particularly its sharpness. It has been observed that flat minima sought by sharpness-aware minimization (SAM)
tend to generalize better than sharp ones (Foret et al. 2020 Kwon et al. 2021 Kim et al. 2022
This underscores the potential of leveraging insights from loss geometry.
Although numerous flatness- seeking methods have been developed to learn generalizable mod-
els (Kwon al. 2021 Kim et al. 2022 ZZhan et al. 2024), emerging research (Zhang al. 2021

---

Published as a conference paper at ICLR 2026
Table 1: Comparisons with other optimization algorithms in update subspaces and rules. The update direction of SGD/SGLD lies in random subspace R: SAM lies in two-dimensional Krylov subspace K2; and our BOA lies in a K K-dimensional Krylov subspace Kg.
Methods
Involving Subspaces
Update Rules
SODISGLD Welling
2011
{g. a N( J).
A0 -9 + 22T=
SAM (Furet 2020
KG = ig Hy). Hg a (e vnie
A0- V + eg)
Ei. 119.
BOA (Ours)
Kx = Hg......... H 9).
Wen et al. 2023) indicates that flatness does not universally guarantee improved generalization. This reveals limitations in over-relying on local sharpness measures, particularly within challenging Domain Generalization (DG) settings. In the DG task, our goal is to learn model from multiple source domains and generalize to unseen target domains with different data distributions (Wang et al. 2022 Zhou et al. 2021; Shen et al., 2021). In such scenarios, the connection between flatness and generalization becomes significantly more complex and context-dependent. Crucially, the rela- tionship between flatness in parameter space and the geometry of the representation space suggests that low sharpness may not adequately capture a model's vulnerability to domain shifts in feature space /Andrrushchenko et al.] 2023). Thus, although pursuing flat minima can be beneficial, it should not be considered a unique solution for DG. It is essential to investigate other geometric properties of the loss landscape beyond flatness to achieve better generalization.
In contrast to local flatness, this paper focuses on a global geometric property termed mode connec- rivity (Freeman & Bruna 2016 Garipovv al. 2018 Draxler al. 2018). That is, distinct local min- ima, discovered via independent training from different initializations, are connected by continuous pathways of low loss. This paper empirically investigates this phenomenon in the context of domain generalization (DG) and observes a similar connectivity: As visually illustrated in Figure D basins corresponding to models with significantly different out-of-domain performance (e.g., non-ideal model with poor DG accuracy and an ideal model with nearly 100% DG accuracy) are connected via low-loss pathways. It suggests that the loss landscape is not composed of isolated basins but exhibits connected structure among separate solutions. This connectivity implies promising possibility: transitioningg from a solution with poor generalization properties to one with strong out-of-domain performance without escaping the low-loss region. Despite its theoretical promise, effectively nav- igating these pathways in high-dimensional parameter spaces remains challenging due to the curse of dimensionality. Conventional optimization methods, such as SGD and its Langevin dynamics ex- tensions (Bussi & Parrinello 2007 Welling & Tch [011). are often ineffective in this context, as they tend become trapped in local regions (Deng et al. 20200] Zheng et al.] 2024), preventing the discovery of superior solutions. This problem motivates the need for more deliberate algorithm capable of actively navigating these low-loss tunnels.
Inspired by the dynamics of billiard motion (Bunimovich 2007 Gutkin 2003). this paper proposes a novel and efficient traversal algorithm, Billiard Optimization Algorithm (BOA), which explicitly encourages movement along these connected pathways for improved domain generalization. Specif- ically, BOA operates through two core operations: (1) line search to locate loss contour boundaries (analogous to billiard ball approaching a cushion), and (2) reflection to redirect the optimiza- tion trajectory upon boundary contact (mimicking momentum pretnrrrreeeennnneee bounces). During this process of high-dimense digeniinenio optimization, the curse of dimensionality profoundly impacts BOA's performance in two key ways. First, high-dimensional theory (Vershynin] (018)) suggests that ran- dom vectors become nearly orthogonal to the optimal initial directions (such directions of oracle negative test gradients), making naive random direction search highly inefficient in locating useful paths. Second, trajectory sparsity emerges in -igeenimensional landscapes, meaning that numerous optimization steps are required to achieve sufficient domain generalization. These issues are com- pounded in deep models, where the parameter space can be overwhelmingly large. Fortunately, key geometric regularity identified in our empirical studies offers a solution: the test gradient exhibits strong alignment with the Krylov subspace (Liesen & Strakos (003)) derived from training gradi- ents across different datasets and architectures. The alignment effectively bridges the gap between training and test domains, and provides opar-optimal initial directions for BOA without requiring access to test data. Furthermore, by leveraging this alignment. BOA can constrain its traversal tra- jectory to a reduced subspace of merely 5-20 dimensions, drastically reducing the vast search space and enabling efficient discovery of models with superior domain generalization capabilities. Empiri-
2

---

Published as a conference paper at ICLR 2026
cal validation conducted on the challenging DomainBed (Gulrajani & Lopez- Paz 2020) benchmark with various vision transformer architectures demonstrates that BOA consistently outperforms pop- ular sharpness-awand methods and other prevalent DG techniques across five diverse datasets. These results underscore the effectiveness of BOA in navigating high-dimensional loss landscapes and its practical utility for real-world domain generalization tasks.
Our key contributions can be summarized as follows:
Mode connectivity is identified in the DG context. It describes phenomenon whereby continu- ous low-loss trajectories in the parameter space connect models exhibiting substantially divergent out-of-domain performance.
A novel and efficient Billiards Optimization Algorithm (BOA) introduced to advance domain generalization. It promotes navigation along low-loss paths connecting distinct local optima, facilitating the identification of models with enhanced generalization capabilities.
Our study reveals the notable geometric regularity that test gradients demonstrate significant alignment with the Krylov subspace derived from training gradients, thereby establishing an effective bridge between training and unseen test domains.
Experiments across diverse architectures confirm that BOA consistently surpasses popular sharpness aware methods and other DG techniques on five datasets of DomainBed.
RELATED WORK
Although deep learning has achieved success in many application areas, the loss landscapes of deep neural networks remain inadequately understood. This area constitutes an actively evolving field of research, primarily divided into two distinct categories.
Local structure. The first category explores the local structure of minima found by SGD and its variants. Researchers have observed that smaller mini-batch sizes often lead to sharp minima, while larger mini-batch sizes tend to yield flat minima (Keskar et al.] 2016). In recent years, numer- studies have established a connection between flatness near minimizers and model general- ization (Keskar et al. Dziugaite & Roy] 2017] Jiang et al.) 2019):: flat minima exhibit stronger generalization capabilities, whereas sharp minima perform poorly on test datasets. This conclusion has also been extended to out-of-distribution generalization scenarios (Zou et al. 2024), attracting widespread attention in the domain generalization research community and inspiring a series of do- main generalization methods that seek flat minima, such as SWAD (Cha et al. 2021). SAM (Foret al. 2020) and its variants (Wang al. 2023 Li al. 2025). Unfortunately. however, flatness does not equate to generalization. It is because: (1) There exist sharp yet well-generalizing models. Dinh et al. (Dinh et al., 2017) and Kaiyue Wen et al. (Wen et al. (023) have theoretically and empirically identified minimizers that generalize well despite being very sharp. (2) While the generalization
benefits of the flattest minimizers have been theoretically established for simple linear models, this classical result does not readily transfer standard neural networks, including relatively simple chitectures such as two-layer ReLU networks (Wen et al. 2023). Unlike previous DG methods, this paper leverages the connectivity of minimizers and attempts to traverse from any given minimizer to one with strong generalization capability by simulating billiard dynamics.
Global structure. The other major category of research focuses the global structure of minima. Over the past few years, numerous studies have revealed connectivity between minima, known as mode connectivity. Freeman el al. (Freeman & Bruna 2016) theoretically proved that half-rectified single-layer networks are asymptotically connected. Their theoretical results are not easily general- izable multi-layer networks. contrast, Garipov al. (Garipov et al., 2018)) proposed a simple training procedure that finds pathways with nearly constant accuracy even in various modern state- of-the-art architectures, with only one bend between optima. Draxler er al. (Draxler al.) 2018) simultaneously and independently discovered curves connecting local optima in DNN loss land- scapes. They adopted a different approach to finding these curves, inspired by the "Nudged Elastic Band" method from quantum chemistry. Recently, mode connectivity is used in algorithm design across multiple application areas, such as machine unlearning (Cheng & Amiri 2025), model merg- ing (Li et al. 2024), and continual learning (Mirzadeh al. 2020). Although mode connectivity offers the potential to find models with stronger generalization capabilities continuously, this prop- erty has not been effectively utilized in the design of out-of-distribution generalization algorithms
3

---

Published as a conference paper at ICLR 2026
For small
For hariagcc
122
104
2(g) in.n
80
(a) Billiard Game
(b) Line Search
(c) Reflection
Figure 2: Overview of the proposed Billiard Optimization Algorithm (BOA). (a) A schematic di- agram of a billiards game on two-dimensional table; (b) The first operation of BOA (Line Search): Like ball approaching a cushion, this operation identifies loss contour boundaries through line search along directional vectors. (c) The second operation of BOA (Reflection): Upon boundary contact, the trajectory redirects via physies-based rules, mimicking a ball's reflection off cushion.
due to the curse of dimensionality. To the best of our knowledge, this paper the first work seek models with better out-of-domain generalization via mode connectivity.
METHODOLOGY
Our Billiard Optimization Algorithm (BOA) aims to effectively navigate connected pathways within loss landscapes and discover models with superior out-of-domain generalization. Its design draws inspiration from the physical dynamics of billiards, as illustrated in Fig. [2 Specifically, it iteratively performs two core operations: (1) line search locate the loss contour boundary, analogous to a billiard ball moving toward a cushion (Fig. 2b). and (2) reflection to redirect the optimization trajectory upon reaching the boundary, mimicking ball's reflection off cushion (Fig. (c)..2c). Due to the space limit, formalized sketch describing the complete procedure of the Billiard Optimization Algorithm (BOA) provided in Appendix A.
3.1 DEFINITION
In the Billiard Optimization Algorithm (BOA), the conceptual "billiard table" is mathematically defined as the sub-level of the training loss landscape bounded by specified contour threshold. Formally, this table corresponds to the parameter domain:
( Rd Craii (0) S (m)
where the contour threshold ln is deliberately constructed as: Lm = Luain + Ae. Here, Og represents the parameter vector of model trained with an ERM/SAM warm-up. and Ae denotes strictly positive loss increment. This construction guarantees that the warm-up model resides strietly within the interior of the billiard table, satisfying the containment condition: {(0p) < Ln. This foundational definition establishes the optimization landscape bounded playfield where subsequent billiaddiinspredd operations (contour-secking line searches and comnentum-preservviie reflections) are performed. The bounded nature of the playfield effectively constrains variations in training loss, ensuring that performance on the training set remains nearly constant while the algorithm searches for model parameters that deliver superior DG performance.
3.2 LINE SEARCH
During the line search phase of the Billiard Optimization Algorithm (BOA), we emulate the linear trajectory of billiard ball approaching a cushion by systematically locating the target loss contour boundary La = E(Oo) + Ae. Unless specified otherwise, we use & represent Grain below.
This procedure mathemattcally corresponds to solving the nonlinear equation with respect at:
((0 + ap:-1)
lab:
4

---

Published as a conference paper at ICLR 2026
Table Out-of-domain accuracies of ViT-B/16 on the DomainBed benchmark. The star symbol (+) marks experiments involving full fine-tuning of the image encoder, with its absence signifying the application of visual prompt tuning instead.
Algorithms
VLCCS
PACS
OfficeHome
Terralncognita
DomainNet
Avg.
ERM Vapnik
1998
81.6
93.2
80.0
54.5
57.7
73.4
MIRO (Cha 2022
83.6
95.8
82.3
58.8
57.2
75.5
ERM (Jia al. 2022
81.9
95.9
84.1
56.1
59.5
73.5
IRM (Arjovsky cl al
2019
82.9
96.1
83.2
56.7
59.1
75.6
DANN Ganin
2016
81.8
96.3
83.0
56.0
58.4
75.1
CDANN (L et al. 2018c
82.4
96.5
82.9
55.6
58.4
75.2
MMD (Li al. 20186
82.3
95.8
83.6
57.4
59.9
75.8
RSC (Huang
2020
82.2
96.5
83.2
58.2
59.0
75.8
CORAT (Sun Saenko
2016
82.6
96.4
83.8
57.5
59.8
76.0
IIB (Li et al. 2022b
82.3
96.5
84.2
58.2
58.6
76.0
SAM (Forel al. Z0201
82.9
96.6
85.4
56.2
59.8
76.2
GSAM (Zhuang et al. 2022
82.9
96.6
85.6
55.4
59.8
76.1
GAM ZZhang et al. 2023
83.6
96.4
85.5
55.3
59.5
76.1
SAGM Wang
2023
82.8
96.8
85.2
58.0
59.1
76.4
DISAM 7Zhang et al.
2024
82.7
97.1
85.4
57.3
59.8
76.5
BOA (Ours)
86.5
97.4
86.0
60.3
60.2
78.1
where 0 represents the current model parameter (analogous the billiard ball's current position) and p denotes the search direction. Candidate points are generated along the ray 24 0 + h&P:-1 (hg 0), with BOA employing an adaptive bracketing strategy efficiently isolate the solution interval [hz., hn]: when f(xx)) < lah step sizes expand exponentially via hx+1 (2 1) (where represents the initial step) until exceeding lni conversely, when {(xx)) > luh step sizes
contract geometrically through h2 / until falling below Lu. Following interval isolation, BOA employs the golden-section search method for precise refinement:
hk
hn
hR hr
(2) 1
hr
hR hr
where 1 25 represents the golden ratio. The interval boundaries [hr,, hg) are then updated based on function evaluations at points hig (1,2) Finally, the approximate optimal solution at equation (2), found via the golden-section search (Noori et al. 2025), used to update parameters:
0, 0 + a P1-1, precisely positioning the model parameter on the loss contour.
3.3 REFLECTION
Upon reaching the contour boundary, BOA initiates its second operation: reflection that mimics momentum-preserviing billiard collisions.
At the contour point 0,, the reflection direction is computed by leveraging the geometry of the local loss landscape, analogous to the physical reflection process illustrated in Fig. 2c this phase, the unit normal vector is expressed as: n; ( which corresponds to the steepest descent direction, plays role equivalent to the surface normal of the cushion in billiard dynamics. As depicted Fig. 2c both the incident direction p:-1 and the normal vector n, define the reflection plane. The specular reflection direction is subsequently updated follows:
ve(O,)
p. = P:-1 (pT_,n:)n (I 2n, P
(5)
Geometrically, this transformation subtracts twice the orthogonal projection of p;-1 onto This operation preserves the magnitude of momentum, i.c., |p = 1p:-1l8 and adheres to the re- flection law Pincident_ Pretlectdd which implies (P:-1, n,;) (p,, n2). The reflection operator
R(n,] = 2n,n, represents an improper rotation (with det(R) (R) = -1) that incorporates curva- ture information via n, x V((0.). This allows efficient exploration along the loss contour without
5

---

Published as a conference paper at ICLR 2026
the need for Hessian recomputation, thereby maintaining computational complexity of (dd) per iteration for d-dimensional parameters.
After multiple iterations of alternating line search and reflection operations, generate trajectory of model parameters and select an optimal model from this trajectory via a validation set for testing.
3.4 OVERCOMING THE "CURSE OF DIMENSIONALITY"
The curse of dimensionality fundamentally impacts optimization in high- high-dimensional parameter spaces through two primary mechanisms: (1) In billiard dynamics, the initial incident direction po determines the trajectory's ability to find model parameters with better DG performance. Unlike intuitive 2D billiard, the high-dimensional setting induces a geometric constraint: random vectors become nearly orthogonal to the oracle optimal directions (Vershynin) 2018), suggesting that naive random search becomes extremely inefficient in high dimensions. (2) Trajectory sparsity emerges in nigh-dimensional landscapes, necessitating numerous optimization steps to achieve sufficient out- of-domain generalization.
To overcome the challenges of nigh-dimensional optimization, we leverage the geometric structure of the Krylov subspace, which is generated from training derivatives:
KK (iinn Iunin) span (go. Hgo
H [=K-1g0]
where 9u = Virain(B) and H (riinii (fo). Krylov subspace (Liesen & Strakos 2013) is fa-
mous as an efficient, low-dimensional framework to find approximate solutions to high-dimensional linear algebra problems. Here, we use the Krylov subspace concerning the Hessian matrix to capture the dominant curvature information. It provides two critical advantages: First, it enables principled selection of the initial search direction through efficient approximation of test gradients. Second, it constrains the trajectory to subspace with limited dimensions that already includes model param- eters with better generalization.
Determine the initial incident direction. Intuitively, in the DG scenario, the negative test gradient -Vleaa(0)) might serve a good choice for the initial incident direction. Fortunately, empirical analysis reveals a remarkable geometric regularity: the test gradient bee (0p) exhibits strong align- ment with the Krylov subspace K H unin Pmmin). This alignment enables efficient approximation via setting proper values for 3y:
K-1 Pe Σ
B 8 gu (eees((o))
Interestingly, experimental evidence demonstrates that setting 3* = yields a particularly effective direction. This configuration achieves small angle between po and (est(()))) across benchmark datasets, effectively overcoming dimensional barriers. To avoid the computational burden of explicit Hessian calculation, we efficiently approximate sessian-vector products using a fiite-difference method:
Ve(Oo + ego) VE(O.)
Hgo
where the scalar is small step size that balances approximation accuracy and numerical enror. This method preserves (d) complexity while avoiding direct second-order derivative calculations.
Constrains the trajectory to limited dimensions. The Krylov alignment mentioned above also means the existence of better out-of-domain (OOD) solutions within the Krylov subspaces. To limit the reflection operation within the Krylov subspace, the reflection direction P; can be updated as follows:
p: = (I - 10-.n, 1P:-1
where n, projx, n. denotes the projection of n., within the Krylov subspace KK-
The Krylov alignment constitutes a "free lunch" in dighediiensional optimization, providing near- optimal initial directions without the availability of test data and reducing the search space substan- tially. This geometric regularity bridges the gap between training and test distributions, enabling effective exploration of loss landscapes for improved domain generalization.

---

Published as a conference paper at ICLR 2026
1.0
1.0
1.0
1.0
Eylorieaed
Kyfoviepeen
Cosine
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.0
2
4
5
6
8
9.
0.0
3
6
8
9.10
0.0
4 15667
8 10
0.0
2
3
10 5055500
7
8
Dimnnnsoos
Dimensions
DDieensions
(a) Caltech101
(b) LabelMe
(c) SUN09
(d) VOC2007
Figure 3: Cosine similarity between test gradients and their approximations derived from ei- ther random Krylov subspaces. Krylov(Optimal); This curve corresponds to the projection of the test gradients onto the Krylov subspace, representing the best possible approximation achievable under the given basis; Krylov(Approx): This curve corresponds to approximation obtained via the heuristic choice of 3y = in equation (7): Random: This curve depicts the theoretical expectation value of the projection length of the unit test gradients onto random subspaces (see Appendix B).
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTING
Datasets and Protocol. Our evaluation is conducted on five widely-used, challenging domain gen- eralization datasets: PACS (Li et al. 2017) (9,991 images, 4 domains and 7 classes), VLCS (Fang et al. 2013) (10,729 images, 4 domains and 5 classes), OfficeHome (Venkateswara et al. 2017) ) (15,588 images, 4 domains and 65 classes), Terralncognita (Beery al. (018)) (24,788 images,
domains and 10 classes), and DomainNet (Peng et al., 2019) (586,575 images, domains and 345 classes). Following the standardized DomainBed benchmark (Gulrajani & Lopez-Paz 2020),
implement a rigorous leave-one-domaing domain as the test set (i.c., test or target domain) while aggregating all remaining domains for train- ing (i.e., training or source domain). To ensure robust model selection and hyperparameter tuning, previous research typically reserves 20% of the data from each source domain to create a validation set. However, we observed that while models may attain similar validation accuracy, their DG per- formance can differ by over 10% at various points along the training trajectory. It suggests that a low-loss region of the validation set may contain models with widely varying generalization. Thus, relying on this set amounts to random guessing, necessitating the use of a test-domain validation set practical alternative. For a fair comparison with prior DG methods, we re-evaluate all baselines using the same test-domain validation set.
evaluation protocol: each experiment designates one
Implementation Details. Given the demonstrated parameter efficiency and effectiveness of Visual Prompt Tuning (VPT) in domain generalization benchmarks like DomainBed (Jia et al.] 2022] Li al., 2022al Zheng et al.) (2022), our study primarily focuses on VPT-based optimization. To val- idate the broad applicability of our approach, we conduct comprehensive experiments across three backbones: VTT-B/32, ViT-B/16, and ViT-L/14, all pretrained with CLIP (Radford al. 202T Our BOA optimization is applied after a SAM warm-up. During warm-up, we employ a batch size of 16 for the ViT-B models and 8 for ViT-L/14 due to GPU memory constraints, whereas the learning rate is fixed at 5e-4 for all architectures. All experiments employ the Adam optimizer for consis- tent optimization. During the billiard optimization, we fix the step size h at 10, and set the number of reflections to 20. Other hyperparameters, including the Krylov subspace dimension K and loss variation A,, are determined through grid search, with model selection using the validation set.
4.2 KRYLOV SUBSPACE ANALYSIS
Prior to the formal evaluation of our BOA algorithm, present comprehensive analysis of the Krylov subspace introduced to overcome the curse of dimensionality in high-dimensional optimiza- tion.
Firstly, to validate the feasibility of the billiard dynamics, we conduct experiments with oracle test domain: when using the ideal negative test gradient - Vless the incident direction, BOA achieves significant performance gains across five datasets (Table 4 The ERM+BOA and SAM+BOA configurations demonstrate consistent improvements averaging +4.9% and +4.8%,
7

---

Published as a conference paper at ICLR 2026
Test Accuracy (80
Test Accuraacy (KG)
12.0
23 22 6J
0. Step
06
1.
92 14 26
14 30
34 Step
3.4
01 02 0.3
24 0.5
06
91 02 03
06 0.7 38
Step
(a) Caltech101
(b) LabelMe
(c) SUN09
(d) VOC2007
Figure 4: OOD accuracies along approximate initial incident directions on four VLCS domains.
Table 3: Out-of-domain test accuracies with
Table 4: Test accuracies when using the ora- cle negative test gradients as the initial inci- dent directions.
different ViT backbones.
Algorithms
VLCS
PACS
OH
TI
DN
Avg.
Algorithms
VLCS
PACS
OH
TI
DN
Avg.
Backhone:
VIT-B/32
ERM
81.9
95.9
84.
56.1
59.5
75.5
ERM
94.5
36.6
69.5
ERM+BOA (Ours)
85.6
97.7
86.3
69.5
62.9
80.4
SAM
81.9
95.1
80.6
42.1
54.6
70.9
SAM
82.9
96.6
85.4
56.2
59.8
76.2
BOA (Ours)
84,9
96.3
80.9
49.7
54.9
73.3
SAM+BOA (Ours)
87,0
97.9
87.6
69.6
62,9
81.0
Backhone: VTT-B/16
ERM
81.9
95.9
84.1
56.1
75.5
Table 5:
Comparisons of out-of-domain ac- curacy when using ResNet50 architecture.
SAM
82.9
96.6
85.4
56.2
59.8
76.2
BOA (Ours)
86.5
97.4
86.0
60.3
60.2
78.1
Backbone: VTT-1/14
Method Caltech101 LabelMe
SUN09
VOC2007
Avg.
ERM
82.7
98.6
90.2
61.3
64.6
79.5
ERM
98.2
67.1
73.0
77.6
79.0
SAM
82.5
98.7 2
90.9
64.1
64.8
80.1
SAM
99.6
65.7
75.1
80.5
80.2
BOA (Ours)
86.4
98.7
91.2
65.7
65.4
81.5
BOA
99.7
69.0
77.2
81.1
81.8
spectively, over baseline methods, confirming the feasibility of the billiard dynamics and the ideality of the negative test gradient as the initial incident direction.
Fortunately, we discovered the test gradient exhibits strong alignment with the training-derivn deriied Krylov subspace, enabling effective approximation in the absence of test data. Specifically, two facts can observed in Figure 5 (1) Rapid improvement in directional alignment (quantified by
cos yx)) subspace dimension K increases can be observed, where cos 7/ is defined by the cosine similarity between test gradient and its optimal approximation with the Krylov subspace Kx:
cos = max cos(VLL feea (0n), p) cos(V (On), proj Rea (fo)),
(10)
PEAx
where ( denotes the angle between two vectors, Remarkably, cos even exceeds 0.8 for K = 7 10 on "LabelMe", which represents an eightfold improvement over the K case (cos 71 < 0.1). This sharply contrasts with random subspaces, where alignment increases marginally at best. More detailed analysis about Krylov and random subspace can be found Appendix B. (2) The approximate initial incident direction defined in equation (7) has similar increasing trend with K, achieving near-optimal alignment on "SUN09" and "VOC2007", Note that the approximate initial incident direction is obtained without using test data, different from the optimal approximation that utilizes projıc, Vees (0,). Additionally, we even find a significant increase in test accuracies along the approximate initial incident direction (+1.4% "Caltech101", +6.5% "LabelMe". +2.7% on "SUN09" and +2.9% on "VOC2007", see Figure 4.. However, this performance gain occurs concurrently with training loss degradation, necessitating BOA's reflection mechanism to maintain the training loss as much as possible while navigating toward generalizable solutions.
4.3 MAIN RESULTS
Comparisons with SOTA domain generalization methods. Table provides a decisive perfor- mance comparison of domain generalization methods on the DomainBed benchmark, clearly estab- lishing our BOA method as the new state-of-the-arr. With an average accuracy of 78.1% across all five datasets, BOA outperforms the previous best methods (DISAM at 76.5%) by a significant 1.6% margin. This performance gain represents the largest inter- method improvement observed in the benchmark, demonstrating BOA's exceptional doos-d generalization capabilities. The dataset- specific analysis reveals particularly striking results; On VLCS, BOA achieves a remarkable 86.5% accuracy, surpassing SAM's 82.9% by dramatic 3.6% margin. Notably, BOA achieves these record
8

---

Published as a conference paper at ICLR 2026
results using only visual prompt tuning rather than full encoder fine-tuning, making its efficiency par- ticularly impressive. BOA's consistent leadership from easier domains like PACS to highly difficult ones like Terralncogniaa demonstrates its effective handling of diverse domain shifts.
Experiments with different Vision Transformers. As shown in Table 3] the proposed BOA method consistently outperforms both ERM and SAM across all evaluated Vision Transformer back- bones (ViT-B/32, ViT-B/16, ViT-L/14), achieving the highest average accuracy on multi-domain benchmarks. Under the ViT-B/32 backbone, BOA attains an average of 73.3%, outperforming SAM (70.9%) and ERM (69.5%) by margins of 2.4% and 3.8%, respectively, with particularly notable gains on challenging domains such TI (49.7% vs SAM's 42.1%). This advantage is maintained with larger backbones: using ViT-B/16, BOA reaches 78. 78.1% 1% on average, surpassing SAM by 1.9% and ERM by 3.6%, and with ViT-L/14, it achieves 81.5%, exceeding SAM by 1.4% and ERM by 2.0%. The performance improvement is consistent not only in the aggregate metric but across all five individual datasets, underscoring BOA's robustness and its capability to enhance generalization irrespective of the model capacity or architectural variant of the backbone network.
Broad applicability to CNN architectures. While much of our analysis has focused on Vision Transformer architectures, we now demonstrate that the core findings can extend effectively to stan- dard CNN backbones. To this end, we conduct additional experiments using a ResNet50 backbone to validate the persistence of Krylov alignment in CNN architectures. As summarized in Table the superiority of Krylov subspaces over random subspaces remains consistent across multiple datasets, similar to the trends observed in VIT architectures (as illustrated in Figure B). Notably, on the SUN09 dataset, Krylov alignment (quantified by cos 7 exceeds 0.75 for K = 10, marking an improve- ment of approximately +0.5 over the K 1 case. This suggests that Krylov subspaces capture geometrically meaningful directions in the loss landscape, even in CNNs.
Table 6: Cosine similarity between test gradients and their approximations derived from either random or Krylov subspaces (using a ResNet50 backbone).
Subspace Dimension (K)
Domain
Subspace Type
2
8
10
Random
0.3121
0.3122
0.3124
0.3125
0.3125
Caltech101
Krylov
0.3317
0.3339
0.3360
0,3381
0.3381
Random
0.4063
0.4064
0,4065
0.4066
0.4067
LabelMe
Krylov
0.4937
0.5235
0,5724
0.5825
0.5855
Random
0.2653
0.2654
0.2655
0.2657
0.2657
SUN09
Krylov
0.4199
0.7547
0.7603
0.7610
0.7617
Random
0.1395
0.1396
0.1397
0.1399
0.1399
VOC2007
Krylov
0.1406
0.2991
0.3260
0.3414
0.3707
We also evaluate the DG performance of our BOA method using ResNet50 backbone. As il- lustrated in Table BOA achieves the highest accuracy across all four domains, with an average accuracy of 81.8%. outperforming both ERM (79.0%) and SAM (80.2%). The consistency perfor- mance gains reinforces BOA's utility an architecture-agnosii optimization framework. Notably, in our ResNet experiments, we observed that most hyperparameters could be directly transferred from ViT-B/16 setups. However, due differences in loss landscape geometry (e.g., CNNs may hibit sharper minima), we found that maintaining the step size = 10 requires excessive line search iterations identify solution intervals. To ensure efficient optimization, adjusted the step size to for ResNet50.
4.4 ABLATION STUDY
Figure 5 presents decisive ablation study on one VLCS domain ("LabelMe"), evaluating the con- tribution Krylov subspace components to the Billiard Optimization Algorithm (BOA) across three loss variation thresholds (A, 0.05, 0.1, 0.3). In Figure Krylov-inspired Po outperforms random initialization by 0.5-1.6% across all A values, confirming the effectiveness of Krylov subspace on the determination of Pu- Across all three Ae values, the complete BOA within Krylov subspace method consistently outperforms intermediate implementations. At A = 0.3, achieves a remark- able 75.7% accuracy, significantly outperforming both random initialization (69.2%) by 6.5% and

---

Published as a conference paper at ICLR 2026
Usine santion Po
reperct
Accuracy
72.0
70.2
70.0
01.1
69.1
06.2
Loss Variation &
(a) Training Loss Landscape (b) Test Loss Landscape
Figure 5: Ablation study on the effect of Krylov subspace using the VLCS
Figure 6: Trajectory Visualization of BOA. The same tra- jectory, derived from the training loss landscape, is shown in both the (a) training and (b) test loss landscapes.
domain "LabelMe" for testing.
the standalone Krylov-inspired direction (70.8%) by 4.9%. This substantial margin demonstrates that Krylov-enhanced reflection provides further benefits beyond initial direction selection.
4.5 LOss LANDSCAPE VISUALIZATION
Our loss landscape visualization examines two key aspects: (1) connectivity between models with different out-of-domain performance, and (2) optimization trajectory analysis of the Billiard Opti- mization Algorithm (BOA).
Mode Connectivity Analysis. A baseline model was first trained on three OfficeHome domains ("Clipart",. "Product", and "Real World"), which may exhibit poor performance when generalized the unseen "Art" domain. In contrast, ideal oracle model trained using all domains. Sub- sequently, building upon the approach in (Garipov et al. 2018)) we connect these two models using a parameterized Bézier curve and optimize it to maintain uniformly low loss throughout the entire pathway. Finally, as illustrated in Figure 1 a 2D visualization demonstrates a continuous low-loss valley bridging the non-ideal (red marker) and ideal (green marker) models. Remarkably, this con- nectivity pattern has been consistently observed across diverse datasets and architectures.
BOA Trajectory Visualization. visually demonstrate the effectiveness of the BOA algorithm, we employed constrained Krylov subspaces with dimension of K 2 to trace its optimization path on the VLCS dataset. As illustrated Figure 6 BOA successfully navigates the training loss landscape and seeks out-of-domain solutions located within optimal regions on the test loss surface. This visualization offers clear visual evidence BOA's capability in discovering better models.
5 CONCLUSION
This study demonstrates that leveraging mode connectivity in loss landscapes offers novel and effective approach for enhancing domain generalization. We empirically established that low-loss pathways connect models with divergent generalization capabilities, revealing a geometric structure that enables transitions to superior solutions without leaving low-loss regions. To navigate this struc- ture, we proposed the Billiard Optimization Algorithm (BOA), which efficiently traverses these con- nected modes using physics-inspired operations. A key insight enabling this efficiency was the strong alignment between test gradients and the low-dimensional Krylov subspace derived from training gradients, allowing BOA to operate effectively in a reduced subspace. Integrated with parameter- efficient vision prompt tuming, BOA achieved superior performance across multiple DomainBed benchmarks, consistently outperforming existing sharpness-aware and domain generalization tech- niques. These results highlight the value of geometric connectivity properties for developing robust models that generalize well to unseen domains. Future work will explore other geometric properties of loss landscapes adapt BOA for dynamic environments where domain shifts occur continuously.
ACKNOWLEDGEMENTS
This work was financially supported by the National Natural Science Foundation of China (NSFC) under Grant No. U20020007 The authors gratefully acknowledge the NSFC for its critical support of this research.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Maksym Andriushchenko, Dara Bahri, Hossein Mobahi, and Nicolas Flammarion. Sharpness-aware minimization leads to low-rank features. Advances in Neural Information Processing Systems, 36: 47032-47051, 2023.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02880 02893, 2019.
Sara Beery, Grant Van Horn, and Pietro Perona. Recognition in terra incognita. In Proceedings of the European conference on computer vision (ECCV), pp. 456-473, 2018.
Leonid Bunimovich. Dynamical billiards. Scholarpedia, 2(8): 2007.
2007.
Giovanni Bussi and Michele Parrinello. Accurate sampling using langevin dynamics. Physical Review E Statistical, Nonlinear, and Soft Matter Physics, 75(5):557707, 2007.
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Infor- mation Processing Systems, 34:22405-22444 22418, 2021.
Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual- information regularization with pre-trained models. arXiv preprint wrXiv:2203.10782 2022.
Jiali Cheng and Hadi Amiri. Understanding machine unlearning through the lens of mode connec- tivity. arXiv preprint 1.Xxv:2550.064007 2025.
Wei Deng. Guang Lin, and Faming Liang. A contour stochastic gradient langevin dynamics algo- rithm for simulations of multi-modal distributions. Advances neural information processing systems, 33:15725-15736, 2020.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. International Conference on Machine Learning, pp. 1019-1028. PMIR, 2017.
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International conference on machine learning, pp. 1309- 1318. PMLR, 2018.
Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. arXiv preprint 17Xiv: 2017.
Chen Fang, Ye Xu, and Daniel N Rockmore. Unbiased metric learning: On the utilization of multiple datasets and web images for softening bias. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1657-1664, 2013.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving generalization. arXiv preprint arXiv:2010.01412. 2020.
Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization.
arXiv preprint arXiv:1611.015.0 2016.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavi- olette, Mario Marchand, and Victor Lempitsky. adverriadversarial training of neural networks. The journal of machine learning research, 171)):009-2030, 2016.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances in neural information pro- cessing systems, 31, 2018.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv:2007.01 2020.
arXiv preprint
Eugene A Gutkin. Billiard dynamics: a survey with the emphasis on open problems on billiards. Regular and chaotic dynamics, 8(1):1-13, 2003.

---

Published as a conference paper at ICLR 2026
Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves dross-domain generalization. In Computer Vision-ECCo 2020: 16th European Conference, Glasgow, UK, Au- gust 23-28, 2020, Proceedings. Part 11 16, pp. 124-140. Springer, 2020.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprini arXiv:2203. (tt::22233.111999 12119, 2022.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. arXiv preprint arXiv: 191.02191, 2019.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: 04836, 2016.
Minyoung Kim, Da Li. Shell X Hu, and Timothy Hospedales. Fisher sam: Information geometry and sharpness aware minimisation. In International Conference on Machine Learning, pp. 11 11148 11161. PMLR, 2022.
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness- aware minimization for scale-invariant learning of deep neural networks. In International Con- ference on Machine Learning, pp. 5905-5914, PMLR, 2021.
Aodi Li, Liansheng Zhuang, Shuo Fan, and Shafei Wang. Learning common and specific visual prompts for domain generalization. Proceedings of the Asian Conference on Computer Vision, pp. 4260-4275, 2022a.
Aodi Li. Liansheng Zhuang, Xiao Long, Minghong Yao, and Shafei Wang. Seeking consistent flat minima for better domain generalization via refining loss landscapes. In Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), pp. 15349-15359, June 2025.
Bo Li, Yifci Shen, Yezhen Wang. Wenzhen Zhu, Dongsheng Li, Kurt Keutzer, and Han Zhao. In- variant information bottleneck for domain generalization. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 7399-7407, 2022b..
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. Proceedings of the IEEE international conference on computer vision, pp- 5542-5550, 2017.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss land- scape of neural nets. Advances in neural information processing systems, 31, 2018a
Haoliang Li. Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adver- sarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5400-5409, 2018b.
Wenyi Li, Huan-ang Gao, Mingju Gao, Beiwen Tian, Rong Zhi, and Hao Zhao Training-free model merging for multi-target domain adaptation. In European Conference on Computer Vision, pp- 419-438. Springer, 2024.
Ya Li, Mingming Gong, Xinmei Tian, Tongliang Liu, and Dacheng Tao. Domain generalization via conditional invariant representations. In Proceedings of the AAAI conference artificial intelligence, 2018c.
Jörg Liesen and Zdenek Strakos. Krylov subspace methods: principles and analysis. Numerical Mathematics and Scie, 2013.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Dilan Gorur, Razvan Pascanu, and Hassan Ghasemzadeh. Linear mode connectivity in multitask and continual learning. arXiv preprint 2:X1::2010.04495. 2020.
Hamidullah Noori, Rahman Besharat, and Mohammad Zarif Mehrzad. Optimization of one- dimensional functions using the golden section search method. Journal of Natural Science Re- view, 3(3):59-80, 2025.
12

---

Published as a conference paper at ICLR 2026
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE/CVE international conference computer vision, pp. 1406-1415, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference Machine Learning, pp. 8748-8763. PMLR, 2021.
Akshay Rangamani et al. Loss landscapes and generalization in neural networks: Theory and ap- plications. PhD thesis, Johns Hopkins University, 2020.
Zheyan Shen, Jiashuo Liu, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108, 13624, 2021.
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European conference computer vision, pp. 443-450. Springer, 2016.
Vladimir Vapnik. Statistical learning theory. (No Tirle), 1998.
Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 5018-5027, 2017.
Roman Vershynin. diigh-dimensiona probability: An introduction with applications in data science, volume 47. Cambridge university press, 2018.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: A survey domain generalization. IEEE Transactions on Knowledge and Data Engineering, 2022.
Pengfei Wang, Zhaoxiang Zhang, Zhen Lei, and Lei Zhang. Sharpness-aware gradient matching for domain generalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3769-3778, 2023.
Max Welling and Yee W Tch. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference machine learning (ICML-/I)), pp. 681-688, 2011.
Kaiyue Wen, Zhiyuan Li, and Tengyu Ma. Sharpness minimization algorithms do not only minimize sharpness achieve better generalization. Advances in Neural Information Processing Systems, 36: 1024-10355 2023.
Lei Wu, Zhanxing Zhu, et al. Towards understanding generalization of deep learning: Perspective of loss landscapes. arXiv preprint 1rXiv:1706.10220 10239, 2017.
Yichu Xu, Xin-Chun Li, Lan Li, and De-Chuan Zhan. Visualizing, rethinking, and mining the loss landscape deep neural networks. arXiv preprint arXiv:2405. 2024.
Ruipeng Zhang, Ziqing Fan, Jiangchao Yao, Ya Zhang, and Yanfeng Wang. Domain -inainn sharpness- aware minimization under domain shifts. arXiv preprint arXiv:2405. 18861, 2024.
Shuofeng Zhang, Isaac Reid, Guillermo Valle Pérez, and Ard Louis. Why flatness does and does not correlate with generalization for deep neural networks. arXiv preprint arXiv:2103.06219, 2021.
Xingxuan Zhang, Renzhe Xu, Han Yu, Hao Zou, and Peng Cui. Gradient norm aware minimization seeks first-order flatness and improves generalization. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 20222022222222 2023.
Haoyang Zheng, Hengrong Du Qi Feng, Wei Deng. and Guang Lin. Constrained exploration via re- flected replica exchange stochastic gradient langevin dynamics. arXiv preprint 2405v:2405.07834 2024.
Zangwei Zheng, Xiangyu Yue, Kai Wang, and Yang You. Prompt vision transformer for domain generalization. arXiv preprint arXiv:2208.00 089144 2022.
13

---

Published as a conference paper at ICLR 2026
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization in vision: A survey. arXiv preprint arXiv:2103.02503, 2021.
Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha Dvomek, Sekhar Tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-aware training, arXiv preprint arXiv:2203.066 08066, 2022.
Yingtian Zou, Kenji Kawaguchi, Yingnan Liu, Jiashuo Liu, Mong-Li Li Lee, and Wynne Hsu. Towards robust disti--dist generalization bounds via sharpness. arXiv preprint arXiv:2403.06333 063392 2024.
14


Reviews:
[{"review_id": "aKj5QjhEHl", "reviewer": "Reviewer_mWsD", "summary": "This paper introduces BOA (Billiard Optimization Algorithm) for domain generalization. The idea is to stay inside a low-loss region of the training loss, move to the loss boundary with a line search, then reflect and keep going. To avoid getting lost in high dimensions, the method searches only inside a Krylov subspace built from training gradients/HVPs. The authors also show that test gradients tend to align with this subspace, and that a train-computed path often also looks good on the test landscape. On DomainBed, BOA beats ERM/SAM and several DG methods; for example, on VLCS with a ViT-B/16 model, BOA improved accuracy by 3.6 percentage points compared to SAM.", "questions": "Is it acceptable not to include CNN backbones, or can you provide BOA vs ERM/SAM results on a standard CNN such as ResNet-50 under the same budget with brief notes on hyperparameter transferability and compute cost?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Clear formulation and intuition. BOA’s use of a training-loss sublevel set, a concrete line-search to the boundary, and a reflection update is straightforward and well motivated.\n\n- Low-dimensional search. Constraining the trajectory to a Krylov subspace is a sensible way to capture salient curvature directions without exploring the full parameter space.\n\n- Useful visual evidence. Overlaying the same trajectory on train and test landscapes helps illustrate why a train-computed path can still navigate good regions on test.\n\n- Strong empirical results. Consistent improvements on DomainBed with ViT backbones (including VPT) and a notable margin over SAM on VLCS.", "weaknesses": "- Limited backbones: Experiments are mostly on ViT. Results on CNNs (e.g., ResNet) or other architectures would strengthen generality.\n\n- Unclear compute cost: Please compare total elapsed (wall-clock) time, memory, and counts of line-search/HVP calls to ERM/SAM under the same settings.\n\n- Heuristic initial direction: The current choice is simple; more analysis or alternatives would improve justification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uut1ZCOuP7", "reviewer": "Reviewer_LAoM", "summary": "This paper proposes a new optimization framework, termed \"Billiard Optimization Algorithm\" (BOA), to improve domain generalization using VPT with ViT backbones. This algorithm leverages the \"mode connectivity\" properties of the loss landscape, instead of relying only on flatness (as done by methods like SAM).\n\nBOA consists of a line search part, where boundaries of the loss contour are reached, and a reflecion part, where the new search direction is selected using a physics-inspired rule based on the local gradients. Moreover, BOA uses the Krylov subspace to select the initial direction and constrain the search trajectory. This choice is motivated by the observation that the test gradients seem to be aligned with the Krylov subspace generated from the training gradients.\n\nEmpirical evaluation using the DomainBed benchmark with VLCS, PACS, OfficeHome, TerraIncognita and DomainNet shows that the proposed BOA method consistently outperforms the evaluated DG baselines. The paper also provides interesting theoretical analyses showing the benefits of approximating gradients using Krylov subspaces compared to random subspaces.", "questions": "- How sensitive is the method to the hyperparameters (e.g., K, reflection count)?\n- What is the computational cost / runtime compared to SAM or GSAM?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This paper presents an interesting and novel optimization approach towards domain generalization introducing several new (for DG) concepts, like mode connectivity, the use of a new search approach and the use of Krylov subspaces\n - The paper also offers theoretical and empirical insights on the properties of the Krylov subspace, especially regarding its alignment with the test gradients.\n - The results presented consistently outperform the compared DomainBed baselines and sharpness-aware methods\n - Although no detailed (theoretical or empirical) analysis of the computational complexity of the proposed method is provided, the method appears to be efficient since it avoids the computation of the Hessian.", "weaknesses": "- The reason / underlying mechanisms behind the observed improvements remain unclear. The paper does not convincingly explain why mode connectivity or Krylov alignment lead to better cross-domain generalization.\n - The experiments are constrained to VPT with ViTs. Although this choice may indeed lead to the best results, the paper would be much more convincing if similar findings were observed e.g., for ResNet backbones and/or with full fine-tuning. Even if performance is degraded, demonstrating consistent improvements over comparable baselines would better support the stated claims. In addition, given the increased dimensionality of the parameter space, the proposed method should benefit even more compared to the baselines in this case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fR4JHUrQPa", "reviewer": "Reviewer_Gedm", "summary": "To find flat minima that perform better on the test data under distribution shift, the paper considers mode connectivity. It proposes a Billiard Optimization Algorithm (BOA) that traverses the flat basin of the loss landscape analogous to the reflection of a billiard on the board. To find the search direction effectively, the paper leverages the observation that test loss gradients align with the Krylov subspace. Experiments on five datasets show the potential of BOA.", "questions": "1. How is $\\alpha^*$ in equation 4 determined?\n\n2. BOA's reflection results in symmetric exploration of the loss landscape. The geometry of the loss landscape and initial search points both affect the efficiency of the symmetric search. How robust is the symmetric exploration in searching flat minima when the flat basin is not symmetric, and the initial search point lies at the center of the flat basin?\n\n3. Does the search angle $\\phi$ affect the search of optimal model? Why is it set to a fixed value instead of as a hyperparameter?\n\n4. How is $\\epsilon$ in equation 8 determined? \n\n5. Is it a fair comparison with other DG methods, given that many of them do not utilize a validation set of test data?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. It is meaningful to use mode connectivity to find better flat minima in domain generalization. The observation that test loss gradients align with the Krylov subspace is interesting and inspiring. \n\n2. BOA proposes a novel strategy to search over the loss landscape. \n\n3. Figure 6 is very helpful in understanding how BOA works.", "weaknesses": "1. Figure 1 suggests that minima that are equally flat on the loss landscape of the training distribution can have different sharpness on the loss landscape of test data. The observation motivates finding better minima through mode connectivity. BOA relies on a validation set of the test data to find an optimal model. However, in domain generalization, it is common to assume that test data is not accessible. My biggest concern is that under this condition, how to guarantee that the minima found via BOA are better? \n\n2. The clarity of section 3 should be improved to aid understanding. For example, some notations in section 3 are not explained in the main paper (e.g. $\\alpha, \\epsilon$), and the role of $h$ and its connection with $\\alpha$ are not explicitly stated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)