Symmetry-Aware Bayesian Optimization via Max Kernels
Authors: Anthony Bardou
Antoine Gonon
Aryan Ahadinia
Patrick Thiran
Venue: ICLR 2026 Poster (2026)
Keywords: Bayesian Optimization
Invariance
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Abstract: Bayesian Optimization (BO) is a powerful framework for optimizing noisy, expensive-to-evaluate black-box functions. When the objective exhibits invariances under a group action, exploiting these symmetries can substantially improve BO efficiency. While using maximum similarity across group orbits has long been considered in other domains, the fact that the max kernel is not positive semidefinite (PSD) has prevented its use in BO. In this work, we revisit this idea by considering a PSD projection of the max kernel. Compared to existing invariant (and non-invariant) kernels, we show it achieves significantly lower regret on both synthetic and real-world BO benchmarks, without increasing computational complexity.

Published as a conference paper at ICLR 2026
SYMMETRY-AWARE BAYESIAN OPTIMIZATION VIA MAX KERNELS
Anthony Bardou
Antoine Gonon Institute of Mathematics EPFL. Lausanne, Switzerland antoine. gow.nne.oonneee....co ch.
School Computer and Communication Sciences EPFL
Lausanne, Switzerland
anthony.bardoueee hww..oeeeddeee...t.tc.c
Aryan Ahadinia & Patrick Thiran
School of Computer and Communication Sciences
EPFL.
Lausanne, Switzerland
(aryan.ahadinna ahadinia, patrick chatriikkthirraneepp.....
ABSTRACT
Bayesian Optimization (BO) is powerful framework for optimizing noisy, expensive-to-evaluat black-box functions. When the objective exhibits invari- under group action, exploiting these symmetries BO efficiency. While using maximum similarity across group orbits has long been considered in other domains, the fact that the max kernel is not positive semidefinite (PSD) has prevented its use in BO. In this work, we revisit this idea by considering a PSD) projection of the max kernel. Compared existing invari- ant (and non-invariant) kernels, we show it achieves significantly lower regret on both synthetic and real-world BO benchmarks, without increasing computational complexity. Code. gulhbbccmmbbbrdoowaaa-eenee
substantially improve
INTRODUCTION
Many real-world problems can be framed as the optimization of noisy, expensive-to-ees to-ellluaee black-box function f* S C Rd + IR. Bayesian Optimization (BO) provides principled and sample-efficient framework for tackling this problem, with asymptotic guarantees of global optimality complementing its empirical success. As result, BO has been widely adopted across diverse domains such as robotics (Lizotte et al. 2007), computational biology (Gonzalez et al. 2015) and computer networks (Bardou al. 2025).
For a black-box function f* belonging to the Reproducing Kernel Hilbert Space (RKHS) H. associated with kernel & S x S - R. BO proceeds by placing Gaussian Process (GP) prior f TP,, k)) over functions H4. The kernel & determines the covariance structure of the GP and thus encodes prior assumptions about ʃ Incorporating suitable prior knowledge can substantially improve convergence and sample efficiency. In many applications, the objective is known to be invariant under the action group that is,
f'(x) fg(z) for g e G.
For instance, molecular property prediction, f* may be invariant to rotations of the underlying molecular structure (Glielmo et al. (017). In such cases, designing kernels that explicitly incorporate G-invariance becomes essential. Ginshourger et al.) (2012) showed that for centered GP to be G-invariant, its covariance function must also be invariant under G. Motivated by this, we revisit simple idea-keep the best alignment over each orbit-and apply to BO.
Given base kernel kn and symmetry group G, define Kmax (x, x')
max kulgæ, g o'x', 9.9' g. EG
(I)
that the similarity between and x' is the best alignment their orbits.

---

Published as a conference paper at ICLR 2026
Radial-Invarian Function
Xx)
KmaxXX,X X*)
|
||
Figure 1: (Left) Colored level sets of wwo-dimensional function (x) invariant under planar rotations (see (16)): if ||>||| = |a'||2- then (*x) = (z'). (Center/Right) Rotation-invarian kemells derived from RBF base kernel (lengthscale 1/2),, visualized as function of (||æ) [|aa)1)))) kmax (right) captures the correct invariance, while hiav (center) only approximates it.
The intuition for using the max max-alignment is that when the objective is invariant under group of transformations, two inputs can become very similar after applying the right group element, even if they differ a lot in their original positions. For instance, in an image-based problem with rotation invariance, two rotated images of the same object (e.g., cats) should principle be treated similarly by the optimizer since they correspond to the same objective value. However, most rotations will not align the images well; and if the optimizer compares images with e² distances, only a small number of them can give good match. In such settings, taking the maximum similarity over all group actions is natural: among all transformations, typically only one (or a few) reveal a true alignment. Averaging transformed pairs look different-wwereen.
over all rotations would dilute this information-mon the max retains the one transformation that matters. This "best-alignment" principle is the core motivation behind mmax and is expected to provide clearer signal to the optimizer about which inputs should be treated similarly, compared, e.g., to an averaging approach.
While kmmx symmetric and G-invariant, it is however not guaranteed be positive semi-definite (PSD), a property required for the standard Gaussian-proees machinery underlying BO (see Ap- pendix C.Ip. To address this, we introduce PSD version of kmax-
A PSD, invariant surrogate via projection + Nyström. On a finite design set D, we form the Gram matrix of kmax and project it onto the PSD cone (eigenvalue clipping), obtaining K,. Denoting by Kt the Penre-Penrose pseudo-inverse, of K+, we then define the G-invariant, PSD kernel
k (xx xx) kma D) K! (mmx((D,n))
Equivalently, (2, xx) (xx) ((x') with features 6(x) K Amax(D,aa which makes
positive semidefiniteness immediate. By construction, k(") (i) coincides with kmax on D whenever knax is already PSD. and (ii) has per-iteration asymptotic cost comparable to orbit-averaged kernels;
details in Section 3.2
Results. The max- aligmmetn heuristic does translate into concrete benefits for BO, which we observe throughout the paper. The resulting kernel is geometrically better aligned with the true structure of the problem (Figures [ and 2) In practice, this makes (i) the acquisition function more faithful as it avoids redundant exploration of points that are already explored up to symmetry, and (ii) uncertainty modeling also more faithful: it gains confidence in unexplored regions that correspond to symmetry- equivalent points. Across synthetic benchmarks with finite and continuous groups, wireless-nettwok design task and a particle packing problem, we show that k(() consistently attains lower cumulative and simple regret than both the base kernel and the orbit-averaged alternative, with gains increasing with (g).
Relation with spectral-based theory. Mainstream BO theory links fast eigendecay of the kernel to small regret upper bounds (Srinivas et al. 2012 Valko et al. 2013 Scarlett et al. 2017 Whitehouse
al. 2023). Surprisingly, find the opposite trend in our setting: x2(2) typically has slower empirical eigendecay than kivvg. yet consistently achieves better (lower) regret in practice. This directly challenges the usual spectral intuition: our results reveal a clear mismatch between spectral predictions and empirical performance, suggesting that eigendecay alone does not capture the advantages of (P)) As discuss later, geometric considerations (the alignment of the kernel
2

---

Published as a conference paper at ICLR 2026
eigenvectors with the directions that matter for optimization) and approximation hardness of the blackbox f* in the RKHS likely play an essential role beyond pure spectral rates.
Summary of the contributions. We propose Kmax max-alignment route to G-invariance, turn it into a valid GP kernel for BO via PSD projection and Nyström, and show A:()) G-invariant, equals kmax on D when kimax is PSD, and matches the asymptotic cost of orbit-averaged kernels (Section 3 We demonstrate consistent BO gains over orbit averaging across BO benchmarks (Section and we analyze why eigendecay alone does not explain these gains (Section
2
BACKGROUND
2.1 BAYESIAN OPTIMIZATION IN NUTSHELL.
Problem. We seek maximize an expensive-to-evaluute black-box objective f S- under the assumption that f is in the RKHS Hr of PSD kernel k: S x S - R. Each query S returns noisy observation y = f'(x) + E, where ε ~ (0, of) Let 2, { y))) denote the dataset after evaluations, and write D;- (I1..... Ft) and y: (y1, (y1.....3t)) 96)
Surrogate model: the GP prior. BO maintains a probabilistic surrogate f over functions in Hy to guide sampling of new queries æ E S with the goal of converging to arg max,es (x)) common choice is zero-mean Gaussian process (GP) (Rasmussen & Williams 2006
f GP(O,k),
Conditionally on the dataset of queried points 2, after evaluations, the posterior f 2 still a GP with posterior mean and covariance
(e(ar) = k(r, D:) (K + og yt.
(3)
Cov,(æ, x²) = k(æ, x') - k(æ, D:) (K, + 1) kD,,,x')
(4)
where K, D,) Rext, I, is the x identity, and k(x,, D,) = (((2, k((,,,,,,,,,,, x,)].
The GP posterior plays the role of a refined surrogate for /" throughout the optimization process. At iteration 1, BO algorithm completes the next steps:
Step 1. Form the Gram matrix K, = k: kDD,, D:) using all past queries.
Step Compute the inverse of K, + of (with fixed hyperparameter oa) and plug it into to obtain the posterior mean and covariance functions (Hz, Cov,).
Step 3. Select the next query by maximizing an acquisition function 0g : S - built from (Me, Cov:) (e.g., GP-UCB (Srinivas et al. (012)) Expected Improvement (Jones et al. 1998)). This where BO balances exploration (learning (") and exploitation (sampling near current optima). The pair (Me, o can be viewed as the algorithm's 's current best estimate of the unknown function and its uncertainty.
The dataset is then updated with the new query:
3r+1 E arg max at (x), V/++ = (*(411 E141,
aeS
and the loop repeats until stopping criterion is met.
Measuring performance with regret. We follow the common practice in BO: for experiments where f* is known, we measure the regret on the deterministic f* E Mg, and when discussing theoretical regret bounds refer to the regret on / (P, k) (Garnett 2023). In both cases, for h ʃ or f", the instanianeous regret at timestep / is r. = max h(x) h((;,) ) the cumulative regret at horizon T is RT ΣT Tr. and the simple regret is ST max h(x)) maxigesT h((x)) BO algorithm with a sublinear regret (i.e., Ry E o(T)) is called no-regret and offers asymptotic global optimization guarantees on f" Most standard cumulative regret upper bounds established in terms of the eigendecay of the operator spectrum of the kernel is (Srinivas et 2012 Valko et al.
2013 Scarlett al. 2017 Whitehouse al. 2023
3

---

Published as a conference paper at ICLR 2026
2.2 INVARIANCE IN BAYESIAN OPTIMIZATION
In many applications, the objective function f is invariant under the action of a known symmetry group 9 on S, i... (x) = f (gx) for all g E G. When such invariances are ignored, BO algorithms may waste evaluations by treating all points within the same [g -orbit as distinct. Given a non-invariant base kernel h;, and an arbitrary symmetry group G. both provided by the user, this section reviews existing strategies for incorporating group invariance into BO and positions our contribution within this literature.
Data augmentation. A popular way to enforce symmetry is to expand the dataset 2 itself, as it is often done in computer vision (Krizhevsky et al. 2012). For each acquired observation (æ,, We), one augments Z with all transformed copies [ga,, We oeg. while leaving the base kernel l;, unchanged. However, since BO scales O( ()),,,, this approach quickly becomes computationally prohibitive and is inapplicable to continuous symmetry groups. For completeness, we include in Appendix [F] numerical comparison of our approach with data augmentation, showing that data augmentation scales poorly with the size of the group, and does not meet the performance of the average or max kernel even when using all symmetry augmentations.
Search space restriction. Another approach is to restrict the search domain to a fundamental region Sg C S whose G-orbit covers S: Ugec gSg S (e.g., Baird et al. (023bb). For example, if S S= s 112 and 9 the group of T rr/2-rotations one may work on Sq [0, 1J while keeping the kernel unchanged. This viewpoint corresponds to working directly with the quotient S/9 embedded in S.
This line of work is complementary to ours. In BO, one must choose both a search domain and kernel: fundamental domains address the former, while our construction helps with the latter. Even we decide to run BO on Sg. one still needs good invariant kernel on Sg. and our invariant kernels can be used in that setting as well. We refer to Appendix G for a short example illustrating the practical difficulties of explicitly optimizing over fundamental domain, and how the design of the kernel is complementary to that decision.
Invariant kernels. A principled way to incorporate prior G-invariance of f* is to consider a G- invariant GP prior f. i.e., a GP whose sample paths x E S - f(x ()) obtained by fixing one outcome in the probability space
themselves invariant under g. [Ginshourger et al. (201)) established that such GPs necessarily admit a G-invariant covariance function"] meaning k(gæ, g x k(æ, x²) for all x, x' E S and g, E G. The central question then becomes: how can one construct an invariant kernel k from an arbitrary base kernel hb and symmetry group G? An elegant solution, dating back to Kondor 2008 and recently advocated for BO by Brown al. (2024) is average hy, over G-orbits:
1
kavg (x, x²) =
ko x')
(5)
9.9'ES
This construction is not only guaranteed to be G-invariant, but also admits clean functional in- terpretation: if Hs, and Hiweg denote the RKHS induced by k;, and kavg respectively, then H Ha coincides exactly with the subspace of G-invariant functions in Ha, (Theorem 4.4.3 in Kondor (2008)) Consequently, kavg (up to normalization) has gained popularity as the standard off-the-shelf kernel for BO in symmetric settings (Glielmo et al. 2017 Kim al. 202 Brown et al. 2024).
A complementary idea in kernel methods is to retain the best latent alignment between two orbits via a maximum, as in convolutiom/best-macch kernels for structured data (Gärtner 2003 Vishwanathan
et al. 2003) and follow-up work across domains (Fröhlich et al. 2005) Zhang 2010 Curtin et al. 2013). Max-alignment kernels, however, are not PSD in general, leading to indefinite Gram matrices.
This has motivated two families of remedies: (i) explicit Kreîn-space formulations (Ong et al. 2004 Oglic & Gärtner 2018), and (ii) simple PSD) corrections such as eigenvalue clipping/ftipping in
SVMs (Luss & D aspremont 2007 Chen et al. 2009), which are empirically effective.
Our adaptation to BO. Guided by the above, we adopt the max-aligmment view for BO. To ensure positive definiteness, we project kmax (see (T) onto a PSD kernel k(P), which coincides with kmax whenever the latter is already PSD. This preserves the sharp, high-contrast orbit alignments of kmax
UU to modification, i.c., there is another GP f' such that for every E S, P (()())) f'(m)) 1 and f has invariant paths and invariant covariance, see Property 3.3 in Ginshourger et al. 2012)
4

---

Published as a conference paper at ICLR 2026
while ensuring compatibility with the BO framework. Moreover, it maintains a per-iteration BO complexity comparable to that of orbit-averaged kemelss (see Section 2.2 In our experiments, R(P) better reflects the intended symmetries of standard synthetic objectives and achieves substantially lower cumulative regret. Interestingly, these empirical gains are not mirrored by existing eigendecay- based upper bounds, point we return in Section 5
3 THE MAX KERNEL
We have introduced the max-alignment kernel mmax and its PSD surrogate ((P) in 2 This section explains why kmax is a natural G-invariant covariance, clarifies how it differs from orbit averaging through examples, and records the practical PSD construction we use in BO.
3.1 MOTIVATION: kimax AS A VALID COVARIANCE
A natural way to motivate kmax is to exhibit G G-invariant GPs whose covariance equals kmax.
Construction. Let h ~ GP(O, ks) with an isotropic base kernel lbs (z, x²) <[I[ x' (l2) with nonincreasing (e.g.. popular ones such as RBF, Matérn). Consider a map Øg such that (i) og (x) =
(gx)) for all g E G and (ii) (z)g(x) og (x') ll22 ming.g' |ga g'x² [|2- Define f(x: (x))
Then f is G- G-invariant and:
Proposition 1. Under the construction above, f ~ GP(O, himax) with kimax given by ①T.
Proof sketch, details in Appendix (ax), deff ko( )g @ g' x<' 222l2). and monotonicity of x converts the min-distance into maxg.0' kn (n g x²).
k(mingg llga
This shows that kmax naturally arises as the covariance of valid G-invariant GPs. In contrast, the common approach to invariance in BO is to build kavg by averaging a base kernel as in (5). But averaging and maximization induce fundamentally different geometries:
Lemma 2.
For any base kernel kis and any (double) orbit ((x, x') := {(ga, g'a')))')))))))) D' E 9). havg kiaxx on O(x, x²) if and only if ki = kimax on that orbit.
Indeed, an average reaches the maximum only when every term is maximal. Thus kavy can never reproduce the geometry of kimax, except in the degenerate case where the base kernel is already kmax. making averaging redundant. One might wonder whether this limitation of havg could be circumvented by building it from a different base kernel than the one used for kmax. In Appendix[A2 show that, under mild assumptions satisfied by standard kernels (upper-bounded by 1, with equality k(x, x) = along the diagonal), kavg and kmax can coincide only in the trivial case where the base kernel of kavy is already invariant for pairs of points belonging to the same orbit. Thus, even in this more general setting, averaging does not reproduce the geometry of maximization (except if the base kernel already had invariances).
To make this contrast concrete, we now examine a simple example (radial invariance with an RBF base kernel) where kmax and hiavg can be computed in closed form.
Example (Radial invariance with kmax). Let G be the group of planar rotations and kis (xx x') [||1| exp(-I|
xx 1/11///22 11/222) be RBF kernel. With dog (a:) ||a2||2-
kmax (x, x²) exp( (mp(-((1zz))) 1222 kavg(I, x') = exp
||
ar 113
Io
|| ||||||||="|||
21
with lo the modified Bessel function (derivation in Appendix B. As illustrated in Figure the two kernels kmax and kiwg induce qualitatively different similarity structures. By construction, kmax assigns large similarity whenever ||æ|||2 R || 12. If || = 1| [|,, the function f* satisfies f*(x)= f*')))) since it is invariant under rotations, and kmax exactly recovers this invariance by In contrast, kavg only approximates this behavior: its similarity curves as a function of {) ||| ||z))12) correspond to distorted balls, and two points with identical norms may be ranked as highly dissimilar (see the diagonal ||æ|| = || of the right plot in Figure TT. This mismatch highlight that while both constructions enforce rotation invariance,
assigning maximal similarity kmax( x')
only kmax preserves the correct notion of similarity.
5

---

Published as a conference paper at ICLR 2026
Table Complexity per BO iteration. Here G denotes either [G] or |G| depending on whether the orbit terms reduce to single sum (when kb (gæ, æ²) suffices) or require double sum over (g. g m the number candidate points used acquisition optimization. The row Per-candidate acquisition evaluation gives the cost of single acquisition evaluation; for one BO iteration this row is multiplied by m and added to the other rows to obtain the total.
Base kernel Ab
Averaged havg
Projected k.
Gram matrix (n n)) SVD inversion
(nn²)
(n
On
(n1))
O(n²)
O(n
PSD projection
O(n O(n]G])
Per-candidate aeq. eval.
(())
O(G))
Total for 1 BO iteration
O + n² + n²)
O( + G²) + +
O(mm + n n²] + n²))
3.2 PSD EXTENSION OF kmax: WHAT WE USE IN PRACTICE
Because kmax is not PSD in general, we apply a standard projection step on the finite design set
DD = (.1....,, Fn }. Let K = kimax (D, D) with eigendecomposition K QAQ" and define
(with the max applied elementwise)
K+ Q max A
(6)
We then use the Nyström extensior Williams & Seeger 2000) evaluate cross-covariances with new points, yielding the PSD, G-invariant surrogate A! given ( and that we reproduce here:
k P (x. x') kmax (x, D) K! kmax x').
Key properties of (D)
PSD & invariance. k. is PSD and inherits argumentwise G-invariance kmax.
Consistency with kmax. K 2 O, then K = K and k4 agrees with kmaxx on x D. Cost. Each BO iteration involves (i) building the Gram matrix on D, (ii) inverting the Gram matrix to build the acquisition function, and (iii) m kernel evaluations when optimizing the acquisition function. Step (ii) has the same cost as the SVD of K needed to compute both K+ and KI, which makes 4 having the same asymptotic per-iteration cost as kavgi i its per-query evaluations are expensive, but this difference is negligible long summary is provided in [ and example of runtimes Table Regularity. For finite groups, kmax is max of finitely many smooth maps and is almost every- where (a.e.) differentiable; the Nyström extension preserves a.e. differentiability in each argument. For continuous groups, smoothness sometimes be obtained via closed-form formulas (e.g.. as in Example 3p.
keep m S A concise complexity
We now illustrate the behavior of k. versus havg (in this situation, Knax is not PSD and the projection step is indeed needed to restore positive semidefinitenesss)
Example (Ackley function with k4). Figure compares k(()) and kavg on the one-dimensional
Ackley function (see (15)). The projected kernel kP)) preserves the expected pairwise symmetries (invariance along x = y and x -y) and spreads mass more evenly across the symmetric regions, whereas kavg concentrates covariance mostly the origin. Thus, k. better reflects the symmetry geometry of the problem, echoing the qualitative difference observed in Example
Beyond the finite view (details in Appendix C.. The PSD projection with Nyström in Equation 7 practical, data-dependent construction. It can be seen as the finite-sample face of broader, intrinsic
2 K2 does not depend on the choice of the eigendecomposition, see Lemma 7 in the appendix.
'It indeed extends K since (P) (a;; æx)) K; K! K.j = (K (K K);; = KK)
*One SVD of K suffices obtain both K, and K!, so the extra PSD projection does not increase asymptotic cost.
Kmaase (gar, x²) Amass (Ir, .x) implies Kmax (gær, D) kmaax (xr, D), hence invariance of (P)

---

Published as a conference paper at ICLR 2026
Ackleyld
k
Karg
1.0
Objective
-1-
Observations D
y
0.5
-2-
Post. Mean
Functonn
0.0
-0.5
0.5
-1.00-20 -0.75 00.50 -.25 0.00 0.25 0.50 0.75
1.00
-1.9
-1.0
-0.5
0.0
0.5
1.0
-1.0
1.0
-0.5
0.0
0.5
1.0
Figure 2: (Left) One-dimensional Ackley function f* (see (55). invariant up to coordinate-wise sign-flips, and GP posterior means (e(x) as in for (D) (orange diamond) and kavg (green circles) built from D (black crosses). (Center) Covariance structure induced by k(_)) (Right) Covariance structure induced by kavg. Both kernels invariant to reflections across y and z -y, but kavg concentrates covariance near O, while (PP) better reflects the underlying symmetry geometry. Consequently, GP posterior mean induced by k_P) is the best at fitting the objective (left).
Table 2: Performance of kb, kaug: and k_P) across benchmarks. For each kernel {kb; kavg k(P)) we report m + Ser. where m is the empirical mean over 10 seeds (lower is better) and 8en is the empirical standard error. Best mean bold; means m whose 95% confidence interval (m + 1.96ser) confidence interval overlap with the best are underlined. Performance is measured by cumulative regret on synthetic benchmarks and by negated simple reward on real-world experiments.
Benchmark
|
kb
kavg
Synthetic (Cumulative Reg.)
Ackley2d
382.7 5.7
128.2 + 10.4
126.4 3.6
Griewank6d
64
3840.3 177.7
3067.4 + 841.9
1832.6
146.3
Rastrigin5d
8.0
3568.5 : 91.3
1583.5 + 341.9
813.4 70.6
Radial2d
00
388.6 20.3
480.9 76.4
199.7 11.6
Scaling2d
1820.6 + 1135.4
3361.8 + 742.9
25.4 6.4
Real-World (Neg. Simple Rew.)
WLAN8d
24
65.0 3.2
-11.8 1.7
-4.44 0.7
PartPack6d
00
0.79 + 0.10
0.69 + 0.01
0.92 0.10
definition that does not depend on D. Since kmax is symmetric, it admits a spectral decomposition kmax(x, x²) =; (z) L2, and we can always define (a.e.)
k4 (x, x') Σ max(0, Ax) (x)))
with h4 kmax whenever kmax already PSD. On finite domains, this precisely reduces to the matrix PSD projection in (6). In Appendix [
we formalize the infinite-domain construction via integral operators, prove that k4 is G-invariant, and show that the finite projection Nyström in converges to k4 at the spectral (Hilbert-Schmidh) level under iid sampling (Appendix C.4).
Takeaway, knax the exact covariance of natural class of G-invariant GPs and induces search geometry that preserves high-contrast orbit alignments (Examples Bjaddd The PSD projection + Nyström step yields valid GP kernel k(P) without introducing extra asymptotic complexity. We now measure its practical impact in Section
EXPERIMENTS
We evaluate { against two baselines: (i) the off-the-shelff kernel k; (no symmetry handling), and (ii) the orbit-averaged kernel kavg (Brown al.) 2024). Benchmarks include standard synthetic objectives and two real-world tasks with known invariances (a wireless network design task and
particle packing problem). We ask: (Q1) Does k(P) reduce simple/cumulative regret kavg? and (Q2) How does performance scale with the size of the symmetry group and dinension? The full experimental setup is described in Appendix
7

---

Published as a conference paper at ICLR 2026
Grieankkdd
Rad al2d
WLAN8d
Cunulative Rogret
4000
alive Regret Rv
500
Karg
RBF
Neg. Wi AN Throughput
-30
Matám-5/2 k
3000
2000
-00
50-
200
-60-
1000
Motémm-52
ki
100
Kag
-70
20
Iteraaion
Iteration
Figure Cumulative regret and negated simple reward under GP-UCB with li (blue crosses), Kavg (orange diamonds), and (P) (green circles) on a selection of benchmarks (all benchmarks in Appendix Hb. Shaded regions show the standard error (#teer)) over 10 seeds.
Headline: kPP)
wins on every task. Across all benchmarks (Table 2p. ki( achieves the best performance with up to 50% of improvement. This answers Q1 positively. Regarding Q2, we will see that the group size increases, k42) stays strong, while Ravg degrades and even underperform the non-invariant base kernel kb.
Setup in one glance. We run GP-UCB with each kernel &s E flib havg. k. k(P)), using the same acqui- sition and optimization budgets. We report results averaged over 10 seeds. All the hyperparameters and group actions are detailed in Appendix E
4.1 SYNTHETIC BENCHMARKS
We consider synthetic functions f (Ackley, Griewank, Rastrigin, etc.) that exhibit symmetries (such as permutations, coordinate- twite sign-flips, rotations, rescaling) and are classically considered as challenging to optimize in the BO literature (Qian et al. 2021 Bardou et al. (024). We cover dimensions = to = and group sizes [C] 8 to
Gl OC. We evaluate performance using the cumulative regret Rt = ΣT (f*(x*) f (xx)) since the global maximizer x' arg max (xx is known.
Finite groups: the gap widens G grows. With Matérn-5/2 base la on Ackley2d (|g|=8), havg and k.( are tied; both dominate lns As G increases (Griewank6d, ç 64; Rastrigin5d, -3,400)
k(P) increasingly outperforms kavg achieving cumulative regrets that are, on average, 40% and 49% lower respectively (Table Figure 3 left panel, and Appendix B for the whole set of figures).
Continuous groups: hivv can underperform even kig. For radial and scaling invariances (continuous
groups; RBF base), kavg degrades relative to kg, while k(?) remains strong (Figure center panel, and Appendix E for the whole set of figures).
4.2 REAL-WORLD EXPERIMENTS
We consider two real-world experiments that are described in detail Appendix [ the design of wireless network (8-dimensional, invariant permutations of pairs of parameters) and particle packing problem (6-dimensional, invariant to the rescaling of some parameters and to permutations of pairs of parameters). For both benchmarks, performance is evaluated using the negated best reward min -f*( attained during optimization (the regret cannot be computed because the max of f' is unknown). Note that we consider min,e - - (ax; instead of the cumulated - -, / (ax) because the goal is to assess the quality of the best combination of parameters discovered by the optimizer, rather than the cumulative negative reward across all explored combinations.
k (D) finds better combinations of parameters. For the design of a wireless network or for the particle packing problem, (P))) consistently discovers combinations of parameters with larger utility than both kavg and kn, (Figure right: Appendix E for more figures).
8

---

Published as a conference paper at ICLR 2026
Avg. Regret [G] (Ackley)
2yperoctaahedral222
Signflip-2d [G)] 44)
Matérn 512 -
10-1
Average Regrat..
Karz
10--
10"
10
10-5
20-1
30-1
10-4
Mutim-32 [-0.33,
20
Nutern-3 (-0.-33.00
10.10
10.113
424 [-1.0,11.7
10"
Number Symmetries [o]
Dipenvatio Index!
E Egeevarus nde
Hyyerootahhsira/22
Signflip: [GG]=))
Avg. Regret w.r.t. [G] (Rastrigin)
Average Regret R.f
80
Kine
Matérn-5/2
10
Eggmooluc AM0ma
10-1
10-
30-9
20
RBF on
1-100.107
10-*
-
RBF on
1-100.100 1.0
20.
on
/-2.39.0.35/
20m10
-.3..0..)
Kuy on1-0.01.00"
Awy on
(-1.0. 1.07
10
KP on
1.0.1.0
10-12
A!t on 1.0.1.9
10
10 Elgenro ue
12
107
167
161
10
18'
Number Symmerrics [q]
Elgenvalie Indeect
Figure 4: Leftt column: Final average regret R T for la, (blue crosses), lavg (orange diamonds), and k.)) (green circles) Ackley (top) and Rastrigin (bottom), averaged over 10 seeds with standard error bars. Middle and right columns: Empirical eigendecays under different bases and groups (ordered eigenvalues of the Gram-matrix divided by n). typical behavior on a single seed.
4.3 ROBUSTNESS TO GROUP SIZE
when the group size []] is small, but its performance deteriorates as |g] grows, whereas (P) remains stable. To investigate this effect more systematically, conduct additional experiments the d-dimensional Ackley and Rastrigin benchmarks, each invariant under the hyperoctahedral group of size [ 2dd! (permutations coordinate-wise sign-flips. We compare the average regret of kavg and (P))) after 50 iterations of GP-UCB for dimensions 1,...,5, 5, and include l;, as a baseline to control for the effect of increasing d.
Both synthetic and real-world benchmarks suggest that kavg performs comparably to (D)
The results are shown Figure (left column). Both experiments reveal the same trend: while kavg consistently outperforms kb, its performance also deteriorates as [G] increases. In contrast, k. remains largely unaffected by the growing number of symmetries, demonstrating a clear robustness to group size. In the next section, we discuss several explanations for these empirical observations.
Takeaway. (P)) consistently matches or outperforms kuvg and kn with the largest gains at large [g].
The evidence suggests that (i) how kernel encodes orbit alignments matters as much whether it is invariant, and (ii) averaging across many alignments can dilute informative similarities. These themes reconnect with our discussion in Section 5 and motivate analyses beyond eigendecay rates.
5 SPECTRAL ANALYSIS AND REGRET BOUNDS
So far, k. has shown consistently lower regret than kavg. despite comparable computational cost. A natural question is: existing BO theory account for such gap? Current regret bounds for GP surrogates proceed via the information gain, which is shaped by the decay of the operator spectrum of the kernel. In particular, faster spectral decay leads tighter regret upper bounds in standard analyses (Srinivas et al. 2012 Valko et al. 2013 Scarlett et al.) 2017 Whitchouse et al. (023).. We
compare the eigendecay of k(P) and liavg and ask whether it can explain the empirical gap.
Empirical eigendecays: similar or faster decay for kavg. Across our benchmarks, the empirical spectra of (P) and kavg exhibit very similar log-log slopes (decay rates). In several settings, kavg eigenvalues decay even faster than those of k see Figure 4 (middle and right columns). Under the usual theory, this would translate into similar, or potentially tighter, upper bounds for methods run with kavg compared to those with k((). A more detailed discussion of the empirical spectra in Figure and further insights are in Appendix

---

Published as a conference paper at ICLR 2026
Limitations of eigendecay as an explanation. Since kavg matches or exceeds A.?) in empirical decay rate, standard theory would predict similar or better regret upper bounds. Yet in practice we consistently observe lower regret for k(P)) (Section 4 This suggests that eigendecay alone does not capture the structural advantages of (P)) We outline possible explanations in the conclusion.
6 CONCLUSION
Our spectral analysis highlights gap between theory and practice: although liavg often exhibits faster empirical eigendecay than k(P), the latter consistently achieves lower regret. Standard eigendecay arguments thus fail to explain the observed advantage of k(P) We hypothesize two complementary explanations.
First, geometry vs. rates: eigendecay quantifies how fast spectra shrink but ignores which eigenfunc- tions are emphasized. In practice, livv often introduces similarity reversals, distorting the search geometry (Figure Ip. whereas k.")) preserves high-contrast alignments between orbits, just as kmax-
Second, approximation hardness: standard BO guarantees assume that the black-box f* belongs to the RKHS Hz of the chosen kernel k. In practice, this assumption unverifiable. One might therefore wonder whether the empirical gap between havy and (P))) could stem from misspecification, i.., different distances d(f", HRwwg) ) and d(f", Hygp,). Indeed, existing work (Bogunovic & Krause
2021 shows that when d(f", Hs)) > 0, regret necessarily degrades linearly with this distance.
However, this explanation does not apply in our setting. With an RBF base kernel kn. Hig is universal (iichhelli al. 2006), HR... contains all invariant functions including f", so d(s", Hk
and cannot exceed d(s", H.(D)). ). Yet hive still performs worse than (D)
This suggests that the relevant issue is not membership in Hkwrz or Hyt0, but how difficult f to approximate within each RKHS. BO ultimately constructs approximations from finite linear combinations of atoms k). Different kernels can thus induce different approximation rates.
This perspective might explain both our results and those of Brown et al. (2024). In their setup. the synthethic benchmarks f* are explicit linear combinations of relatively few Kavg (æ), -) atoms (between 64 and 512, depending on dimension; see their Appendix B. In such regimes, havg performs very well, likely because the GP posterior mean can in principle exactly recover f once those o, are sampled. Typical BO objectives do not exhibit this sparse expansion structure, which may explain why in experiments kavg consistently underperforms k k(P) and sometimes even the base kernel.
Developing regret bounds that capture this notion of approximation hardness, beyond mere member- ship in He. appears essential to bridge the gap between theory and empirical performance.
Finally, while focus has been empirical, note that the intrinsic data-independent version of k4P) which we called k4 and which we mentioned at the end of Section 3.2 (introduced formally
in Appendix C. provides a natural, data-independent analogue of the practical kernel k(P) We see k4 as convenient object for future theoretical work, as it cleanly isolates the PSD projection of kmax from the additional data dependence introduced by Nyström. We believe that makes K4 convenient starting point for any future theoretical work, in the same spirit as gradient flow serving as an idealized analogue of gradient descent.
ACKNOWLEDGMENTS
We thank the anonymous reviewers for their constructive suggestions, which led several improve- ments and clarifications in the final version. This work was supported in part by the Swiss State Secretariat for Education, Research and Innovation (SERI) under contract number MB22.00027.
*Consider (af)(x)= Eveo /(gx)/ G). the projection onto Haws (Brown al. 2024 Appendix A), and observe that if fm f with fo Ho, then Pfn + f with Pfa E Haww So universality of Hs, implies universality of H Hesse
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Sterling Baird, Jason R. Hall, and Taylor D. Sparks. Compactness matters: Improving bayesian optimization efficiency of materials formulations through invariant search spaces. chemrxiv, 2023a. doi: 1126644/chemxxi12222-22222
Sterling G. Baird, Jason R. Hall, and Taylor D. Sparks. Compactness matters: Improving bayesian optimization efficiency of materials formulations through invariant search spaces. Computational Materials Science, 224: 12134, 2023b. ISSN 0927-0256. doi: https://doi.org/10.. http://bbboor//1110001//.ommmmmo 2023.11212
Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G Wil- son, and Eytan Bakshy. Botorch: A framework for efficient monte-carlo bayesian optimization. Advances in neural information processing systems, 33:21524-21538, 2020.
Anthony Bardou, Patrick Thiran, and Giovanni Ranieri. This too shall pass: Removing stale observations in dynamic bayesian optimization. Advances in Neural Information Processing Systems, 01444266-477777 2024.
Anthony Bardou, Jean-Marie Gorce, and Thomas Begin. Assessing the performance of noma in multi-cell context: general evaluation framework. IEEE Transactions on Wireless Communica- tions, 2025.
Abibasheer Basheerudeen and Sivakumar Anandan. Particle packing approach for designing the mortar phase of self compacting concrete. Engineering Journal, 18(2):127-140, 2014.
hoffman-wielandt inequality in infinite dimensions. Pro- Rajendra Bhatia and Ludwig Elsner. The hoffman wielandt
ceedings of the Indian Academy of Sciences Mathematical Sciences, 104(4):483 Aug 1994.
doi: 10. 1..1000/FF0867116666 URL httph:: springer..nggee.ee com wtt.:/omwmatteeeeoooommmmoom
BFC2867116
Ilija Bogunovic and Andreas Krause. Misspecified gaussian process bandit optimization. Advances in neural information processing systems, 34:3004-3011 2021.
Theodore Brown, Alexandru Cioba, and Ilija Bogunovic. Sample-efficient bayesian optimisation using known invariances. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10- 15, 2024, 2024.
Yihua Chen, Maya R Gupta, and Benjamin Recht. Learning kernels from indefinite similarities, In Proceedings of the 26th Annual International Conference Machine Learning, pp. 145-152, 2009.
John B. Conway. A Course Functional Analysis, volume 96 of Graduate Texts in Mathematics. Springer, 2007. ISBN 978-1-4757-4335 doi: 10.1007/978 1-0119777----77---- 1-4757-4383-8
Ryan R Curtin, Parikshit Ram, and Alexander Gray. Fast exact max max-kernel search. Proceedings of the 2013 SIAM International Conference on Data Mining, pp. 1-9, SIAM, 2013.
Holger Fröhlich, Jörg K Wegner, Florian Sieker, and Andreas Zell. Optimal assignment kernels for attributed molecular graphs. In Proceedings of the 22nd international conference Machine learning, pp. 225-232, 2005.
Roman Garnett. Bayesian optimization. Cambridge University Press, 2023.
Thomas Gärtner. survey of kernels for structured data. ACM SIGKDD explorations newsletter, (1):49-58, 2003.
David Ginsbourger, Xavier Bay, Olivier Roustant, and Laurent Carraro. Argumentwise invariant kernels for the approximation of invariant functions. Ann. Fac. Sci. Toulouse Math. (6), 21(3): 501-527, 2012. ISSN 29000-996,2222775711 doi: 10.5802/afst.l 1343. URL https: /doi 0/5/10.58022. fat. 1343
11

---

Published as a conference paper at ICLR 2026
Aldo Glielmo, Peter Sollich, and Alessandro De Vita. Accurate interatomic force fields via machine learning with covariant kernels. Physical Review B, 95(21):214302, 2017.
Javier Gonzalez, Joseph Longworth, David C James, and Neil D Lawrence. Bayesian optimization for synthetic gene design. arXiv preprint wrxiv::505.01660 2015.
Nicholas Higham. Computing a nearest symmetric positive semidefinite matrix. Linear
Algebra and its Applications, 103:103-118, 1988. ISSN 0024-3795. doi: https//doi.org/
10. 10.101660022477999833392222--3-3 URL https::///// http::://wwwwwt// http:/wwwwwwweedd//////e rect. www..ereete..e.eeeeeeeeeeeeeeeee
BTt.ccle/p11100223375583888
Donald R Jones, Matthias Schonlau, and William J Welch. Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4):455-492, 1998.
Jungtaek Kim, Michael McCourt, Tackgeun You, Sachoon Kim, and Seungjin Choi. Bayesian optimization with approximate set kernels. Machine Learning, 1106))857-879, 2021.
Vladimir Koltchinskii and Evarist Giné. Random matrix approximation of spectra of integral
operators. Bernoulli, 6(1):113-167, 2000. ISSN 1350077265,1573--759 doi: 10.2307/3318636.
URL htRL:tttpp://oo.ogg/000.. 2307
Risi Kondor, Group theoretical methods machine learning. PhD thesis, Columbia University,
2008. URL.htttss://ppppleee. www.suchhcagg..ooo edwe -risi hctp//rris..ppppess/ooooo.. Kaappeers/oooorTheeeee
pdf
Ph.D. thesis.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C.J. Burges, L. Bottou, and K.Q. Wein- berger (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Asso- ciates, Inc., 2012. URL http:: http:/tppps/ppooceccisss.cooee [207///1|02239222333366676666663
cc/paper files/paper) pdf
John M. Lee. Introduction smooth manifolds, volume 218 of Graduate Texts in Mathematics. Springer, New York, second edition, 2013. ISBN 978-1-4419-999 9999999-988.
Pengfei Li, Xiaoyan Wang, and Hanbo Cao. Empirical compression model of perf--aah concrete considering the effect of cement hydration on particle packing characteristics. Materials, 16(13), 2023. ISSN 1996-1944 1944. 1996 1996-1944/66/3/4588
doi: 10.3390/ma16134555 URL https: /twww//wwwww///.// mdpi. com/
Daniel J Lizotte, Tao Wang, Michael H Bowling, Dale Schuurmans, et al. Automatic gait optimization with gaussian process regression. In IJCAI, volume 7, pp. 944-949, 2007.
Ronny Luss and Alexandre D' aspremont. Support vector machine classification with indefinite kernels. In J. Platt, D. Koller, Y. Singer, and S. Roweis (eds.), Ad- vances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., www.nne.ueeeeeaeeeeeee 2007. URL Flle://ccecccbbbbbbeertte12111e2 Paper Paper..p-
ffless/paper/2007/
Charles A Micchelli, Yuesheng Xu, and Haizhang Zhang. Universal kernels. Journal of Machine Learning Research, 7(12), 2006.
Dino Oglic and Thomas Gärtner. Learning in reproducing kernel krein spaces. In International conference on machine learning, pp. 3859-3867. PMLR, 2018.
Cheng Soon Ong, Xavier Mary. Stéphane Canu, and Alexander Smola. Learning with non-pon nosppositive kernels. In Proceedings of the twenty-first international conference Machine learning, pp. 81, 2004.
Peter Petersen.
Riemannian geometry, volume 171. Springer, 2006.
Chao Qian, Hang Xiong, and Ke Xue. Bayesian optimization using pseudo-points. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp- 3044-3050, 2021.
12

---

Published as a conference paper at ICLR 2026
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning. Adaptive Computation and Machine Learning. MIT Press, Cambridge, MA, 2006. ISBN 978-0- 262-18253-9.
Michael Reed and Barry Simon. Vi bounded operators. In Methods of Modern Mathematical Physics, pp. 182-220. Academic Press, 1972. ISBN 978-0-12-585001-8 doi: https///oii.rr// 1016 B978- 177880011--8550008850000-222 /
a
Jonathan Scarlett, Ilija Bogunovic, and Volkan Cevher. Lower bounds on regret for noisy gaussian process bandit optimization. Conference on Learning Theory, pp. 1723-1742. PMLR, 2017.
Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. Information-theorele regret bounds for gaussian process optimization in the bandit setting. JEEE Transactions Information Theory, 5855)3320--3265, 2012. doi: doi: 109/tit 1.2182033
Sylia Mekhmoukh Taleb, Yassine Meraihi, Asma Benmessaoud Gabis, Seyedali Mirjalili, and Amar Ramdane- Cheriff Nodes placement in wireless mesh networks using optimization approaches: survey. Neural Computing and Applications, 34(7):5283- 5319, 2022.
Aidan P. Thompson, H. Metin Aktulga, Richard Berger, Dan S. Bolintincanu, W. Michael Brown, Paul Crozier, Pieter J. in Veld, Axel Kohlmeyer, Stan G. Moore, Trung Dac Nguyen, Ray Shan, Mark J. Stevens, Julien Tranchida, Christian Trott, and Steven J. Plimpton. Lammps flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales. Computer Physics Communications, 271:108171, 2022. ISSN 0010-4655. doi: https://doi. org/10. ng/10..1016fj.cpe22222 108171. URL httpps:/ wwww///wwwww///./////////// www.:w/wceecee..c rect, com/ science/ Sttiiclees1//500145522008333
Michal Valko, Nathaniel Korda, Rémi Munos, Ilias Flaounas, and Nelo Cristianini. Finite-time 2013.
analysis of kernelised contextual bandits. arXiv preprint
SVN Vishwanathan, Alexander Smola, et al. Fast kernels for string and tree matching. Advances in neural information processing systems, 15:569-576, 2003.
Justin Whitehouse, Aaditya Ramdas, and Steven Z Wu. On the sublinear regret of gp-ucb. Advances in Neural Information Processing Systems, (4235266-35276, 2023.
Christopher Williams and Matthias Seeger. Using the nyström method to speed up kernel machines. In T. Leen, T. Dietterich, and V. Tresp (eds.), Advances in Neural Information Processing Syy- tems, volume 13. MIT Press, 2000. URL, http:htttps//pprcceeeeeeee
neuri cc wwwpappr.rpp.....ppccpp
Files/ Files/paper hwtps/aaeer//eee 2 10.00001111 http:/19111lddaa http://iie..eteaaaaaaaaeee///3/.
Paper pdf
Mohamed Younis and Kemal Akkaya. Strategies and techniques for node placement in wireless sensor networks: survey. Ad Hoc Networks, 6(4):621-655, 2008
Ziming Zhang. Maximum Similarity Based Feature Matching and Adaptive Multiple Kernel Learning for Object Recognition. PhD thesis. Simon Fraser University, 2010. PhD thesis.
13

---

Published as a conference paper at ICLR 2026
A PROOFS FOR SECTION 3
A.1 FULL STATEMENT AND PROOF OF PROPOSITION
We state Proposition formally and give slightly more detailed proof.
Proposition (Max-keme) covariance for invariant GPs). Let S. R2 be measurable spaces and let (finite or compact) group 9 aci measurably S. Let h ~ GP(O, ki.) be GP on S₁ with isotropic base kernel kn : (ae, x²) SxS > k[[ x' [2) where x Rz0 + RRo nonincreusing. Assume there exists Øg S + Sn satisfying (i) invariance: (g(x)) (g for all g E G, S
and (ii) minimal-distance representativity: (Øg(()) ) óg(x')ll2 ming.o eg lg g'x' 2. Define f() h()g(x))). Then f ~ GP(O, kmax) and it is G-invariant.
Proof. Since is a GP, f also a GP, and invariance follows from (i). Its covariance kernel is kimax since:
Cov ()))), [(r')]
Cov (M(do h((g
(dg((z) (z
R( min 11a g' (2)
(8)
9.9'E9
max x(llgæ: - g' at (12)
(9)
9.8 9.9'E9 E9
kimax (ax, x')
(10)
where we used (ii) in Equation { and monotonicity of in Equation (9) Note that compactness of G' guarantees that the minimum in (ii) is indeed achieved, which makes Equation true even when is not necessarily continuous.
A.2 AVERAGING MAXIMIZATION WITH DIFFERENT BASE KERNELS
We extend Lemma to the case where kavz and kimax are built from different base kernels. The result shows that even this more flexible setting, the coincidence kavg and kimax can only occur in degenerate situations.
Lemma 6. Let kn and ki, be two base kernels such that l| = || = and ky, (xx 1 for all x. Let kavg be the group-averaged kernel built from lb and kinax be the maximization kernel built from k. It holds
kavg
Kmax on the orbit (xx, gz) == {(hæ, h' h, 9}
for every x E X and g G, and only if
ko gær) kimax (I, ga) =1 for every and g G.
In particular, this forces ki, already exhibit a form of G-invariance on pairs (æ, ga)).
Proof. (=) Fix x and E G. Since by assumption K, is bounded by 1 and (x, x) = 1:
2 kimax (xc, gz)) max ki, (hær, gar)) 2 ki, (zx, xx) =
h,
SO kimx (x, gz) = 1. Now consider kiavg By definition,
Kavy (I. gx)
Σ kba h' h'
h.V'E
Each summand is bounded by 1 and the average is equal to 1 as kavg Therefore each term is equal 1, which proves hn kmax on (vx, gar). As this true for every G, this shows the result. The converse is immediate.
(x, (x. gx) = kmax( (2. gx) = 1.
This shows that even when allowing different base kernels for kavg and kmax. equality between the two kernels requires ks to already be argumentwise G-invariant pairs (x, gzx). This fails for standard choices (e.g. RBF kernels with translation or rotation groups), so averaging cannot replicate maxinization in practice.
14

---

Published as a conference paper at ICLR 2026
B RADIAL INVARIANCE: CLOSED FORM FOR Kavg
We prove the formulas provided in Example Let g SO(2) act on R2² by in-plane rotations, and let ,n, be the RBF kernel with lengthscale l: (a. x²) = exp( | xx x211/ Writing
= (r, 0) and x' = (s, (p) in polar coordinates, we have
kavg (x. x'
exp
poa0000ptta-p--p00 21
de d3.
(2th
Integrating the absolute angle and keeping only the relative angle v5 0 1 + a yields
kavg x') exp( r²0 where Io(2)
1
exp(
f
cos v) du exp (-12te²
1 Iol Io(
21
2π
2 Jo e² e v dib is the modified Bessel function of order 0.
C AN INTRINSIC PSD PROJECTION K4 AND ITS PROPERTIES
We first recall in Appendix C.I
why positive semidefiniteness is essential in the BO framework. Then, we turn to the intrinsic structure underlying our practical construction.
In the main text we defined data-dependent kernel (PP obtained by projecting the Gram matrix of kmax on finite set of samples D onto the PSD cone and extending via Nyström. This finite-sample
construction (P) is the star of the show in practice, as it convenient to compute and exhibits strong empirical performance. However, its data-dependence can complicate theoretical analysis.
In this appendix, we show that (P))) is the finite-sample facet of broader, intrinsic data-indepee iata-independent PSD projection k4 of kmmx which (i) preserves the G-invariance of kumax (ii) coincides with kmax whenever kmax is already PSD. Since the PSD projection kmax discussed here can also be applied to any other indefinite kernel k. directly introduce it for arbitrary kernel k.
We present the finite-domain "matrix" construction to build intuition in Appendix and then lift general domains via integral operators.
C.I WHY PSDNESS OF i MATTERS
In this paper, we consider = kmax and then project it onto a PSD kemel. Although there is no technical impossibility in rumning BO loop as in Section 2.1 with kernel that is not PSD doing so is poorly motivated: the fundamental assumptions underlying BO no longer apply, and the key quantities lose their meaning. particular:
the assumption f E Hy no longer makes sense because Hy is not defined for non-PSD kernels; the usual interpretation of BO as maintaining GP prior whose posteriors provide increasingly refined approximations of I* no longer holds (in particular μa, and Cov, are no longer GP posterior mean or covariance), since is not valid covariance structure for the prior; acquisition functions (UCB, EI, etc.) lose their principled exploration-cxploitatie meaning and may now behave unpredictably.
C.2 WARMUP: FINITE DOMAINS
We start finite domain S to build intuition. In that case, K, is simply Frobenius-nearest PSD truncation of the Gram matrix on the full domain S, which is unique, basis-independent, preserves G-invariance, and coincides with & when k is already PSD.
Let S (..,.... NN be finite, and let G act on S. Consider any symmetric kernel A on S with Gram matrix K E N (possibly indefinite) given by Kij k( v.). We define kG as the kernel corresponding to the Crobenius-nearee PSD projection of K (Higham 1988).
"Only step 2 in Section 2.1 may fail if K. + of & is non-invertible. One can use pseudo-invedd inverse or very large doa but the latter makes the posterior variance nearly flat, degenerating the procedure into blind exploitation.
15

---

Published as a conference paper at ICLR 2026
Lemma (Frobenius PSD projection and explicit form (Higham 1988). The optimization problem
K+ = arg mzn P K|FF has a unique solution and, for any eigendecomposition K
QAQ it is given by
K+ Q m,
where max(0,-) acts entrywise on A. In particular, the matrix k depends only on K (not on the chosen eigenbasis), satisfies K = O, and K = K iff K
= 0.
We define k+, the (Frobenius) PSD projection of k, as:
(412,, = K Digt i,j E [N].
(11)
Inheritance of G-invariance. Each element g induces permutation of the elements of S: let Ty be the permutations of the integers j E [I,...,N] defined by gæg = (xx(j). Denote by Pg the permutation matrix associated with Tg. For every vector v, the matrix Pg acts as (P = xx 2x550 (i)
which is equivalent to the action canonical vectors Pgej Cx₂(i) (Pg) Limmn())-
Invariance in the first component guarantees kmax (t)) Uz)) kimax ( 2;)) kmax (ri, 2.2) for every i, E {1,.... N,,,,,, i.e., the rows of K = (k(=;; x. )). are invariant under the permutation Tg. hence P. K = K. Thus, for any positive integer m. Pg km = (P, K) KKKKK) Km-1 Km so for any polynomial p such that p(0) = 0, Pap(K) K) = p(K). Now consider a sequence (px)n, of polynomials such that (0) = and (A) max(0, A) 50 for any À in the spectrum of K. In the
limit P. R2 K. hence k4 invariant under the action of G on the first variable (k4 (')xxx))= ky (ae, oz)),, and invariance along the second one follows by symmetry (K P. K4). This shows that b+ inherits from the G-invariance of & (equivalently, P,K K KP, for all g). We collect this result in the next lemma.
Lemma 8 (Invariance is preserved by the projection). Consider g g. P.K K, then P,K, K = KTP. Hence the projected kernel K4 is G- G-invariant on S x S.
Pj.
Relation to the practical Nyström kernel. If the set D = [0),.... (n.) used to build k4,) (Equa- tion (7) equals the whole domain D = S, then ( k4. Indeed, (P))) (x), xz) = K, K; K.j (K K K); = (K+)s )ij on D D, and the latter is the definition of k on finite domains.
We now generalize the matrix considerations above using integral operators. The finite-domain construction is recovered as special case.
C.3 GENERAL DEFINITION (VIA INTEGRAL OPERATORS THEORY)
We lift the finite-domain construction of the previous subsection to general domains by viewing k as a Schhidt operator and defining k4 the positive part of Tk: this yields a PSD, iata-independent kernel that inherits any G-invariance and equals i whenever lo is PSD.
Let (S, T.,4) u be probability space. For a measurable, symmetric kernel k : S x S + E with 1: E [,2(2 @ (4), let the (compact, self-adjoint) Hilbert-Schmian operator Tk [p)(p) + [1) be
(Txf)()) = S k(xx,)) x') f(x') du(x').
(Note that in the finite-domain case, f is a vector indexed by the domain and if μ is the uniform measure then T3 is simply multiplication by the Gram matrix K normalized by the domain size.) By the spectral theorem, there exist (A,, Φ1) with { orthonormal in L (²( and (Ax) of mixed signs) such that Te Eiz1 A di @ Φi in T,2(u) where for every u, E T2)()), the rank-one operator L (2) + (1)) such that (u @ v)f = (S, v) u for every f E L²())).
(possibly is
'ee can impose (0) = 0 since f(0) = O. Indeed, take pn (A) = (A) - gm (0) where is a sequence
given by Weierstrass" theorem, which converges f()) = max(0, X) the spectrum of K. We have (A) - f()) < Ig (A) f()) Ign (0) and because f(0) 0 we get gg. (0) = l4, (0) f(0) + 0.
16

---

Published as a conference paper at ICLR 2026
Generic definition of k+ via operator theory. Define the positive part of T3 = Ei di by
TE [x+++ @; 2 oi where (t)+ max(t, 0}. Since ((+++)) s E, 13 < c0, the series
(x,, = ((i)+ (z)) (μ @ v-a.e.).
(12)
converges in I.2 (14 @ (4) and defines a kernel 69 j-almost everywhere. By construction Tee Tt. hence 4+ is PSD as kernel a.c., and PSD in the operator sense: (f. 2 0 for all f (m).. In particular, if was already PSD (all X; 2 0), then k+ (up to null sets). It also inherits G-invariance of & if k is indeed invariant (the proof mimics the finite- moite-domain case, we give the full details for completeness Appendix
C.4 FROM THE FNNTTESAMPLE PROJECTION TO THE INTRINSIC LIMIT: WHAT CONVERGES TO WHAT?
We relate the practical, data-dependent Nyström kernel k (Equation ? to the intrinsic k4: under iid sampling, the empirical spectra of k.*2) /|D| converge to that of Tk with rates under mild moment assumptions. This shows that eigendecay-
regret analysis
Notations. Let 11, X2, ... 1 i.i.d. and Dn = (XI,..., ".,...,,)), We write K,, := :D,,, D,), K == arg minero |P- K, Kn == Kn/n, and recall that the practical (data-dependent)
kernel defined in Equation is
P (z,x') k(x, Dr,) ( k(D,.
We denote by X(T) the (ordered, nonincreasing, each counted with its multiplicity) sequence of eigen- values of compact self-adjoint operator T, and by 62( (e(A(T) A(S) == (E, (N((T) {ss / 1/2 the spectral & distance. For symmetric matrices M, X(M) denotes the nonincreasing sequence of eigenvalues of M (with multiplicity) padded with an infinite number of zeros. For a bounded operator A, ||A||s and ||Allop denote the Hilbert-Schmidt and operator norms, respectively. We include in Appendix C.5 reminder on the different notions of norms and convergence, and we now recall the essentials.
Relations between convergence notions. For compact self-adjoint operators:
max (5x( (TT), [T, - Tllpp) S ||T, Ti|ss (Reed & Simon 1972 Bhatia & El-
sner 1994); (ii) converse inequalities do not hold in infinite dimension (see Appendix C.5
examples). Thus, HS convergence is the strongest notion of convergence we manipulate here.
We now present convergence guarantees of the data-dependent construction k(Px)/ /n to the intrinsic k4 under progressively stronger assumptions. With minimal assumptions we obtain almost-sure spectral consistency in the d2 metric; with stronger assumptions we obtain quantitative rates in HS norm (hence also spectral lg in probability).
(a) Weak a.s. spectral consistency of positive parts (minimal assumptions).
Proposition 9. Assume the symmetric (not necessarily PSD) kernel k in L² 1)) so that Tk is Hilbert-Schmiae Let S, (pn) ) + ((²) be the integral operator with kernel k4 (Da) (x, x')/n defined by:
(Sn((z)) = (Da) (2)
(x. Xx))(X,).
(13)
Assume the X; are pairwise distinct almost surely. Then, almost surely,
82 (2(3))n), X(Tx.))
n=300 0.
IIdeed, by definition (Tk, (z) = Js ( ((a))) ((E)) du(x'))
(A)+ (x(=) =
do 2 ()) ((E) (aattaa))(((z(0000
(T (z).
17

---

Published as a conference paper at ICLR 2026
Proof. Let K, be the empirical operator on R" with matrix 1 (*((,, XX)x), and let X(K,,, be its ordered spectrum (nonincreasing, with multiplicity) padded with an infinite number of zeros. Theorem 3.1 of Koltchiissi & Giné 2000) shows that 62 (,,((K),,(TTA)) - 0 as + cc.
Let K be the positive part of K, (i.c., its Frobenius PSD projection). Since + max(0, A) is 1-Lipschitz, have for any operators S:
(2(((4)) X(S+)
Σ max A, (T)) maa-mmaxx--- (0, (O)) &
Σ (A)(T)) (((()))
We deduce that (2(A(K ((TR.)) 0 as 77 oc.
remains to observe that the spectrum of K as an operator on R the same as S, 1,2 (M
(.2(())))))))
T,3 (p,). This identification is standard (e.g., see above Equation 1.2 in Koltchinskii & Giné (2000)) For completeness, we include the formal arguments of Koltchinskii & Gine (2000) in Lemma [2] which shows that we can identify the spectrum of k4 (P.)) (Dn. Dz)/n with the one of K a.s. if the ii X; ~ Il are pairwise distinct a.s, which is true as soon as μ non-atomic; otherwise one can index the distinct atoms and work in Ih with m #supp(/",). obtaining the same spectral identity on that subspace.
(b) Expected HS convergence with Onn21/2 rate (stronger assumption). Define the empirical integral operator (f,.f) := E.= k(x,, (X,) and D, := T, Tg. Let (A,, Φ3) be an eigensystem of Tj in I,2 (p)). Assume the following fourth-order summability condition holds:
:
Σ x
0x(x)) 6x(x) du(z))
co.
(14)
Proposition 10 (Expected HS rate). Under & E (2 @ n) and
C
E 1 [is
n
Consequently, [|D, Op(n 122 and therefore using the same notations as in Proposition
(<(((K)) ((T=E)) Oppnn111))) ((Tx))).(TT,)) Op(n)1/2).
Proof. Fix any f E (1)).. By Fubini-Tonel for non-negative functions, we have:
B [lI
B [{((Dn()(=))) du(æ)).
By definition
(Dff((x) = Σ klx, - / k(xx ('), du(x')
where the randomness comes from the iid. X; Hence B[(D,..)(z)) 0 and for any fixed æ
a(z, d())) E ((D.)(()))) Var((D..f(zz) = Var x)(()) < n
n
The Hilbert-Schmidt spectral theorem gives the expansion k(æ, x')
"r)=x')== = E, (x)6,(() in (2( X
n), with (At); E e2 and (dh)) an orthonormal set of (1) (see Equation 3.2 in Koltchinskii & Giné (2000), Corollary 5.4 in Conway (2007)). Thus
E ((Dn))a))) du(x) = f( k(x, 3'2xx') f(x') du(a') (d))))
Edddy / (1(a')o (pr, dy) du((')
=limy
Σ x [
18

---

Published as a conference paper at ICLR 2026
Taking f Øj for fixed j yields
E[| Eno (w)) - Σ x
(a)
Since |DDIIIIss - E; D 05 we get the main claim:
BIID [ss S C'
Jensen gives the bound for D inequality infinite dimension (Bhatia & Elsner () and X A (0, X) is I-Lipschitz on R, hence the spectral bound probability claim using Markov's 's inequality, and Lemma 12 transfers this claims to Sn.
Finally, 82 A(K ) A(Ta)) < |DDaIlus (Hoffman-Wielan Wielandt
Condition 14 is a fourth-order integrability requirement that controls eigenfunction overlaps. Ir is standard in random Nyström analyses (see, egg. Equations (4.3) and (4.11) of Koltchinskii & Giné (2000) and stronger than L? but it yields dimension-free
Remark 11 (On assumption (14),
Onn 112) rate in HS norm.
(c) High-probability HS rates (heavier but more precise). Under slightly stronger -t-tyype conditions on eigenfunctions, the section 4 in Koltchinskii & (2000) gives more more precise statements on the rates in Proposition 10 and we directly refer the reader to it.
Application to kiax and to the BO kernels in the paper. When h kmax is bounded compact domain S (as in all our experiments), E L² @ μ)) for any probability measure μ on S, so T Kimps Hilbert-Schmide and Proposition appplies. particular, the integral operator associated (D.) /n, called Sn (Equation (3)) satisfies with k
82 (Sa),, A(R..)
a.s.
0.
This clarifies the two objects introduced the main text: the intrinsic k4 the unique data- independent target, while the practical kernel k<(Px) (finite PSD projection + Nyström) is on-path approximation whose spectrum converges (once normalized by n) to that of ki under i.i.d. sampling.
The following subsections only optional complementary materials added help building intuitions on the convergence results stated above.
C.5 REMINDERS ON THE DIFFERENT TYPE OF CONVERGENCES FOR BOUNDED LINEAR OPERATORS
This subsection recalls standard notions of operator convergence, included only as background to help build intuition for the convergence results above.
Definitions (operator norm, HS norm, spectral distance). Let be separable Hilbert space with orthonormal basis {e/321 For a bounded linear operator T' 7 H.
|T = Σ Te 1 1/2
TTlp = sup |Tflk
II bx=1
The HS norm is basis-independent When T is an integral operator with kernel L2 @ (²) (Reed & Simon 1972)
I /1 [k(x, v² du(()) du(y)-
For finite matrices, ||A||ss ||Al|e (Frobenius). We say T, T in HS norm if [[, Tilss 0. and we say T, T spectrally if & (A(T.)) A(T) O, where we recall that X(T) is the ordered eigenvalues a compact self-adjoint operator T, and where the spectral disdtstncce is 52(X(T), = (E (A((T) (2
19

---

Published as a conference paper at ICLR 2026
Which convergences matter, and how they relate (reminders well-known facts). We compare
three notions: (i) operator norm convergence |[T,, - TT op - 0: (ii) CelberrtSchhide (HS) convergence [[, - HS + 0: (ii) spectral convergence in 82, i.c., (2(A(In),
== (E, (T,)
(T)))))) /2 + O, where X(-) denotes the ordered eigenvalues of a compact self-adjoint operator. We recall the following well-known facts, useful to grasp the convergence results state next.
(1) HS = spectral 82. For compact self-adjoint operators the (infnite-dimensionaa) Hoffman- Wielandt inequality yields (Bhatia Elsner 1994)
82 8a(A(T)) ((T) < 1[Th TH|
(2) HS - operator norm. For every Hilbert-Schmice operator S. ||S||pp I ||S||s. Indeed for unit vectors x, E H, using x = (x, es)es. we have (Sx, y) = Eier (r, Ga) (Se,,)) y). By Buuchy-Schwarz:
(Sxx v)l
Σ (o, (o) ie7
1/2
Σ ET
(Se 2)} 1/2
The first factor equals ||>||| = 1. and for the second we use (Se, y>! = || Illll = ||Se,] to get
(Se,.
||eey| |S]
TEl
Hence ((5x, yRl S ||S||SS. Taking the supremum over all unit y gives
Sxxl
sup
(Sx, v) < ||||S
||v|==1
and then taking the supremum over all unit yields
||S||op
sup
|Sx|||
|S|]ss-
|
|=|==
(3) Spectral 82 does not imply HS nor operator norm. Even if eigenvalues match in Eg, the
operators may be far in norm because eigenvectors rotate. Let T (ia 1/3....) in the canonical basis (ei):21 and let Un swap ej and en. Set T, == TU* n Then X(T, = X(T) for all (same ordered spectrum), so 52 (A(T,), ((T) 0. Yet (T, T)e [IU TU" T))
[[L1/n 1)eill = 1 - 1/n, hence [T, - Tllop 2 1 1/n + and, a fortiori, [[T; TH A 0.
(4) Operator norm does not imply spectral 82. Let T and Tn be diagonal with the first mn entries equal to en and the rest Choose en -1/2 and m, Then ||Tn|lop En but 52 ((TT),
(E 22) vn. (1/n) 1.
(5) Two useful corollaries. (a) Spectral 6y-convereence implies convergence of the largest eigenvalue, since sup; (Ar(T,) Xx(T)) < (2(A(,,) X(T)). (b) Operator-norm convergence forces uniform eigenvalue deviations to vanish by Weyl's inequality: sup, (T,) Ax(T)) < [[,, Tllop- but does not control the fy-sum of all deviations.
Takeaway: HS is the strongest notion here: it simultaneously implies spectral òz-convergence (and thus convergence eigenvalue-based quantities) and operator-norm convergence. The converses fail in infinite dimension because eigenvectors can drift and an infinite number of tiny eigenvalue errors can accumulate.
C.6 IDENTIFICATION OF THE SPECTRUM OF AN EMPIRICAL OPERATOR IN (²())) AND ITS MATRIX COUNTERPART
Here show how the spectrum of the empirical operator can be identified with that of its matrix form. This is complementary material meant to clarify how operator-level and matrix-level viewpoints connect (which is useful, e.g., in the proof of Proposition
20

---

Published as a conference paper at ICLR 2026
Lemma 12 (Empirical Nystróm spectral identity). Let K, (k(z; 2))) and let K be its spectral positive part (the Frobenius-nearest PSD projection). Define the empirical measure Nn === ow: and the Nyström kernel
k+
(x, (x,x') x') k(x, D,)) (K k(Dn,,'')
Let Sn (pm) - (1a) be the integral operator with kernel k4 (Pa)) (ze, ) ie.
(an1)(z) 1 (ar, f(xz)).
The map E (2 RR. ES = ((m)) I( is isometry: || || [ll))))))) and we have the intertwining identity
E. If. in addition, the sample points æ1..... En are pairwise distinct, then E is an isometric isomorphism (hence invertible) and
ES.
(SS))) = ((;)) = 1(k (Dn: Dw)/m).
Proof. First note the on-sample identity (P))) (x,,,,)) = (K+)as for the unscaled K = (k(2,, 2,)),,. which follows from K(K+) ' Kt Hence k ((Da) Dn) K + and therefore
1(Da) (Dn, Dn)/n K. For f L2 (Mm) and each i E [I....,m).
1
Vnl (EŠ.f): (zi
(Z,, f(z,)
Σ (kt) f(X)) Vn EK)
which proves E S. = K E. Since is an isometry by definition of the 1,2 inner product, the X; are pairwise distinct then E is bijective and conjugates S, with K so the spectra (with multiplicities) coincide.
C.7 PROOF OF G-INVARIANCE OF k4 FOR GENERAL DOMAINS
We conclude this appendix with the formal proof that K4 defined in inherits from any group- invariance of k. This proof is not needed for the main results but is included for completeness. makes explicit why k4 preserves any G-invariance of k. The proof follows the one for finite domains but is heavier in notations because it is now stated using integral operators to generalize the matrix manipulations of finite domains. For finite domains, denoting by K the Gram matrix of k over the whole domain and P, the permutation matrix induced by the action of E G on the domain, invariance of & is equivalent to P,K = KP = K. Thus any polynomial pl K(K) of K such that p(0) 0 inherits from this invariance since still have Pop(K K) P p( p(K). And the limit, we get invariance of K4. Here, we mimic this proof, and we start by introducing the equivalent integral operator form of the characterization PyK K K KP. K for general domains.
operator commutation). Let (S.T.H be probability space and let G act measurably on S. Assume is G-invariant. Let U, T,²(K) + T.²()) be the unitary
Lemma 13 (Kernel invariance
representation (U) == f(g-tx) Let E L²(H @ μ)) be symmetric kernel with integral operator (Txf)(x)) = k(æ, f''(((()))) du(x'). Then the following are equivalent:
(i) k is argumentvise G-invariant: k(gæ, x²) = kkk,, gx') k(æ, for μ @ p-a.e. x²) and all
g E G. (1i) Ty satisfies UgTx = T2U, -T on I (1)) for all E G.
Proof. (ii)) For any E &T?(1)),
(U,Tif)(z) (T x)= / "2,'''( dp(x').
21

---

Published as a conference paper at ICLR 2026
By invariance of lo in the first argument U,Tx = Tμ. Hence T T and T Ta (self-adjoint) and U = Ug-1 so Th = Tg. This is true for all E g hence UgTh = TuU = Tk-
(i)) For p, v E (H),,
|| M(z x²) ((z) du(a))d((z')
( = (p, Twv) = (p. v.)
Expanding the last inner product, we get by change of variable and invariance of μ
If (zz, x') (())))(g - '') dp(z))d((z') /] (( ga') ((x)) /
dz(z))du(z))
Hence for all p.v. [[[ 2²) - kax gat) )] (l)))())))))))) ((')) du(ar)du(a") = 0, which implies k(x, gx') = k(æ, x²) μ @ p-a.c. Symmetry implies argumentwise G-invariance.
We now show that UgT= = T is preserved we apply a function f such that f(0) = to the spectrum of
Lemma 14 (Borel functional calculus preserves invariance). Let T a self-adjoint compact operator on Hilbert space H with eigendeccomposition T E, r,o; @ di and let geg be unitary representation such that UgT = TU = T for all g E g. For a bounded Borel function f R. - R, define f(T) E, f(x)), @ Φ₁. Then for such with f(0) O, we have UT) U
f(T) for all g g.
Proof. Proof sketch: The assumption U,T = T forces Ug to act as the identity on each nonzero eigenspace of T, which directly yields U, (T) f(T) for any bounded Borel f with f(0) 0.
Step 1 (spectral decomposition for compact self-adjoint T without measures), Since T is compact and self-adjoint, its spectrum is o (T)) = {0} U {An
I} where finite or countable, each An * 0 is an eigenvalue of finite multiplicity, and An - 0 if infinite. Let Ex denote the eigenspace for x * 0, and Fug = ker T. We have the orthogonal decomposition
H
Eo +
Ex:
and T acts as scalar multiplication on each Ex: Tl AIdg TE 0. Let PA be the orthogonal projector onto Ex (for X & 0) and Po onto Eg. Then for every E H with expansion
v 1o Ex4o UX (VA = Pxv), we have
Tu Σ AuX AgE
Step 2 (U, fixes each nonzero eigenspace pointwise). From UgT = we get, for any v E Ex with X 0,
AU,e U. (Tv) = Tu Au, hence Ug = v. Thus Ug acts as the identity on each Ex (A + 0). Equivalently, UgPs = PAU, Px for all > # O. (There is no restriction on Ug inside Eg ker T.)
Step 3 (defining f(T) for bounded Borel f with f(0) = 0). Because (T)) \ {0) is at most countable and T diagonal on {Ex}. we can define f(T) by applying f on the spectrum of T as
f(T) v = Σ
f(A) ux.
0 0o
UY ex Ex.
(T
The series converges in norm since the Ex are mutually orthogonal and [T = Exso | S (sup ([( Σ Exto || < | Thus S(T) is a bounded operator with [I) & || (When f(0) = 0, there is no contribution on Eg.)
Step 4 (invariance and commutation). For Un Σx Ut as above and any g E G, Step gives UyV Ug + Exs0 Ut and P2Ug- PA for + O. Hence
U2J(T) v Ug Σ 1())
Σ f(A)
Σ f(A) S(T)
0
22

---

Published as a conference paper at ICLR 2026
Le., UTJ(T) f()). In particular = f(T)
for all g g.
Consequence. If A is G-invariant, then so is K+ (Equation 12
D EIGENDECAY COMPARISON
In this appendix, we discuss more details the empirical observations made in Section 5 and formally derive some inequalities between Schatten norms of integral operators associated with kavg and k4-
D.1 EMPIRICAL OBSERVATIONS
Here, we further discuss the empirical spectra reported in Figure C (middle and right columns),
Computation of spectra. The normalized Gram matrices K (where K (k(2,, 25)ssss reported in Figure are computed from n = 3000 i..dd samples 2; E S. We compare the spectra obtained with & {kib kavgs k(P)) } with D [ and each æ; being chosen uniformly in S S --1,,, 1j. We also report the spectrum of k when observations æ; are instead sampled from an alternative domain S' of reduced volume, chosen such that vol(S') vol(S)//)) Finally, note that because D is set of i.i.d. observations, the spectrum of (PP approximates the one of k4 on S (see Appendix C.4) so our observations transfer to k4.
(D) k on S vs. lb on S'. For the base kernels hn and groups & considered, the spectrum of k. on
= [-1,1] 1] exactly matches that of kb on the reduced domain S'. This indicates that A (P) faithfully incorporates the extra similarities induced by G-invariance: it retains the eigendecay of kb, but as if it were defined on the quotient space S/g of effective volume vol(S)/
(D) on S kavg on S. From Figure 4 (middle and right columns). it is clear that the spectrum
of kavg decays at least fast that of k They coincide for the RBF kernel and kavv decays even faster for the Matérn kernel. In principle, this suggests that havg should admit tighter information gain bounds and thus better regret guarantees. However, our empirical results contradict this prediction, as x(P) consistently outperforms kuvg- This discrepancy highlights the fact that eigendecay alone does not fully explain BO performance, as pointed out in Sections and
D.2 SCHATTEN NORM INEQUALITIES
While the empirical spectra in Appendix D. already highlight a mismatch between eigendecay and observed BO performance,
may ask whether formal inequalities between the operators induced by kavg and k4 can be established. We record here for completeness that is possible to control the Schatten class of k4 in terms of the one of kavg-
Assume: (S, J1) is a probability space on which a finite group G acts measurably, and the base kernel lo is bounded, symmetric, PSD, and nonnegative. Define
Kavg (x, x')
kolgæ, g'x'), 9.9'E9
Σ
Kmax (x, (x,x') x'))===== max kbb g' x')
g
9.9 9.9'E9
and h4 as the kernel corresponding to the positive part of Tkmms: Tk., (Tkmax)+
Schatten norm interpolation. Let H (n) be the separable Hilbert space of squared integrable functions on (8, T H + H compact operator, and write (T") for the singular values of T, i.e. (T) (T*T), arranged in nonincreasing order and counted with multiplicity. The Schatten-p norm is defined as
ITIs, = (TT)))
1 1sp<o
|| == sups;(T)
For finite group of isometries, indeed has vol(S/9) vol(S)/1g) (Petersen 2006
23

---

Published as a conference paper at ICLR 2026
Lemma 15 (Monotonicity for pointwise kernels). If two kernels k. k' are bounded and satisfy I k I K pointwise, then ||T I [Tx for p = 2, oc. If k and are also PSD, then [T& ss Isslls,, for too.
Proof. For p = oo, the Schatten p-norm is the operator norm || = /uPpl/ ITS Point- wise 0 - k < k'' implies [Txf S |f I Is Il
so taking the supremum over Ilfll 1 yields [Talls> S [|T If = Tk the integral operator associated with a nonnega- tive kernel k. then || = ||k|] (=(#)))- Hence pointwise 0 < k S k' gives [[Tallss S [[Te-lss for 2 as well. Finally when A: is PSD, we have ||Txlls, f k(x, x))p(x) (and similarly for k) and again a pointwise comparison yields the result.
From this immediately obtain, for specific kernels that for p 20, and also p if kiax PSD:
Kavg S kmax
S |9 havg
[T lls, S | [TRmma lls, < | I (YTRavall Ils,
Lemma 16 (Interpolation inequalities for Schatten norms). For any nonnegative sequence (0:)121 one has
|allee> - ||al| 2/2 |al|
(p 2 2),
||a| $ ||a| ||a| (((-1
(1 p 2
Proof. For 2, E; = E; Ej 2 < ||a|b E; at giving the stated inequality, For 1 S p S write
Σef of
2 p-P p 2(p-1)
Let 2-p and p-1 1 < r, s < x and by Hölder,
(with the usual convention 1/0 = oo). For < p <
we have
ΣΦp
aa $ (Ca 1/r
((aa 2(p-11) 1/8
Σ (a,
ai
1/r
a
1/+
Since 1/r 2 p and 1/s = p 1, this gives
| ||a ||a || |||| 2(0-1)
The endpoint cases 1,2 follow by continuity (and are trivial directly).
Applied to a; = 8; (T), Lemma
16 yields the standard Schatten interpolation inequalities:
ITTls, I ||T113
2/v I|T1|
2/p
(p 2p2,,
|Tlls, z (1TTIss) 3 (1T1
(1
Since the spectrum Tk, is the positive part of the one of T Thmas we have The Ils, S |T Ils, We deduce the next lemma.
Lemma 17. For 2 2:
TK Ils, S WKTh Is, S T R 11 TiTkw
11
1-2/p
and if kinas is already PSD then for 1 S S
Tik lls, = ||Timmeel 1lsp 5 19 [Gl([/iaac lls.)) )1.s)
(1Tiave) 2-111-1/0
1-1/p
and
Tk Ils, 5 (11T [1s:) 2/0-1 (1
Knn 112,) 1-1/v
24

---

Published as a conference paper at ICLR 2026
Ackley2d
Griewank6d
Rastrigin5d
Matérn-5/2
- Kwy
4000
Cumulattvv Regieet R:
Matér
Cumulative Ragret
Matern-5/2 (DA)
Cumulative Regrett
3000
Kavy
5000
2000
200
2000
1000
--- Matém 5/2 - k
1000
Kan
0-
40
40
40
Iteration
Iteration
Iteration
Radial2d
Scaling2d
autveative Regret &r
500
RBF
Cumulative Regret R.
Kava
20*
400
300-
10a
200
10
100
Kaup
20
75
100
Iteration 7
Iteration
Figure 5: Cumulative regret under GP-UCB with ky (blue crosses), Kavg (orange diamonds), and
(P)
(green circles) on synthetic benchmarks. Shaded areas: standard error over 10 seeds.
WLAN8d
Neg. Particle Packing Fraction
PartPackéd
Neg. WLAN Throughout
Matérn-5/2
kD
Kavo
-40
-50-
-0.8
-60
-0.9
Matérn-5/2 + kim
70
Kava
0
40
10
20
30
Iteration
Iteration
Figure 6: Negated simple reward under GP-UCB with lb (blue crosses), kavg (orange diamonds),
and k (green circles) on real-world experiments. Shaded areas: standard eтror over seeds.
E BENCHMARKS
In this appendix, we present additional results and describe the experimental setup of Section in detail.
E. EXPERIMENTAL FIGURES
We provide the whole set of figures generated from our experiments synthetic benchmarks (Figure 5) and on real- problems (Figure
E.2 EXPERIMENTAL DETAILS
In our experiments, every BO algorithm is implemented with the same BO library, namely BOTorch (Balandat et al. 2020). All of them are initialized with five observations sampled uniformly in S. After that, at each iteration I, every BO algorithm must:
Fit its kernel hyperparameters. This done by gradient ascent of the Gaussian likelihood, as recommended by BOTorch. The hyperparameters are the signal variance X, the lengthscale and the observational noise level oi Optimize GP-UCB to find æt. This is done by multi-start gradient ascent, using the oot imi a function from BOTorch. As values of 3, recommended by SSinivas et al. (2012) turn out be too exploratory in practice, set 3, 0.5d log(t).
Observe y(x;)) = f (xx,) + &,. Function values are corrupted by noise whose variance is 2% the signal variance.
25

---

Published as a conference paper at ICLR 2026
We optimize over 50 iterations and typically measure the cumulated regret along the optimizer's trajectory.
laptop equipped with an Intel Core 19-9980HK 2.40 GHz with cores (16 threads). No graphics card was used to speed up GP inference. The typical time for each maximization problem ranged from ~1 minute (two-dimensional Ackley, [ = 8) to 1~5 minutes dive-dimenssonal Rastrigin, |g] = 3840). The particle packing problem
All experiments replicated across ten independent seeds and
by far the most time-consuming experiment due to the expensive physics simulator used for computing the objective value of each new query (~44 hours for 30 BO iterations, which we repeated 10 seeds for each kernel).
E.3 BENCHMARKS
We maximize the following functions.
Ackley. The d-dimensional Ackley function fAckley on [-16, 16) with global maximum fAckley (0) with fAckley defined by:
d
SAckley (x:) a exp
exp
Σ cos()):)
+ + exp(1), (15)
d
where we set = 20, = 0.2 and
2T as recommended.
The d-dimensional Ackley is invariant the hyperoctahedral group in dimensions, which includes
permutations composed with coordinate-wise sign-flips. Consequently, in d dimensions, [G]
2d
sign flips permutations
The d-dimensional Griewank function fGriewank S maximum fCriewank (0) = 0, with - fCriewank defined by:
Griewank.
[-600, 600j4 with global
d
Srrewwakk (x)
COS
+ 1.
4000
1=1
The d-dimensional Griewank is invariant to coordinate-wise sign-flips of all d coordinates. Therefore, dimensions, [ 2d.
Rastrigin. The d-dimensional Rastrigin Rusttiiin S .5.12,5.12" with global maximum
Rastrigin (0) = O, with RRastrigin defined by:
RRsttigin (x: = 10d +
Σ
(x? 10 (2mT::)
1=1
The d-dimensional Rastrigin is invariant the hyperoctahedral group in d dimensions, which includes permutations composed with coordinate-wise sign-flips. Consequently, in d dimensions, 2d
G
sign-tlips pennutations
Radial Our radial benchmark is defined on S = 10, 10 with global maxima fRadial (x") 0,
where any x E S such that |x) l2 ab. has the following expression:
||ا|||
[Radial (xx)
JRastrigin
(16)
where 10 0.8 and where fRassigin is the one- dimeeninsio Rastrigin benchmark.
Our radial benchmark is invariant to planar rotations. Consequently, G comprises an uncountably infinite number of symmetries.
26

---

Published as a conference paper at ICLR 2026
Scaling. Our scaling benchmark is defined S [0.1, 10]² with global maxima fscaling (x=) 0, where x" is any æ (I1, x2) E S such that #1 = 222 The function - -fscaling has the following expression:
J11
-fecaling (x: =
I22
Our scaling benchmark invariant to rescaling of both coordinates. Consequently, comprises an uncountably infinite number of symmetries.
WLAN. The goal of the WLAN benchmark is to place m access points (APs) inside a square region A = [-50, 50]2 so to maximize the total communication quality over p users located in A. a recurring problem in wireless network design (Younis & Akkaya 2008 Taleb et al. (022). Given set of AP positions, each user connects to its closest AP, and the resulting network throughput- computed from the Signal to Interference plus Noise Ratio (SINR) and Shannon capacities defines the value of the objective function.
The user positions {(u,, U5)}Eb C A and all physical parameters (W, L, X, N) are given. The region A itself is fixed.
The variables of the problem are the AP locations
(a. y) ((z1,... Em), (yi....,, Vm)) =AA"
so the search space is 2m-dimensional. Every quantity below below-AP received powers, SINRs, and capacities-dependds on (æ, y).
AP--APPuu user associations, distances,
For candidate placement {I }: each user attaches its nearest AP. Thus AP serves the users in
U( ui) = {j E ip] dis S dky for all & ş ii),
and the SINR is (ties are resolved arbitrarily) where the distance to user j
dij / (z, u2)² + (yi U3 2
For any associated pair (i,j), the power received by user j from AP is
Poj = 1014 min(d,7 1),
Ps
Mis = N + Enzs Pki
The corresponding Shannon capacity is
Cj W logz + 7g).
Maximizing the WLAN performance amounts to maximizing the total throughput (the cumulated sum of Shannon capacities for every AP-user association):
fWLAN(æ y) = Σ
Σ
Cj
jEl((zE.3n)
viewed as a function of the AP locations (x., y)-
In experiment, set W MHz, L 46.67 dBm, X 3, N -85 dBm, m 4 APs and 16 users.
Our objective SWLAN invariant to any permutation of the APs: permuting both and y with the same permutation leaves the objective value unchanged. Therefore, |g] m!!
Figure shows the best AP AP-placement found by GP-UCB using (PP)
training run.
Particle packing problem. The partiele packing fraction (PPF) problem models how a mixture of spherical particles settles under gravity inside fixed rectangular box. This setting originates from
27

---

Published as a conference paper at ICLR 2026
WLAN
67
40
2.2
20
(m)
Users
APs
20-
40
-40
-20
20
40
x (m)
WN with the best positions of APs found by GP-UCB with k.P), APs are depicted by red triangles and users with blue circles. The throughput for each user shown in Mbps.
Figure 7:
gramular-material physics and is routinely used in materials science and civil engineering (e.g., in the design of concrete mixes (Li et al., 20233 Basheerudeen & Anandan 2014) by tuning the size distribution and proportions of aggregates to maximize packing density; for instance to need less cement and water, and get better mechanical properties).
People literally design concrete mixes by tuning the size distribution and proportions of aggregates to maximize packing density (so you need less cement and water, and get better mechanical properties).
In this problem, mixture of particles is first instantiated inside the box according to prescribed mixture parameters, and the particles are then allowed to fall under gravity. Collisions, frictions and rearrangements determine the final configuration, and the packing fraction is defined as the ratio between the total particle volume and the volume of the smallest axis-aligned box that contains all particles after settling.
We fix the number of particle types to n. Each type is described by:
diameter di in a prescribed interval [dmin: dmax);
a share 8, in mmii $max], representing the relative proportion of particles of that type in the mixture.
Thus the optimization variable is
x = (dj, dn, $1..... (1.))
The box size and the total initial particle volume Vp (which then remains constant during the simulation) are fixed in all experiments.
Given a mixture specification x = (di,..., dz, $1....,5,) sn), the initial particle configuration is generated by repeatedly sampling particles until a fixed total particle volume V, is reached. Particles are sampled independently as follows: (i) sample a type E (1...., n) with probability proportional to its share 8;, (ii) sample location uniformly diameter d, there. If any overlap of particles occurs during initialization, positions are adjusted locally so that the configuration becomes valid. From this randomized initial state, the system evolves under gravity, in practice we use a physics- The simulation proceeds until the particles reach mechanically stable configuration. as illustrated in Figure 8 If (a( denotes the volume of the smallest axis- axis-aligned box enclosing all particles at the end of the dynamics (i.e., the container volume after settling), the particle packing fraction is
random in the container and put a partiele of
simulator (LAMMPS (Thompson et al.) (022)) ) for that.
Vp PPF(x)
Vx(x)'
and we aim at maximizing this function of the mixture parameters æ. To our knowledge, there is no accurate closed-form expression for this dynamical packing fraction our setup, so evaluating PPF(x) requires running the full physical simulation. Indeed: PFF(() is actually a random variable: given any mixture parameters σ, V=( depends on the random initialization of the particles in the container, so there observational noise induced by this random initialization. Moreover, even if the random seed was fixed, because V.(x)) depends on complex interactions during the fall fall- collisions, friction, and rearrangements, there is still closed form available: evaluating PPF(x:) always
28

---

Published as a conference paper at ICLR 2026
requires running this full physical simulation. This makes the objective function costly and genuinely black-box, a typical regime where BO well motivated.
Figure 8: Particles settling under gravity in fixed-size box. A single evaluation of (aF requires simulating the fall from randomized initial configuration (left) to mechanically stable state (right), making the objective expensive and simulation-based
Two symmetries are inherent to this formulation:
Share scaling: multiplying all 8; by the same positive factor leaves the resulting mixture unchanged (the mixture only involves normalized shares).
Permutation symmetry: permuting the (d,, 8;) pairs does not change the mixture either.
In practice, we take n = 3, which is the smallest setting where the problem starts to be interesting (no easy solution) while keeping simulation costs manageable. We constrain the diameters and shares to
dj [0.35, 0.80].
8; E [0.1, 1.0],
chosen so that (i) all particles remain sufficiently small relative to the fixed box size, and (ii) each type is represented non-negligible quantity.
Baird et al. (023a) previously applied BO to this problem (for solid rocket fuel design) and handled these symmetries by restricting the search a fundamental domain and applying standard kernels there. In contrast, we keep the domain unchanged and instead use kernels that are invariant under the symmetries of the problem. A conceptual comparison between these coametry-handllinn strategies is provided in Appendix
F COMPARISON OF SYMMETRY-INVARIANT KERNELS WITH THE AUGA-TTNMNNNTTION APPROACH
Given the widespread use of data augmentation (DA), we compare symmetry-invariad kernels with the simple baseline corresponding to using the base kernel combined with DA. We find that commetry-invarian kernels perform better overall.
DA consists of replacing each input in the dataset by (gz)geg; with G- G, and BO is run this augmented dataset. We consider two scenarios: (i) using all augmentations for small groups (G), = G for all 2)) so that (g=) simply the orbit of z, and (ii) using random subset G'z g for larger groups (chosen independently for every drawn uniformly without replacement).
the two-dimensional Ackley function (left panel of Figure D). ka is applied to dataset augmented with all symmetries (g = 8). In this case, k, with DA achieves slightly better (lower) cumulative regret than As alone. Its performance, however, remains worse that of the average kernel kavg and the
PSD projection of the max kernel k(P) A similar pattern appears the three-diment dinenssiia Ackley function (right panel of Figure where DA uses 20 augmentations sampled without replacement from ((9) 48).
We also report the runtime of each method. These results show that k; +DA scales less favorably than kavg and (P) even when using only moderate random subset of augmentations. Overall, these experiments suggest that using commetry-invariann kernels directly is more practical for Bayesian optimization than relying data augmentation.
29

---

Published as a conference paper at ICLR 2026
Ackley2d
Ackley3d
400
Cumulative Regret Rt
Matérn-5/2
Kava
Matérn-5/2
kavo k12n
Matérn-5/2 (DA)
k(
Cumulative Regret
Matérn-5/2 (DA)
300
400
200
200
100
20
40
20
40
Iteration T
Iteration
Figure 9: Cumulative regret the two-dimensional (resp., threc-dimensional) Ackley function, with
[G] = (resp.. G 48).
Table 3: Average wall-clock time in seconds per iteration for each method on the two-dimensional (resp., three-dimensional) Ackley function.
Benchmark
IG]
l 0.416
k with DA
lavg
(D)
Ackley2d
8
0.253
0.599 + 0,279
0.451
0.273
0.924 0.444
Ackley3d
48
0.506 + 0.336
2.665 = 2.950
0.590 + 0.384
1.307 = 0.724
G WORKING WITH FUNDAMENTAL DOMAINS AND QUOTIENTS
This appendix expands on the brief discussion in Section 2.2 about search-space restriction and explains why our approach targets kernel design rather than the choice of domain. The goal is to clarify that both ingredients, good domain and good kernel, are needed and complementary.
G.1 FUNDAMENTAL DOMAINS AS QUOTIENT REPRESENTATIONS
Given a domain and a group action G, restricting the search to fundamental domain amounts to choosing concrete embedded representation of the quotient space S/G in S. While this is conceptually elegant, the practical implementation depends heavily on the pair (S,9) and must be re-derived for each new problem.
G.2 EXAMPLE: PERMUTATIONS OF Rd
In several of our experiments, S [a, 5/d and g = S, acts by permuting coordinates. Two vectors equivalent if one is a permutation of the other. A natural choice of fundamental domain is the sorted cone
C {ar [a, bj : S.... xa)
which is one possible representation the quotient S/G (other equivalent views include multisets or d-atomic probability measures, but these views does lead to subsets of the original domain so they do not qualify as "fundamental domains").
Even in this simple case, practical issues appear.
(1) One must characterize and project onto the quotient, and check that it "smooth enough". ". Most BO implementations assume that the search domain is a box [a, b] for which enforcing feasibility of the iterates is straightforward (via coordinatewise clipping x -> max(a, min(b, x))). If we optimize an acquisition function over the fundamental domain C instead, any gradient-based or heuristic optimizer will typically propose points that lie outside and these must be projected back. This requires (i) describing the quotient S/ / via an explicit embedded representation (here, C S) and (ii) figuring out how to implement the projection. For C, projecting & onto amounts to solving
(roje E arg min
||y
=||²
y1 $I..-ppaa
which can be solved efficiently using known algorithms (c.g. the pool adjacent violators algorithm). Our point is that this particular projection is hard, but that for each new pair (S, 9) the user must
30

---

Published as a conference paper at ICLR 2026
again derive an explicit model of the quotient and practical projection operator, which can be burden depending on their goals and familiarity with quotients and the problem at hand.
Smoothness assumptions also need to be checked. The cone C' is not smooth manifold, implying that the projection is not smooth everywhere and gradients are not smooth (or even properly defined) at certain points. Here, the singularities form zero-measure set: they occur at points with some equal coordinates (this is because the action of G is not free; in contrast, if the action were free, proper, and smooth, Theorem 21.10 in Lee (2013) would guarantee that the quotient is smooth manifold). For many constrained sets, singularities similarly form a negligible set and may be harmless for optimization (initialization and gradient descent are likely avoid them), but this depends on the specific quotient and must be verified on a case-by-case basis.
Overall, working in the quotient means that the user must (i) characterize and project onto a potentially non-smooth quotient, and (ii) check that its singularities do not cause difficulties for the optimization method they use. Doing this for each new (S. 9) may burdensome. This why, this paper, we choose to avoid optimizing in fundamental domain and instead provide kernels that can be used in plug-and-play manner directly on S. These same kernels could also be used on the quotient space (by interpreting them kernels equivalence classes), so our approach is complementary to, rather than in competition with, the choice of the search domain.
(2) One must still choose a kernel on equivalence classes. Working on S/G does not remove the modelling choice: one still needs to pick kernel [[[ar), [v]). and there no canonical option even in the permutation example. The quotient can be described in several equivalent ways (sorted vectors in C, multisets, or atomic measures), and each viewpoint naturally suggests different classes of kernels distances. This is precisely the type of question kernel that is invariant to the symmetries? We study a natural construction: start with a "good" kernel on S (e.g. one that makes sense locally S measure similarity before accounting for symmetries), and then make it invariant by aggregating via mean or max. The resulting kernels are G-invariant and thus well-defined on the quotient, and our results show that the max-based construction shows good properties, both empirically and geometrically.
paper addresses: how to construct good
H USE OF LLMS
We made limited use of large language models (GPT-5) during the preparation of this manuscript. Their role was strictly restricted to grammar correction, improving clarity and conciseness, emphasiz- ing text (e.g., bolding), and formatting tables. They were not used for generating technical content, suggesting new concepts, or contributing to proofs or results. All ideas, proofs, experiments, and findings are entirely our own. Every rephrased passage was carefully reviewed and validated by the authors to ensure correctness and faithfulness to our original intent. No unverified or plagiarized content was introduced.
31


Reviews:
[{"review_id": "lXRksBzdlT", "reviewer": "Reviewer_6LKt", "summary": "This paper proposes Symmetry-Aware Bayesian Optimization (SABO), a framework for exploiting known or learnable symmetries in black-box optimization problems to improve sample efficiency. Traditional approaches, such as Brown et al. (NeurIPS 2024), incorporate known invariances into Gaussian Processes through orbit-averaged kernels. This paper introduces a max-alignment view that aligns input pairs via the most similar symmetry transformations.\nTo address the fact that the resulting kernel may violate positive semi-definiteness, the authors introduce the projected-max kernel, which applies a PSD projection based on spectral (eigenvalue) analysis to restore validity. \nOverall, the paper contributes a more general method for incorporating symmetry structure into BO, extending symmetry exploitation beyond fixed and known invariance groups to settings where symmetries can be learned or adaptively aligned.", "questions": "Repeat from the weakness section:\n1. Why PSD projection matters?\n2. Why consider a spectral analysis?\n3. The authors can explain more about how kernel validity relates to PSD properties, and why “valid covariance” matters in GP inference, and what specific advantages “max alignment” brings in the Bayesian optimization setting (e.g., in terms of acquisition behavior, uncertainty modeling, or convergence properties).", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper presents an original and conceptually appealing extension of symmetry-exploiting Bayesian optimization methods, offering a practical and theoretically grounded approach that goes beyond prior invariant-kernel formulations. By introducing the max-alignment view and the projected-max kernel, the authors propose a framework allowing Bayesian optimization to leverage approximate or continuous symmetries without requiring exact group knowledge. The integration of spectral analysis and PSD projection is technically sound and shows thoughtful consideration of kernel stability, while the empirical results convincingly demonstrate the benefits of sharper, symmetry-aware modeling.", "weaknesses": "This paper's presentation significantly limits accessibility and clarity, especially for a broader audience. It does not clearly explain why PSD matters and what is the motivation of spectral analysis. Similarly, although the paper includes justification of the max-alignment view, the explanation remains largely heuristic. It does not provide an analytical argument or formal characterization of when or why this leads to better Bayesian optimization performance. As a result, the motivation still feels conceptually underdeveloped and difficult to follow for readers without deep kernel-theoretic background.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r8GzpG3dPw", "reviewer": "Reviewer_4Tfz", "summary": "The paper introduces a Gaussian process kernel that handles symmetries (invariance over a group action), for the purpose of Bayesian optimization. One approach is to average the kernel over the symmetries. This paper proposes instead taking a max, which is not necessarily PSD, so then a straightforward PSD projection is included as well.\n\nThe paper does a theoretical analysis that provides strong motivation for the use of the max rather than the average over the group for the kernel. Empirical results show that this new kernel performs much better than baseline approaches for handling symmetries. Spectral analysis shows that existing generalization bounds do not predict the improved performance, and the discussion raises ideas for why that might be.", "questions": "N/A", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The paper is very clear and is well written.\n\n* The paper addresses an important issue (the detrimental effect of symmetries on GP modeling with standard kernels, and thus on BO performance). The core idea to take the max instead of the average over group orbits is not novel in the general case, but its application to Bayesian optimization is novel, and the paper provides a significant amount of useful analysis to go along with this that make it, in my mind, a very original work.\n\n* I especially appreciated the strong theoretical motivation in Section 3.\n\n* The paper seems particularly appropriate for ICLR as it is about representation learning with symmetries.", "weaknesses": "In Section 2.2 the paper describes data augmentation as \"a popular way to enforce symmetry.\" I certainly appreciate that it is intractable in many settings, but by virtue of being popular, it should be compared to in this work. In Ackley2d it could be done directly, since |G| is only 8, and so with 50 total iterations, it becomes a lot of data but still feasible. On other problems, the popular strategy would be to randomly sample a few (still 8 perhaps?) items from G and just augment with those. These are pretty easy to implement and would strengthen the results of the paper significantly. It should come paired with including results showing wall times; the paper suggests that k_avg and k_max will have similar wall time, but it does not actually compare them both to k_b, and so it isn't clear if increasing the amount of data by 8x will make k_b slower than k_max and k_avg.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qS34LScY8v", "reviewer": "Reviewer_mZAc", "summary": "The authors claim to contribute:\n- Introduce a PSD projection of the max-alignment kernel over a symmetry group to avoid typical kernel averaging strategies, which can dilute high-similarity alignments when the symmetry group is large. This kernel is a valid GP kernel and doesn’t change the asymptotic cost of GP inference, so can be used for BO when the objective function is invariant under a known symmetry group.\n- Their max kernel behaves as one expects on toy problems and does really well on standard BO test cases in comparison to the average kernel, which is the closest invariant kernel in the literature so far. It also outperforms standard kernels.", "questions": "I’m not an expert; why is it difficult to identify a suitable fundamental domain to restrict the search space to, if one knows the symmetry group? Is there an easy example to demonstrate this? It would be good to explain this to readers less familiar with symmetry (which includes much of the BO community).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Paper is overall well written, clear, and concise. It outlines relevant related work and explains why their method might be preferred to each, situating their method well within the literature. Exposition of method is clear.\n- Method seems general—can be used with any base kernel and be PSD thanks to their construction. Can be used with any symmetry group, just need to maximize over orbits. This operation scales quadratically with the size of the symmetry group but this is a problem for previous invariant kernels too (e.g., average kernels).\n- Clear 1-D examples showcase how the max kernel more accurately models the objective than the average kernel.\n- Regret is impressive compared to base and average kernels.\n- Compares findings with spectral analyses and finds a gap, and suggests new ways to explain this gap, which may lead nicely into future work.", "weaknesses": "- I would appreciate more real-world BO test cases that are group invariant beside the one WLAN problem, to demonstrate that this is actually an important innovation for applications people care about. More argument about the relevance of this area in the intro, e.g., examples, could also help.\n\n\nGeneral feedback for improving paper: I think the intro is a little abrupt and a smoother motivation of why we care about max alignment before introducing notation and the PSD projection of the max kernel would help the reader.\n\n\nI strongly recommend acceptance — the paper is well-written, introduces a max-alignment kernel which is not in the BO literature yet (to best of my knowledge) by approximating it with a projection to make it PSD. This leads to really good results on both toy, interpretable settings and on standard (group invariant) BO test functions. It seems clear that this work would be interesting to the BO community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)