Why High-rank Neural Networks Generalize?: An Algebraic Framework with RKHSs
Authors: Yuka Hashimoto
Sho Sonoda
Isao Ishikawa
Masahiro Ikeda
Venue: ICLR 2026 Poster (2026)
Keywords: Generalization bound
Deep neural network
Koopman operator
Reproducing kernel Hilbert space
Primary area: learning theory
TL;DR: We derive a new Rademacher complexity bound that describes why the models with high-rank weight matrices generalize well, which is valid for a wide range of models.

Abstract: We derive a new Rademacher complexity bound for deep neural networks using Koopman operators, group representations, and reproducing kernel Hilbert spaces (RKHSs).
The proposed bound describes why the models with high-rank weight matrices generalize well.
Although there are existing bounds that attempt to describe this phenomenon, these existing bounds can be applied to limited types of models.
We introduce an algebraic representation of neural networks and a kernel function to construct an RKHS to derive a bound
for a wider range of realistic models.
This work paves the way for the Koopman-based theory for Rademacher complexity bounds to be valid for more practical situations.

Published as a conference paper at ICLR 2026
WHY HIGH-RANE NEURAL NETWORKS GENERALIZE?: AN ALGEBRAIC FRAMEWORK WITH RKHSs
Yuka Hashimoto ,2, Sho Sonoda Isao Ishikawa Masahiro Ikeda 5.2 NTT, Inc., 2. RIKEN AIP, 3. CyberAgent, Inc., 4. Kyto University, 5. The University of Osaka
ABSTRACT
We derive a new Rademacher complexity bound for deep neural networks using Koopman operators, group representations, and reproducing kernel Hilbert spaces (RKHSs). The proposed bound describes why the models with high-rank weight matrices generalize well. Although there are existing bounds that attempt describe this phenomenon, these existing bounds can be applied to limited types of models. We introduce an algebraic representation of neural networks and kernel function to construct an RKHS to derive bound for wider range of realistic models. This work paves the way for the Koopman-basee based theory for Rademacher complexity bounds to be valid for more practical situations.
INTRODUCTION
Understanding the generalization property of deep neural networks has been one of the biggest
challenges in the machine learning community. The generalization property describes how the
model can fit unseen data. Classically, the generalization error is bounded using the VC-dimension
theory (Harvey al. 2017 Anthony & Bartlet] 2009). Norm-based (Neyshabur al.
2015 Bartlett
it al. 2017 Golowich et al. [018
et al.] Wei & 2a| Li et al. 2021 Ju et al.
2022 Weinan E et al. (2022) and compression-basee based (Arora et al.[20118) Suzuki al. (0200) bounds
have also been investigated. The norm-based bounds depend on the matrix (p, 7) norm of the weight matrices, and the compression-bed based bounds are derived by investigating how much the networks can
be compressed. These bounds imply that low-rank weight matrices and weight matrices with small singular values, i.e., nearly low-rank matrices, have good effects for generalization. See Appendix C for more details about the existing bounds.
the other hand, phenomena in which models with weight matrices that are high-rank and have large singular values generalize well have been empirically observed (Goldblum et al., 2020). Since the norm-based and compression-basee bounds focus only on the low-rank and nearly low-rank cases, they cannot describe these phenomena. To theoretically describe these phenomena, the Koopman- based bound was proposed (Hashimoto et al., 2024). Koopman operators are linear operators that describe the compositions of functions, which essential structures of neural networks. This existing bound is described by the ratio of the norm to the determinant of each weight matrix as
o ( Grl| o, II, [WI] E=1 VS de (W;
where S is the sample size, si represents the smoothness of the ith layer, Gj is factor determined by the 1 ~~ Lth layers, Ka; is the Koopman operator with respect to the activation function di, and ||. ||æ, represents the operator norm in a Sobolev space H,. Since the determinant factor appears in the denominator of the bound, even if the weight matrices are high rank and have large singular values, this bound can be small. The Koopman based bound theoretically sheds light why neural networks with high-rank weight matrices generalize well.
However, the existing analysis for the Koopman-based bound strongly depends on the smoothness of models and the unboundedness of the data space, which excludes realistic models with bounded data space and with activation functions such as the hyperbolic tangent, sigmoid, and ReLU-type nonsmooth functions. In addition, the dependency of the bound on the activation function not clear. In fact, the factors Ka, Hr, and G in the bound (1 hard to evaluate in many cases.

---

Published as a conference paper at ICLR 2026
Modal: unbounded aclivation (eg) smooch Lusky
space (RKH5)
Kooppan operslar feeprdduring property
*Applicabls unredisttic
Existing
.Hard
Sabolev
Modal: vider cless of activation functions includ ReLUU
Moolicablee
parameter space constructed function space
-Reproducing
-Easy factors
sigmoid, and tanh
Figure Summary of the framework of the existing and proposed Koopman- based bounds
In this paper, propose new Koopman-based bound that resolves the issues of the existing Koopman-based bounds. The proposed bound is described as
o ii
Gill llc. 1= VS det det(W; (WWW))//
where || - llc, is the operator norm in I function space. Similar to the existing Koopman-based bounds, the proposed bound describes why high-rank neural networks generalize well. On the other hand, the difference of the function space Li from H: gives significant benefit to the proposed bound. We note that L; is larger than Hr, and L; enables us to analyze nonsmooth deep models and bounded data space. In addition, enables Lemmas 2.5) and understand the effect of the activation functions on the deep model. As result, the proposed bound significantly improves the existing bound the sense that can be applied to wider range of models and enables us to understand the models well.
evaluate the factors |Ko,, lc. and Gi easily (see
To achieve the above improvement, we introduce a kernel function defined on the parameter space using linear operators to construct reproducing kernel Hilbert space (RKHS) that describes realistic deep models with nonsmooth activation function and bounded data space. We use the Rademacher complexity to derive generalization bounds. The Rademacher complexity measures the complexity of the model, which also describe the generalization property. Using the reproducing property of the RKHS, the Rademacher complexity with the operator norms of the linear operators. For linear operators, we group representations and Koopman operators. We first focus algebraic representations of models using group representations. A typical example is the representation the affine group, which describes invertible neural networks. We then focus on representations using Koopman operators with respect to the weight matrices, which describe neural networks with non-constant width. We schematically show the summary of the framework of the existing and proposed Koopman- based bounds in Figure
Hilbert space to which models belong. This kernel function allows us
bound
The main contributions of this paper are as follows:
We introduce an algebraic representation of models that represent deep neural networks as typi- cal examples. To describe the action of parameters on models, we focus on group representations, which enables us to represent invertible neural networks, and Koopman operators, which enables us to represent more general neural networks (Subsections 3.1 and (.T).
We define a kernel function to construct RKHS that deseribes the model. We derive Rademacher complexity bound using this kernel (Subsection 3.2). The proposed bound describes why the models with high-rank weight matrices generalize well for wider range of models than the existing bounds (Section 4 and Subsections 5.2 5.4).
Notations and remarks For E N and a Lebesgue measure space X' C Rd of complex-valued squared Lebesgue-integrable functions on X' We denote by μu the Lebesgue measure on X. For Hilbert space H. let >x be the inner product H. We omit the subscript 7. when it is obvious. We denote by B(H1, H2) be the space of bounded linear operators from H4 to H2. particular, we denote B(H, H) = B(H). All the technical proofs are in Appendix
d let (²)) be the space
PRELIMINARIES
2.1
KOOPMAN OPERATOR
Koopman operator is a linear operator that represents the composition nonlinear functions. Since neural networks constructed using compositions, Koopman operators play essential role in
2

---

Published as a conference paper at ICLR 2026
analyzing neural networks. Let X' R be Lebesgue measure space. Koopman operators defined as follows. We also introduce weighted Koopman operator, which a generalization of Koopman operator.
Definition 2.1 (Koopman operator and weighted Koopman operator). Let X' C R and X C IRd The Koopman operator K,, with respect to map : X - X' is a linear operator from (²)xx) to (X') that is defined as K, hx) = h(a(z)) for h E (2)). In addition, the weighted Koopman operator K v. with respect to maps 10 X + C and X' + X' is a linear operator from L (X) to 1,3 (20) that is defined (a))(()) ())))(((())) for (2)).
We will consider the Koopman operators with respect to activation functions. Throughout this paper, we assume these Koopman operators are bounded.
Assumption 2.2 (Boundedness of Koopman operators). The Koopman operator Ko with respect to a map a is bounded, i.e., the operator norm defined as || K, = sup | is finite.
Indeed, we have the following lemma regarding the sufficient condition of the boundedness of Koopman operators.
Lemma 2.3. Assume a X + X' is bijective, aー is differentiable, and the Jacobian of a is bounded in X. Then, we have ||all S supzex (=)= where Jo-! 1 is the Jacobian of In particular, the Koopman operator Kg is bounded.
The following lemma is regarding the boundedness of well-known elementwise activation functions defined as or ([x1,.....
..ci.................. ,..,...,,,.. x=]) = [ō(=).. ... [ā(z)] for a map R R.
Lemma 2.4. Let X [an, b1] x... (aa, bal S Rd be a bounded rectanglar domain, and let X (x)) If a the elementwise hyperbolic tangent defined as (x) = tanh(x), then we have X c [-1, 1] and [K,] I (TT suPnEa((,, 1/(1 x²)))²/ 1/2 If a is the elementwise sigmoid defined
as (x) = 1/(1 + e), then we have X C [-1, and || S (II suPsea(las.bi) 1/(x 1/2
Even if a is not differentiable, the Koopman operator is bounded, and we can evaluate the upper bound some cases.
Lemma 2.5. Let X = X = Rd. Let a be the elementwise Leaky ReLU defined as ö(x) = or for S 0 and ((x) = r for > 0, where a > 0. Then, we have |Koll max
2.2 REPRODUCING KERNEL HILBERT SPACE (RKHS)
In addition to the 2 function space, we also consider reproducing kernel Hilbert spaces. Let θ be a non-empty set for parameters. We first introduce positive definite kernel.
Definition 2.6 (Positive definite kernel). A map le θ x θ > C called a positive definite kernel it satisfies the following conditions:
k(0,, 02) k(02, 81) for 0,, 02 E θ Σij= (cckk(0, 0,) 2 for n E N, cs E C, and 0, E e.
Let Φ θ C be the feature map associated with k. defined as x(0) = k(: for 0 θ
and let RR,0 (E ((0,)cc) n N, es C. O, O (i I,..., n)}. We can define a map
(,)) Re,0 R2,0 + Cas
Σ (0,)cc
(E)d
Ed)
Ra
The reproducing kernel Hilbert space (RKHS) Ra associated with A is defined as the completion of Ry.D- One important property of RKHSs is the reproducing property ()( f() for RR and 0 O which is also useful for deriving Rademacher complexity bound.
3

---

Published as a conference paper at ICLR 2026
2.3 GROUP REPRESENTATION
Group representation is also useful tool to analyze the deep structure of neural networks (Sonoda
ct al. 2025). Let G be locally compact group. A unitary representation p G + B(H) for Hilbert space H is map whose image is in the space of unitary operators on H, that satisfies p(9122) = p(gn }((g)) and p(gi") = p(g))" for 91, 92 E G, and for which 9 + p(g) is continuous for any h H. Here, means the adjoint. If there exists nontrivial subspace M of H such that P(g) S M for any g E G, then the representation p is called irreducible.
For irreducible unitary representations, we have the following fundamental result (see, e.g. Folland [1995 Lemma 3.5)). which we will apply to show the universality of the model. Here, the commutant of subset A S B(H) is defined as the set {A E (HH) AB BA for B E A}.
Lemma 2.7 (Schur's ' lemma). A unitary representation p G is irreducible if and only the commutant of p(G) contains only sealar multiples of the identity.
We also apply the following fundamental result (sce, c.g. Davidson 1996 Theorem 1.7.1)).
Lemma 2.8 (von Neumann double commutant theorem). Let A be a subalgebra of B(H) that satisfies "A "A E A = A E A" and is closed with respect to the operator norm. Then, the double commutant (i.e., the commutant of the commutant) of A is equal to the closure of A with respect to the strong operator topology.
3 PROBLEM SETTING
We formulate deep models, which include the neural network model as a special example, using operators. Then, we define an RKHS to analyze the deep model.
3.1 ALGGEBRAC REPRESENTATION OF DEEP MODELS WITH GROUP REPRESENTATIONS
Let G be a locally compact group and p G + B(H) be a unitary representation on a Hilbert space H. We consider an algebraic representation of L-layered deep model in H
fin, 9t.) A1
A1-11((LLV,
where Ø1, ......... 91 G are learnable parameters, A AL B(H) and v E H fixed.
Example 3.1 (Scaled neural network with invertible weights). Let G GL(d) Rd be the affine group and 76 = L ( Here, GL(d) is the group of d by d invertible matrices. Let p G + B(H) be the representation of G on H defined as p(g)h(()) E L (RR). and x E Rd. Note that p is an irreducible unitary representation. In addition, let E (Rdd) be the final nonlinear transformation, 0r Rd - Rdd be an activation function satisfying Assumption 2.2 and A₁ Ka, be the Koopman operator with respect to oi for 1,.... For example, is the elementwise Leaky ReLU. Then, the deep model
det (/ for g = (W.
G.
(2) is
..0......aa f(gh.....92) v( ( (W 01 (W1 a W1b1) ... x det W1 1/2... det Wr-11 1/2
W Wi--bb-1) Wyb1)
Example 3.2 (Deep model with new structures). In addition to describing existing neural networks, can develop new model using the abstract model (2). Let G {(a, c) a, Rd C R) be the Heisenberg group (Thangavelu) 1998). The product in G is defined as (a1,b,, (1) (az, b2, 62) = (a) + a2 b2, 1/2 (a1, b2)), where (a), b2) is the Euclidean inner product of aj and by. Let H = L2 (Rd) and G + B(H) be the representation of G on H defined as p(g)h(x) = (c (() 11 b) for = (a, c)., where is the imaginary unit. Note that p is an irreducible unitary representation. Let and A be the same as in Example 3.1 Then, the deep model (2) is
f(g f(g......g)()))) (z) (lll--aann (ac
Hars) Mos (x bi)) ... (1----- (z.b......bb--- bb..)) b1,-)
bı) bI-1) bz)-
4

---

Published as a conference paper at ICLR 2026
Instead of directly considering the model ( we focus on the following regularized model with
parameter с > on a data space Xy:
Felgh...... ,.. IL. x) (p(())
AL-1P(OL),, Pc,z).
where Pc.x: E H for c > and x E Xo. We assume for any c > O, there exists E(c) > 0 such that |po.=|] < E(c). This regularization is required to technically derive the Rademacher complexity
bound using the framework of RKHSs. However, the following example indicates, the regularized model (3) sufficiently approximates the original model (2).
Example 3.3. Consider the same setting in Example 3.1 Let Pe,. = (c/x)4/2 -oll clla-=] for c >> and x E Ko. Then, Pe,z E T.2 (TR) and ll || 2c/m)d/2 Since pe,. goes to the Dirac
delta function centered at æ as c + 0o, we have
Fel91.-...91.. x)
Al11)) A2 ... A
Note that for any TRd and any 91,...,90 91 E G. lima- Felgi.... IL, x) f(g .L)( Thus, if e is sufficiently large, Felgi..... IL, x) approximates f(gi..... ,9) (z) well.
3.2 RKHS FOR ANALYZING DEEP MODELS
We use the Rademacher complexity to derive a generalization bound. According to Theorem 3.5 in Mohri et al. (2018), the generalization error is bounded by the Rademacher complexity. Thus, we obtain a Rademacher complexity bound, then we can also bound the gencralization error. To derive Rademacher complexity bound, we apply the framework of RKHSs. The Hilbert space H to which the models belong does not always have the reproducing property. Indeed, a typical discussed in Example 3.1 Thus, we consider RKHS that is function space on the parameter space G and isomorphic to a subspace of H. We can regard the deep model on the data space Xn as a function on G through this isomorphism and make of the reproducing property on G. Here, the isomorphism ensures that the mathematical structure of the RKHS is the same as the subspace of H. We define the following positive definite kernel k: (G x --. x G) x (G x ... x G) + C to construct an RKHS to analyze the deep model ( k((g1.... "llgi.....gII)) .LL), (gi..... (L)) = (p(g1)A
example of H is (PR)))
A p( A1 AL-IP(ōL)O)
We denote the RKHS associated with k as RR.
Let g (91 (g1.....,hu 9L). Ø(g) k(.g), and o(g) p(g))A1 ... AL-1P(OL)U. Let Ko {E Gā(g) n E N,g. E G E C} and Ko. Note that K. is a sub-Hilbert space of H. L K + R2 defined as u(h) = (g + ())) The map enables us to regard the Hilbert space K. where the deep model defined. as the RKHS R.. Proposition 3.4. The map i is isometrically isomorphic.
If is irreducible and A AL are invertible, then we have K. = H. which means that the deep model (2) has universality. The following lemmas are derived using Lemmas 2.7 and 2.8 Lemma 3.5. Assume p is irreducible. Let A = {Ei c(p())) n N.
E CJ. Then, A
dense in B(H) with respect to the strong operator topology.
Lemma 3.6. Assume p is irreducible and A AL-1 invertible. Then, K. = Ko H.
4 RADEMACHER COMPLEXITY BOUND
We apply the isomorphism in Proposition 3.4 to derive a Rademacher complexity bound with the aid of the reproducing property in the RKHS R&- If Pc.x E K., Eq.. (3) implies Fe(:, ((Pe.)) E Rx for x E Xy and e > O. Thus, we can apply the reproducing property with respect to the model ar). Let Ω be a probability space equipped with probability measure P. Let S E N be the sample size, I1,....25 E Xo. and Eg Ω + C be i.i.d. Rademacher variables (random variables following the uniform distribution on {-1, For a measurable function
- C. we denote by Fi(]] the integral Jg <( The empirical Rademacher complexity R(FF, (1FF,1......0
of function class F is defined as R(F .i..... Øg) E[suprer Σ F(x,)ce//S. We denote by F. the function class (F.(g)......1... .t..) 91.... 9r. E G}. The Rademacher complexity of F. upper bounded as follows.
5

---

Published as a conference paper at ICLR 2026
Theorem 4.1. Assume Pe.x ffr E No. Then, the Rademacher complexity of the function class Fe is bounded as
R(Fes 21 DI............88 ks) _ |Al
|| |IIv||B(c)
VS
Remark 4.2. If p is irreducible and , AL-1 are invertible, then by Lemma 3.6 the assumption of Theorem 4.7 is satisfied automatically.
then we have E(c) = (2c/m)d/2 d/2, Combining with that there is tradeoff between F. being close to the original model f and the constant E(c) becoming large.
Remark 4.3. If Po.x = (c/m)//2eeddl))
the discussion in Example 3.3 we
An important example of models that can be analyzed using this framework is invertible neural networks.
4.1 INVERTIBLE NEURAL NETWORKS
Consider the same setting in Example 3.1 Note that since P is irreducible, the as- sumption of Theorem 4.1 is satisfied in this case (see Remark 4.2). Let nn(g, x) = (WLoL-I(WL-1" 01 ( W1b1).... -.---W---bee -Wz-bb-1) WLbr) be neural network model.
Then, we have f(g) nn(g. -) det W,/2/2... /et W2/1/2. Thus, we have Fe(g.)) NN det Wi] W//1/2.... det W1 1/2, where (N. x) = J (()))e..yyddd Let D -1/2 -1/2 det Wr| W < D}. We assume A is
and NN, = (NN.(g.) g G det
invertible for =1, 1,....,. 1.
Theorem 4.4. The Rademacher complexity bound of NN is
E(c)[oo I L-1 ||ll
z8)
II det Wi -1/- t=1
sup
det W//-1/2
VS
For example, if oy is the elementwise Leaky ReLU, then ||| is bounded as Lemmal 2.5 Since det Wi the product the singular values of Wi, and is in the denominator of the bound, Theorem 4.] implies that the model can generalize well even if W, has large singular values.
5
GENERALIZATION TO NON-CONSTANT WIDTH NEURAL NETWORKS
5.1 ALGEBRAIC REPRESENTATION OF DEEP MODELS WITH KOOPMAN OPERATORS
In pervious sections, we focused single Hilbert space H and consider operators on H. This corresponds to considering a neural network with constant width. In addition, H is determined by the group representation, which forces us to consider a certain data space such Rd However, in general, the width is not always constant. In addition, the data space is bounded in many cases. To meet this situation, we consider multiple Hilbert spaces Ho, a set of parameters and let 9, - BH,, H4-1) for I = 1. 1...., L. In addition, let v E Hy and A E B(H. Hi) be fixed. Consider the model
HI-11 Hi,,. Hy. Let O be
... 02) 7n (172())) AL AL-17Z(01)e) where 0g 0 for L,..., L.
In the same manner as Subsection
3.1 we consider a regularized model
F.(O,,... ...., Oz, .)
(01))102((4 ... Ar..17r.. po,.,)
where Pe,x E Ho for x E X, and c >>>>>>>>000 0 with ||po.zl| < E(c) for F(e) > O. We also define a positive definite kernel & (O) deep model p
x θ2) (0,
θτ) C construct RKHS analyze the
A0((,, 0z.,, (ō) ........ θr.)) = (m (0:)A, ... Ar. 1/.. (0n) mn (0:)A) ... Ar.- 1n. (n. No
We set R2 and IC in the same manner as in Subsection 3.2 This generalization allows us to derive Rademacher complexity bounds for wide range of models.

---

Published as a conference paper at ICLR 2026
5.2 ( E Chidi (W -...
NEURAL NETWORK WITH INJECTIVE WEIGHT MATRICES
Let di
E
N and O
W is injective).
Let
No c
Rd,
(WWa-----aa (W1)(WW
X S 124,
and
oW( W... (21)(W1)))) 5 N s Ied that satisfy Maa (R4
0 and Maa (Ri)
> 0.
Starting from Nu. we re- currently construct & and Xi for I = I,.... I. Since Wi injective, the space W is d--1- dimensional. If di > d-1, then the measure Mg4 11) becomes O, and setting X = - makes the analysis mean- ingless. Thus, we set a space X that includes W and Mad; (WiX)-1) 0. Figure 2 schematically shows the con- struction of X and X, Let Hi = (X)). Hi = (X)). and m (Wi) operator from H4 to Hi-1 with respect to W,. addition, let A = Ka, be the Koopman operator from Hi to Hi with respect to an activation function Øj Xi - Xi that satisfies Assumption 2.2 Then, have
Wi
XG
X1-1
X
Mer (X = 0 Me ({-1)
(g=
(xx >
Figure Construction of X and X;
Kw; be the Koopman
S(W (WII...,W.(())) = (WWoo (I(W))))))
Let D > 0 and Fe = (11........ ,1.- det W 1 -1/4 ..... det W/ < D}. Let dppx)) (x)] (d(=) (z)) for h H4. This value depends how large set X; compared with W, and by setting & sufficiently large, can bound it by 1 with a reasonable assumption (see Remark for more details). In the same way as in Theorem 4.1 obtain the following bound.
(h) = ( 1h(x)
Theorem 5.1. Assume Pe,z K. for E No. Let Si vo WT W0 W4+1 Then, have
E(C)|I|II[ [All(())
R(FF,)21,...., x8)
sup det WT W/A VSII det W/Wij'/4"
As for ||Arl since As Koy, we evaluate the upper bound of |||i|| by Lemma 2.3 For example, X% is bounded, we can apply Lemma to4 the sigmoid and hyperbolic tangent.
Remark 5.2. For simplicity, we consider models without bias terms. We obtain the same result for models with bias terms since the norm of the Koopman operator with respect to the shift function is 1.
Remark 5.3. Assume there exist b >> 0 such that S [f((x)))2 < b. We set X, sufficiently large so that b (R..W,) (W/X-)) S pmga (&.). Then, have
affi)² SwiN (2) (I) b- PR(()) (Widz-1) ( $1 J& (z(())) (x)
a Mgg (2i)
Remark 5.4. There is a tradeoff between the magnitudes of the denominator and the numerator of the bound ( When 0)(2)) tends to be constant as ||>||| -> oc, such as the hyperbolic iangent and sigmoid, the derivative of ) tends to be large as the magnitude of ||ж|| becomes large. In this case, according to Lemma 2.3 if det Wt is large, then ||i]] is also large since the volume of Xi becomes large. The activation function plays a significant role in increasing the complexity in this case. When o1...., 0L-1 are unbounded, such as the Leaky ReLU, Xr. becomes large if det WI...., det Wr, are large, which makes ||l|| large. The final nonlinear transformation v plays a significant role in increasing the complexity in this case.
Advantage over existing Koopman-basee based bounds Hashimoto et al. 20244) proposed Rademacher complexity bounds using Koopman operator norms. Since the norm is defined by the Sobolev space,
7

---

Published as a conference paper at ICLR 2026
the framework accepts only smooth and unbounded activation functions. In addition, although they include factors of the norms of Koopman operators with respect to the activation functions, their evaluation is extremely challenging. making the effect of the activation function unclear. On the other hand, our bound can be applied to various types of activation functions, such as the hyperbolic tangent, sigmoid, and Leaky ReL.U, can evaluate the Koopman operator norms using Lemmas 2.3 2.5 and we can understand the effect of the activation function as discussed in Remark (54
5.3 GENERAL NEURAL NETWORK
If W is not injective, the Koopman operator Kw is unbounded. Thus, instead of the standard Koopman operators, we consider weighted Koopman operators. Let di E N and O = {W E Cdixd-1). For 1 O,...,L 1, let di dim(ker(Wy,z II...., Lă be an orthonormal basis of ker(W1+1), 0.+1--... 4d be an orthonormal ba- sis of ker(Wi+1)', & = 1 C:g. (W, W2 S.t N, 3i ELd Giui
Gi E [,,b for some at < b, such that c E [a,, bill and 2 1E Giai |
G E b;l.. Let WW ... 1201 (W226) In this case, to decompose the integral on ker(W(+1)
s & s Rd satisfying Mud (Xi) >
Xi-1
WiXi-1 WiXi-10 o1
and that on ker(W(+1)+ we set the orthonormal ba- sis along ker(Wi+1) and define Yi and Zr. Fig- ure 3 schematically shows the construction of X; Vi. Z, and Xt. Let Hi and Hi be the same space as in Subsection [.2] and Let m (WW) K K w be the weighted Koopman operator from H4 to H4-1 with respect to W and th.
Wi
Z
V1-1 4:
(X = 0
Mgg (X;-1)
(x,1) >
(x) > 0
Figure 3: Construction of 2; V%. 2,, and X,
where 24 is defined as (z) th (2,) = 1 for æ E X where x = 21 + 22 with #1 V and I2 E - and (x)) = 0 for & X In addition, let A be the same operator as in Subsection 5.2
Then, we have
f(W1.....
- (1) wwt.
(Wxz))
u (WV00L--1WW-002-2--00000
The factor 1i ... (W-- L (W(x)))) for x X for x E X/-1, we have f(W1, i......, (2)(=)
is an auxiliary factor. Since the (z) = 1
in the data space, exactly the same structure as that of neural networks. Thus, we can regard f as the original neural network vWW, (W,, (Wr.-or (W
Let D 0 and F. (Fo(O (FI........ 1......,...)
det W1 ker(W))4 11/2 ...., det Wz WE [ker 1-11)1111- 1/2 <
DJ. In the same way in Theorem 5.1 we obtain the following bound.
Theorem 5.5. Assume Pe E K. E Xg. Then, we have
E(c)|10111I L-1 |A Mker(W,) W.) (2)-1)
R[Fo.
sup
VSTI dot Wi (el) |1/2
det Wi 11/2
Remark 5.6. If the output of the lth layer has small values in the direction of ker())4))) then the factor ( (Di) is small. We expect that the magnitude of the noise smaller than that of the essential signals. This implies that if the weight W is learned so that ker (Wi-1) becomes the direction of noise, i.e, so that the noise is removed by WL4, the model generalizes well. Arora et al. (2018) insist that the noise stability property implies that the model generalizes well. The resuli of Theorem 5.5 does not contradict the results of Arora al. 20118)
5.4 CONVOLUTIONAL. NEURAL NETWORK
Let I = J1,1 X
Jr,d C Zd be a finite index set and θ = {0 E R x O + is invertible). Let 0: E Oi, Pi be the matrix representing the average pooling with pool size my, which is defined
... ...
8

---

Published as a conference paper at ICLR 2026
0.93
Test Accuracy
0.87
Test Accuracy
0.91
Out bound
10
0.85
0.89
Ours
5
0.84
Existing
With regularization Without regularization
Without regularization
0.2
Generalization error
0.4
0.6
0.83
400
Epochs
800
1200
1600
0.85
500
Epochs
1000
1500 2000
(a)
(b)
(c)
Figure 4: (a) Scatter plot of the generalization error versus our bound (for 3 independent runs). The
color is set to get dark as the epoch proceeds. (b) Test accuracy with the regularization based on our bound and that based on the existing bound (deep neural net with dense layers). (c) Test accuracy with and without the regularization based on our bound (LeNet).
as (P)),s 1/m, for j.j E I the jth element of the input is pooled in the ith element of the output, and or be the same as in Subsection 5.22 Let No S Rt, 01 P1-110--(01-1 +..-02 02 Piai (0, (o)) S & S R and (0 * 02 * Pioi (01 Xo)) S X S R satisfying Mgn (Ki) > 0 and Mgnit (X) > 0. Let di = dim(ker Q1...., , qå. be an orthonormal basis of (P)). Id.+1:-..,, 4d be an orthonormal basis of ker & EE c.li 4.E (as, b:]3
02 * (00 Xo)) S d S R Di = 12 4:ai c [a); and 2 (_i=d4 G9 | C E b.]. Let H4 (2), H4 (2),,
for some 0g < b; such that 0:(0 * +..002
Hi -2) and m O; + B(H, H4-1) be defined as ((0)((x) h(O * x), where is the convolution. Note that the convolution is linear operator whose eigenvalues are Fourier components 7' (Or) = Ejen, 0fe"(S: f) for m E 4t, where S; is the diagonal matrix whose diagonal is the scaling factor [1/(2m] Jı,l)..... 1/(2m D). Let A Ko, Kr.. where Ky.. and K₂, are weighted Koopman and Koopman operators from Hi to Hi and from Hi to Hi, respectively. Here, 4/z is defined as (x) v)1))) 1 for æ E Rr, where I1 + 22 with 0ı }) and x2 E 2,, and ()(())= = 0 for x & &g Then, we have
......... (x) - (0, # x)) .. 11-))) (0 (0r,-1 * 2- P1 * +x))))
vv(A, P ..... Pio (01 * xx)x ...)).
Let ((( [Ime, 7m (f) and F. (Fe .2. [3(01)) 1/2 18(0L) ((00))111 -1/2 D). Proposition 5.7. Assume Pe.z E K. E No. Then, we have
E(c)[[oll II |aa)) VSII (3) 1/2
ie Ri FI,..,,.... xs)
sup 18(0.)1 1/2CD
Remark 5.8. If ai is bounded, then we can set X1 independent of 01..... Or so that it covers the range of or Since Pt is a fixed operator, the factor ( is a constant this case.
NUMERICAL RESULTS
We numerically confirm the validity of the proposed bound. Experimental details are in Appendix B
Validity of the bound To show the relationship between the generalization error and the proposed
bound, we consider regression problem with synthetic data on Xo = [],, The target function t
is (2x) 112=-1111 We constructed a network f(x) v(W2o (W,,:(Wi>00 +xx bb))))) where Wi E R3x3
W2 E 16x3 b, E R3, 62 E 6 v(x) = wze ||=1|2 w3 E R, and a is the elementwise hyperbolic tangent. We created training dataset from randomly drawn samples from the uniform distribution
on 1-, 1|3 The training sample size is 1000. Our bound is proportional to the value r :=
|w3 sup [2) 2W,,
1/(1 (011==))/C =²))(1 -
(0(1111=//// /11) - x33) det W1| W/1/41 1/4 | det W; -1/4
since ||||| < |uay Jge e || and according to Lemma 2.4 We added 0.1r regularization term.
Figure 4 (a) illustrates the relationship between the generalization error and our bound throughout the learning process. We that the generalization bound gets small in proportion bound.

---

Published as a conference paper at ICLR 2026
Comparison with existing bounds To compare bound with existing bounds, considered the same classification task with MNIST as in Hashimoto et al. (2024). We constructed the same model f(x) = a. (W (W, + b1) b2) b3) b4) as Hashimoto et al. (2024) with dense layers. Based on the bound, we tried to make the factors || ||. 1/ det W,W,?'? and ||||| small, where [ō ), ................................ [(xa) is the elementwise smooth Leaky ReL.U proposed by BBiwwas et al. (2022) and 04 is the softmax. This setting is for meeting
((z) o. (W (Wa + b3) + ba), (z 1............................... (.a
the setting in HHashimoto et al. 2024 We set Xo = [0, 11784 2 = (1) || + [[b]]|=>[-1,,)
1 1021 2
W + b1, Xi ( 2 (W + b1), 82
(I]W2]I( W I1 + ||6 (l + ||/2// [-] 1--11,1111 1
and X₂ = (Xb) 2 ( + 6,) + b2). To make the factor || small, we applied
SUPMe(Aa))) 1(- (z)). Lemma 2.3 and set a regularization term ri SuPrE(X)) 1(@ (x=)) +
Here, (11) is the set the first elements of the vectors in X, addition, we set 72 = 1/(1
det Wi W 1/1 + 1/(1 + det W.W. Regarding |||||, we set r3 ||Will + [W2]] since we have ||ul| [xx [dx S p((222 S ((E)). We added the regularization term 0.01(r1 + 72 ra) to the loss function. The training sample size is S 1000. We compared the regularization based on our bound with that based on the bound proposed by Hashimoto et al. (2024). The result is shown in Figure (b). Note that since the training sample size S is small, obtaining high test accuracy is challenging. We can see that with the regularization based on our bound, we obtain a better performance than that based on the existing bound.
Validity for existing CNN models (LeNet) To show that our bound is valid for practical models, we applied the regularization based on our bound LeNet on MNIST (Lecun et af.) 1998). We set the activation function a of each layer as the elementwise hyperbolic tangent function and the final nonlinear transformation v as the softmax. We used the same training and test datasets as the previous experiment. In addition, we set Xn = [0, 1 784 X = Wi + [[b]]]=)[-11, 1] 1J1021 2 Wio o (W 2o + b1) +...) )) + bt. X = (xt) 2 (Wior(-.. (W ( + bb)) +...) + br). Here. W, is the matrix that represents the /th convolution layer. We note that the bound by Haahimoto eet al. (2024) is not valid for the models with hyperbolic tangent and softmax functions. To make
the factor |[Ail] small, we applied Lemmas 2.3 and 2.4 and tried to make inf (1 - x²) large.
Thus, we set a regularization term 71 Ei= supze 1/(1 + - x²). 1-x2). Regarding the factor 1/(0.01+ (Wi)), to make siii (Wi) large, where 5min (Wi) is the smallest singular value of Wi since the determinant is described as the product of the singular values. For ||||],, we set = ||z/| in the same way as in the previous experiment according to the definition of Xr, We added the regularization term 0.1 + r2 + ra) to the loss function and compared with the case without regularization. The result is shown in Figure (c). We can see that with the regularization, the model performs better than in the case without the regularization, which shows the validity of our bound for LeNet.
det W2 [ker(w,)= - 1/2 we set = [(0.01r+
7 CONCLUSION AND LIMITATION
In this paper, we derived a new Koopman- based Rademacher complexity bound. Analogous the existing Koopman-bor bbsed bounds, our bound describes that neural networks with high-rank weight matrices can generalize well. Existing Koopman-based bounds rely on the smoothness of the function space and the unboundedness of the data space, which makes the result valid for limited neural network models with smooth and unbounded activation functions. We resolved this issue by introducing an algebraic representation of neural network models and constructing an RKHS associated with a kernel defined with this representation. Our bound is valid for a wide range of models, such as those with the hyperbolic tangent, sigmoid, and Leaky ReLU activation functions. Our framework is the first step to filling the gap between the Koopman-baa based analysis of generalization bounds and practical situations.
Although our bound can be applied to models more realistic than the existing Koopman-bast based bounds, it is not valid for activation functions whose derivative is zero in some domain, such the exact ReLU. Introducing a variant of the Koopman operator such as the weighted Koopman operator may help us deal with this situation, but more detailed investigation is left for future work.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENT
This work was supported by JSPS KAKENHI 24K21316, 25H01453, JP24K21316, JST BOOST JPMJBY24E2, and JST CREST JPMICR2015, JPMICR2515, JPMJCR24Q6, JPMJCR24Q1.
REFERENCES
Martin Anthony and Peter L. Bartlett. Neural network learning: Theoretical foundations. Cambridge University Press, 2009.
Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via compression approach. In Proceedings the 35th International Conference Machine Learning (ICML), 2018.
Peter L Bartlett, Dylan Foster, and Matus J Telgarsky. Spectrally-nommaliin normalized margin bounds for neural networks. In Proceedings of the 31st Conference on Neural Information Processing Systems (NIPS), 2017.
Koushik Biswas, Sandeep Kumar, Shilpak Banerjee, and Ashish Kumar Pandey. Smooth maximum unit: Smooth activation function for deep networks using smoothing maximum technique. In Proceedings of 2022 IEEE/CVF Conference on Computer Vision and Pultern Recognition (CVPR), 2022.
Kenneth R. Davidson. C -Agebbras by Example. American Mathematical Society, 1996.
Gerald B. Folland. A Course in Abstract Harmonic Analysis. CRC Press, 1995.
Micah Goldblum, Jonas Geiping, Avi Schwarzschild, Michael Moeller, and Tom Goldstein. Truth or backpropaganda? an empirical investigation of deep learning theory. Proceedings of the 8th International Conference Learning Representations (ICLR), 2020.
Noah Golowich, Alexander Rakhlin, and Ohad Shamir. Size-independent sample complexity of neural networks. In Proceedings of the 2018 Conference On Learning Theory (COLT), 2018.
Nick Harvey, Christopher Liaw, and Abbas Mchrabian. Nearly-tight VC-dimension bounds for piecewise linear neural networks. In Proceedings the 2017 Conference Learning Theory (COLT), pp. 1064-1068, 2017.
Yuka Hashimoto, Sho Sonoda, Isao Ishikawa, Atsushi Nitanda, and Taiji Suzuki. Koopman based generalization bound: New aspect for full-rank weights. In The 12th International Conference on Learning Representations (ICLR), 2024.
Haotian Ju, Dongyue Li, and Hongyang R Zhang. Robust fine-tuning of deep neural networks with Hessian-based generalization guarantees. In Proceedings of the 39th International Conference on Machine Learning (ICML), 2022.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2015.
Yann Lecun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278- 2222222224 1998.
Shuai Li, Kui Jia, Yuxin Wen, Tongliang Liu, and Dacheng Tao. Orthogonal deep neural networks.
IEEE Transactions Pattern Analysis and Machine Intelligence, 43(04)):352-1338. 2021.
Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar. Foundations of Machine Learning. MIT press, 2nd edition, 2018
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Proceedings of the 2015 Conference on Learning Theory (COLT), 2015.
Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A PaC-bayesian approach to norttallly-oommlizedd margin bounds for neural networks. Proceedings of the 6th Interna- fional Conference Learning Representations (ICLR), 2018.

---

Published as a conference paper at ICLR 2026
Sho Sonoda, Yuka Hashimoto, Isao Ishikawa, and Masahiro Ikeda. Deep ridgelet transform and unified universality theorem for deep and shallow group- machines. In The 42nd International Conference Machine Learning (ICML), 2025.
Taiji Suzuki, Hiroshi Abe, and Tomoaki Nishimura. Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network. In Proceedings of the 8th International Conference Learning Representations (ICLR), 2020.
Sundaram Thangavelu. Harmonic Analysis the Heisenberg Group. Birkhäuser Boston, 1998.
Colin Wei and Tengyu Ma. Data-depentee data-dependent sample complexity of deep neural networks via Lipschitz augmentation. In Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS), 2019.
Colin Wei and Tengyu Ma. Improved sample complexities for deep neural networks and robust classification via an all-layer margin. In Proceedings of the 8th International Conference on Learning Representations (ICLR), 2020.
Weinan E. Chao Ma, and Lei Wu. The Barron space and the flow-induced function spaces for neural network models. Constructive Approximation, 55:369-406, 2022.
APPENDIX
A PROOFS
We show the proofs of the theorems, propositions, and lemmas in the main text.
Lemma 2.3 Assume a E + X is bijective, a is differentiable, and the Jacobian of -1 bounded in X. Then, huve Kall S sup (xx) where Jo 11 is the Jucobian of In particular, the Koopman operator K, bounded.
Proof. For h E (x))),, we have
| [b(o(z))) =s Jo(z)(()o-'(x)))
|K,h||
S sup IJo (w)) { [Mxx) sup 2EX
JJ [x)]
=EX
Lemma 2.5 Let X X Rd. Let be the elementwise Leaky ReLU defined as 5(x) az for
x and o xx) x for where > Then, have ||Kll S max(1, 1/ad)
Proof. For h E (2), we have
|| =
1b(o()))
Rd
[
[M(diag[1, ...eesl................. a))) +.. + [xxx)) Pdr
(-xc,o
-(000)x(-x000--
(0,20)
$ max 11,1/
[hxx) = max {1, 1/a4) ||1|2.
Rd
Proposition 3.4
The map is isometrically isomorphic.
Proposition 3.4 is derived using the following lemmas.
12

---

Published as a conference paper at ICLR 2026
Lemma A.1. The map is injective.
Proof. Assume (h) = 0. Then, for any g E G. (Ø(g), = 0. Thus, for any E N, gi...., En. and C1,..., E C, we have (E== (g)())) 3 h) 0, which means for any E Ko. (h. = 0. Thus, we obtain 0.
Lemma A.2. The map preserves the norm and is surjective.
Proof. By definition, & is a linear map that maps o(g) E Kop to @(g) E Rx,0- Thus, we have (Ko)) = R2,0-
For h Ko. there exist N. gh.... gn G and cp... E C such that h E Go(g)). We have
n Σ cg)(g
Σ i,=1
(enk(gn)))
Σ (o)(o()). 1111.2.
Rx
Thus, preserves the norm, and in particular, it is bounded.
For any r R2, there exists a sequence T1 T2.... E RkD0 such that lom Ta r. Since i(Ko)) = Rx there exists h; E Ko such that ((ha) т, for 1,2, Thus, we have r co Ti so c(h;) = ol h2) (h)..
Lemma 3.5
Assume p is irreducible. Let A= {E c.p(g)) n N.N, G.q E C). Then, A dense in B(H (H) with respect to the strong operator topology.
Proof. By the Schur's 's lemma (Lemma 2.7 the commutant of A SOT the closure of A with respect the strong operator topology, CT. Thus, the double commutant of A SOT B(H). By the von Neumann double commutant theorem (Lemma 2.8), the double commutant of A SoT SOT is A itself.
Therefore, we have ASOT B(H).
Lemma 3.6
Assume p is irreducible and , are invertible. Then, K. = Ko H.
Proof. Let E H. Then, there exists B E B(H) such that = Br. Let E > By Lemma 3.5 there exist ny E N. (..,,..,II,,,, E G, and c1,1,..., E C such that ||Âg ا| I E, where AL EBz=1 CL,a P(9Laa)) In addition, there exist n N. CL-I.,,,. E C such that (AAL-((LL-1LL))) We continue this process, and for I-2, 2, ........ 2, we obtain n, E N, Il.1........ I1 E G, and c1,, Ctm. such that (A A Ear=1 a,,,
91- IL -m E G. and
A 2 Aلvلl e, where Åz- Enh-1 11-1ac--P(9L-1..0
A - A [A|A
A < s, where
C. (() Finally, get nı E N, 91.1, ... 91,n; G. and C1 ... Ci,. C A,v) A[A
such that [A
A,, A - A < e, where A Σm C1.01 p Let A,A ... - AL. Then, we have
|CC' A|ll
CC BA A,
A || + |BA, A A BAyds ... A ALvil
+... |BA - -A BA BÂ
BA
< |Br +... |BA + |B
13

---

Published as a conference paper at ICLR 2026
Theorem 4.1 Let Fe the function class [Fe(91..... ................................ ...))L 91....,91 9L G}. Assume Pe,x K. for x E No. Then, the Rademacher complexity of the function class Fe is bounded as
R(F 21. |A
||A
[[())
VS
Proof. Since Fe(:,z) ((Pe.)) E RR, by the reproducing property, we have
s Σ
S
sup gEGL ==1
Folg, 1g
sup gEGL
((g),
Σ
==1
Ru
8
< sup gEGL
[k( E
Fel: 2s)e.
S
RR
1/2 Do)s (Fd )"
1
[
s
sup REGL
9,1=1
Fe,, Fe(s 1) 1/2
S sup
[p( ... ......A..... A
E
S
gEGL
e,1=1
1/2
S A1ll || |o|| Σ
where the third equality is by Lemma final inequality is derived since p(91)
A.2 the fourth inequality is by the Jensen's inequality, and the p(LL) are unitary.
Since Fo(:,20 2) ((ee.)), apply Lemma A.2 again and obtain
1/2
1/2
5 ||Alll ...
Σ |.
||AIll -.. | Σ Ilpe.e. 11
S
A1ll ... |
Vs
where the last equality is derived since Pe,r is the regularizer that satisfies Xg.
||pe, 1 for any
Theorem 4.4
Let NN (NN. :
g GL, /eet W, -1/2 ..... det W1 -1/2 < D}. The
Rademacher complexity bound of N
F(c) ||l
2I x8)
sup d W/ 11/ 1=1
II
det Wil
VS
Proof. By Theorem 4.1 we have
R(NN T1...... .......))...............))..)... = E
sup
NN. 23)0
LEGL det WII-1/2DD a=1
s
L gEGL, det W. W 1/2 s=1
sup
1/2 Folg. x.) det W W 1/2 ... det Wz -1/2 0.
|
E(c) ... || |||||
<
sup
det W 11/2. 1/2
det Wz 112 11/2.
VS
det wi 1
14

---

Published as a conference paper at ICLR 2026
Let Fe (F.(01,...,,01 det Wi 1 ........ det W - 1/4 < D). Assume Pe E K for a E No. Let fi = vo o WL o JL-1
Theorem 5.1
W441 oy. Let a(h) = [b(z)]d (x) for h E Hy. Then, we have
Uwix
sup E[C)|II| [All((() det WW VSIIL det W////
31,
Proof. In the same way as Theorem 4.1 have the same inequality (5) but p(g) is replaced by (0)) Kw,- For H4, we have
KKw 1kmhh1 =J
[b(Wa))| du =
1n(x)1
det W7 ((RRAW)
WiXi-
Swi [1(x)) (d(z))d
ax (<h)) 2
[1(x)) (pp4 det W Wi|
(6)
det W W/
Sx [h(x) (eg (x)
Applying the inequality
to the inequality for this case, we obtain the result.
Theorem 5.5 Let (Fal0..... .),)) det W [ker(W,) 1/2 -/.,,..... dot W, (er( 1-12 1/2
D). Assume Pa,x K for x E No. Then, we have
E(c) 1=1 (At)l IT Mker (Wi)()1-1)
sup
det. hor 1 1/25D
vS 1S det W/ (er((,)) 11/2
Proof. For l E Hi, we have
| [Ke,,, h112
(=
[h(Waz) Pdx (e)(
Xx-1
2 Vi-1
1
J 1(x))1 det Welker(W.) duR(w;) (ker(W,) (J/-1)
Swix (
s det Wi ( a SR.
(x) Jxx 1 (egggii (x) (ker(Wi) (2%-1)
[ (wer()))))) (Dt-1)
det Wi Wilker(W:)
Applying the inequality
7 to the inequality
5 for this case, we obtain the result.
Let F. = (F.(...... Pe,x K. for No. Then, have
Proposition
5.7
O2. [3(0,) -3(2 1/2 ,....... 18(0 ",,00)),.......(()))111) s D}. Assume
E(c) 1=1 |A (5i)
II, ((.,,.).....s))
sup 1//2
VS VSII (B))))) 1/2
Proof. Since the convolution is a linear operator whose eigenvalues are Fourier components, we have [m)( < (8)) 1/2, In the same way as the proof of Theorem 5.5 we have
Mker (Di)
KK S det Pi 11/2 ( ( (31),
which proves the result.
15

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of our bound to existing bounds.
Authors
Rate
Type
2" I Willaa
Neyshabur et al.
2015
VS
L maxi de I [Will
Will
1/2
Neyshabur al.
2018
v/
1=11
Norm-based
Golowich et al.
2018
(I [1
min
S
3
3/2
||Wall
Σ IW: -A7
12..1
Bartlett et al.
(017
||
(E Mi min (1
(W:- 212.
Wei & Ma
2020
vS BellWa-Allee
Σ W,
Ju et al.
2022)
v8
Li et al.
(021
X
II [Will
1| + + 7xx +
Ex
1/2
Arora al.
(018
F+
maxx [[
Ei=1 p.
AVS
Compression
Suzuki et
(020))
((E Pr(di-1 -1 + di))
1/2
VS
Hashimoto et al.
2024)
||||| |
Il==1
G. Kel 11 [| 11"!
vS
det :
W1)
Koopman-based
Ours
|eller /9
I
Gi Key
W.)77
B EXPERIMENTAL DETAILS
All the experiments were executed with Python 3.10 and TensorFlow 2.15.
B.1 VALIDITY OF BOUNDS
We set Wi, W2, and Wy as learnable parameters. We set the loss function as the mean squared error and the optimizer the SGD with learning 0.001. The learnable parameters are initialized with the orthogonal initialization.
B.2 COMPARISON TO EXISTING BOUNDS
We constructed network f(x) oa ( (W + b2) + 63) + b4) with dense layers, where W1 E IR 1024x W2 E 222048x1021 W3 E 20204x2045 W4 E IR 20482048 by E R 1024 by E 2048 by E 2048 b, 1000 is the elementwise smooth Leaky ReLU (Biswas et al. 2022) with a = 0.1 and μ 0.5, and 04 is the softmax. The learnable parameters Wi...., Wa are initialized by the orthogonal initialization for 1, and by samples from the truncated normal distribution for = 3, 4, and we used the Adam optimizer (Kingma & Ba 2015) for the optimizer with learning rate of 0.001. We the loss function the categorical cross-entropy loss. The result in Figure 4 (b) is the averaged value + the standard deviation in 3 independent runs.
B.3 VALIDITY FOR EXISTING CNN MODELS (LENET)
We constructed 5-layered LeNet with the hyperbolic tangent activation functions and the averaged pooling layers. We set the optimizer as the Adam optimizer with learning rate of 0.001. The result in Figure (c) is the averaged value + the standard deviation in 3 independent runs.
C COMPARISON OF THE KOOPMAN -OOMD BOUNDS TO EXISTING BOUNDS
We show the summary the existing bounds and the proposed bound in Table I Here, K₁ and 6, determined by the Jacobian and Hessian of the network with respect the jth layer and Wi,
16

---

Published as a conference paper at ICLR 2026
Table 2: Notation table
G Locally compact group for parameters
O Set of parameters for the /th layer Number of layers
L
d Width of the Ith layer
H Hilbert space for models
Unitary representation of on H
K.
Koopman operator with respect to function
W Weight matrix for the ith layer Activation function for the lth layer
A
Linear operator corresponding the activation function for the /th layer Original deep model
f
Fe
Regularized model with parameter
Fo
Function class for models
k
Positive definite kernel defined as k((g).....9I) .2.), (#1... ..))
(f(g......9L) f( (1))
Feature map defined ó(g) k(.g)
Feature map representing models defined as e(g) = [(n,,... where g (01,....11)
K. Hilbert space defined the closure of (Ei Gg(gi) N.g. E G
Isomorphism that maps ø(g) ((g)
respectively. addition, Fz and di are the rank and dimension the jth weight matrices for the compressed network and || p.g is the matrix (p, g) norm. We note that although the form of the existing Koopman-based bound and the proposed bound is similar, our bound is applicable to a wider range of deep models, and the factors G; and Ka, || are more easily evaluated.
D NOTATION TABLE
We provide notation table 2 that summarizes important notation in the main text.
17


Reviews:
[{"review_id": "oroF7d2LUA", "reviewer": "Reviewer_JANi", "summary": "The paper introduces a new bound on the Rademacher complexity of deep neural networks, with the goal of better accounting for the observed good generalization of neural networks with high-rank weight matrices.  The paper improves upon earlier work by Hashimoto et al. 2024 that introduced an approach based on Koopman operators and RKHSs to represent and reason about neural networks.  The main improvement is the analysis of models in a different RKHS than in prior work, thus accounting for a larger and more practical model class, for example including non-smooth activation functions.  A small-scale experiment on MNIST digit classification shows that a regularizer derived from the bound improves on one derived from the earlier bound.", "questions": "My remaining questions are mainly about low-level details:\n\n- Regarding these sentences:  \"... phenomena in which models with high-rank weight matrices generalize well have been empirically observed (Goldblum et al., 2020).  Since the norm-based and compression-based bounds focus only on the low-rank and nearly low-rank cases, they cannot describe these phenomena.\"  The second sentence need not be true if the high-rank matrices in Goldblum et al. 2020 are in fact nearly low-rank.  I guess that is not the case, but could you please confirm?\n\n- There is some imprecise language that would be good to clean up.  Examples:\n  - \"It describes how the model can fit unseen data\" -- not clear what \"It\" refers to.\n  - \"This bound is described by the ratio of the norm to the determinant of the weight matrix\" -- since there are multiple layers and multiple weight matrices, this can't be precise.\n  - \"the generalization bound is described by the Rademacher complexity\" -- unclear what it means for the Rademacher complexity to \"describe\" a bound.\n  - \"Our framework fills the gap between the Koopman-based analysis of generalization bounds and practical situations\" -- I think this is a bit stronger than intended (i.e. the paper does not fully fill the gap).\n\n- Typo:  \"The Hilbert space H to which the modes belong...\" --> \"... the models belong\"\n\n- In Fig. 1(a), I have a hard time seeing the color gradient.  Perhaps a different gradient would be visually clearer.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Expanding the reach of existing complexity bounds to a more realistic model class is a very important contribution.\n\n- The style of Koopman-based bounds appears to differ from much of the other existing work studying generalization in neural networks (besides the cited Hashimoto et al. 2024), and so may be complementary to other results.\n\n- The inclusion of empirical evidence, while limited, is helpful.", "weaknesses": "- The main weakness is with the presentation.  I fear that in its current form, the paper will be accessible only to an very narrow segment of the ICLR audience.  While I am admittedly not an expert in the recent theoretical ML literature, I believe the paper should be able to make its results and implications much clearer to most of the ICLR community.  By the end of the paper, it is not clear to me in what way exactly the paper improves over Hashimoto et al. 2024, and how this improvement was achieved.  It is possible that parts of my summary above are incorrect as a result (please let me know if so!).  I would have expected a clear statement of the Hashimoto et al. 2024 bound and the new bound, both mathematically and intuitively, and ideally also other existing bounds for comparison.  I also would have expected a more intuitive \"walk-through\" of the key steps.  There are two paragraphs devoted to introducing RKHSs -- a broadly familiar concept, I believe -- while there are many other more obscure concepts and notation that are relied on but not explained.  While they cannot all be defined for the novice reader, some intuitive explanations of the ideas would go a long way.\n\n- A second possible and more minor weakness (which may be not a weakness at all, and rather due to my own misunderstanding -- again, please let me know if so!) is with the empirical results.  I am not able to understand why the bound was not compared directly to the prior bound (e.g. in Fig. 1(a)) but only as a form of regularization (in Fig. 1(b, c)).  I also wonder why there are no empirical results with more practical, larger datasets and models (say, ImageNet).  It is reasonable for a theoretical paper to limit itself to small experimental settings, if needed.  But it should be stated what are the limitations that prevent scaling up the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bTwbgrfiPl", "reviewer": "Reviewer_y36o", "summary": "The paper develops a new Rademacher complexity bound for deep neural networks by combining Koopman operator theory, group representations, and RKHS methods. Unlike traditional norm- or rank-based bounds, the proposed framework explains why high-rank networks can still generalize well. The authors construct a kernel on the parameter space and prove an isometric correspondence between this Koopman-RKHS and the function space of the network. Overall, the work provides a mathematically elegant perspective on neural network generalization, though some aspects of proofs raise some concerns about their validity.", "questions": "Example 3.3 introduces $p_{c,x}$ which is  Gaussian centered at $x$ whose variance goes to zero, as $c$ goes to infinity. \n\nThis function satisfies \n$||p_{c,x}||_{1}=1$ \n\nbut not \n\n$$||p_{c,x}||_{2}=1$$ \n\nas claimed in paper. \nThe latter claim, i.e. \n$$||p_{c,x}||_{2}=1$$ \nis then used in the proof of Theorem 4.3 where the dependence on $c$ vanishes (which I doubt is correct).", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "The paper offers a novel and mathematically rigorous framework that unifies Koopman operator theory, RKHS analysis, and group representations for studying neural network generalization. It extends existing Koopman-based approaches to handle nonsmooth activations and bounded domains, broadening their practical relevance.", "weaknesses": "The main weakness is a couple of issues that I found in proofs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TIPAnDRBzb", "reviewer": "Reviewer_DC2X", "summary": "This paper proposes a new Koopman-operator-based Rademacher complexity bound to explain why neural networks with high-rank weight matrices can generalize well—an observation not captured by existing norm-based or compression-based generalization theories.\n\nThe key idea is to construct a reproducing kernel Hilbert space (RKHS) on the parameter space of neural networks using group representations and Koopman operators, enabling a generalization bound applicable to realistic models with bounded input domains and nonsmooth activations (e.g., tanh, sigmoid, Leaky ReLU).", "questions": "1) Can your framework extend to ReLU networks via non-smooth or distributional Koopman operators?\n\n2) How sensitive is the determinant-based bound to weight scaling (e.g., if |det (W)| >> 1 due to scaling, not diversity)?\n\n3) In practical networks, rank deficiency often correlates with overfitting. Does your theory predict the same phenomenon in the inverse direction?\n\n4) How does the bound behave for randomly initialized networks versus trained ones?\n\n5) Could the RKHS kernel you define (Eq. 3.2) be computed empirically for small models to verify the bound numerically?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Addresses a known theoretical gap:\nMost generalization bounds (norm-, compression-, or PAC-Bayes–based) explain why low-rank networks generalize, but fail for empirically observed high-rank networks. This work provides a principled explanation via Koopman-based operator theory.\n\nMathematical generality:\nThe approach unifies algebraic, operator-theoretic, and kernel methods, supporting both bounded and nonsmooth activations—something prior Koopman-based bounds (Hashimoto et al., 2024) could not handle.\n\nElegant use of RKHS:\nBy defining an RKHS on the parameter space, the authors convert the non-linear deep network mapping into a linear setting where standard Rademacher complexity tools apply — a conceptually clean and general construction.\n\nPractical implications validated by experiments:\nThe three plots in Figure 1 show consistent improvement in generalization across synthetic regression, MNIST dense nets, and LeNet CNNs when regularization terms derived from the bound are added.\n\nConnections to high-level mathematical structures:\nThe use of group representations (affine, Heisenberg) and operator algebras (Schur’s Lemma, von Neumann double commutant theorem) to describe neural networks is novel and could inspire deeper theoretical analyses of neural architectures.", "weaknesses": "Dependence on bounded activation assumptions:\nThe proofs assume Koopman operator boundedness (Assumption 2.2), which fails for ReLU (derivative = 0 on half-line).\nThe authors acknowledge this limitation (Sec. 7) and suggest exploring weighted Koopman operators, but current results exclude the most popular activations.\n\n\nInterpretability of constants:\nThe bounds depend on abstract constants that are not easy to estimate in practice. Consequently, the bound is more qualitative than quantitative.\n\nComparative evaluation:\nWhile it compares against the 2024 Koopman bound, other modern generalization frameworks (e.g., PAC-Bayes, spectral norms, neural tangent kernel analyses) are not benchmarked.\n\nNo discussion of sample complexity scaling:\nThe 1/sqrt(S) dependence mirrors classical Rademacher bounds, but no new scaling exponents are derived.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gxgKGFhdG5", "reviewer": "Reviewer_X6tr", "summary": "The paper presents a Rademacher complexity bound for deep NNs derived using Koopman operators, group representations and RKHSs. Distinct from other bounds, the bound includes the determinant of the weight matrices in its denominator, and is therefore tighter than comparable bounds in the case of high-rank weight-matrices.", "questions": "See weaknesses section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper builds on previous work on Rademacher complexity bounds via Koopman operators, removing restrictions which limited the applicability of the previously derived bounds.\n2. The resulting bounds scale much better than comparable bounds when the weight matrices are high-rank.\n3. The experimental results appear to back up the claims make in the paper.", "weaknesses": "My main problem with this paper is readability. While I understand that the material is intrinsically difficult, it would nevertheless be helpful if more time was spent systematically introducing the various notations/definitions etc that are used. In particular, I found section 5 to be quite heavy going, and section 5.3 in particular was extremely difficult to parse. Perhaps the authors could include a table (even if it is located in the appendices?) presenting the various notations/definitions (and their inter-connections) in a more systematic way so that the reader isn't left constantly scanning back and forward through paragraphs of dense definitions/notations interleaved with prose to remind themselves what something means when it is eventually used?\n\nMore generally it seems to me that this is a somewhat incremental improvement on the previous work in Hashimoto et al, but I am open to argument in this regard as this is open to interpretation.\n\nSpecific points and questions:\n\n- Lemma 2.5: it seems to me that this result effectively rules out applying the method to ReLU networks as it blows up in the relevant limit. Can you see any way around this?\n- Example 3.3: is the limit in line 205(ish) due to $p_{c,x}(y)$ approaching the Dirac-delta as $c \\to \\infty$?\n- Theorem 4.3: is the supremum here actually bounded? The determinant cannot be zero for the invertible case, but it can be arbitrarily small, so the supremum can presumably be arbitrarily large. Also I assume $A_l$ here is the Koopman operator?\n- Theorem 5.1: it is confusing to use $\\alpha (f_l)$ here before introducing it (in remark 5.3). Also what role does this factor play? Could it just be replaced by $1$ for clarity in the theorem?\n- Section 5.3: the first paragraph here is far to mathematically dense and needs to be expanded for clarity. Also it strikes me that many of these are essentially geometric constructs, so perhaps a diagram could help?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)