Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Group-Equivariant Machines
Authors: Sho Sonoda
Yuka Hashimoto
Isao Ishikawa
Masahiro Ikeda
Venue: ICML 2025 poster (2025)
Keywords: ridgelet transform
group representation
harmonic analysis
Primary area: theory->deep_learning

Abstract: We present a constructive universal approximation theorem for learning machines equipped with joint-group-equivariant feature maps, called the joint-equivariant machines, based on the group representation theory. ``Constructive'' here indicates that the distribution of parameters is given in a closed-form expression known as the ridgelet transform. Joint-group-equivariance encompasses a broad class of feature maps that generalize classical group-equivariance. Particularly, fully-connected networks are *not* group-equivariant *but* are joint-group-equivariant. Our main theorem also unifies the universal approximation theorems for both shallow and deep networks. Until this study, the universality of deep networks has been shown in a different manner from the universality of shallow networks, but our results discuss them on common ground. Now we can understand the approximation schemes of various learning machines in a unified manner. As applications, we show the constructive universal approximation properties of four examples: depth-$n$ joint-equivariant machine, depth-$n$ fully-connected network, depth-$n$ group-convolutional network, and a new depth-$2$ network with quadratic forms whose universality has not been known.

Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Group-Rquivariant Machines
Sho Sonoda 12
Yuka Hashimoto
31
Isao Ishikawa 41
Masahiro Ikeda" 51
Abstract
The proof of a universality theorem contains hints for un- derstanding the internal data processing mechanisms inside neural networks. For example, the first universality the- orems for depth-2 neural networks were shown in 1989 with four different proofs by Cybenko (1989), Hornik et al. (1989), Funahashi (1989), and Carroll & Dickinson (1989), Among them. Cybenko and Hornik et al. presented exis- tential proofs by using Mahn-Banach and Stone-Weiersie respectively, meaning that it not clear how assign the parameters. On the other hand, Funahashi and Carroll-and- Dickinson presented constructive proofss by reducing net- works to the Fourier transform and Radon transform respec- tively, meaning that it is clear how to assign the parameters. The latter constructive methods were refined as the so-called integral representation by Barron (1993) and further culmi- nated as the ridgelet transform, the main objective of this study, discovered by Murata (1996) and Candès (1998).
We present a constructive universal approxima- tion theorem for learning machines equipped with honn-ggruup-equvvvriantt feature maps, called the joint-qquivariant machines, based on the group representation theory. "Constructive" here indi- cates that the distribution of parameters is given in a closed-form expression known as the ridgelet transform. caant-roou--qquvariinne encompasses a broad class of feature maps that generalize classical group-equivarianee. Particularly, fully- connected networks are not eouu-arauvaai but are entt-grouu--quivariann Our main theorem also unifies the universal approximation theorems for both shallow and deep networks. Until this study, the universality of deep networks has been shown in a different manner from the universality of shallow networks, but our results discuss them common ground. Now we can understand the approximation schemes of various learning machines in unified manner. As applications, we show the constructive universal approxima- tion properties of four examples: depth-n joint- equivariant machine, depth-n fully-connected net- work, depth-n group-convolutional network, and new depth-2 network with quadratic forms whose universality has not been known.
To show the universality in constructive manner, we formu- late the problem as functional equation: Let M[y] denote a certain learning machine (such as a deep network) with parameter y, and let F denote a class of functions to be expressed by the learning machine, Given a function f E F. find an unknown parameter y so that the machine M[7] rep- resents function f. i.e.
Mh1 f.
which we call a learning equation. This equation is a stronger formulation of learning than an ordinary formu- lation such as minimizing empirical risk Σ [M))(2;)) (xx))22 with respect to 7, as the latter is weak form (or variational form) of this equation. Therefore, characterizing the solution space of this equation leads to understanding the parameters obtained by risk minimization. Following previous studies (Murata, 1996; Candès, 1998; Sonoda et al., 2021a;b; 2022a;b), we call a solution operator R satisfying M[ = f a ridgelet transform. Once such an R is found in closed-form manner, we can present constructive proof of universality because the reconstruction formula M M[R[S]] = f implies for any ʃ E F there exists a machine that implements f.
1. Introduction
One of the technical barriers in deep learning theory is that the relationship between parameters and functions is black box. For this reason, the majority of authors build their theories on extremely simplified mathematical models. Such theories can explain the complex phenomena in deep learning only at highly abstract level.
RIKEN AIP *CyberAgent, Inc. 'NIT Corporation "Kyoto University 'The University of Osaka. Correspondence to: Sho Sonoda <sho. connod@rriknnj....
For 2 neural networks, the equation has been solved with several closed-form ridgelet transforms by using either Fourier expression method (Sonoda et al., 2024b), or group
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Croup-Equivarinn Machines
representation method (Sonoda al., 2024a). For example, the closed-form ridgelet transforms have been obtained for 2epth-2 fully-connected networks (Sonoda et al., 202 lb), 2 2 fully-connn cully-connected networks on manifolds (Sonoda et al., 2022b), depth-2 group convolution networks (Sonoda et al., 2022a), and 2epth-2 fully-connected networks on fi- nite fields (Yamasaki et al., 2023). Furthermore, Sonoda et al. (2021a) have revealed that the distribution of param- eters inside depth-2 fully-connecte empirical risk minimization asymptotically converges to the ridgelet transform. In other words, the ridgelet transform can also explain the solutions obtained by risk minimization.
(Theorem 3.10), which shows the universal approxima- tion theorem for wide range of learning machines in a constructive and unified manner.
As applications, we show the universal approximation properties of four examples: depth-n joint-equivariant machine (Section 4), depth-n fully-connected net- work (in Section 5), depth-n group-convolutional net- work (in Section 6), and a new depth-2 network with quadratic forms whose universality has not been known (in Section 7).
networks obtained by
On the other hand, for depth-n neural networks, the equa- tion is far from solved, and it is common to either con- sider infinitel-ddeep mathematical models such as Neural ODEs (Sonoda & Murata, 2017b; E, 2017; Li & Hao, 2018; Haber& & Ruthotto, 2017; Chen et al., 201 (), or handcraft networks that approximate another universal approxima- tors such as piecewise polynomial functions and indica- tor functions. For example, construction methods such as the Telgarsky sawtooth function (tent map, or the Yarot- sky scheme) and bit extraction techniques (Cohen et al., 2016; Telgarsky, 2016; Yarotsky, 2017; 2018; Yarotsky & Zhevnerchuk, 2020; Daubechies et al., 2022; Cohen al., 2022; Siegel, 2023; Petrova & Wojtaszezyk, 2023; Grohs et al., 2023) have been developed (not only to investigate the expressivity but also) to demonstrate the depth separation, super-convergence, and minmax optimality of deep ReLU networks. Various feature maps have also been handcrafted in the contexts of geometric deep learning (Bronstein al., 2021) and deep narrow networks (Lu et al., 2017; Hanin & Sellke, 2017; Lin & Jegelka, 2018; Kidger & Lyons, 2020; Park et al., 2021; Li et al., 2023; Cai, 2023; Kim et al., 2024). However, for the purpose of understanding the parameters obtained by risk minimization, these results are less satis- factory because there is no guarantee that these handerafted solutions are obtained by risk minimization in manner presented by Sonoda et al. (2021a).
Until this study, the universality of deep networks has been shown in different manner from the universality of shallow networks, but our results discuss them common ground. Now we can understand the approximation schemes of vari- ous learning machines in unified manner.
2. Preliminaries
We quickly overview the original integral representation and the ridgelet transform, mathematical model of depth-2 fully-connected network and its right inverse. Then, we list few facts in the group representation theory. In particular, Schur's 's lemma play key roles in the proof of the main results.
Notation. For any topological space X, Cx(X) denotes the Banach space of all compactly supported continuous functions on X. For any measure space X, L L*(X) denotes the Banach space of all p-integrable functions on X. ((Rd) and S' ('(R)) denote the classes of rapidly decreasing func- tions (or Schwartz test functions) and tempered distributions on Rd, respectively.
2.1. Integral Representation and Ridgelet Transform for Depth-2 Fully-Connected Network
Definition 2.1. For any measurable functions a : R + C and : RD x R + put
In order to investigate the relation between parameters and functions, we need to write down general solution (i.c., the ridgelet transform) rather than handcrafting a particular so- lution. However, conventional ridgelet transforms have been limited to 2epth-2 networks. In other words, existing meth- ods cannot construct solutions for networks that repeatedly compose nonlinear activation functions more than twice- such as af (4 (Aiæ b,) - b2). In this study, inspired by the group- troor-the approach of Sonoda et al. (2024a), we derive the ridgelet transform for depth-n learning machines.
Mo (x)())) := [ JRm JR XR
7(a,b) b)dadb, R
We call Mo [T] an (integral representation of) neural network, and y a parameter distribution.
The integration over all the hidden parameters (a, b) RT F. means all the neurons {x + aa- (a, Rd x R} are summed (or integrated, to be precise) with weight 7, hence formally [2] is understood as a contin- uous neural network with a single hidden layer. We note, however, when y is a finite sum of point measures such as Ppp = -' (a (by appropriately extending the class of y to Borel measures), then it can also reproduce a finite
The contributions of this study are summarized as follows.
We derive the ridgelet transform (solution operator for learning equation) for a general class of learning machines called the ceint-grouu--quivvae machine
2

---

Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Croup-Apuivvriaan Machines
width network
2.2. Irreducible Representation and Schur's Lemma
In the main theorem, we use Schur's lemma, a fundamental theorem from group representation theory. We refer to Folland (2015) for more details on group representation and harmonic analysis on groups.
Mo (x))
Σ 4: (a x bb).
In other words, the integral representation is mathematical model of 2 network with any width (ranging from finite to continuous).
In this study, we assume group G to be locally compact. This is a sufficient condition for having invariant measures. It is not a strong assumption. For example, any finite group, discrete group, compact group, and finite-dimensional Lie group are locally compact, while infinite-dimensional Lie group is not locally compact.
Next, we introduce the ridgelet transform, which is known to be a right-inverse operator to Mg.
Definition 2.2, For any measurable functions p - C
and f: Imm - C, put
Let H. be nonzero Hilbert space, and (H) be the group of unitary operators on H. A unitary representation π of G on H a group homomorphism that is continuous with respect to the strong operator topology that is. a map πT : G + U(H) satisfying Tgh " and 7--1 = "y and for any + E H. the map G 3 g ++ Ty [vb] E H is continuous.
bal = [ (())p(aa b)dæ, (a, E R. x RR.
We call Rp ridgelet transform.
To be precise, it satisfies the following reconstruction for- mula.
Suppose M is a closed subspace of H. M is called an invariant subspace when [y[M] C .M for all g E G. Par- ticularly, π is called irreducible when it has only trivial invariant subspaces, namely {0} and the whole space H.
Theorem 2.3 (Reconstruction Formula). Suppose a and are a tempered distribution (S') and a rapid decreasing function (S) respectively. There exists a bilinear ((a, p)) such that
The following theorem a basic and useful characterization of irreducible representations.
Theorem 2.4 (Schur's 's lemma). A unitary representation π G + U(H) is irreducible iff any bounded operator T on H. that commutes with π is always a constant multiple of the identity. other words, if kg o T = T mg for all
Mr o RRlff = P))::
for any square integrable function f E (. Further, the bilineur form given by ((, Jg (w)))(() where denotes the 1- dimensional Fourier transform.
g G. then T cldx for some E C.
See Folland (2015, Theorem 3.5(a)) for the proof. As we will see in the proof of the main theorem, an irreducible rep- resentation (or more generally, a simple object) is a standard unit for expressive power. Namely, suppose X is a simple object (such as a simple abelian group, and an irreducible representation), and N is non-trivial sub-object (such as a normal group, and uub-representation) then we can con- clude N = X, which means the universality of N in X. Schur's 's lemma restates this in terms of morphism. That is, the commutative property π o T = T o π implies T is a homomorphism, and thus it has to be either zero or identity.
See Sonoda al. (2021b, Theorem 6) for the proof. In particular, according to Sonoda et al. (2021b, Lemma 9), for any activation function o, there always exists p satisfying (() = 1. Here, a being a tempered distribution means that typical activation functions are covered such as ReLU, step function, tanh, gaussian, etc... We can interpret the reconstruction formula as a universality theorem of contin- uous neural networks, since for any given data generating function f. a network with output weight 7f Rf[ re- produces if (up to factor ((a, P))), i.e. Shy]] = f. In other words, the ridgelet transform indicates how the network pa- rameters should be organized so that the network represents an individual function f.
As a concrete example of an irredueible representation, we use the following regular representation of the affine group (ff on L (I"").
Theorem 2.5. Let G == -ff(mm) = -GG(m) x Rm be the affine group acting on = Rmm by (L, x LL + t. and let H. == (RT)) be the Hilbert space of square- integrable functions. Let ग. : Aff(m)
The original ridgelet transform was discovered by Murata (1996) and Candès (1998). It is recently extended to few modern networks by the Fourier slice method (see e.g. Sonoda et al., 2024b). In this study, we present a system- atic scheme to find the ridgelet transform for a variety of given network architecture based on the group theoretic arguments.
(ITT) be the regular representation of the affine group Aff(m) on (RR",, namely n [[]))) := det L) _ t))
for any g = (L, E G. Then x is irreducible.
See Appendix E for the proof.
3

---

Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Crrou--quuvariin Machines
3. Main Results
input
parameter
output
input
parameter
output
We introduce unitary representations т and R, a joint- equivariant feature map @ X x U.E+Yaajonn
machine M[y; øl X + Y. and present the ridgelet trans- form R[f; []] = + C for joint-qquivariant machines, yielding the universality M[][
BY
G-equvvrraace
joint sent----uhanaanee
co.v.. We note that т plays key role in the main theorem, and the joint-qquivariance is an essential property of depth-n fully- connected network.
Figure The classical G-equivariant feature map Φ XxE is subclass of -equi---uvvari map where the G-action on parameter domain is trivial, i.e. g =
Let G' be a locally compact group equipped with a left invari- ant measure dg. Let X and =\t be G-spaces equipped with G-invariant measures dæ and df, called the data domain and the parameter domain, respectively. Let Y be a separable Hilbert space, called the output domain. Let (YY) space of unitary operators on Y, and let u G - (YY be unitary representation of G on Y. We call a vvalued map e on the data-pant parameterrre domain X x i.e. á X x Y, feature map.
Definition 3.2 Srinnt--Eqqivariin Feature Map). We say feature map Φ X E - Y is joint---equivarr when
6(g x, E) = g E): (x,))
) be the
holds for all g E G. Especially, when G-action on Y is trivial, i.e. Φ(g. x, 9 {) ((x, f), we say joint- G-invariant.
A basic example is the fully-connected layer
p(æ, (a. b)) = o a(a x ---28 b),
Let LY denote the space of -vauued square- integrable functions on X equipped with the inner product (é (a(:))) == fx (ő(x), vb(r)) dæ; and let (=) de- note the space of C-valued square-integrable functions on E.
which is not G-equivariant but joint-G-equivariant. See Appendix B for more details.
Remark 3.3 (Relation to classical G-equivariance). The joint-G-equivariance is not a restriction but an extension of the classical notion of G-equivariance, i.c. p(g xz, xx g o(I, E) In fact, G-equivariance is special case of joint- G-equivariance where G acts trivially on parameter domain, i.e. g { = (see Figure 1). Thus, all G-equivariant maps are automatically joint-G-qquivarian
If there is no risk of confusion, we use the same symbol for the G-actions on X, Y, and E (e.g.. 9 xt g y. and g))). On the other hand, to avoid the confusion between G-actions on output domain Y and Y-valued function f x Y, both "g- f(x)" and "xy[fxx]]" always imply G-action on Y, and "xy (x))" (introduced soon below) for G-actions on f X
3.1.1. INTERPRETATION OF JOINT- EQUIVARIANT MAPS
Obviously, Φ is G-map, namely a homomorphism between G-sets X x E and Y. We denote the collection of all joint- G-equivariant maps as homo (X Y). Equivalently, is identified with a G-map de E - y through currying (x(()(z) = (xx {). satisfying -g (z) (z) Further, & is identified with the third G-map &' X + y through (=x( = 6(x, These identifications are summarized as tensor-hom adjunction: homo x Y) = homg(E, ~ homg(X, Y²).
Additionally, we introduce two unitary representations π and गे of G on function spaces Y² and (E( follows.
Definition 3.1. For each g E G. f e L2(X;Y) and y (E).
[f(f((x)= = --gfflg0>> -x)+)))=-- =9 f(g xx, * h)(()) == >= +). E
In terms of geometric deep learning, for example, Cohen et al. (2019) formulate the feature map as a vector field (or section). In their formulation, the joint-equivariant feature map Φe : + y is understood as a global section of a trivial G-bundle p E x y + E over base E with fiber yx, where structure group G acts on fiber y by x.
In the main theorem, the irreducibility of π will be a suffi- cient condition for the universality. On the other hand, the irreducibility of स is not necessary. We have shown that æ and स are unitary representations in Lemmas A.I and A.2.
We note, however, such geometric understanding is not unique. For example, in terms of learning equation M o R Idy the learning machine M = - yx is a feature map, and the ridgelet transform R. y + E is a section (right- inverse). In this perspective, we can conversely understand the feature map o. E + y itself as a vector bundle (or projection) with base space y and total space E.
3.1. Noint-Equivariant Feature Map
We introduce the joint-group-equivvriant feature map, ex- tending the classical notion of group-equivarian feature maps. One of the major motivation to introduce this is that the depth-n fully-connected network, the main subject of this study, is not equivariant but joint-equivariant.
4

---

Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Grruu--gguivvrinn Machines
3.1.2. CONSTRUCTION OF JOINT-EQUTVARIANT MAPS
where the integral is understood the Bochner integral. We also write M M[ Øb) for short. If needed, we call the image M[n; ø) x + Y joint-equivarian machine, and the integral transform M M[+;q] Ø] of a j voint-equivarian transform.
In the following, we list several construction methods of coint-quuivariant maps in Lemmas 3.4, 3.5 and 3.7 (in the next subsection), indicating the richness of the proposed concept. Whereas to construct (non-joint) G-equivariant network, we must carefully and precisely design the network architecture (see, e.g., a textbook of geometric deep learning Bronstein et al., 2021), to construct a jointG--euiivariant network, we can easily and systematically obtain the one.
The joint-equivariant machine inherits the concept of the original integral representation-inleenne all the available parameters & to indirectly select which parameters to use by weighting on them, which linearize parametrization by lifting nonlinear parameters & to linear parameter 7
First, we can synthesize joint-equivariant map from any map do X + Y.
Moreover, the G-action g E on parameter domain also linearized to linear representation ă of G on (=) (defined in Definition 3.1). As important consequence, a joint- G-equivariant machine Mş is jeint-G-equivariant, For later use, we formulate this slogan as the following formula.
Lemma 3.4. Let X and Y be G-sets. Fix an arbitrary map Øu : X + Y, and put Φ(2, g) Tg [ou] g-oulg -z))) for every x. E X and E G. Then, o : X x G -equiv--qvi.
Lemma 3.7. Suppose & + Y be Geiii---quivariant. Then, the associated joint-G-euuivaiins machine Ms T,( (X) intertwines it and x: For every g E G,
Proof. For any g. h E G, we have 60g 2.g --g--- h) (gh)- dol(gh) (g. x)) g o(æ, ht.
Mg kg ng o Mg.
In particular, the case of X = Y = =
= G. namely & G x G - G, is understood as a primitive type of joint-G- equivariant maps.
See Appendix A.3 for the proof. In other words, M is a functor from homg Y*) to (E), L2;;)))
The next lemma suggests the compatibility with function compositions, or deep structures.
3.3. Ridgelet Transform
We introduce the ridgelet transform for joint-equivarintt machines, extending the one for depth-2 fully-connected networks.
Lemma 3.5 (Depth-n Joint-Equivarian Feature Map dim)). Given a sequence of G-equivariant
maps oi : X,-1 x EE, X; (i ..), let E := = x.... En be the n-fold parameter space with the component-wise G-action f1. == (g- EI...., 5.) for each n-fold parameters Eim E Eim. and let @in" Xo x Eim + Xn
Definition 3.8 (Ridgelet Transform). For any joint- equivariant feature map vi X x E - Y and Y-valued Borel measurable function ff on X, put a vallur map by
be the featuee map given by
B[f; (1(() = 1
E
Φ1 {im) = En) (1 (1)
We also write R R(o; 1 for short. If there is no risk of confusion, we call both the image Rff; 4/ X + Y and the integral transform R(e; of f ridgelet transform.
Then, &1: is Sequii--equivarian..
See Appendix A.2 for the proof. In other words, the com- position of joint- equivariaat maps defines cascade prod- uct of morphisms: hom xx") x h5m X;Ko) XxX9)
Formally, it measures the similarity between target function f and feature v() at As long as the integrals are convergent, the ridgelet transform is the dual operator of the joint-equivariant transform (with common Φ):
homo(E, Xxx0.
3.2. Joint- Equivariui Machine
= | (8) (())) xxx
(7,
We introduce the joint-equivariant machine, extending the integral representation.
-M MM ()
Definition 3.6 (Joint- -quiva-rqivaiiant Machine), Fix an arbitrary joint-equivariant feature map Φ X x E + Y. For any scalar-valued measurable function 7 E + C, define a y-valued map on X by
As a dual statement for Lemma 3.7, the ridgelet transform is also joint- Saitt---ivariant and particularly an intertwiner.
Lemma 3.9. Suppose ib hhomg(E, Y YX), then we have
Ry Rg = म. for every g G.
M[n (xx) = (E)) x E X,
In other words, Ry E home (X² (2)) See Ap- pendix A.4 for the proof.
5

---

Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Croou--guuiariin Machines
3.4. Main Theorem
G-space
space
feoture
6-spoce
sector-space
At last, we state the main theorem, that is, the reconstruction formula for joint-equivariant machines.
Theorem 3.10 (Reconstruction Formula). Assume (1) fea- ture maps 4, 1.4/: X x - Y are Deint-G-equivaiinn, (2) composite operator Mş Ry YY) + L)) is bounded (i.e., Lipschitz continuous), and (3) the unitary representation क. G + (XL ()) defined in Defini- tion 3.1 is irreducible. Then, there exists bilinear form ((o, +)) C (independent of f) such that for any Y -valued square sauare-integrbbee function (X;Y),
Figure 2. Deep Y-valued joint- -equi---quiaaiiann machine on G-space X is Y Juniivesal when unitary representation of G on (2(X; Y) is irreducible, and the distribution of parameters for the machine represent given map f X + Y is exactly given by the ridgelet transform R[f]
Ms R.y [/]] = / (fx)) (x ())dr))) E)dE = ((), V.
5.
The assumptions on feature maps Φ 1/ that they are joint-equivariant and not orthogonal need to be verified in a case-by case-by-case by--cse manner. Fortunately, we can use the dloeed-form expression of the ridgelet transform to our advantage. For example, for fully-connected networks (Section 5) and quadratic-form networks (Section 7), the joint-qquivariance holds for any activation function. For the case of depth-2 fully-connected networks, it is known that the constant is zero if and only if the activation function is a polynomial function (see e.g.. Sonoda & Murata, 2017a).
In practice, once the irreducibility of the representation π on (x; is verified, the ridgelet transform R becomes right inverse operator of joint-qquivariant transform Mo as long as ((o +)) # 0, co. Despite the wide coverage of examples, the proof is brief and simple as follows.
Proof. Put T = Mg o Ry Y) + ()) By Lemmas 3.7 and 3.9, commutes with π as follows
Mg R.g. Tg Mp ng o Rs ng Mg B
4. Example: Depth-n Joint-Equivariant Machine
for all E G. Hence by Schur's lemma (Theorem 2.4), there exist a constant Co.s E C such that M, RR = Co.v [d)=(x)- Since MS o Ry is bilinear in & and vb, Co.. d is bilinear in Φ and vh.
As pointed out in Lemma 3.5, the depth-n feature map Φi (z. {im) = (n.) o Φ (1) is joint- G-equivariant when each component map Φi is joint- equivariant. Hence, universal deep joint-cquivariant machine DM(y; @1ml (see also Figure 2).
Remark 3.11. 1. When æ is not irreducible (thus re- ducible) and admits an irreducible decomposition such as (X; Y)= +i H4 then the reconstruction for- mula M e R[f] holds for every f E H4 for some k. This is another consequence from Schur's lemma.
construct ();
Corollary 4.1 (Deep Ridgelet Transform). For any maps ?y L'( and f E (X;;),,, put
2. The irreducibility is required only for π, and not for म. This asymmetry originates from the fact that our main theorem focuses on the universality of the learn- ing machine, namely Mo Mn] X Y, not on its dual [f[ E + IR. When से is irreducible, we can fur- ther conclude Ryp. Mp[7] = for any 7 E T, (E) (the order of composition is reverted from M2 R3)). In practical examples such fully-connected networks and wavelet analysis. however, Ry Mg is only pro- jection due to the redundancy of parameter distribution 7(a,b) b).
DM(y; (1:n { (E1sm) ddind x E X,
JE Riʃ; ((imn)) = / (f(x), (1 Yy da, Eim E Eim-
Under the assumptions that DM o Re is bounded, and that a is irreducible, there exists a bilin- ear form (orm vin)) satisfying DM o B (0 Idxx)x)))
3. Relations to cc-universality, and sufficient conditions on I bo-uddudsssssss of M o R, are discussed in Appen- dices and D respectively.
Again, it extends the original integral representation, and inherits the linearization trick of nonlinear parameters fum by integrating all the possible parameters (beyond the dif- ference of layers) and indirectly select which parameters to use by weighting on them.
4. As also mentioned in Section 3.1.1, M₁ (²(X; is a G-equivariant vector bundle, and Ry T2(X;Y) > (B) is a G-equivariant section.
6

---

Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Grrou-Eguuivaiin Machines
5. Example: Depth-n Fully-Connected Network
So by putting depth-n neural network and the corresponding ridgelet transform as below
We explain the case of depth-n (precisely, depth-n + 1) fully-connected network.
DNN(y: - / =mm
Eis) t
Set X = Y = IR. (input and output domains), and for each i E {l...., n). set X; == IR. (with X1 = X and Xn+1 Y). E == xn x R x S9: domain), where Su denotes the d 1-dim. d:t3 (parameter sphere, o, : RP: - R (activation functions), and define the feature map (vector-valued fully-connected neurons) as
R (h:00(mn)))) = / pm f(xr) 1a Eis)dd,
Theorem 3.10 yields the reconstruction formula DNN R = ((pmm (1 (Rm)))))
6. Example: Depth-n Group Convolutional Network
(x; Ei) == C,,,(A, - b)),
for every 2; E R & = (A,, by Ci) E E, Specifically, di = d = m. If there is no risk of confusion, we omit writing i for simplicity.
As mentioned in Remark 3.3, all the classical equivariant feature maps, namely Φ : X x = - Y with trivial G-action on parameters: 6(g xz () = 9 Φ(x, E). are automatically joint-equivarint. Therefore, once the irreducibility of rep- resentation π is verified, our main theorem can state the ridgelet transform for classical G-equivariant networks.
Let (m) denote the orthogonal group in dimension m. Let G== (m) x (m) be the product group of (m) and (m) = GL(m) x R... We suppose G acts on the input and output domains as below: For any = (Q, L, G= (m) x (GL(m) k ).
In fact, in the case of group convolutional networks (GCNs) with vector inputs, tion for affine groups (f). In the following, we explain the ridgelet transform for depth-n GCNs, extending gen- eral 2epth-2 GCNs formulated by Sonoda et al. (2022a), which covers a wide range of typical group equivariant net- works such as an ordinary G-convolution, DeepSets and E(n) rqqiivvaaa maps unified manner.
reuse the irreducible representa-
g x == Lx + t, xx
g g vgluy]= == Qy, y Y.
Namely, the group actions of both ((m) on X and (f])m on Y are trivial.
Let x be the unitary representation of G on the vector-valued square-integrable functions f (xx defined by
In the previous study, the ridgelet transform was derived only for depth-2 GCNs, which is due to the proof tech- nique based on the Fourier expression method (see Sonoda et al., 2024b, for more details), another proof technique for ridgelet transforms that does not require the irreducibility assumption but is limited to depth-2 learning machines.
t)), Tg [xx) == det (|- t)), xE I,,,,
for each g = L,, E (m)) x (GL(m) x R.).
Lemma 5.1. The above x : G - U(? is irre- ducible.
See Appendix A.5 for the proof. Additionally, we put the dual action of G on parameter domain E, as below:
In the following, we extend the GCNs from depth-2 to depth- n and derive the ridgelet transform by reviewing it from the group theoretic perspective. The main idea is to turn the depth-n fully-connected network (FCN) é1 in Section 5 to a depth-n G-convolutional network, denoted oi... by following the construction of the previous study.
(A,L-1,b) + A, i=1
g. (A, bi C,) =
(Ai,b;, Ci), bi),
i + 1,
(Ai; bi, QC,),
i n
for all 9 L,) E 0(m) x (GL(m) K RTT), (A,,,, C,)
6.1. Notations
Besides the primary group G for convolution, we introduce an auxiliary group : O()) x Aff = 0(m) x (GL(m) x I."). where A and G need not be homomorphic. Eventually, the irreducibility assumption of i required not for G but for A. Hence, different from Section 5, the group acting on Yx; by is not G but A. In accordance with the previous study, we write T [ for G-action, a for A-action if needed, and [f](()) == (T for G-action on function f X + Y. By LX; we denote the space of G-equivariant Y -valued functions
Then, the composition of feature maps 61 (x, E1s1) ==
É1) is joint-G-equivariaant In fact, (n) o...o o (1 (x,E1)
Gi g.g.tt. +1) = Cia (A, + t)- (b, t ALL-
= C - b1) (x,
(1 3 = C,((A,0 b.)= = (m, Ei), i 1,
(n g. {n) = QC - bn) = g Φn (x, En),
Therefore @1 (g- x. g: (Im) = g Pin (ar, E1.m
7

---

Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Croup-Equuvvrinn Machines
on X that is square-integrable at the identity element le of G, namely Y); = (f E homa(X, YG) (La))lxx)))))) oo) = [T-l6]]]
Theorem 6.1 (Reconstruction Formula). There exists bi- linear form (Crna sl )) such that for any f E ( :Y), GCN [f: Pin)] Pin]] = ( 1.
LX)
From the next subsections, we will turn a joint-A- equivariant map Φlnn to G-equivariant map Vin.
See Appendix A.7 for the proof. When - 2. the argument here reproduces the one for depth-2 GCNs presented in Son- oda et al. (2022a). We remark that the base feature map é and auxiliary group A need not be the fully-connected network and affine group. In fact, we have never used the specific property of Co(Ax b). but only used the group actions. Thus A and ø can be arbitrary group and joint- A-equivariant map. When A is the affine group, then the irreducibility of x has already been verified in 5.1. On the other hand, when A is another general group, we need to ver- ify the irreducibility of representation π of A on (X;
6.2. G-Convolutional Feature Map
For each i, let o, X; x E; + X1+1 be the conneceodd map di (a:;, Ei) = (A, b;) (as in Section 5). We
define the G-convolutional map oī : X; x E, + x as follows: For every z; E X; and &: = (A, b, C.) E E,.
0. (g) == Ty Tgl
[C [x:] b,)l, g E G.
By appropriately specifying the G-action T. the expres- sion (,Tg-1(2.] can reproduce a variety of general G- convolution products, say a #T z, such as an ordinary G-convolution, the ones employed in DeepSets and B(n)- equivariant maps (see Section 5 of Sonoda et al., 2022a).
7. Example: Quadratic-form with Nonlinearity
Here, we present a new network for which the universality was not known.
Similarly to Lemma 3.4, each G-convolutional map o7 is G- equivariant in the classical sense because for any g, h E G,
Let M denote the class of all m x m-symmeerie matrices equipped with the Lebesgue measure dA Auzs da,y. Set X = R = = M x Th x IR, and
0 (Tg [.... = T, (T. = (o [2,] <.)]] = T (x;, (M))
<<x, E) = ox T Ax + x b+c)
for any fixed function o : R + R. Namely, it is a quadratic- form in x followed by nonlinear activation function d.
Remarkably, the G-equivariance holds for any activation function o;, because is applied element-wise in G.
Then, it is joint-invariant with G = Aff(m) under the fol- lowing group actions of g = Lt, E R GL(m):
6.3. G-Convolutional Network and Ridgelet Transform
Next, we define the depth-n G-convolutional map 6T.. X x iim + Y by their compositions:
Lt. x Lxx Lt. (A,, == : (L- (L-TLLL-- - b- AL-
(x, (o))) = o% En)(g) . (1. (g)(
c+ - AL- (T).
and define the depth-n G-convolutional network and ridgelet transform follows. For any 7 E (= and E L2 Y))
See Appendix A.8 for the proof of joint-invariann By Theorem 2.5, the regular representation of Aff(m) on ('RI)T)) is irreducible. Hence as consequence of the general result, the following network is L2( -universal.
(())) = [ (EIm) (1, (o) = Reonv (f: = [ (f(x)(1o), din (im)) dæ.
QNN (x)) = / 7(A, b(A,b Ax + To c)dAdbde.
8. Discussion
See Appendix A.6 for more technical details on GCNs. The ridgelet transform encodes the information of function f only at single point Lg (see also Lemma A.5). This is due to the G-equivariance of / that the image y can be copied from the image at 1G by translation: (g) = T.If In fact, the G-convolutions in depth-n GCN has mechanism to expand the image at lg to entire G by using G-equivariance (see Lemma A.4 for more precise meanings).
We have developed a systematic method for deriving a ridgelet transform for wide range of learning machines de- fined by joint-group-ert group-equivariant feature maps, yielding the universal approximation theorems as corollaries. Tradition- ally, the techniques used in the expressive power analysis of deep networks were different from those used in the analy- sis of shallow networks, as overviewed in the introduction. Our main theorem unifies the approximation schemes of
8

---

Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Group-Equivvriinn Machines
both deep and shallow networks from the perspective of joint-group-action on the data-parameter domain. Techni- cally, this unification is due to the irreducibility of group representations. From the traditional analytical viewpoint, universality refers to density. In this study, we have reviewed universality as irreducibility (or more generally, simplicity of objects) from an algebraic viewpoint. This switch of viewpoints has enabled us reunify various universality theorems in clear perspective.
oretical framework that can precisely handle compositions of functions.
Acknowledgements
This work was supported by JSPS KAKENHI 24K21316, 25H01453,, JST PRESTO JPMJPR2125, 25, JST BOOST JP- MJBY24E2, JST CREST JPMJCR2015 and JPMJCRI913.
Impact Statement
Additional Comments on Significance after Rebuttal
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
The main theorem of this work is not merely unification of existing results; a key strength lies in its ability to system- atically and mechanically assess universality even for novel architectures. For example, it enables the straightforward design of expressive networks with universal approximation capability, such as quadratic networks.
References
In the era of large language models, the diversity of pro- posed architectures has significantly increased. Manually analyzing the expressive power of each new model on case-by-case basis is no longer feasible from theoretical standpoint. Our framework addresses this challenge by of- fering a general principle.
Barron, A. R. Universal approximation bounds for super- positions of sigmoidal function. IEEE Transactions on Information Theory, 39(3):330-945, 1993.
Bronstein, M. M., Bruna, J., Cohen, T., and Veličković, P. Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges. arXiv preprint: 2104.13478 2021.
In particular, algebraic tools such as Schur's lemma provide a notable advantage in that they allow us to assess universal- ity (or density) even when the inputs and outputs are vec- tors something that traditional techniques often struggle with. While classical theorems like the Weieesssssssssss or Maan-Banachh theorems offer generic tools for establishing universality, applying them to deep neural networks with hicrarchical structures requires considerable ingenuity.
Cai, Y. Achieve the Minimum Width of Neural Networks for Universal Approximation. In The Eleventh International Conference on Learning Representations, 2023.
Candès, E. J. Ridgelets: theory and applications. PhD
thesis, Standford University, 1998.
Carroll, S. M. and Dickinsso,, B. W. Construction of neural nets using the Radon transform. In International Joint Conference Neural Networks 1989, volume I, pp. 607- 611. IEEF, 1989.
By contrast, proof techniques originating with Hornik al. (1989), which rely on repeatedly differentiating activation functions to construct polynomial approximations, often obscure the intuitive mechanism by which neural networks perform approximation. In comparison, our use of joint group equivariance and ridgelet transforms as inverse opera- tors provides a more direct and interpretable understanding of the approximation process in deep networks.
Chen, R. T. Q., Rubanova, Y., Bettencourt, J., and Duvenaud, D. Neural Ordinary Differential Equations, In Advances in Neural Information Processing Systems, volume 31, pp. 6572-6583, 2018
Regarding generalization: Theoretical analysis of general- ization error is often based on estimates of the Rademacher complexity of the hypothesis class. However, in deep leamn ing, the Rademacher complexity typically scales exponen- tially with network depth. This is at odds with practical observations, where deeper networks tend to generalize bet- ter.
Cohen, A., DeVore, R., Petrova, G., and Wojtaszezyk, P. Optimal Stable Nonlinear Approximation. Foundations of Computational Mathematics, 22(3):607-648, 2022.
Cohen, N., Sharir, O., and Shashua, A. On the Expressive Power of Deep Learning: A Tensor Analysis. In 29th Annual Conference on Learning Theory, volume 49, pp. 1-31, 2016.
This discrepancy arises from the limitations of current tech- niques for analyzing composite functions: the reliance on coarse Lipschitz estimates leads to overly pessimistic bounds. To address this gap, we revisited expressive power analysis and developed the deep ridgelet transform as a the-
Cohen, T. S., Geiger, M., and Weiler, M. A General The- ory of Equivariant CNNs on Homogeneous Spaces. In Advances in Neural Information Processing Systems, vol- ume 32, 2019.

---

Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Croup--Apuivvriann Machines
Cybenko, G. Approximation by superpositions of sig- moidal function. Mathematics of Control, Signals, and Systems (MCSS), 2(4):303-314, 1989.
Lang, S. Real and Functional Analysis. Graduate Texts in Mathematics. Springer New York, edition, 1993.
Li, L., Duan, Y., Ji, G., and Cai, Y. Minimum Width of Leaky-ReLL Neural Networks for Uniform Universal Approximation. In Proceedings of the 40th Interna- tional Conference on Machine Learning, volume 202, pp. 19460-19470, 2023.
Daubechies, I., DeVore, R., Foucart, S., Hanin, B., and Petrova, G. Nonlinear Approximation and (Deep) ReLU Networks. Constructive Approximation, 15(1):127-172, 2022.
E. W. Proposal on Machine Learning via Dynamical Systems. Communications in Mathematics and Statistics, 5(1):1-11, 2017.
Li, Q. and Hao, S. An Optimal Control Approach to Deep Learning and Applications to Discrete- Weight Neural Networks. In Proceedings of The 35th International Con- ference on Machine Leurning, volume 80, pp. 2985-2994, 2018.
Evans, L. C. and Gariepy, R. F. Measure Theory and Fine Properties of Functions. CRC Press, revised edition, 2015.
Lin, H. and Jegelka, S. ResNet with one-neuron hidden lay- ers is Universal Approximator. In Advances Neural Information Processing Systems, volume 31, 2018.
Folland, G. B. Course in Abstract Harmonic Analy- sis. Chapman and Hall/CRC, New York, second edition, 2015.
Lu, Z., Pu, H., Wang, F., Hu, Z., and Wang. L. The Expres- sive Power of Neural Networks: A View from the Width. In Advances in Neural Information Processing Systems, volume 30, 2017.
Funahashi, K.-I. On the approximate realization of contin- uous mappings by neural networks. Neural Networks, 2 (3):183-192, 1989.
Grohs, P., Klotz, A., and Voigtlaender, F. Phase Transitions in Rate Distortion Theory and Deep Learning. Foun- dations of Computational Mathematics, 23(1):329-392, 2023.
Murata, N. An integral representation of functions using three-layered networks and their approximation bounds. Neural Networks, 96):947-956, 1996.
Park, S., Yun, C., Lee, J., and Shin, J. Minimum Width for Universal Approximation. In International Conference on Learning Representations, 2021.
Haber, E. and Ruthotto, L. Stable architectures for deep neural networks. Inverse Problems, 34(1):1-22 2017.
Hanin, B. and Sellke, M. Approximating Continuous Func- tions by ReLU Nets of Minimal Width. arXiv preprint: 1710. 1 1278, 2017.
Petrova, G. and Wojtaszezyk, P. Limitations on approxi- mation by deep and shallow neural networks. Journal of Machine Learning Research, 24(353):1-38, 2023.
Hornik, K., Stinchcombe, M., and White, H. Multilayer feedforward networks are universal approximators. Neu- ral Networks, 2(5):359-366, 1989.
Siegel, J. W. Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spaces. Journal of Machine Learning Reseurch, 24(357): 2023.
Sonoda, S. and Murata, N. Neural network with unbounded Kainen, P. C., Kūrková, V., and Sanguineti, M. Approx- imating multivariable functions by feedforward neural nets. In Handbook on Neural Information Processing, volume 49 of Intelligent Systems Reference Library. pp- Springer Berlin Heidelberg, 2013.
activation functions is universal approximator. Applied and Computational Harmonic Analysis, 43(2):233-266 2017a.
Kidger, P. and Lyons, T. Universal Approximation with Deep Narrow Networks. In Proceedings of Thirty Third Conference on Learning Theory, volume 125, pp. 2306- 2327, 2020.
Sonoda, S. and Murata, N. Transportation analysis of de- noising autoencoders: novel method for analyzing deep neural networks. In NIPS 2017 Workshop on Optimal Transport & Machine Learning (OTML), pp. 1-10, Long Beach, 2017b.
Kim, N., Min, C., and Park. S. Minimum width for universal approximation using ReLU networks on compact domain. In The Twelfth International Conference on Learning Representations, 2024.
Sonoda, S., Ishikawa, I., and Ikeda, M. Ridge Regression with Pver-Parametrizee Two-Layer Networks Converge to Ridgelet Spectrum. In Proceedings of The 24th Interna- tional Conference on Artificial Intelligence and Statistics, volume 130, pp. 26822 2021a.
Kobayashi, T. and Oshima, T. Lie group and representation theory (in Japanese). Iwanami Shoten, 2005.
10

---

Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Gaarrruu-Euuivvraae Machines
Sonoda, S., Ishikawa, I,, and Ikeda, M. Ghosts in Neu- ral Networks: Existence, Structure and Role of Infinite- Dimensional Null Space. arXiv preprint: 2021b.
Sonoda, S., Ishikawa, I., and Ikeda, M. Universality of Group Convolutional Neural Networks Based on Ridgelet Analysis on Groups. In Advances in Neural Information Processing Systems 35, pp. 38680-386944 2022a.
Sonoda, S., Ishikawa, and Ikeda, M. Colly-Connected Network on Noncompact Symmetric Space and Ridgelet Transform based on MilgassonFForrice Analysis. In Pro- ceedings of the 39th International Conference on Ma- chine Learning, volume 162, pp. 20405-20422, 2022b.
Sonoda, S., Ishi, H., Ishikawa, and Ikeda, M. Joint Group Invariant Functions
Domain Induce Universal Neural Networks. In Proceedings of the 2nd NeurIPS Workshop on Symmetry and Geometry in Neu- ral Representations, Proceedings Machine Learning Research, pp. 129-144. PMLR, 2024a.
Sonoda, S., Ishikawa, I., and Ikeda, M. A unified Fourier slice method to derive ridgelet transform for a variety of 2 neural networks. Journal of Statistical Planning and Inference, 233:106184, 2024b.
Telgarsky, M. Benefits of depth in neural networks. In 29th Annual Conference on Learning Theory, pp. 1 1-23, 2016.
Yamasaki, H., Subramanian, S., Hayakawa, S., and Son- oda, S. Quantum Ridgelet Transform: Winning Lottery Ticket of Neural Networks with Quantum Computation. In Proceedings of the 40th International Conference on Machine Learning, volume 202, pp. 19008-39034 2023.
Yarotsky, D. Error bounds for approximations with deep ReLU networks. Neural Networks, 94: 103-114, 2017.
Yarotsky, D. Optimal approximation of continuous func- tions by very deep ReLU networks. In Proceedings of the 31st Conference On Learning Theory, volume 75, pp. 639 649, 2018.
Yarotsky, and Zhevnerchuk, A. The phase diagram ap- proximation rates for deep neural networks. In Advances in Neural Information Processing Systems, volume 33, pp. 13005-13015, 2020.
Yosida, K. Functional Analysis. Springer- Verlag Berlin Heidelberg, edition, 1995.

---

Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Croup-Equivvrian Machines
A. Proofs
A.1. Unitarity of Representations
In Definition 3.1, ı and are defined as below: For each g E G, (Y) and y E (²)
(x) : =1gls xx]] xx]==] = g.
n [g) = 10-00--1000
Lemma A.1. æ is unitary representation of on (Y:
Proof. Recall that the representation G on unitary. So, for any g, h E G and f (Xx;
Tg [=]= g9. (h- fa- (o- xx)) (gh) f((gh)( x) = gihh (z)
and for any g G and E
(T ( J (vol6i(0 "g(2(9 r))l/drr
= efgb (ft (2)
Lemma A.2. &t is unitary representation G on (E).
Proof. For any g, h G and 7 (E)
R (xll))((E) = ((h (g) = -(gh))" .g) = (f))
and for any G and 71, E L3)),
(T ( / - -002l0-111111000 E)dg
[ (()))(()dd = (2)
A.2. Proof of Lemma 3.5
Proof. For any G x E X, and Eim E Eim we have
Φin(g: g.g {im) = on ( Ex) o... 22(0,9 E2) o x. E1)
dn( ( En) o ... Φ2 E2) di (x,
= Ón
En. o 0
= @n En) o...... 02 (z, (2,
A.3. Proof of Lemma 3.7
Proof. We use the left-invariance of measure dg. and joiut-G-equivariiance of o X x x + Y. For any E G, x E X, we have
My T (x))()))) =
7 ()
7(E)d(1, E)dg
xx, xx, (()),,000
E)]dE TyMM)lVl(<).
12

---

Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Crrou--gquivvrinn Machines
A.4. Proof of Lemma 3.9
Proof. We the unitarity of representation + UY), left-invariance of measure dx, and join-CC-epuivariianee of
X = + Y. For any g E G.C E, we have
/ (wglf(r x)1 (a, yz
By By(T)(f]((E)
/ (F( ,)) dr
/ (s(x), ugl(( 2,
/ (f(z). 5)) Ry (f]((E.
A.5. Proof of Lemma 5.1
Proof. We use the following fact.
Lemma A.3 (Folland (2015, Theorem 7.12)), Let and To be representations of locally compact groups G1 and G2, and let Tı @ 22 be their outer tensor product, which is a representation of the product group G1 x Gz. Then, Tı and ne are irreducible and only if mı @ ι2 is irreducible.
Recall representations of O(m) on R and of Aff on (IR are respectively irreducible (see Theorem 2.5), and 2 (R";; IT") is equivalent the tensor product IR'' @ L (m Hence by Lemma A.3, the representation ı of the product group (m) (ff on the tensor product Th @ I,2 (Rm) T,2 (R.; I")) is irreducible.
A.6. Connection between GCN and FCN
Recall that the depth-n FCN and its ridgelet transform introduced in Section are given as below.
DN (im)))))))))) = [ 7(11n) (x. Ein dd
Reelf: (1 = =s (f(x), s/ E1:n))
As consequence of Lemmas 3.5 and 3.7, have the following.
Lemma A.4. GCN(y: (I.,)(=))) = Ty [NNNNN[2: (x)).
Proof.
TT.n (z, (g) Ty[on ( 0 .... (T E1)
Tylpin ( (x))
Ty Tm(0 [x0 Eim)
and thus
GCN(T; (()))(()))))))))))))))))(())
J 7(E1m) 1n ( (i.)dE [N(DNN(>s (().)())))))
Lemma A.S. Reonv [J: (1
Proof. Immediate from the definition.
13

---

Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Croup-Equivarian Machines
A.7. Proof of Theorem 6.1
Proof. By Lemmas A.4 and A.5,
GCN //:m: Pfn
Ty (f (1ml((e) Toll(oims vin f (Lc))(()) ((olnss 1m (z)))
A.8. Joint-equivariance of quadratic-form network The feature map and group actions are given as follows.
((x,{) = ox Ax x b c)2
L) x = 1 Læx
Lt (A. b, c) == AL-!A- L 21 -1-
- - L=
Then, is joint-invariant. fact,
<(g g E) a(Lxx L. (La (Lx+))
tt (I. 2T- ALL- + +..)
o oxx Ax - 2x AL L + 2x AL A L-
-2t L + L- -
aa Ax + x. b +c) c) = <lg a. g E)
B. Example: Depth-2 Fully-Connected Network Here we reproduce the ridgelet transform for depth-2 fully-connected network, originally presented in Sonoda al. (2024a).
Set X R.'h (data domain), = R" x IR (parameter domain), and G == Aff(m) G..(m) Rhm be the m-dimensional affine group, acting on data domain X by
Lx +t, g (L,t) E GL(m) x R". x E X. g. x
Addition to this, let be the regular representation of Aml on L2² namely
म (x)lf(()) det -/ t)), E (X) and g (I, GL(m) T..
Further, define dual action of (f) on the parameter domain = as
g (a. (L-T T b L a). g == = t) e GL(m) R", (a. E.
Then, we can see the feature map (bxr, (a) = (a b) is joint-G-invariany
fact,
p(g x. g (a.b)) = (L-Ta (Lx + t) - (b (b+ TTa) a)) = o(a x - b) = o(x, (a, b)).
Because the regular representation т G Aff is irreducible, by Theorem 3.10, the depth-2 neural network and corresponding ridgelet transform:
(x)) = =f -
b)dadb, and B[fl(a,b) S f(x)p(a bjdæ.
((a.))o
satisfy the reconstruction formula NN R ((a Id)=(Rm)-
14

---

Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Crouup-Aquivvrinn Machines
C. cc- Universality by Bochner-Integral Representation
summarize a general scheme to show the co-universality by discretizing the integral representation.
C.1. Notes on Bochner Integral
We refer to Chapter VI of of Lang (1993), where the chapter title The General Integral refers to Banach-valued integrals. Let (X, M, ui) be a measure space, and let be a Banach space.
Lemma C.1. A strongly M-mmasurable function f + E Bochner ju-itterablee iff [[f is u-integrable, i.e. [x (I) < oc.
See Yosida (Theorem 1, V.5 1995) for the proof.
Lemma C.2 (Dominated Convergence Theorem for Bochner Integral). Let {fn X - E} be a sequence of Bochner integrable functiins.
Assume {fa} converges almost everywhere to some function f f X -
Let X I. be integrable function such that for all (zf S g(z) for almost every X.
Then is Bochner integrable and T.- -.onverrg to ie..
||I - falldæ O, and / fdu= lim
n-t00 /
lim 1=000
Sadu
See Lang (Theorem 5.8, VI, 5 1993) for the proof.
Lemma C.3 (Bounded Convergence Theorem), Assume additionally that X is a finite measure space. Let {fn X E} be sequence of Bochner integrable functions.
Assume
{fn] converges almost everywhere some function f X E.
Let G > be a uniform constant such that for all n, [lff(())]| & G for almost every x E X.
Then is Bochner integrable and Ll' -converrg to i.e.
/ Ilf fn falldp- O, and [ fdw fdu= lim
n / fadp
lim +00
This is an immediate corollary of DCT with dominant function being 9 = G.
C.2. Application to Discretization of Integral Representation
Theorem C.4 (Uniform Approximation of Integral Representation by Finite Networks). Let X be topological space, Y be a Banach space, and E = C(X; Y) be the Banach space of Y -valued continuous functions on X equipped with the uniform norm [lfl)c(x.x) == supaex [x)])
Let be a compact metric space with finite Borel measure, let & X x Y and 7 (or d: X E and y Y) be measurable functions. Assume that the E-valued function (p{) = (E) {) is Lipsehitz continuous, namely
1 {) E') <
L == sup E.E'E=
dz(E,E")
Then, the Y-valued continuous model
NN(x) 7()), xeX
J
15

---

Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Croup-Equivvaiin Machines
exists in the sense of C(X;Y) YYXxuludd Bochner integral, and there exists a sequence of finite models: For each nt,
No
INN, (x)
Σ (2n,
EEX
with some (Wn,s,nn)) E R x E (or E B) that uniformly converges to the continuous model, namely
lim - f.w. = 0. -
Remark C.5. We do need assume a measure on X since only consider continuous functions with sup-norm, neither the compactness of X since the Lipschitzness of is simply assumed. We use the compactness of E to take finite covering and obtain finite representative points {En,i). ). and to guarantee the existence of maximum supses [1 z (with uniform continuity of (p), We use the metric of E\t to assume and use the Lipschitzness of to show the a.c. convergence. We use the finiteness of v=) to use the bounded convergence theorem instead of dominated convergence theorem, and to show the Bochner integrability of discretized models. We do not need to assume the Bochner integrability of yp as it is consequence.
Proof. Let An, be a sequence of finite disjoint covering of with diameter at most 1/n. Namely each A, is a finite family {Ana i [Nv]] (with cardinarity N, < oo) of measurable sets satisfying =\t = L An,i and supe,eeA., de(s, S 1/n.
Put sequence (4pn : 3 >> F: n E NJ of functions: For each n,
N₂
(n(() Σ XAm.. (E)p(En,))
E E.
Then, (1) the sequence uniformly bounded: For any n,
Nn XA XA.,, (5)p((
Nn. (S)l( S sup [|{(()] <0. EEE
[()(()) =
E
1=1
The last maximum exists because E\t is compact and yo is uniformly continuous. As consequence, (2) each function is Bochner integrable: For any n.
[ [In((E)l s vol(E) sup || < <0. EEE
Finally, (3) a.e.-converges to yp: At almost every [ E E, there uniquely exists An,i including ç, so I(((E) pp, (E)l (1x(() (pn,. E Tu En,ill E Lu/n O,
Hence, by the bounded convergence theorem (Lemma C.3), &o is Bochner integrable and
[ (n(()d
1 E XA (5)(,,))
J i=1
| p(()dS- [ ((E))lo, E)SE, +
In other words, putting Wn,i (Em.i)) for short, we have shown
lim
Σ
wn 3(. En.i)
(E)))((EE)d
0.
((XYY)
Notes. In the proof, we only used the existence of the partition with diameter at most E by the compactness. Once and Φ are specified, we may make use of the structures of them in a computable and effective manner. For example, a better discretization scheme investigated in the so-called Maurey-Jones-Bee Jonrs-Baron (MJB) theory to achieve the dimension independent Barron's bound (see, e.g., Kainen al., 2013).
16

---

Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Noint-Croup-Equivariie Machines
D. Boundedness of M R
We present several sufficient conditions for the L'-bounnednees of M R.
Lemma D.1. For any f E L2) put
T[f(() == = MoR[f(()) = | V (w))ly o(2, E)d5, x X.
The I' is bounded, namely there exists a constant OT such that for any f L=
|T((||l S Crlli/laa(x)
when at least one of the following conditions hold:
T1. The integral kernel k(x,, y) := J=
E)dE is square integrable.
T2. The two feature maps ④, vp are square integrable.
T3. For some topological vector space I, both R (X) - I' and M: P (X) are bounded.
Proof. Case TI.
IT = 1 1 Ux Soowers)aa b(a; s)de
2
dx
Sx 1 f(v) [L (.v.))o(e s)dE dy dx
2
Jx 1Sx (())k(a de
- U ( 1 (k(2,,)) Pyddr = 11
Case T2.
[T|]I|1a(X) = / / R(S((E)((,,) s)dE da
< S (R/ RdEd
z (() [ //// f()) de
s ( / ( E) dyde
s /1l.
Case T3.
| = ||o I Ca||R|f[IIr s CxCallflex(x)
Lemma D.2. Suppose Φ2 (1(E1,)) : 1 det D( oJ (y)) satisfying [Millae < oc. Then, d2 o òi Xi E + is square integrable.
X2 x 3g - R is square integrable, and Φ1 : X1 x E1 + X2 has inverse jacobian
Remark D.3. When input and output dimensions differ, replace the jacobian J1 with V o J (2015) for more details.
See Evans & Gariepy
17

---

Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Crouu--gguivriinn Machines
Proof. For almost every & 1, put x = (51, v). dæ JI(S1, with J1(51. det [e D( (v))) (v))) Put Y = Usiez (I X). Then,
J
JXXEI (22(E2, si (S,,
1db(EE2 ()idd)d
3 (2)()..))
< 1112 Vorcon
(02((2, Rdl
(1 d5
(E(I(EL))))
I (oba(e:.))
sup (1
df1:
1112
LUEY
|| =10b|lB2rExx)) (illlr
E. Irreducibility of the infnite-dimensionall representation of the Affine group
We refer to Folland (2015, Theorem 6.42) for more details on the infiiie-dimmnssionl irreducible representations. The following proofs are attributed to Kobayashi Oshima (2005), Japanese textbook.
Lemma E.1 (Kobayashi & Oshima (2005, Lemma 2.8)). Let T be the translation group in dimension m, and T:T ((L((R be the unitary representation T defined by the regular action. A closed subspace V of L (RRM) 7-invariant iff there exists measurable E of Th such that
V V=F-((>())
Proof. Let VL denote the orthocomplement of V. Since V is r-invariant, for all f E V and E VL,
f (x) = f f(x ()))(y) = f f(z + vlglr)dy = (r- = O.
Therefore, for all f E V and V-,
f(E)((E) = 0, a. E.".
(I)
Let be the Gaussian measure on R defined by du({))
Put
E(V) = U C I. measurable | E C supp f)
fEV
and
A: sup ((E) S u(R²") < oc. BEE(V)
Let Ej be sequence of measurable sets of &(V) satisfying limj-soo 1 EEE)) A, and F == U, Ej. Then F is measurable, and u(F)) 2 A because lim sup μu( E,) u(lim sup E,) in general. By Equation (1), 3(E) 0 at e. Ej for all j and V- Hence
v c CF-
[R (L' [))
thus
V VS VJ (L
(BT(A))) - [LP(F))
Conversely, suppose that there exist / V and measurable subset F' C Rm satisfying f(E) + 0a.e. F", Then ](E) 0 at a. e. F' for all g 1 and have
- C
LL (I m (FUF))))
18

---

Deep Ridgelet Transform and Unified Universality Theorem for Deep and Shallow Joint-Grouu--Aquivvrinn Machines
thus
V (F-1 [ (FFU (F
which means FU F' E EVV). By the definition of A, 2 ((FFF)) ((F) + (FF)). On the other hand, (p() 2 A. So, j(F") O, but this implies f(E) 0a ea.e. E Im F. Therefore
V VCF1 [L)(
thus
V V= (1)()]
Theorem E.2 (Kobayashi & Oshima (2005, Theorem 2.10)). Let G Af"( = GL(m, R)) x Rm be the affine group. For any X E C, let be the representation of G on L (IR defined by
mg (x] == det A f(g x), x ZER". -9 (A,b) GL R) RR..
Additionaly, let स. be another representation of G I,2 (TRT) defined by
[g]h)(E) = e det Al A|1 E), EER,. .g (A, GL(mm, R) x I
so that F. intertwines π and R, namely F Fong = Rg o F for all E G. Suppose X E + iR, then a is an irreducible unitary representation of on (3()..
Proof. Note that æ is unitary iff À E 2 + i.. We identify the translation group T = RT as a subgroup of the affine group G Aff( by embedding b T to (I, E G. Suppose V of be a m-invariant closed subspace L (IR"). Then is also r-invariant because <(I, b) T(b). Hence by Lemma E.1, there exists a measurable subset of R satisfying V ===-1 [722 [ [F)] Further, the image F(V) 7.? (TR") is --invariant. Therefore by the definition of म, E has to be invariant (except measure zero subset) under the dual action of A GL(m,, I) given by
E A EER".
But this action is known be ergodic, that is, such measurable set is either RT Hence E R and V F-1 [E)) LRR"), which yields the irreducibility of æ.
19


Reviews:
[{"review_id": "5K8q4h8uu4", "reviewer": "Reviewer_RWHG", "summary": "The goal of the paper is to subscribe a general framework for proving universality type theorems for a generalized class of models, that the authors call joint-group-equivariant machines. Joint-group-equivariant machines are models consisting of a sequence of joint-group-equivariant features maps, maps from the joint input-parameter space to an output (feature) space that are group equivariant (jointly in the input and parameter space). Central to the proof is an interesting application of Schur’s lemma on irreducible representations, which concludes that a certain construction for a Ridgelet transform (which is introduced in the paper) is indeed one the “dual” which maps functions of a certain class to parameters of the joint-group-equivariant model.\n\nThe authors proceed to exemplify the proof for various particular types of models, such as “Depth-n Fully-Connected\nNetworks”, “ Depth-n Group Convolutional Networks” and a “Quadratic-form with Nonlinearity”.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> On times the paper uses abstract mathematical language to give alternative descriptions to constructions which may obfuscate the reading (for example some category-theoretical parallels, whose validity I cannot testify). However, I find them redundant/complementary to the actual message of the paper, and conditioned that they are true, they do not alter its story.\n\nThank you for the suggestion. We will retain the mathematically detailed supplementary explanation as an optional message intended for readers with a strong background in algebra or representation theory. Since it is strictly supplementary, skipping it will not affect the main narrative or the overall understanding of the paper.\n\n> The unification under the generalized model is theoretically appealing, however I am not sure if the generalized model (joint-group-equivariant machines) are any further useful. It is not clear to me what is the impact of the unification of mostly known results. Have we learnt something insightful about machine learning, or generalization, in the process of unifying universality theorems?\n\nThe main theorem of this work is not merely a unification of existing results; a key strength lies in its ability to systematically and mechanically assess universality even for novel architectures. For example, it enables the straightforward design of expressive networks with universal approximation capability, such as *quadratic networks*.\n\nIn the era of large language models, the diversity of proposed architectures has significantly increased. Manually analyzing the expressive power of each new model on a case-by-case basis is no longer feasible from a theoretical standpoint. Our framework addresses this challenge by offering a general principle.\n\nIn particular, algebraic tools such as *Schur's lemma* provide a notable advantage in that they allow us to assess universality (or density) even when the inputs and outputs are not vectors—something that traditional techniques often struggle with. While classical theorems like the Stone–Weierstrass or Hahn–Banach theorems offer generic tools for establishing universality, applying them to deep neural networks with hierarchical structures requires considerable ingenuity.\n\nBy contrast, proof techniques originating with Hornik et al (1989), which rely on repeatedly differentiating activation functions to construct polynomial approximations, often obscure the intuitive mechanism by which neural networks perform approximation. In comparison, our use of *joint group equivariance* and *ridgelet transforms* as inverse operators provides a more direct and interpretable understanding of the approximation process in deep networks.\n\nRegarding generalization: Theoretical analysis of generalization error is often based on estimates of the Rademacher complexity of the hypothesis class. However, in deep learning, the Rademacher complexity typically scales exponentially with network depth. This is at odds with practical observations, where deeper networks tend to generalize better.\n\nThis discrepancy arises from the limitations of current techniques for analyzing composite functions: the reliance on coarse Lipschitz estimates leads to overly pessimistic bounds. To address this gap, we revisited expressive power analysis and developed the *deep ridgelet transform* as a theoretical framework that can precisely handle compositions of functions.\n\n> The depth-N fully connected neural network described in Section 5 is different that what the community is used to and it looks more general. Does the provided formulation of fully-conntected NNs generalize a ReLU layer for example?\n\nYes, the presented example extends typical networks, and especially it includes the case when $\\sigma$ is ReLU.", "claims_and_evidence": "This paper is about a pure theoretical construction and universal approximation results on them, empirical evidence is not applicable.", "methods_and_evaluation_criteria": "N/A", "theoretical_claims": "The theoretical claims seem reasonable, the writing is easy to follow and introduces clearly the constructions about which the proofs is about. On times the paper uses abstract mathematical language to give alternative descriptions to constructions which may obfuscate the reading (for example some category-theoretical parallels, whose validity I cannot testify). However, I find them redundant/complementary to the actual message of the paper, and conditioned that they are true, they do not alter its story.", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "I am not up-to-date with the literature on this topic", "essential_references_not_discussed": "See above", "other_strengths_and_weaknesses": "The unification under the generalized model is theoretically appealing, however I am not sure if the generalized model (joint-group-equivariant machines) are any further useful. It is not clear to me what is the impact of the unification of mostly known results. Have we learnt something insightful about machine learning, or generalization, in the process of unifying universality theorems?", "other_comments_or_suggestions": "Small typo (double ‘the’) on L30, right column", "questions_for_authors": "The depth-N fully connected neural network described in Section 5 is different that what the community is used to and it looks more general. Does the provided formulation of fully-conntected NNs generalize a ReLU layer for example?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JBA30dXDEr", "reviewer": "Reviewer_ETza", "summary": "This paper introduces a framework for proving constructive universal approximation theorems for general classes of neural networks. By invoking ideas from representation theory, the paper generalizes the ridgelet transform to a larger class of models which satisfy \"joint G equivariance.\" As a consequence, the paper establishes universal approximation theorems for deep fully connected networks, deep group convolutional networks, and a depth 2 network on a quadratic form.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> This paper only handles the infinite width (i.e continuous) setting, and does not provide any quantitative results on how much width is needed for a certain architecture to approximate a target function, akin to the Barron norm in (Barron, 1993).\n\nIn our main result, the inverse operator for the integral representation is explicitly obtained. This allows us to apply the so-called Barron’s argument to evaluate the approximation error using the Maurey–Jones–Barron (MJB) bound. In general, applying the MJB bound yields an approximation error of $O(1/\\sqrt{n})$ when using a finite-width model with $n$ units.\n\n> The implications of this work (such as depth separations between different classes of architectures) is not clear, which limits the significance.\n\n(Please also refer to our fourth response for related discussion.)\n\nIn addition, algebraic methods such as Schur's lemma offer a significant advantage: they allow us to assess density (or universality) even when the input and output are not vectors. While there are several classical theorems—such as the Stone–Weierstrass and Hahn–Banach theorems—that provide general tools for determining universality, applying them to neural networks with deep hierarchical structures requires substantial effort and ingenuity.\n\nFor example, proof techniques that trace back to Hornik et al. (1989) often rely on repeatedly differentiating activation functions to construct polynomial approximations. While these techniques are mathematically valid, they tend to obscure the underlying mechanism by which neural networks perform approximation, and may not offer much intuitive understanding.\n\nIn contrast, conditions such as *joint group equivariance* and tools like the *ridgelet transform*, which serves as an inverse operator, provide a more direct and interpretable explanation of how neural networks approximate functions.\n\n> The paper could be improved by motivating the various definitions in Section 3 with a concrete example, such as the deep feedforward network.\n\nThank you for the suggestion. We will incorporate a concrete example, such as the deep fully connected network, immediately after the definition to improve clarity.\n\n> Additionally, the universal approximation result for deep feedforward networks in Section 5 is unsurprising, given the universal approximation guarantee for two-layer networks and the fact that deep networks are a strictly larger hypothesis class. Could the authors please comment further on this point?\n\nThank you for your request. The value of this work does not lie solely in the illustrative example presented in Section 5. Rather, it lies in providing a unified, simple, and constructive principle for analyzing the expressive power of a wide range of learning machines, as demonstrated in Sections 4 through 7.\n\nIn deep learning, diverse architectures are proposed on a daily basis. Traditionally, analyzing the expressive power of each new architecture required manual, case-by-case efforts by experts. This study identifies a key condition—*joint-group-equivariance*—that enables such analysis to be performed systematically. We believe this is where the main contribution of our work lies.", "claims_and_evidence": "The claims in this paper (i.e the main result Theorem 3.10, and the following examples in sections 4-7), are indeed supported by clear evidence.", "methods_and_evaluation_criteria": "This paper is theoretical, and so the \"methods\" (i.e theorems proven) do indeed make sense for the problem at hand.", "theoretical_claims": "The theoretical claims appear to be sound.", "experimental_designs_or_analyses": "N/A", "relation_to_broader_scientific_literature": "This paper unifies prior works on universal approximation of neural networks via the integral representation approach (i.e Barron, 1993), and generalizes prior works on closed form solutions for the ridgelet transform (Sonoda et al., 2021, 2022a, 2022b, 2024a, 2024b).", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n- The framework developed by this paper seems to be very general purpose, and can allow for constructive proofs of universal approximation theorems for other architectures in the future.\n- To the best of my knowledge, the contribution of this paper appears to be novel.\n\nWeaknesses:\n- This paper only handles the infinite width (i.e continuous) setting, and does not provide any quantitative results on how much width is needed for a certain architecture to approximate a target function, akin to the Barron norm in (Barron, 1993).\n- The implications of this work (such as depth separations between different classes of architectures) is not clear, which limits the significance.", "other_comments_or_suggestions": "- From a clarity perspective, the machinery developed in Section 3 is quite technical and at times difficult to follow (especially for readers less familiar with representation theory). The paper could be improved by motivating the various definitions in Section 3 with a concrete example, such as the deep feedforward network.", "questions_for_authors": "- Can the authors please address my comments in the weaknesses section above?\n- Additionally, the universal approximation result for deep feedforward networks in Section 5 is unsurprising, given the universal approximation guarantee for two-layer networks and the fact that deep networks are a strictly larger hypothesis class. Could the authors please comment further on this point?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8CvVUu6ECe", "reviewer": "Reviewer_9cKy", "summary": "The main result of this paper is Theorem 3.10, which provides a closed form formula for a ridgelet transform of learning machines with joint-group-equivariant maps. Previous works have been deriving the closed form formula for the ridgelet transforms of depth-2 networks. This paper has generalized these results to the wider class of neural networks that covers joint-equivariant machines, fully connected networks, group-convolutional networks, and a depth-2 network with quadratic forms.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The authors mention several times that this result unifies the universal approximation theorem. But it is not clear how Theorem 3.10 can imply the universal approximation theorem, say the one established by (Pinkus, 1999).\n\nThere is no unique definition of \"universality.\" In the context of machine learning, the term \"universality\" typically corresponds to what is referred to as \"density\" in standard mathematical terminology. Accordingly, there exist various types of universality depending on the choice of topology.\n\nFor example, the notion treated in Pinkus (1999) corresponds to $cc$-universality, or density in the topology of compact convergence, which is also known as the topology of uniform convergence on compact sets. In contrast, the type of universality we have demonstrated is $L^2$-universality, or density in the $L^2$-topology.\n\nThe relationships among various notions of universality commonly used in machine learning theory are discussed in detail in the following reference:\n\nSriperumbudur, Fukumizu, and Lanckriet. Universality, Characteristic Kernels and RKHS Embedding of Measures, JMLR, 2011.\n\nMoreover, it is generally known that once an $L^2$-universal integral representation is established, $cc$-universality of a finite-width model can be shown by employing quasi-Monte Carlo integration (more precisely, by applying the law of large numbers). Therefore, the claims of our work indeed imply $cc$-universality as well.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence. All theorems and lemmas are provided with complete proofs.", "methods_and_evaluation_criteria": "No method nor evaluation criteria is proposed in this paper.", "theoretical_claims": "I checked some of the main proofs but did not go through all of them in details. The proofs seem to be correct.", "experimental_designs_or_analyses": "No experiment is provided.", "relation_to_broader_scientific_literature": "This paper provides a closed form formula for a ridgelet transform of learning machines with joint-group-equivariant maps. This result is already a great generalization of previous results in the literature on ridgelet transforms of neural networks.", "essential_references_not_discussed": "Essential related works on ridgelet transforms and universal approximation theorems are included in the introduction section.", "other_strengths_and_weaknesses": "**Strengths**\n\n- The paper successfully provides a closed form formula for a ridgelet transform of learning machines with joint-group-equivariant maps, thus proposes a unified theoretical framework for understanding universal approximation in both deep and shallow networks, addressing a fundamental topic in machine learning.\n\n- The results are strongly supported by rigorously proofs, making a valuable theoretical contribution.\n\n- Several concrete examples are provided to demonstrate the applicability of the main theorem, enhancing its relevance to various machine learning architectures.\n\n**Weaknesses**\n\n- The paper relies on dense mathematical formalism, making it less accessible to a broader audience, especially those outside mathematical machine learning.\n\n- The paper has no numerical experiments or empirical validation as it focuses entirely on theoretical derivations. Personally, I think this paper is more suitable for mathematics community than machine learning community. \n\n- The authors mention several times that this result unifies the universal approximation theorem. But it is not clear how Theorem 3.10 can imply the universal approximation theorem, say the one established by (Pinkus, 1999).\n\nPinkus, Allan (January 1999). \"Approximation theory of the MLP model in neural networks\". Acta Numerica. 8: 143–195.", "other_comments_or_suggestions": "See weaknesses.", "questions_for_authors": "How does Theorem 3.10 imply the traditional universality approximation theorem in (Pinkus, 1999)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GT8jfz0XbW", "reviewer": "Reviewer_Jwhk", "summary": "The paper studies so called \"joint-equivariant networks\", which are neural networks that are simultaneously equivariant with respect to a group action on the input data as well as the parameter space. This can be viewed as a generalization of previously studied group-equivariant neural network, in the sense that the joint-equivariant networks reduces to the aforementioned networks upon taking a trivial group action on the parameter space. This framework also includes standard fully connected neural networks, which are not equivariant. The main result is a universality theorem for the joint-equivariant that unifies certain previous universality results in the literature.\n\n## update after rebuttal\nI stand by my original score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> How are your depth-n G-convolutional layers related to the group convolutional layers by Cohen et al?\n\nThank you for your question regarding concrete examples. We fully acknowledge the pioneering contributions of Cohen and colleagues in the development of group-equivariant neural networks.\n\nThe focus of this study lies in universality and a general geometric perspective. Therefore, in this paper, we chose to highlight a single representative example with a particular emphasis on differential geometric aspects, specifically the 2019 paper.\n\nThe relationship between the ridgelet transform of group-convolution networks and existing works—especially in the context of universal approximation theorems—is thoroughly discussed in Sections 5 and 6 of Sonoda et al. (2022a). We kindly refer the reviewer and future readers to these sections for further details.\n\nThe three papers suggested by the reviewer do not explicitly analyze the universality or expressive power of the proposed networks. The group convolutions in Group Equivariant Convolutional Networks and Steerable CNNs correspond to group convolutions where a finite group $G$ acts on $X = \\mathbb{R}^{n \\times n \\times k}$ for some $n$ and $k$, and thus their universality follows as a corollary of Theorem 6.1. Similarly, the convolution in Spherical CNNs corresponds to the case where compact group $G = SO(3)$ acts on $X = \\mathrm{SO}(3)$ (or homogeneous space $S^2$), and thus the universality can also be shown in the sense of Theorem 6.1.", "claims_and_evidence": "Yes, see below.", "methods_and_evaluation_criteria": "NA", "theoretical_claims": "Yes, it is my opinion that the claims in the paper are supported by convincing evidence. The theorems are proven rigorously and the mathematical framework is consistent. \n\nIt is worth mentioning that the same group of authors have written several papers on this topic before. Although the papers address related points the present paper represents sufficiently novel results to warrant publication on its own.", "experimental_designs_or_analyses": "NA", "relation_to_broader_scientific_literature": "See below", "essential_references_not_discussed": "It strikes me as strange that the authors do not cite the original references for group equivariant CNNs by Cohen et al. They mention one paper from 2019 by Cohen, Geiger and Weiler which provides some of the underlying mathematical structure of GCNNs. However, there are several papers before that introducing the key structure of GCNNS, such as\n\nGroup Equivariant Convolutional Networks - Cohen and Welling\nhttps://arxiv.org/abs/1602.07576\n\n\nSteerable CNNs - Cohen and Welling\nhttps://arxiv.org/abs/1612.08498\n\n\nSpherical CNNs - Cohen, Geiger, Koehler and Welling\nhttps://arxiv.org/abs/1801.10130", "other_strengths_and_weaknesses": "In section 6 they discuss G-convolutional networks of Depth n. However, nowhere do they cite the original papers on group convolutional layers mentioned above. They also do not discuss how their result relate to these previous results. I would certainly have liked to see a detailed discussion of how their GCN-layer is related to previously constructed group equivariant layers by Cohen et al. I think this is a weakness of the paper.", "other_comments_or_suggestions": "To reiterate the comment above: How are your depth-n G-convolutional layers related to the group convolutional layers by Cohen et al?", "questions_for_authors": "See above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviews are all positive. The method proposes a novel and more general architecture to leverage symmetries, as supported by a complete and rigorous theoretical analysis. I am in favor of accepting this paper.
Decision: Accept (poster)