Unified Universality Theorem for Deep and Shallow Joint-Group-Equivariant Machines

## Abstract

Abstract We present a constructive universal approximation theorem for learning machines equipped with joint-group-equivariant feature maps, based on the group representation theory. “Constructive” here indicates that the distribution of parameters is given in a closed-form expression known as the ridgelet transform. Joint-group-equivariance encompasses a broad class of feature maps that generalize classical group-equivariance. Notably, this class includes fully-connected networks, which are not group-equivariant but are joint-group-equivariant. Moreover, our main theorem also unifies the universal approximation theorems for both shallow and deep networks. While the universality of shallow networks has been investigated in a unified manner by the ridgelet transform, the universality of deep networks has been investigated in a case-by-case manner.

## 1 Introduction

The proof of a universality theorem contains hints for understanding the internal data processing mechanisms inside learning machines such as neural networks. For example, the first universality theorem*s* for depth-2 neural networks were shown in 1989 with *four* different proofs by Cybenko (), Hornik et al. (), Funahashi (), and Carroll and Dickinson (). Among them, Cybenko’s proof using Hahn-Banach and Hornik et al.’s proof using Stone-Weierstrass are existential proofs, meaning that it is not clear how to assign the parameters. On the other hand, Funahashi’s proof reducing to the Fourier transform and Carroll and Dickinson’s proof reducing to the Radon transform are constructive proofs, meaning that it is clear how to assign the parameters. The latter constructive methods, which reduce to integral transforms, were refined as the so-called integral representation by Barron () and further culminated as the *ridgelet transform*, the main objective of this study, discovered by Murata () and Candès ().

To show the universality in a constructive manner, we formulate the the problem as a functional equation:
Let $\mathtt{LM}[\gamma]$ denote a certain learning machine (such as a deep network) with parameter $\gamma$, and let $\mathcal{F}$ denote a class of functions to be expressed by the learning machine. Given a function $f\in\mathcal{F}$, find an unknown parameter $\gamma$ so that the machine $\mathtt{LM}[\gamma]$ represents function $f$, i.e.

$$ $\displaystyle\mathtt{LM}[\gamma]=f,$ (1) $$

which we call a *learning equation*.
This equation is understood as a stronger formulation of learning than an ordinary formulation by the empirical risk minimization such as minimizing $\sum_{i=1}^{n}|\mathtt{LM}[\gamma](x_{i})-f(x_{i})|^{2}$ with respect to $\gamma$, as the latter is understood as a weak form (or a variational form) of this equation. Therefore, characterizing the solution space of this equation leads to understanding the parameters obtained by risk minimization. Following previous studies (Murata, ; Candès, ; Sonoda et al., , , , ), we call a solution operator $\mathtt{R}$ that satisfies $\mathtt{LM}[\mathtt{R}[f]]=f$ a *ridgelet transform*.
Once such a solution operator $\mathtt{R}$ is found, we can conclude a *universality* of the learning machine in consideration because the reconstruction formula $\mathtt{LM}[\mathtt{R}[f]]=f$ implies for any $f\in\mathcal{F}$ there exists a machine that represents $f$.
In particular, when $\mathtt{R}[f]$ is found in a closed-form manner, then it leads to a *constructive* proof of the universality since $\mathtt{R}[f]$ could indicate how to assign parameters.

For depth-2 neural networks (particularly with an infinitely-wide hidden layer), the equation has been solved with several closed-form ridgelet transforms. For example, the closed-form ridgelet transforms have been obtained for depth-2 fully-connected layers (Sonoda et al., ), depth-2 fully-connected layers on manifolds (Sonoda et al., ), depth-2 group convolution layers (Sonoda et al., ), and depth-2 fully-connected layers on finite fields (Yamasaki et al., ). The essential technique to obtain these ridgelet transforms are to construct a Fourier expression corresponding to the network in consideration. We refer to Sonoda et al. () for more tecnnical backgrounds behind these results. Furthermore, Sonoda et al. () have revealed that the distribution of parameters inside depth-2 fully-connected networks obtained by regularized empirical risk minimization assymptotically converges to the ridgelet transform. In other words, the ridgelet transform can also explain the solutions obtained by risk minimization.

On the other hand, for depth-$n$ neural networks, the equation is far from solved, and it is common to either consider infinitely-deep mathematical models such as Neural ODEs (Sonoda and Murata, ; E, ; Li and Hao, ; Haber and Ruthotto, ; Chen et al., ), or handcraft solutions depending on the network specifications. For example, construction methods such as the so-called Telgarsky sawtooth function (or the Yarotsky scheme) and bit extraction techniques
(Cohen et al., ; Telgarsky, ; Yarotsky, , ; Yarotsky and Zhevnerchuk, ; Daubechies et al., ; Cohen et al., ; Siegel, ; Petrova and Wojtaszczyk, ; Grohs et al., )
have been developed (not only to investigate the expressivity but also) to demonstrate the depth separation, super-convergence, and minmax optimality of deep ReLU networks. Various feature maps have also been handcrafted in the contexts of
geometric deep learning (Bronstein et al., ) and deep narrow networks (Lu et al., ; Hanin and Sellke, ; Lin and Jegelka, ; Kidger and Lyons, ; Park et al., ; Li et al., ; Cai, ; Kim et al., ). However, for the purpose of understanding the parameters obtained by risk minimization (in a manner presented by Sonoda et al. ()), these results are less satisfactory because there is no guarantee that these handcrafted solutions are obtaiend by risk minimization.

Recently, Sonoda et al. () developed a novel technique to show the universality based on the group representation theory, and discovered a rich class of ridgelet transforms for learning machines with joint-group-*invariant* feature maps. However, their technique was essentially limited to depth-2 networks and could not cover depth-$n$ networks defined by composites of nonlinear activation functions such as $\sigma(A_{2}\sigma(A_{1}\bm{x}-\bm{b}_{1})-\bm{b}_{2})$. By carefully reviewing their group theoretic arguments, we found that the joint-*invariance* is the bottleneck and it can be resolved by relaxing the assumption to the joint-*equivariance*. In this study, we present a wider class of ridgelet transforms for learning machines with joint-*equivariant* feature maps so to cover the depth-$n$ (as well as depth-2) fully-connected networks.

The contributions of this study include

- •
We derived the ridgelet transform (solution operator for the learning equation) for learning machines with joint-group-equivariant feature maps. Since the solution of the learning equation can be written in closed form for any $f\in\mathcal{F}$, it is a constructive and unified proof of the universal approximation theorem for joint-group-equivariant machines.
- •
As a corollary, we have shown the constructive universal approximation property of deep fully-connected neural networks. Until this study, the universality of deep networks has been shown in a different manner from the universality of shallow networks, but our results discuss them on common ground. Now we can understand the approximation schemes of various learning machines in a unified manner.
- •
In addition, as an example of a learning machine whose universality has not been known, we presented a network with quadratic forms and showed its universality.

## 2 Preliminaries

We quickly overview the original integral representation and the ridgelet transform, a mathematical model of depth-2 fully-connected network and its right inverse. Then, we list a few facts in the group representation theory.
In particular, *Schur’s lemma* and the *Haar measure* play key roles in the proof of the main results.

##### Notation.

For any topological space $X$, $C_{c}(X)$ denotes the Banach space of all compactly supported continuous functions on $X$.
For any measure space $X$, $L^{p}(X)$ denotes the Banach space of all $p$-integrable functions on $X$.
$\mathcal{S}(\mathbb{R}^{d})$ and $\mathcal{S}^{\prime}(\mathbb{R}^{d})$ denote the classes of rapidly decreasing functions (or Schwartz test functions) and tempered distributions on $\mathbb{R}^{d}$, respectively.

### 2.1 Integral Representation and Ridgelet Transform for Depth-2 Fully-Connected Network

###### Definition 1 .

For any measurable functions $\sigma:\mathbb{R}\to\mathbb{C}$ and $\gamma:\mathbb{R}^{m}\times\mathbb{R}\to\mathbb{C}$, put

$$ $\displaystyle S_{\sigma}[\gamma](\bm{x}):=\int_{\mathbb{R}^{m}\times\mathbb{R} }\gamma(\bm{a},b)\sigma(\bm{a}\cdot\bm{x}-b)\mathrm{d}\bm{a}\mathrm{d}b,\quad \bm{x}\in\mathbb{R}^{m}.$ (2) $$

We call
$S_{\sigma}[\gamma]$ an (integral representation of) neural network, and $\gamma$ a parameter distribution.

The integration over all the hidden parameters $(\bm{a},b)\in\mathbb{R}^{m}\times\mathbb{R}$ means all the neurons $\{\bm{x}\mapsto\sigma(\bm{a}\cdot\bm{x}-b)\mid(\bm{a},b)\in\mathbb{R}^{m}
\times\mathbb{R}\}$ are summed (or integrated, to be precise) with weight $\gamma$, hence formally $S_{\sigma}[\gamma]$ is understood as a continuous neural network with a single hidden layer.
We note, however, when $\gamma$ is a finite sum of point measures such as $\gamma_{p}=\sum_{i=1}^{p}c_{i}\delta_{(\bm{a}_{i},b_{i})}$ (by appropriately extending the class of $\gamma$ to Borel measures), then it can also reproduce a finite width network

$$ $S_{\sigma}[\gamma_{p}](\bm{x})=\sum_{i=1}^{p}c_{i}\sigma(\bm{a}_{i}\cdot\bm{x} -b_{i}).$ (3) $$

In other words, the integral representation is a mathematical model of depth-2 network with *any* width (ranging from finite to continuous).

Next, we introduce the ridgelet transform, which is known to be a right-inverse operator to $S_{\sigma}$.

###### Definition 2 .

For any measurable functions $\rho:\mathbb{R}\to\mathbb{C}$ and $f:\mathbb{R}^{m}\to\mathbb{C}$, put

$$ $\displaystyle R_{\rho}[f](\bm{a},b):=\int_{\mathbb{R}^{m}}f(\bm{x})\overline{ \rho(\bm{a}\cdot\bm{x}-b)}\mathrm{d}\bm{x},\quad(\bm{a},b)\in\mathbb{R}^{m} \times\mathbb{R}.$ (4) $$

We call $R_{\rho}$ a ridgelet transform.

To be precise, it satisfies the following reconstruction formula.

###### Theorem 1 (Reconstruction Formula) .

Suppose $\sigma$ and $\rho$ are a tempered distribution ($\mathcal{S}^{\prime}$) and a rapid decreasing function ($\mathcal{S}$) respectively.
There exists a bilinear form $(\!(\sigma,\rho)\!)$ such that

$$ $\displaystyle S_{\sigma}\circ R_{\rho}[f]=(\!(\sigma,\rho)\!)f,$ (5) $$

for any square integrable function $f\in L^{2}(\mathbb{R}^{m})$. Further, the bilinear form is given by
$(\!(\sigma,\rho)\!)=\int_{\mathbb{R}}\sigma^{\sharp}(\omega)\overline{\rho^{
\sharp}(\omega)}|\omega|^{-m}\mathrm{d}\omega,$
where $\sharp$ denotes the 1-dimensional Fourier transform.

See Sonoda et al. (, Theorem 6) for the proof.
In particular, according to Sonoda et al. (, Lemma 9), for any activation function $\sigma$, there always exists $\rho$ satisfying $(\!(\sigma,\rho)\!)=1$.
Here, $\sigma$ being a tempered distribution means that typical activation functions are covered such as ReLU, step function, $\tanh$, gaussian, etc…
We can interpret the reconstruction formula as a universality theorem of continuous neural networks, since for any given data generating function $f$, a network with output weight $\gamma_{f}=R_{\rho}[f]$ reproduces $f$ (up to factor $(\!(\sigma,\rho)\!)$), i.e. $S[\gamma_{f}]=f$. In other words, the ridgelet transform indicates how the network parameters should be organized so that the network represents an individual function $f$.

The original ridgelet transform was discovered by Murata () and Candès (). It is recently extended to a few modern networks by the Fourier slice method (see e.g. Sonoda et al., ).
In this study, we present a systematic scheme to find the ridgelet transform for a variety of given network architecture based on the group theoretic arguments.

###### Definition 1 .

###### Definition 2 .

###### Theorem 1 (Reconstruction Formula) .

### 2.2 Irreducible Unitary Representation and Schur’s Lemma

In the main theorem, we use *Schur’s lemma*, a fundamental theorem from unitary group representation theory. Group representation is a method for investigating properties of an abstract group $G$ by mapping $G$ to another (much computable) group of invertible linear operators. We refer to Folland () for more details on group representation and harmonic analysis on groups.

In this study, we assume group $G$ to be *locally compact*. This is a sufficient condition for having invariant measures. It is not a strong assumption. For example, any finite group, discrete group, compact group, and finite-dimensional Lie group are locally compact, while an infinite-dimensional Lie group is *not* locally compact.

Let $\mathcal{H}$ be a nonzero Hilbert space, and
$\mathcal{U}(\mathcal{H})$ be the group of unitary operators on $\mathcal{H}$. A *unitary representation* $\pi$ of $G$ on $\mathcal{H}$ is a group homomorphism that is continuous with respect to the strong operator topology—that is,
a map $\pi:G\to\mathcal{U}(\mathcal{H})$ satisfying $\pi_{gh}=\pi_{g}\pi_{h}$ and $\pi_{g^{-1}}=\pi_{g}^{-1}$, and for any $\psi\in\mathcal{H}$, the map $G\ni g\mapsto\pi_{g}[\psi]\in\mathcal{H}$ is continuous.

Suppose $\mathcal{M}$ is a closed subspace of $\mathcal{H}$. $\mathcal{M}$ is called an *invariant* subspace when $\pi_{g}[\mathcal{M}]\subset\mathcal{M}$ for all $g\in G$. Particularly, $\pi$ is called *irreducible* when it does not admit any nontrivial invariant subspace $\mathcal{M}\neq\{0\}$ nor $\mathcal{H}$.
The following theorem is a fundamental result of group representation theory that characterizes the irreducibility.

###### Theorem 2 (Schur’s lemma) .

A unitary representation $\pi:G\to\mathcal{U}(\mathcal{H})$ is irreducible iff
any bounded operator $T$ on $\mathcal{H}$ that commutes with $\pi$ is always a constant multiple of the identity. In other words, if $\pi_{g}\circ T=T\circ\pi_{g}$ for all $g\in G$, then $T=c\operatorname{Id}_{\mathcal{H}}$ for some $c\in\mathbb{C}$.

See Folland (, Theorem 3.5(a)) for the proof. We use this as a key step in the proof of our main theorem.

As a concrete example of an irreducible representation, we use the following regular representation of the affine group $\operatorname{Aff}(m)$ on $L^{2}(\mathbb{R}^{m})$.

###### Theorem 3 .

Let $G:=\operatorname{Aff}(m):=GL(m)\ltimes\mathbb{R}^{m}$ be the affine group acting on $X=\mathbb{R}^{m}$ by $(L,\bm{t})\cdot\bm{x}=L\bm{x}+\bm{t}$, and let $\mathcal{H}:=L^{2}(\mathbb{R}^{m})$ be the Hilbert space of square-integrable functions.
Let $\pi:\operatorname{Aff}(m)\to\mathcal{U}(L^{2}(\mathbb{R}^{m}))$ be the regular representation of the affine group $\operatorname{Aff}(m)$ on $L^{2}(\mathbb{R}^{m})$, namely $\pi_{g}[f](\bm{x}):=|\det L|^{-1/2}f(L^{-1}(\bm{x}-\bm{t}))$ for any $g=(L,\bm{t})\in G$. Then $\pi$ is irreducible.

See Folland (, Theorem 6.42) for the proof.

###### Theorem 2 (Schur’s lemma) .

###### Theorem 3 .

### 2.3 Calculus on Locally Compact Group

By Haar’s theorem, if $G$ is a locally compact group, then there uniquely exist left and right invariant measures $\mathrm{d}_{l}g$ and $\mathrm{d}_{r}g$, satisfying for any $s\in G$ and $f\in C_{c}(G)$,

$$ $\displaystyle\int_{G}f(sg)\mathrm{d}_{l}g=\int_{G}f(g)\mathrm{d}_{l}g,\quad \mbox{and}\quad\int_{G}f(gs)\mathrm{d}_{r}g=\int_{G}f(g)\mathrm{d}_{r}g.$ $$

Let $X$ be a $G$-space with transitive left (resp. right) $G$-action $g\cdot x$ (resp. $x\cdot g$) for any $(g,x)\in G\times X$.
Then, we can further induce the left (resp. right) invariant measure $\mathrm{d}_{l}x$ (resp. $\mathrm{d}_{r}x$) so that for any $f\in C_{c}(G)$,

$$ $\displaystyle\int_{X}f(x)\mathrm{d}_{l}x:=\int_{G}f(g\cdot o)\mathrm{d}_{l}g, \quad\mbox{resp.}\quad\int_{X}f(x)\mathrm{d}_{r}x:=\int_{G}f(o\cdot g)\mathrm{ d}_{r}g,$ $$

where $o\in X$ is a fixed point called the origin.

## 3 Main Results

We introduce the joint-group-equivariant feature map $\phi:X\times\Xi\to Y$,
the learning machine $\mathtt{LM}[\gamma;\phi]:X\to Y$ defined by joint-group-equivariant feature maps, called the joint-equivariant machine for short, and present the ridgelet transform $\mathtt{R}[f;\psi]:\Xi\to\mathbb{C}$ for joint-equivariant machines, yielding the universality $\mathtt{LM}[\mathtt{R}[f;\psi];\phi]=c_{\phi,\psi}f$.

Let $G$ be a locally compact group equipped with a left invariant measure $\mathrm{d}g$.
Let $X$ and $\Xi$ be $G$-spaces equipped with $G$-invariant measures $\mathrm{d}x$ and $\mathrm{d}\xi$, called the *data domain* and the *parameter domain*. respectively. Let $Y$ be a separable Hilbert space, called the *output domain*.
Let $\mathcal{U}(Y)$ be the space of unitary operators on $Y$, and let $\upsilon:G\to\mathcal{U}(Y)$ be a unitary representation of $G$ on $Y$.
We call a $Y$-valued map $\phi$ on the data-parameter domain $X\times\Xi$, i.e. $\phi:X\times\Xi\to Y$, a *feature map*.

If there is no risk of confusion, we use the same symbol $\cdot$ for the $G$-actions on $X$, $Y$, and $\Xi$ (e.g., $g\cdot x$, $g\cdot y$, and $g\cdot\xi$).
On the other hand, to avoid the confusion between $G$-actions on output domain $Y$ and $Y$-valued function $f:X\to Y$, both “$g\cdot f(x)$” and “$\upsilon_{g}[f(x)]$” (if needed) always imply $G$-action on $Y$, and “$\pi_{g}[f](x)$” (introduced soon below) for $G$-actions on $f:X\to Y$.

Let $L^{2}(X;Y)$ denote the space of $Y$-valued square-integrable functions on $X$ equipped with the inner product $\langle\phi,\psi\rangle_{L^{2}(X;Y)}:=\int_{X}\langle\phi(x),\psi(x)\rangle_{Y
}\mathrm{d}x$.
In the main theorem, the irreducibility of the following unitary representation $\pi:G\to\mathcal{U}(L^{2}(X;Y))$ will be a sufficient condition for the universality.
Put

$$ $\displaystyle\pi_{g}[f](x):=\upsilon_{g}[f(g^{-1}\cdot x)]=g\cdot f(g^{-1} \cdot x),\quad x\in X,\ f\in L^{2}(X;Y),\ g\in G.$ (6) $$

###### Lemma 1 .

$\pi$ is a unitary representation of $G$ on $L^{2}(X;Y)$.

###### Proof.

In fact,

$$ $\displaystyle\pi_{g}[\pi_{h}[f]](x)=g\cdot(h\cdot f(h^{-1}\cdot(g^{-1}\cdot x) ))=(gh)\cdot f((gh)^{-1}\cdot x)=\pi_{gh}[f](x),$ $$

and

$$ $\displaystyle\langle\pi_{g}[f_{1}],\pi_{g}[f_{2}]\rangle_{L^{2}(X;Y)}$ $\displaystyle=\int_{X}\langle\upsilon_{g}[f_{1}(g^{-1}\cdot x)],\upsilon_{g}[f _{2}(g^{-1}\cdot x)]\rangle_{Y}\mathrm{d}x$ $\displaystyle=\int_{X}\langle f_{1}(x),\upsilon_{g}^{*}[\upsilon_{g}[f_{2}(x)] ]\rangle_{Y}\mathrm{d}x=\langle f_{1},f_{2}\rangle_{L^{2}(X;Y)}.\qed$ $$

In addition, let $L^{2}(\Xi)$ denote the space of $\mathbb{C}$-valued square-integrable functions on $\Xi$, and let $\widehat{\pi}$ be the left-regular representation of $G$ on $L^{2}(\Xi)$ given by

$$ $\displaystyle\widehat{\pi}_{g}[\gamma](\xi):=\gamma(g^{-1}\cdot\xi),\quad\xi \in\Xi,\ \gamma\in L^{2}(\Xi),\ g\in G.$ (7) $$

Similarly to $\pi$, $\widehat{\pi}$ is also a unitary representation of $G$ on $L^{2}(\Xi)$.

###### Lemma 1 .

###### Proof.

### 3.1 Joint-Equivariant Feature Map

###### Definition 3 (Joint- $G$ -Equivariant Feature Map) .

We say a feature map $\phi:X\times\Xi\to Y$ is *joint-$G$-equivariant* when

$$ $\displaystyle\phi(g\cdot x,g\cdot\xi)=g\cdot\phi(x,\xi),\quad(x,\xi)\in X \times\Xi,$ (8) $$

holds for all $g\in G$.
In other words, $\phi$ is a homomorphism (or $G$-map) of $G$-sets from $X\times\Xi$ to $Y$. So
by $\hom_{G}(X\times\Xi,Y)$, we denote the collection of all joint-$G$-equivariant maps.
Additionally, when $G$-action on $Y$ is trivial, i.e. $\phi(g\cdot x,g\cdot\xi)=\phi(x,\xi)$, we say it is *joint-$G$-invariant*.

###### Remark 1 .

The joint-$G$-equivariance extends an ordinary notion of *$G$-equivariance*, i.e. $\phi(g\cdot x,\xi)=g\cdot\phi(x,\xi)$. In fact, $G$-equivariance is a special case of joint-$G$-equivariance where $G$ acts trivially on parameter domain, i.e. $g\cdot\xi=\xi$ (see also Figure 1).

Figure: Figure 1: An ordinary $G$-equivariant feature map $\phi:X\times\Xi\to Y$ is a subclass of joint-$G$-equivariant map where the $G$-action on parameter domain $\Xi$ is *trivial*, i.e. $g\cdot\xi=\xi$
Refer to caption: x1.png

In order to construct a (non-joint) group-equivariant network, we must carefully and precisely design the network architecture (see, e.g., a textbook of geometric deep learning Bronstein et al., ). On the other hand, we can easily and systematically construct joint-$G$-equivariant network from (not at all equivariant but) *any* map $f:X\to Y$ as performed in Lemmas 2, 3 and 4 (explained later).

###### Lemma 2 .

Let $X$ and $Y$ be $G$-sets.
Fix an arbitrary map $\phi_{0}:X\to Y$, and put $\phi(x,g):=\pi_{g}[\phi_{0}](x)=g\cdot\phi_{0}(g^{-1}\cdot x)$ for every $x\in X$ and $g\in G$. Then, $\phi:X\times G\to Y$ is joint-$G$-equivariant.

###### Proof.

Straightforward. For any $g\in G$, $\phi(g\cdot x,g\cdot h)=(gh)\cdot\phi_{0}((gh)^{-1}\cdot(g\cdot x))=g\cdot\phi
(x,h)$.
∎

We note that since $G$-set is a representation of $G$, the above construction $\phi:G\times G\to G$ (with $X=Y=G$) is understood as a primitive type of joint-$G$-equivariant maps $\phi:X\times\Xi\to Y$.

###### Lemma 3 (Depth- $n$ Joint-Equivariant Feature Map $\phi_{1:n}$ ) .

Given a sequence of joint-$G$-equivariant feature maps $\phi_{i}:X_{i-1}\times\Xi_{i}\to X_{i}\ (i=1,\ldots,n)$, put $\phi_{1:n}:X_{0}\times\Xi_{1}\times\cdots\times\Xi_{n}\to X_{n}$ by

$$ $\displaystyle\phi_{1:n}(x,\xi_{1},\ldots,\xi_{n}):=\phi_{n}(\bullet,\xi_{n}) \circ\cdots\circ\phi_{1}(x,\xi_{1}).$ (9) $$

Then, $\phi_{1:n}$ is joint-$G$-equivariant.
Following the custom of counting the number of parameter domains $(\Xi_{i})_{i=1}^{n}$, we say $\phi_{1:n}$ is depth-$n$.

###### Proof.

In fact,

$$ $\displaystyle\phi_{1:n}(g\cdot x,g\cdot\xi_{1},\ldots,g\cdot\xi_{n})$ $\displaystyle=\phi_{n}(\bullet,g\cdot\xi_{n})\circ\cdots\circ\phi_{2}(\bullet, g\cdot\xi_{2})\circ\phi_{1}(g\cdot x,g\cdot\xi_{1})$ $\displaystyle=\phi_{n}(\bullet,g\cdot\xi_{n})\circ\cdots\circ\phi_{2}({\color[ rgb]{0,0,1}g{\ \cdot\ }}\bullet,g\cdot\xi_{2})\circ\phi_{1}(x,\xi_{1})$ $\displaystyle\phantom{==}\vdots$ $\displaystyle=\phi_{n}({\color[rgb]{0,0,1}g{\ \cdot\ }}\bullet,g\cdot\xi_{n}) \circ\cdots\circ\phi_{2}(\bullet,\xi_{2})\circ\phi_{1}(x,\xi_{1})$ $\displaystyle={\color[rgb]{0,0,1}g{\ \cdot\ }}\phi_{n}(\bullet,\xi_{n})\circ \cdots\circ\phi_{2}(\bullet,\xi_{2})\circ\phi_{1}(x,\xi_{1})$ $\displaystyle=g\cdot\phi_{1:n}(x,\xi_{1},\ldots,\xi_{n}).\qed$ $$

###### Definition 3 (Joint- G 𝐺 G italic_G -Equivariant Feature Map) .

###### Remark 1 .

###### Lemma 2 .

###### Proof.

###### Lemma 3 (Depth- n 𝑛 n italic_n Joint-Equivariant Feature Map ϕ 1 : n subscript italic-ϕ : 1 𝑛 \phi_{1:n} italic_ϕ start_POSTSUBSCRIPT 1 : italic_n end_POSTSUBSCRIPT ) .

###### Proof.

### 3.2 Joint-Equivariant Machine and Ridgelet Transform

###### Definition 4 (Joint-Equivariant Machine) .

Fix an arbitrary $Y$-valued joint-equivariant feature map $\phi:X\times\Xi\to Y$. For any scalar-valued measurable function $\gamma:\Xi\to\mathbb{C}$, define a $Y$-valued map on $X$ by

$$ $\displaystyle\mathtt{LM}[\gamma;\phi](x):=\int_{\Xi}\gamma(\xi)\phi(x,\xi) \mathrm{d}\xi,\quad x\in X,$ (10) $$

where the integral is understood as the Bocher integral.
We also write $\mathtt{LM}_{\phi}:=\mathtt{LM}[\bullet;\phi]$ for short.
If needed, we call the image $\mathtt{LM}[\gamma;\phi]:X\to Y$ a joint-equivariant *machine*,
and the integral transform $\mathtt{LM}[\bullet;\phi]$ of $\gamma$ a joint-equivariant *transform*.

The joint-equivariant machine extends the original integral representation. In particular, it inherits the concept of integrating all the possible parameters $\xi$ and indirectly select which parameters to use by weighting on them, which *linearize* parametrization by lifting nonlinear parameters $\xi$ to linear parameter $\gamma$.

We use the following fact in the proof of the main theorem.

###### Lemma 4 .

A joint-$G$-equivariant machine is joint-$G$-equivariant.

###### Proof.

Recall that $G$ acts on $\gamma$ by $\widehat{\pi}$. Thus

$$ $\displaystyle\mathtt{LM}_{\phi}[\widehat{\pi}_{g}[\gamma]](g\cdot x)=\int_{\Xi }\gamma(g^{-1}\cdot\xi)\phi(g\cdot x,\xi)\mathrm{d}\xi=\int_{\Xi}\gamma(\xi) \phi(g\cdot x,g\cdot\xi)\mathrm{d}\xi=g\cdot\mathtt{LM}_{\phi}[\gamma](x).$ $$

∎

###### Definition 5 (Ridgelet Transform for Joint-Equivariant Machine) .

For any $Y$-valued joint-equivariant feature map $\psi:X\times\Xi\to Y$ and $Y$-valued Borel measurable function $f$ on $X$, put a scalar-valued integral transform

$$ $\displaystyle\mathtt{R}[f;\psi](\xi):=\int_{X}\langle f(x),\psi(x,\xi)\rangle_ {Y}\mathrm{d}x,\quad\xi\in\Xi.$ (11) $$

We also write $\mathtt{R}_{\psi}:=\mathtt{R}[\bullet;\psi]$ for short.
If there is no risk of confusion, we call both the image $\mathtt{R}[f;\psi]:X\to Y$ and the integral transform $\mathtt{R}[\bullet;\psi]$ of $f$ a ridgelet transform.

Similar to the joint-equivariant machine, the ridgelet transform is again joint-$G$-*invariant*. In fact,

$$ $\displaystyle\mathtt{R}_{\psi}[\pi_{g}[f]](g\cdot\xi)=\int_{X}\langle\upsilon_ {g}[f(g^{-1}\cdot x)],\psi(x,g\cdot\xi)\rangle_{Y}\mathrm{d}\xi=\int_{X} \langle f(x),\upsilon_{g}^{*}[\psi(g\cdot x,g\cdot\xi)]\rangle_{Y}\mathrm{d} \xi=\mathtt{R}_{\psi}[f](\xi).$ $$

Furthermore, as long as the integrals are convergent, the ridgelet transform is the dual operator of the joint-equivariant transform (with common $\phi$), since

$$ $\displaystyle\langle\gamma,\mathtt{R}[f;\phi]\rangle_{L^{2}(\Xi)}=\int_{X \times\Xi}\gamma(\xi)\langle\phi(x,\xi),f(x)\rangle_{Y}\mathrm{d}x\mathrm{d} \xi=\langle\mathtt{LM}[\gamma;\phi],f\rangle_{L^{2}(X;Y)}.$ (12) $$

###### Definition 4 (Joint-Equivariant Machine) .

###### Lemma 4 .

###### Proof.

###### Definition 5 (Ridgelet Transform for Joint-Equivariant Machine) .

### 3.3 Main Theorem

###### Theorem 4 (Reconstruction Formula) .

Assume (1) $Y$-valued feature maps $\phi,\psi:X\times\Xi\to Y$ are joint-$G$-equivariant, (2) composite operator $\mathtt{LM}_{\phi}\circ\mathtt{R}_{\psi}:L^{2}(X;Y)\to L^{2}(X;Y)$ is bounded (i.e., Lipschitz continuous), and (3) the unitary representation $\pi:G\to\mathcal{U}(L^{2}(X;Y))$ defined in (6) is irreducible.
Then, there exists a bilinear form $(\!(\phi,\psi)\!)\in\mathbb{C}$ (independent of $f$) such that for any $Y$-valued square-integrable function $f\in L^{2}(X;Y)$,

$$ $\displaystyle\mathtt{LM}_{\phi}\circ\mathtt{R}_{\psi}[f]=(\!(\phi,\psi)\!)f.$ (13) $$

In other words, the ridgelet transform $\mathtt{R}_{\psi}$ is a right inverse operator of joint-equivariant transform $\mathtt{LM}_{\phi}$ as long as $(\!(\phi,\psi)\!)\neq 0,\infty$.

###### Proof.

By using the unitarity of representation $\upsilon:G\to\mathcal{U}(Y)$, left-invariance of measure $\mathrm{d}x$, and $G$-equivariance of feature map $\psi$, for all $g\in G$, we have

$$ $\displaystyle\mathtt{R}_{\psi}[\pi_{g}[f]](\xi)$ $\displaystyle=\int_{X}\langle g\cdot f(g^{-1}\cdot x),\psi(x,\xi)\rangle_{Y} \mathrm{d}x=\int_{X}\langle f(x),g^{-1}\cdot\psi(g\cdot x,\xi)\rangle_{Y} \mathrm{d}x$ $\displaystyle=\int_{X}\langle f(x),\psi(x,g^{-1}\cdot\xi)\rangle_{Y}\mathrm{d} x=\widehat{\pi}_{g}[\mathtt{R}_{\psi}[f]](\xi).$ (14) $$

Similarly,

$$ $\displaystyle\mathtt{LM}_{\phi}[\widehat{\pi}_{g}[\gamma]](x)$ $\displaystyle=\int_{\Xi}\gamma(g^{-1}\cdot\xi)\phi(x,\xi)\mathrm{d}\xi=\int_{ \Xi}\gamma(\xi)\phi(x,g\cdot\xi)\mathrm{d}\xi$ $\displaystyle=\int_{\Xi}\gamma(\xi)\left(g\cdot\phi(g^{-1}\cdot x,\xi)\right) \mathrm{d}\xi=\pi_{g}[\mathtt{LM}_{\phi}[\gamma]](x).$ (15) $$

As a consequence, $\mathtt{LM}_{\phi}\circ\mathtt{R}_{\psi}:L^{2}(X;Y)\to L^{2}(X;Y)$ commutes with $\pi$ as below

$$ $\displaystyle\mathtt{LM}_{\phi}\circ\mathtt{R}_{\psi}\circ\pi_{g}=\mathtt{LM}_ {\phi}\circ\widehat{\pi}_{g}\circ\mathtt{R}_{\psi}=\pi_{g}\circ\mathtt{LM}_{ \phi}\circ\mathtt{R}_{\psi}$ (16) $$

for all $g\in G$.
Hence by Schur’s lemma (Theorem 2), there exist a constant $C_{\phi,\psi}\in\mathbb{C}$ such that
$\mathtt{LM}_{\phi}\circ\mathtt{R}_{\psi}=C_{\phi,\psi}\operatorname{Id}_{L^{2}
(X)}$.
Since $\mathtt{LM}_{\phi}\circ\mathtt{R}_{\psi}$ is bilinear in $\phi$ and $\psi$, $C_{\phi,\psi}$ is bilinear in $\phi$ and $\psi$.
∎

###### Theorem 4 (Reconstruction Formula) .

###### Proof.

## 4 Example: Depth- n 𝑛 n italic_n Joint-Equivariant Machine

As pointed out in Lemma 3, the depth-$n$ feature map $\phi_{1:n}$ is joint-$G$-equivariant.
Therefore, the following depth-$n$ $Y$-valued deep joint-equivariant machine $\mathtt{DLM}[\gamma;\phi_{1:n}]$ is $L^{2}(X;Y)$-universal.

Figure: Figure 2: Deep $Y$-valued joint-$G$-equivariant machine on $G$-space $X$ is $L^{2}(X;Y)$-universal when unitary representation $\pi$ of $G$ on $L^{2}(X;Y)$ is irreducible, and the distribution of parameters for the machine to represent a given map $f:X\to Y$ is exactly given by the ridgelet transform $\mathtt{R}[f]$
Refer to caption: x2.png

###### Corollary 1 (Deep Ridgelet Transform) .

For any maps $\gamma:X\to\mathbb{C}$ and $f\in L^{2}(X;Y)$, put

$$ $\displaystyle\mathtt{DLM}[\gamma;\phi_{1:n}](x)$ $\displaystyle:=\int_{\Xi_{1}\times\cdots\times\Xi_{n}}\gamma(\xi_{1},\ldots, \xi_{n})\phi_{n}(\bullet,\xi_{n})\circ\cdots\circ\phi_{1}(x,\xi_{1})\mathrm{d} \bm{\xi},\quad x\in X,$ (17) $\displaystyle\mathtt{R}[f;\psi_{1:n}](\bm{\xi})$ $\displaystyle:=\int_{\Xi}\langle f(x),\psi_{n}(\bullet,\xi_{n})\circ\cdots \circ\psi_{1}(x,\xi_{n})\rangle_{Y}\mathrm{d}x,\quad\bm{\xi}\in\Xi_{1}\times \cdots\times\Xi_{n}.$ (18) $$

Under the assumptions that $\mathtt{DLM}_{\phi_{1:n}}\circ\mathtt{R}_{\psi_{1:n}}$ is bounded, and that $\pi$ is irreducible, there exists a bilinear form $(\!(\phi_{1:n},\psi_{1:n})\!)$ satisfying $\mathtt{DLM}_{\phi_{1:n}}\circ\mathtt{R}_{\psi_{1:n}}=(\!(\phi_{1:n},\psi_{1:n
})\!)\operatorname{Id}_{L^{2}(X;Y)}$.

Again, it extends the original integral representation, and inherits the *linearization* trick of nonlinear parameters $\bm{\xi}$ by integrating all the possible parameters (beyond the difference of layers) and indirectly select which parameters to use by weighting on them.

###### Corollary 1 (Deep Ridgelet Transform) .

## 5 Example: Depth- n 𝑛 n italic_n Fully-Connected Network

We explain the case of depth-$n$ (precisely, depth-$n+1$) fully-connected network. We use the following fact.

###### Lemma 5 ( Folland ( 2015 , Theorem 7.12) ) .

Let $\pi_{1}$ and $\pi_{2}$ be representations of locally compact groups $G_{1}$ and $G_{2}$, and let $\pi_{1}\otimes\pi_{2}$ be their outer tensor product, which is a representation of the product group $G_{1}\times G_{2}$.
Then, $\pi_{1}$ and $\pi_{2}$ are irreducible if and only if $\pi_{1}\otimes\pi_{2}$ is irreducible.

Set $X=Y=\mathbb{R}^{m}$ (input and output domains),
and for each $i\in\{1,\ldots,n\}$, set
$\Xi_{i}:=\mathbb{R}^{p_{i}\times d_{i}}\times\mathbb{R}^{p_{i}}\times\mathbb{R
}^{d_{i+1}\times q_{i}}$ (parameter domain), $\sigma_{i}:\mathbb{R}^{p_{i}}\to\mathbb{R}^{q_{i}}$ (activation functions), and define the feature map (vector-valued fully-connected neurons) as

$$ $\displaystyle\phi_{i}(\bm{x}_{i},\bm{\xi}_{i}):=C_{i}\sigma_{i}(A_{i}\bm{x}_{i }-\bm{b}_{i}),\quad\bm{x}_{i}\in\mathbb{R}^{d_{i}},\bm{\xi}_{i}=(A_{i},\bm{b}_ {i},C_{i})\in\Xi_{i}$ (19) $$

Specifically, $d_{1}=d_{n+1}=m$. If there is no risk of confusion, we omit writing $i$ for simplicity.

Let $O(m)$ denote the orthogonal group in dimension $m$.
Let $G:=O(m)\times\operatorname{Aff}(m)$ be the product group of $O(m)$ and $\operatorname{Aff}(m)=GL(m)\ltimes\mathbb{R}^{m}$.
We suppose $G$ acts on the input and output domains as below: For any $g=(Q,L,\bm{t})\in G=O(m)\times\left(GL(m)\ltimes\mathbb{R}^{m}\right)$,

$$ $\displaystyle g\cdot\bm{x}:=L\bm{x}+\bm{t},\ \bm{x}\in X,\quad\mbox{and}\quad g \cdot\bm{y}:=\upsilon_{g}[\bm{y}]:=Q\bm{y},\ \bm{y}\in Y.$ (20) $$

Namely, the group actions of both $O(m)$ on $X$ and $\operatorname{Aff}(m)$ on $Y$ are trivial.

Let $\pi$ be the induced representation of $G$ on the vector-valued square-integrable functions $L^{2}(X;Y)$, defined by

$$ $\displaystyle\pi_{g}[\bm{f}](\bm{x}):=|\det L|^{-1/2}Q\bm{f}(L^{-1}(\bm{x}-\bm {t})),\quad\bm{x}\in X,\ \bm{f}\in L^{2}(X;Y)$ (21) $$

for each $g=(Q,L,\bm{t})\in O(m)\times(GL(m)\ltimes\mathbb{R}^{m})$.

###### Lemma 6 .

The above $\pi:G\to\mathcal{U}(L^{2}(\mathbb{R}^{m};\mathbb{R}^{m}))$ is irreducible.

###### Proof.

Recall the representations of $O(m)$ on $\mathbb{R}^{m}$ and of $\operatorname{Aff}(m)$ on $L^{2}(\mathbb{R}^{m})$ are respectively irreducible (see Theorem 3), and $L^{2}(\mathbb{R}^{m};\mathbb{R}^{m})$ is equivalent to the tensor product $\mathbb{R}^{m}\otimes L^{2}(\mathbb{R}^{m})$. Hence by Lemma 5,
the representation $\pi$ of the product group $O(m)\times\operatorname{Aff}(m)$ on the tensor product $\mathbb{R}^{m}\otimes L^{2}(\mathbb{R}^{m})=L^{2}(\mathbb{R}^{m};\mathbb{R}^{m})$ is irreducible.
∎

Additionally, we put the dual action of $G$ on parameter domain $\Xi_{i}$ as below:

$$ $\displaystyle g\cdot(A_{i},\bm{b}_{i},C_{i}):=\begin{cases}(A_{i}L^{-1},\bm{b} _{i}+A_{i}L^{-1}\bm{t},C_{i}),&i=1\\ (A_{i},\bm{b}_{i},C_{i}),&i\neq 1,n\\ (A_{i},\bm{b}_{i},QC_{i}),&i=n\end{cases}$ (22) $$

for all $g=(Q,L,\bm{t})\in O(m)\times(GL(m)\ltimes\mathbb{R}^{m}),\ (A_{i},\bm{b}_{i},C
_{i})\in\Xi_{i}$.

Then, the composition of feature maps $\phi_{1:n}(\bm{x},\bm{\xi}_{1:n}):=\phi_{n}(\bullet,\bm{\xi}_{n})\circ\cdots
\circ\phi_{1}(\bm{x},\bm{\xi}_{1})$ is joint-$G$-equivariant. In fact,

$$ $\displaystyle\phi_{1}(g\cdot\bm{x},g\cdot\bm{\xi}_{1})$ $\displaystyle=C_{1}\sigma\left(A_{1}L^{-1}(L\bm{x}+\bm{t})-(\bm{b}_{1}+A_{1}L^ {-1}\bm{t})\right)=C_{1}\sigma(A_{1}\bm{x}-\bm{b}_{1})=\phi_{1}(\bm{x},\bm{\xi }_{1}),$ $\displaystyle\phi_{i}(\bm{x},g\cdot\bm{\xi}_{i})$ $\displaystyle=C_{i}\sigma(A_{i}\bm{x}-\bm{b}_{i})=\phi_{i}(\bm{x},\bm{\xi}_{i} ),\quad i\neq 1,n$ $\displaystyle\phi_{n}(\bm{x},g\cdot\bm{\xi}_{n})$ $\displaystyle=QC_{n}\sigma(A_{n}\bm{x}-\bm{b}_{n})=g\cdot\phi_{n}(\bm{x},\bm{ \xi}_{n}),$ $$

Therefore $\phi_{1:n}(g\cdot\bm{x},g\cdot\bm{\xi}_{1:n})=g\cdot\phi_{1:n}(\bm{x},\bm{\xi}
_{1:n})$.

So by putting depth-$n$ neural network and the corresponding ridgelet transform as below

$$ $\displaystyle\mathtt{DNN}[\gamma;\phi_{1:n}](\bm{x})$ $\displaystyle=\int_{\Xi_{1:n}}\gamma(\bm{\xi}_{1:n})\phi_{1:n}(\bm{x},\bm{\xi} _{1:n})\mathrm{d}\bm{\xi}_{1:n},$ (23) $\displaystyle\mathtt{R}[\bm{f};\psi_{1:n}](\bm{a},b)$ $\displaystyle=\int_{\mathbb{R}^{m}}\bm{f}(\bm{x})\cdot\overline{\psi_{1:n}(\bm {x},\bm{\xi}_{1:n})}\mathrm{d}\bm{x},$ (24) $$

Theorem 4 yields the reconstruction formula $\mathtt{DNN}_{\phi_{1:n}}\circ\mathtt{R}_{\psi_{1:n}}=(\!(\phi_{1:n},\psi_{1:n
})\!)\operatorname{Id}_{L^{2}(\mathbb{R}^{m};\mathbb{R}^{m})}$.

###### Lemma 5 ( Folland ( 2015 , Theorem 7.12) ) .

###### Lemma 6 .

###### Proof.

## 6 Example: Quadratic-form with Nonlinearity

Here, we present a new network for which the universality was not known.

Let $M$ denote the class of all $m\times m$-symmetric matrices
equipped with the Lebesgue measure $\mathrm{d}A=\bigwedge_{i\geq j}\mathrm{d}a_{ij}$.
Set $X=\mathbb{R}^{m}$, $\Xi=M\times\mathbb{R}^{m}\times\mathbb{R}$, and

$$ $\displaystyle\phi(\bm{x},\xi):=\sigma(\bm{x}^{\top}A\bm{x}+\bm{x}^{\top}\bm{b} +c)$ (25) $$

for any fixed function $\sigma:\mathbb{R}\to\mathbb{R}$.
Namely, it is a quadratic-form in $x$ followed by nonlinear activation function $\sigma$.

Then, it is joint-invariant with $G=\operatorname{Aff}(m)$.
In fact, we can put the group actions of $g=(\bm{t},L)\in\mathbb{R}^{m}\rtimes GL(m)$ on $X$ and $\Xi$ by

$$ $\displaystyle(\bm{t},L)\cdot\bm{x}$ $\displaystyle:=\bm{t}+L\bm{x},$ (26) $\displaystyle(\bm{t},L)\cdot(A,\bm{b},c)$ $\displaystyle:=(L^{-\top}AL^{-1},L^{-\top}\bm{b}-2L^{-\top}AL^{-1}\bm{t},c+\bm {t}^{\top}L^{-\top}AL^{-1}\bm{t}-\bm{t}^{\top}L^{-\top}\bm{b}).$ (27) $$

Then, the joint-$G$-action on $X\times\Xi$ remains the feature map joint-invariant as below.

$$ $\displaystyle\phi(g\cdot\bm{x},g\cdot\bm{\xi})$ $\displaystyle=\sigma((L\bm{x}+\bm{t})^{\top}L^{-\top}AL^{-1}(L\bm{x}+\bm{t})+( L\bm{x}+\bm{t})^{\top}(L^{-\top}\bm{b}-2L^{-\top}AL^{-1}\bm{t})+...)$ $\displaystyle=\sigma(\bm{x}^{\top}A\bm{x}+2\bm{x}^{\top}AL^{-1}\bm{t}+\bm{t}^{ \top}L^{-\top}AL^{-1}\bm{t}+$ $\displaystyle\qquad+\bm{x}^{\top}\bm{b}-2\bm{x}^{\top}AL^{-1}\bm{t}+\bm{t}^{ \top}L^{-\top}\bm{b}-2\bm{t}^{\top}L^{-\top}AL^{-1}\bm{t}$ $\displaystyle\qquad+c+\bm{t}^{\top}L^{-\top}AL^{-1}\bm{t}-\bm{t}^{\top}L^{- \top}\bm{b})$ $\displaystyle=\sigma(\bm{x}^{\top}A\bm{x}+\bm{x}^{\top}\bm{b}+c)=\phi(g\cdot \bm{x},g\cdot\bm{\xi}).$ $$

By Theorem 3, the regular representation $\pi$ of $\operatorname{Aff}(m)$ on $L^{2}(\mathbb{R}^{m})$ is irreducible.
Hence as a consequence of the general result, the following network is $L^{2}(\mathbb{R}^{m})$-universal.

$$ $\displaystyle\mathtt{NN}[\gamma](\bm{x}):=\int_{M\times\mathbb{R}^{m}\times \mathbb{R}}\gamma(A,\bm{b},c)\sigma(\bm{x}^{\top}A\bm{x}+\bm{x}^{\top}\bm{b}+c )\mathrm{d}A\mathrm{d}\bm{b}\mathrm{d}c.$ (28) $$

## 7 Discussion

We have developed a systematic method for deriving a ridgelet transform for a wide range of learning machines defined by joint-group-equivariant feature maps, yielding the universal approximation theorems as corollaries.
Traditionally, the techniques used in the expressive power analysis of deep networks were different from those used in the analysis of shallow networks, as overviewed in the introduction. Our main theorem unifies the approximation schemes of both deep and shallow networks from the perspective of joint-group-action on the data-parameter domain. Technically, this unification is due to Schur’s lemma, a basic and useful result in the representation theory. Thanks to this lemma, the proof of the main theorem is simple, yet the scope of application is wide. The significance of this study lies in revealing the close relationship between machine learning theory and modern algebra. With this study as a catalyst, we expect a major upgrade to machine learning theory from the perspective of modern algebra.

#### Acknowledgments

This work was supported by JSPS KAKENHI 20K03657, JST PRESTO JPMJPR2125, JST CREST JPMJCR2015 and JPMJCR1913, and JST ACTX JPMJAX2004.

## Peer Reviews

confidence: 3
contribution: 4
initial_rating: 6
presentation: 4
questions: - Does the derivation completely hold for the case in which the irreducible trivial representation is required for the parameter domain to include the group-invariant case?

- Are there any specific underlying assumptions for the non-linearities and how does this relate to the special requirements that non-linearities usually need to fulfill to preserve group equivariance?
soundness: 3
strengths: This work presents some important insights and most importantly a constructive universal approximation theorem for (joint) group equivariant neural networks. The presentation is clear, the proofs concise, and the introduction includes all necessary lemmas from group representation theory. In addition to the derivation of a ridgelet transform for joint group invariant networks, section 5 provides the corresponding ridgeless transform for n-layer fully connected neural networks.
summary: This work derives a closed-form ridgelet transform for a specific class of neural networks, that is defined as joint-group-equivariant networks. In this class of neural networks, the assumption is that not only input and output are equivariant to a group action (group-equivariant networks), but that instead also the model's parameters are equivariant to a representation of the same group action acting on the networks parameters. While this at first seems like a limitation of the presented theory, it provides insights to a broad range of models as demonstrated in the experiments
weaknesses: First it appears to be a limitation of the work, that the presented theory only applies to joint group equivariant networks (that is, the same group action is applied to the network's parameters as to the input and output), instead of to the widely used group equivariant networks (in which no group action acts on the model parameters to reach equivariance). In Remark 1, also Figure 1, it is however stated that the group-equivariant networks are included as a subclass of joint equivariant networks, as those for which G acts trivially on the parameter domain.

This raises some concerns, if all of the developed theory applies to this case. Theorem 4, which is the core of the established theory and defines the ridgelet transform, relies on irreducible representations of the group actions, both for the input as for the parameter space.
If however we would require the irreducible trivial representation for the parameter space but not for the input space then the question arises if all of the proofs still hold for this case. This might trivially be the case but because of its importance a comment on this special case would strengthen the manuscript.

Further information on the underlying assumptions for the non-linearities is needed. For group equivariant networks it is well-known that arbitrary non-linearities might destroy group-equivariance. It would be important to state which assumptions need to hold for the non-linearities for the joint group equivariant case and group equivariant case (in which G acts trivially on the parameter domain).

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 2
questions: - Can the authors clarify the differences between the presented paper and [1] and explain how this work provides novel insights relative to the cited paper?
- Could the authors elaborate on the relevance of the examples presented in Sections 5 and 6 within the context of current equivariant ML literature?

---
[1] S. Sonoda et al., Joint Group Invariant Functions on Data-Parameter Domain Induce Universal Neural Networks. In Proceedings of the 2nd NeurIPS, Workshop on Symmetry and Geometry in Neural Representations, Proceedings of Machine Learning Research, PMLR, 2024.
soundness: 4
strengths: - The paper generalizes existing work on constructive approximation and the ridgelet transform on invariant machines to the equivariant case.
- Mathematical notation is clear, and the proofs are correct.
summary: The proposed paper presents a constructive universal approximation theorem for learning machines equipped with joint-group equivariant feature maps, extending previous work on convolutional and joint-group invariant feature maps. Additionally, it provides examples of depth-$n$ joint-equivariant machines, depth-$n$ fully connected networks, and quadratic forms with nonlinearity.
weaknesses: 1. Significant overlap with previous work [1].
2. Lack of examples covering models commonly used in the equivariant ML literature.
3. Numerous technical lemmas hinder readability and obscure the main message.

Regarding the first weakness, I understand that group equivariance generalizes invariance, extending the scope of this work beyond [1]. However, if I am not mistaken, the technical advancements over [1] to achieve this generalization are minimal. Indeed, the *principal contributions* of this paper—the statement and proof of the main theorem (Theorem 4)—appear *nearly identical* to those of Theorem 9 in [1].

Moreover, it appears that Section 3 (Main Results) of the paper presents only minor differences from Section 3 of [1], which can be summarized as follows:

- Jointly-Equivariant Machines (Def 4) are simply the equivariant analogs of Generalized Neural Networks ([1], Def 7), with no further differences in definition.
- The Ridgelet Transform for Jointly-Equivariant Machines (Def 5) is analogous to the Ridgelet Transform of [1] but is integrated over a scalar product on $Y$ instead of a scalar product on $\mathbb{C}$.
- This section presents four technical lemmas with simple, straightforward proofs that, in my opinion, hinder readability and could be moved to an appendix.

Additionally, Section 2 of the paper and Section 2 of [1] are again nearly identical, though this may be more understandable as they introduce notation.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: In so far as I could tell, the mathematics is correct. I once read Folland's book (a primary reference), so I should have some appropriate background, but still this is a long time ago and with the time I could invest in the paper I could not ascertain with 100% confidence that all results are correct. I am fairly confident in the main theorem, but I found it a bit hard to see when exactly the conditions are satisfied. Seeing more examples of concrete networks where the conditions are satisfied, and importantly, examples where they fail, would help me better appreciate the theorem and convince myself of its significance.
soundness: 3
strengths: The paper is written in a clear and precise way, and the authors are clearly expert on the topic. Although the result is ultimately a simple application of Schur's lemma, the insight that this tool can be applied here to answer universality questions, and the whole setup that enables the statement of the theorem, is in my opinion quite significant.
summary: The paper presents a new theorem establishing universality of joint group-equivariant learning machines (where a locally compact group acts on input, parameter and output space). The proof is constructive, in that an integral transform is given that maps the function to be represented to a parameter distribution (generalized function). As a special case, this covers universality of deep fully-connected networks, for which such a constructive proof had not been available (it was only known for shallow ones). The approach of this paper applies equally to shallow and deep networks. The paper also introduces new techniques based on group representation theory, that may find wider application in the theory of neural networks.

The basic idea of the proof is to show that the map LM(R(f)) commutes with an irreducible representation and is therefore by Schur's lemma a multiple of the identity. Here R is the operator taking functions to parameters, and LM is the operator mapping parameters to functions.
weaknesses: I could not find very significant weaknesses, but one limitation seems to be that although these techniques establish universality for potentially infinitely wide (and finitely deep) neural networks, they do not show that the approximation error can be reduced by increasing width (or at what rate this error reduces).

Meta-review: This paper presents a constructive universal approximation theorem for learning machines equipped with joint-group-equivariant feature maps, utilizing group representation theory. The authors introduce the concept of joint-group-equivariance, which generalizes classical group-equivariance and includes fully-connected networks. The main theorem aims to unify universal approximation theorems for both shallow and deep networks, providing a closed-form expression for parameter distribution via the ridgelet transform. The reviewers acknowledged the novel theoretical contributions of this paper and the validity of the results. However, a concern was raised about the similarity of this paper and [1], in terms of the presentation of the results and the proof techniques. This concern was not resolved after a lengthy discussion process.

[1] S. Sonoda et al., Joint Group Invariant Functions on Data-Parameter Domain Induce Universal Neural Networks. In Proceedings of the 2nd NeurIPS, Workshop on Symmetry and Geometry in Neural Representations, Proceedings of Machine Learning Research, PMLR, 2024.

Decision: Reject