Title: Constructive Universal Approximation Theorems for Deep Joint-Equivariant Networks by Schur's Lemma
Venue: NeurIPS 2024
Accepted: no
Keywords: Schur's lemma

deep neural network

joint-group-equivariant

universality

ridgelet transform
Abstract: We present a unified constructive universal approximation theorem covering a wide range of learning machines including both shallow and deep neural networks based on the group representation theory. Constructive here means that the distribution of parameters is given in a closed-form expression (called the *ridgelet transform*). Contrary to the case of shallow models, expressive power analysis of deep models has been conducted in a case-by-case manner. Recently, Sonoda et al. (2023a,b) developed a systematic method to show a constructive approximation theorem from *scalar-valued joint-group-invariant* feature maps, covering a formal deep network. However, each hidden layer was formalized as an abstract group action, so it was not possible to cover real deep networks defined by composites of nonlinear activation function. In this study, we extend the method for *vector-valued joint-group-equivariant* feature maps, so to cover such real networks.

Abstract

We present a unified constructive universal approximation theorem covering a wide range of learning machines including both shallow and deep neural networks based on the group representation theory. Constructive here means that the distribution of parameters is given in a closed-form expression (called the ridgelet transform ). Contrary to the case of shallow models, expressive power analysis of deep models has been conducted in a case-by-case manner. Recently, Sonoda et al. [33, 32] developed a systematic method to show a constructive approximation theorem from scalar-valued joint-group-invariant feature maps, covering a formal deep network. However, each hidden layer was formalized as an abstract group action, so it was not possible to cover real deep networks defined by composites of nonlinear activation function. In this study, we extend the method for vector-valued jointgroup-equivariant feature maps, so to cover such real networks.

1 Introduction

An ultimate goal of the deep learning theory is to characterize the internal data processing procedure 14 inside deep neural networks obtained by deep learning. We may formulate this problem as a functional 15 equation problem: Let F denote a class of data generating functions, and let DNN [ γ ] denote a certain 16 deep neural network with parameter γ . Given a function f ∈ F , find an unknown parameter γ so that 17 network DNN [ γ ] represents function f , i.e. 18

which we call a DNN equation . An ordinary learning problem by empirical risk minimization, such as minimizing ∑ n i =1 | DNN [ γ ]( x i ) -f ( x i ) | 2 with respect to γ , is understood as a weak form (or a variational form) of this equation. Therefore, characterizing the solution space of this equation leads to understanding the parameters obtained by deep learning. Following previous studies [21, 3, 2831], we call a solution operator R that satisfies DNN [ R [ f ]] = f a ridgelet transform . Once such a solution operator R is found, we can conclude a universality of the DNN in consideration because the reconstruction formula DNN [ R [ f ]] = f implies for any f ∈ F there exists a DNN that represents f . In particular, when R [ f ] is found in a closed-form manner, then it leads to a constructive proof of the universality since R [ f ] could indicate how to assign parameters.

19

20

21

22

23

24

25

26

27

28

29

30

31

32

When the network has only one infinitely-wide hidden layer, though it is not deep but shallow, the characterization problem has been well investigated. For example, the learning dynamics and the global convergence property (of SGD) are well studied in the mean field theory [22, 25, 20, 5] and the Langevin dynamics theory [35], and even closed-form solution operator to a 'shallow' NN equation, the original ridgelet transform, has already been presented [28-31].

On the other hand, when the network has more than one hidden layer, the problem is far from 33 solved, and it is common to either consider infinitely-deep mathematical models such as Neural 34

ODEs [27, 9, 17, 12, 4], or handcraft inner feature maps depending on the problem. For example, 35 construction methods such as the Telgarsky sawtooth function (or the Yarotsky scheme) and bit 36 extraction techniques [7, 36-39, 8, 6, 26, 24, 11] have been developed to demonstrate the depth 37 separation, super-convergence, and minmax optimality of deep ReLU networks. Various feature maps 38 have also been handcrafted in the contexts of geometric deep learning [1] and deep narrow networks 39 [19, 13, 18, 14, 23, 16, 2, 15]. Needless to say, there is no guarantee that these handcrafted feature 40 maps are acquired by deep learning, so these analyses are considered to be analyses of possible 41 worlds. 42

43

44

45

46

47

48

49

50

51

52

53

54

Recently, Sonoda et al. [33, 32] discovered a rich class of ridgelet transforms for learning machines defined by scalar-valued joint-group-invariant feature maps, covering both depth-2 fully-connected networks and the formal deep network (FDN), yielding the first ridgelet transform for deep models. Their theory is indeed a breakthrough because it could cover both deep and shallow models simultaneously. However, each hidden layer in the FDN has to be formalized as an abstract group action, so it was not possible to cover real deep networks defined by composites of nonlinear activation function. In this study, we extend their arguments for vector-valued joint-group-equivariant feature maps (Theorem 3 and Corollary 1), so to cover such real networks. As an important example, in § 4.2, we obtained the ridgelet transform for a more realistic DNN, the depthn fully-connected network with an arbitrary activation function (not limited to ReLU), without handcrafting network architecture. In other words, it is a constructive proof of the L 2 ( R m ; R m ) -universality of the DNNs, and an explicit characterization of the solution space of the DNN equation for more realistic setup.

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

Thanks to Schur's lemma, a basic and useful result in the representation theory, the proof of the main theorem is surprisingly simple, yet the scope of application is wide. The significance of this study lies in revealing the close relationship between machine learning theory and modern algebra. With this study as a catalyst, we expect a major upgrade to machine learning theory from the perspective of modern algebra.

2 Preliminaries

We quickly introduce the original integral representation and the ridgelet transform, a mathematical model of depth-2 fully-connected network and its right inverse. Then, we list a few facts in the group representation theory. In particular, Schur's lemma and the Haar measure play key roles in the proof of the main results.

Notation. For any topological space X , C c ( X ) denotes the Banach space of all compactly supported continuous functions on X . For any measure space X , L p ( X ) denotes the Banach space of all p -integrable functions on X . S ( R d ) and S ′ ( R d ) denote the classes of rapidly decreasing functions (or Schwartz test functions) and tempered distributions on R d , respectively.

2.1 Integral Representation and Ridgelet Transform for Depth-2 Fully-Connected Network

Definition 1. For any measurable functions σ : R → C and γ : R m × R → C , put

We call S σ [ γ ] an (integral representation of) neural network, and γ a parameter distribution.

The integration over all the hidden parameters ( a , b ) ∈ R m × R means all the neurons { x ↦→ 72 σ ( a · x -b ) | ( a , b ) ∈ R m × R } are summed (or integrated, to be precise) with weight γ , hence 73 formally S σ [ γ ] is understood as a continuous neural network with a single hidden layer. We note, 74 however, when γ is a finite sum of point measures such as γ p = ∑ p i =1 c i δ ( a i ,b i ) (by appropriately 75 extending the class of γ to Borel measures), then it can also reproduce a finite width network 76

In other words, the integral representation is a mathmatical model of depth-2 network with any width 77 (ranging from finite to continuous). 78

Definition 2. For any measurable functions ρ : R → C and f : R m → C , put 80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

We call R ρ a ridgelet transform.

To be precise, it satisfies the following reconstruction formula.

Theorem 1 (Reconstruction Formula) . Suppose σ and ρ are a tempered distribution ( S ′ ) and a rapid decreasing function ( S ) respectively. There exists a bilinear form ( ( σ, ρ ) ) such that for any square integrable function f ∈ L 2 ( R m ) . Further, the bilinear form is given by ( ( σ, ρ ) ) = ∫ R σ ♯ ( ω ) ρ ♯ ( ω ) | ω | -m d ω, where ♯ denotes the 1-dimensional Fourier transform.

See Sonoda et al. [29, Theorem 6] for the proof. In particular, according to Sonoda et al. [29, Lemma 9], for any activation function σ , there always exists ρ satisfying ( ( σ, ρ ) ) = 1 . Here, σ being a tempered distribution means that typical activation functions are covered such as ReLU, step function, tanh , gaussian, etc... We can interpret the reconstruction formula as a universality theorem of continuous neural networks, since for any given data generating function f , a network with output weight γ f = R ρ [ f ] reproduces f (up to factor ( ( σ, ρ ) ) ), i.e. S [ γ f ] = f . In other words, the ridgelet transform indicates how the network parameters should be organized so that the network represents an individual function f .

The original ridgelet transform was discovered by Murata [21] and Candès [3]. It is recently extended to a few modern networks by the Fourier slice method [34, see e.g.]. In this study, we present a systematic scheme to find the ridgelet transform for a variety of given network architecture based on the group theoretic arguments.

2.2 Irreducible Unitary Representation and Schur's Lemma

Let G be a locally compact group, H be a nonzero Hilbert space, and U ( H ) be the group of unitary operators on H . For example, any finite group, discrete group, compact group, and finite-dimensional Lie group are locally compact, while an infinite-dimensional Lie group is not locally compact. A unitary representation π of G on H is a group homomorphism that is continuous with respect to the strong operator topology-that is, a map π : G → U ( H ) satisfying π ( gh ) = π ( g ) π ( h ) and π ( g -1 ) = π ( g ) -1 , and for any ψ ∈ H , the map G ∋ g ↦→ π ( g )[ ψ ] ∈ H is continuous.

Suppose M is a closed subspace of H . M is called an invariant subspace when π ( g ) M⊂M for all g ∈ G . Particularly, π is called irreducible when it does not admit any nontrivial invariant subspace M̸ = { 0 } nor H . The following theorem is a fundamental result of group representation theory that characterizes the irreducibility.

Theorem 2 (Schur's lemma) . A unitary representation ( π, H ) is irreducible iff any bounded operator T on H that commutes with π is always a constant multiple of the identity. In other words, if π ( g ) T = Tπ ( g ) for all g ∈ G , then T = c Id H for some c ∈ C .

See Folland [10, Theorem 3.5(a)] for the proof. We use this as a key step in the proof of our main theorem.

2.3 Calculus on Locally Compact Group

By Haar's theorem, if G is a locally compact group, then there uniquely exist left and right invariant measures d l g and d r g , satisfying for any s ∈ G and f ∈ C c ( G ) ,

Let X be a G -space with transitive left (resp. right) G -action g · x (resp. x · g ) for any ( g, x ) ∈ G × X . Then, we can further induce the left (resp. right) invariant measure d l x (resp. d r x ) so that for any f ∈ C c ( G ) ,

where o ∈ X is a fixed point called the origin.

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

g.x

input

ξ

g.y

output

φ

parameter

G-equivariance

g.x

input

g.ξ

g.y

output

φ

parameter

joint-G-equivariance

3 Main Results

We introduce the joint-group-equivariant feature map, and present the ridgelet transforms for learning machines defined by joint-group-equivariant feature maps, yielding the universality of deep models.

Let G be a locally compact group equipped with a left invariant measure d g . Let X and Ξ be G -spaces equipped with G -invariant measures d x and d ξ , called the data domain and the parameter domain, respectively. Particularly, we call the product space X × Ξ the data-parameter domain (like time-frequency domain), and call any map ϕ on data-parameter domain X × Ξ a feature map . Let H be a separable Hilbert space, let U ( H ) be the space of unitary operators on H , and let υ : G →U ( H ) be a unitary representation of G on H . If there is no danger of confusion, we use the same symbol · for the G -actions on X , H , and Ξ (e.g., g · x , g · v , and g · ξ ).

In the main theorem, the irreducibility of the following unitary representation π will be a sufficient condition for the universality. Let L 2 ( X ; H ) denote the space of H -valued square-integrable functions on X equipped with the inner product ⟨ ϕ, ψ ⟩ L 2 ( X ; H ) := ∫ X ⟨ ϕ ( x ) , ψ ( x ) ⟩ H d x . Put

Then, it is a unitary representation of G on L 2 ( X ; H ) . In fact, π g [ π h [ f ]]( x ) = g · h · f ( h -1 · g -1 · x ) = ( gh ) · f (( gh ) -1 · x ) = π gh [ f ]( x ) , and ⟨ π g [ f 1 ] , π g [ f 2 ] ⟩ L 2 ( X ; H ) = ∫ X ⟨ υ g [ f 1 ]( g -1 · x ) , υ g [ f 2 ]( g -1 · x ) ⟩ H d x = ∫ X ⟨ f 1 ( x ) , υ ∗ g [ υ g [ f 2 ]]( x ) ⟩ H d x = ⟨ f 1 , f 2 ⟩ L 2 ( X ; H ) .

In addition, let L 2 (Ξ) denote the space of C -valued square-integrable functions on Ξ , and let ̂ π be the left-regular representation of G on L 2 (Ξ) given by

Similarly to π , ̂ π is also a unitary representation.

Definition 3 (Joint G -Equivariant Feature Map) . Let X,Y be data domains, and Ξ be a parameter domain (with G -actions). We say a feature map ϕ : X × Ξ → Y is jointG -equivariant when

holds for all g ∈ G . In other words, ϕ is a homomorphism (or G -map) of G -sets from X × Ξ to Y . So by hom G ( X × Ξ , Y ) , we denote the collection of all jointG -equivariant maps. Additionally, when G -action on Y is trivial, i.e. ϕ ( g · x, g · ξ ) = ϕ ( x, ξ ) , we say it is jointG -invariant .

Remark 1 . The jointG -equivariance extends an ordinary notion of G -equivariance , i.e. ϕ ( g · x, ξ ) = g · ϕ ( x, ξ ) . In fact, G -equivariance is a special case of jointG -equivariance where G acts trivially on parameter domain, i.e. g · ξ = ξ (see also Figure 1).

In order to construct a (non-joint) group-equivariant network, we must carefully and precisely design the network architecture [see, e.g., a textbook of geometric deep learning 1]. On the other hand, we can easily and systematically construct jointG -equivariant network from (not at all equivariant but) any map f : X → Y according to the following Lemmas 1 and 2.

Lemma 1. Suppose group G acts on sets X and Y . Fix an arbitrary map f : X → Y , and put ϕ ( x, g ) := g · f ( g -1 · x ) for every x ∈ X and g ∈ G . Then, ϕ : X × G → Y is jointG -equivariant.

Proof. Straightforward. For any g ∈ G , ϕ ( g · x, g · h ) = ( gh ) · f (( gh ) -1 · ( g · x )) = g · ϕ ( x, h ) .

Lemma 2 (Depthn Feature Map ϕ 1: n ) . Given a sequence of G -equivariant feature maps ϕ i : 156 X i -1 × Ξ i → X i ( i = 1 , . . . , n ) , put ϕ 1: n : X 0 × Ξ 1 ×··· × Ξ n → X n by 157

158

159

Then, ϕ 1: n is G -equivariant. Following the custom of counting the number of parameter domains (Ξ i ) n i =1 , we say ϕ 1: n is depthn .

Proof. In fact, 160

161

162

Definition 4 ( ϕ -Network) . For any vector-valued map ϕ : X × Ξ → H and scalar-valued map γ : Ξ → C , define a vector-valued map X →H by

163

164

165

166

167

168

169

170

171

172

where the integral is understood as the Bocher integral.

We call the integral transform NN [ · ; ϕ ] a ϕ -transform, and each individual image NN [ γ ; ϕ ] a ϕ -network for short. The ϕ -network extends the original integral representation. In particular, it inherits the concept of integrating all the possible parameters ξ and indirectly select which parameters to use by weighting on them, which linearize parametrization by lifting nonlinear parameters ξ to linear parameter γ .

Definition 5 ( ψ -Ridgelet Transform) . For any H -valued feature map ψ : X × Ξ →H and H -valued Borel measurable function f on X , put a scalar-valued integral transform

We call the integral transform R [ · ; ψ ] a ψ -ridgelet transform for short.

As long as the integrals are convergent, ϕ -ridgelet transform is the dual operator of ϕ -transform, since

Theorem 3 (Reconstruction Formula) . Assume (1) H -valued feature maps ϕ, ψ : X × Ξ →H are 173 jointG -equivariant, (2) composite operator NN ϕ ◦ R ψ : L 2 ( X ; H ) → L 2 ( X ; H ) is bounded (i.e., 174 Lipschitz continuous), and (3) the unitary representation π defined in (6) is irreducible. Then, there 175 exists a bilinear form ( ( ϕ, ψ ) ) ∈ C (independent of f ) such that for any H -valued square-integrable 176 function f ∈ L 2 ( X ; H ) , 177

178

179

In other words, the ψ -ridgelet transform R ψ is a right inverse operator of ϕ -transform NN ϕ as long as ( ( ϕ, ψ ) ) = 0 , ∞ .

̸

Proof. We write NN [ · ; ϕ ] as NN ϕ and R [ · ; ϕ ] as R ϕ for short. By using the unitarity of representation 180 υ : G →U ( H ) , left-invariance of measure d x , and G -equivariance of feature map ψ , for all g ∈ G , 181 we have 182

x

G-space

input

ξ1

vector-space

output

φ1

G-space

parameter

x1

ξ2

φ2

x2

ξ3

φ3

y

G-space

feature

Similarly, 183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

Here, ̂ π ∗ denotes the dual representation of ̂ π with respect to L 2 (Ξ) .

As a consequence, NN ϕ ◦ R ψ : L 2 ( X ; H ) → L 2 ( X ; H ) commutes with π as below for all g ∈ G . Hence by Schur's lemma (Theorem 2), there exist a constant C ϕ,ψ ∈ C such that NN ϕ ◦ R ψ = C ϕ,ψ Id L 2 ( X ) . Since NN ϕ ◦ R ψ is bilinear in ϕ and ψ , C ϕ,ψ is bilinear in ϕ and ψ .

In particular, because depthn feature map ϕ 1: n is G -equivariant (Lemma 2), the following depthn H -valued deep network DNN [ γ ; ϕ 1: n ] is L 2 ( X ; H ) -universal.

Corollary 1 (Deep Ridgelet Transform) . For any maps γ : X → C and f ∈ L 2 ( X ; H ) , put

Under the assumptions that DNN ϕ 1: n ◦ R ψ 1: n is bounded, and that π is irreducible, there exists a bilinear form ( ( ϕ 1: n , ψ 1: n ) ) satisfying DNN ϕ 1: n ◦ R ψ 1: n = ( ( ϕ 1: n , ψ 1: n ) ) Id L 2 ( X ; H ) .

Again, it extends the original integral representation, and inherits the linearization trick of nonlinear parameters ξ by integrating all the possible parameters (beyond the difference of layers) and indirectly select which parameters to use by weighting on them.

4 Example: Depthn Fully-Connected Network with Arbitrary Activation

As a concrete example, we present the ridgelet transform for depthn fully-connected network. First, we show the depth-2 case based on a joint-affineinvariant argument, which was originally demonstrated by Sonoda et al. [33]. Then, we show the depthn case based on a jointequivariant argument by extending the original arguments.

We use the following known facts.

Lemma 3. The regular representation π of the affine group Aff( m ) on L 2 ( R m ) (defined below) is irreducible.

See Folland [10, Theorem 6.42] for the proof.

Lemma4. Suppose σ and ρ are a tempered distribution ( S ′ ) and a Schwartz test function, respectively. Then, S σ ◦ R ρ : L 2 ( R m ) → L 2 ( R m ) is bounded.

See Sonoda et al. [29, Lemmas 7 and 8] for the proof.

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

4.1 Depth-2

Set X := R m (data domain), Ξ := R m × R (parameter domain), and G := Aff( m ) = GL ( m ) ⋉R m be the m -dimensional affine group, acting on data domain X by

Addition to this, let π be the regular representation of Aff( m ) on L 2 ( X ) , namely

Further, define a dual action of Aff( m ) on the parameter domain Ξ as

Then, we can see the feature map ϕ ( x , ( a , b )) := σ ( a · x -b ) is jointG -invariant. In fact,

By Lemma 3, the regular representation π of G = Aff( m ) is irreducible. Therefore, by Theorem 3, the depth2 neural network and corresponding ridgelet transform:

satisfy the reconstruction formula NN ◦ R 2 = ( ( σ, ρ ) ) Id L 2 ( R m ) . In Appendix A, we supplemented a detailed proof. In Appendix B, we discussed a geometric interpretation of dual G -action (21).

4.2 Depthn

Following Corollary 1, we derive the ridgelet transform for depthn fully-connected network by constructing a joint-equivariant network.

Let O ( m ) be the m -dimensional orthogonal group acting on R m by Q v for Q ∈ O ( m ) and v ∈ R m , and (re)set G := O ( m ) × Aff( m ) be the product group, acting on the data domain X by

Namely, we set the O ( m ) -action on X is trivial. Define a unitary representation π of G on vectorvalued square-integrable functions L 2 ( X ; X ) as

Lemma 5. The above π : G → L 2 ( R m ; R m ) is an irreducible unitary representation.

Proof. Recall that a tensor product of irreducible representations is irreducible. Since both O ( m ) -action on R m and Aff( m ) -action on L 2 ( R m ) are irreducible, and L 2 ( R m ; R m ) is a tensor product R m ⊗ L 2 ( R m ) , so the action π of product group O ( m ) × Aff( m ) on tensor product R m ⊗ L 2 ( R m ) = L 2 ( R m ; R m ) is irreducible.

Following the same arguments in Lemma 1, we first construct a depth2 jointG -equivariant network. Take an arbitrary square-integrable (not yet jointG -equivariant) vector-field f 0 ∈ L 2 ( X ; X ) . Then, the following network is jointG -equivariant:

for every x ∈ X,ξ = ( Q,L, t ) ∈ O ( m ) × GL ( m ) ⋉ R m . Here R 2 is the ridgelet transform for depth-2 case (applied for vector-valued function by element-wise manner). This is jointG -equivariant because NN ( x , ξ ) = π ξ [ f 0 ]( x ) . Henceforth, we (re)set Ξ := G .

Finally, we construct a depthn jointG -equivariant network by composing the above depth2 networks as below. Write ξ := ( ξ 1 , . . . , ξ n ) ∈ Ξ n for short. For any measurable function γ : Ξ n → C and vector-field f : R m → R m , put

Then, as a consequence of Corollary 1, there exists a constant c ∈ C satisfying DNN ◦ R n [ f ] = c f for 239 any f ∈ L 2 ( X ; X ) . 240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

5 Example: Formal Deep Network

We explain the formal deep network (FDN) introduced by Sonoda et al. [32]. Compared to the depthn fully-connected network introduced in the previous section, the FDN (introduced in the previous study) is more abstract because the network architecture is not specified. Yet, we consider this is still useful for theoretical study of deep networks as it covers a wide range of groups and data domains (i.e., not limited to the affine group and the Euclidean space).

5.1 Formal Deep Network

Let G be an arbitrary locally compact group equipped with left-invariant measure d g , let X be a G -space equipped with left-invariant measure d x , and set Ξ := G with right-invariant measure d ξ . The key concept is to identify each feature map ϕ : X × Ξ → X with a G -action g : X → X with parameter domain Ξ being identified with group G , and the composite of feature maps, say g ◦ h , with product gh . Since a group is closed under its operation by definition, the proposed network can represent literally any depth such as a single hidden layer g , double hidden layers g ◦ h , triple hidden layers g ◦ h ◦ k , and infinite hidden layers g ◦ h ◦ · · · . Besides, to lift the group action on a linear space, the network is formulated as a regular action of group G on a hidden layer, say ψ ∈ L 2 ( X ) .

Definition 6 (Formal Deep Network) . For any functions ψ ∈ L 2 ( X ) and γ : Ξ → C , put

Here, G = G 1 ⋊ · · · ⋊ G n denotes the semi-direct product of groups, suggesting that the network gets much complex and expressive as it gets deeper.

To see the universality, define the dual action of G on the parameter domain Ξ = G as

Then, we can see ϕ ( x, ξ ) := ψ ◦ ξ ( x ) is jointG -invariant . In fact,

Therefore, by Theorem 3, assuming that the regular representation π : G →U ( L 2 ( X )) is irreducible, the ridgelet transform is given by

satisfying NN ◦ R = ( ( σ, ρ ) ) Id L 2 ( X ) .

5.2 Depth Separation

To enjoy the advantage of abstract formulation, we discuss the effect of depth. For the sake of simplicity, we assume G to be a finite group, which may be acceptable given that the data domain X in practice is often discretized (or coarse-grained) into finite sets of representative points, say X ≈ X := { x i } p i =1 , and if so the G -action is also reduced to finite representative actions.

Next, let us define the width of a single layer G as the cardinality | G | . This is reasonable because the set G parametrizes each map g : X → X . Then, under the assumption that each G i is simple, the depthn network G 1 ⋊ · · · ⋊ G n can express the same class of depth-1 network exponentiallyeffectively, because the total widths are ∑ n i =1 | G i | = O ( n ) for depthn and ∏ n i =1 | G i | = exp O ( n ) for depth-1. This estimate can be interpreted as the classical thought that the hierarchical models such as deep networks can represent complex functions combinatorially more efficient than shallow models.

Following the concept of the formal deep network, we call group G acting on X a network. Let us consider depth-1 network G and depthn network G 1 ⋊ · · · ⋊ G n satisfying G = G 1 ⋊ · · · ⋊ G n . The equation indicates that two networks have the same expressive power, because they can implement the same class of maps g : X → X .

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

6 Discussion

We have developed a systematic method for deriving a ridgelet transform for a wide range of learning machines defined by joint-group-equivariant feature maps, yielding the universal approximation theorems as corollaries. The previous results by Sonoda et al. [33] was limited to scalar-valued joint-invariant functions, which were insufficient to deal with practical learning machines defined by composite mappings of vector-valued functions, such as deep neural networks. For example, they could only deal with abstract composite structures like formal deep network [32]. By extending their argument to vector-valued joint-equivariant functions, we were able to deal with deep structures. Traditionally, the techniques used in the expressive power analysis of deep networks were different from those used in the analysis of shallow networks, as overviewed in the introduction. Nonetheless, our main theorem cover both deep and shallow networks from the unified perspective (joint-groupaction on the data-parameter domain). Technically, this unification is due to Schur's lemma, a basic and useful result in the representation theory. Thanks to this lemma, the proof of the main theorem is simple, yet the scope of application is wide. The significance of this study lies in revealing the close relationship between machine learning theory and modern algebra. With this study as a catalyst, we expect a major upgrade to machine learning theory from the perspective of modern algebra.

6.1 Limitations

In the main theorem, we assume the following: (1) joint-equivariance of feature map ϕ , (2) boundedness of composite operator NN ◦ R , (3) irreducibility of unitary representation π . In addition, throughout this study, we assume (4) local compactness of group G , and (5) that the network is given by the integral representation.

As discussed in the main text, satisfying (1) is much easier than (non-joint) equivariance. Also, (2) is often a textbook excersise when the specific expression is given. (3) is required for Schur's lemma, and it is often sufficient to synthesize the known results such as the one for the example of depthn fullyconnected network. (4) is quite a frequent assumption in the standard group representation theory, but it excludes infinite-dimensional groups. When formulated natively , nonparametric learning models including DNN can be infinite-dimensional groups. However, from the perspective of learnability, it is nonsense to consider too large a model, and it is common to assume regularity conditions such as sparsity and low rank in usual theoretical analysis. So, it is natural to impose additional regularity conditions for satisfying local compactness. (5) may be rather an advantage because there are established techniques to show the cc -universaity of finite models by discretizing integral representations. Moreover, there is a fast discretization scheme called the Barron's rate based on the quasi-Monte Carlo method. On the other hand, problems like the minimum width in the field of deep narrow networks are analyses of finite parameters, and they could be a different type of parameters. Yet, the current mainstream solutions are the information theoretic method by Park et al. [23] and the neural ODE method by Cai [2], and both arguments contain the discretization of continuous models. Therefore, we may expect a high affinity with the integral representation theory.

This study is the first step in extending the harmonic analysis method, which was previously applicable only to shallow models, to deep models. The above limitations will be resolved in our future works.

7 Broader Impact

This work studies theoretical aspects of neural networks for expressing square integrable functions. Since we do not propose a new method nor a new dataset, we expect that the impact of this work on ethical aspects and future societal consequences will be small, if any. Our work can help understand the theoretical benefit and limitations of neural networks in approximating functions. Our work and the proof technique improve our understanding of the theoretical aspect of deep neural networks and other learning machines used in machine learning, and may lead to better use of these techniques with possible benefits to the society.

References

M. M. Bronstein, J. Bruna, T. Cohen, and P. Veliˇ ckovi´ c. Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges. arXiv preprint: 2104.13478 , 2021.

Y. Cai. Achieve the Minimum Width of Neural Networks for Universal Approximation. In The Eleventh 330 International Conference on Learning Representations , 2023. 331

E. J. Candès. Ridgelets: theory and applications . PhD thesis, Standford University, 1998. 332

R. T. Q. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud. Neural Ordinary Differential Equations. In 333 Advances in Neural Information Processing Systems , volume 31, pages 6572-6583, Palais des Congrès de 334 Montréal, Montréal CANADA, 2018. 335

L. Chizat and F. Bach. On the Global Convergence of Gradient Descent for Over-parameterized Models 336 using Optimal Transport. In Advances in Neural Information Processing Systems 32 , pages 3036-3046, 337 Montreal, BC, 2018. 338

A. Cohen, R. DeVore, G. Petrova, and P. Wojtaszczyk. Optimal Stable Nonlinear Approximation. Founda339 tions of Computational Mathematics , 22(3):607-648, 2022. 340

N. Cohen, O. Sharir, and A. Shashua. On the Expressive Power of Deep Learning: A Tensor Analysis. In 341 29th Annual Conference on Learning Theory , volume 49, pages 1-31, 2016. 342

I. Daubechies, R. DeVore, S. Foucart, B. Hanin, and G. Petrova. Nonlinear Approximation and (Deep) 343 ReLU Networks. Constructive Approximation , 55(1):127-172, 2022. 344

W. E. A Proposal on Machine Learning via Dynamical Systems. Communications in Mathematics and 345 Statistics , 5(1):1-11, 2017. 346

G. B. Folland. A Course in Abstract Harmonic Analysis . Chapman and Hall/CRC, New York, second 347 edition, 2015. 348

P. Grohs, A. Klotz, and F. Voigtlaender. Phase Transitions in Rate Distortion Theory and Deep Learning. 349 Foundations of Computational Mathematics , 23(1):329-392, 2023. 350

E. Haber and L. Ruthotto. Stable architectures for deep neural networks. Inverse Problems , 34(1):1-22, 351 2017. 352

B. Hanin and M. Sellke. Approximating Continuous Functions by ReLU Nets of Minimal Width. arXiv 353 preprint: 1710.11278 , 2017. 354

P. Kidger and T. Lyons. Universal Approximation with Deep Narrow Networks. In Proceedings of Thirty 355 Third Conference on Learning Theory , volume 125 of Proceedings of Machine Learning Research , pages 356 2306-2327. PMLR, 2020. 357

N. Kim, C. Min, and S. Park. Minimum width for universal approximation using ReLU networks on 358 compact domain. In The Twelfth International Conference on Learning Representations , 2024. 359

L. Li, Y . Duan, G. Ji, and Y . Cai. Minimum Width of Leaky-ReLU Neural Networks for Uniform Universal 360 Approximation. In Proceedings of the 40th International Conference on Machine Learning , volume 202 of 361 Proceedings of Machine Learning Research , pages 19460-19470, 2023. 362

Q. Li and S. Hao. An Optimal Control Approach to Deep Learning and Applications to Discrete-Weight 363 Neural Networks. In Proceedings of The 35th International Conference on Machine Learning , volume 80, 364 pages 2985-2994, Stockholm, 2018. PMLR. 365

H. Lin and S. Jegelka. ResNet with one-neuron hidden layers is a Universal Approximator. In Advances in 366 Neural Information Processing Systems , volume 31, Montreal, BC, 2018. 367

Z. Lu, H. Pu, F. Wang, Z. Hu, and L. Wang. The Expressive Power of Neural Networks: A View from the 368 Width. In Advances in Neural Information Processing Systems , volume 30, 2017. 369

S. Mei, A. Montanari, and P.-M. Nguyen. A mean field view of the landscape of two-layer neural networks. 370 Proceedings of the National Academy of Sciences , 115(33):E7665-E7671, 2018. 371

N. Murata. An integral representation of functions using three-layered networks and their approximation 372 bounds. Neural Networks , 9(6):947-956, 1996. 373

A. Nitanda and T. Suzuki. Stochastic Particle Gradient Descent for Infinite Ensembles. arXiv preprint: 374 1712.05438 , 2017. 375

S. Park, C. Yun, J. Lee, and J. Shin. Minimum Width for Universal Approximation. In International 376 Conference on Learning Representations , 2021. 377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

G. Petrova and P. Wojtaszczyk. Limitations on approximation by deep and shallow neural networks. Journal of Machine Learning Research , 24(353):1-38, 2023.

G. Rotskoff and E. Vanden-Eijnden. Parameters as interacting particles: long time convergence and asymptotic error scaling of neural networks. In Advances in Neural Information Processing Systems 31 , pages 7146-7155, Montreal, BC, 2018.

J. W. Siegel. Optimal Approximation Rates for Deep ReLU Neural Networks on Sobolev and Besov Spaces. Journal of Machine Learning Research , 24(357):1-52, 2023.

S. Sonoda and N. Murata. Transportation analysis of denoising autoencoders: a novel method for analyzing deep neural networks. In NIPS 2017 Workshop on Optimal Transport & Machine Learning (OTML) , pages 1-10, Long Beach, 2017.

S. Sonoda, I. Ishikawa, and M. Ikeda. Ridge Regression with Over-Parametrized Two-Layer Networks Converge to Ridgelet Spectrum. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics (AISTATS) 2021 , volume 130, pages 2674-2682. PMLR, 2021.

S. Sonoda, I. Ishikawa, and M. Ikeda. Ghosts in Neural Networks: Existence, Structure and Role of Infinite-Dimensional Null Space. arXiv preprint: 2106.04770 , 2021.

S. Sonoda, I. Ishikawa, and M. Ikeda. Universality of Group Convolutional Neural Networks Based on Ridgelet Analysis on Groups. In Advances in Neural Information Processing Systems 35 , pages 38680-38694, New Orleans, Louisiana, USA, 2022.

S. Sonoda, I. Ishikawa, and M. Ikeda. Fully-Connected Network on Noncompact Symmetric Space and Ridgelet Transform based on Helgason-Fourier Analysis. In Proceedings of the 39th International Conference on Machine Learning , volume 162, pages 20405-20422, Baltimore, Maryland, USA, 2022.

S. Sonoda, Y. Hashimoto, I. Ishikawa, and M. Ikeda. Deep Ridgelet Transform: Voice with Koopman Operator Proves Universality of Formal Deep Networks. In Proceedings of the 2nd NeurIPS Workshop on Symmetry and Geometry in Neural Representations , Proceedings of Machine Learning Research. PMLR, 2023.

S. Sonoda, H. Ishi, I. Ishikawa, and M. Ikeda. Joint Group Invariant Functions on Data-Parameter Domain Induce Universal Neural Networks. In Proceedings of the 2nd NeurIPS Workshop on Symmetry and Geometry in Neural Representations , Proceedings of Machine Learning Research. PMLR, 2023.

S. Sonoda, I. Ishikawa, and M. Ikeda. A unified Fourier slice method to derive ridgelet transform for a variety of depth-2 neural networks. Journal of Statistical Planning and Inference , 233:106184, 2024.

T. Suzuki. Generalization bound of globally optimal non-convex neural network training: Transportation map estimation by infinite dimensional Langevin dynamics. In Advances in Neural Information Processing Systems 33 , pages 19224-19237, 2020.

M. Telgarsky. Benefits of depth in neural networks. In 29th Annual Conference on Learning Theory , pages 1-23, 2016.

D. Yarotsky. Error bounds for approximations with deep ReLU networks. Neural Networks , 94:103-114, 2017.

D. Yarotsky. Optimal approximation of continuous functions by very deep ReLU networks. In Proceedings of the 31st Conference On Learning Theory , volume 75 of Proceedings of Machine Learning Research , pages 639-649. PMLR, 2018.

D. Yarotsky and A. Zhevnerchuk. The phase diagram of approximation rates for deep neural networks. In Advances in Neural Information Processing Systems , volume 33, pages 13005-13015, 2020.

A Depth2 Fully-Connected Neural Network and Ridgelet Transform

A non group theoretic proof by reducing to a Fourier expression is given in Sonoda et al. [29, 421 Theorem 6]. 422

A.1 Proof 423

424

425

426

427

In the following, we identify the group G acting on data domain R m with the affine group Aff( R m ) , and introduce the so-called twisted dual group action that leaves a function θ invariant. Then, we see the regular action π of G on functions space L 2 ( R m ) commutes with composite S σ ◦ R ρ . Hence, by Schur's lemma, S σ ◦ R ρ is a constant multiple of identity, which concludes the assertion.

428

429

430

431

432

433

Proof. Let G be the affine group Aff( R ) = GL ( R ) ⋉R . For any g = ( L, t ) G , let be its action on R m , and let

be its left-regular action on L 2 ( R m ) .

Besides, putting

we define the twisted dual action of G on R m × R as so that the following invariance hold:

To see this, use matrix expressions with extended variables 434

which suggests ˜ g · ( a , b ) := ˜ I ˜ L -⊤ ˜ I ˜ a , and we have 436

Further, we define its regular-action by 437

Then we can see that, for all g = ( L, t ) ∈ G , 438

In fact, at every g = ( L, t ) ∈ G and ( a , b ) ∈ R m × R , 439

by putting x = g · y = L y + t with d x = | det L | d y , 440

and calculate 435

for all f ∈ L 2 ( R m ) . 447

450

A.2 Proof for (33) 451

Use matrix expressions with extended variables 452

which suggests ˜ g · ( a , b ) := ˜ I ˜ L -⊤ ˜ I ˜ a , and we have 454

and calculate 453

Similarly, at every g = ( L, t ) ∈ G and x ∈ R m , 441

by putting ( a , b ) := g · ( ξ , η ) = ( L -⊤ ξ , η + ξ · ( L -1 t )) with d a d b = | det L | d ξ d η , 442

Hence S σ ◦ R ρ commutes with π ( g ) because 443

Since S σ ◦ R ρ : L 2 ( R m ) → L 2 ( R m ) is bounded (Lemma 4), and ( π, L 2 ( R m )) is an irreducible 444 unitary representation of G (Lemma 3), Schur's lemma (Theorem 2) yields that there exist a constant 445 C σ,ρ ∈ C such that 446

Finally, by directly computing the left-hand-side, namely S σ ◦ R ρ [ f ] , we can verify that the constant 448 C σ,ρ is given by 449

A.3 Proof for (39) 455

In fact, at every g = ( L, t ) ∈ G and ( a , b ) ∈ R m × R , 456 R ρ [ π ( g )[ f ]]( a , b ) = | det L | -1 / 2 ∫ R m f ( g -1 · x ) ρ ( θ (( a , b ) , x ))d x by putting x = g · y = L y + t with d x = | det L | d y , 457 = | det L | 1 / 2 ∫ R m f ( y ) ρ ( θ (( a , b ) , g · y )))d y = | det L | 1 / 2 ∫ R m f ( y ) ρ ( θ ( g -1 · ( a , b ) , y )))d y = ̂ π ( g )[ R ρ [ f ]]( a , b ) . (47) Similarly, at every g = ( L, t ) ∈ G and x ∈ R m , 458 S σ [ ̂ π ( g )[ γ ]]( x ) = | det L | 1 / 2 ∫ R m × R γ ( g -1 · ( a , b )) σ ( θ (( a , b ) , x ))d a d b by putting ( a , b ) := g · ( ξ , η ) = ( L -⊤ ξ , η + ξ · ( L -1 t )) with d a d b = | det L | d ξ d η , 459 = | det L | -1 / 2 ∫ R m × R γ ( ξ , η ) σ ( θ ( g · ( ξ , η ) , x ))d ξ d η = | det L | -1 / 2 ∫ R m × R γ ( ξ , η ) σ ( θ (( ξ , η ) , g -1 · x ))d ξ d η = π ( g )[ S σ [ γ ]]( x ) . (48)

460

461

462

463

464

465

466

B Geometric Interpretation of Dual Action for Original Ridgelet Transform

We explain a geometric interpretation of the dual action (33) in the previous section. We note that in general θ does not require any geometric interpretation as long as it is joint group invariant on data-parameter domain.

For each ( a , b ) ∈ R m × R , put ξ ( a , b ) := { x ∈ R m | a · x -b = 0 } . Then it is a hyperplane in R m through point x 0 = b a / | a | 2 with normal vector u := a / | a | .

o

ξ

(

a

, b

)

u

y

0

y

ξ

(

a

,

a

x

)

x

x

0

y

x

·

For any point y in the hyperplane ξ ( a , b ) , by definition a · y = b , thus 467

But this means a · x -b is a scaled distance between point x and hyperplane ξ ( a , b ) , 468

and further a scaled distance between hyperplanes ξ ( a , a · x ) through x with normal a / | a | and ξ ( a , b ) , 469

Now, we can interpret the invariant θ (( a , b ) , x ) := a · x -b in a geometric manner, that is, it is the 470 distance between point and hyperplane, or between hyperplanes. We note that we can regard entire 471 σ ( a · x -b ) -the distance modulated by both scaling and nonlinearity-as the invariant, say ϕ . 472

Furthermore, the dual action g · ( a , b ) is understood as a parallel translation of hyperplane ξ ( a , b ) to 473 ξ ( g · ( a , b )) so as to leave the scaled distance θ invariant, namely 474

Indeed, for any g = ( L, t ) ∈ G ,

475

meaning that the hyperplane with parameter ( a , b ) translated by g is identical to the hyperplane with 476 parameter g · ( a , b ) . 477

To summarize, in the case of fully-connected neural network (and its corresponding ridgelet trans478 form), the invariant is a modulated distance σ ( a · x -b ) , and the dual action is the parallel translation 479 of hyperplane so as to keep the distance invariant. Further, from this geometric perspective, we can 480 rewrite the fully-connected neural network in a geometric manner as 481

where a ∈ R denotes signed scale and Ξ denotes the space of all hyperplanes (not always through 482 the origin). Since each hyperplane is parametrized by normal vectors u ∈ m -1 and distance p ≥ 0 483 from the origin, we can induce the product of spherical measure d u and Lebesgue measure d p as a 484 measure d ξ on the space Ξ of hyperplanes. 485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Theorem 3 and Corollary 1

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: § 6.1

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We put the proof right after Theorem 3

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This study does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA] .

Justification: This study does not include experiments.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This study does not include experiments

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This study does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This study does not include experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: § 7

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This study does not contain any code, data nor trained model

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This study does not contain any code, data nor trained model

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

747

748

749

750

751

752

753

754

755

756

757

758

759

760

761

762

763

764

765

766

767

768

769

770

771

772

773

774

775

776

777

778

779

780

781

782

783

784

785

786

787

788

789

790

791

Answer: [NA]

Justification: This study does not provide any code, data nor trained model

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This study does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This study does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a unified approach to universal approximation theorems for neural networks using group representation theory. It extends to vector-valued joint-group-equivariant feature maps, providing a systematic method for both shallow and deep neural networks with nonlinear activation functions. By leveraging Schur's lemma, the paper shows that these networks can universally approximate any function within a certain class. It main contribution is the closed-form ridgelet transform, which offers a constructive proof and explicit parameter distribution for these networks.\n\nSTRENGTHS:\n1.The paper introduces a unified constructive universal approximation theorem that applies to both shallow and deep neural networks using group representation theory. This is an innovative approach. It also extends previous work by incorporating vector-valued joint-group-equivariant feature maps.\n\n2.  The paper is theoretically sounding, leveraging concepts from group representation theory and Schur's lemma. They perform the thorough and systematic development of the ridgelet transform, providing a closed-form solution for parameter distributions and ensuring the findings are theoretically well justified.\n\n3. The paper is well-structured and clearly written. Definitions, theorems, and proofs are presented in a coherent manner, making it easier for readers to follow the details of the argument and understand the implications of the results.\n\n4.  This work is significant since it provides a relationship between deep learning theory and modern algebra. By providing a unified framework that applies to a wide range of network architectures, the paper incentivize further research and development in the field of machine learning.\n\nWEAKNESSES:\n1.  While the paper is strong in its theoretical contributions, it lacks empirical validation through experiments or simulations. Demonstrating the practical applicability and effectiveness of the proposed ridgelet transform and the unified framework on real-world datasets or benchmark problems would strengthen the paper. Including even a small set of experiments could provide evidence of the practical relevance and performance of the theoretical results.\n\n2. This work makes several assumptions, such as the local compactness of the group \\( G \\) and the boundedness of the composite operator \\( \\text{NN} \\circ R \\). While these assumptions are standard in group representation theory, the paper could benefit from a more detailed discussion on their implications and limitations. Exploring scenarios where these assumptions might not hold or providing guidance on how to relax these assumptions.\n\n3. Some of the technical details, particularly those related to advanced concepts in group representation theory and the ridgelet transform, might be challenging for readers who are not experts in these areas. Providing additional intuitive explanations, diagrams, or examples to illustrate these concepts could enhance the clarity of the paper.\n\nQUESTIONS:\n1. The work introduces a theoretically robust framework for universal approximation using the ridgelet transform and group representation theory. How feasible is it to implement these theoretical constructs in practical neural network architectures? Have the authors considered the computational complexity and resource requirements for applying these methods to real-world datasets, and if so, what strategies or approximations can be employed to make this approach computationally efficient?\n\n2. The main results rely on several key assumptions, such as the local compactness of the group \\( G \\) and the irreducibility of the unitary representation \\( \\pi \\). How robust are the findings to deviations from these assumptions? Specifically, can the authors provide more insights or alternative approaches for cases where these assumptions might not hold, such as in networks involving infinite-dimensional groups or non-compact groups?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work generalizes the ridgelet transform to equivariant neural networks, providing constructive proofs of universality in the general case as integrations over parameter distributions. Although such a direction had been taken up in prior work [33], they generalize it from scalar activations to vector activations, therefore encompassing more practical equivariant networks. The authors consider the form of the ridgelet transform for deep networks, and groups including the affine group and orthogonal group.\n\nSTRENGTHS:\nThe authors provide a constructive universal approximation result, which is in contrast to many non-constructive universality results. They strictly improve on the past work of Sonoda et al [33] by extending from scalars to vectors, which is more realistic. They consider the implications of their framework on depth separations for equivariant networks.\n\nWEAKNESSES:\nSignificance/novelty: The novelty relative to Sonoda et al [33] is limited, and the significance of this work to the universality and equivariance literatures is unclear. For example, many universality results already exist in equivariance (see e.g. work by Yarotsky [3], by Dym et al [2], etc.) — it is not clear how much value this extension of the ridgelet transform adds. \n\n\nClarity: I found the writing of the paper extremely hard to follow. It did not provide sufficient background on the ridgelet transform, universality results for equivariant networks (whether constructive or non-constructive), or perhaps most importantly, motivation for why one should value constructive approximation theorems for equivariant networks. It felt that one had to have read the previous works by Sonoda et al, in order to grasp why this work was important or where its novelty was, such as how vector-valued equivariant feature maps are superior to scalar-valued feature maps, what exactly formal networks are, what the practical use or theoretical value of the ridgelet transform is, etc. The work would also benefit from an outline of the sections earlier in the paper, and a more concise and early statement of what the authors consider their main theorem/s. It was not clear what the central result about the ridgelet transform was, as the transform seemed to still involve an integral in all equivariant cases, without simplification. \n\nAs a demonstration of the power of their theoretical formulation, the authors claim to show a depth separation, in which some class of networks is exponentially wide when shallow (constant number of layers), and only linearly wide when deep (linear number of layers). However, it is not clear whether they show that any shallow network is exponentially wide when representing a given function, or just the one constructed by the ridgelet transform — is this a strict depth separation?\n\nMathematical rigor: Although I did not check all of the math, some glaring errors stood out to me. First, the proof of Lemma 5 begins with, “Recall that a tensor product of irreducible representations is irreducible.” This is incorrect — for example, the tensor product of the irreps of the group of 3D rotations, SO(3), are reducible, and the irreps that appear in the decomposition of their tensor products are famously given by the Clebsch-Gordan coefficients (see e.g. [1]). Moreover, in the limitations section (6.1), the authors discuss the assumption that the group is locally compact, but say that this “excludes infinite-dimensional groups”. Yet, this is also false: for example, the infinite group SO(3) is compact (and therefore locally compact). In fact, several of the authors’ examples pertain to infinite groups, such as the affine group. These errors are surprising. \n\nAlso, the mathematical techniques themselves do not appear to be novel (for instance, Schur’s lemma is quite standard, and the proofs included in the main body are rather simple — Lemmas 1 and 2 are in fact widely known), and there are no experiments or practical implications, so the merit of the paper must rest on the significance of the results themselves. Unfortunately, the the broader significance of the results are not clearly demonstrated. The authors claim to reveal “the close relationship between machine learning theory and modern algebra,” but the mathematical tools they use seem like the standard ones used already throughout the equivariance literature. I am not sure what the “major upgrade to machine learning theory from the perspective of modern algebra” will therefore be. \n\n[1] Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network by Kondor, Lin, and Trivedi 2018\n\n[2] On the Universality of Rotation Equivariant Point Cloud Networks by Nadav Dam and Haggai Maron 2020\n\n[3] Universal approximations of invariant maps by neural networks by Dmitry Yarotsky 2018\n\nQUESTIONS:\n1. Do the techniques in this paper enabling proving universality of any networks for which universality (even non-constructive) was not already known? If so, it would be great to highlight these cases. \n\n2. Can the authors clarify the two mathematical errors pointed out in the previous section (on the tensor product of irreps, and on local compactness)?\n\n3. Is the depth separation a strict depth separation, I.e. is it the case that every shallow network representing the hypothetical function has to be exponentially wide?\n\n4. Is there intuition for why semi-direct product arises in formal deep networks? And intuition for how formal deep networks differ from standard neural networks?\n\nLIMITATIONS:\nYes, the authors discussed limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a generalization of the work by Sonoda et al. by extending their formulation of universal approximation theorems applicable to a specific class of neural networks namely scalar-valued joint-group-invariant feature maps for \"formal deep network\" to a much larger class of learning machines. Their theory using tools from group representation theory allows them to uniformly treat both shallow and deep neural networks with a larger class of activation functions. They provide an explicit construction for parameter assignment (aka Ridgelet Transform) and apply it to vector valued joint group-equivariant feature maps.\n\nSTRENGTHS:\n- The topic is well motivated and the writing is clear and understandable. The interspersed explanations in plain english are quite helpful in understanding a paper that leans quite heavily on sophisticated mathematical formalisms. (eg line 93-94). \n- The proofs and the notation are clear and succinct.\n- The authors extend an earlier work to a much more practical and real world class of NNs by introducing *vector-valued joint group-equivariant* feature maps, which yields universal approximation theorems as corollaries. They also unify the treatment of both shallow and deep networks by leveraging Schur's Lemma.\n- They provide explicit examples for depth 2 and depth $n$ fully connected network with an arbitrary activation in Section 4.2 which helps ground their method and significantly helps the reader understand how to leverage the tooling introduced by the authors.\n- The paper provides formal support for the popular interpretation for the efficacy of DNNs compared to shallow networks, namely that they construct hierarchical representations which would take an exponential number of neurons to represent using a single layer.\n- The limitations section is well written and is explicit about the assumptions made so that the reader is aware of the regime in which the proofs are applicable.\n\nWEAKNESSES:\n**Major**\n- The biggest weakness of the work seems to be that it shares a vast amount of technical analysis, machinery and the fundamental proofs are shared with the earlier work by Sonoda et al. While the extension to a larger class of networks and the introduced vector values feature maps is certainly valuable, I am not fully convinced of the differential novelty of the work. Most of the (valuable) effort has been spent in a mostly natural extension of the previous work on the topic.\n\n\n**Minor**\n-  The authors mention that assumption (5) (that the network is given by the integral representation) in limitations is potentially an \"advantage\". If that is so, a discretized version would be the preferred model since it is also closer to real world NNs\n- Typo on line 77  - mathmatical -> mathematical\n- Typo on line 310 - cc-universaity -> cc-universality\n- lines 135 - 137 would be significantly easier to read when broken into multiple lines\n\nQUESTIONS:\n- The authors allude to cc-universality in the Limitations section, can you briefly explain the term and is it the same as defined in Micchelli et al., 2006 etc\n- See major weakness\n\nLIMITATIONS:\nNo limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-31"}