Universal approximation and model compression for radial neural networks
Authors: Iordan Ganev
Twan van Laarhoven
Robin Walters
Venue: NeurIPS 2022 Submitted (2022)
Keywords: universal approximation
model compression
radial functions
parameter space symmetries
projected gradient descent

Abstract: We introduce a class of fully-connected neural networks whose activation functions, rather than being pointwise, rescale feature vectors by a function depending only on their norm. We call such networks radial neural networks, extending previous work on rotation equivariant networks that considers rescaling activations in less generality. We prove universal approximation theorems for radial neural networks, including in the more difficult cases of bounded widths and unbounded domains. Our proof techniques are novel, distinct from those in the pointwise case. Additionally, radial neural networks exhibit a rich group of orthogonal change-of-basis  symmetries on the vector space of trainable parameters. Factoring out these symmetries leads to a practical  lossless model compression algorithm. Optimization of the compressed model by gradient descent is equivalent to projected gradient descent for the full model.

Universal approximation and model compression for radial neural networks
Anonymous Author(s)
Affiliation
Address
email
Abstract
We introduce a class of fully-connected neural networks whose activa- tion functions, rather than being pointwise, rescale feature vectors by a function depending only on their norm. We call such networks radial neural networks, extending previous work on rotation equivariant net- works that considers rescaling activations less generality. We prove universal approximation theorems for radial neural networks, including in the more difficult cases of bounded widths and unbounded domains. Our proof techniques are novel, distinct from those in the pointwise case. Additionally, radial neural networks exhibit rich group of orthogonal change-of of-b-sis symmetries on the vector space of trainable parameters. Factoring out these symmetries leads to practical lossless model com- pression algorithm. Optimization of the compressed model by gradient descent is equivalent to projected gradient descent for the full model.
5
11
12
Introduction
14
15 Inspired by biological neural networks, the theory of artificial neural networks has largely focused on pointwise (or "local") nonlinear layers [46, 14], in which the same function IR is applied to each coordinate independently:
17 o: IR
(1.1) 18 In networks with pointwise nonlinearities, the standard basis vectors in IR" can be inter- 19 preted as "neurons" and the nonlinearity as "neuron activation." " Research has generally 20 focused on finding functions which lead to more stable training, have less sensitivity to initialization, are better adapted to certain applications [42, 38, 37, 10, 29]. Many o have 22 been considered, including sigmoid, ReLU, arclangent, ELU, Swish, and others.
R" R" V (v.,..., vn) + (o(v)), o(v)), ((v))).
However, by setting aside the biological metaphor, it is possible to consider much broader class of nonlinearities, which are not necessarily pointwise, but instead depend 26 simultaneously on many coordinates. Freedom from the pointwise assumption allows one to design activations that yield expressive function classes with specific advantages. Additionally, certain choices of non-pointwise activations maximize symmetry in the 28 parameter space of the network, leading to compressibility and other desirable properties.
In this paper, introduce radial neural networks which employ non-pointwise nonlin- earities called radial rescaling activations. Such networks enjoy several provable properties including high model compressibility, symmetry in optimization, and universal approxi- mation. Radial rescaling activations defined by rescaling each vector by scalar that
s1
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

Furthermore, we prove that the loss of the compressed model after one step of gradient 60 descent is equal to the loss of the original model after one step of projected gradient descent. As explained below, projected gradient descent involves zeroing out certain parameter W 1- 0
W
W
W
P
0
U
U
Figure 1: (Left) Pointwise activations distinguish specific basis of each hidden layer and treat each coordinate independently, see equation 1.1. (Right) Radial rescaling activations rescale each feature vector by a function of the norm, see equation 1.2.
depends only on the norm of the vector:
33
p: R R"
D A(Do),,
(1.2)
where À is a scalar-valued function of the norm. Whereas in the pointwise setting, only the linear layers mix information between different components of the latent features, for radial rescaling, all coordinates of the activation output vector are affected by all coordinates of the activation input vector. The inherent geometric symmetry of radial rescalings makes them particularly useful for designing equivariant neural networks [55, 47, 56, 57]
35
36
38
39 We note that radial neural networks constitute simple and previously unconsidered type of multilayer radial basis functions network [4], namely, one where the number of hidden activation neurons (often denoted N) in each layer is equal to one. Indeed, pre-composing equation 1.2 with a translation and post-composing with a linear map, one obtains a special case of the local linear model extension of radial basis functions network.
41
42
44 In our first set of main results, we prove that radial neural networks are in fact universal approximators. Specifically, we demonstrate that any asymptotically affine function can be approximated with radial neural network, suggesting potentially good extrapolation 47 behavior. Moreover, this approximation can be done with bounded width. Our approach
45
48 to proving these results departs markedly from techniques used in the pointwise case. Additionally, our result is not implied by the universality property of radial basis functions 50 networks in general, and differs in significant ways, particularly in the bounded width property and the approximation of asymptotically affine functions.
51
In our second set of main results, we exploit parameter space symmetries of radial neural networks to achieve model compression. Using the fact that radial rescaling activations commute with orthogonal transformations, we develop a practical algorithm system- atically factor out orthogonal symmetries via iterated QR decompositions. This leads to 58 another radial neural network with fewer neurons in each hidden layer. The resulting model compression algorithm is lossless: the compressed network and the original network both have the same value of the loss function on any batch of training data.
53
54
57
59
62 values after each step of gradient descent. Although training the original network may
63 result in a lower loss function after fewer epochs, in many cases the compressed network takes less time per epoch to train and is faster in reaching a local minimum.
66 To summarize, our main contributions are:
A formalization of radial neural networks, new class of neural networks;
87
Universal approximations results for radial neural networks, including: a) approxi- mation of asymptotically affine functions, and b) bounded width approximation;
68
Implementation of lossless compression algorithm for radial neural networks;
2

---

70
A theorem providing the precise relationship between gradient descent optimiza- tion of the original and compressed networks.
71
72 2
Related work
Radial rescaling activations. As noted, radial rescaling activations special case of the activations used in radial basis functions networks [4]. Radial rescaling functions have the symmetry property of preserving vector directions, and hence exhibit rotation equivariance. Consequently, examples of such functions, such as the squashing nonlinearity and Norm- RELU, feature in the study of rotationally equivariant neural networks [55, 47, 56, 57, 26]. However, previous works apply the activation only along the channel dimension, and consider the orthogonal group O(1) only for = 2,3. In contrast, we consider a radial rescaling activation across the entire hidden layer, and (())--quivariance where иI is the hidden layer dimension. Our constructions echo the vector neurons formalism [15], in
74
75
78
79
81
B2 which the output of a nonlinearity is a vector rather than a scalar.
83 Universal approximation. Neural networks of arbitrary width and sigmoid activations
84 have long been known to be universal approximators [14]. Universality can also be achieved
65 by bounded width networks with arbitrary depth [36], and generalizes to other activations and architectures I24, 60, 43, 50]. While most work has focused on compact domains,
87 some recent work also considers non-compact domains [28, 54]. The techniques used for
88 pointwise activations do not generalize radial rescaling activations, where all activation output coordinates are affected by all input coordinates. Consequently, individual radial
90 neural network approximators of two different functions cannot be easily combined to an approximator of the sum of the functions. The standard proof of universal approximation for radial basis functions networks requires an unbounded increase the number of hidden activation neurons, and hence does not apply to the case of radial neural networks [40].
91
88
94
Groups and symmetry. Appearances of symmetry in machine learning have generally focused on symmetric input and output spaces. Most prominently, equivariant neural networks incorporate symmetry as an inductive bias and feature weight-sharing constraints based on equivariance. Examples include G-convolution, steerable CNN, and Clebsch- Gordon networks [13, 55, 11, 9, 30, 2, 58, 12, 57, 16, 31, 44]. By contrast, our approach to radial neural networks does not depend on symmetries of the input domain, output space, or feedforward mapping. Instead, we exploit parameter space symmetries and thus obtain more general results that apply to domains with no apparent symmetry.
96
97
99
100
101
102
Model compression. A major goal in machine learning is to find methods to reduce the number of trainable parameters, decrease memory usage, or accelerate inference and training 18, 61]. Our approach toward this goal differs significantly from most existing methods in that it is based on the inherent symmetry of network parameter spaces. One prior method is weight pruning, which removes redundant weights with little loss in accuracy [20, 3, 27]. Pruning can be done during training [18] or at initialization 134, 53]. Gradient-banl based pruning removes weights by estimating the increase in loss resulting from their removal [33, 22, 17, 39). A complementary approach is quantization, which decreases the bit depth of weights 159, 25, 19]. Knowledge distillation identifies small model mimicking the performance of a larger model [5, 23, 1]. Matrix Factorization methods replace fully connected layers with lower rank or sparse factored tensors [6, 7, 52, 32, 45, 35] and can often be applied before training. Our method involves a type of matrix factorization
103
104
105
106
107
108
109
110
112
113
114 based the QR decomposition however, rather than aim for rank reduction, we leverage
115 this decomposition to reduce hidden widths via change-of-basis operations on the hidden
116 representations. Close to our method are lossless compression methods which remove
117 stable neurons in ReLU networks [49, 48] or exploit permutation parameter space symmetry
118 to remove neurons [51]; our compression instead follows from the symmetries of the radial
119 rescaling activation. Finally, the compression results of [26], while conceptually similar to
120 ours, are weaker, as (1) the unitary group action on disjoint layers instead of moving through all layers, and (2) the results are only stated for the squashing nonlinearity.
3

---

(1) Step-ReI.U())
Shifted ReLU
1.5
Squashing function
0.5
-2
-1.5
0.5
0.5
1
1.5
2
0.5
-1
- 1.5
2
Figure 2: Examples of different radial rescaling functions in R', see Example 1.
122 3 Radial neural networks
123
In this section, we define radial rescaling functions and radial neural networks. Let
h:R- + R be function. For any 2 1, set:
1 IR" R
11()) h(lll)))))))))>>)))) |
125 for * O, and (0)
function p: R² R" is called radial rescaling function if p = (((m) for some piecewise differentiable R + R. Hence, p sends each input vector to
128
scalar multiple of itself, and that scalar depends only on the norm of the vector'. It is
127
easy to show that radial rescaling functions commute with orthogonal transformations.
Example 1. (1) Step-ReLU, where h(r) = r i 2 1 and 0 otherwise. In this case, the radial rescaling function is given by
130
p: R" R"
V 4サト v if |0| 2 1;
D U |0| <1
(3.1)
(2) The squashing function, where h(r) (12 (3) Shifted ReLU, where h(r) max(0,r b) for 0 and b is real number. See Figure We refer to [55] and the references therein for more examples and discussion of radial functions.
151
182
134
radial neural network with L layers consists of a positive integer n; indicating the width of each layer = 0, 0, L; the trainable parameters, comprising of matrix W; E R=x-=;- of weights and bias vector b, E RR for each i = 1,..., L; and radial rescaling function R" >> Rd for each i = 1, L. We refer to the tuple n = (no, ",,..,, n1) as the widths vector of the neural network. The hidden widths vector is n hid = 12, ... M1 1). The feedforward function F: RR - R of radial neural network is defined in the usual way as an iterated composition of affine maps and activations. Explicitly, set Fo idg" and recursively define the partial feedforward functions for i = L,
155
137
139
140
F Ro + R" x P; (W; F 1xx bb)
Then the feedforward function is F= FL- Radial neural networks are a special type of radial basis functions network; we explain the connection in Appendix F.
Remark 2. If b; 0 for all i, then the feedforward function takes the form F(x) = W (μ(x)x) where и R" - R is a scalar-valued function and W = Wt ... W1 E R the
145
product of the weight matrices. If any of the biases are non-zero, then the feedforward
146
function lacks such simple form.
A function + R that depends only on the norm vector is known as radial function. Radial rescaling functions rescale each vector according to the radial function + A([e)) = M(P)) This explains the connection to Equation 1.2.
[

---

Universal Approximation
148
149
In this section, we consider two universal approximation results. The first approxi- mates asymptotically affine functions with network of unbounded width. The second generalizes to bounded width networks. Proofs appear in Appendix B. Throughout, Br(c) {x E R" |x - cl < r denotes the r-ball around a point e, and an affine map RR + R" is one of the from L(x) Ax + b for matrix A E RR"x" and R
150
151
152
153
154
4.
Approximation of asymptotically affine functions
A continuous function f R" + R" is said to be asymptotically affine if there exists an affine map L: R > IR such that, for every é > 0, there is compact subset K of R" such that -
158
157
< e for all x E R" \ K. In particular, continuous functions with compact support are asymptotically affine. The continuity of f and compactness of K imply that, for any e > 0, there exist CN,..., E K and INN..., E (0, 1) such that, first, the union of the balls Br, (c) covers K and, second, for all i, we have f (B) S Be(f())). Let N(f, e) be the minimal? choice of N.
158
159
160
161
Theorem 3 (Universal approximation). Let f: R" R" be an asymptoticalty affine function. For any E > O, there exists a compact set K c IK" and a function F: R" - IR" such that:
163
164
1. F is the feedforward function of radial neural network with N K,) layers whose hidden widths are (n + 1, 11 + 2, ....,, n1 + N).
2. For any x E RR", we have [F(x) f(x) <.
168
167
We note that the approximation in Theorem 3 is valid on all of RR. To give an idea of the proof, first fix CN E and IN (0,1) as above. Let ep....,e be orthonormal basis vectors extending R" to RR+N, For = I,...,N N define affine maps
168
169
T; R + Ruti and S Ruti + R by
170
T2(2) = c; + h4;
S,(z) = z-
(1+ =- h (ej, + +
where 12? = 1-1 + and (e) is the coefficient of e; in z. Setting Pi to be Step-ReL.U (Equation 3.1) on Ru*1, these maps are chosen that the composition S; o; T; maps the points in B, to c; + e; while keeping points outside this ball the same. We now describe a radial neural network with widths (n, n + 1,..., + N, m) whose feedforward function approximates f. For i = N, N the affine map from layer i to layer i is given by Z + T, (z), with Sn- = idg=. The activation at each hidden layer is Step-ReLU, Let be the affine map such that [L- - fl < E on R" K. The affine map from layer N to the output layer is Φo SN where ① : R N + R" is the unique affine map determined by + L(x) if E R", and e; + f(;) L(c)). This construction illustrated in Figure 3.
171
172
178
174
175
176
177
178
179
180
Corollary 4. Radial neural networks dense in the space of all continuous functions with respect to the topology of compact convergence, and hence satisfy cc-universality.
182
Bounded width approximation
183
We now turn our attention to a bounded width universal approximation result.
164
Theorem 5. Let f: R² -> RM be an asymptotically affine function. For any e > 0, there exists
185
compact set K C IR" and function F: RR" + n such that:
166
1. F is the feedforward function of a radial neural network with N = N(f, K,e, hidden layers whose widths are all n + m +
187
2. For any x E R"" we have [F(x) fx)) <.
188
The proof, which is more involved than that of Theorem 3, relies on using orthogonal dimensions to represent the domain and the range of f, together with an indicator
169
Inn many cases, the constant N(f,K,)) e) can be bounded explicitly. For example, if K is the unit cube in R" and f is Lipschitz continuous with Lipschitz constant R, then N(f, - RvT

---

So T2
C7
82
C8
f(22)
K
C6
R
RR
Figure 3: Two layers of the radial neural network used in the proof of Theorem 3- (Left) The compact set K is covered with open balls. (Middle) Points close c2 (green ball) are mapped to + e2, all other points are kept the same. (Right) In the final layer, C2 + is mapped to f(2)).
dimension to distinguish the two. We regard points in R=+ as triples (x,y,0) where 182 x E R" E R" and E R. The proof of Theorem 5 parallels that of Theorem 3, but instead
(c;) to c; e we map the points in B,, ((; 0,0)) to (0, f()))((0) 1), where $ is chosen such that different balls do not interfere. The final layer then uses
196
of mapping points
194
affine map (x, y) > L(x) + sy, which takes (x, 0,0) to L(x), and (0, [(a)-L(0) 1) to f(c)..
196
We remark on several additional results; see Appendix for full statements and proofs. The bound of Theorem 5 can be strengthened to max(n, m) + in the case of functions K + RR" defined on compact domain K C R" (i.e., ignoring asymptotic behavior). Furthermore, with more layers, it is possible to reduce that bound to ma)
198
199
5 Model compression
201
In this section, we prove a model compression result. Specifically, we provide an algorithm which, given any radial neural network, computes different radial neural network with smaller widths. The resulting compressed network has the same feedforward function
202
204
as the original network, and hence the same value of the loss function on any batch of training data. In other words, our model compression procedure lossless. Although our algorithm is practical and explicit, it reflects more conceptual phenomena, namely, change-of-basic action on network parameter spaces (Section 5.1).
206
5.1 The parameter space
209 Suppose a fully connected network has layers and widths given by the tuple n 210 no H1, "1 nt).. In other words, the i-th layer has input width n;-1 and output width n, The parameter space is defined the vector space of all possible choices of parameter values. Hence, it is given by the following product of vector spaces:
Param(n) = (R" X x R x ... X R=) x (R" X R x ...
. x R"
We denote element therein pair of tuples (W,b) where W = (W, E R***-1){
214 are the weights and (b; E R are the biases. To describe certain symmetries of the parameter space, consider the following product of orthogonal groups, with sizes 216 corresponding to the widths of the hidden layers:
On O(m) x O(n2) .... ((H1-1)
There is change-of-base action of hid) on the parameter space Param(n), Explicitly, the tuple of orthogonal matrices Q (Q:)==1 e hidhid transforms the parameter values
218
219 (W,b) to Q- W (Q,W,0, and Q (Q,b,) where Qo idny and Qr. idny

---

5.2 Model compression
221 In order to state the compression result, we first define the reduced widths. Namely,
222 the reduction ned = (mgd, _n ...., (ed) of a widths vector n is defined recursively by setting n/d No, then nnnd min(n; nr.ed + 1) for i ,, 1, and finally nidd NI. For a tuple p (p:: IR" - R of radial rescaling functions, we write pted ( RM R for the corresponding tuple of restrictions, which are all radial rescaling functions. The following result relies on Algorithm 1 below.
226
Theorem 6. Let pW) be a radial neural network with widths n. Let Wred and bred be the weights and biases of the compressed network produced by Algorithm 1. The feedforward function of the original network (W, b, b,p) coincides with that of the compressed network (Wred, b pred, pred).
227
Algorithm 1: QR Model Compression (QR-compress)
input
W,, E (n)am(n)
output :Q E O(nhid) hid) and Wred breed E Param
Q W bred < [][]]]]
// initialize output lists |/ matrix of size "1 x (no + // iterate through layers // A; Q,Inc,
A1 + [bq Wi]
for i + 1 to L- do
Q R; t QR-decomp (A; mode = ccmmleeee)
Append Q to Q
230
Append first column of R to bred
// reduced bias for layer i // reduced weights for layer i // matrix of size Mi41 (n 1)
Append remainder of R; to Wred
Set Ai+1 + [b;+1 1,nnQ
end
Append the first column of AL to bred
// reduced bias for last layer // reduced weights for last layer
Append the remainder of A to Wred
return Q Wred, red
We explain the notation of the algorithm. The inclusion matrix Inc; Raxaa has ones along the main diagonal and zeros elsewhere. The method QR-decomp with mode complete' computes the complete QR decomposition of the n; (1 + nedd matrix A₁
)
234 as R, where Q; E O(n,) and R is upper-triangular of size nr.ed x (1 + nedd). The definition of nde implies that either nied nedd 1 niedd = n4 The matrix R, is of size ned x anned in the former case and of size n; x (1 + = in the latter case.
205
237
Example 7. Suppose the widths of radial neural network are (1,8, 16,8,1), 1). Then has - n1 1 + 1)n, = 305 trainable parameters. The reduced network has widths (1,2,3,4,1] and _ (n;o + 1) (nffed = 34 trainable parameters. Another example appears in Figure 4-
208
240
We note that the tuple of matrices Q produced by Algorithm I does not feature in the 241 statement of Theorem 6, but is important in the proof (which appears in Appendix C).
x
Figure 4: Model compression in 3 steps. Layer widths can be iteratively reduced to greater than the previous. The number of trainable parameters reduces from 33 to 17.

---

To state our results, fix tuple of widths n and a tuple p p= (p:: R -> R 1 of radial batch of training Namely, an induction argument shows that the i-th partial feedforward function of the 242
243
original and reduced models are related via the matrices Qi and Inc, A crucial ingredient in the proof is that radial rescaling activations commute with orthogonal transformations.
244
Projected gradient descent
248
The typical use case for model compression algorithms is to produce a smaller version of the fully trained model which can be deployed to make inference more efficient. It is also worth considering whether compression can be used to accelerate training- For example, for some compression algorithms, the compressed and full models have the same werd function after a step of gradient descent is applied to each, and so one can compress before training and still reach the same minimum. Unfortunately, in the context of radial neural networks, compression using Algorithm and then training does not necessarily give the same result as training and then compression (see Appendix D.6 for counterexample) However, QR- compresss does lead to a precise mathematical relationship between optimization of the two models: the loss of the compressed model after one step of gradient descent is equivalent to the loss of (a transformed version of) the original model after one step of projected gradient descent. Proofs appear in Appendix D.
247
248
250
251
253
254
256
257
258
250
rescaling functions. The loss function Param(n) - IR associated
data (xx, C "R R is defined as taking parameter values (W,b) to the sum 260
, (()( where R x R IR is a cost function the output space, and F( is the feedforward the radial neural network with parameters (W,b) and activations p. Similarly, we have loss function Gred on the parameter space Param(n with reduced widths vector. For any learning rate 1/ O, we obtain gradient descent maps:
262
263
Param (n) + Param(n)
Yred Param (n red + Param (n
(W, ~> (W,b) - -V (W.))
(V.c) > (V,c) - v (red
265 We will also consider, for k 2 0, the k-fold composition n = 7 yo.............. ................................ o and similarly
268 for Tred- The projected gradient descent map on Param(n) is given by:
Proj Param (n) Param(n),
(W,b) + Proj (7(W,
267 where the map Proj zeroes out all entries in the bottom left (n; (eiin need submatrix of
268 Wi V C, and the bottom (n; nieed entries in b; - V Vi,C, for each i. Schematically:
W, W
b V VbC
+
To state the following theorem, let Wred bed Q DR-compress: W,b) be the outputs 270 of Algorithm 1 applied to (W, E Param(n). Hence (Wredd, E Param(n are the parameters of the compressed model, and Q E O( is an orthogonal parameter symmetry. We also consider the action (Section 5.1) of Q applied to (W,b).
260
Theorem 8. Lef Wred, bred Q QR-compress(WW (W, be the outputs of Algorithm 1 applied to b) E Param(n) Set U = Q-1 11 (W, (Wred, bred). For any k 2 O, we have:
273
274
* (w. Q (Q 1. (W)
yprof (Q 1. (W, - (Wred bred + U.
We conclude that gradient descent with initial values (W,b) is equivalent to gradient 276 descent with initial values Q 1. b),b) since at any stage we can apply Q=1 to move from one to the other. Furthermore, projected gradient descent with initial values Q (W, b) equivalent gradient descent on Param (n red) with initial values (Wred bred since at any stage we can move from one to the other by =U. Neither Q nor U depends on k.
278
8

---

Experiments
280
In addition to the theoretical results in this work, we provide an implementation of Algorithm 1, in order to validate the claims of Theorems 6 and 8 empirically, as well as to quantify real-world performance. Full experimental details are in Appendix E.
282
283
(1) Empirical verification of Theorem 6. We learn the function -(x)=->* from samples using a radial neural network with widths n (1,6,7,1) and activation the radial shifted sigmoid h(x) = 1/ e Applying QR- QR-compress gives compressed radial neural network with widths n ned (1,2,3,1). Theorem 6 implies that the respective neural functions F and Fred are equal. Over 10 random initializations, the mean absolute error is negligible up to machine precision: (1/N) Ej [F(x)) Fred (xy) 1.31 10-8 + 4.45 10-9.
284
=
285
Applying
267
290
(2) Empirical verification of Theorem 8. The claim is that training the transformed model with parameters Q 1. (W, and objective L by projected gradient descent coincides with training the reduced model with parameters (Wred, brd)) and objective Led by usual gradient descent. We verified this on synthetic data as above. Over 10 random initializations, the loss functions after training match: --- rrdd = 4.02 10 1 + -001-10 10
252
290
(3) The compressed model trains faster. Our compression method may be applied before training to produce a smaller model class which trains faster without sacrificing accuracy,
296
296
We demonstrate this in learning the function f IR2 >> IR² sending (41,12 to i B) using& radial neural network with widths n (2, 16, 64, 128, 16,2) and activation the radial sigmoid h(r) = 11 +e 7). Applying cR-compress gives compressed network with widths red = (2,3,4,5,6,2). We trained both models until the training loss was
297
296
299
300
< 0.01. Over 1o random initializations on our system, the reduced network trained in
15.32 2.53 seconds and the original network trained in 31.24 + 4.55 seconds
302
Conclusions and Discussion
303
304
This paper demonstrates that radial neural networks are universal approximators and that their parameter spaces exhibit rich symmetry group, leading to a model compression algorithm. The results of this work combine to build a theoretical foundation for the use of radial neural networks, and suggest that radial neural networks hold promise for wider practical applicability. Furthermore, this work makes an argument for considering the ad advantages of non-pointwise nonlinearities in neural networks.
305
306
307
306
309
310
There are two main limitations of our results, each providing an opportunity for future work. First, our universal approximation constructions currently work only for Step-ReLU radial rescaling radial activations; it would be desirable to generalize to other activations.
311
313 Additionally, Theorem achieves compression only for networks whose widths satisfy
314 ni > n; 1 1+1 for some Neural networks which do not have increasing widths anywhere
315 in their architecture, such encoders, would not be compressible.
316
Further extensions of this work include: First, little is currently known about the stabil- ity properties of radial neural networks during training, as well as their sensitivity to initialization. Second, radial rescaling activations provide an extreme case of symmetry; there may be benefits to combining radial and pointwise activations within a single net- work, for example, through 'block' radial rescaling functions. Our techniques may yield weaker compression properties for more general radial basis functions networks; radial neural networks may be the most compressible such networks. Third, the parameter space symmetries may provide a key ingredient in analyzing the gradient flow dynamics of radial neural networks and computation of conserved quantities. Fourth, radial rescaling activations can be used within convolutional or group-equivariant NNs. Finally, based on the theoretical advantages laid out in this paper, future work will explore empirically
317
318
319
320
321
322
323
324
325
326
applications in which we expect radial networks to outperform alternate methods. Such potential applications include data spaces with circular or distance based class boundaries.
327

---

References
330
[1] Lei Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? arXio:131266184
351
2013. 3
S52
[2]
Erkao Bao and Linqi Song. ar ('Xi:11906.0717,, 2019. 3
Equivariant neural networks and equivarification.
388
384
[3] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? arXio:2003.03033, 2020. 3
$35
[4] David S Broomhead and David Lowe. Radial basis functions, multi-variable functional interpolation and adaptive networks. Technical report, Royal Signals and Radar Establishment Malvern (United Kingdom), 1988. 2, 3
357
398
[5] Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 535-541, 2006.
339
341
[6] Yu Cheng, X Yu Felix, Rogerio S Feris, Sanjiv Kumar, Alok Choudhary, and Shih-Fu Chang. Fast neural networks with circulant projections. arXiv:1502.03436, 2, 2015.
342
343
17| Yu Cheng, Felix X Yu, Rogerio S Feris, Sanjiv Kumar, Alok Choudhary, and Shi-Fu Chang. An exploration of parameter redundancy in deep networks with circulant projections. In Proceedings of the IEFE international conference on computer vision, pages
344
345
2857-2865, 2015. 3
347
348
[8] Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. arXio:1710.09282, 2017. 3
349
350
[9] Benjamin Chidester, Minh N. Do, and Jian Ma. Rotation equivariance and invariance in convolutional neural networks. arXio:1805.12301 2018. 3
352
10] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep
network learning by exponential linear units (elus). arviv preprint arXiv:1111.07289, 2015.
853
354
1
366
[11]
Taco S. Cohen and Max Welling. Group equivariant convolutional networks. In
International conference on machine learning (ICML), pages 2990-29998 2016. 3
356
357
12 Taco S Cohen and Max Welling. Steerable CNNs. In Proceedings of the International
Conference on Learning Representations (ICLR), 2017. 3
359
[13] Taco S. Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equiv- ariant convolutional networks and the icosahedral CNN. In Proceedings of the 36th International Conference on Machine Learning (ICML), volume 97, pages 1321-1330, 2019.
361
362
3
[14] George Cybenko. Approximation by superpositions of sigmoidal function. Mathe- matics of control, signals and systems, (4):303-3114, 1989. 1, 3
363
[15] Congyue Deng, O. Litany, Yueqi Duan, A. Poulenard, A. Tagliasacchi, and L. Guibas. Vector Neurons: A General Framework for SO(3)-Equivariant Networks. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021. doi: 10.1109/ iccv489p2.2021.01199 3
368
367
[16] Sander Dieleman, Jeffrey De Fauw, and Koray Kavukcuoglu. Exploiting cyclic symme- try in convolutional neural networks. In International Conference on Machine Learning (ICML), 2016.
370
371
[17] Xin Dong, Shangyu Chen, and Sinno Jialin Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. arXiv preprint ar 175701705.00555, 2017.
372
373
3
10

---

[18] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. ar arXio:1803.0365 2018. 3
378
[19] Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional networks using vector quantization. arXiv:1412.61115 2014. 3
378
20
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neu- ral networks with pruning, trained quantization and huffman coding. 51Xii:1500.0000:
380
381
2015.
3
382
[21]
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gerard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825):357-362, September 2020. doi: 10.1038/ 10.1038/s41566000
385
385
366
387
URL https:
389
aidoi.org/10 10w.////...111///// www.:/100///1880000 000-5666-20-2699--1 35
[22] Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon. Morgan Kaufmann, 1993- 3
391
[23] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. ar arXio:2503.0253, 2015. 3
388
[24] Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks, (2))2251-277, 1991.
396
[55] Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient con- volutional neural networks for mobile vision applications. arXiv:1704.04661, 2017.
397
398
359
3
400
[26] George Jeffreys and Siu-Cheong Lau. Kähler Geometry of Quiver Varieties and Machine Learning. arXit:2101.11487, 2021. URL http://arxiv,, wttp://www..rgg///////1//11 11487.
3
[27] Ehud D Karnin. A simple procedure for pruning back-propagatioon trained neural networks. IEEE transactions on neural networks, 122):239-242, 1990. 3
404
406
[28]
Patrick Kidger and Terry Lyons. Universal approximation with deep narrow networks. In Conference on learning theory, pages 2306-2327. PMLR, 2020. 3
407
[29] Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self- normalizing neural networks. Advances in neural information processing systems, 30,
409
2017.
1
[30] Risi Kondor and Shubhendu Trivedi. On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups. In International conference on machine learning (ICML), 2018. 3
411
412
413
[31] Leon Lang and Maurice Weiler. A Wigner-Eckart theorem for group equivariant convolution kernels. In International Conference on Learning Representations (ICLR), 2021.
414
415
3
[32] Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempit- sky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. arXinx1412.6553, 2014-
416
417
418
419
[33] Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pages 598-605, 1990. 3
420
11

---

[34] Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip HS Torr. A signal propagation perspective for pruning neural networks at initialization. arXiv preprint ar arXic:1906.06007, 2019. 3
422
423
[35] Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogerio Feris. Fully-adaptive feature sharing in multi-task networks with applications in person attribute classification. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 5334-5344 2017. 3
424
425
426
427
428
[36] Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: A view from the width. Advances in neural
429
430
information processing systems, 30, 2017. 3
431
137 Mirco Milletari, Thiparat Chotibut, and Paolo E Trevisanutto. Mean field theory of activation functions in deep neural networks. arXiv preprint orXtv:105500876, 2018.
432
433
[38] Diganta Misra. Mish: A self regularized non-monotonic activation function. arXiv preprint ar arXto:1908.08666 088681, 2019.
434
435
[39] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Prun- ing convolutional neural networks for resource efficient inference. arXiv preprint ar axXio:6611.06440, 2016. 3
438
440 Jooyoung Park and Irwin W Sandberg. Universal approximation using radial-basis- function networks. Neural computation, 3(2):246-257, 1991. 3
41 Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, d Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems (NeurIPS) 32, pages 8024-8035. Curran Associates, Inc., 2019. URL http:/ http://papers neurips cc/paper/ 1015-pytorch-ao an- imperative-style high seehh--prrrrmmnneeeeeee -ee---ennnne-- 1ibraryb..... pdf. 35
442
443
444
445
446
448
449
450
[42] Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. ar Xiv preprint arXiv:1710.05944, 2017. 1
451
[43] Siamak Ravanbakhsh. Universal equivariant multilayer perceptrons. In International 452
Conference on Machine Learning, pages 8006. PMLR, 2020. 3
453
454
[44] Siamak Ravanbakhsh, Jeff Schneider, and Barnabas Poczos. Equivariance through parameter-sharing. In International Conference on Machine Learning, pages 2892-2901.
455
PMLR, 2017.
457
[45] Roberto Rigamonti, Amos Sironi, Vincent Lepetit, and Pascal Fua. Learning separable filters. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2754-2761, 2013. 3
459
[46] Frank Rosenblatt. The perceptron: probabilistic model for information storage and organization in the brain. Psychological review, 65(6):386, 1958. 1
461
462
47 Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. arXioo1770.09822, 2017. 2,
463
464
[48]
Thiago Serra, Abhinav Kumar, and Srikumar Ramalingam. Lossless compression of deep neural networks. In International Conference on Integration of Constraint Pro- gramming, Artificial Intelligence, and Operations Research, pages 417-430. Springer, 2020.
466
466
3
12

---

[49] Thiago Serra, Xin Yu, Abhinav Kumar, and Srikumar Ramalingam. Scaling up exact neural network compression by relu stability. Advances Neural Information Processing Systems, 34, 2021. 3
469
470
[50] Sho Sonoda and Noboru Murata. Neural network with unbounded activation func- tions is universal approximator. Applied and Computational Harmonic Analysis, 43(2): 233-268, 2017.
471
472
[5t] Gustav Sourek, Filip Zelezny, and Ondrej Kuzelka. Lossless compression of structured convolutional models via lifting, arXiv preprint aXXiv:2007.06567, 2020. 3
52] Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, et al. Convolutional neural networks with low-rank regularization. arXiv:1511.06067 2015.
53 Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before
training by preserving gradient flow. arXiv preprint arXiv:2002.07376, 2020. 3
[54] Ming-Xi Wang and Yang Qu Approximation capabilities of neural networks unbounded domains. Neural Networks, 145:56-67 67, 2022.
55l Maurice Weiler and Gabriele Cesa. General E(2)-Equivariant Steerable CNNs. Confer- ence on Neural Information Processing Systems (NeurIPS), 2019. 2, 3,
483
[6)
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3D steerable CNNs: Learning rotationally equivariant features in volumetric data. Proceedings of the 32nd International Conference on Neural Information Processing Systems (NeurlPS), 2018. 2, 3
485
486
467
468
[57] Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant CNNs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 849-858, 2018. 3,
58 Daniel E Worrall, Stephan Garbin, Daniyar Turmukhambetov, and Gabriel Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5028-5037, 2017. 3
498
494
[59] Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng, Quantized convolutional neural networks for mobile devices. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4820-4828, 2016. 3
496
497
[60] Dmitry Yarotsky. Universal approximations of invariant maps by neural networks. Constructive Approximation, 55(1)4407474, 2022.
[6t] Tianyun Zhang, Shaokai Ye, Kaiqi Zhang, Jian Tang, Wujie Wen, Makan Fardad, and Yanzhi Wang. A systematic DNN weight pruning framework using alternating direction method of multipliers. In Proceedings of the European Conference on Computer Vision (ECCV), pages 184-199, 2018. 3
502
503
13

---

Checklist
506
1. For all authors...
506
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope? [Yes]
508
(b) Did you describe the limitations of your work? [Yes] See Section 8.
509
(c) Did you discuss any potential negative societal impacts of your work? [N/A] Our work theoretical and does not hold specific risks of negative impacts.
510
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
512
513
2. If you are including theoretical results...
514
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] Most of the proofs appear in the supplementary material.
516
517
3. If you ran experiments..
518
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or a URL)? [Yes]
519
(b) Did you specify all the training details (e.g,, data splits, hyperparameters, how they were chosen)? [Yes]
621
623
(c) Did you report error bars (e.g,, with respect the random seed after running experiments multiple times)? [Yes]
525
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix E.
4. If you using existing assets (e.g., code, data, models) curating/ rretiinn/eeeaiingg new assets...
528
529
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
531
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
593
(d) Did you discuss whether and how consent was obtained from people whose data you're seing/curating'" [N/A]
594
696
(e) Did you discuss whether the data you are using/ /cur/tirrnng contains personally
696
identifiable information or offensive content? [N/A]
597
5. If you used crowdsourcing or conducted research with human subjects...
598
(a) Did you include the full text of instructions given to participants and screen- shots, if applicable? [N/A]
540
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
542
14

---

A
Organization of the appendices
644
645
This paper is a contribution to the mathematical foundations of machine learning, and our results are motivated by expanding the applicabilityy and performance of neural networks. At the same time, we give precise mathematical formulations of our results and proofs. The purposes of these appendices are several:
548
548
549
1. To clarify the mathematical conventions and terminology, thus making the paper more accessible.
To provide full proofs of the main results.
3. To develop context around various construction appearing in the main text.
4. To discuss in detail examples, special cases, and generalizations of our results.
554
We now give summary of the contents of the appendices.
555
Appendix contains proofs the universal approximation results (Theorems 3 and 5) stated in Section of the main text, as well as proofs of additional bounded width results. The proofs use notation given in Appendix B.1, and rely on preliminary topological considerations given in Appendix B.2.
657
558
In Appendix C, we give proof of the model compression result given in Theorem 6, which appears in Section For clarity and background we begin the appendix with a discussion of the version of the QR decomposition relevant for our purposes (Appendix C.1). We also establish elementary properties of radial rescaling activations (Appendix C.2).
560
561
568
The focus of Appendix D is projected gradient descent, elaborating on Section We first prove result the interaction of gradient descent and orthogonal transformations (Appendix D.1), before formulating projected gradient descent in more detail (Appendix D.2), and introducing the so-called interpolating space (Appendix D.3). We restate Theorem
566
567
8 in more convenient notation (Appendix D.4) before proceeding the proof (Appendix D.5).
569
A Appendix E contains implementation details for the experiments summarized Section 7. Our implementations use shifted radial rescaling activations, which we formulate in Appendix E.1.
571
572
Appendix explains the connection between our constructions and radial basis functions networks. While radial neural networks turn out to be specific type of radial basis functions network, our universality results are not implied by those for general radial basis functions networks.
574
575
B Universal approximation proofs and additional results
577
In this section, we provide full proofs of the universal approximation (UA) results for radial neural networks, as stated in Section 4. In order do so, we first clarify our notational conventions (Appendix B.1), and collect basic topological results (Appendix B.2).
578
560
B.
Notation
Recall that, for point c in the Euclidean space R" and positive real number r, we denote the r-ball around c by (c(c) = {x E R" - < All networks in this section have the Step-ReLU radial rescaling activation function, defined as:
582
563
if 2 otherwise
p R
R"
Throughout, denotes the composition of functions. We identify linear map with corresponding matrix (in the standard bases). In the case of linear maps, the operation
584
15

---

566 can be be identified with matrix multiplication. Recall also that an affine map L : R² - R"
587 is one of the from L(x) = Ax + b for a matrix A E R=* and E R
B.2 Topology
Let K be compact subset of IR" and let f K - IR"h be continuous function.
Lemma 9. For any O, there exist CN E K and FI....,, E (0,1) such that, first, the union of the balls B,, (c;) covers K; second, all i, we have f (B,, n)) S (f(f(c))
691
552
Proof. The continuity of f implies that for each (Br, (c) C Be(f(c).. The subsets B,, (c) n K form an open cover of K. The compactness of implies that there is a finite subcover. The result follows.
K, there exists r re such that
504
We also prove variation of Lemma 9 that additionally guarantees that none of the balls in the cover of K contains the center point of another ball.
Lemma 10. For any > O, there exist Ci...., CM K and II,..,, E (0,1) such that, first, the union of the balls Br, (c:) covers K; second, for all i, we have f (B, (G)) C Be (f())); and, third,
598
590
C; Cj 2 ri.
Proof. Because f is continuous compact domain, it is uniformly continuous. So, there exists r > 0 such that ((B,(c) C Be ((()() for each c E K. Because is compact it has a finite volume, and so does 22 (K) = UceK Bc)2 Hence, there exists a finite maximal packing of (K)) with balls of radius r/2. That is, collection cI,...,M CM E (2)2(K) such that, for all i, (c( S (K) and, for all + B,/2(ci n B,/2(cj) = @. The first condition implies that E K. The second condition implies that |c; cyl 2 Finally, we argue that C 1 (c))). To see this, suppose, for a contradiction, that x E K does not belong to UM (c)). Then B,,2(Gi n (x)2 = Ø, and x could be added to the packing, which contradicts the fact that the packing was chosen to be maximal. So the union of the balls (c;) covers K.
601
802
608
606
608
600
610
We turn our attention to the minimal choices of N and M in Lemmas and 10. Definition 11. Given f K >> R continuous and > 0, let N(f,K,)) be the minimal choice of N in Lemma and let M(f,K, K, be the minimal choice of M in Lemma 10.
812
Observe that M(f, Z N(f,K,e) e). In many cases, it is possible to give explicit bounds for the constants ((f, and M(f, e) As an illustration, we give the argument in the case that K is the closed unit cube in R" and f: R" is Lipschtiz continuous.
613
614
Proposition Let K [0, C R" be the (closed) unit cube and let f K. - RM be Lipschitz continuous with Lipschitz constant R. For any >> 0, we have:
616
RRnn" 2E
T(n/2+1) (
2R
N(f,K,e) <
and M(f.K,e) <
2+
721/2
RR/n 2z
Proof. For the first inequality, observe that the unit cube can be covered with 2c cubes of side length Ryn Each cube is contained in a ball of radius & centered at the center of the cube. (In general, cube of side length in R" is contained in ball of
619
800
radius av/l 2 ) Lipschitz continuity implies that, for all x, x' E K, if |x x!| e/R then
fxx) f(x)] S R|x-XX x'I <.
622
For the second inequality, let r = e/R. Lipschitz continuity implies that, for all x, x' e K, - then ((x))f(xx) S R|x 1xx-xx x'1 e. The n-dimensional volume of the set of
623
624
points with distance at most r/2 to the unit cube is vol(B,/2(K)) S (1", The volume
16

---

608 of a ball with radius r/2 is vol (ol(B,/((0))
[(x/2+1] (r/2)". Hence, any packing of (2))
x2//2
with balls of radius r/2 consists of at most
vol vol(B,;2(K)) vol(B,/2(0)
- T(n/2+1)
(2 2 2R
7t//2
698
such balls. So there also exists a maximal packing with at most that many balls. This packing can be used in the proof of Lemma 10, which implies that it is bound on e,).
629
651
We note in passing that any differentiable function f K + RD on a compact subset of R" is Lipschitz continuous. Indeed, the compactness of K implies that there exists R such that ff (x) R for all E K. Then one can take R to be the Lipschitz constant of f-
652
688
3.3 Proof of Theorem 3: UA for asymptotically affine functions
635
In this section, we restate and prove Theorem which proves that radial neural networks are universal approximators of asymptotically affine functions. We recall the definition of such functions:
636
637
638
Definition 13. A function f R" - R" is asymptotically affine if there exists an affine function L R" + R such that, for all e > 0, there exists a compact set K C IR" such that for all x R" K. We say that L is the limit of f.
639
[xx - fx))
640
641
Remark 14. An asymptotically linear function is defined in the same way, except L is taken to be linear (i.e., given just by applying matrix multiplication without translation). Hence any asymptotically linear function is in particular an asymptotically affine function, and Theorem 3 applies to asymptotically linear functions as well.
642
643
644
645
Given asymptoticllly affine function f R" - R and O, let K be compact set as in Definition 13. We apply Lemma 9 to the restriction flx of f to K and produce a minimal constant N = Nf e) as in Definition 11. We write simply N(f, K, for this constant.
646
647
Theorem 3 (Universal approximation). Let f: R" R" be an asymptotically affine function.
648
649
For any E > O, there exists a compact set K C RR" and a function F: RR - IRT such that:
650
F is the feedforward function of radial neural network with N K, layers whose hidden widths are (n + 1, + .,..., + N).
2. For any x E R", we have |F(x) f(x) <.
652
653
Proof. By the hypothesis on f, there exists an affine function L R" + R and a compact
set C R" such that [x(x ((x))ffll fxx) < e for all x E R" \. Abbreviate N(f, e,) by N. As
654
in Lemma 9, fix CN....,CN E K and IN E (0,1) such that, first, the union of the balls
656
656 B (c:) covers K and, second, for all i, we have f (B,, (G)) S Be(f(c)). Let U = U Br, (c)),
so that K C u. Define F: R" + R" as:
[(x)
if & U
F(x)=
f(c))
where is the smallest index with x E Bc,
e5e If x & U, then [(x)) [L(x)= - f(x) < e. Hence suppose x U. Let j be the smallest index such that x E B,, (c,)) Then F(x) = f(c;), and, by the choice of rj, we have: [F(x) - f(x) = If(c,) - f(x) <e
ese
We proceed to show that F the feedforward function of radial neural network. Let 661 en.... ey be orthonormal basis vectors extending IR" to R*+N We regard each RR++i-1 as
860
862 a subspace of R i by embedding into the first n + i - 1 coordinates. For = 1,..., N, we
/1 ? and define the following affine transformations:
663 set h;
T; RR"
+ Ri
S R" n + R²"
I H - Ci + h;
h z (1 + h; (eiz)) +.::+:1+.. +G
17

---

where ()),2) is the coefficient of e; in z. Consider the radial neural network with widths (n,n n + 1,..., n + N, m), whose affine transformations and activations are given by:
065
For ,..., N the affine transformation from layer i i- to layer i is given by 2 + T, o (2)), where So = idR".
667
The activation function at the i-th hidden layer is Step-ReLU on R*+,, that is:
Pi: IR" T+
RR*,,
2
if | 2
0
otherwise
The affine transformation from layer i N to the output layer is
609
= L,fs S2(z)
where LL.f is the affine transformation given by:
670
@ Pife : R 1*+N + R" + Σ aie + L(x) + L - -((c))
which can be shown be affine when L is affine. Indeed, write L(x) Ax + where A is matrix in R=*x and b E R" is vector. Then LL is the composition of the linear map given by the matrix
671
672
[A f(g) - L(c)) f((2) - L(e2)
f(ex) L(cN)] E (==((+NN)
674
and translation by b R". Note that we regard each f(c) L(c)) E R" as column vector in the matrix above.
675
678
We claim that the feedforward function of the above radial neural network is exactly F. To show this, we first state a lemma, whose (omitted) proof is an elementary computation.
677
Lemma 3.1. For i = 1,..,,,, N, the composition S o T; is the embedding R < R"ti
678
Next, recursively define G : R - R"+ via
679
G; S p; o Ti Gi-1,
680 where Go = idge. The function G; admits an direct formulation:
661
Proposition 3.2. For = 0,1,..., N, we have:
x E U/-1 (c)
x
(x)
Cj tej
where S i is the smallest index with x E Br,(c))
682
Proof. We proceed by induction. The base step i = 0 is immediate. For the induction step, 683 assume the claim is true for i 11,, where 0 < i < N. There are three cases to consider.
Case 1. Suppose U/ Br, (c;). Then in particular & U (c)(),,, so the induction hypothesis implies that G (x) = x. Additionally, x & B,, (c;), so:
665
[T(x)] |x c + hee v cal + + 17 +1 17 =.
686
Using the definition of p; and Lemma 3.1, we compute:
G((x) S₁ o p: Ti (x) S o:op pj o (x(x) S Ti(x) x.
687 Case 2. Suppose x E Bj U Br (cx) for some j i i - 1. Then the induction hypothesis
implies that Gx)((x) = Gj tej We compute:
[T + ej)l = lcj ej Ci + heei > leyl 1.
18

---

Therefore,
Gx(x) S o P: Ti(c tej) S Ti(c; tej) cj + +
Case 3. Finally, suppose x E B; U (G). The induction hypothesis implies that
690
Gz- = x. Since x E Br, we have:
[T(x)) = = -:- + heil = V1 + h2 V ? +1 =1
692
Therefore:
(x)x = S o 1i(x) S,(0) = = +e.
This completes the proof of the proposition.
886
604
Finally, we show that the function F defined at the beginning of the proof is the feedforward function of the above radial neural network. The computation elementary:
606
Feedfforward = LL o SN o pN o In o SN-1 PN-1 o TN-1 0 S o TT
L.f. GN
=F
where the first equality follows from the definition of the feedforward function, the second 696
from the definition of GN, and the last from the case = N of Proposition 3.2 together with the definition of Li This completes the proof of the theorem.
eee
899
B.
Proof of Theorem 5: bounded width UA for asymptotically affine functions
We restate and prove Theorem 5, which strengthens Theorem 3 by providing bounded width radial neural network approximation of any asymptotically affine function.
700
Theorem 5. Let f: R² >> R be an asymptotically affine function. For any e >> 0, there exists compact set K C IR" and function F: R² + Rd such that:
1. F is the feedforward function of a radial neural network with N = N(f, K,e) hidden layers whose widths are all n + m +1. 1. 2. For any x E R", we have [F(x) f(x) <e
705
706
Proof. By the hypothesis on there exists an affine function L: R" R and compact set
K C IR" such that L f(x) < e for all x E IR" \ K. Given e > 0, let N = N K, e) and (0,1) such that the union of the balls B, (c;) covers K, and, for all i, we have (B, (c)) C Be(f(c)). Lets be the minimal non-zero
708
709
use Lemma 9 to choose Ci,..., E K and F1, .....,, IN
value of [f(c:) f(c;) for i,j E N),, that is, s = ((n) [f(c)) - f(ci)..
711
Using the decomposition R" mll RD x RD x R, we write elements of IR" mll as (x, 0), where E RR", E R" and I R. For 1,..., N, set:
712
713
(x + x (1 (1-0)c; y o f(e)) ( L()) (1 -)-
T; +++++1 R++++
where h; V p7. Note that Ti is an invertible affine transformation, whose inverse is given by:
715
T (x,,,0) x G ci, y (1- 0 f(a) - L(0) L(O)
f(a)-
1 - 0
h;
For i 1,.., N, define G R" Ru+ via the following recursive definition:
= opoo: G--

---

n7 where (x) = (x,0,0)
R + R"* is the inclusion, and p Ru+=+1 - Ru*m=1 is Step- on R quimil We claim that, for x E RR", we have:
[x, 0,0)
if x E U/-1 (r)(c)
Gx(x) (0. /
where S i the smallest index with x E Br,
719
This claim can be verified by straightforward induction argument, similar to the one given the proof of Proposition 3.2, and using the following key facts:
720
For R", (T,((xx0,0)) (x -, 0,0,,,) < 1 if and only if xx - cil < T..
721
(0) (o, fla)= ,
T, ((( flo) (o. 110) 11) (o, f(() f(s), .). s(e)),)) which, by the choice of has norm at least 1 if f(()) + f()), and is 0 if f(c) f(c)).
728
Let ④ RR mi
R" denote the affine map sending (z, to L(x) sy. It follows that F = oo GN satisfies
[x(x) f(c))
if x & U (c)
F(x)=
where is the smallest index with E Br)
727 By construction, F is the feedforward function of a radial neural network with N hidden
728 layers whose widths are all + m Let x E RR. If x E K, let j be the smallest index such that x E By (qj). Then F(x) = f(c),, and, by the choice of rj. we have |F(x) f(x) =
If(e)) - fxx] < e. Otherwise, x E IR" K, and [F(x) fx)) = [x(x - f(x) e.
730
B.5
Additional result: bound of max(n, m) +1
752
We state and prove an additional bounded width result. In contrast to the results above, the theorem below only holds for functions defined on compact domain, without assumptions about the asymptotic behavior. The proof is an adaptation of the proof of Theorem 5, so we give only a sketch.
733
735
738
Theorem 15. Let f - IR be a contimuous function, where K is a compact subset of R". For any e > O, there exisls F: IR" Rm such that:
757
F is the feedforward function of radial neural network with N(f,K,e) hidden layers whose widths are all mnx(n,m m) 1.
2. For any x E K, we have Fx() f(x) <.
Sketch of proof. The construction appearing in the proof of Theorem with L = 0 can 742 be used produce a radial neural network with N(f, hidden layers with widths
741
+ that approximates f on K. (Note that the approximation works only on K, as f is not defined outside of K.) All values in the hidden layers are of the form (x, 0,0) or (0, y, 1). We therefore replace yx, E R#+=++ by (x (x++g 0) ma,((n.,) R ~ Rmax(in,)+11 everywhere, without affecting any statements about the hidden layers. In particular, the transformation T; becomes
744
746
747
T; max - mmax(nm)=11 (x + x (1-(1- 11-00G, f)c; f(c))
s (1 (1-
748
With this change the final affine map sends (x,0) to sx. From the rest of the proof of Theorem follows that the feedforward function of the radial network satisfies [(x) - f(x) for all K.
749
20

---

B.6 Additional result: bound of max(n, m)
752
In this section, we prove a different version of the result of the previous section. Specifically, reduce the bound on the widths max(n, m) the cost of using more layers. Again, focus functions defined on a compact domain without assumptions about their asymptotic behavior. Recall the notation M(f,K, c) from Lemma 10 and Definition 11.
753
755
756
Theorem 16. Let f K ++ R be continuous function, where K is a compact subset of IR" for n
757
2. For any E > O, there exists F: RR - R such that:
F is the feedforward function of a radial neural network with 2M e,e/) hidden layers whose widths are all max(n, m). For any x E K, we have [Fxx) f(x)] <c.
Proof. We first consider the proof in the case m. Set M K,, As in Lemma 10, fix C E K and FI...., E (0, 1) such that, first, the union of the balls B,, (G) covers K; second, for all i, we have fB, (c)) S (f)2(((;)); and third, |c; - cjl 2 ri for # For
763
764 j I,... M,
x x- Ci
T: R R""
Ti
and recursively define G : RR" + R" G = o opo T, G where Go idg" is the identity on R" and p: R² + R" is Step-ReLU.
T
766
787
Lemma 16.1. For =
N, we have:
x
if x & U/_11 B,,(c)
Gx(x) =
Cj
where S is the smallest index with x E Br,
We omit the full proof of Lemma 16.1, as it is standard induction argument similar to Proposition 3-2, relying the following two facts. First, [T](x] < if and only if
768
x E Br (c).. Second, by the choice of e; we have - cjl 2 for all + This implies that
770
[i(c)) for i +
772
Next, perform the following loop over I,...,
Set P; 1 = (CI (cI.-.., U (d (dl....,d; 1}
d;
Choose d, in (c)) that is not colinear with any pair of points in PA- This is where use the hypothesis that > 2.
774
778
Let s; be the minimum distance between any point on the line through c; and d; and any point in P: {G}.
Let U, R" + R" be the following affine transformation:
U: : IR" R" x dj + (1e- (1;--a 10-d) d;l 3s 1c; - d,2
1 (x d. di) (c; d:)
Define H, R + RR recursively as H; u 1 opo U, H;- 1, where Ho idga.
780 We note that the transformation U; can also be written as A,(x - d;) where A is the linear map given by A; 1 (ro + Te- proj(c which involves the projections onto the line spanned by c; d, and onto the orthogonal complement of this line.
782
783
Lemma 16.2. For i,j 1, 1,...,M, M, we have:
d
if
Hi(c;) =
Cj
ifj>i
21

---

Proof. It is immediate that (d,)) = 0 and [4(c]] 1/2. It is also straightforward to show,
784
using the choice of s; that [p)p 2 1 for all p E P:-1 \ (G). It follows that U, p U sends to d; and fixes all other points in P:-1-
786
787
Lemma 16.3. For X E K, we have HM o GM = d; where i is the smallest index with
788 X E (c)
709
Proof. Let E K. By Lemma 16.1, we have that (m) cj where is the smallest index with E Br, (ci). (We use the fact that the balls (c)) cover K.) By Lemma 16.2, we have that (G) dj for all i. The result follows.
791
Set F HM o GM- We see that, for K: 792
[F(x) - fxx) d; f()) s [d; - --c((())++(())
< < + e/2
where is the smallest index with E (G). We show that the foedforward function of a radial neural network with 2M hidden layers, all of width equal to n. Indeed, take the affine transformations and activations as follows:
793
795
796
For 1,..., M the affine transformation from layer i- to layer i is given by x T T (x), where To
797
idg".
788
For = -1,------ -,,..,,M MM the affine transformation from layer M+i ++i- 1 to layer M + is given by x + U; o u, (x), where Up -N=
799
800
The activation at each hidden layer Step-ReLU on IR" that is p(x) = x if |x| 2 and 0 otherwise.
Layer 2M + has the affine transformation UM-
802
800
It is immediate from definitions that the feedforward function of this network is F.
To conclude the proof, we discuss the cases where # m. Suppose max(n, m) Then we can regard K as compact subset of R" and apply the above 808
804
so that
constructions. Suppose иt >> n so that max m) = n. Let inc : R" + R". Apply the above constructions to the function = inc f K + R"
807
C
Model compression proofs
808
The aim of this appendix is to give proof of Theorem 6. In order to do so, we first (1) provide background on a relevant version of the QR decomposition, and (2) establish basic properties of radial rescaling activations.
810
811
812
C.1 The QR decomposition
813 In this section, we recall the QR decomposition and note several relevant facts. For integers
614 n and m, let (R"*m upper denote the vector space of upper triangular n by m matrices.
815 Theorem 17 (QR Decomposition). The following map is surjective:
O(n) (Ruxm) upper
Rum
Q, R
QoR
816 In other words, any matrix can be written as the product of an orthogonal matrix and an urper-triangular matrix. When m < n, the last и mr rows of any matrix in (R"xm) upper 818 are zero, and the top m rows form an upper-triangular m by m matrix. These observations
819 lead to the following "complete" version of the QR decomposition, which coincides with
the above result when m 2
22

---

Corollary 18 (Complete QR Decomposition). The following map is surjective:
: (()) x Rkxm
upper
R nx
Q
R
Q inc
R
where k = min(n, m) and inc IR* + IR" is the standard inclusion into the first k coordinates.
823
We make some remarks:
There are several algorithms for computing the QR decomposition of given matrix. One is Gram-Schmidt orthogonalization, and another is the method of Householder reflections. The latter has computational complexity O(f'm) in the case of tion numpy. winall..geee qr that computes the QR decomposition of matrix using Householder reflections.
825
827
m matrix with 2 m. The package numpy includes a func-
898
830
2. In each iteration of the loop in Algorithm 1, the method QR-decomp with mode = 'complete' takes as input a matrix A₁ of size n; (m + 1), and pro- duces an orthogonal matrix Q; E O(n,) and upper-triangular matrix R, of size min(n; n/dd + 1) (nted + 1) such that A Qi inc; o R, Note that nnted min (i, piddd + 1).
893
3. The QR decomposition is not unique in general, or, in other words, the map y is not injective in general. For example, if n > m, each fiber of ş contains a copy of the orthogonal group O(1 mt).
896
857
4. The QR decomposition is unique (in a certain sense) for invertible square matrices. To be precise, let B be the subset of of (R"xn)upper consisting of upper triangular n by n matrices with positive entries along the diagonal. Both B and O(n) are subgroups of the general linear group GL, (R), and the multiplication map (n) x B - GL (R) is bijective. However, the QR decomposition not unique for non-invertible square matrices.
809
841
842
C.2 Radial rescaling functions
845
We now prove the following basic facts about radial rescaling functions:
Lemma 19. Let p h(m) R" - R" be radial rescaling function on IR"
847
1. The function commutes with any orthogonal transformation R". That is, po Q = Qop for any Q E O()).
848
849
2. If m I n and inc : R" + R" is the standard inclusion into the first m coordinates, then; h(m) o inc = inc I!)
Proof. Suppose Q E O(n) is an orthogonal transformation of R". Since Q is norm- 852 preserving, we have [Qul |o| for any RR. Since Q is linear, we have Q(Av) AQ for any A E R and u R". Using the definition of a = (n))) we compute:
851
853
p(Qv) h([Qol) Q h(lv)) |o]
Qv Q [((01) |
v) (
Q(p(v)).
[Qal
854
The first claim follows. The second claim is an elementary verification.
856 More generally, the restriction of the radial rescaling function p to a linear subspace of RR" is radial rescaling function on that subspace. Given tuple radial rescaling functions p
(p: R -> R suited to widths = (mi) we write red (pped RRnded R
858 for the tuple of restrictions suited to the reduced widths red, so that pped = pi Rod
23

---

C33 Proof of Theorem 6
Adopting notation from above and Section 5, we now restate and prove Theorem 6.
860
Theorem 6. Let (W,b,p) be a radial neural network with widths n. Let Wred and bred be the weights and biases of the compressed network produced by Algorithm 1. The feedforward function of the original network (W, (,,,pp) b, coincides with that of the compressed network (Wred b ,peed)
862
Proof. Let (Wred, bred, Q) QR- Compressee.bbb be the output of Algorithm 1, so that 664
865 Q E O(nhid) and (Wreed red red is a neural network with widths nrdd and radial
866
rescaling activations pf Pi
Let F F(W.b.p) denote the feedforward function
867
of the radial neural network with parameters (W,b) and activations p. Similarly, let Fedd F(wred bred pred denote the feedforward function of the radial neural network with 868
paraeeters (Wred, bred) and activations pred. Additionally, we have the partial feedforward functions Fi and Fred. We show by induction that
869
F = Qi o inc; o Fred
871 for any i = 0, N. (Continuing conventions from Sections 5.1 and 5.2, we set Qo
872 id Qr. id and inc; R + R" to the inclusion map.) The base step
873 immediate, For the induction step, let R Then:
Fi(x) P: (W₂ Fx-1(x) + bi)
pi (W: o Qi-1 o inc; ( (x) + +b;b)
Pi (br W Qi- inc,
Frod (x))
Pi Q o inc;
[bred
Wped]]
( (x) (x))
Frod
(wree
(x) + bped)
Qi inc; pi
RePed
Q inc
pred
874
The first equality relies on the definition of the partial feedforward function Fú the second 878 on the induction hypothesis; the fourth on an inspection of Algorithm I, noting that
878 R = bard Wred]; the fifth on the results of Lemma 19, observing that Pj inc; Pil Re =
877 inc; gredd and the sixth on the definition of Fred, In the case L, we have:
== FL QL inc, Fred pred
since Q1 = incy = idR", and Fred = pred The theorem now follows.
878
The techniques of the above proof can be used to show that the action of the group Onnhid)
879
of orthogonal change-of-basis symmetries on the parameter space Param (n) leaves the feedforward function unchanged. We do not use this result directly, but state is precisely nonetheless:
881
862
Proposition 20. Let (W, be radial neural network with widths vector n. Suppose g E
863
hnhid). Then the original and transformed networks have the same feedforward function:
864
F g W. gb, p) F(w. b, p)
In other words, fix parameters (W,b) E Param(n), radial rescaling activations p, and E hid) Then the radial neural network with parameters (W,b) has the same feedforward
885
24

---

function as the radial neural network with transformed parameters (g W. b), where we take radial rescaling activations in both cases.
008
We remark that Proposition 20 is analogous to the "non-iegative homogeneity" (or "positive scaling invariance") of the pointwise ReLU activation function3, In that setting, instead of considering the product of orthogonal groups O O(nhid), one considers the rescaling action of the following subgroup of I GLa,
L-1 GL
G= g
(g)) E I
each gi is diagonal with positive diagonal entries
Note that G is isomorphic to the product I 1 Rn and the action on (aram(n) is given by the same formulas as those appearing near the end of Section 5.1. The feedforward
669
860
function of MLP with pointwise ReLU activations invariant for the action of G (n).
D Projected gradient descent proofs
894
In this section, we give a proof of Theorem 8, which relates projected gradient descent for a representation with dimension n to (usual) gradient descent for the corresponding reduced representation with dimension vector red This proof requires some set up and background resutls.
896
896
897
D.1 Gradient descent and orthogonal symmetries
899
We first prove a result that gradient descent commutes with invariant orthogonal trans- formations. This section is general and departs from the specific case of radial neural
900
networks.
D.I.1 Setting
Let L: V V= R' + R be a smooth function. Semantically, is a the parameter space of
908
904
neural network and L the loss function with respect to batch of training data. The
differential dL,, of L at v E V is row vector, while the gradient V. o of L at v is column vectort:
906
ac
dL,
2c dx
ac dxp
VoL =
ac dxp
907 Hence VoL is the transpose of d',,, that is: V,C = (d.x)T. A step of gradient descent 908 with respect to L at learning rate > 0 is defined as:
7 7p -
0 PIULル
3Se Armenta and Jodoin, The Representation Theory of Neural Networks, arXiv:2007.12211 Dinh, Pascanu, Bengio, and Bengio, Sharp Minima Can Generalize For Deep Nets, ICML 2017; Meng, Zheng, Zhang, Chen, Ye, Ma, Yu, and Liu, G-SGD: Optimizing ReLli Neural Networks in its Positively Scale- Invariant Space, 2019; and Neyshabur, Salakhutdinov, and Srebro. Path-SGD: path-normalized optimiza- tion in deep neural nefworks, NIPS'15..
FFllowing usual conventions, we regard column vectors as elements of and vectors as elements of the dual vector space V*, The differential dC, at v E V is also known as the Jacobian of L at V.
25

---

Figure 5: Illustration of Lemma 22. If the loss is invariant with respect an orthogonal transformation Q of the parameter space, then optimization of the network by gradient descent is also invariant with respect to Q. (Note: in this example, projected and usual gradient descent match; this is not the case in higher dimensions, as explained in D.6.)
We drop 1j from the notation when it is clear from context. For any k 2 0, we denote by 7 910 the k-fold composition of the gradient descent map y
7070...07
D.I.2
Invariant group action
Now suppose p: - GL(V) is an action of Lie group G on V such that L is G-invariant, 918 Lee:
912
((p(g)())) = L(v)
914 for all g E G and E V. We write simply 8.v a for (v)) and g for p(g).
Lemma 21. For any a E V and g E G, we have:
VoL === (V vl)
916 Proof.
The proof is a computation:
VoL = (dxf)² = (d(()))))' = (df. o dgo)² = o og) =g- (dfgg))
(VL..)
916
The second equality relies on the hypothesis that Log L, the third on the chain rule, 917 and the fourth on the fact that dgæ = g since is linear map.
918 One can perform the computation of the proof in coordinates, for i p:
dc
a(Log)
dC 3g
(V = (dL,)²
dxi
dxi
dxi ge
πxi
1
= (V 8) = (8 ( (V gg = (g V Vg C
D.1.3 Orthogonal case
Furthermore, suppose the action of G is by orthogonal transformations, so that p(g)T= = p(g)-1 for all E G. Then Lemma 21 implies that
800
VgoL=g V.L
(D.1)
for any D E V and g E G. The proof of the following lemma is immediate from Equation D.1, together with the definition of y. See Figure 5 for an illustration.
922
Lemma 22. Suppose the action of G on V is by orthogonal transformations, and that L is G- invariant. Then the action of G commutes with gradient descent (for any learning rate). That is,
824
8es
* . v) = 7
for any E V.g E G, and k 200 0.
26

---

D.2
Gradient descent notation and set-up
629
We now turn our attention back to radial neural networks. In this section, we recall notation from above, and introduce new notation that will be relevant for the formulation and proof of Theorem 8.
950
952
D22.1
Merging widths and biases
Let n (No 11, R2, ......... RL 1, (1) be the widths vector of an MLP. Recall the definition of (n) as the parameter space of all possible choices of trainable parameters:
894
(nram(n)
[R X "
( (R" R X R"
We have been denoting an element therein as a pair of tuples (W,b) where W = (W; E 805
R R" ) are the weights and b (b; E R are the biases. However, in this appendix we adopt different notation. Observe that, placing each bias vector as a extra
807
column on the left of the weight matrix, we obtain matrices:
908
A = [b, Wi] E ((*((1+))1)
Thus, there is an isomorphism:
L Param(n) = + R" (1 1 (1)x(= x (2 11) x... ( 11)
In this appendix, we regard an element of Param(n) tuple of 'merged' matrices (A, E R= 1=1- We now define convenient maps to translate between the merged notation and the split notation. For each i, define the extension-by-one map from
940
941
942
R to R R R 11 as follows:
943
ext; R + Ru
v= = U2,,, Unl) + (1, 01,0,.. U2, Un,
(D.2)
Observe that, for any and x e R" 1, we have
A; ext; = Wix bj.
Consequently, the i-th partial feedforward function can be defined recursively as:
F = Pi A; o ext; 1
(D.3)
where p:: I > R is the activation at the i-th layer, and Fo is the identity R
946 D.2.2 Orthogonal change-of-base action
847 To describe the orthogonal change-of-basis symmetries of the parameter space in the
948 merged notation, recall the following product of orthogonal groups, with sizes correspond- ing to the widths of the hidden layers:
O(nhid) (m;) 0(2)) ... O(FL-1)
In the merged notation, the element Q (Q:) O(nhid) transforms A E Param (n) as:
:
001)
0
A + A- ==
Q; o A
(D.4)
0
851 where Qo = idin and Qt = idny-
Sn this general formulation, Pj can be any piece-wise differentiable function; for most of the rest of the paper will be interested in the where p; is radial rescaling function.
27

---

962
D.2.3 Model compression algorithm
We restate Algorithm in the merged notation. We emphasize that Algorithms and are mathematically equivalent; the later simply uses more compact notation.
964
Algorithm 2: QR Model Compression (QR-compress)
input
:A E (n)am(n)
output
:Q E On and V E Param(n
Q V [].[]
initialize output matrix lists
M1 + A1
for + to -- do
// iterate through layers
Q R; t QR-decomp (M; mode (
// Mi Qi inc R
966
Append Q: to Q
Append R; to V
// reduced merged weights for layer i
Set M,11 + A41
// transform next layer
end
Append M, to V
return Q V
We explain the notation. As noted in Appendix B.I, the symbol denotes composition of maps, or matrix multiplication in the case of linear maps. The standard inclusion inc; R + R" maps into the first need coordinates. As matrix, Inc; R= has ones along the main diagonal and zeros elsewhere. The method QR-decome with mode complete' computes the complete QR decomposition of the X (1 + n; matrix Mi as Q inc, R where Q; E (n;; and R is upper-triange trianglaa of size ne.ed x (1 + m The 961
856
967
definition of nde implies that either nifed neld 1 need H, The matrix R; is of size
962
red x ned in the former case and of size n; x (1 + n;de in the latter case.
963
964
D.2.4 Gradient descent definitions
965
As in Section 6, we fix:
966
a widths vector n = (no, N1 (n)).
a tuple p= (PI...., of radial rescaling activations, where p: : R" - R", for I.
967
968
batch of training data Vx) S 1 x R = Rod R
cost function C: IR x !R - IR
As result, we have a loss function on Param(n):
L Param + R
L(A)= (x))
where F(A,p) is the feedforward of the radial neural network with (merged) parameters A and activations p. We emphasize that the loss function L depends on the batch of training data chosen above; however, for clarity, we omit extra notation indicating this dependency since the batch of training data is fixed throughout this discussion. Similarly, we have:
974
975
the reduced widths vector red = (nred, red ni
878
the restrictions p red (pted, ....,
,....,tttt p(ed) where pred Redded
R ned for i 1,..,,L L.
Using the fact that r nip and ntte n1, there is loss function on Perd 878
Lred Param(n + R
fred (B) = CF (x)).
28

---

block of where (( is the feedforward of the radial neural network with parameters B E (n and activations pTdd (Again, technically speaking, the loss function Fedd 961 depends on the batch of training data fixed above.) For any learning rate Yf > 0, we obtain gradient descent maps:
Param n)
+ Param(n)
Trd Param (ndd)
red
+ Param (n red)
A + A 17 V
- V BLedd
D.3 The interpolating space
In this section, we introduce subspace Param""((n) (n) of Param(n), that, as we will later see, interpolates between Param(n) and Param(n
Let Param int (n) denote the subspace of (nram(n) consisting of those T (T Tt) E 987 Param (n) for which the bottom left (ı;- nife (1+ = block of T; is zero for each i. Schematically:
T;
where the rows are divided as nited on top and n1; nided on the bottom, while the columns divided (1 mted) on the left and n; nee the right. Let
969
860
41 Param int (n) Param(n)
be the inclusion. The following proposition follows from an elementary analysis of the workings of Algorithm 2 (or, equivalently, Algorithm 1).
Proposition 23. Let E Param(n) and let Q E (nhidd be the tuple of orthogonal matrices produced by Algorithm 2. Then Q A belongs to Param""((n).
993
994
996
Define map
(n)am(n) + Param int (n)
by taking (nram(n) and zeroing out the bottom left (n; no,dd (1 + n/de)) A; for each i. Schematically:
996
1) i=1
(10 *
A
Aj
= *
(1(A) -
*
*
i=1
It is straightforward to check that 91 well-defined, surjective linear map. The transpose of 91 is the inclusion (1- We summarize the situation in the following diagram:
4
Param int n
Param(n)
(D.5)
41
We observe that the composition 91 o is the identity on Paraminl(n)
1000
1001
D.4
Projected gradient descent and model compression
Recall from Section 6 that the projected gradient descent map on Param(n) is given by:
1002
Yproj Param n)) - (n)am(n),
A + Proj (A vVVCC
1008
where A = b) are the merged parameters (Appendix D.2), and, in the notation of the previous section, the map Proj is 4 o 911 To reiterate, while all entries each weight matrix
1004
and each bias vector contribute to the computation of the gradient VAL V [Wwp) C, only those in the bottom left submatrix get updated under the projected gradient descent map /proj.
1006
29

---

Let Q = QR- (A-CooprressA) be the outputs of Algorithm 2 (which is equivalent to A Algorithm 1), that V (Wred, bled) E Param(n are the parameters of the com- pressed model corresponding to the full model with merged parameters A = (W,b), and 1010
1008
Q E ( is an orthogonal change-of-bastee symmetry of the parameter space. Moreover, the change-of-bast action from Appendix D.2 and Proposition 23. We have the following rephrasing of Theorem 8.
1011
set T Q A Param""(nn) where
1018
1014
Theorem 24 (Theorem 8). Let A E (nram(n), and V,Q,T be as above. For any k 2
1. (A)) Q (T) 1015
1016
2. Xroy (T)
(T) - yfad (V) +T-V.
More precisely, the second equality is ? (T) ( (V)) + T (V) where Para - (n)am(n) is the inclusion into the top left corner in each coordinate. Also, in the statement of Theorem 8, we have T
1017
1018
1019
1020
We summarize this result in the following diagram. The left horizontal maps indicate the addition of = T v, the right horizontal arrows indicate the action of Q, and the vertical maps are various versions of gradient descent. The shaded regions indicate the (smallest) vector space to which the various representations naturally belong.
1021
1022
Param n red
Param int
Param (n)
W
GD on Pel)
proj on Param(n)
CD on Param(n)
CD on Param(n)
Fed (V)
g Proj
(T)
(T) (W)
+T
1024
D.5 Proof of Theorem
1025 We begin by explaining the sense which Paramint(n) interpolates between Param and
Param (nram(n red). One extends Diagram D.5 as follows:
1026
Param(n"*)
Param int (n)
Param (n)
92
91
The map
1: Param(n red) < Paramint(n)
takes B (B,)= Param(n red and pad each matrix with n; nied rows zeros on the bottom and 11-1 red columns of zeros on the right:
1028
( B 9) :=1
(B.) + (B)
1080
It is straightforward to check that ia well-defined injective linear map.
1031
The map
92 Param
(n) (n)
Param(n
30

---

extracts from T the top left niee
x (1 + needd matrix:
T = T, = T 11)
(2)
- 92(T) -T,) (T
0
T
(4)
i=1
i=1
1083
It is straightforward to check that 92 is a surjective linear map. The transpose of 92 is the inclusion 12.
1034
1035
Lemma 25. We have the following:
1036
1. The inclusion : Param(n - Param(n) coincides with the composition (1 o 12, and commutes with the loss functions:
1057
Param (n red )
4=2==1
Param(n)
Lied
R
2. The following diagram commutes:
1039
Param int (n)
92
& Param n red
fed
(n
L
R
3. For any T E Param!nt(n)) (n), we have: 91 (V = (Vez (T) fred)
1040
1041
Proof. We have the following standard inclusions into the first coordinates and projections onto the first coordinates, for 0, 0,1, L:
inc;
inc nee
R
R inc; = inc nefed 1
R
Rl-
1043
M, : R" R" RT
m4 ++ - 1+
Observe that Param int (n) is the subspace of Param consisting of those T (Ti...,, Tτ) E Param(n) such that:
10M
1045
(idm inc, 77,)) Ti inc; 7;-1
1046
for i I,..,,
1047
By the definition of radial rescaling functions, for each 1,..., I,, there is a piece-wise
differentiable function h;; IR RR such that pi = h (n)).. Note that padd (ntedd
1048
h and
J (m) o inc; inc; (n
1049
1050
The identity (1 follows directly from definitions. To prove the commutativity of the first diagram, it is enough to show that, for any X in Param(n the feedforward functions of X and (() coincide. This follows easily from the fact that, for i = I,..., L, we
1052
1053
have:
π1 (mi) inc; n2, inc; ((( = (n.
For the second claim, let T E Pnr) It suffices show that ( (T) and (T) have the same feedforward function. Recall the ext, maps and the formulation of the feedforward function in the merged notation given in Equation D.3. Using this set-up, the
1064
1056
1057
key computation is:
inc, o (n!" o π, o T; extr oinc
(1i))) inc; n. T, inc o extr,
((ii)) T, inc; extn
(n:)) Ti extr inc;-1
31

---

1058
which uses the fact that (idæ, inc; πT) o T; o inc; 1 = 0, or, equivalently, inc; π T, inc = T, inc;-1, as well as the fact that ext, inc; inc; ext, Applying this relation successively starting with the second-to-last layer (i L- and ending in the first (i = 1),
1059
one obtains the result. For the last claim, one computes V 44) in two different ways.
1061
1062
The first way is:
Vi(Con) = (d(CTo = (dC,, odT!)* = (d))()))u)*
= (dc%((m) = (Vn(T)()
1063
where we use the fact that 4 is linear map whose transpose is 91- The second way uses the commutative diagram of the second part of the Lemma:
1064
Vi(Con) VT (eeed o 92) (d (Ered) 92) (a (fred) od (92)2)
= (Cred) o 992 93 (Cred) 12 (v (Tee
1065
We also use the fact that 92 is a linear map whose transpose is 12.
1068
Proof of Theorem 8. As above, let Q QR-cQ-o---o OR-compress(( (A) be the outputs of Algorithm 1, so that V (Wreed bred) Param(n red is the dimensional reduction of the merged
1067
parameters = bW, and Q E O( hid). Set T - Q -, A E Paramin((n)) (n).
1068
The action of Q E Oidhid on Param(n) is an orthogonal transformation, so the first claim follows from Lemma
hid)
1069
1070
1071
For the second claim, it suffices to consider the case 1] 1. The general case follows similarly. We proceed by induction. The base case k 0 amounts to Theorem For the
1072
1073
induction step, we set
z(k) 111 (V)) +T (V).
1074 Each (k) belongs to Param int (n), (n),, so i (z))) = (k), Moreover, q2 (z(k) = =7ted (V).. We
1075
compute:
Q (-1.A) AA) =
Yproj /proj (Q (Q1 (A))
proj
Prooj (1T (V)) +T -V-)
= 91 (1(Tred (V)) +T- - (Tata (V))+
=I(rieal (V)) 11 91 (Va(21082)
+T +T-I(V)
ulrreell ( (V)) - 11 12 ) +T ((V)
(rrea (V) V 1ind (V) (red
((V)
= 1 (7red (V) + T (V))
where the second equality uses the induction hypothesis; the third invokes the definition of Tproji the fourth uses the fact that ((k) ( (V)) +T ((V) belongs to Param (n);
1076
1077
the fifth and sixth use Lemma 25 above; and the last uses the definition of Tred-
1078
1079
D.6 Example
We now discuss an example where projected gradient descent does not match usual gradient descent.
1080
1061
32

---

1082 Let (1,3,1) be widths vector. The space of parameters with this widths vector is
1083
10-dimensional:
Param (n) = Hom(R², @ Homm IR = R
1084 We identify choice of parameters (in the merged notation)
A1
A2 18 i il E Param (1,33)
(D.6)
with the point p (a, b, d, f, & h,,, in R10 To be even more explicit, the weights for
1085
b
the bias in the first hidden hidden layer is by (a, (,,,,)), the the first layer are Wi d
weights for the second layer are W2 = [h f), and the bias for the output layer is b2 g- The action of the orthogonal group (nn) ((3 on Param (n) ~ R can be expressed as:
0
0
0 0 0
0
Q
1069
where the rows and columns are divided according to the partition 3 3+3+1+3 10. 1090 Consider the function
L: Param(n) + R
p = (a, b, c, d,,,c, i, + h(a +) + i(c + +f(e+
1091
By the product rule, we have:
vp (b,,,,
One easily checks that L(Q p) Cp) and that V Q Vp for any Q E O(3).
1092
The interpolating space is the eight-dimensional subspace of Param(n) R with (using the notation of Equation D.6), Suppose p' = b, 0, 0, g, i, belongs to the interpolating space. Then the gradient is
1094
1096
V P (h, i. + b,c d,0)
1096
which does not belong to the interpolating space. So one step of usual gradient descent, with learning rate 1j > yields:
1097
-p' = (a, c,
0, 0, i,) + (a - 17h s1h, c c- nji, 1p -nj -1 - 11
11 i ( +d), 1) On the other hand, one step of projected gradient descent yields:
Yproj : pp (a, b,,.b,,,,,00,g,i k, (a - 1/h 17h c qi. 0, - 17 b)) i- (
0,
Direct computation shows that the difference between the evaluation of L after step of gradient descent and the evaluation of L after one step of projected gradient descent is:
1100
C(r(p') - L (Tproj(''))
2nj²
6Fr A E (n)am(n), the neural function of the neural network with affine maps determined by and identity activation functions is R - R; x + L(W)x. The function L can appear as a loss function for certain batches of training data and cost function on IR
33

---

E
Experiments
1101
1102
As mentioned in Section 7, we provide an implementation of Algorithm 1 in order to (1) empirically validate that our implementation satisfies the claims of Theorems and Theo- rem 8 and (2) quantify real-world performance. Our implementation uses generalization of radial neural networks, which we explain presently.
1108
1106
1106
Ei
Radial neural networks with shifts
1107
In this section, we consider radial neural networks with an extra trainable parameter in each layer that shifts the radial rescaling activation. Adding such parameters allows for more flexibility in the model, and (as shown in Theorem 26) the model compression of Theorem holds for such networks. It is this generalization that we use in our experiments.
1108
110e
Let R- + R be a function. For any n 2 1 and any t E IR, the corresponding shifted radial rescaling function R" is given by:
1111
h(lvl t)
p = (m +
|
1113
if v # and p(0) = 0. A radial neural network with shifts consists of the following data:
1114
1. Hyperparameters: A positive integer and a widths vector n (no, f1, .2,...,, NL.)
1115
2. Trainable parameters:
1116
(a) A choice of weights and biases (W,b) E Param(n).
1117 (b) A vector of shifts = 12..,,...,, (L) E IR..
[L)
1118
3. Activations: A tuple h = (hi,...,hy h1,) of piecewise differentiable functions IR + R.
Together with the shifts, we have the shifted radial rescaling activation pi h (mit))
1119
1120
R + R each layer.
The feedforward function of radial neural network with shifts is defined in the usual recursive way, as in Section 3. The trainable parameters form the vector space Param(n) x RL, and the loss function of a batch of training data {(x;, C R x RRt is defined as
1121
L (n)amm()) x R² R;
(W, + [C(F(wibAn))>)) (x)).
1124 where F(W.bh)) is the feedforward function of radial neural network with weights W, 1125 biases b, shifts t, and radial rescaling activations produced from h. We have the gradient
descent map:
Y Param(n) x R" Param(n) x IR.
which updates the entries of W, b, and The group (nn O(n)) ... O(1L-1) acts on Pnram(n) as usual (see Section 5.1), and on RL trivially. The neural function is unchanged by this action. We conclude that the Oinhid) action on Param(n) x RL commutes with gradient descent 7. We now state generalization of Theorem 6 for the case of radial neural networks with shifts. We omit proof, as it uses the same techniques as the proof of Theorem 6.
1127
1128
1129
1130
1181
1152
1153
Theorem 26. Let b,,,,t,h) be a radial neural network with shifts and widths vector n. Let Wred and bed be the weights and biases of the compressed network produced by Algorithm The feedforward function of the original network (W, b, coincides with that of the compressed network (Wred bred ,t,
1135
1136
1137
Theorem also generalizes to the setting of radial neural networks with shifts, using projected gradient descent with respect to the subspace Param int (nt(n) IR_ of Param(n) x R
34

---

1189
E.2
Implementation details
Our implementation is written in Python and uses the QR decomposition routine in NumPy [2T]. We also implement a general class RadNet for radial neural networks using PyTorch [41]. For brevity, we write W for (W,b) and Wred for (Wred, (Wred,bred)
1141
1148
(1) Empirical verification of Theorem 6. We use synthetic data to learn the function f(x) e with N 121 samples xj -3 j/20 for S 121. We model fw as radial neural network with widths (1,6,7,1) and activation the radial shifted sigmoid h(x) 1/(1 x Applying QR-compress gives radial neural network Wied with widths ned (1,2,3,1). Theorem implies that the neural functions of
1144
1145
1147
1148 fw and fwied are equal. Over 10 random initializations of W, the mean absolute error
(1/N) Σ (xy) fared (xx) = 1.31 10 - 4.45- 10-9. Thus fw and fiee agree up to
1149
machine precision.
1151
(2) Empirical verification Theorem 8. Adopting the notation from above, the claim is that training fo w.w with objective L by projected gradient descent coincides with training fweed with objective Lred by usual gradient descent. We verified this on synthetic data using 3000 epochs at learning rate 0.01. Over 10 random initializations of W, the loss functions match up to machine precision with -- Lred 4.02 10-9 + 7.01 10-
1152
1153
1154
1166
1156
(3) Reduced model trains faster. Due to the relation between projected gradient descent of the full network W and gradient descent of the reduced network Wreed our method may be applied before training to produce a smaller model class which trains faster without sacrificing accuracy. We test this hypothesis in learning the function f: R² + IR² sending
1157
1158
1159
(41, 12) to (e H, using N 121 samples
1160
- k/20) for 0 < j,k < 121. We model fw as a radial neural network with layer widths = (2,16, 64, 128, 16,2) and activation the radial sigmoid hrr) (1(1++=)) Applying comcress gives radial neural network fined with widths ned = (2,3,4,5,6,2) We trained both models until the training loss was < 0.01. Running on system with an Intel 15-8257U@1.40CC
1161
1163
1164
and 8GB of RAM and averaged over 10 random initializations, the reduced network trained in 15.32 + 2.53 seconds and the original network trained in 31.24 + 4.55 seconds.
1165
1168
Relation to radial basis function networks
1167
1168
In this appendix, we show that radial neural networks are equivalent to a particular class of multilayer radial basis functions networks. This class is obtained by imposing the condition that the so-called 'hidden dimension' at each layer is equal to one; the total number of layers, however, is unconstrained. To our knowledge, the literature contains no universal approximation result for this class of radial basis functions networks.
1169
1170
1171
1173
F1 Single layer case
1174
We first recall the definition of radial basis function network. A local linear model extension of radial basis function network (henceforth abbreviated simply by RBFN) consists of:
An input dimension n1 an output dimension m, and a 'hidden' dimension N.
1178
For = 1, matrix W; E R"*", vector by R", and weight a; R
A nonlinear function? A R R.
7 A more general version allows for different nonlinear function for every N.
35

---

The feedforward function of RBFN is defined as:
F:R² R
Σ (a: (x A([x b;l).
1179
The integer N is commonly referred to as 'the hidden number of neurons'. This is a bit of misnomer. Really there is only one layer with input dimension and output dimension m; the integer N is part of the specification of the activation function.
1180
We observe that if = and 41 O, then the feedforward function given by:
F: IR" - T
x x Wp(x+
1182
where p is the radial rescaling function determined by A. In words, one adds
E IR" to the input vector x, applies the activation p to obtain new vector in R", and then applies the linear transformation determined by the matrix W1 = W to obtain the output vector in R Motivated by this observation, we say that RBFN is constrained if N and aj
1884
185
1186
E2
Constrained multilayer case
1187
Next, we consider the constrained multilayer case of a radial basis functions network. Specifically, constrained multilayer RBFN consists of:
1189
A widths vector (Ho....,, where L is the number of layers.
1190 A matrix W₂ E R for ( = I,...,L.
1191 A vector by & R for e= -,,,,,.,,L-1
1192
1-1. 1. (Equivalently, the corresponding radial rescaling function Pt R - R for 0,..., 1.)
A nonlinear function 2e : R + R for = 0,1,,..., L-1
188
The feedforward function is defined follows. For l = O,...,,, we recursively define Fy R + R" by setting Fo(x) and
Fx(x) = W (F + b
for f= 1,.., L. The feedforward function is Fz.
1104
1186
E3 Relation to radial neural networks
We now demonstrate that radial neural networks are equivalent to multilayer RBENs.
1196
1197
Proposition 27. For any radial neural network, there a constrained multilayer RBFN with the same feedforward function. Conversely, for any constrained multiplayer RBFN, there is radial neural network with the same feedforward function.
1198
Proof. For the first statement, let p) be a radial neural network with L layers and widths vector (Hg,..., nI.). Recall the partial feedforward functions Gj R - R defined recursively by setting Go(x) and
Gz(x) = Pe (W,G,-1xx) bb)
1200
The feedforward function is Gr. Consider the constrained multilayer RBFN with L+ layers and the following:
1201
Widths vector (No, Mi... "i...., UI. 1t) The last two layers have the same dimen- sion.
1208
1204
Weight matrices W2 E R for = 1,..,, L and Wr.+1 = idny E IR
1206
A vector by E Rh for L = 1,..., and by = 0 E Ro,
A radial rescaling activation Pt R - R for = 1,..., L, and po id
36

---

Let Fy be the partial feedforward functions for this RBFN, defined recursively as above. We
claim that
Fx(x) We G G
for any R and t= 1,.., L. We prove this by induction. The base case is l 1:
(x W1 o Po (Fo(x) + bo) Wix W1 Go(x)
For the induction step, take f and compute:
(x(x) Wo Pt 1 (Fe (xx + by 1) Wz pt 1 (We G 2(x) + by 1) We Ge 1xx) The first claim now follows from the case = L, using the fact that WL-1 is the identity,
1208
For the second statement, let (W,,,p) be a constrained multilayer RBFN with L layers and widths vector (No....., NL)) Consider the radial neural network with L L+ layers and the following:
1209
1211
Widths vector (No, foy ",,,..,,,,../// The first two layers have the same dimen- sion.
Weight matrices given by W1 idn, and WN We-1 for L 2,..., 1.
Bias vectors given by by = b for l =
and bL+1 = 0.
1214
1215
Radial rescaling activations given by #o Pt for l = 1,,..,,,,, and PL1 idmi-
One uses the recursive definition of the partial feedforward functions show that, for {= L, we have (x) Wy Gę(x), where F, and G, are the partial feedforward functions of the RBFN and radial neural network, respectively. Then:
(x) Pr.+ (Wi+1 (x. b1+ Wi. (x) Fr,xx),
so the two feedforward functions coincide.
F4
Conclusions
1218
While radial neural networks are equivalent to certain class of radial basis function network, we point out differences between our results and the standard theory of radial basis functions network. First, RBENs generally only have two layers; we consider ones with unbounded depth. Second, to our knowledge, ours is the first universal approximation result such that:
1219
1220
1221
1222
it uses networks in the subclass of multilayer RBFNs satisfying the constraint that all the number of 'hidden neurons' in each layer is equal to 1.
1225
it approximates functions with networks of bounded width.
it can be used to approximate asymptotically affine functions, rather than functions defined on a compact domain.
1228
Our compressibility result may apply to multilayer RBENs where the number of 'hidden neurons' N₂ at each layer is not equal 1, but expect the compression to be weaker, and that constrained mulitlayer RBFNs are in some sense the most com compressible type of RBFN.
1229
1251
37


Reviews:
[{"review_id": "ln3bStd9I2", "reviewer": "Reviewer_g15c", "summary": "Activation functions are a critical part of the design of neural networks. The choice of activation function in the hidden layers controls how well the network can model a training dataset. In this manuscript, the authors consider so-called radial neural networks, which have similarities to radial basis function neural networks. The authors show universal approximation in both infinite width and depth as well as present a projected gradient descent algorithm for training.  ", "questions": "- Can you better connect the NN architecture to the literature and suggest why the NN architecture is novel? \n\n- Do the authors observe similarly training difficulties with radial neural networks as radial basis function neural networks?\n\n- Can the authors add experiments to convince readers to use radial neural networks?", "limitations": "The work has little potential negative societal impact. ", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "SIMILARITY TO RBF NNs (WITH LOCAL LINEAR MODEL):\nA radial basis function NN is given by (in the 2-layer case):\n\nNN(x) = sum_{i=1}^N a_i*h(||x+b_i||)\n\nIt is sometimes convenient to extend a radial basis function NN with a local linear model to obtain (in the 2-layer case): \n\nNN(x) = sum_{i=1}^N (a_i + w_i^T(x+b_i))*h(||x+b_i||).\n\nThis model is more general than the radial neural networks introduced in this manuscript. In particular, set a_i = 0, w_i = c_i*e_i/||x+b_i||, where c_i is a scalar and e_i is the ith unit vector to obtain: \n\nNN(x) = sum_{i=1}^N c_i (x+b_i)/||x+b_i||*h(||x+b_i||) (= sum_{i=1}^N c_i rho(||x+b_i||)),\n\nwhich I believe is an expression for 2-layer radial NN (as defined by the manuscript). Therefore, I do not find the NN architecture to be sufficiently novel to support publication.  \n\nTRAINING: \nSensitivity analysis explains the difficulties of gradient descent for radial basis function neural networks with smooth radial basis activation functions (see Karayiannis, \"Gradient descent learning of radial basis neural networks.\" Proc. Inter. Conf. Neural Netw., IEEE, 1997). Do the authors observe similarly difficulties with radial neural networks? \n\nEXPERIMENTS: \nSeparate from the novelty of the architecture and training, one would hope for experimental results that demonstrate a significant benefit over existing NN architectures (at least with regard to one example or metric).  The three experiments currently do not demonstrate any benefits but verify the theoretical contributions. \n\nWRITING \nThe manuscript is well-written. Two typos are: \n\n- p1, in (1.1): I think you want rho : R^n->R^n at the start of (1.1). \n- p6, line 195: \"strengthend\" should read \"strengthened\"\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PSBYzm5siRB", "reviewer": "Reviewer_RXP7", "summary": "This study introduces a new deep radial neural network, shows its density in the space of `asymptotically affine' continuous functions w.r.t the sup norm, and presents a compression algorithm that has a similar taste with QR decomposition.", "questions": "Suggestion:\nThe authors introduced a new class of functions: Asymptotically affine. As pointed out in l.158, this class covers compactly supported functions. As a consequence, Theorem 3 further implies the so-called cc-universality (i.e., the density in the space of all continuous functions w.r.t. the topology of compact convergence.) I recommend the authors to clearly state that the cc-universality is a corollary of Theorem 3.\n", "limitations": "The authors have addressed the limitation in Section 8.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe proof of the main theorem is constructive as well as simple. In traditional approximation theory, the decomposition of a function into ‘width’ direction has often been investigated. That is, decomposition into coefficients and bases (or frames). On the other hand, deep learning decomposes a function into function composites, a method that has not been well investigated by traditional approximation theory. This study provides a concrete example of how to decompose a nonlinear function in the ‘depth’ direction, and I consider this is highly valuable as a result of pioneering research.\n\nWeakness:\nAs the authors also pointed out in the conclusion and discussion section, the main theorem (universality) holds only for step-relu function.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MzAKWu3sxH5", "reviewer": "Reviewer_nux4", "summary": "The paper provides an analysis of radial neural networks from the viewpoint of expressivity and model compressivity.", "questions": "Why should we use these networks instead of the radial basis function networks, for which the theory is more understood?\nIntuitively, one may expect that applying the nonlinearity on the activation norm instead of each component of the activation vector decreases expressivity. The result you provide is very generic; would it be possible to characterize the loss of expressivity for more complicated functions?\nHow does the number of layers/width needed to approximate a continuous function to accuracy epsilon compare to those needed for other types of architectures?\nCould you please give intuition of why radial neural nets are more compressible than other architectures?\nAm I right that Q in Thm 7 depends on k? This should be clarified.", "limitations": "I do not expect this work to have any societal impact.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I have appreciated the originality of the work, namely, exploring simultaneously in a same paper the pros and cons of these architectures. \nThe main weaknesses is to me a lack of motivations: why should we use these networks instead of the radial basis function networks, for which the theory is more understood? After having read the paper, I am still unsure whether I should use this architecture or another, mostly for the following reason: it is unclear to me what expressivity we loose using those architectures. The authors only prove the ability of the model to represent simple continuous functions, but do not say exactly how expressivity compares to common architectures regarding state-of-the-art expressivity guarantees. Regarding paper clarity, I found the section about network compression hard to understand. The rest of the paper was well-presented. The paper is overall well written, I found few typos.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject