Universal approximation and model compression for radial neural networks 
Authors: Iordan Ganev
Twan van Laarhoven
Robin Walters
Venue: Submitted to ICLR 2023 (2023)
Keywords: universal approximation
model compression
radial functions
parameter space symmetries
projected gradient descent

Abstract: We introduce a class of fully-connected neural networks whose activation functions, rather than being pointwise, rescale feature vectors by a function depending only on their norm. We call such networks radial neural networks, extending previous work on rotation equivariant networks that considers rescaling activations in less generality. We prove universal approximation theorems for radial neural networks, including in the more difficult cases of bounded widths and unbounded domains. Our proof techniques are novel, distinct from those in the pointwise case. Additionally, radial neural networks exhibit a rich group of orthogonal change-of-basis symmetries on the vector space of trainable parameters. Factoring out these symmetries leads to a practical lossless model compression algorithm. Optimization of the compressed model by gradient descent is equivalent to projected gradient descent for the full model.

Under review as a conference paper at ICLR 2023
UNIVERSAL APPROXIMATION AND MODEL COMPRES- SION FOR RADIAL NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
We introduce a class of fully-connected neural networks whose activation func- tions, rather than being pointwise, rescale feature vectors by function depending only on their norm. We call such networks rudial neural networks, extending pre- vious work on rotation equivariant networks that considers rescaling activations less generality. We prove universal approximation theorems for radial neural net- works, including in the more difficult cases of bounded widths and unbounded do- mains. Our proof techniques Additionally, radial neural networks exhibit a rich group of orthogonal change- of-basis symmetries on the vector space of trainable parameters. Factoring out these symmetries leads to a practical lossless model compression algorithm. Op- timization of the compressed model by gradient descent equivalent to projected gradient descent for the full model.
novel, distinct from those in the pointwise
INTRODUCTION
Inspired by biological neural networks, the theory of artificial neural networks has largely focused on pointwise (or "local") nonlinear layers (Rosenblatt 1958, Cybenko 1989), in which the same function R IR is applied to each coordinate independently:
R- R"
(VI ... v,) (o(v1) (
(v,)).
(1.1)
In networks with pointwise nonlinearities, the standard basis vectors in En can be interpreted as "neurons" and the nonlinearity as "neuron activation." Research has generally focused on finding functions which lead to more stable training, have less sensitivity to initialization, or are better adapted to certain applications (Ramachandran et al.) [017;; Misra 2019); Milletarí et al. [0188] Clevert et al.] 20155] Klambauer et 2017). Many have been considered, including sigmoid, ReLU, arctangent, ELU, Swish, and others.
However, by setting aside the biological metaphor, it is possible to consider a much broader class of nonlinearities, which are not necessarily pointwise, but instead depend simultaneously many coordinates. Freedom from the pointwise assumption allows one to design activations that yield expressive function classes with specific advantages. Additionally, certain choices of non-pointwise activations maximize symmetry in the parameter space of the network, leading to compressibility and other desirable properties.
In this paper, we introduce radial neural networks which employ non-pointwise nonlinearities called radial rescaling activations. Such networks enjoy several provable properties including high model compressibility, symmetry in optimization, and universal approximation. Radial rescaling activa- tions defined by rescaling each vector by scalar that depends only on the norm of the vector:
p: R R
+ A(ul)ee,
(1.2)
where is a scalar-valued function of the norm. Whereas the pointwise setting, only the linear layers mix information between different components of the latent features, for radial rescaling, all coordinates of the activation output vector are affected by all coordinates vector. The inherent geometric symmetry of radial rescalings makes them particularly useful for designing equivariant neural networks (Weiler & Cesal 2019 Sabour et al. 2017: Weiler et al, 2018a b).
the activation input

---

Under review as a conference paper at ICLR 2023
W 1-
W
W
W
0
0
0
0
Figure 1: (Left) Pointwise activations distinguish a specific basis of each hidden layer and treat each coordinate independently, see equation [.] (Right) Radial rescaling activations rescale each feature vector by a function of the norm, see equation 1.2
We note that radial neural networks constitute a simple and previously unconsidered type of multi- layer radial basis functions network (Broomhead & Lowe 1988), namely, one where the number of hidden activation neurons (often denoted N) in each layer is equal to one. Indeed, pre-composing equation 1.2 with translation and post-composing with linear map, one obtains a special case of the local linear model extension of radial basis functions network.
In our first tors. Specifically, we demonstrate that any asymptotically affine function can be approximated with a radial neural network, suggesting potentially good extrapolation behavior. Moreover, this approx- imation can be done with bounded width. Our approach to proving these results departs markedly from techniques used the pointwise case. Additionally, our result is not implied by the universality property of radial basis functions networks in general, and differs in significant ways, particularly in the bounded width property and the approximation of asymptotically affine functions.
main results, we prove that radial neural networks are in fact universal approxima-
In our second set of main results, exploit parameter space symmetries of radial neural networks to achieve model compression. Using the fact that radial rescaling activations commute with orthogonal transformations, we develop a practical algorithm systematically factor out orthogonal symmetries via iterated QR decompositions. This leads to another radial neural network with fewer neurons in each hidden layer. The resulting model compression algorithm is lossless: the compressed network and the original network both have the same value of the loss function on any batch of training data.
Furthermore, prove that the loss of the compressed model after one step of gradient descent is equal to the loss of the original model after one step of projected gradient descent. As explained below, projected gradient descent involves zeroing out certain parameter values after each step of gradient descent. Although training the original network may result in a lower loss function after fewer epochs, in many cases the compressed network takes less time per epoch to train and is faster in reaching local minimum.
To summarize, our main contributions and headline results are:
Radial rescaling activations are an alternative pointwise activations: We provide a for- malization of radial neural networks, a new class of neural networks;
Radial neural networks are universal approximators: Results include a) approximation of asymptotically affine functions, and b) bounded width approximation;
Radial neural networks are inherently compressible: We prove lossless compression al- gorithm for such networks and a theorem providing the relationship between optimization of the original and compressed networks.
Radial neural networks have practical advantages: We describe experiments verifying all theoretical results and showing that radial networks outperform pointwise networks on a noisy image recovery task.
2 RELATED WORK
Radial rescaling activations. As noted, radial rescaling activations special case of the tivations used in radial basis functions networks (Broomhead & Lowe 1988). Radial rescaling functions have the symmetry property of preserving vector directions, and hence exhibit rotation
2

---

Under review as a conference paper at ICLR 2023
equivariance. Consequently, examples of such functions, such the squashing nonlinearity and Morm-ReLLU feature in the study of rotationally equivariant neural networks (Weiler & Cesa 2019 Sabour et al., 2017; Weiler et al. [2118a b Deftreys & Lau 2021). However, previous works apply the activation only along the channel dimension, and consider the orthogonal group (n) only for
2, In contrast, we apply the activation across the entire hidden layer, and (()) -equivvarraa where nt is the hidden layer dimension. Our constructions echo the vector neurons formalism (Deng
al. 202I). in which the output of nonlinearity is a vector rather than scalar.
Universal approximation. Neural networks of arbitrary width and sigmoid activations have long been known to be universal approximators (Cybenko 1989). Universality can also be achieved by bounded width networks with arbitrary depth (Lu et al. [017b)) and generalizes to other activations and architectures [Hornik 1991, Yarotsky, 2022] Ravanbakhsh 2020 [oonoda Murata 2017). While most work has focused on compact domains, some recent work also considers non-compact domains (Kidger & Lyons] 2020 Wang & Qu 2022p, but only for L,p functions, which are less general than asymptotically affine functions. The techniques used for pointwise activations do not generalize to radial rescaling activations, where all activation output coordinates are affected by all input coordinates. Consequently, individual radial neural network approximators of two different functions cannot be easily combined to an approximator of the sum of the functions. The standard proof of universal approximation for radial basis functions networks requires unbounded increase the number of hidden activation neurons, and hence does not apply to the case of radial neural networks (Park& & Sandberg 1991).
Groups and symmetry. Appearances of symmetry in machine learning have generally focused on symmetric input and output spaces. Most prominently, equivariant neural networks incorporate symmetry as an inductive bias and feature weight-sharing constraints based on equivariance. Ex- amples include G-convolution, steerable CNN, and Clobsch-Gordon networks (Cohen et al., 2019 Weiler & Cesa 2019: Cohen & Welling, 2016 CChdesteer et al. 2018 Kondor & Trivedi [008]] Bao & | Worrall et al. al.] 2017 Cohen & Welling 2017; Weiler et al., 20T8b] Dieleman et al., 2016 Lang & Weiler 2021 Ravanbakhsh et 2017). By contrast, our approach does not depend on symmetries of the imput domain, output space, or feedforward mapping. Instead, we exploit parameter space symmetries and obtain results that apply to domains with no apparent symmetry.
Model compression. A major goal in machine learning is to find methods to reduce the number of trainable parameters. decrease memory usage, or accelerate inference and training (Cheng al., 2017 Zhang et al.] 2018). Our approach toward this goal differs significantly from most existing methods in that it is based the inherent symmetry of network parameter spaces. One prior method
is weight pruning. which removes redundant weights with little loss in accuracy (Han et al. 2015
Blalock al. 2020 Karnin|
1990). Pruning can be done during training (Frankle & Carbin 2018
or at initialization (Lee et al. [2019] Waag et al. 2020). Gradient-based pruming removes weights by estimating the increase in loss resulting from their removal (LeCun et al. 1990 Hassibi & Stork
1993 Dong et al., 2017 Molchanov et al. 2016). A complementary approach is quantization, which decreases the bit depth of weights (Wu al.) [2016] [Hwward et al.] [017]] Gong et al., 2014).
Knowledge distillation identifies a small model mimicking the performance of larger model (Bu- ciluă et al. |206)] Hinnon et al., 20155 Ba Caruana 2013). Matrix Factorization methods replace fully connected layers with lower rank or sparse factored tensors (Cheng et al., 2015a b Tai et al.) 2015 Lebedev et al. 2014 Rigamonti et al. 2013 Lu et al., 2017a) and can often be applied before training. Our method involves a type of matrix factorization based the QR decomposition; how- ever, rather than aim for rank reduction, we leverage this decomposition to reduce hidden widths via change-of-basis -asiis operations on the hidden representations. Close to our method are lossless compres- sion methods which remove stable neurons in ReLU networks (Serra et al. 2021 2020) or exploit permutation parameter space symmetry to remove neurons (Sourek et al.) 2020); our compression instead follows from the symmetries of the radial rescaling activation. Finally, the compression re- sults of Jeffreys & Lau (2021), while conceptually similar ours, are weaker, as the unitary group action is only on disjoit layers, and the results are only stated for the squashing nonlinearity.
3

---

Under review as a conference paper at ICLR 2023
(1) Step-ReLU(r)
2
Shifted ReLU
1.5
1
(2) Squashing function
0.5
-2
-1.5
0.5
0.5
1.5
0.5
-1
1.5
-2
Figure 2: Examples of different radial rescaling functions in IR,, see Example 1
RADIAL NEURAL NETWORKS
In this section, we define radial rescaling functions and radial neural networks. Let h. IR TR be function. For any 1, set:
h (m) RRM + R"
(0 (v) h(1=))
for * O, and (0) 0. A function p R R is called radial rescaling function if (!m)) for some piecewise differentiable h R + R. Hence, P sends each input vector scalar multiple itself, and that scalar depends only on the norm of the vectof
easy to show that
radial rescaling functions commute with orthogonal transformations.
Example 1. (1) Step-ReLU, where h(r)) = r 2 1 and 0 otherwise. In this case, the radial rescaling function given by
p: P: RR.
リ u 2
0ff <1
(3.1)
(2) The squashing function, where h(r) = r /r2 +1). (3) Shifted ReLU, where h(r) = max (0, r- b) for 0 and b is real number. See Figure 2 We refer to Weiler & Cesa (2019) and the references therein for more examples and discussion of radial functions.
A radial neural network with I layers consists of positive integers n; indicating the width of each layer i = 0,1, .......;; L; the trainable parameters, comprising of a matrix W; E x of weights and bias vector E m for each for each i = 1.... L. We refer to the tuple n = (No, fl, nL) as the widths vector of the neural network. The hidden widths vector F R + R2 of radial neural network is defined in the usual way an iterated composition of affine maps and activations. Explicitly, set Fg = idg.0 and the partial feedforward functions are:
I,..., , L; and radial rescaling function p; 12 R
hid (n1 n2.. n nz-1)). The feedforward function
F. Ro IR" x Ps (W; Fz-1(x) + b,)
for 1,..., L. Then the feedforward function is F = Fr. Radial neural networks are a special type of radial basis functions network; we explain the connection in Appendix
Remark 2. If b, = 0 for all i, then we have F(x) = W (µ(x)r) where μ R" + a scalar- valued function and W WLWL-1' W1 E Ro is the product of the weight matrices. If any of the biases are non- zero, then the feedforward function lacks such a simple form.
UNIVERSAL APPROXIMATION
We consider two universal approximation results. The first approximates asymptotically affine functions with a network of unbounded width. The second generalizes to bounded width. Proofs
A function I² - IR that depends only on the norm of vector is known as a radial function. Radial rescaling functions rescale each vector according to the radial function e + x((xl) = MBll), This explains the connection Equation 1.2
4

---

Under review as a conference paper at ICLR 2023
S2 ooopo 2
C7
62
C1
C8
f(02)
K
C6
Rm
R"
Two layers of the radial neural network used in the proof of Theorem 3/ (Left) The compact set K is covered with open balls. (Middle) Points close to C2 (green ball) are mapped to 2 + e2, all other points are kept the same. (Right) the final layer, Co + e2 mapped to f(c2).
Figure 3:
appear in Appendix Throughout, (c) = for E R" Ja --- < <r) is the r-ball around a point e, and affine map g R. is one of the from L(x) = Ar + for A Ruxn and b E R"
4.1 APPROXIMATION OF ASYMPTOTICALLY AFFINE FUNCTIONS
continuous function f RR -> IR is asymptotically affine if there exists an affine map L R Rm such that, for every 0, there is compact subset K of IK" such that [(x) f()) < for all x E IR K. In particular, continuous functions with compact support are asymptotically affine. The continuity of f and compactness of K imply that, for any e and Fi.....IN E (0,1) such that, first, the union of the balls Br, (c:) covers K and, second, for all i, we have f (B,
there exist CI,- CI,.... . CN E K
S (f((c))). Let (f, K, be the minima?? choice of N.
Theorem 3 (Universal approximation), Let f IR" th Rm be asymptotically affine function. For any > there exists a compuct set K Rn and a function F: R" - IRm such that:
1. F is the feedforward function of radial neural network with N K,) layers whose hidden widths are (n + 1, + 2...., N).
2. For any x: E IR", we have Fxx) f(x) <c
We note that the approximation in Theorem is valid all Rf" not only K. To give idea of the proof, first fix ci....,cy CN E K and Fi, , E (0, 1) above. Let e EN be orthonormal basis vectors extending 1. to gu+N For 1, ......,. define afline maps T3 R gnti-1 g=+i and S, gu+i + Ruti by
T2(2) = ei + hGe;
S;(=) 2 (1 h1+ (es, z)e; + +6
where h2 /1 11-rt .? and (e,, 2) is the coefficient of e; in z. Setting P; to be Step ReUp (Equation 3.1 on Ru+i, these maps chosen so that the composition S; p maps the points in B,, to t, while keeping points outside this ball the same. We now describe radial neural network with N, m) whose feedforward function approximates f. For = 1, N the affine map from layer - 1 to layer is given by + T; (2), with So = idg=. The activation each hidden layer is Step-ReL.U. Let L be the affine map such that L fl affine map from layer to the output layer is $ o Sy where tb 1 - IR. is the unique affine map determined by x + L(x) if R", and e; + f(c,) L(c). See Figure 3 for an illustration of this construction. Theorem 3 has the following straightforward corollary:
widths (n, n +1.,.... 1,...,
T8 K. The
Corollary 4. Radial neural networks are dense the space of all continuous functions with respect to the topology of compact convergence, and hence satisfy cc-universality.
4.2 BOUNDED WIDTH APPROXIMATION
We now turn our attention to a bounded width universal approximation result.
In many cases, the constant N NJ,, K, can be bounded explicitly. For example, if K is the unit cube in Ra and f is Lipschitz continuous with Lipschitz constant R, then N(f, K,e)
RvT

---

Under review as a conference paper at ICLR 2023
Theorem 5. Let f R Rm be asymptotically affine function. For any 0, there exists compact set K C R" and a function F: : IR" + R such that:
1. F is the feedforward function of a radial neural network with N N,,K,,, hidden layers whose widths are all n. + m +1
2. For any E IR.", we have F( [(x)]<<
The proof, which is more involved than that of Theorem 3 relies on using orthogonal dimensions to represent the domain and the range of f, together with an indicator dimension distinguish the two. We regard points in *++++ as triples (x. y, 8) where x E IR", y E R and E R. The proof of Theorem parallels that of Theorem 3 but instead of mapping points in Br, (c:) 6j + 6i
map the points in B,, ((c, O, 0)) to (0, [tc) ( 1), where is chosen such that different balls do not interfere. The final layer then uses affine map (I, y, L(x) + sy. which takes (a, 0, to L(x), and (0, [(e))-(()) 1) to (c:)
We remark on several additional results; see Appendix B for full statements and proofs. The bound of Theorem[ccan be strengthened to max(n, m) + 1 in the case functions f K - R defined on compact domain K C _ (i.e., ignoring asymptotic behavior). Furthermore, with more layers, it is possible to reduce that bound to max(n, m))
MODEL COMPRESSION
In this section, we prove a model compression result. Specifically, we provide an algorithm which, given any radial neural network, computes different radial neural network with smaller widths. The resulting compressed network has the same feedforward function as the original network, and hence the same value of the loss function any batch of training data. In other words, model compression procedure is lossless. Although our algorithm is practical and explicit, it reflects more conceptual phenomena, namely, change-of- ofabassss action network parameter spaces.
5.1 PARAMETER SPACE SYMMETRIES
Suppose fully connected network has L layers and widths given by the tuple (no, 71, 72, : NL). In other words, the i-th layer has input width nı-1 and output width The parameter space is defined as the vector space of all possible choices of parameter values. Hence, it is given by the following product of vector spaces:
RL-1,
Param(n)
[I Xno x R22xX1 X ... X R X (R X R x X RTL)
An element is a pair of tuples (W, b) where W = (W; E B are the weights and = (b; E ) are the biases. To describe certain symmetries of the parameter space, consider the following product of orthogonal groups, with sizes corresponding hidden layer widths:
O(n hid) O(n1) O(n2) x ... O(NL-1)
There is a change-of-basis action of this group on the parameter space. Explicitly, the tuple of orthogonal matrices Q (Q+) E (nd transforms the parameter values (W, b) to Q W = (Q.WQQZ) and Q b = (Q, where we set Qo and Qr. to be identity matrices.
5.2 MODEL COMPRESSION
In order to state the compression result, we first define the reduced widths. Namely, the reduction add (nged .nfe.....,, ...., red) of widths vector n is defined recursively by setting red nn. then nnded min(ng, need + 1) for 1,..., I 1, and finally nied ni.- For tuple p (p. R". + )= | of radial rescaling functions, we write rod (pred 12 *
for the corresponding tuple of restrictions, which are all radial rescaling functions. The following result relies on Algorithm [
Theorem 6. Let (W. b. p) be radial neural network with widths Let Wied and bed be the weights and biases of the compressed network produced by Algorithm [ The feedforward function of the original network (W, b, p) coincides with that of the compressed network (Wied baed pred)

---

Under review as a conference paper at ICLR 2023
Algorithm 1: QR Model Compression (QR-compress)
input : :W, w,bb (aram(n)
output : Q O(n"i)) and W b Param(n
Q, W b [1.11.11
// initialize output lists
A1
+ [6 Wi]
// matrix of size nn x (no + +1) // iterate through layers // A Q,lnc,R,
for i 4 1 to -
Q, R, dd-coomp (A, mode complete') Append Q to Q
Append first column of R2 to bred
// reduced bias for layer i // reduced weights for layer i
Append remainder of R2 to Wied
Set A,+1 + [b++ W [ncc;]
// matrix of size #411 x (n.m) +1)
end
Append the first column of Ag to bad
// reduced bias for last layer // reduced weights for last layer
Append the remainder of Ag to Weed return Q Wied b
We explain the notation of the algorithm. The inclusion matrix Inc; E Buxnn has ones along the main diagonal and zeros elsewhere. The method QR- de-decoop with mode 'complete' computes the complete QR decomposition of the nı X (1 na matrix A, as Q,Inc, where Q: E (m²) and R3 is upper-triangular of size nied (1 m/a) ). The definition of noed implies that either neded [erd + or n. n4. The matrix R² of size nefed x noo the former case and of size n, x (1 + n in the latter case.
Example 7. Suppose the widths of a radial neural network are 16,8,1,,, Then it has EL (n,1 + 1)m; = 305 trainable parameters. The reduced network has widths 3,4,1,
and Σ 1 (m!=) + ()(""dd 34 trainable parameters. Another example appears in Figure
We note that the tuple of matrices Q produced by Algorithm I does not feature the statement of Theorem 6 but is important in the proof (which appears in Appendix ment shows that the i-th partial feedforward function of the original and reduced models are related via the matrices Q; and Inc, A crucial ingredient in the proof is that radial rescaling activations commute with orthogonal transformations.
Namely. Namely, induction argu-
6 PROJECTED GRADIENT DESCENT
The typical use case for model compression algorithms is produce smaller version of the fully trained model which can be deployed to make inference more efficient. It is also worth considering whether compression can be used to accelerate training. For example, for some compression algo- rithms, the compressed and full models have the same feedforward function after step of gradient descent is applied to each, and so one can compress before training and still reach the same mini- mum. Unfortunately, in the context of radial neural networks, compression using Algorithm [ and then training does not necessarily give the same result as training and then compression (see Ap- pendix [D66 for counterexample). However, QR-compress does lead precise mathematical relationship between optimization of the two models: the loss of the compressed model after one
Figure 4: Model compression in 3 steps. Layer widths can be iteratively reduced to 1 greater than the previous. The number of trainable parameters reduces from 33 to 17.
7

---

Under review as a conference paper at ICLR 2023
step of gradient descent is equivalent the loss of (a transformed version of) the original model after one step of projected gradient descent. Proofs appear in Appendix D
To state results, fix widths n and radial rescaling functions p above. The loss function L Param(n) - R associated to a batch of training data is defined as taking parameter values (W,b) b) to the sum Σ, C(F Y5) where C is cost function the output space, F ((W)b.) is the feedforward of the radial neural network with the specified parameters. and (zzz My) the data points. Similarly, we have loss function faed on the parameter space Param (n*dd) with reduced widths vector. For any learning rate 77 > O, we obtain gradient descent maps:
) E 12 RR
y Param (n) - Param(n);
(W, b) (W, b) - v C
Tred Param(n "d) Param (
(V,c) (V, (v.)
Ppnoj Param(n) (n) Param(n);
(W, b) Proj (W),
where the last is the projected gradient descent map on Param(n). The map Proj zeroes out all entries in the bottom left (n; -n?"d) ) n submatrix of W; - Ww. and the bottom (n; m!
entries in b; Vb, L. for each Schematically:
W - Vw,C=
- 6.-V
To state the following theorem, recall that, applying Algorithm Dto parameters (W, b), we obtain the reduced model (Wred brd) and an orthogonal parameter symmetry Q. We consider, for k 2 0, the k-folld composition 7* y o 7 o and similarly for Trea and "ied-
Theorem 8. Ler Wred b Q OR-compress) (W, W. b) be the outputs of Algorithm 7 applied to
(W,b) Param(n). Set U = Q -1. (W. - (Wred bred). For any k 2 0, we have:
/ (W, b) Q 7 (Q (W,b)
& Pproj (Q -(,,)) Thoa Prrca(("W (Wred b + U.
We conclude that gradient descent with initial values (W, b) is equivalent to gradient descent with apply Q to move from one the other initial values Q- (W, b) since at any stage projected gradient descent with initial values (using the action from Section 5.I). Furthermore,
Q-1 (W, b) is equivalent to gradient descent on Param(n with initial values (Wred b since at any stage we can move from one to the other by +U. Neither Q nor U depends on k.
7 EXPERIMENTS
In addition to our theoretical results, we provide an implementation of Algorithm [ in order to validate the claims of Bem outperforms a MLP on a noisy image recovery task. Full experimental details are in Appendix
well as demonstration that radial network
(1) Empirical verification of Theorem We learn the function ((x) = e== from samples us- ing radial neural network with widths
(1,6, 1) and activation the radial shifted sigmoid f(x))= = 1/(1 + e -r+)). Applying OR-compress gives compressed radial neural network with widths red (1, 2, 3, 1). Theorem 6 implies that the respective neural functions F' and Fied are equal. Over 10 random initializations, the mean absolute error is negligible up to machine precision:
(1/N) E, FF(,)) Fred (2,,) 1.31 10-8 4.45 10-2
(2) Empirical verification of Theorem g See Appendix B for details.
We verified the claim on same synthetic data as above.
(3) The compressed model trains faster. Our compression method may be applied before training to produce a smaller model class which trains faster without sacrificing accuracy. We demonstrate this in learning the function 2 + R² sending (t1, 12) to (e- -t) using radial neural network with widths (2, 16, 64, 128, 16, 2) and activation the radial sigmoid (((r) = 1/(1+ + e e""). Applying QR-comp ress gives compressed network with widths n ndd (2, 3, 4, 5, 6, 2). We trained both models until the training loss was < 0.01. Over 10 random initializations on our system, the reduced network trained in 15.32 + 2.53 seconds and the original network trained in 31.24 + 4.55 seconds.
(4) Noisy image recovery. A Step-ReLU radial network performs better than an otherwise compa- rable network with pointwise ReLU on noisy image recovery task. Using samples of MNIST with
8

---

Under review as a conference paper at ICLR 2023
3
Campartuus convergence
14-
38-
Noiso
3
3
02
3.5
Neloo kovel =
Noie lovel 1
Nokaz lovel
Figure 5: (Left) Different levels of noise. (Right) Training five Step-ReLU radial networks and five ReLU MLPs data with n=3 original images, m=100 noisy copies of each.
significant added noise, the network must identify from which original sample the noisy sample derives (see Figure 5 We observe that the radial network 1) is able to obtain better fit, 2) has faster convergence, and 3) generalizes better than the pointwise ReLUU, We hypothesize the radial nature of the random noise makes radial networks well-adapted to the task. Our data takes n = original MNIST images with the same label, and produces m = 100 noisy images for each, with a 240 train / 60 test split. Over 10 trials, each training for 150 epochs, the radial network achieves training loss 0.00256 +3.074 10-1 with accuracy + O. while the ReLU MLP has training loss 0.295 - 259 10 with accuracy 0.768 2.1199. 10 1 On the test set, the radial network has loss 0.00266 749 10-4 with accuracy 1 + O, while the ReLU MLP has loss 0.305 222.888 10-1 with accuracy 0.757 +:2.644 10-1 The convergence rates are illustrated in Figure 5 with the radial network outperforming the ReLU MLP, and 150 epochs are sufficient for all methods to converge.
In another experiment, we sampled 1000 MNIST digits and added spherical noise to each. Using a 800/200 train/test split, we trained both a Step-ReLU and pointwise ReLU network for the task of recovering the original digit (so there are ten classes total). Both networks had dimension vector 1784, 785, 786, 10]. After 1000 epochs, the Step-ReLL ReLU achieved training loss of 0,008248 and test loss of 1.193811, while for pointwise ReLU these were 7.534195 and 1.374735, respectively.
CONCLUSIONS AND DISCUSSION
This paper demonstrates that radial neural networks are universal approximators and that their pa- rameter spaces exhibit a rich symmetry group, leading to a model compression algorithm. The results of this work combine build a theoretical foundation for the use of radial neural networks, and suggest that radial neural networks hold promise for wider practical applicability. Furthermore, this work makes an argument for considering non-pointwise nonlinearities in neural networks
There are two main limitations of our results, each providing opportunity for future work. First, our universal approximation constructions currently work only for Step-ReLU radial rescaling ra- dial activations; it would be desirable to generalize to other activations. Additionally, Theorem 6 achieves compression only for networks whose widths satisfy n, > f4-1 + 1 for some i. Networks which do not have increasing widths anywhere, such encoders, would not be compressible.
Further extensions of this work include: First, little is currently known about the stability properties of radial neural networks during training, as well their sensitivity initialization. Second, radial rescaling activations provide an extreme case of symmetry; there may be benefits to combining radial and pointwise activations within a single network, for example, through 'block' radial rescaling functions. Our techniques may yield weaker compression properties for more general radial basis functions networks; radial neural networks may be the most compressible such networks. Third, radial rescaling activations can be used within convolutional or group-equivariant NNs. Finally, based on the theoretical advantages and experiments laid out in this paper, future empirical work will further explore applications in which we expect radial networks to outperform alternate methods. Such potential applications include data spaces with circular or distance-based class boundaries.

---

Under review as a conference paper at ICLR 2023
ETHICS STATEMENT
Our work primarily focused on theoretical foundations of machine learning, however, it does have a direct application in the form a model compression. Model compression largely beneficial to the world since it allows for inference to run on smaller systems which use less energy. On the other hand, when models may be run on smaller systems such as smartphones, easier use deep models covertly, for example, for facial recognition and surveillance. This may make abuses of deep learning technology easier hide.
REPRODUCIBILITY STATEMENT
The theoretical results of this paper, namely Theorem 3 Theorem Theorem 6 and Theorem may be independently verified through their proofs, which we include in their entirety in the appendices, including all necessary definitions, lemmas, and hypotheses in precise and complete mathematical language. The empirical verification of Section may be reproduced using the code included with the supplementary materials. In addition, Algorithm is written detailed pseudocode, allowing readers to recreate our algorithm in programming language of their choosing.
[
10

---

Under review as a conference paper at ICLR 2023
REFERENCES
Lei Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? arXiv:1312.6184, 2013: 2013.
Erkao Bao and Linqi Song. Equivariant neural networks and equivarification. arXiv:1906.07172 2019.
Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? arXiv:2003.03033, 2020.
David S Broomhead and David Lowe. Radial basis functions, multi-variable functional interpolation and adaptive networks. Technical report, Royal Signals and Radar Establishment Malvern (United Kingdom), 1988.
Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceed- ings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 535-541, 2006.
Yu Cheng, X Yu Felix, Rogerio S Feris, Sanjiv Kumar, Alok Choudhary, and Shih-Fu Chang. Fast neural networks with circulant projections. arXiv:1502.00 1502.03436, 2, 2015a.
Yu Cheng, Felix X Yu, Rogerio S Feris, Sanjiv Kumar, Alok Choudhary, and Shi-Fu Chang. An exploration of parameter redundancy in deep networks with circulant projections. In Proceedings of the IEEE international conference computer vision, pp. 2857-28666 2015b.
Yu Cheng, Duo Wang, Pan Zhou, and Tao Zhang. A survey of model compression and acceleration for deep neural networks. 17Xiv:1710.09282, 2017.
Benjamin Chidester, Minh N. Do. and Jian Ma. Rotation equivariance and invariance in convolu- tional neural networks. 18Xiv: 10255.23300 12301, 2018.
Djork-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint 1XXi:255/ 2015.
Taco S. Cohen and Max Welling. Group equivariant convolutional networks. In International
conference on machine learning (ICML), pp. 2990-2999, 2016.
Taco S Cohen and Max Welling. Steerable CNNs. In Proceedings of the International Conference Learning Representations (ICLR), 2017.
Taco S. Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling, Gauge equivariant convo- lutional networks and the icosahedral CNN. In Proceedings of the 36th International Conference on Machine Learning (ICML), volume 97, pp. 1321-1330, 2019.
George Cybenko. Approximation by superpositions a sigmoidal function. Mathematics of control, signals and systems, 2(4):303-314 1989.
Congyue Deng, O. Litany, Yueqi Duan, A. Poulenard, A. Tagliasacchi, and L. Guibas. Vector Neurons: A General Framework for SO(3)- (0(3)-Equivarian Networks. 2021 IEEE/CVF International Conference Computer Vision (ICCV), 2021. doi: 10.1 (U99iccv489222.2021..001
Sander Dieleman, Jeffrey De Fauw, and Koray Kavukcuoglu. Exploiting cyclic symmetry in convo- lutional neural networks. In International Conference on Machine Learning (ICMI., 2016.
Xin Dong, Shangyu Chen, and Sinno Jialin Pan. Learning to prune deep neural networks via layer- wise optimal brain surgeon. arXiv preprint arXiv:1705.07566 2017.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. 18X3:1180330333 2018.
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional net- works using vector quantization. arXiv:1412.6115, 2014.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. /10001/5 2015.
11

---

Under review as a conference paper at ICLR 2023
Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre
Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Ar- ray programming with NumPy. Nature, 585(7825):357-362 September 2020. doi: 10.1038/ 1415860020-2649-1 URL http::ttts//tt http::ttssss...rppppeeeee .tg.ii.rrgggooo wwt.:o.gg//oo...... /4158844166666 020 2649-2
Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon. Morgan Kaufmann, 1993.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv:1503.02531, 2015.
Kurt Hornik. Approximation capabilities of multilayer feedforward networks. Neural networks, (2):251-257, 1991.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv: 1ee.17044.4444 040004004461.. 2017.
George Jeffreys and Siu-Cheong Lau. Kähler Geometry of Quiver Varieties and Machine Learning. wttp://///aaaa/aaaaa///////////a 2144
arXiv:2101.11487, 2021, URL
Ehud D Karnin. A simple procedure for pruning back-propagation trained neural networks. IEEE transactions on neural networks, 1(2):239 242, 1990.
Patrick Kidger and Terry Lyons. Universal approximation with deep narrow networks. In Conference on learning theory, pp. 2306-2327. PMLR, 2020.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems, 30, 2017.
Risi Kondor and Shubhendu Trivedi. On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups. In International conference on machine learning (ICML), 2018
Leon Lang and Maurice Weiler. A Wigner-Eckart theorem for group equivariant convolution ker- nels. In International Conference on Learning Representations (ICLR), 2021.
Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan Oseledets, and Victor Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. arXiv:1412.6553, 2014.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598-605, 1990.
Namhoon Lee, Thalaiyasingam Ajanthan, Stephen Gould, and Philip HS Torr A signal propagation perspective for pruning neural networks at initialization. arXiv preprint arXiv: (006::9960003000 2019.
Yongxi Lu, Abhishek Kumar, Shuangfei Zhai, Yu Cheng, Tara Javidi, and Rogerio Feris. Fully- adaptive feature sharing in multi-task networks with applications in person attribute classification. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pp.
5334-5343, 2017a
Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang. The expressive power of neural networks: view from the width. Advances neural information processing systems, 30, 2017b.
Mirco Milletari, Thiparat Chotibut, and Paolo E Trevisanutto. Mean field theory of activation func- tions in deep neural networks. arXiv preprint arXiv:1805.08780 2018
Diganta Misra. Mish: A self regularized non-monotonic activation function. arXiv preprint
2019.
12

---

Under review as a conference paper at ICLR 2023
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440, 2016.
Jooyoung Park and Irwin W Sandberg. Universal approximation using radial-baske function net-
works. Neural computation, 3(2):246-257, 1991.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary VeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, pegforerfrr deep learning library. In H. Wallach, H. Larochelle. A. Beygelzimer, F. d Alché- Bucé E. Fox, and R. Garnett (eds.), Advances Neural Information Processing Systems (NeurIPS) 32, pp. 8024 8035. Curran Associates, Inc., 2019. URL http::tttp::ppppeepeee neur ccppppprrrcppprr
9015 pytorch an www.retteeeeeeeeeeeeeee style perforrrerrorccccee deneaaanceeeeee---eeee Sectp-eearnnn learning library. pdf
Prajit Ramachandran. Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv 2017.
preprint
Siamak Ravanbakhsh. Universal equivariant multilayer perceptrons. In International Conference on Machine Learning, pp- 7996-8006. PMLR, 2020.
Siamak Ravanbakhsh, Jeff Schneider, and Barnabas Poczos. Equivariance through parameter- sharing. In International Conference on Machine Learning, pp. 2892-2901. PMLR, 2017.
Roberto Rigamonti, Amos Sironi, Vincent Lepetit, and Pascal Fua. Learning separable filters. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2754-2761, 2013.
Frank Rosenblatt. The perceptron: probabilistic model for information storage and organization in the brain. Psychological review, 65(6):386, 1958.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules.
arXiv:1710.09829 2017.
Thiago Serra, Abhinav Kumar, and Srikumar Ramalingam. Lossless compression of deep neural networks. In International Conference on Integration of Constraint Progrumming, Artificial felligence, and Operations Research, pp. 417-430. Springe,,, 2020.
Thiago Serra, Xin Yu, Abhinav Kumar, and Srikumar Ramalingam. Scaling up exact neural network compression by relu stability. Advances in Neural Information Processing Systems, 34, 2021.
Sho Sonoda and Noboru Murata. Neural network with unbounded activation functions is universal approximator. Applied and Computational Harmonic Analysis, 43(2):233-268, 2017.
Gustav Sourek, Filip Zelezny, and Ondrej Kuzelka. Lossless compression of structured convolu- tional models via lifting. arXiv preprint arXiv:2007.06567, 2020.
Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, et al. Convolutional neural networks with low- rank regularization.
2015.
Chaoqi Wang. Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.03370 2020.
Ming-Xi Wang and Yang Qu. Approximation capabilities of neural networks on unbounded do- mains. Neural Networks, 145:56-67, 2022.
Maurice Weiler and Gabriele Cesa. General Equi--qqivariant Steerable CNNs. Conference on Neural Information Processing Systems (NeurIPS), 2019.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3D steerable CNNs: Learning rotationally equivariant features in volumetric data. Proceedings of the 32nd International Conference Neural Information Processing Systems (NeurIPS), 2018a.
13

---

Under review as a conference paper at ICLR 2023
Maurice Weiler, Fred A Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant CNNs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 849-858, 2018b.
Daniel E Worrall, Stephan Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Patter Recognition (CVPR). pp. 5028-52 1728-5037.2 2017.
Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. Quantized convolutional neural networks for mobile devices. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition (CVPR), pp. 4820-4828, 2016.
Dmitry Yarotsky. Universal approximations of invariant maps by neural networks. Constructive Approximation, 55(11:407-4445 2022.
Tianyun Zhang, Shaokai Ye, Kaiqi Zhang, Jian Tang, Wujie Wen, Makan Fardad, and Yanzhi Wang. A systematic DNN weight pruning framework using alternating direction method of multipliers. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 184-199, 2018.
14

---

Under review as a conference paper at ICLR 2023
A ORGANIZATION OF THE APPENDICES
This paper is contribution to the mathematical foundations of machine learning, and results motivated by expanding the applicability and performance of neural networks. At the same time, we give precise mathematical formulations of our results and proofs. The purposes of these appendices are several:
To clarify the mathematical conventions and terminology, thus making the paper more accessible.
2. To provide full proofs of the main results.
3. To develop context around various construction appearing in the main text.
4. To discuss in detail examples, special cases, and generalizations of our results.
5. To specify implementation details for the experiments.
We now give summary of the contents of the appendices.
AppendixBJcontainn proof' the universal approximation results (Theorems and stated in Section of the main text, as well as proof's of additional bounded width results. The proof's use notation given in Appendix B.I and rely on preliminary topological considerations given in Appendix B.2
In Appendix we give proof of the model compression result given in Theorem which appears in Section For clarity and background we begin the appendix with a discussion of the version of the QR decomposition relevant our purposes (Appendix C.T
We also establish elementary
properties of radial rescaling activations (Appendix C.2)
The focus of Appendix D is projected gradient descent, elaborating on Section We first prove result on the interaction of gradient descent and orthogonal transformations (Appendix(D.l) before formulating projected gradient descent in more detail (Appendix (.22). and introducing the so-called interpolating space (Appendix D.3). We restate Theorem 8 in more convenient notation (Appendix D.4 before proceeding to the proof (Appendix D.55)
Appendix E contains implementation details for the experiments summarized in Section 7 Several of our implementations use shifted radial rescaling activations, which we formulate in Appendix
E.1
Appendix F exxlains the connection between our constructions and radial basis functions networks. While radial neural networks turn out be specific type of radial basis functions network, our universality results are not implied by those for general radial basis functions networks.
B UNIVERSAL APPROXIMATION PROOFS AND ADDITIONAL RESULTS
In this section, we provide full proofs the universal approximation (UA) results for radial neu- ral networks, as stated in Section 4 In order to do we first clarify our notational conventions (Appendix B.I. and collect basic topological results (Appendix B.2).
B.1 NOTATION
Recall that, for a point in the Euclidean space R2 and positive real number denote the r-ball around e by (c) = {o R" Iz - cl < r). All networks in this section have the Step-ReLU radial rescaling activation function, defined as:
f2 )
if | otherwise
R
R"
Throughout, o denotes the composition of functions. We identify a linear map with a corresponding matrix (in the standard bases). In the case of linear maps, the operation be be identified with matrix multiplication. Recall also that an affine map I R one of the from L(x) = A for matrix A E R and l mm..
15

---

Under review as a conference paper at ICLR 2023
B.2 TOPOLOGY
Let K be a compact subset of R" and let f + Rn be continuous function.
Lemma 9, For any > O, there exist C1,..., CN K and ri..... ,, TN E (0, such that, first, the union of the balls B,, (ca covers K; second, for all we have (Br((c) K)) S Bc(((e)).
Proof. The continuity of f implies that for each с E K, there exists re such that f(Br, (c) n K) S (f((()). The subsets By, (c) n K form an open cover of K. The compactness of K implies that there is a finite subcover. The result follows.
We also prove a variation of Lemma of K contains the center point of another ball.
9 that additionally guarantees that none of the balls in the cover
Lemma 10. For any 0, there exist Cl..... CM E K and ri..... TM E (0,1) such that, first, the union of the balls Br, (ci) covers K; second, for all i, we have f (Br, (c:) C B.(((e)) and, third, |c Cyl T.-
Proof. Because f is continuous on compact domain, it is uniformly continuous. So, there exists > such that (c) K) _ (c) for each e E K. Because K compact has a finite volume, and so does (K) = UvER Br/z(c). Hence, there exists a finite maximal packing of (K) with balls of radius r/2. That is, a collection CI....,CM E (2))(K)) such that, for all i, (r,)((s) S (K) and, for all j * i, B,,2((a) n (x/2((;) = 0. The first condition implies that O2 K. The second condition implies that |4 - 91 2 r. Finally, we argue that K C U (c+). To see this, suppose, for contradiction, that E K does not belong to 0. and could be added the packing, which contradicts the fact that the packing was chosen to maximal. So the union of the balls (c(c,) covers K. C
U. (c).. Then B,,)(()))B,//(z1)
We turn attention to the minimal choices of N and M in Lemmas D and 10
Definition 11. Given f K - IR" continuous and 0, let N(.. K, c) be the minimal choice of N in Lemma 9 and Mff, c) be the minimal choice of M Lemma
Observe that M(, constants NJ, K) and MJ, K, e). As an illustration, we give the argument in the case that K is the closed unit cube in R" and ʃ K R is Lipschtiz continuous.
N(S, e)... In many cases, is possible to give explicit bounds for the
Proposition 12. Let K [0, 1j" R" be the (closed) unit cube and let f K - R be Lipschitz continuous with Lipschitz constant For any > O, we have:
RRn
T(n/2+11 +
(
2R n
K,f.Kec)
and
M(f.K,c)
2
2E
xn/2
RR/ " cubes of side Each cube is contained in a ball of radius & centered at the center of the cube. (In length Ryn
Proof. For the first inequality, observe that the unit cube can be covered with
2c
general, cube of side length a in R" is contained in ball of radius SyT Lipschitz continuity implies that, for all x. x' E K, J - x'l < </ then (f(z) - f(x')( S R|
For the second inequality, let r = c/R. Lipschitz continuity implies that, for all x, E K, if [x x'I < r then [f(x) [xx)] S R|æ x'l < The n-dimennional volume of the of points with distance most r/2 to the unit cube is vol vol(B,/22(K) < (1 +++) r)". The volume of ball with
x" n/2 radius r r/2 is vol( (0)) T(#/2+1) (r/2)". Hence, any packing of (ryz(K) with balls of radius r r/2 consists of most
vol vol(B,/2(K)) T(n/2+1) vol(B,/2(0)
(
2R
2
nn/2
such balls. So there also exists a maximal packing with at most that many balls. This packing can be used in the proof of Theorem 10, which implies that it is bound M K,e). n
16

---

Under review as a conference paper at ICLR 2023
We note in passing that any differentiable function f K R" compact subset K of R is Lipschitz continuous. Indeed, the compactness of K implies that there exists R such that (f() - R for all x E K. Then one can take R to be the Lipschitz constant of f.
B.3 PROOF OF THEOREM 3 UA FOR ASYMPTOTICALLY AFFINE FUNCTIONS
In this section, restate and prove Theorem 3] which proves that radial neural networks uni- versal approximators of asymptotically affine functions. We recall the definition of such functions:
Definition 13. A function ʃ : IR + R is asymptotically affine if there exists an affine function L: 1R + TT such that, for all >> 0, there exists a compact set K C IR" such that [T(x) f(x) <
e for all x E R" \ K. We say that L the limit f.
Remark 14. An asymptotically linear function is defined the same way, except is taken to be linear (i.e., given just by applying matrix multiplication without translation). Hence any asymptot- ically linear function is particular an asymptotically affine function, and Theorem applies to asymptotically linear functions well.
Given an asymptotically affine function f R + Rh and O, let K be a compact set as in Definition 13 We apply Lemma 9 to the restriction flk of f to K and produce minimal constant N = (J k; e) as in Definition I1
We write simply NJ, K, é) for this constant.
Theorem 3 (Universal approximation). Let f R² + Rm be an asymptotically affine function. For any there exists a compact set K R" and a function F :R IR such that:
1. F is the feedforward function of radial neural network with N K,) layers whose hidden widths are (n + 1, ....., + N).
2. For any IR", we have [F(x) f(x) <E
Proof. By the hypothesis f. there exists affine function L R - Rh and compact set K C IR" such that [L(x)= - f(x]] < c for all x E IR" \ K. Abbreviate N(f. K, c) by N. As in Lemma 9 fix CI,...,CN' E K and .,....,,'' E (0, such that, first, the union of the balls B, (c;)
covers K and, second, for all i, we have f (B,, (c)) S Be(((e)).. Let U U Br, (c:), so that
K C U. Define F: R. + 1 as:
L(x)
i & U
F(x) = (f(e)
where is the smallest index with Br, (c:)
If & U. then F(x)- - [(x)] = [[(x) f(x) < e. Hence suppose x E Let j the smallest index such that (c)). Then F(x) = f(g)). and, by the choice of ry, we have:
[F(x) - f()) = If(c)) f(x)] <c
We proceed to show that F' is the feedforward function of a radial neural network. Let el, eN be orthonormal basis vectors extending R R=+N We regard each Ra+i-1 subspace of gu+i by embedding into the first n + 1 coordinates. For I....,N, N, we set h4 /I r? and define the following affine transformations:
T: R pati-l + B=++
S, Bati + Ra+i
h + hes
H (1 (1+1) )) z)e + +
where (e,, 2) is the coefficient of e; in 2. Consider the radial neural network with widths (n, n 1,..., n + N, m), whose affine transformations and activations are given by:
For i 1,..., N the affine transformation from layer 1 layer i is given by z T, (z)1 where Sn idga.
The activation function at the i-th hidden layer is Step-ReLU on t that is:
if otherwise
Pj g
E B
0
17

---

Under review as a conference paper at ICLR 2023
The affine transformation from layer N to the output layer is
+ 2 L. SN
where Pr.fe is the affine transformation given by:
N
N 04e + L(x) Σ (a(((c) - I(ee)) ==1
T L. R - RT T
which can be shown to be affine when L is affine. Indeed, write L(x) Az + where A is matrix in Rmxx and Rm is vector. Then PL.fe is the composition of the linear map given by the matrix
[A f(a) - M(a) f(e2) [(e2)
s(eN) [(cN)] E m=*(4+N)
and translation by E R Note that we regard each ((G) - L(ei) E I as a column vector in the matrix above.
We claim that the feedforward function of the above radial neural network is exactly F. To show this, first state lemma, whose (omitted) proof is an elementary computation.
Lemma 3.1. For i = I,...,M, N, the composition S; o T; is the embedding Ri++--1 + Ruti
Next, recursively define G, R - R via
G. = S; o pi o T: o Gi-1,
where Go idga. The function G; admits an direct formulation:
Proposition 3.2. For i = 00, 1,..... N, we have:
x & Ui=1 (c)
Gi(z)
Cj tey
where j S is the smallest index with Bry(cs)"
Proof. We proceed by induction. The base step = 0 immediate. For the induction step, assume the claim is true for i 1, where ( < i- < N. There are three cases to consider.
Case 1. Suppose x & U)=11 Br, (gz). Then in particular x & U B,, (c)), so the induction Additionally, Br, (c+), S0:
hypothesis implies that G;-1(z) (z)
[T(x)] = j q + hacs
cl + h
1 + 12
Using the definition of and Lemma 3.1 we compute:
G₂(x)) = S, Ps o T2 Gz- = S, p TT(x) S, T7(x) (7(x))=0 x.
Case 2. Suppose By S B,, (c+) for some j < i-11 1. Then the induction hypothesis implies
that Gz-1(z) = Cj + ej. We compute:
[Tice + e;)l lcg t - + heed > =1
Therefore,
G,(z) S. Tilc + e,)) S, Ti(c + ej) Cj teg.
Case 3. Finally, suppose & E B, A B, (cz). The induction hypothesis implies that Gz- = Since x E Br, (c:), we have:
[T(()) = |r a heel = = + 113 < + 1
Therefore:
G((x) = o Ri()) 5, = c +
This completes the proof of the proposition.
18

---

Under review as a conference paper at ICLR 2023
Finally, show that the function F defined the beginning of the proof is the feedforward function of the above radial neural network. The computation is elementary:
feodfoorwrr L, SN PN TN SN-1 PN-1 Tv-1 o........ 0 TT
iI.ff Gn
where the first equality follows from the definition of the feedforward function, the second from the
definition of GN and the last from the case N of Proposition 3.2 together with the definition of fr.f.e- This completes the proof of the theorem.
L
B.4 PROOF OF THEOREM 5 BOUNDED WIDTH UA FOR ASYMPTOTICALIY AFFINE FUNCTIONS
We restate and prove Theorem 5 which strengthens Theorem by providing bounded width radial neural network approximation of any asymptotically affine function.
Theorem 5. Let f : TR Th be an asymptotically affine function. For any 6 > 0, there exists
compact set K C. IR and a function F: IR"" R such that:
1. F is the feedforward function of a radial neural network with N = =N(f, K, hidden layers whose widths are all n + m 1.
2. For any x IR"" we have [F(x) - f(x) <.
Proof. By the hypothesis f. there exists affine function L R Rh and compact
K C R such that 1(x)-f((x) < e for all x E R" K. Given > 0, let N = N(f. f K, e) and use Lemma to choose CN E and r1, ,....,,N''
.,,,,,, TN E (0, 1) such that the union of the balls Br, (c:) covers K, and, for all i, we have f(Br, (c)) C Bc(f(c)). Let be the minimal non-zero value of [f(c) f(c)) for f, [1....,.N] N}. that is, ming
[f(c,) f(c,)).
Using the decomposition R=+m+1 = R x I x IR, we write elements m=++++ as (x. y, 0),
where x E R", E RT, and 0 I. For
N, set:
(I) + x (1 - 0)a y - (ce) L(0) (1 - 0)h;
T B - Rutm-1
where h; V1 r.. Note that T; is invertible affine transformation, whose inverse is given by:
T: (x,,00) (x x h
0 G y (1- mm.) 1 L(0)
f(c))-(())
θ
hg
For i I.....,,. N, define G, : R gn+m+1 via the following recursive definition:
G, G o o T₃ o G
where Gy(x) = (x, 0, 0) R" + Ra+m+1 is the inclusion, and p Ra+m+1 + Ru+m+1 is Step-ReLU pat We claim that, for E R,, have:
(xx 0,0)
if x & U₁=1 (cg)
Gx(x) (0. (a))
where j S is the smallest index with x E Br,
This claim can be verified by straightforward induction argument, similar the one given in the proof of Proposition 3.2 and using the following key facts:
For x E R", [ ((x, 0, 0,0) 1(2 c,, 0,h,, < 1 if and only if æ cal < T/.
-77 = ((0, f(c.)-((1) 1).
. T. ((.(() [()) ,1) -0,, ((ss)-((e)) .). which, by the choice of s, has norm least
1 (ff((c) M(x),, and is 0f ((G) f(c)).
19

---

Under review as a conference paper at ICLR 2023
Let + 1=tm++ - R denote the affine map sending (I. to L(x) + sy. It follows that
F = @ GN satisfies
L(x) F(x) - (f(e)
x & Uن Bry Bry(c,)
where j is the smallest index with Br, (cz)
By construction, F is the feedforward function a radial neural network with N hidden layers whose widths are all n + + 1. Let E R If x E K, let j be the smallest index such that x B,, (c)). Then F(z) = (c and, by the choice of rj, we have (())=f((ee lf(c)) f(z) < c.
Otherwise, I K, and [F(x) [(x)] [L(x) f()) <e
B.5 ADDITIONAL RESULT: BOUND max(n, m)
We state and prove an additional bounded width result. In contrast to the results above, the theo- rem below only holds for functions defined on compact domain, without assumptions about the asymptotic behavior. The proof is an adaptation of the proof of Theorem 5| so we give only a sketch. R be continuous functio,, where K a compact subset of R.. For
Theorem 15. Let f
any 0, there exists F R + R. such that:
1. F is the feedforward function of a radial neural network with Nff, K,c) hidden layers whose widths are all max(n, m) + 1.
2. For any E K, we have [F(x) - f(x)] <.
Sketch of proof. The construction appearing in the proof of Theorem with = 0 can be used produce radial neural network with N(,, Kee) hidden layers with widths + approximates I on K. (Note that the approximation works only on K, as is not defined outside of K.) All values in the hidden layers are of the form (z, 0, or (0, y, 1). We can therefore replace (ar, y, 0) E g=+m+1 by (x + y, E mm, = [ everywhere, without affecting any statements about the hidden layers. In particular, the transformation T; becomes
that
[ (1 (1-0)h;)
T: I (n, mm
(z, + x -- 0)c;
With this change the final affine map & sends (x, 0) sr. From the rest of the proof of Theorem it follows that the feedforward function F' of the radial network satisfies [F(x) f(x) < e for all K.
B.6 ADDITIONAL RESULT: BOUND max(n, m)
In this section, we prove different version of the result of the previous section. Specifically, we reduce the bound on the widths to max(n, m) at the cost using more layers. Again, we focus on functions defined on compact domain without assumptions about their asymptotic behavior. Recall the notation Mff, K, <) from Theorem and Theorem 11
Theorem 16. Let f K + R be continuous function, where K is compact subset of R for n 2 2. For any > 0, there exists F: IR. + R. such that:
1. F is the feedforward function of radial neural network with 2M (f, (f,K,)/2) hidden layers whose widths are all max(n, m).
2. For any x. E K, we have [F(x) - s(x)) <E
Proof. We first consider the proof in the case m. Set M M, c). As in Lemma 10
fix Ci...., CM E K and ri,..., PM E (0. 1) such that, first, the union of the balls Br, (c:) covers K; second, for all i, we have (B,((e)) Bepz((())) and third, |Gs egl 2 r; for i # j. For
i i= 1,..., M, set
T RD + R"
and recursively define G, : + Ra as G. = T: o ooo o G where Go = idg is the identity on I2" and p R R is Step-ReL.U.
20

---

Under review as a conference paper at ICLR 2023
Lemma 16.1. For i 0,1,- 0.l,..., N. we have:
x G((z)
x & Uj=1 (c;)
cj
where j S is the smallest index with e Br, (c)).
We omit the full proof of Lemma T6. it is standard induction argument similar Proposition 3.2 relying on the following two facts. First, [z) < 1 if and only if x E Br, (4). Second, by the choice of c,, we have c; cj r; for all + j. This implies that [Ti(e;)] 2 for + j.
Next, perform the following loop over i = I....,:: M:
Set P:-1 (a (CI.-..,CM)) .CM) U {d1 : d=113
Choose d; in (e))(((()) that is not colinear with any pair of points in P:-1- This is where we use the hypothesis that 2 2.
Let be the minimum distance between any point on the line through and d; and any point P;-1 \ {...
Let U3: R" + R" be the following affine transformation:
di 1
1
(x d,, di) (c. - d:)
Up R2 R",
$i c4 dil
31
1a - d;1
Define H,
R R recursively as H, = U,² o po U. o H1-1, where Ho = idga.
We note that the transformation Uj can also be written A,(x - d:) where A, is the linear map given by A, = J, proj -d)+ + -a (roj(c which involves the projections onto the line spanned by c: - d, and onto the orthogonal complement of this line.
Lemma 16.2. For i,j 1,..., M, we have:
(1] - [d,
i j >
H:(e,)
[cg
Proof. It is immediate that (i(d)) = and [U(())) = 1/2. It is also straightforward to show, using choice of 8,, that [U:()) 2 1 for all p E P:-1 \ {G). follows that U, o p o sends e; d, and fixes all other points in P--1-
Lemma 16.3. For & E K, we have M (x) d, where is the smallest index with x E (c:)
Proof. Let x E K. By Lemma 16.1 we have that Gm(()) = G where i is the smallest index with Br, (c:). (We use the fact that the balls (B,, cover K.) By Lemma 16.2 we have that
HM (G.) = d, for all i. The result follows.
Set F Hy Gas- We see that, for x K:
[F(x) f(x) = [d. f(x) < Id: (
+
- f(x)) </22+/22-
where i is the smallest index with æ E Br, (c). We show that is the feedforward function of Indeed, take the affine
radial neural network with 2M hidden layers, all of width equal transformations and activations as follows:
For i 1, 1,..., M the affine transformation from layer i layer i is given by T. [ where To = idg
For i = 1, 1.....M. the affine transformation from layer M + - to layer M + i given by x U, U. (x), where Un T
The activation at each hidden layer is Step-Rett ReLU on R" that p(x) = x if |x| and otherwise.
21

---

qr that computes the Under review as a conference paper at ICLR 2023
Layer 2M + 1 has the affine transformation Unr-
It is immediate from definitions that the feedforward function of this network is F.
To conclude the proof, we discuss the cases where n * m. Suppose n < m so that max(n, m) = m.
Then we can regard K compact subset of R and apply the above constructions. Suppose n > m so that max(n, m) = n.. Let inc R + IR". Apply the above constructions to the function inc o f K R"
=
C MODEL COMPRESSION PROOFS
The aim of this appendix to give proof of Theorem In order to so, we first (1) provide background relevant version of the QR decomposition, and (2) establish basic properties of radial rescaling activations.
C.1 THE QR DECOMPOSITION
In this section, recall the QR decomposition and note several relevant facts. For integers and m, let (Raxm) uppperrerrrrrre denote the vector space of upper triangular n by m matrices.
Theorem 17 (QR Decomposition). The following map is surjective:
((n) (Ruxm) upper
Raxm
Q.R.
QoR
In other words, any matrix can be written as the product of an orthogonal matrix and an upper- triangular matrix. When n, the last n rows of any matrix in (IR "R**")"P" y zero, and the top m rows form an upper-triangulan m by m matrix. These observations lead to the following "complete" version of the QR decomposition, which coincides with the above result when 2
Corollary 18 (Complete QR Decomposition). The following map surjective:
::: (n)
(R (R+xm) upper
Raxm
Q.
inc R
where k = min(n, m) and inc R + Ra is the standard inclusion into the first k coordinates.
We make some remarks:
1. There are several algorithms for computing the QR decomposition of a given matrix. One is Gram-Schmide orthogonalization, and another the method of Houscholder reflections. The latter has computational complexity O(n² in the case of n matrix with n 2 m. The package numpy includes a function QR decomposition of matrix using Householder reflections.
2. each iteration of the loop in Algorithm [ the method QR-decomp with mode ecomplete takes input matrix A, of size x (nted + 1), and produces an or- thogonal matrix Qi E (n,)) and an upper-triangular matrix R; of size min(n,, need m, +
1) x (n-r) 1) such that A, = Q, inc, R, Note that not min(n4, na + 1).
3. The QR decomposition is not unique in general, or, in other words, the map ju is not injec- tive in general. For example, if m, each fiber of y contains copy of the orthogonal group 0(n-m) m).
4. The QR decomposition is unique (in certain sense) for invertible square matrices. To be precise, let B be the subset of (R"*a) upper consisting of upper triangular n by n matrices with positive entries along the diagonal. Both B, and ((n) subgroups of the general linear group CL,,R),, and the multiplication map (n) x B + GL,, (R) is bijective. However, the QR decomposition is not unique for non-invertible square matrices.
22

---

Under review as a conference paper at ICLR 2023
C.2 RADIAL RESCALING FUNCTIONS
We now prove the following basic facts about radial rescaling functions:
Lemma 19. Let p = h(m) R + R" be a radial rescaling function on IR..
1. The function p commutes with any orthogonal transformation Rf. That is, po Qop for any Q E (n).
2. If m Σ n and inc Imm + RR is the standard inclusion into the first m coordinates, then: h(m) inc inc (mm)
Proof. Suppose Q (n) is an orthogonal transformation of R Since Q is norm-preserving, we have [Qul = |u] for any TR. Since Q linear, we have Q()v) XQu for any X E and E R Using the definition of h(m) we compute:
pQv)) = h( Qv1) [Qul Qu
h(vl))
(
( :) Q(((v)).
Qu =0=Q/
The first claim follows. The second claim is an elementary verification.
More generally, the restriction of the radial rescaling function to a linear subspace of IR" radial rescaling function that subspace. Given tuple radial rescaling functions p
(p: R + ) suited to widths = (n:)/2 we write pr (pted
Rn!ed Ra! for
the tuple of restrictions suited to the reduced widths nd so that ppod pi IRased
C.3 PROOF OF THEOREM 6
Adopting notation from above and Section 5 we now restate and prove Theorem 6
Theorem 6. Let (W, b. p) be radial neural network with widths n. Let Wied and b be the weights and biases of the compressed nenwork produced by Algorithm The feedforward function of the original network (W, b,P) p) coincides with that of the compressed network (Wied bred pred).
Proof. Let (Wred brd Q) OR-Compress(W, b) be the output of Algorithm Q E Onhiid) and (Wred bra r is a neural network with widths n red and radial rescaling ac-
that
tivations p.dd Pc Let F(W,b.)) denote the feedforward function of the radial neural
network with parameters (W, b) and activations p. Similarly, let Fred = F F(weelbee brett denote the feedforward function of the radial neural network with parameters (Wred bred) and activations p peed Additionally, we have the partial feedforward functions F; and Frred We show by induction that
F; Qi inc; pred
for any 0, I, ...... ........ N. (Continuing conventions from Sections 5.1 and 5..2 we set Qo idg QL = idges. and inc, Ra > R to the inclusion map.) The base step = 0 immediate. For the induction step, let & R0 Then:
F((x)) Ps (W, o F,-1(x) + by)
p (W, o Q o inc,-1 prod (x) + by)
W, Qi-1 inc;-1] Ai bb
Fred (x))
Q Q
inc, [irpd Winw]
prodd (x))
Q o inc; o
(Wred,
pred + (inn))
Royed
Qi inc; pred
23

---

Under review as a conference paper at ICLR 2023
The first equality relies the definition of the partial feedforward function F;; the second on the induction hypothesis; the fourth on an inspection of Algorithm |
noting that R, = [6 W? the
fifth on the results of Lemma 19 observing that Pr inc; Ps Polg = inc, p. and the sixth on the definition of pred. In the case i I., have:
F F2 Qz incz FL prod
since Qr. = inc,, = idga and Fged = pred, The theorem now follows.
The techniques of the above proof can be used to show that the action of the group O(niil) of orthogonal change-of-basie symmetries on the parameter space Param(n) leaves the feedforward function unchanged. We do not use this result directly, but state is precisely it nonetheless:
Proposition 20. Let (W,b, p) be a radial neural network with widths vector n. Suppose g O(n")) Then the original and transformed networks have the same feedforward function:
F Fig W. P) F(w,b, b.
In other words, fix parameters (W, b) E (aram(n), radial rescaling activations p. and g O(nhid) Then the radial neural network with parameters (W, b) has the same feedforward function as the radial neural network with transformed parameters (g W.g b), where we take radial rescaling activations p in both cases.
We remark that Proposition 20 is analogous to the "non-negative homogeneity" (or "positive scaling invariance") of the pointwise ReLU activation functior(] In that setting, instead of considering the product orthogonal groups O(n hid), one considers the rescaling action of the following subgroup of GL,,::
T--1
G g (g:) E II GL,, each gi is diagonal with positive diagonal entries 1=1
Note that G is isomorphic to the product II R and the action on Param(n) is given by the same formulas those appearing the end of Section 5.1 The feedforward function of a MLP with pointwise ReLU activations is invariant for the action of G' on Param(n).
D PROJECTED GRADIENT DESCENT PROOFS
In this section, we give a proof of Theorem g
which relates projected gradient descent for a repre- sentation with dimension (usual) gradient descent for the corresponding reduced representation with dimension vector rdd This proof requires some set up and background resutls.
D.1 GRADIENT DESCENT AND ORTIIOGONAL SYMMETRIES
We first prove a result that gradient descent commutes with invariant orthogonal transformations. This section is general and departs from the specific case of radial neural networks.
D.1.1 SETTING
Let L V IRP + I be smooth function. Semantically, V is a the parameter space of neural network and L the loss function with respect to a batch of training data. The differential dC, of L
"See Armenta and Jodoin, The Representation Theory of Neural Networks, arXiv:2007. 12213: Dinh. Pas- canu, Bengio, and Bengio, Sharp Minima Can Generalize For Deep Nets, ICML 2017: Meng. Zheng, Zhang, Chen. Ye, Ma. Yu, and Liu, G-SGD: Optimizing ReLU Newral Networks in its Positively Scale- Invarian Space, 2019; and Neyshabur, Salakhutdinov, and Srebro. Path-SGD: path-normalized optimization in deep neural nenworks, 15.
24

---

Under review as a conference paper at ICLR 2023
E V is row vector, while the gradient V. of C at is column vector
8C
8e1
dC,
9C
ac EtEp
voL=
D2
ac Hip
Hence V.C the transpose of dl.,, that is: voL= (dd.)² A step of gradient descent with respect to C at learning rate y 0 is defined as:
V V
V.L
We drop from the notation when it is clear from context. For any &o 2 0, we denote by Tth the k-fold composition the gradient descent map 7
7 = -0700700007
D.1.2 INVARIANT GROUP ACTION Now suppose p: G + GL(V) is action of Lie group G' V such that L is G-invariant, i.e.: (()(g)(v) [(v)
for all g G and v E V. We write simply v for (v)) and for p(g).
Lemma 21. For any u E V and g G, we huve:
v.C= g" (V g.
Proof. The proof computation:
V.C. (d,f)* = =g" (V
(dLg.v dg ------0- (dL o g)T gT od
The second equality relies on the hypothesis that C L, the third on the chain rule, and the fourth on the fact that dg, = g since g a linear map.
One can perform the computation of the proof in coordinates, for I,..., p:
OL
o(Log) 0L 0g;
(V.L), = (dd,)' dr, bv Ør;
dx, gu dx,
= (V  = ( (V g) (9" vg0 C)
D.1.3 ORTHOGONAL CASE
Furthermore, suppose the action of by orthogonal transformations, so that p(g)² p(gg) for all g E G. Then Lemma 21 implies that
Vg V.L
(D.1)
for any E V and E G. The proof of the following lemma is immediate from Equation D.I together with the definition of y. See Figure an illustration.
Lemma 22. Suppose the action of G V is by orthogonal transformations, and that L is G- invariant. Then the action of commutes with gradient descent (for any learning rate). That is,
1th (g- u) 10- (v))
for any E V, g E G. and 2 O.
'Following usual conventions, regard column vectors elements of V and row vectors elements of the dual vector space V", The differential dh. of u also known the Jacobian of at V.
25

---

Under review as a conference paper at ICLR 2023
Figure 6: Illustration of Lemma 22 If the loss invariant with respect to an orthogonal transforma- tion Q of the parameter space, then optimization of the network by gradient descent is also invariant with respect to Q. (Note: in this example, projected and usual gradient descent match; this is not the case in higher dimensions, as explained in (.6)
D.2 GRADIENT DESCENT NOTATION AND SET-UP
We now urrn our attention back to radial neural networks. In this section, we recall notation from above, and introduce new notation that will be relevant for the formulation and proof of Theorem B
D.2.1 MERGING WIDTHS AND BIASES
Let n = (no, 71, ......, f1-1, R2) be the widths vector an MLP. Recall the definition of (nram(n) the parameter space of all possible choices of trainable parameters:
Param(n) = (IR" X x 2 x ... x Ra.xas-)) x (IR"* x IR x ... X R*
We have been denoting element therein R", R=,xn(-1){_ are the weights and b = (b; E R.: ) are the biases. However, this appendix we adopt different notation. Observe that, placing each bias vector extra column the left of the weight matrix, we obtain matrices:
pair of tuples (W, b) where W (W;
A = [bs W.] (=
Thus, there is an isomorphism:
Param(n) Rn.((_-111) p = (mxx(m)1)) .... (
In this appendix, we regard an element of Param(n) as a tuple 'merged' matrices A = (A, e R (2x )/
We now define convenient maps to translate between the merged notation and the split notation. For each i, define the extension-by-one map from R to R x Rd; R as
follows:
ext, : Rt - R
V (vi, 02, .2....., Un.) + (1, 01, 02 .
(D.2)
Observe that, for any i and x E R we have
A; ex(;)1 Wi + b;-
Consequently, the i-th partial feedforward function can be defined recursively as:
F; Pt A₁ ext;-1 F;
(D.3)
where P: R. + Ro: the activation at the i-th layer, and Fg is the identity R
D.2.2 ORTHOGONAL CHANGE-OF-BASIE ACTION
To describe the orthogonal change-of-basse symmetries of the parameter space in the merged nota- tion, recall the following product of orthogonal groups, with sizes corresponding to the widths of the hidden layers:
O hid O(ni) O(n2) x O(nI.-1
'In this general formulation, P: he any piece-wise differentiable function; for most of the rest of the paper will be interested the case where Pi radial rescaling function.
26

---

Under review as a conference paper at ICLR 2023
In the merged notation, the element Q (Q:) O(nhid)
transforms A Param(n) as:
1
-221
0
)
1
A
Q-A: A :=
Qi
0
0 @
(D.4)
=
where Qo id and Qr. -dn..
D.2.3 MODEL COMPRESSION ALGORITIIM
We now restate Algorithm D in the merged notation. We emphasize that Algorithms mathematically equivalent; the later simply uses more compact notation.
| and
Algorithm 2: QR Model Compression (OR-compress)
input : : E (aram(n)
output QQ E O( and V E Param(n*d)
Q. V < []:[]
// initialize output matrix lists
M1 < A1
for f- 1 to L - do
// iterate through layers
Qi, R2 +- QR QR-decomp (M, mode 'complete') // M₁ = Qi inc, R
Append Q. to
Append R. to
// reduced merged weights for layer i // transform next layer
1 0
Set M,+1 + A+t1 o
0
Qi o ine;
end
Append Mt V
return Q, V
We explain the notation. As noted in Appendix B.1 the symbol "o" denotes composition of maps, matrix multiplication in the case of linear maps. The standard inclusion inc; : R R. maps xn!"
into the first ned coordinates. As a matrix, Inc; E R"*,", has ones along the main diagonal and zeros elsewhere. The method QR-decomp with mode QR decomposition of the nı x (1 + m matrix M, as Q o inc; o RG where Qj E ((n,) and RG is upper-triangular of size nz! x (1 + m1 ). The definition of nied implies that either no.dd = -n!"d +1
'complete' computes the complete
niped n4- The matrix R, is of size nend nndd in the former case and of size n, X (1 n=:o in the latter case.
D.2.4 GRADIENT DESCENT DEFINITIONS
As in Section 6 we fix:
widths vector = (no. Ri,..., m..)
tuple p (PI....,PP) of radial rescaling activations, where : R - Rd for i 1.....L.
batch of training data ((rg,15)) C RR.00 x R Rindd x Ro
cost function C T x 1 + IR
As result, have loss function Param (n):
L: (a)am(n) - IR
((A) =
where F(A,pi is the feedforward of the radial neural network with (merged) parameters A and ac- tivations P. We emphasize that the loss function L depends the batch of training data chosen above; however, for clarity, we omit extra notation indicating this dependency since the batch of training data fixed throughout this discussion. Similarly, have:
the reduced widths vector nod_ (nge .n . n
27

---

Under review as a conference paper at ICLR 2023
the restrictions pmm (pp ...., PL p) where pi : R! for i 1, 1,..., L.
Using the fact that mi nog and L n,, there is loss function Param(n
red Param(n mo)) + R
Crdd B) = Σ C(F (2y))
where peat) is the feedforward of the radial neural network with parameters B E Param(n and activations pred, (Again, technically speaking, the loss function Leed depends on the batch of training data fixed above.) For any learning rate > O, we obtain a gradient descent maps:
y Param(n)
+ (n)am(n)
rid Pn rody
+ Param(n
A + A VAC
B B 1/V
D.3 THE INTERPOLATING SPACE
In this section, we introduce a subspace Param (n) of Param(n), that, as we will later see, inter- polates between Param(n) and Param(n'd))
Let Param" (n) denote the subspace of (aram(n) consisting of those T (Ti...., ...... Tt) Param(n) for which the bottom left (n,-n, - x x01 (1+m,=1) block of is zero for each i. Schemat- ically:
T
where the rows are divided re!ed top and n; aod on the bottom, while the columns are divided as (1 + n=:4) on the left and n4-1 n2;1 on the right. Let
Param (aram""(n) (n) + Param(n)
be the inclusion. The following proposition follows from an elementary analysis of the workings of Algorithm (or, equivalently, Algorithm
Proposition 23. Let A E Param(n) and let Q E (nhid) be the tuple of orthogonal matrices produced by Algorithm 2 Then Q-1 belongs to Param (n))
Define map
Q1 Param(n) - Paraminl(1) (n) by taking A E (n)am(n) and zeroing out the bottom left (n; - nned) (1 + ++(4)) block of A; for each i. Schematically:
1)² L
(A( ((0
L
A A
==1
i=1
It is straightforward to check that 91 is well-defined, surjective linear map. The transpose of qı is the inclusion 41- We summarize the situation in the following diagram:
Param
int (n)
(n)am(n)
(D.5)
91
We observe that the composition o i is the identity on Param int (n).
D.4 PROJECTED GRADIENT DESCENT AND MODEL. COMPRESSION
Recall from Section 6 that the projected gradient descent map on Param(n) is given by:
Yprjj (aram(n) - Param(n). A + Proj (A VC)
where = (W, b) are the merged parameters (Appendix D.2). and, in the notation the previous section, the map Proj is 4 41- To reiterate, while all entries of each weight matrix and each bias vector contribute to the computation of the gradient V VAL V V(W,b)C only those not the bottom left submatrix get updated under the projected gradient descent map 7proj.
28

---

Under review as a conference paper at ICLR 2023
Let Q.Q = (R-Compress(A) be the outputs of Algorithm 2 (which is equivalent to Algorithm
D. so that (W ( Param (n are the parameters of the compressed model corre- sponding to the full model with merged parameters A (W, b), and Q E O(n (nhh is orthogonal change-of-basis symmetry of the parameter space. Moreover, where we use the change-of-basis action from Appendix D.2 and Proposition 23 We have the following rephrasing of Theorem
Q-
Param t (n),
Theorem 24 (Theorem
Let A E Param(n), and let V, Q, T be above. For any k 2 0:
1.
2. A Projj (T) nadl +
More precisely, the second equality is "prio (T) ( (V)) + - (V) where :: Param(n
Param(n) is the inclusion into the top left comerr in each coordinate. Also, in the statement of Theorem we have U T
We summarize this result in the following diagram. The left horizontal maps indicate the addition of U T V, the right horizontal arrows indicate the action of Q, and the vertical maps various versions gradient descent. The shaded regions indicate the (smallest) vector space to which the various representations naturally belong.
Param red
Param
n
Param n
+T
V
T
W
GD Pa
proj-GD Param(()
GD Param(n)
GD on Param(n)
Frod V)
d: proy T
* T
7 (W)
D.5 PROOF OF THEBOREM g
We begin by explaining the sense in which Param inl (n) interpolates between Param(n) and Param m)). One extends Diagram D.5 as follows:
Param(n rd)
Param ((n)
Param(n)
95
The map
Param(n - Param (m) (n)
takes B (B,) E Param (n)) and pad each matrix with nı nieed rows of zeros the bottom and n need columns of zeros the right:
L
B = (B)L. + (B) = |
2=1
It is straightforward to check that is well-defined injective linear map.
The map
Ø2 : "aram""(()) Param(n
29

---

Under review as a conference paper at ICLR 2023
extracts from T the top left niad
(1 + n;= matrix:
T (1) T;
T )
+ (a(T) -T.!1) 4
T
It is straightforward to check that 92 is a surjective linear map. The transpose of d2 is the inclusion 22.
Lemma 25. We have the following:
The inclusion L: Param (n 4 Param(n) coincides with the composition 41 1g and commutes with the loss functions:
Param (n 06
(n)am(n)
Ciel
2. The following diagram commutes:
Param n
q2
> Param (n
End
Param(n)
I
3. For any T E Param (n), we have: (V,, = (V Cre))
Proof. We have the following standard inclusions into the first coordinates and projections onto the first coordinates, for i 0, L:
inc, = inc red
IR" R
inc; = inc 1n wtw.eedd.t1.. ,1+m
R 1ngg
+ R 1+,,
2 R
म. R 1tn; IR !
Observe that Param int (n) is the subspace of (aram(n) consisting of those T (Tı,..., T2) Param(n) such that:
(idn, inc; mT) o Ti inc,-1 7
for i 1, I,
By the definition of radial rescaling functions, for each i L,..., L. there is pisce-wise differentiable function h4 R R such that P h (n:). Note that pndd h (m:)) inc; = inc; o (x
(m and
The identity
12 follows directly from definitions. To prove the commutativity of the first diagram, enough to show that, for any X in Param md)) ). the feedforward functions of and (() coincide. This follows easily from the fact that, for =1...., L, we have:
o 1 (n.) inc; ग. o inc; o (n (n!"
For the second claim, let E Param feedforward function. Recall the ext, maps and the formulation of the feedforward function in the merged notation given in Equation D33 Using this set-up, the key computation is:
1nlm (n). suffices to show that e (T) and 02 (T) have the same
inc h (n To, o T, ext,,-1 inc;-1
(mm)) inc; n. T. inc;- ext
(aas)) o T inc; o ext
(m:) T₁ ext,,-, inc;-1
30

---

Under review as a conference paper at ICLR 2023
which uses the fact that (id,, inc, o kz) o T, onc;-1 = 0, or, equivalently, inc, ग. oT, inc,-1 T; o inc;-1, as well as the fact that ext; inc; = inc; ext, Applying this relation successively starting with the second-to-lass layer (i = L - 1) and ending in the first (a = 1), one obtains the result. For the last claim, one computes V u)) in two different ways. The first way is:
Vi(Cou) (d(CT oI)) (T) dTT))² (d ou)
= (T) (T))
where the fact that is linear map whose transpose is Ωı- The second way the commutative diagram of the second part of the Lemma:
Vi(Cou) VT (Cred o 22) (Croa) o 090)² (dd o (92)2)
= (Cred) 92) 4 (d (L(ed)) 12 (Vea(T)
We also use the fact that 92 is a linear map whose transpose is
Proof of Theorem
As above, let R, Q = QR (R-compress(AA (A) be the outputs of Algorithm so that V (W b E Param n rd) is the dimensional reduction of the merged parameters A (W, b). and Q O(nid). Set T Q-1 A Param" (n).
The action of Q E O( on (aram(n) is an orthogonal transformation, so the first claim follows from Lemma 22
For the second claim, it suffices to consider the case 1. The general case follows similarly. We proceed by induction. The base k 0 amounts Theorem 6 For the induction step, we
(k) ( (V)) (V).
Each (*)) belongs to Param int (n). so in (Z(k) = (k) Moreover, Q (Z(+) = - (V). We compute:
1-1 1 (Q-1. A) Ppoij (1iing) (Q- 1-- AA)
proj
Pproo (1(Tred (V)) + T - ((V))
(V)) 11 41 (V)) + +T -(V) -V (Vn
=<(Stta()) 11 91 (Vi(zzi))) C) (VV)
=((ria(V)) 02 (V )) + T - -(V)
= (1ta (V) V / (v) Food +T T-(())
1 = (V)) + T (VV)
where the second equality uses the induction hypothesis; the third invokes the definition of Tproj: the (n):
fourth uses the fact that (k) a (V)) + T ((V) belongs to Parami" [n:; the fifth and sixth
use Lemma 25 above; and the last uses the definition of Tred-
D.6 EXAMPLE
We now discuss an example where projected gradient descent does not match usual gradient descent.
Let n (1,3,1) be widths vector. The space of parameters with this widths vector is 10- dimensional:
Param(n) Hom R R @ Hom(R R) R0
We identify a choice of parameters (in the merged notation)
aa h7
A
Ai
Le A = lg
d
h
5] Param((1,3, 1))
(D.6)
fl
31

---

Under review as a conference paper at ICLR 2023
with the point p b, ( in R To be more explicit, the weights for the
first layer are W1 bb the second layer are W = [h jl. and the bias for the output layer is b2 = g.
the bias in the first hidden hidden layer is b1 (a, c, e), the weights for [f]
The action of the orthogonal group (nn) ((3) on Param(n) 10 can be expressed as:
Q 0
9 0
0
0
0 0
Q
where the rows and columns are divided according to the partition 3-3-1++ 10. Consider the function
L Param(n) + R
p= (a, d, f. h, i, h(a 6) +j +g
By the product rule, we have:
v,C (i,,,,,,,,,,,,,, d,
One easily checks that C(Q p) C(p) and that VQ. = Q vpL for any Q E 0(3).
The interpolating space is the (using the notation of Equation D.6). Suppose p' (a, b, c, d, 0,0, h, i,j) belongs to the interpo- lating space. Then the gradient is
subspace of Param(n) ~ R 10 with = f =
Cp (h, 1, 1, 6, d,
which does not belong to the interpolating space. So one step of usual gradient descent, with learning rate 17 > 0 yields;
:p' = (a, b, d 0, 0, h, i.
(a 1ph nh c m/i rji, - T -1 1. h v1aa + b) n(c d) i)
On the other hand, step of projected gradient descent yields:
Tproj : p' = (a, b, d, 0, 0. g, h i, +
(a n b 1/h, c mpi - rp 0, 0, 9 n, 11 + b) i 17c + d) j)
Direct computation shows that the difference between the evaluation of L after one step of gradient descent and the evaluation of L after one step of projected gradient descent is:
[(>(p') ((Ypoo((p)) = 2nj².
E EXPERIMENTS
As mentioned in Section we provide an implementation of Algorithm iin order to (1) empirically validate that our implementation satisfies the claims of Theorems 6 Theorem 8 and (2) quantify real-world performance. Our implementation uses generalization of radial neural networks, which we explain presently.
E. RADIAL NEURAL NETWORKS WITH SHIFTS
In this section, consider radial neural networks with extra trainable parameter in each layer that shifts the radial rescaling activation. Adding such parameters allows for more flexibility in the model, and (as shown in Theorem 26 the model compression of Theorem Tooldd for such networks. is this generalization that we use in our experiments.
For A Param(n). the neural function of the neural network with affine maps determined by A and identity activation functions is R + R; J. L(W)x. The function C can appear loss function for certain batches of training data and function IR.
32

---

Under review as a conference paper at ICLR 2023
Let h R R- R be function. For any 2 and any IR, the corresponding shifted radial rescaling function on I= given by:
p (m.4) + h(1ol t)
|
if # 0 and p(0) 0. A radial neural network with shifts consists of the following data:
1. Hyperparameters: A positive integer L and a widths vector n = (no. R2, .2...... mL).
2. Trainable parameters:
(a) A choice of weights and biases (W, b) E Param(n).
(b) A vector of shifts =
E R²,
3. Activations: A tuple h (h1,...,,hy) h2) of piecewise differentiable functions IR + R.
Together with the shifts, we have the shifted radial rescaling activation Ps h1 (n..t)) I Rd in each layer.
The feedforward function a radial neural network with shifts is defined in the usual recursive way, in Section The trainable parameters form the vector space Param(n) x R4, and the loss function of a batch of training data {(T,, w)} C. I2 x 1 is defined as
C (aram(n) R_ R;
(W,t) EC(F(w.aan(())..)
where F(w.b,t.)) is the feedforward function of a radial neural network with weights W, biases b, shifts t, and radial rescaling activations produced from h. We have the gradient descent map:
7 (n)am(n) x R- - Param(n) R2
which updates the entries of W. b. and t. The group Onnidd) 0(m)) x ... x O(nz-1) acts on Param(n) as usual (see Section 5.1 and on IR_ trivially. The neural function is unchanged by this action. We conclude that the (n mid action on Param(n) R commutes with gradient descent y. We now state generalization of Theorem 6 for the case of radial neural networks with shifts. We omit a proof, as it uses the same techniques as the proof of Theorem 6
Theorem 26. Let (W, b, h) be radial neural network with shifts and widths vector Let Wred and brdd be the weights and biases of the compressed network produced by Algorithm The feedforward function of the original network (W, b, t, h) coincides with that of the compressed network (Wred) bred h)
Theorem 8 also generalizes to the setting of radial neural networks with shifts, using projected gradient descent with respect to the subspace Param int (n) x IR' of Param(n) RR..
E.2
IMPLEMENTATION DETAILS
Our implementation written in Python and uses the QR decomposition routine in NumPy Harris et al. (2020). We also implement general class RadNet. for radial neural networks using PyTorch Paszke et al. 2019). For brevity, we write W for (W, b) and Wred for (Wred, b
(1) Empirical verification of Theorem 6 We synthetic data learn the function f(x) with N = 121 samples 2 = -3 -3+ j/20 < < 121. We model fw as radial neural network with widths n = 61,6,7 6, 1) and activation the radial shifted sigmoid ((x)) = 1/(1+e-=1t
Apply- ing CR-comprees gives radial neural network f with widths n ned (1,2,3, 1). Theorem implies that the neural functions of fw and fwis are equal. Over 10 random initializations W, the mean absolute error (1/N) E, [fw (x,) (waa(z,)) = 1.31 10-8 + 4.45 10-9. Thus Jw and fin agree up to machine precision.
(2) Empirical verification of Theorem 8 Adopting the notation from above, the claim is that training fq- with objective L by projected gradient descent coincides with training fwee with objective Lred by usual gradient descent. We verified this on synthetic data using 3000 epochs at learning rate 0.01. Over 10 random initializations of W, the loss functions match up to machine precision with f Lreal 4.02 10-9 9 + 7.01 10-9
33

---

Under review as a conference paper at ICLR 2023
(3) Reduced model trains faster. Due to the relation between projected gradient descent of the full network W and gradient descent of the reduced network W our method may be applied before training to produce a smaller model class which trains faster without sacrificing accuracy. We test this hypothesis in learning the function f : I22 + R² sending = (f1,ty to (e (e- "e- e. using N 1212 samples (-3
+ J/20, -3 + k/20) for S j, < 121. We model fW as radial neural network with layer widths n = (2, 16, 64, 128, 16, 2) and activation the radial sigmoid h(r)) = 1/(1 + e e- Applying QR-compress gives radial neural network / with widths red =
We trained both models until the training loss was Σ 0.01. Running on system with an Intel i5-8257U@) I. 1.40GHz and 8GB of RAM and averaged over 10 random initializations, the reduced network trained in 15.32 2.53 seconds and the original network trained in 31.21..4 + 4.55 seconds,
(4) Comparison with ReLU MLP on noisy image recovery. We show that Step-ReLU radial network performs better than an otherwise comparable network with pointwise ReLU on a noisy im- age recovery task. Using samples of MNIST with significant added noise the network classification task to identify from which original sample the noisy sample derives.
Specifically, choose samples from MNIST, all with the same MNIST label, and produce noisy samples from each by adding noise. The noise added by considering each sample as a point in 18784, and adding uniform random noise in ball around each. The radius of the ball around a given point is the product of the noise level variable (noise_scarl _ccale,, which is the same for all points) and the minimal distance to another sample point (which varies from point to point). As indicated in Figure 5 when prossesscales the classification task is difficult for the human eye.
3 original MNIST images with the same label, and produces images for each, with noiss noise_scaal se_scalee3.. We perform 240 train / 60 test split of the 300 data points. Both models have three layers with widths (d, + d + 2, n = 3), where d 28 784; hence, both models have 620, 158 trainable parameters
Our data takes
100 noisy
Over 10 trials, each training for 150 epochs and learning rate 0.05 for both models, the radial network achieves training loss 0,00256 -3.074 10 4 with accuracy + O, while the ReLU MLP has training On the test set, the radial network has loss 0.00266 +3.7799 10-1 with accuracy + O, while the ReL.U MLP has loss 0.305 +2.588 10-1 with accuracy 0.757 +-2.664. 464-10-1 The convergence rates are illustrated in Figure with the radial network outperforming the ReLU MLP. We note that 150 epochs is sufficient for all methods to converge, although the ReLU MLP does not always converge to zero loss.
loss 0.295 + 2.259. 10 with accuracy 0.768 +.9.999.
We observe that the radial network 1) is able to obtain a better fit, 2) has faster convergence, and 3) generalizes better than the pointwise ReLU. We hypothesize the radial nature of the random noise makes radials networks well eellted to the task.
F RELATION TO RADIAL BASIS FUNCTION NETWORKS
In this appendix, we show that radial neural networks are equivalent to a particular class of mul- tilayer radial basis functions networks. This class is obtained by imposing the condition that the so-called 'hidden dimension' each layer is equal to one; the total number of layers, however, unconstrained. To our knowledge, the literature contains no universal approximation result for this class of radial basis functions networks.
F.1 SINGLE LAYER CASE
We first recall the definition of radial basis function network. A local linear model extension of radial basis function network (henceforth abbreviated simply by RBFN) consists of:
An input dimension n, an output dimension m, and a 'hidden' dimension N.
For i = 1,...,,,, N, a matrix W. E Rmx", a vector b, E R", and a weight ag E R
A nonlinear function F A : + R.
AA general version allows for different nonlinear function for every 1,...,
34

---

Under review as a conference paper at ICLR 2023
The feedforward function of RBFN is defined as:
F R
(a: + Wi( b.)) A( + bzl).
The integer N is commonly referred as 'the hidden number of neurons'. This is bit of a mis- nomer. Really there is only one layer with input dimension n and output dimension m; the integer N is part of the specification of the activation function.
We observe that if N = 1 and a1 = O, then the feedforward function is given by:
F R
R
+ W p(x b)
where P the radial rescaling function determined by A. In words, one adds 61 b E R" to the input vector I, applies the activation p to obtain new vector in R", and then applies the linear transformation determined by the matrix W1 W to obtain the output vector R" Motivated by this observation, we say that RBFN is constrained if N = 1 and a1 = 0.
F.2 CONSTRAINED MULTILAYER CASE
Next, consider the constrained multilayer case of radial basis functions network. Specifically, a constrained multilayer RBFN consists of:
A widths vector (no...., n1) where L is the number of layers.
A matrix W, E R for = I....... L.
A vector by RR for E 0, 0.......5 --1. 1.
A nonlinear function A + R for & 0, ........L radial rescaling function R te for l. = 0, .......... L 1.)
(Equivalently, the corresponding
The feedforward function is defined as follows. For e = 0...., I. we recursively define Fj R > Th by setting (z) = x and
Pa(x) =
+ bz-1)
for l = 1, I,...,,.. L. The feedforward function is FL.
F.3 RELATION TO RADIAL NEURAL NETWORKS
We now demonstrate that radial neural networks equivalent constrained multilayer RBFNs.
Proposition 27. For any radial neural network, there constrained multilayer RBFN with the same feedforward function. Conversely, for any constrained multiplayer RBFN, there is radial neural network with the same feedforward function.
Proof. For the first statement, (W, b, p) be radial neural network with L layers and widths vector (no..... ,.)). Recall the partial feedforward functions G. I - Be defined recursively by setting Go(z) x and
Gz(z) = pe (W,G,-1(z) + b,)
The feedforward function is Gr. Consider the constrained multilayer RBFN with L + layers and the following:
Widths vector (no, 10,,,., nt.).
The last layers have the same dimension.
Weight matrices We E [Rnyxaz=- for 8 1,..., and W1+1= = idny R
A vector by "h for & = 1,..., L, and by 0 Ro
A radial rescaling activation P: R R for l = 1,... L, and Po idng-
35

---

Under review as a conference paper at ICLR 2023
Let Fe be the partial feedforward functions for this RBFN, defined recursively above. We claim that
Fe(x) We Gz)
for any E Ruo and 8 1,.... L. We prove this by induction. The base case is l = 1:
(x( = W1 po (Fo(z) + bu) Wix x = W1 o Go(x)
For the induction step, take & >> and compute:
(z( We PE-1 (F) be-1) We o Pt-1 (We-iGe-200 bE-1) Weo
The first claim follows from the case L, using the fact that WL+1 is the identity.
For the second statement, let (W, b, p) be a constrained multilayer RBFN with L layers and widths vector (no. ..... n...- Consider the radial neural network with T. + layers and the following:
Widths vector (No: Roy Rl,.... (1-11; The first two layers have the same dimension.
Weight matrices given by Wi idno and We We-1 for 2, ......
Bias vectors given by be b for = 1,2, ......... L, and b = 0.
Radial rescaling activations given by P for l I....,,a and PL+1 idni.
One uses the recursive definition the partial feedforward functions show that, for I....... L, we have Fr(x)) Wf Ge(x), where Pt and G3 are the partial feedforward functions of the RBFN and radial neural network, respectively. Then:
GL+1(F) = PL+I (Wi-i o Gr.(x) bb+1) W.. a (x) Fz(x),
so the two feedforward functions coincide.
F.4 CONCLUSIONS
While radial neural networks equivalent to certain class of radial basis function network, point out differences between our results and the standard theory radial basis functions network. First, RBFNs generally only have layers; consider ones with unbounded depth. Second, to our knowledge, ours the first universal approximation result such that:
it uses networks in the subclass multilayer RBFNs satisfying the constraint that all the number of 'hidden neurons' in each layer is equal to 1.
it approximates functions with networks of bounded width.
it can be used to approximate asymptotically affine functions, rather than functions defined compact domain.
Our compressibility result may apply to multilayer RBFNs where the number of 'hidden neurons' N₂ at each layer is not equal to 1, but we expect the compression to weaker, and that constrained mulitlayer RBFNs are in some sense the most compressible type of RBFN.
36


Reviews:
[{"review_id": "ZcaZwnbuj6c", "reviewer": "Reviewer_Vbpq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes *radial neural networks*, neural networks with a non-pointwise nonlinearity that is a scalar rescaling of the vector of activations as a function of its norm. The paper argues that such neural networks are rotation equivariant and proves that they are universal function approximators. It further introduces a lossless compression technique for these networks and supports its claims empirically.", "summary_of_the_review": "I think this is interesting research in general, however the core contribution and future directions are not clear from the manuscript to me, so I will recommend rejection for now", "strength_and_weaknesses": "Strengths:\n* Nonlinearities are a fundamental building block of neural network architectures, so new techniques are potentially broadly applicable.\n* The writing of the paper is clear and focused within the respective sections.\n* The range of contributions is quite broad, ranging from the theoretical results on the proposed networks being universal function approximators to a practical compression method taking advantage of their structure.\n\nWeaknesses:\n* Probably as a result of its breadth, the paper somewhat lacks a clear red thread or headline result. I am not really sure who the target audience of the paper is. Is the rotation equivariance a means to the end of constructing neural networks that are easy to compress, i.e. is this a practically minded paper? Do the authors want to establish radial neural networks as a new direction of research? Then what are promising directions for future work and what specific limitations of current architectures are they meant to overcome?\n* There are essentially no quantitative results. I really don't mind if the methods can't be immediately plugged into ResNets or Transformers and give state-of-the-art results, but the paper should at least give a general idea of how well the method works on standard tasks (MNIST, CIFAR, ...). What's the rationale behind choosing the benchmark on the noised MNIST images? Is it expected that radial neural networks perform better on this task?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YfWbNiIDL2", "reviewer": "Reviewer_MQ2f", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new neural network architecture which is radial: instead of pointwise activation, the activation is applied to the whole layer which rescales the output vector according to its norm. Universal approximation results are shown, with additional model compression algorithms.", "summary_of_the_review": "The results are solid and novel but not as significant: the new radial architecture doesn't show improvement over previous bounds. As a result I lean to weak accept rather than accept.", "strength_and_weaknesses": "Strength:\n\n1, the proposed radial neural network is novel. The way it generalizes pointwise activation is natural and elegant, that a simple rescaling of vector is actually expressive.\n\n2, the universal approximation results are solid. The proof method is intuitive and resembles classic ideas in literature.\n\nWeaknesses:\n\n1, I don't see any improvement over previous bounds for universal approximation: for the usual Lipschitz function case, the construction of Theorem 3 requires $O((\\frac{\\sqrt{n}}{\\epsilon})^{3n})$ number of parameters (ignoring poly factors). Theorem 5 improves it to the standard $O((\\frac{\\sqrt{n}}{\\epsilon})^{n})$.\n\n2, the compression algorithm is useful only if there are subsequent layers with notably larger width. However, such neural networks are already sub-optimal theoretically and wouldn't be used in the first place, then there is no need for compression.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "A9JQgbLvD3", "reviewer": "Reviewer_vdPZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Instead of considering entry wise nonlinearities in neural networks, the paper studies radial neural networks, where the activation function acts to rescale the output of a whole layer. Asymptotic universal approximation ability of radial neural networks is proved. In addition, it is shown that radial neural networks can be compressed with an efficient algorithm based on QR decompositions. Experiments on synthetic data and MNIST are provided to support the theory.", "summary_of_the_review": "From a pure theoretical point of view, the paper can be a good study on radial neural networks, if a clearer exposure of technical contributions is provided. Further, I believe the paper would attract more attention if the practicality of radial neural networks is discussed and advantages over pointwise neural networks are demonstrated. The best result would be using the developed theory to explain the advantage of radial neural networks.\n\n=============== Post author response ===============\n\nThank you for the detailed response. The explanation on approximation in $\\mathbb{R}^d$ makes sense to me. Examples of $\\sin x$ are difficult to approximate in $\\mathbb{R}^d$ since they do not vanish at infinity. The ultimate idea is always to truncate the domain.\n\nWith the experiments in Section 7 and Appendix E, the practical motivation of radial neural networks might still be limited. As pointed out, radial neural networks can be advantageous when spherical decision boundaries appear in tasks.\n\nOverall, I think the paper makes interesting contributions to universal approximation theories of neural networks. Nonetheless, practical implications are on the weak side.\n\n", "strength_and_weaknesses": "=============== Strength ===============\n\nThe paper manages to contain a rich amount of related references and provide graphical illustration that makes the ideas easy to understand.\n\nFrom a technical point of view, the asymptotic universal approximation property of radial neural networks is new to me. Besides, the compressive nature of radial neural networks and its resulting fast optimization are also not studied in literature.\n\nThe organization of the paper is good and easy to follow.\n\n=============== Weakness ===============\n\nA strong motivation of the practicality of radial neural networks seems to be lacking. It will be helpful to provide working examples of radial neural networks and indicate that in some important applications, radial neural networks outperform pointwise networks.\n\nThe universal approximation theory allows the network size to grow to infinity. It seems unfair to claim that not requiring bounded domain is a contribution. In early asymptotic results, e.g., Cybenko and Barron, neural networks are already shown to be dense in differential function spaces with unbounded domain. Later results established the rate of approximation requires a compact domain. As there is no rate of approximation provided, considering unbounded domain does not add much technical challenges. Moreover, the asymptotic results does not provide understanding of potential advantages of radial neural networks over pointwise neural networks in terms of function approximation.\n\nSynthetic data is relatively toy and the comparison with ReLU network is not fully convincing. On the one hand, synthetic data are low-dimensional and the target function is simple in that the activation function is almost the form of the target function. On the other hand, Step-ReLU radial network outperforms ReLU MLP might correlate to multiple reasons: 1) different activation functions; 2) different network architectures; 3) hyper parameter tuning. By the way, is it possible to simply perform classification on MNIST using radial and pointwise neural networks?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject