Year: 2021

Abstract:
We provide a theoretical framework for neural networks in terms of the representation theory of quivers, thus revealing symmetries of the parameter space of neural networks. An exploitation of these symmetries leads to a model compression algorithm for radial neural networks based on an analogue of the QR decomposition. The algorithm is lossless; the compressed model has the same feedforward function as the original model. If applied before training, optimization of the compressed model by gradient descent is equivalent to a projected version of gradient descent on the original model.

heading: 1 INTRODUCTION
text: Recent work has shown that representation theory, the formal study of symmetry, provides the foundation for various innovative techniques in deep learning (Cohen & Welling, 2016; Kondor & Trivedi, 2018; Ravanbakhsh et al., 2017; Cohen & Welling, 2017). Much of this previous work considers symmetries inherent to the input and output spaces, as well as distributions and functions that respect these symmetries. By contrast, in this paper, we expose a broad class of symmetries intrinsic to the parameter space of the neural networks themselves. We use these parameter space symmetries to devise a model compression algorithm that reduces the widths of the hidden layers, and hence the number of parameters. Unlike representation-theoretic techniques in the setting of equivariant neural networks, our methods are applicable to deep learning models with non-symmetric domains and non-equivariant functions, and hence pertain to some degree to all neural networks.
Specifically, we formulate a theoretical framework for neural networks inspired by quiver representation theory, a mathematical field with connections to symplectic geometry and Lie theory (Kirillov Jr, 2016; Nakajima et al., 1998). This approach builds on that of Armenta & Jodoin (2020) and of Wood & Shawe-Taylor (1996), but is arguably simpler and encapsulates larger symmetry groups. Formally, a quiver is another name for a finite directed graph, and a representation of a quiver is the assignment of a vector space to each vertex and a compatible linear map to each edge.
Our starting point is to regard the vector space of parameters for a neural network as a representation of a particular quiver, namely, the neural quiver (Figure 1). The advantage of this viewpoint is that representations of quivers carry rich symmetry groups via change-of-basis transformations; such operations can be viewed as symmetries of the neural network parameter space.
Moreover, these symmetries may be factored out without affecting the feedforward function, making our method a lossless model compression algorithm (Serra et al., 2020). Model compression has
become critically important as models have grown to billions of parameters; with compression, enormous models may be reduced and run on smaller systems with faster inference (Buciluǎ et al., 2006; Cheng et al., 2017; Frankle & Carbin, 2018; Zhang et al., 2018). Whereas many previous approaches to model compression are based on weight pruning, quantization, matrix factorization, or knowledge distillation, similar to Sourek et al. (2020), our approach exploits symmetries of neural network parameter spaces.
The size of the parameter space symmetry group is determined by properties of the activation functions. We focus on radial activation functions, as in Weiler & Cesa (2019); Sabour et al. (2017); Weiler et al. (2018a;b); these interact favorably with certain QR decompositions, and, consequently, the model compression is significant compared to the more common pointwise (or ‘local’) activations. We refer to neural networks with radial activation functions as radial neural networks.
Given a radial neural network, our results produce a new network with fewer neurons in each hidden layer and the same feedforward function, and hence the same loss for any batch of training data. Moreover, the value of the loss function after a step of gradient descent applied to the compressed model is the same as the value of the loss function after a step of projected gradient descent applied to the original model. As we explain below, in projected gradient descent, one subtracts a truncation of the gradient, rather than the full gradient. When the compression is significant enough, the compressed model takes less time per epoch to train and reaches local minima faster.
To state these results slightly more precisely, recall that the parameters of a neural network with layer widths (n0, n1, . . . , nL) consist of an ni×(ni−1+1) matrixWi of weights for each layer i, where we include the bias as an extra column. These are grouped into a tuple W = ( Wi ∈ Rni×(ni−1+1) ) i .
We define the reduced widths recursively as nredi = min ( ni , n red i−1 + 1 ) for i = 1, . . . , L− 1, with nred0 = n0 and n red L = nL. Note that n red i ≤ ni for all i. Theorem 1.1 (Informal version of Theorems 4.3 and 4.7). Suppose a neural network has L layers with widths (n0, . . . , nL), parameters W, and radial activation functions. Let fW : Rn0 → RnL be the feedforward function of the network.
1. There exists an algorithm to produce a reduced radial neural network with layer widths (n0, n red 1 , . . . , n red L−1, nL), parameters R, and the same feedforward function fR = fW.
2. Training fR with gradient descent is an equivalent optimization problem to training fW with projected gradient descent.
This theorem can be interpreted as a model compression result: the reduced (or compressed) neural network R has the same accuracy as the original neural network W, and there is an explicit relationship between the gradient descent optimization problems for the two neural networks. This result is not just theoretical; it emerges from a practical and efficient algorithm depending on successive QR decompositions. We describe this procedure below (Algorithm 1) and implement it in Python.
To summarize, our contributions are as follows:
1. A theoretical framework for neural networks based on quiver representation theory; 2. A QR decomposition for radial neural networks; 3. An implementation of a lossless model compression algorithm for radial neural networks; 4. A theorem relating gradient descent optimization of the original and compressed networks.
We view this work as a step in the direction of improving learning algorithms by exploiting symmetry inherent to neural network parameter spaces. As such, we expect our framework and results to generalize in several ways, including: (1) further reductions of the hidden widths, (2) incorporating certain non-radial activation functions, (3) encapsulating networks beyond MLPs, such as convolutional, recurrent, and graph neural networks, (4) integration of regularization techniques. We include a detailed discussion of the limitations of our results, as well as future directions, in Section 6.

heading: 2 RELATED WORK
text: Quiver Representation Theory and Neural Networks. Armenta & Jodoin (2020) give an approach to understanding neural networks in terms of quiver representations. Our work generalizes
their approach as it (1) accommodates both pointwise and non-pointwise activation functions, (2) taps into larger symmetry groups, and (3) connects more naturally to gradient descent. Jeffreys & Lau (2021) and Manin & Marcolli (2020) also place quivers in the context of neural networks; our approach is inspired by similar algebro-geometric and categorical perspectives, but differs in our emphasis on practical consequences for optimization techniques at the core of machine learning. These works have a number of precursors. One is the study of the “non-negative homogeneity” (also known as “positive scaling invariance”) property of ReLU activation functions (Dinh et al., 2017; Neyshabur et al., 2015; Meng et al., 2019), which is a special case of the symmetry studied in this paper. Wood & Shawe-Taylor (1996) regard layers in a neural network as representations of finite groups and consider only pointwise activation functions; by contrast, our framework captures Lie groups as well as non-pointwise activations. Our quiver approach to neural networks shares parallels with the “algebraic neural networks” of Parada-Mayorga & Ribeiro, and special cases of their formalism amount to representations of quivers over base rings beyond R, such as the ring of polynomials. In a somewhat different data-analytic context, Chindris & Kline (2021) use quiver representation theory in order to untangle point clouds; though they do not use neural networks.
Equivariant Neural Networks. Previously, representation theory has been used to design neural networks which incorporate symmetry as an inductive bias. A variety of architectures such as G-convolution, steerable CNN, and Clebsch-Gordan networks are constrained by various weightsharing schemes to be equivariant or invariant to various symmetry groups (Cohen et al., 2019; Weiler & Cesa, 2019; Cohen & Welling, 2016; Chidester et al., 2018; Kondor & Trivedi, 2018; Bao & Song, 2019; Worrall et al., 2017; Cohen & Welling, 2017; Weiler et al., 2018b; Dieleman et al., 2016; Lang & Weiler, 2021; Ravanbakhsh et al., 2017). Our approach, in contrast, does not rely on symmetry of the input domain, output space, or mapping. Rather, our method exploits symmetry of the parameter space and thus applies more generally to domains with no obvious symmetry. From the point of view of model compression, equivariant networks do achieve reduction in the number of trainable parameters through weight-sharing for fixed hidden dimension widths; however, in practice, they may use larger layer widths and consequently have larger memory requirements than non-equivariant models. Sampling or summing over large symmetry groups may make equivariant models computationally slow as well (Finzi et al., 2020; Kondor & Trivedi, 2018).
Model Compression and Weight Pruning. A major goal in machine learning is to find methods for compressing models in order to reduce the number of trainable parameters, decrease memory usage, or accelerate inference and training (Cheng et al., 2017). Our approach toward this goal differs significantly from most existing methods in that it is based on the inherent symmetry of neural network parameter spaces. One prior model compression method is weight pruning, which removes redundant, small, or unnecessary weights from a network with little loss in accuracy (Han et al., 2015; Blalock et al., 2020; Karnin, 1990). Recent work has shown pruning can be done during training by identifying and removing weights of less relevance and reverting other weights to earlier values (Frankle & Carbin, 2018). Related work shows effective pruning may be done at initialization (Lee et al., 2019; Wang et al., 2020). Gradient-based pruning identifies low saliency weights by estimating the increase in loss resulting from their removal (LeCun et al., 1990; Hassibi & Stork, 1993; Dong et al., 2017; Molchanov et al., 2016). A complementary approach is quantization, in which the bit depth of weights is decreased (Wu et al., 2016; Howard et al., 2017; Gong et al., 2014). Knowledge distillation works by training a small model to mimic the performance of a larger model or ensemble of models (Buciluǎ et al., 2006; Hinton et al., 2015; Ba & Caruana, 2013). Matrix Factorization methods replace fully connected layers with lower rank or sparse factored tensors (Cheng et al., 2015a;b; Tai et al., 2015; Lebedev et al., 2014; Rigamonti et al., 2013; Lu et al., 2017) and can often be applied before training. Our method involves a generalized QR decomposition, which is a type of matrix factorization; however, rather than aim for a rank reduction of linear layers, we leverage this decomposition in order to reduce hidden layer widths via change-of-basis operations on the hidden representations. Closest to our method are lossless compression methods. Serra et al. (2021; 2020) identify and remove stable neurons in ReLU networks. Sourek et al. (2020) also exploit symmetry in parameter space to remove redundant neuron. However, their symmetries are induced from permutation equivariance; ours follow from the symmetries of the radial activation.

heading: 3 NEURAL NETWORKS AS QUIVER REPRESENTATIONS
text: We turn our attention to quivers and their representations. Neural networks may be viewed as representations of a specific quiver with the additional data of activation functions; backpropogation can also be formulated in these terms. The advantage of such perspectives is that representations come equipped with symmetry groups. We provide proofs and context for this section in Appendix D.

heading: 3.1 QUIVER REPRESENTATIONS
text: A quiver is a finite directed graph. A dimension vector for a quiver Q consists of a non-negative integer for every vertex, that is, a tuple d = (di)i indexed by the vertices ofQ. A representation of Q with dimension vector d consists of a matrix for each edge, where the matrix Ae corresponding to an edge
s• e−→ t• must be of size dt × ds. In other words, a representation is a tuple of matrices (Ae)e indexed by the edges of Q, with sizes constrained by the dimension vector. Note that each Ae defines a linear map Rds → Rdt . The space of representations Rep(Q,d) is defined as the set of all possible representations of a quiver Q with dimension vector d. This space can be expressed as the product of matrix spaces: Rep(Q,d) = ∏ e∈E Rdt(e)×ds(e) . Here E is the edge set of Q, and s(e) and t(e) denote the source and target vertices of an edge e.

heading: 3.2 THE NEURAL QUIVER
text: For the remainder of the paper, we focus on a particular quiver. Let L be a positive integer. The neural quiverQL is the quiver with L+2 vertices shown in Figure 1. The vertices in the top row are indexed from i = 0 to i = L. The first vertex (i = 0) in the top row is called the input vertex, the last vertex (i = L) is the output vertex, and every other vertex (1 ≤ i ≤ L− 1) in the top row is a hidden vertex. The vertex at the bottom is called the bias vertex and is connected to each vertex in the top row except for i = 0. When discussing representations of the neural quiver, we exclusively consider dimension vectors whose value at the bias vertex is 1. Hence, a dimension vector for the neural quiver will refer to an (L+ 1)-tuple of positive integers n = (n0, . . . , nL).
With these conventions in hand, we observe that, by definition, a representation of the neural quiver with dimension vector n consists of an ni × ni−1 matrix Ai and a vector bi ∈ Rni , for each i = 1, . . . , L. Correspondingly, there is an affine map W̃i : Rni−1 → Rni given by x 7→ Aix + bi, and an ni × (1 + ni−1) matrix given by Wi = [bi Ai]. As a result, we have: Lemma 3.1. A representation of QL with dimension vector n is equivalent to the data of an affine map Rni−1 → Rni for each i = 1, . . . , L. There is an isomorphism of vector spaces:
Rep(QL,n) ' Rn1×(1+n0) × Rn2×(1+n1) × · · · × RnL×(1+nL−1).
Notation: We henceforth denote a representation of QL as a tuple W = (Wi)Li=1, where each Wi belongs to Rni×(1+ni−1), and denote the corresponding affine maps as W̃i : Rni−1 → Rni .

heading: 3.3 CHANGE-OF-BASIS SYMMETRY
text: Recall that the general linear group GLn(R) consists of the set of invertible n by n matrices. The unit is the identity n by n matrix, denoted idn. There is a linear action of the general linear group GLn(R) on the vector space Rn given by matrix multiplication: ~x 7→ G~x for G ∈ GLn(R) and ~x ∈ Rn. This action encodes change-of-basis transformations: the entries of the vector G~x are the coefficients when expressing the original vector ~x in the basis specified by the columns of G−1.
Given a dimension vector n for the neural quiver, the parameter symmetry group is defined as:
GLhiddenn := GLn1(R)×GLn2(R)× · · · ×GLnL−1(R)
An element of this group consists of the choice of a transformation of Rni for each hidden vertex i, and results in a corresponding transformation of the matrices Wi appearing in any representation W of the neural quiver with dimension vector n. To be explicit, a particular choice G = (Gi)L−1i=1
of transformations results in the following linear transformation of representations1:
W = (Wi) L i=1 7→ G ·W := ( Gi Wi [ 1 0 0 G−1i−1 ] )L i=1 , (3.1)
whereG0 = idn0 andGL = idnL . In other words: there is a linear action of the parameter symmetry group GLhiddenn on the vector space Rep(QL,n) of representations of the neural quiver.

heading: 3.4 NEURAL NETWORKS
text: For simplicity, in the body of this paper, we focus on the most basic neural network architecture, namely that of multi-layer perceptrons (MLPs). These the neural networks whose underlying architecture is specified by the neural quiver. We include a discussion of more complex quivers (such as that in Figure 1b) in the appendix. We begin with the following somewhat abstract definition: Definition 3.2. An QL-neural network consists of the following:
1. Hyperparameters. A positive integer L and a dimension vector n = (n0, n1, n2, . . . , nL) for the neural quiver QL.
2. Trainable parameters. A representation W = (W1, . . . ,WL) of the neural QL with dimension vector n. So each Wi is a matrix in Rni×(1+ni−1).
3. Activation functions. Piece-wise differentiable functions ai : Rni → Rni for i = 1, . . . , L. These are grouped into a tuple a = (a1, . . . , aL).
For the remainder of this paper, unless specified otherwise, “neural network” will refer to “QLneural network”. We emphasize, however, that there is a general notion of a Q-neural network for any acyclic quiver Q with unique sink (see Appendix D). Additionally, as indicated above, in the case of the neural quiver, a QL-neural network is equivalent to an MLP. We denote by Neur(QL,n) the vector space of neural networks with L layers and dimension vector n, and regard elements therein as pairs (W,a). The neural function (or feedforward function) of a QL-neural network (W,a) is defined as:
f(W,a) : Rn0 → RnL ~x 7→ aL ◦ W̃L ◦ · · · ◦ a2 ◦ W̃2 ◦ a1 ◦ W̃1(~x)
where W̃i : Rni−1 → Rni is the affine map corresponding to Wi (see Section 3.2). The optimization problem for neural networks can be formulated in terms of representations of the neural quiver. Fix a dimension vector n for QL, and activation functions a. To a batch of training data {(~xj , ~yj)} ⊆ Rn0 × RnL , the associated loss function on the space Rep(QL,n) is defined as
L : Rep(QL,n)→ R L(W) = ∑ j C ( f(W,a) (~xj) , ~yj ) where C : RnL × RnL → R is a cost function. The map
γ : Rep(QL,n)→ Rep(QL,n) γ(W) = W − η∇WL
performs one step of gradient descent with learning rate η. Hence, the backpropogation algorithm can be regarded as taking place on the space Rep(QL,n) of representations of the neural quiver.

heading: 4 MAIN THEORETICAL RESULTS
text: In this section, we first define radial neural networks. Then, adopting the framework developed in the previous section, we proceed to state a QR decomposition for such networks, present an algorithm to compute the decomposition, and relate the decomposition to gradient descent.
1For each i, the matrix [ 1 0 0 G−1i−1 ] is block diagonal with one 1 × 1 block consisting of the single entry
‘1’, and one ni−1 × ni−1 block consisting of the inverse of the matrix Gi−1 ∈ GLni−1(R).

heading: 4.1 RADIAL NEURAL NETWORKS
text: We begin by stating the definition of radial functions. A piece-wise differentiable function a : Rn → Rn is called radial if a(v) = h(|v|) v|v| for some h : R → R. Hence, a sends each input vector v ∈ Rn to a scalar multiple of itself, and that scalar depends only on the norm of the vector.
Example 4.1. (1) The squashing function, where h(r) = r 2
r2+1 . (2) Shifted ReLU, where h(r) = ReLU(r − b) for r > 0 a real number b. See Weiler & Cesa (2019) and the references therein for more examples and discussion of radial functions. See Figure 2
Let n be a dimension vector for the neural quiver QL. A neural network (W,a) ∈ Neur(QL,n) is a radial neural network if each ai is a radial function on Rni . Hence, there are functions hi : R → R such that ai(v) = hi(|v|) v|v| for each i. Let O(n) denote the following product of the orthogonal groups:
O(n) = O(n1)×O(n2)× · · · ×O(nL−1).
Note that O(n) is a subgroup of the parameter symmetry group GLhiddenn and hence acts on Rep(Q,n). Radial functions have the flexibility of defining the activation functions of a neural network with arbitrary dimension vector. To be more explicit:
Construction 4.2. Suppose h = (h1, . . . , hL) is a tuple of functions2 R → R, and suppose n is a dimension vector for QL. Consider the tuple of activation functions: a(n)h = ( h (n1) 1 , . . . , h (nL) L ) .
Any representation W of QL with dimension vector n defines a neural network ( W,a (n) h ) as in
Definition 3.2. We write simply (W,h) for this network and f(W,h) for its neural function.

heading: 4.2 THE QR DECOMPOSITION
text: We now formulate the QR decomposition for representations of the neural quiver, which interacts favorably with radial neural networks. Proofs and further context appear in Appendix E.
The first step is to associate a ‘reduced’ dimension vector to each dimension vector of the neural quiver. Let n = (n0, n1, . . . , nL) be a dimension vector for QL. Its reduction nred = (nred0 , n red 1 , . . . , n red L ) is defined recursively 3 by setting nred0 = n0, then n red i = min(ni , n red i−1 +1) for i = 1, . . . , L − 1, and finally nredL = nL. We abbreviate Rep(QL,n) by Rep(n) and Rep(QL,nred) by Rep(nred). Observe that nredi ≤ ni for each i. Therefore, taking a representation with dimension vector nred, i.e., a certain tuple of matrices, one can pad each matrix with the necessary number of rows of zeros below and columns of zeros on the right in order to produce a representation with dimension vector n. Thus we have an inclusion: ι : Rep(nred) ↪→ Rep(n). Fix a tuple h = (h1, . . . , hL) of functions hi : R → R. As in Construction 4.2, we attach a radial neural network (and hence a neural function) to every representation of QL. Theorem 4.3. Let n be a dimension vector and fix h as above. For any W ∈ Rep(n) there exist:
Q ∈ O(n), R = (R1, . . . , RL) ∈ Rep(nred), and U ∈ Rep(n),
such that:
1. The matrices R1, . . . , RL−1 are upper triangular.
2. The following equality holds: W = Q ·R+U 3. The networks defined by W and R have identical neural functions: f(W,h) = f(R,h).
2We assume each hi is piece-wise differentiable and the limit limr→0 hi(r)r exists. 3For neural networks without bias vectors, one defines the reduced dimension vector nred using instead the
recursion nredi = min(ni, n red i−1). See Appendix H.1.
In the the second point above, we identify R with its image in Rep(n) under ι, and invoke the action of O(n) on Rep(n). We say that R is the reduced representation corresponding to W. The proof of Theorem 4.3 (appearing in Appendix E.2) relies on Algorithm 1 – a constructive algorithm proceeding layer by layer and computing a QR decomposition at each stage.
Remark 4.4. Part (2) of Theorem 4.3 is a quiver representation analogue of the usual QR decomposition (c.f. Section B.2); indeed, each matrix (except possibly the last) in R is upper triangular and Q is a tuple of orthogonal matrices.
Algorithm 1: QR Dimensional Reduction (QRDimRed) input : W ∈ Rep(n) output : Q,R,U from Theorem 4.3 Q,R,U← [ ], [ ], [0] // initialize output matrix lists V1 ←W1 // next layer transformed weights for i← 1 to L− 1 do // iterate through layers
if nredi < ni then Qi, Ri ← QR-decomp(Vi, mode = complete) // Vi = Qi ◦ inci ◦Ri else Qi, Ri ← QR-decomp(Vi) // Vi = Qi ◦Ri end Append Qi to Q Append Ri to R // reduced weights for layer i
Append Wi+1 ◦ [ 1 0 0 Qi ] ◦ pi ◦ [ 1 0 0 Q−1i ] to U // matrix multiplication
Set Vi+1 ←Wi+1 ◦ [ 1 0 0 Qi ] ◦ ĩnci // transform next layer
end Append VL to R
return Q, R, and U
Notation: The symbol ‘◦’ denotes matrix multiplication. We initialize U as a list with a single element: the zero n1 × (1 + n0) matrix. For i = 1, . . . , L − 1, we have the standard inclusions inci = incnredi ,ni : R nredi ↪→ Rni and ĩnci = inc1+nredi ,1+ni : R 1+nredi ↪→ R1+ni into the first nredi and 1 + nredi coordinates, respectively. As matrices, they have ones along the main diagonal and zeros elsewhere. The map pi : Rni+1 → Rni+1 zeros the first 1 + nredi coordinates and leaves the remaining coordinates unchanged. As a matrix, pi is block diagonal with two square blocks; the top block is the zero matrix of size nredi + 1, and the bottom is the identity matrix of size ni − nredi .
By definition, either nredi = n red i−1 + 1 or n red i = ni. In the former case, n red i−1 + 1 ≤ ni and QR-decomp with ‘mode = complete’ computes the QR decomposition of the ni× (1+nredi−1) matrix Vi as Qi ◦ inci ◦Ri where Qi ∈ O(ni) and Ri is upper-triangular of size nredi × nredi . In the latter case, nredi−1+1 ≥ ni and QR-decomp computes the QR decomposition of the ni× (1+nredi−1) matrix Vi as Qi ◦Ri where Qi ∈ O(ni) and Ri is upper-triangular of size ni × (1 + nredi−1).
Example 4.5. Suppose the dimension vector is n = (1, 8, 16, 8, 1), so the original radial neural network model has ∑4 i=1(ni−1 + 1)ni = 305 trainable parameters. The reduced dimension vector
is nred = (1, 2, 3, 4, 1). One computes that the compressed model has ∑4 i=1(n red i−1 + 1)(n red i ) = 34 trainable parameters. Another example appears in Figure 3.

heading: 4.3 PROJECTED GRADIENT DESCENT
text: In this section, we use the QR decomposition of Theorem 4.3 to relate projected gradient descent for a representation to (usual) gradient descent for the corresponding reduced representation. Proofs and further context appear in Appendix F.
We adopt the notation of the previous sections. In particular, we fix a tuple h = (h1, . . . , hL); as in Construction 4.2, this can be used to define a neural network for any representation ofQL, with any dimension vector. Furthermore, we fix a batch of training data in Rn0 × RnL . Using h, we obtain loss functions L : Rep(n) → R and Lred : Rep(nred) → R (see Section 3.4). For a fixed learning rate, there are resulting one-step gradient descent maps given by:
γ : Rep(n)→ Rep(n) γred : Rep(nred)→ Rep(nred) W 7→W − η · ∇WL X 7→ X− η · ∇XLred
Definition 4.6. Define Pr : Rep(n) → Rep(n) to be the map that, given a representation T ∈ Rep(n), zeros the bottom left (ni − nredi )× (1 + nredi−1) submatrix of Ti, for each i. Schematically,
Pr : T = ([ ∗ ∗ ∗ ∗ ])L i=1 7−→ Pr(T) = ([ ∗ ∗ 0 ∗ ])L i=1 .
The one-step projected gradient descent map on Rep(n) at learning rate η > 0 is defined as:
γproj : Rep(n)→ Rep(n) W 7→W − η · Pr(∇WL)
Hence, while all entries of each matrix Wi in the representation W contribute to the computation of the gradient ∇WL, only those not in the bottom left submatrix get updated under the projected gradient descent map γproj. To ease notation in the statement of the following result, we identify elements of Rep(nred) with their images in Rep(n) under ι.
Theorem 4.7. Let W ∈ Rep(n) and let Q, R, and U be the outputs of Algorithm 1. Fix a learning rate η > 0. For any k ≥ 0, we have:
γk(W) = Q · γk(Q−1 ·W) and γkproj(Q−1 ·W) = γkred(R) +Q−1 ·U.
While the proof of the second statement of Theorem 4.7 is quite technical, the proof of the first relies on a basic interaction of orthogonal transformations with gradient descent, which we briefly illustrate (Appendices C and F contain more details). Let L : Rp → R be a function, and let γ : Rp → Rp be the gradient descent map with respect to L. Suppose Q ∈ O(p) is an orthogonal transformation of Rp that preserves the level sets of L, that is, L(Q(v)) = L(v) for all v ∈ Rp. Then Q commutes with gradient descent, i.e. γ(Q(v)) = Q(γ(v)) for all v ∈ Rp. See Figure 4.

heading: 5 EXPERIMENTS
text: In addition to the theoretical results in this work, we provide an implementation of QRDimRed as described in Algorithm 1. We (1) empirically validate that our implementation satisfies the claims of Theorems 4.3 and Theorem 4.7 and (2) quantify real-world performance.
Empirical verification of Theorem 4.3 We verify the Theorem using a small model and synthetic data. We learn the function f(x) = e−x 2
usingN = 121 samples xj = −3+j/20 for 0 ≤ j < 121. We model fW as a radial neural network with layer widths n = (1, 6, 7, 1) and activation the radial shifted sigmoid h(x) = 1/(1+e−x+b). Applying QRDimRed gives a radial neural network fR with widths nred = (1, 2, 3, 1). Theorem 4.3 implies that the neural functions of fW and fR are equal. Over 10 random initializations of W, the mean absolute error (1/N) ∑ j |fW(xj) − fR(xj)| = 1.31 · 10−8 ± 4.45 · 10−9. Thus fW and fR agree up to machine precision. Empirical verification of Theorem 4.7 Similarly, we verify the conclusions of Theorem 4.7 using synthetic data. The claim is that training fQ−1·W with objective L by projected gradient descent coincides with training fR with objective Lred by usual gradient descent. We verified this for 3000 epochs at learning rate 0.01. Over 10 random initializations of W, the loss functions match up to machine precision with |L − Lred| = 4.02 · 10−9 ± 7.01 · 10−9. Reduced model trains faster. Due to the relation between projected gradient descent of the full network W and gradient descent of the reduced network R, our method may be applied before training to produce a smaller model class which trains faster without sacrificing accuracy. We test this hypothesis in learning the function f : R2 → R2 sending x = (t1, t2) to (e−t 2 1 , e−t 2 2) usingN = 1212 samples (−3+j/20,−3+k/20) for 0 ≤ j, k < 121. We model fW as a radial neural network with layer widths n = (2, 16, 64, 128, 16, 2) and activation the radial sigmoid h(r) = 1/(1 + e−r). Applying QRDimRed gives a radial neural network fR with widths nred = (2, 3, 4, 5, 6, 2). We trained both models until the training loss was ≤ 0.01. Running on a system with an Intel i5-8257U@1.40GHz and 8GB of RAM and averaged over 10 random initializations, the reduced network trained in 15.32± 2.53 seconds and the original network trained in 31.24± 4.55 seconds.

heading: 6 CONCLUSIONS AND FUTURE WORK
text: In this paper, we have adopted the formalism of quiver representation theory in order to establish a theoretical framework for neural networks. While drawing inspiration from previous work, our approach is novel in that it (1) reveals a large group of symmetries of neural network parameter spaces, (2) leads to a version of the QR decomposition in the context of radial neural networks, and (3) precipitates an algorithm to reduce the widths of the hidden layers in such networks.
Our main results, namely Theorems 4.3 and 4.7, may potentially generalize in several ways. First, these theorems are only meaningful if ni > ni−1 + 1 for some i (otherwise nred = n). Many networks such as decoders, super-resolution mappings, and GANs fulfill this criteria, but others do not. Our techniques may achieve meaningful width reductions in other cases. Second, the hypothesis that the activation functions are radial is crucial for our results, as they commute with orthogonal matrices, and hence interact favorably with QR decompositions. However, similar dimensional-reduction procedures may well be possible for other activation functions which relate to other matrix decompositions. In particular, the bottleneck effect discovered by Serra et al. (2018) for ReLU networks bears a striking similarity to our ascending condition. Third, though Theorem 4.7 does not hold for coordinate-dependent optimizers like Adam, other optimizers in the same spirit may be compatible. Fourth, there may be enhancements of our results that incorporate regularization and assert provable improvements to generalization. For example, the loss function of a radial neural network with dimension vector n remains invariant for the O(n) action after adding an L2 regularization term. Finally, the conceptual flexibility of quiver representation theory can encapsulate neural networks beyond MLPs, including CNNs, equivariant neural networks, RNNs, GNNs, and others.
Our result in Theorem 4.7 relates projected gradient descent γproj for the original network to gradient descent γ for the reduced network. In order to make theoretical conclusions about optimization of the original versus the reduced network, it is necessary to relate gradient descent on both spaces. To realize this, we conjecture, based on preliminary experimental evidence, that our version of projected gradient descent is a first-order approximation of usual gradient descent; we are pursuing work to make this relationship precise.

heading: ETHICS STATEMENT
text: Our work is primarily focused on theoretical foundations of machine learning, however, it does have a direct application in the form a model compression. Model compression is largely beneficial to the world since it allows for inference to run on smaller systems which use less energy. On the other hand, when models may be run on smaller systems such as smartphones, it is easier to use deep models covertly, for example, for facial recognition and surveillance. This may make abuses of deep learning technology easier to hide.

heading: REPRODUCIBILITY STATEMENT
text: The theoretical results of this paper, namely Theorem 4.3 and Theorem 4.7, may be independently verified through their proofs, which we include in their entirety in Appendices E and F, including all necessary definitions, lemmas, and hypotheses in precise and complete mathematical language. The empirical verification of Section 5 may be reproduced using the code included with the supplementary materials. In addition, Algorithm 1 is written in detailed pseudocode, allowing readers to recreate our algorithm in a programming language of their choosing.

heading: A ORGANIZATION OF APPENDICES
text: This paper is a contribution to the mathematical foundations of machine learning, and our results are motivated by expanding the applicability and performance of neural networks. At the same time, we use precise mathematical formalism throughout, and emphasize the rich abstract structures underlying neural networks. The purposes of these appendices are several:
1. To clarify the mathematical conventions and terminology, thus making the paper more accessible.
2. To provide full proofs of the main results.
3. To develop context around various constructions appearing in the main text.
4. To discuss in detail examples, special cases, and generalizations of our results.
We now give a summary of the contents of the appendices. Appendix B consists of preliminary material. We first review the necessary background on linear algebra and group theory, and then state the version of the QR decomposition that is relevant for this paper.
Appendix C serves to establish notation used in the context of gradient descent, and to prove an interaction between gradient descent and orthogonal transformations (Proposition C.2).
Appendix D supplements 3. Specifically, in Appendix D, we provide expanded foundations for our proposed framework of neural networks as quiver representations. In particular, we give a proof of Lemma 3.1, explain symmetries on the space of all neural networks with a specified dimension vector, and prove equivariance results. Finally, we explain how the “non-negative homogeneity” property of pointwise ReLU activation functions is a special case of our results (c.f Proposition D.5 and Remark D.7).
In Appendix E, we first establish elementary results related to radial functions and radial neural networks in Section E.1, and then give a proof of the QR Decomposition result (Theorem 4.3) in Section E.2. The proof relies on Algorithm 1.
In Appendix F, we focus on projected gradient descent. Specifically, we first introduce an ‘interpolating space’ that relates the space of representations with a given dimension vector to the space of representations with the reduced dimension vector (Section F.1). Using the interpolating space, we define a projected version of gradient descent (Section F.2) and give a proof of Theorem 4.7, which relates projected gradient descent and the QR decomposition for radial neural networks.
We devote Appendices H and I to special cases and generalizations of the constructions and results of the main text. Specifically, we consider a version of our results for neural networks with no biases (Section H.1), a generalization involving shifts in radial functions (Section H.2), and a formulation and generalization of our results using the language of category theory (Appendix I).

heading: B LINEAR ALGEBRA
text: In this appendix, we first review basic notation and concepts from linear algebra, group theory, and representation theory. We then state a version of the QR decomposition. References include Petersen et al. (2012), Quarteroni et al. (2007), and Dummit & Foote (2003).

heading: B.1 BASICS
text: For positive integers m and n, let Hom(Rm,Rn) or Rm×n denote the vector space of n by m matrices with entries in R, that is, the vector space of linear functions (also known as homomorphisms) from Rm to Rn. The general linear group GLn(R) consists of the set of invertible n by n matrices, with the operation of matrix multiplication. Equivalently, GLn(R) consists of all linear automorphisms of Rn, with the operation of composition. Such automorphisms are given precisely by change-of-basis transformations. The unit is the identity n by nmatrix, denoted idn. The orthogonal group O(n) is the subgroup of GLn(R) consisting of all matrices Q such that QTQ = idn. Such a matrix is called an orthogonal transformation.
Let G be a group. An action (or representation) of G on the vector space Rn is the data of an invertible n by n matrix ρ(g) for every group element g ∈ G, such that (1) for all g, h ∈ G, the matrix ρ(gh) is the product of the matrices ρ(g) and ρ(h), and (2) for the identity element 1G ∈ G, we have ρ(1G) = idn. In other words, an action amounts to a group homomorphism ρ : G → GLn(R). We often abbreviate ρ(g)(v) as simply g · v, for g ∈ G and v ∈ Rn. A function f : Rn → R (non-linear in general) is invariant for the action of G if f(g · v) = f(v) for all g ∈ G and v ∈ Rn. Suppose m ≤ n. Then incm,n : Rm ↪→ Rn denotes the standard inclusion into the first m coordinates: (v1, . . . , vm) 7→ (v1, . . . , vm, 0, . . . , 0) The standard projection πn,m : Rn Rm onto the first m coordinates is defined as:
(v1, . . . , vm, vm+1, . . . , vn) 7→ (v1, . . . , vm)
An affine map from Rm to Rn is one of the form
x 7→ Ax+ b
for some n × m matrix A and n-dimensional vector b. The set of affine maps from Rm to Rn can be identified with the vector space Hom(Rm+1,Rn), as we now explain. First, given A and b as above, we form the n × (m + 1) matrix [b A]. Conversely, the affine map corresponding to W ∈ Hom(Rm+1,Rn) is given by
W̃ =W ◦ extm : Rm → Rn; x 7→W (extm(x)) ,
where extm is defined as follows: Definition B.1. For m ≥ 1, let
extm : Rm → Rm+1
be the map taking v = (v1, v2, . . . , vm) ∈ Rm to (1, v1, v2, . . . , vm) ∈ Rm+1.

heading: B.2 THE QR DECOMPOSITION
text: In this section, we recall the QR decomposition and note several relevant facts. For integers n and m, let Homupper(Rm,Rn) denote the vector space of upper triangular n by m matrices. Theorem B.2 (QR Decomposition). The following map is surjective:
O(n)×Homupper(Rm,Rn) −→ Hom(Rm,Rn) Q , R 7→ Q ◦R
In other words, any matrix can be written as the product of an orthogonal matrix and an uppertriangular matrix. When m ≤ n, the last n − m rows of any matrix in Homupper(Rm,Rn) are zero, and the top m rows form an upper-triangular m by m matrix. These observations lead to the following special case of the QR decomposition: Corollary B.3. Suppose m ≤ n. The following map is surjective:
µ : O(n)×Homupper(Rm,Rm) −→ Hom(Rm,Rn) Q , R 7→ Q ◦ incm,n ◦ R
We make some remarks:
1. There are several algorithms for computing the QR decomposition of a given matrix. One is Gram–Schmidt orthogonalization, and another is the method of Householder reflections. The latter has computational complexity O(n2m) in the case of a n × m matrix with n ≥ m. The package numpy includes a function numpy.linalg.qr that computes the QR decomposition of a matrix using Householder reflections.
2. The QR decomposition is not unique in general, or, in other words, the map µ is not injective in general. For example, if n > m, each fiber of µ contains a copy of O(n−m).
3. The QR decomposition is unique (in a certain sense) for invertible square matrices. To be precise, let B+n be the subset of of Hom
upper(Rn,Rn) consisting of upper triangular matrices with positive entries along the diagonal. If n = m, then both B+n and O(n) are subgroups of GLn(R), and the multiplication map O(n) × B+n → GLn(R) is bijective. However, the QR decomposition is not unique for non-invertible square matrices.

heading: C SYMMETRY AND GRADIENT DESCENT
text: In this appendix, we establish a gradient descent formalism that appears throughout the paper. Additionally, we explain the interaction between gradient descent and orthogonal transformations, which is a key aspect of our main results.
Fix a positive integer p. Let V = Rp and L : V → R be a differentiable function4. Semantically, p is the number of independent parameters, V is the parameter space, and L is the loss function associated to a batch of training data. We write elements of V as tuples, e.g. v = (v1, . . . , vp), or as column vectors. We write elements of the dual vector space V ∗ as row vectors. The differential and gradient of L are defined as:
dL : V −→ Hom(V,R) = V ∗ ∇L : V −→ V v 7→ dvL = [ ∂L ∂x1 ∣∣∣∣ v · · · ∂L∂xp ∣∣∣∣ v ] v 7→ ∇vL = ( ∂L ∂x1 ∣∣∣∣ v , . . . , ∂L ∂xp ∣∣∣∣ v ) Remark C.1. Suppose that L is in fact a linear map. Then L belongs to V ∗ and dL is the constant map taking each v ∈ V to L. That is, dvL = L as maps V → R.
A step of gradient descent with respect to L at learning rate η > 0 is the function
γ = γη : V −→ V v 7→ v − η · ∇vL
Hence, gradient descent updates the i-th coordinate vi of v to vi − η · ∂L∂xi |v , for i = 1, . . . , p. When η is clear from context, we abbreviate γη by γ. For any k ≥ 0, the k-fold composition of γ is γk := γ ◦ γ ◦ · · · ◦ γ , with γ0 being the identity map on V . Throughout the appendices, all proofs involving gradient descent reduce to the η = 1 case. The arguments for general η > 0 and can be obtained either by inserting the scalar η where necessary, or by rescaling the function L. Proposition C.2. Let Q be an orthogonal transformation of V = Rp such that L ◦ Q = L. Then, for any v ∈ V and k ≥ 0, we have:

heading: Q (∇vL) = ∇Q(v)L and γk(Q(v)) = Q(γk(v)).
text: Proof. We give a proof in the case η = 1. The general case follows similarly. The hypothesis L ◦Q = L implies that:
∇vL = (dvL)T = (dv(L ◦Q))T = (dQ(v)L ◦ dvQ)T = (dQ(v)L ◦Q)T = QT (dQ(v)L)T
= QT (∇Q(v)L)
where dQ(v)Q = Q since Q is a linear map. The fact that Q is an orthogonal transformation implies that Q−1 = QT . The first claim follows. For the second claim, we first compute:
γ(Q(v)) = Q(v)−∇Q(v)L = Q(v)−Q(∇vL) = Q(v −∇vL) = Q(γ(v))
Inductive reasoning shows that γk(Q(v)) = Q(γk(v)) for all k ≥ 0.
The following two results are not necessary for the results of the current paper, but may be of interest to readers. We omit the proofs (which are straightforward).
4This discussion extends easily to the case where L is piece-wise differentiable.
Proposition C.3. Suppose that a group G acts on V by orthogonal transformations and that L is G-invariant. Then there is a well-defined gradient descent map on the set of G-orbits on V :
γ̃ : V/G→ V/G. Proposition C.4. Let M be a linear endomorphism of V = Rp such that L ◦M = L.
1. For any v ∈ V , we have: ∇vL =MT (∇M(v)L).
2. If M = P is an orthogonal projection, i.e. P 2 = P = PT , then, for any v ∈ V and k ≥ 0, we have:
γk(v) = v − P (v) + γk(P (v)).

heading: D THE NEURAL QUIVER
text: In this appendix, we expand on Section 3 of the main text. We explain in more detail the relation between representations of the neural quiver and affine maps, describe an action of GLhiddenn on the space of neural networks Neur(QL,n), and prove equivariance results related to this action. References for general quiver representation theory include (Nakajima et al., 1998) and (Kirillov Jr, 2016).

heading: D.1 THE NEURAL QUIVER
text: We begin by recalling the definition of the neural quiver. For a positive integer L, the neural quiver QL is the following quiver with L+ 2 vertices:
• // • // • // · · · // • // •
•
77==aagg
The vertices in the top row are indexed from i = 0 to i = L. The first vertex (i = 0) in the top row is the sole input vertex, the last vertex (i = L) is the output vertex, and every other vertex (1 ≤ i ≤ L − 1) in the top row is a hidden vertex. The vertex at the bottom is the sole bias vertex and is connected to each vertex in the top row except for i = 0. We only consider dimension vectors whose value at the bias vertex is equal to 1. Hence, a dimension vector for QL will refer to an (L + 1)-tuple of positive integers n = (n0, n1, . . . , nL). We now restate and give a full proof of Lemma 3.1: Lemma D.1. (Lemma 3.1.) A representation of QL with dimension vector n is equivalent to the data of an affine map Rni−1 → Rni for each i = 1, . . . , L. There is an isomorphism of vector spaces:
Rep(QL,n) ' L⊕ i=1 Hom(R1+ni−1 ,Rni).
Proof. By definition, a representation ofQL with dimension vector n consists of the data of a matrix Ai ∈ Hom(Rni−1 ,Rni) and a vector bi ∈ Rni , for each i = 1, . . . , L. For each i, the corresponding affine map W̃i : Rni−1 → Rni is given by x 7→ Aix+bi. Conversely, given the data of an affine map Rni−1 → Rni , for each i, the linear part defines an element of Hom(Rni−1 ,Rni) and the translation part defines an element of Rni . This proves the first claim.
The second claim follows from the discussion of affine maps appearing in Section B. Explicitly, given an affine map x 7→ Aix + bi from Rni−1 to Rni for each i = 1, . . . , L, one defines an element of ⊕L i=1 Hom(R1+ni−1 ,Rni) whose i-th factor is the matrix Wi = [bi Ai]. Conversely, pre-composing a matrix in Hom(R1+ni−1 ,Rni) with the map extni−1 : Rni−1 → R1+ni−1 gives an affine map W̃i from Rni−1 to Rni .
Notation: In light of the preceding lemma, we henceforth denote a representation of QL as a tuple W = (Wi) L i=1, where each Wi belongs to Hom(R1+ni−1 ,Rni).
Recall that the parameter symmetry group GLhiddenn is defined as the product of the groups GLni(R) as i runs over the hidden vertices of the neural quiver QL:
GLhiddenn := L−1∏ i=1 GLni(R).
Hence an element of GLhiddenn consists of the choice of a transformation of Rni for each hidden vertex i. There is an action of GLhiddenn on Rep(QL,n) given by5:
W = (Wi) L i=1 7→ g ·W := ( gi ◦Wi ◦ [ 1 0 0 g−1i−1 ])L i=1 ,
where g = (gi)L−1i=1 is an element of GL hidden n , and g0 = idn0 and gL = idnL .

heading: D.2 NEURAL NETWORKS
text: Recall the following definition, which appears as Definition 3.2 in Section 3.4 of the main text, and is a special case of the more general Definition G.6 appearing below in Appendix G. Definition D.2. Let L be a positive integer. An QL-neural network consists of the following:
1. Hyperparameters. A dimension vector n = (n0, n1, . . . , nL) for the neural quiver QL.
2. Trainable parameters. A representation W = (W1, . . . ,WL) of the neural quiver QL with dimension vector n. So, for i = 1, . . . , L, we have a matrix Wi ∈ Hom(R1+ni−1 ,Rni).
3. Activation functions. Piece-wise differentiable functions ai : Rni → Rni for i = 1, . . . , L. These are grouped into a tuple a = (a1, . . . , aL).
As in Section 3.4, we denote by Neur(QL,n) the vector space of neural networks with L layers and dimension vector n, and define the neural function of a neural network (W,a) as:
f(W,a) : Rn0 → RnL x 7→ aL ◦ W̃L ◦ · · · ◦ a2 ◦ W̃2 ◦ a1 ◦ W̃1(x)
where W̃i : Rni−1 → Rni is the affine map corresponding to Wi.
There is an action of the group GLhiddenn on Neur(QL,n) given as follows. The element g = (gi) L−1 i=1 ∈ GL hidden n transforms (W,a) ∈ Neur(QL,n) as:
(W,a) 7→ g · (W,a) := ( ( gi ◦Wi ◦ [ 1 0 0 g−1i−1 ])L i=1 , ( gi ◦ ai ◦ g−1i )L i=1 ) , (D.1)
where g0 = idn0 and gL = idnL . The following lemma shows that the neural function is unchanged by the GLhiddenn action, that is, if two neural networks are related by the action of an element in GLhiddenn , then their neural functions are the same.
Lemma D.3. For any neural network (W,a) in Neur(QL,n) and any g in GLhiddenn , the neural functions of (W,a) and g · (W,a) coincide:
fg·(W,a) = f(W,a)
Proof. The element g = (gi) ∈ GLhiddenn transforms W̃i =Wi ◦ extni−1 to gi ◦Wi ◦ extni−1 ◦ g−1i−1 and ai to gi ◦ ai ◦ g−1i , for i = 1, . . . , L (where g0 = idn0 and gL = idnL ). Thus, g transforms the composition ai ◦ W̃i to the composition gi ◦ ai ◦ W̃i ◦ g−1i−1. The result now follows from the definition of the neural function.
5As in the main text, for each i, the matrix [ 1 0 0 g−1i−1 ] is block diagonal with one block of size 1 × 1
consisting of the single entry ‘1’, and one block of size ni−1 × ni−1 consisting of the matrix g−1i−1.
We now define a certain subgroup of GLhiddenn that will feature in later discussion. Definition D.4. The stabilizer in GLhiddenn of a choice of activation functions a is defined as:
Z(a) = {g = (gi) ∈ GLhiddenn : gi ◦ ai ◦ g−1i = ai for all i = 1, . . . , L− 1}.
Note that each ai is a function from Rni to Rni , as is each gi, so the equality gi ◦ ai ◦ g−1i = ai is as functions from Rni to Rni . As a subgroup of GLhiddenn , the stabilizer Z(a) acts on Rep(QL,n). We now recall from Section 3 the formulation of the optimization problem for neural networks in the language of quiver representations. For the remainder of this section, we fix a dimension vector n for the the neural quiver QL, and activation functions a = (a1, . . . , aL). To a batch of training data X = {(xj , yj)} ⊆ Rn0 × RnL , there is an associated loss function on Rep(QL,n) defined as
L = LX : Rep(QL,n)→ R L(W) = ∑ j C(f(W,a)(xj), yj)
where C : RnL ×RnL → R is a cost function on RnL . Following the formalism of Appendix C, we obtain a gradient descent map
γ : Rep(QL,n)→ Rep(QL,n) for any learning rate η > 0. Hence:
→ Upshot: the backpropogation algorithm can be regarded as taking place on the vector space Rep(QL,n) of representations of the neural quiver QL.
To conclude this section, we demonstrate that the loss function is invariant for the action of the group Z(a). Proposition D.5. Let a be activation functions, and let L be the loss function associated to a batch of training data. Then, for all g ∈ Z(a) and W ∈ Rep(QL,n), we have:
L(g ·W) = L(W).
Proof. One first verifies using the definition of Z(a) that:
(g ·W,a) = g · (W,a) for any g ∈ Z(a) and W ∈ Rep(QL,n), where the left-hand side invokes the action of Z(a) on Rep(QL,n), while the right-hand side invokes the action of Z(a) on Neur(QL,n). Using this fact and Lemma D.3, we see that: f(g·W,a) = fg·(W,a) = f(W,a) and hence, for any g ∈ Z(a) and W ∈ Rep(QL,n), we have:
L(g ·W) = ∑ j C(f(g·W,a)(xj), yj) = ∑ j C(f(W,a)(xj), yj) = L(W).
Remark D.6. An alternative proof is as follows. The loss function can be written as the composition
Rep(QL,n) sa−→ Neur(QL,n) −→ R
where the first map takes W to (W,a) (and hence is a section of the natural projection from Neur(QL,n) to Rep(QL,n)), and the second map sends (W,a) to the sum ∑ j C(f(W,a)(xj), yj). One verifies directly that sa is equivariant for the action of the group Z(a), while the second map is invariant for GLhiddenn , of which Z(a) is a subgroup. Remark D.7. Suppose ai : Rni → Rni is pointwise ReLU for each i. Then Z(a) consists of those g = (gi) ∈ GLhiddenn such that each gi ∈ GLni(R) is a diagonal matrix with positive entries along the diagonal. In symbols:
Z(a) = L−1∏ i=1 (R>0)ni .
Hence, the “non-negative homogeneity” (or “positive scaling invariance”) property of pointwise ReLU activation functions appearing in Dinh et al. (2017); Neyshabur et al. (2015); Meng et al. (2019) emerges as a special case of Proposition D.5.

heading: E RADIAL NEURAL NETWORKS AND THE QR DECOMPOSITION
text: In this appendix, we give a proof of the main theoretical result (Theorem 4.3) involving a QR decomposition for radial neural networks. To this end, we first collect results about radial functions and radial neural networks.

heading: E.1 RADIAL FUNCTIONS
text: We first recall the definition of radial functions from Section 4.1 above. Let h : R → R be a piece-wise differentiable function such that limr→0 h(r) r exists. For any n ≥ 1, set:
h(n) : Rn → Rn; h(n)(v) := h(|v|) v |v| .
A function a : Rn → Rn is called radial if a = h(n) for some h : R→ R. Lemma E.1. Let a = h(n) : Rn → Rn be a radial function on Rn.
1. The radial function a commutes with any orthogonal transformation of Rn. In symbols: a ◦Q = Q ◦ a for any Q ∈ O(n).
2. If m ≤ n and incm,n : Rm ↪→ Rn is the standard inclusion, then: h(n) ◦ incm,n = incm,n ◦ h(m).
Proof. Suppose Q ∈ O(n) is an orthogonal transformation of Rn. Since Q is norm-preserving, we have |Qv| = |v| for any v ∈ Rn. Since Q is linear, we have Q(λv) = λQv for any λ ∈ R and v ∈ Rn. Using the definition of a = h(n) we compute:
a(Qv) = h(|Qv|) |Qv| Qv = h(|v|) |v| Qv = Q ( h(|v|) |v| v ) = Q(a(v)).
The first claim follows. The second claim is an elementary verification.
Remark E.2. More generally, it is straightforward to show that the restriction of the radial function a to a linear subspace of Rn is a radial function on that subspace.
Next, we recall the definition of a radial neural network (as stated in Section 4.1). Let L be a positive integer and n be a dimension vector for the neural quiver QL. A neural network (W,a) ∈ Neur(QL,n) is said to be a radial neural network if each activation function ai is a radial function on Rni . Let O(n) denote the following product of the orthogonal groups:
O(n) = O(n1)×O(n2)× · · · ×O(nL−1). Note that O(n) is a subgroup of GLhiddenn and hence acts on Rep(QL,n). The following proposition involves the loss function associated to a batch of training data as formulated in Section 3.4. Proposition E.3. For any radial neural network in Neur(QL,n) and any batch of training data, the loss function L : Rep(QL,n)→ R is invariant for the action of O(n).
Proof. Let g = (g1, . . . , gL−1) ∈ O(n). Lemma E.1 implies that the orthogonal transformation gi ∈ O(ni) commutes with the radial function ai = h(ni), for each i, or, equivalently: gi◦ai◦g−1i = ai. Hence O(n) is a subgroup of the centralizer Z(a), and we apply Proposition D.5.
For clarity, we restate Construction 4.2, which illustrates the flexibility of radial functions in defining the activation functions of a neural network for any given representation of QL with arbitrary dimension vector. Construction E.4. Suppose h = (h1, . . . , hL) is a tuple of functions 6 hi : R → R, and suppose n = (n0, . . . , nL) is a dimension vector for QL. Consider the tuple of activation functions:
a (n) h = ( h (n1) 1 , . . . , h (nL) L ) .
Any representation W of QL with dimension vector n defines a neural network ( W,a (n) h ) as in
Definition 3.2. We write simply (W,h) for this network and f(W,h) for its neural function. 6We assume each hi is piece-wise differentiable and the limit limr→0 hi(r)r exists.

heading: E.2 PROOF OF THEOREM 4.3
text: In this section, we give a proof of the QR decomposition appearing in Theorem 4.3. In order to do so, we make some recollections from Section 4.2.
Given a dimension vector n = (n0, n1, . . . , nL) for the neural quiver QL, recall that its reduction nred = (nred0 , n red 1 , . . . , n red L ) is defined recursively by setting n red 0 = n0, then n red i = min(ni , n red i−1 + 1) for i = 1, . . . , L − 1, and finally nredL = nL. We abbreviate Rep(QL,n) by Rep(n) and Rep(QL,nred) by Rep(nred). We also had in Section 4.2 the inclusion ι : Rep(nred) ↪→ Rep(n), which can be understood in terms of padding matrices with rows and columns of zeros. Another description is as follows. Let X = (X1, . . . , XL) ∈ Rep(nred). Then ι(X)i = inci ◦Xi ◦ π̃i−1. Indeed, post-composing Xi with inci amounts to adding ni − nredi rows of zeros on the bottom, while pre-composing Xi with π̃i−1 amounts to adding ni−1 − nredi−1 columns of zeros on the right. Fix a tuple h = (h1, . . . , hL) of functions hi : R → R. As in Construction 4.2, we attach a radial neural network (and hence a neural function) to every representation of QL. As a reminder, we restate Theorem 4.3. Theorem E.5. (Theorem 4.3) Let n be a dimension vector for QL and fix h as above. For any W ∈ Rep(n) there exist:
Q ∈ O(n), R = (R1, . . . , RL) ∈ Rep(nred), and U ∈ Rep(n),
such that:
1. The matrices R1, . . . , RL−1 are upper triangular.
2. The following equality holds: W = Q · ι(R) +U 3. The neural networks defined by W and R have identical neural functions: f(W,h) = f(R,h).
Before stating the proof, we first adopt all notational conventions of Section 4.2, especially those listed after the statement of Algorithm 1. Additionally, we note the projection maps:
πi : Rni Rn red i π̃i : R1+ni R1+n red i
onto the first nredi and 1 + n red i coordinates, respectively. We also have the identity:
idni+1 = pi−1 + ĩnci−1 ◦ π̃i−1 (E.1)
Next, we collect the following basic identities related to the map extn : Rn → Rn+1 (see Definition B.1) taking (v1, . . . , vn) to (1, v1, . . . , vn). Their proofs are elementary. Lemma E.6. For m ≤ n, we have: extn ◦ incm,n = incm+1,n+1 ◦ extm.
Lemma E.7. For any n×m matrix A, we have7: extn ◦A = [ 1 0 0 A ] ◦ extm.
Proof of Theorem 4.3. We claim that the outputs Q, R, and U of Algorithm 1 satisfy the required conditions. The matrices R1, . . . , RL−1 are upper-triangular by construction, so the first part of the theorem holds.
To show that the neural functions of (W,h) and (R,h) coincide, we first show that, for all i = 1, . . . , L, we have:
h(ni) ◦ W̃i ◦Qi−1 ◦ inci−1 = Qi ◦ inci ◦ h(n red i ) ◦ R̃i (E.2)
7The matrix [ 1 0 0 A ] is (n+1)× (m+1), so the top right 0 abbreviates a row of m zeros and the bottom
left 0 abbreviates a column of n zeros.
where Q0 = idn0 and QL = idnL . The justification is a computation:
h(ni) ◦ W̃i ◦Qi−1 ◦ inci−1 = h(ni) ◦Wi ◦ extni−1 ◦Qi−1 ◦ inci−1 = h(ni) ◦Wi ◦ [ 1 0 0 Qi−1 ] ◦ ĩnci−1 ◦ extnredi−1
= h(ni) ◦Qi ◦ inci ◦Ri ◦ extnredi−1 = Qi ◦ inci ◦ h (nredi ) ◦ R̃i.
The first equality uses the definition of W̃i in terms of Wi; the second uses the commutativity properties of ext stated in Lemmas E.6 and E.7; the third uses the definitions of Qi, Ri, and Vi; and the fourth uses the commutativity properties of radial functions as well as the definition of R̃i. The fact that f(W,h) = f(R,h) now follows from the definition of the neural function and iterative applications of Equation E.2, noting that Q0 ◦ inc0 = idn0 and QL ◦ incL = idnL . Finally, to show that W = Q · ι(R) +U, we compute, for i = 1, . . . , L: (Q · ι(R) +U)i = (Q · ι(R))i +Ui
= Qi ◦ inci ◦Ri ◦ π̃i−1 ◦ [ 1 0 0 Q−1i−1 ] +Wi ◦ [ 1 0 0 Qi−1 ] ◦ pi−1 ◦ [ 1 0 0 Q−1i−1 ] =Wi ◦ [ 1 0 0 Qi−1 ] ◦ ĩnci−1 ◦ π̃i−1 ◦ [ 1 0 0 Q−1i−1
] +Wi ◦ [ 1 0 0 Qi−1 ] ◦ pi−1 ◦ [ 1 0 0 Q−1i−1
] =Wi ◦ [ 1 0 0 Qi−1 ] ◦ ( ĩnci−1 ◦ π̃i−1 + pi−1 ) ◦ [ 1 0 0 Q−1i−1
] =Wi ◦ [ 1 0 0 Qi−1 ] ◦ id1+ni−1 ◦ [ 1 0 0 Q−1i−1
] =Wi.
The first two equalities follow from definitions; the third uses the definitions of Qi, Ri, and Vi; the fourth uses the distributive law; and the fifth appeals to the Equation E.1.

heading: F PROJECTED GRADIENT DESCENT
text: In this section, we give a proof of Theorem 4.7, which relates projected gradient descent for a representation with dimension n to (usual) gradient descent for the corresponding reduced representation with dimension vector nred.
We adopt all notational conventions from Section 4. In particular, we fix:
• a tuple h = (h1, . . . , hL) of functions, where hi : R → R for i = 1, . . . , L. These can be used to define a neural function f(W,h) (resp. f(X,h)) for every representation W in Rep(n) (resp. X in Rep(nred)). See Construction 4.2 for more details.
• a batch of training data {(xj , yj)} ⊆ Rn0 × RnL = Rn red 0 × RnredL .
• a cost function C : RnL × RnL → R
As a result, we have loss functions on Rep(n) and Rep(nred) given by:
L : Rep(n)→ R Lred : Rep(nred)→ R W 7→ ∑ j C(f(W,h)(xj), yj) X 7→ ∑ j C(f(X,h)(xj), yj)
as in Section 3.4. Fixing a learning rate η > 0, there are resulting gradient descent maps on Rep(n) and Rep(nred) given by:
γ : Rep(n)→ Rep(n) γred : Rep(nred)→ Rep(nred) W 7→W − η · ∇WL X 7→ X− η · ∇XLred

heading: F.1 THE INTERPOLATING SPACE
text: In this section, we introduce a subspace Repint(n) of Rep(n), that, as we will later see, interpolates between Rep(nred) and Rep(n).
Recall the standard inclusions and projections, for i = 0, 1, . . . , L:
inci = incnredi ,ni : R nredi ↪→ Rni ĩnci = inc1+nredi ,1+ni : R 1+nredi ↪→ R1+ni
πi : Rni Rn red i π̃i : R1+ni R1+n red i
Definition F.1. Let Repint(n) denote the subspace of Rep(n) consisting of those T = (T1, . . . , TL) ∈ Rep(n) such that:
(idni − inci ◦ πi) ◦ Ti ◦ ĩnci−1 ◦ π̃i−1 = 0
for i = 1, . . . , L. Write ι1 : Repint(n) ↪→ Rep(n) for the natural inclusion.
In other words, the space Repint(n) consists of representations T = (T1, . . . , TL) ∈ Rep(n) for which the bottom left (ni − nredi )× (1 + nredi−1) block of Ti is zero for each i:
Ti = [ ∗ ∗ 0 ∗ ] .
We omit the proof of the following proposition, as it involves an elementary analysis of the workings of Algorithm 1.
Proposition F.2. Let W ∈ Rep(n) and let Q, R, and U be the outputs of Algorithm 1, so W = Q ·R+U. Then the representation Q−1 ·W = ι(R) +Q−1 ·U belongs to Repint(n).
Definition F.3. Define a map q1 : Rep(n)→ Repint(n)
by taking T ∈ Rep(n) to the representation whose i-th slot is
Wi − (idni − inci ◦ πi) ◦Wi ◦ ĩnci−1 ◦ π̃i−1.
It is straightforward to check that q1 is well-defined, i.e., that the resulting representation belongs to Repint(n) and that q1 is a surjective linear map. The transpose of q1 is the inclusion ι1. We summarize the situation in the following diagram:
Repint(n) ι1
,, Rep(n)
q1 mm (F.1)
We observe that the composition q1 ◦ ι is the identity on Repint(n), while the endomorphism ι1 ◦ q1 of Rep(n) takes a representation W ∈ Rep(n) and, for each i, zeros all entries in the bottom left (ni − nredi )× (1 + nredi−1) submatrix of Wi.

heading: F.2 PROJECTED GRADIENT DESCENT AND THE QR DECOMPOSITION
text: In this section, we define the projected gradient descent map and state a theorem relating projected gradient descent on Rep(n) with usual gradient descent on Rep(nred).
Definition F.4. The projected gradient descent map on Rep(n) with respect to Repint(n) at learning rate η < 0 is defined as:
γproj : Rep(n)→ Rep(n) W 7→W − η · Pr(∇WL)
where Pr = ι1 ◦ q1.
One verifies easily that the above definition of projected gradient descent is the same as that given in Definition 4.6. To reiterate, while all entries of each matrix Wi in the representation W contribute to the computation of the gradient ∇WL, only those not in the bottom left submatrix get updated under the projected gradient descent map γproj.
We now restate Theorem 4.7. To ease notation, we identify elements of Rep(nred) with their images in Rep(n) under ι.
Theorem F.5. (Theorem 4.7) Let W ∈ Rep(n) and let Q, R, and U be the outputs of Algorithm 1. Fix a learning rate η > 0. For any k ≥ 0, we have:
γk(W) = Q · γk(Q−1 ·W) and γkproj(Q−1 ·W) = γkred(R) +Q−1 ·U.
We summarize this result in the following diagram. The left horizontal maps indicate the addition of Q−1 ·U, the right horizontal arrows indicate the action of Q, and the vertical maps are various versions of gradient descent. The shaded regions indicate the (smallest) vector space to which the various representations naturally belong.
R Q−1 ·W W
γkred(R) γ k proj(Q −1 ·W) γk(Q−1 ·W) γk(W)
+Q−1 ·U
proj-GD on Rep(n)
+Q−1 ·U
GD on Rep(nred) GD on Rep(n)
Q·
Q·
GD on Rep(n)
Rep(nred) Repint(n) Rep(n)
Remark F.6. The map Rep(n)→ Rep(n) taking W to Q · γproj(Q−1 ·W) is a projected gradient descent map on Rep(n) with respect to the subspace of Rep(n) formed by translating Repint(n) by the action of Q, i.e. the subspace Q · Repint(n) = {Q ·T | T ∈ Repint(n) }.

heading: F.3 PROOF OF THEOREM 4.7
text: We begin by explaining the sense in which Repint(n) interpolates between Rep(n) and Rep(nred). One extends Diagram F.1 as follows:
Rep(nred) ι2
-- Repint(n)
q2
mm
ι1 ,, Rep(n)
q1 mm
• The map ι2 : Rep(n
red) ↪→ Repint(n) takes X to the representation whose i-th coordinate is inci ◦Xi ◦ π̃i−1. It is straightforward to check that ι2 is a well-defined injective linear map.
• The map q2 : Rep int(n) Rep(nred)
takes T to the representation whose i-th slot is πi ◦ Ti ◦ ĩnci−1. It is straightforward to check that q2 is a surjective linear map. The transpose of q2 is the inclusion ι2.
Lemma F.7. We have the following:
1. The inclusion ι : Rep(nred) ↪→ Rep(n) coincides with the composition ι1 ◦ ι2, and commutes with the loss functions:
Rep(nred) ι1◦ι2=ι //
Lred ##
Rep(n)
L ||
R
2. The following diagram commutes:
Repint(n) q2 // //
_
ι1
Rep(nred)
Lred
Rep(n) L // R
3. For any T ∈ Repint(n), we have: q1 ( ∇ι1(T)L ) = ι2 ( ∇q2(T)Lred ) .
Proof. The identity ι = ι1 ◦ ι2 follows directly from definitions. To prove the commutativity of the diagram, it is enough to show that, for any X in Rep(nred), the neural functions of X and ι(X) coincide. This follows easily from the fact that, for i = 1, . . . , L, we have:
πi ◦ h(ni) ◦ inci = πi ◦ inci ◦ h(n red i ) = h(n red i ).
For the second claim, let T ∈ Repint(n). It suffices to show that ι1(T) and q2(T) have the same neural function. The key computation is:
inci ◦ h(n red i ) ◦ πi ◦ T̃i ◦ inci−1 = h(ni) ◦ inci ◦ πi ◦ Ti ◦ extni−1 ◦ inci−1
= h(ni) ◦ inci ◦ πi ◦ Ti ◦ ĩnci−1 ◦ extni−1 = h(ni) ◦ Ti ◦ ĩnci−1 ◦ extni−1 = h(ni) ◦ Ti ◦ extni−1 ◦ inci−1 = h(ni) ◦ T̃i ◦ inci−1
which uses the fact that (idni − inci ◦ πi)◦Ti ◦ ĩnci−1 = 0, or, equivalently, inci ◦πi ◦Ti ◦ ĩnci−1 = Ti ◦ ĩnci−1. Applying this relation successively starting with the second-to-last layer (i = L − 1) and ending in the first (i = 1), one obtains the result. For the last claim, one computes ∇T(L ◦ ι1) in two different ways. The first way is:
∇T(L ◦ ι1) = (dT (L ◦ ι1))T = ( dι1(T)L ◦ dTι1 )T = ( dι1(T)L ◦ ι1 )T = ιT1 (( dι1(T)L )T) = q1 ( ∇ι1(T)L
) where we use the fact that ι1 is a linear map whose transpose is q1. The second way uses the commutative diagram of the second part of the Lemma:
∇T(L ◦ ι1) = ∇T (Lred ◦ q2) = (dT (Lred ◦ q2))T = ( dq2(T)Lred ◦ dZq2 )T = ( dq2(T)Lred ◦ q2 )T = qT2 (( dq2(T)Lred )T) = ι2 ( ∇q2(T)Lred ) .
We also use the fact that q2 is a linear map whose transpose is ι2.
Proof of Theorem 4.7. The action of Q ∈ O(n) on Rep(n) is an orthogonal transformation, so the first claim follows from Proposition C.2. For the second claim, it suffices to consider the case η = 1. The general case follows similarly. We proceed by induction. The base case k = 0 amounts to Theorem 4.3. For the induction step, we set
Z(k) = ι(γkred(R)) +Q −1 ·U.
Each Z(k) belongs to Repint(n), so i1(Z(k)) = Z(k). Moreover, q2 ( Z(k) ) = γkred(R). We compute:
γk+1proj (Q −1 ·W) = γproj ( γkproj(Q −1 ·W) )
= γproj ( ι(γkred(R)) +Q −1 ·U )
= ι(γkred(R)) +Q −1 ·U− ι1 ◦ q1 ( ∇ι(γkred(R))+Q−1·UL ) = ι(γkred(R))− ι1 ◦ q1 ( ∇ι1(Z(k))L ) +Q−1 ·U
= ι(γkred(R))− ι1 ◦ ι2 ( ∇q2(Z(k))Lred ) +Q−1 ·U
= ι ( γkred(R)−∇γkred(R)Lred ) +Q−1 ·U
= ι ( γk+1red (R) ) +Q−1 ·U
where the second equality uses the induction hypothesis, the third invokes the definition of γproj, the fourth uses the relation between the gradient and orthogonal transformations, the fifth and sixth use Lemma F.7 above, and the last uses the definition of γred.

heading: F.4 EXAMPLE
text: We now discuss an example where projected gradient descent does not match usual gradient descent.
Let n = (1, 3, 1) be a dimension vector for the neural quiverQL. The space of representations with this dimension vector is 10-dimensional:
Rep(QL,n) = Hom(R2,R3)⊕Hom(R4,R) ' R10. We identify a representation
W = ( W0 = [ a b c d e f ] , W1 = [g h i j] ) ∈ Rep(QL, (1, 3, 1))
with the point p = (a, b, c, d, e, f, g, h, i, j) in R10. The action of the orthogonal group O(n) = O(3) on Rep(QL,n) ' R10 can be expressed as:
Q 7→ Q 0 0 00 Q 0 00 0 1 0 0 0 0 Q  . Consider the function8:
L : Rep(QL,n)→ R p = (a, b, c, d, e, f, g, h, i, j) 7→ h(a+ b) + i(c+ d) + j(e+ f) + g
By the product rule, we have:
∇pL = (h, h, i, i, j, j, 1, a+ b, c+ d, e+ f) One easily checks that L(Q · p) = L(p) and that∇Q·pL = Q · ∇pL for any Q ∈ O(3).
The interpolating space is the subspace of Rep(QL,n) ' R8 with e = f = 0. Suppose p′ = (a, b, c, d, 0, 0, g, h, i, j) belongs to the interpolating space. Then the gradient is
∇p′L = (h, h, i, i, j, j, 1, a+ b, c+ d, 0) which does not belong to the interpolating space. So one step of usual gradient descent, with learning rate η > 0 yields:
γ : p′ =(a, b, c, d, 0, 0, g, h, i, j) 7→ (a− ηh , b− ηh , c− ηi , d− ηi , −ηj , −ηj , g − η , h− η(a+ b) , i− η(c+ d) , j)
8For W ∈ Rep(QL,n), the neural function of the neural network with affine maps determined by W and identity activation functions is R→ R; x 7→ L(W)x. The function L can appear as a loss function for certain batches of training data and cost function on R.
On the other hand, one step of projected gradient descent yields: γproj : p
′ = (a, b, c, d, 0, 0, g, h, i, j) 7→ (a− ηh , b− ηh , c− ηi , d− ηi , 0 , 0 , g − η , h− η(a+ b) , i− η(c+ d) , j)
Direct computation shows that the difference between the evaluation of L after one step of gradient descent and the evaluation of L after one step of projected gradient descent is:
L(γ(p′))− L(γproj(p′)) = 2ηj2.

heading: G QUIVER NEURAL NETWORKS IN GENERAL
text: In this section, we generalize some of our main results to a larger class of quivers beyond the neural quiver. These more general quivers are known as “neural-adapted quivers”, defined presently.

heading: G.1 NEURAL-ADAPTED QUIVERS
text: Definition G.1. A neural-adapted quiver is a quiver Q = (I, E) together with a partition I = Iin ∪B ∪H ∪ {iout} of the vertex set such that the following hold:
1. The quiver Q is acyclic and no two edges share the same source and target.
2. The vertex iout is the unique sink of Q. This is the output vertex.
3. The set Iin is non-empty, and every vertex therein is a source of Q. These are the input vertices.
4. Every vertex in B is a source of Q. These are the bias vertices.
5. Every vertex in H is neither a source nor a sink, and admits a path from an input vertex. These are the hidden vertices.
Given a quiver satisfying condition (1) above, and with a unique sink, one can partition its vertex set in order to give it the structure of a neural-adapted quiver. This can be done, for example, by setting Iin to be the set of sinks, H to be the non-sink and non-source vertices, andB to be empty. In general, there are many possible partitions, since some sources can be input vertices and some can be bias vertices (keeping in mind that any hidden vertex must admit a path from a input vertex). As a final note, we observe that an acyclic directed graph with a unique sink must be connected. Example G.2. We list examples of neural-adapted quivers:
1. The neural quiver with five layers: • // • // • // • // • // •
•
77??OO__gg
One can see that there is a unique neural-adapted structure on this quiver. Namely, the unique sink is the output vertex, the source on the far left is the sole input vertex, and the source on the bottom is the sole bias vertex. All other vertices are hidden vertices, and each admits a unique path from the input vertex.
2. The no-bias neural quiver with five layers: • // • // • // • // • // •
There is a unique neural-adapted structure on this quiver, and it has the property that the set of bias vertices is empty.
3. A skip-connection neural quiver with five layers: • // • // &&• // • // &&• // •
•
77??OO__gg
Once again, there is a unique neural-adapted structure on this quiver.
Given a neural adapted quiver, one exclusively considers dimension vectors whose value at the bias vertices is equal to 1. Thus, a dimension vector for a neural-adapted quiver amounts to a tuple of nonnegative integers indexed by the vertices in I \B = Iin∪H ∪{iout}. We abbreviate din := ∑ i∈Iin di and dout := diout . Remark G.3. As a directed acyclic graph, a neural-adapted Q has a “topological ordering” of the vertices. In what follows, we will be implicitly be choosing an enumeration {i1, . . . , i|I|} of the set I of vertices that is compatible with the topological ordering. That is, if ij ≺ ij′ in the topological ordering, then j < j′. In particular, i|I| = iout. Furthermore, we will assume all source vertices (that is, those in Irm in ∪B) appear before all non-source vertices. Hence,
{i|Iin|+|B|+1, . . . , i|I| = iout}
is an enumeration of the non-source vertices.
Additionally, whenever necessary, we will implicitly fix an enumeration of the edges {e1, . . . , e|E|}. For example, this occurs in the following definition: Definition G.4. Let W be a representation of a neural-adapted quiver Q with dimension vector d. For each vertex in H ∪ {iout}, set Wi to be the following matrix:
Wi = [We1 . . . Wer ]
where {e1, . . . , er} = t−1(i) is the set of edges whose target is i. Note that the size of Wi is di × (dj1 + · · ·+ djr ). Lemma G.5. Let Q be a quiver9, and fix an enumeration of the edges. Specifying a representation ofQ with dimension vector d is the same as specifying a di× (dj1 + · · ·+ djr ) matrix Wi for every non-source vertex i.
G.2 Q-NEURAL NETWORKS
We now define the notion of a Q-neural network, that is, a neural network whose underlying architecture is specified by a neural-adapted quiver. Definition G.6. Let Q be a neural-adapted quiver. A Q-neural network consists of the following:
1. Hyperparameters. A dimension vector d for the quiver Q. 2. Trainable parameters. A representation W of Q with dimension vector d. 3. Activation functions. Piece-wise differentiable functions ai : Rdi → Rdi for i ∈ H ∪ {iout}. These are grouped into a tuple a.
We denote by Neur(Q,d) the vector space of Q-neural networks layers and dimension vector d, and regard elements therein as pairs (W,a). The neural function (also known as the feedforward function) of a Q-neural network is defined recursively, as explained in the following definition. Definition G.7. Let Q be a neural-adapted quiver. Let (W,a) be a Q-neural network with dimension vector d. For every non-bias vertex i ∈ I , define a function:
fi,(W,a) = fi : Rdin → Rdi
by setting:
• If i is an input vertex, fi is the projection map.
• If i is a bias vertex, fi ≡ 1 is the constant function at 1 ∈ Rdi = R. • For all other vertices, we use the recursive definition: fi(x) = ai (∑
j e→iWe ◦ fj(x) ) The neural function of the neural network (W,a) is defined as:
f(W,a) := fiout : Rdin → Rdout
9This works for any quiver, and in particular for neural-adapted quivers
Note that the neural function is well-defined due to the topological ordering of a neural-adapted quiver, regarded as a connected directed acyclic graph. Indeed, one can implement this definition using an ordering of the vertices compatible with the topological ordering (see Remark G.3). Definition G.8. Let W be a representation of a neural-adapted quiver Q with dimension vector d. Let i be a non-source vertex10 of Q. Let {e1, . . . , er} = t−1(i) be the (ordered) set of edges whose target is i, and let j1 be the source of e1, j2 of e2, and so forth.
1. Set Wi to be the following matrix:
Wi = [We1 . . . Wer ]
Note that the size of Wi is di × (dj1 + · · ·+ djr ).
2. For x ∈ Rdin , set f prei (x) to be the following vector:
f prei (x) = fj1(x)... fjr (x)  . Hence, f prei (x) is a column vector of dimension (dj1 + · · ·+ djr ).
The proof of the following lemma is elementary. Lemma G.9. Let (W,a) be a representation of a neural-adapted quiver Q. For any non-source vertex i ∈ I , we have:
fi,(W,a)(x) = ai ◦Wi ◦ f prei,(W,a)(x)

heading: G.3 SYMMETRY GROUP
text: Throughout this section, we fix a neural-adapted quiver Q with partition I = Iin ∪ B ∪H ∪ {iout} of the vertex set as described above. Definition G.10. The parameter symmetry group for representations of a neural-adapted quiver Q with dimension vector d is:
GLhiddend = ∏ i∈H GLdi .
This group acts on Rep(Q,d) as: g ·W = ( gt(e) We g −1 s(e) ) e
This group acts on Neur(Q,d) as: g · (W,a) = (( gt(e) We g −1 s(e) ) e , (gi ai g −1 i )i ) In both cases, by convention, we have gi = iddi for any non-hidden vertex i. Remark G.11. Following the notation appearing in Definition G.4, we have that g ∈ GLhiddend transforms the matrix Wi corresponding to a vertex i to:
gi Wi  g−1j1 0 · · · 0 0 g−1j2 · · · 0 ... . . . ...
0 0 · · · g−1jr  We have the following generalization of Lemma D.3:
Lemma G.12. For any neural network (W,a) in Neur(Q,d) and any g in GLhiddend , the neural functions of (W,a) and g · (W,a) coincide:
fg·(W,a) = f(W,a) 10The non-source vertices are precisely H ∪ {iout}.
Sketch of proof. It suffices to show that: fi,g·(W,a) = gi ◦ fi,(W,a) for all vertices i, where, by convention, gi = iddi for non-hidden vertices. One proceeds by induction using the topological ordering of the quiver, viewed as a connected directed acyclic graph. The base step is when i is a bias or input vertex. The induction step involves the following calculation:
fi,g·(W,a)(x) = gi ◦ ai ◦ g−1i ∑ j e→i gi ◦We ◦ g−1j ◦ gj ◦ fj,(W,a)(x)  = gi ◦ ai
∑ j e→i We ◦ fj,(W,a)(x)  = gi ◦ fi,(W,a)(x) Alternatively, one can use the matrices Wi and Lemma G.9.
Definition G.13. The stabilizer in GLhiddend of a choice of activation functions a is defined as:
Z(a) = {g = (gi) ∈ GLhiddend : gi ◦ ai ◦ g−1i = ai for every hidden vertex i}.
Note that each ai is a function from Rdi to Rdi , as is each gi, so the equality gi ◦ ai ◦ g−1i = ai is as functions from Rdi to Rdi . As a subgroup of GLhiddend , the stabilizer Z(a) acts on Rep(Q,d).

heading: G.4 THE LOSS FUNCTION
text: For the remainder of this section, we fix a neural-adapted quiverQ and a dimension vector d forQ. Furthermore, fix activation functions a = (ai : Rdi → Rdi), one for each hidden vertex, and one for the output vertex. To a batch of training data X = {(xj , yj)} ⊆ Rdin × Rdout , there is an associated loss function on Rep(Q,d) defined as
L = LX : Rep(Q,d)→ R L(W) = ∑ j C(f(W,a)(xj), yj)
where C : Rdout × Rdout → R is a cost function. Following the formalism of Appendix C, we obtain a gradient descent map γ : Rep(Q,d)→ Rep(Q,d) for any learning rate η > 0.
We demonstrate that the loss function is invariant for the action of the group Z(a). We omit a proof of the following proposition, as it is completely analogous to that of Proposition D.5. Proposition G.14. Let a be activation functions, and let L be the loss function associated to a batch of training data. Then, for all g ∈ Z(a) and W ∈ Rep(Q,d), we have:
L(g ·W) = L(W).
G.5 RADIAL Q-NEURAL NETWORKS
Suppose h = (h1, . . . , hL) is a tuple of functions 11 hi : R → R, and suppose d is a dimension vector for a neural-adapted quiver Q. Consider the tuple of activation functions:
a (n) h = ( h (di) i ) i ,
where i runs over H ∪ {iout}. Any representation W of Q with dimension vector d defines a radial neural network ( W,a
(n) h ) . We write simply (W,h) for this radial neural network and f(W,h) for
its neural function.
The following definition implicitly invokes the topological ordering of a neural-adapted quiver, regarded as a directed acyclic graph. Recall that, as always, the dimension vector of a neural-adapted quiver has value 1 on each bias vertex.
11We assume each hi is piece-wise differentiable and the limit limr→0 hi(r)r exists.
Definition G.15. Given a dimension vector d for a neural-adapted quiver Q, one defines the reduced dimension vector dred as follows. First, dredi = di for every i ∈ Iin ∪ B. Next, for any hidden vertex i, recursively set dredi to be minimum of di and ∑ j∈I→i d red j . Finally, d red out = dout for the output vertex.
There is an inclusion ι : Rep(Q,dred) ↪→ Rep(Q,d).
Definition G.16. Let d be a dimension vector for a neural-adapted quiver Q. Define a subgroup of GLhiddend as follows:
O(d) = ∏ i∈H O(di).
We state the following conjecture:
Conjecture G.17. Let d be a dimension vector fora neural-adapted quiver Q and fix h as above. For any W ∈ Rep(Q,d) there exist:
Q ∈ O(d), R = (R1, . . . , RL) ∈ Rep(Q,dred), and U ∈ Rep(Q,d),
such that:
1. The matrix Ri is upper triangular for any hidden vertex i.
2. The following equality holds: W = Q · ι(R) +U 3. The neural networks defined by W and R have identical neural functions: f(W,h) = f(R,h).
Moreover, we conjecture that a generalization of Algorithm 1 produces R, Q and U from W. We first make the following notes:
• Fix an enumeration {i1, . . . , i|I|} of the set I of vertices that is compatible with the topological ordering, as in Remark G.3. Furthermore, we will assume all source vertices (that is, those in Irm in ∪B) appear before all non-source vertices. Hence,
{i|Iin|+|B|+1, . . . , i|I|−1}
is an enumeration of the hidden vertices.
• Fix an enumeration of the edges of Q. Hence, a representation of Q with a specified dimension vector can be regarded either as a tuple W = (We)e of matrices indexed by the edges, or as a tuple of matrices W = (Wi)i of matrices indexed by the vertices (see Lemma G.5). We use the latter formulation in the algorithm, except for one step.
• It is enough to produce R and Q, as we can subsequently set: U = W −Q ·R.
Algorithm 2: General version of QR Dimensional Reduction (QRDimRed) input : W ∈ Rep(Q,d) output : Q,R Q,R← [ ], [ ] // initialize output matrix lists for j ← |Iin|+ |B|+ 1 to |I| − 1 do // iterate through hidden vertices
i← ij // current vertex if dredi < di then
Qi, Ri ← QR-decomp(Vi, mode = complete) // Vi = Qi ◦ inci ◦Ri else
Qi, Ri ← QR-decomp(Vi) // Vi = Qi ◦Ri end Append Qi to Q Append Ri to R // reduced weights for layer i for e such that s(e) = i do // update next layers
We ←WeQi a end
end Append Wiout to R
return Q,R
aIn terms of indexing-by-vertices, we have that Wt(e) gets updated to the product of Wt(e) and the block diagonal matrix Diag(id, . . . , Qi, . . . , id), with the identity matrix (of the appropriate size) in each block, except for the block corresponding to e, which has Qi. Note that we have fixed an ordering of the edges.

heading: H DISCUSSION
text: In this section, we first consider how our framework specializes to the case of no bias, and then how it generalizes to shifts within the radial functions.

heading: H.1 SPECIAL CASE: NO-BIAS VERSION
text: We now consider neural networks with only linear maps between successive layers, rather than the more general setting of affine maps. In other words, there are no bias vectors.
Let L be a positive integer. The no-bias neural quiver QL is the following quiver with L + 1 vertices:
• // • // • // · · · // • // •
A representation of this quiver with dimension vector n = (n0, . . . , nL) consists of a linear map from Rni−1 to Rni for i = 1, . . . , L; hence Rep(QL,n) = ⊕L i=1 Hom(Rni−1 ,Rni). The corresponding no-bias reduced dimension vector nred = (nred0 , nred1 , . . . , nredL ) is defined recursively by setting nred0 = n0, then n red i = min(ni , n red i−1) for i = 1, . . . , L − 1, and finally nredL = nL. Since nredi ≤ ni for all i, we have an obvious inclusion Rep(QL,nred) ↪→ Rep(QL,n) and identify Rep(QL,nred) with its image in Rep(QL,n). Given functions h = (h1, . . . , hL), one adapts Construction 4.2 to define a radial neural network for every representation of QL, where the trainable parameters define linear maps rather than affine maps.
Proposition H.1. Theorem 4.3 holds with the neural quiver replaced by the no-bias neural quiver QL and the reduced dimension vector replaced by the no-bias reduced dimension vector nred.
We illustrate an example of the reduction for no-bias radial neural networks in Figure 5 (which is the same as Figure 3). Versions of Algorithm 1 and Theorem 4.7 also hold in the no-bias case, where one uses projected gradient descent with respect to the subspace Repint(QL,n) of representations T having the lower left (ni − nredi )× (nredi−1) block of each Ti equal to zero.
Example H.2. We give an example where projected gradient descent does not match usual gradient descent. This example is a simpler version of that appearing in F.4.
Let n = (1, 2, 1) be a dimension vector for the no-bias quiver QL. The space of representations with this dimension vector is 4-dimensional:
Rep(QL,n) = Hom(R,R2)⊕Hom(R2,R) ' R4. We identify a representation
W = ( W0 = [ a b ] , W1 = [c d] ) ∈ Rep(QL, (1, 2, 1))
with the point p = (a, b, c, d) in R4. The action of the orthogonal group O(n) = O(2) on Rep(QL,n) ' R4 is the diagonal action:
Q 7→ [ Q 0 0 Q ] .
To be explicit: if Q = [ cos(θ) − sin(θ) sin(θ) cos(θ) ] is an orthogonal transformation, then Q · (a, b, c, d) =
(a cos(θ)− b sin(θ), a sin(θ) + b cos(θ), c cos(θ)− d sin(θ), c sin(θ) + d cos(θ)). Consider the function12:
L : Rep(QL,n)→ R p = (a, b, c, d) 7→ ac+ bd
By the product rule, we have: ∇pL = (c, d, a, b)
One easily checks that L(Q · p) = L(p) and that∇Q·pL = Q · ∇pL for any Q ∈ O(2).
The reduced dimension vector is nred = (1, 1, 1), so Rep(QL,nred) = Hom(R,R)⊕Hom(R,R) ' R2. The reduced version of L is given by:
Lred : Rep(QL,nred)→ R (a, c) 7→ ac
The QR decomposition of the representation W corresponding to p = (a, b, c, d) is: (a, b, c, d) = Q · (√
a2 + b2 , 0 , ac+ bd√ a2 + b2 , 0
) + ( 0 , 0 ,
−abd+ b2c a2 + b2 , a2d− abc a2 + b2 ) where Q = 1√
a2+b2 [ a −b b a ] .
The interpolating space is the subspace of Rep(QL,n) ' R4 with b = 0. Suppose p′ = (a, 0, c, d) belongs to the interpolating space. Then the gradient is
∇p′L = (c, d, a, 0) 12For W ∈ Rep(QL,n), the neural function of the neural network with affine maps determined by W and identity activation functions is R→ R; x 7→ L(W)x. The function L can appear as a loss function for certain batches of training data and cost function on R.
which does not belong to the interpolating space. So one step of usual gradient descent, with learning rate η > 0 yields: γ : p′ = (a, 0, c, d) 7→ (a− ηc , −ηd , c− ηa , d) On the other hand, one step of projected gradient descent yields:
γproj : p ′ = (a, b, c, d) 7→ (a− ηc , 0 , c− ηa , d)
The difference between the evaluation of L after one step of gradient descent and the evaluation of L after one step of projected gradient descent is:
L(γ(p′))− L(γproj(p′)) = ac− η(a2 + c2 + d2) + η2(ac)− ac− η(a2 + c2)− η2(ac) = ηd2.

heading: H.2 GENERALIZATION: RADIAL NEURAL NETWORKS WITH SHIFTS
text: In this section we consider radial neural networks with an extra trainable parameter in each layer which shifts the radial function. Adding such parameters allows for more flexibility in the model, and (as shown in Theorem H.4) the QR decomposition of Theorem 4.3 holds for such radial neural networks.
Let h : R → R be a function13. For any n ≥ 1 and any t ∈ R, the corresponding shifted radial function on Rnis given by:
h(n,t) : v 7→ h(|v| − t) |v| v.
The following definition is a modification of Definition 3.2. Definition H.3. A radial neural network with shifts consists of the following data:
1. Hyperparameters. A positive integer L and a dimension vector n = (n0, n1, n2, . . . , nL) for the neural quiver QL.
2. Trainable parameters.
(a) A representation W = (W1, . . . ,WL) of the quiverQL with dimension vector n. So, for i = 1, . . . , L, we have a matrix Wi ∈ Hom(R1+ni−1 ,Rni). (b) A vector of shifts t = (t1, t2, . . . , tL) ∈ RL.
3. Radial activation functions. A tuple h = (h1, h2, . . . , hL), where hi : R → R. The activation function in the i-th layer is given by ai = h(ni,ti) : Rni → Rni for i = 1, . . . , L.
The neural function of a radial neural network with shifts is defined as:
f(W,t,h) : Rn0 → RnL ; x 7→ h(nL,tL) ◦ W̃L ◦ · · · ◦ h(n2,t2) ◦ W̃2 ◦ h(n1,t1) ◦ W̃1(x)
where W̃i = Wi ◦ extni−1 : Rni−1 → Rni is the affine map corresponding to Wi. The trainable parameters form the vector space Rep(n) ⊕ RL, and the loss function of a batch of training data is defined as
L = LX : Rep(n)⊕ RL −→ R; (W, t) 7→ ∑ j C(f(W,t,h)(xj), yj)
We have the gradient descent map:
γ : Rep(n)⊕ RL −→ Rep(n)⊕ RL
which updates the entries of both W and t. The group O(n) = O(n1) × · · · × O(nL−1) acts on Rep(n) as usual (see Section 4.1), and on RL trivially. The neural function is unchanged by this action. We conclude that the O(n) action on Rep(n)⊕ RL commutes with gradient descent γ. We now state a generalization of Theorem 4.3 for the case of radial neural networks with shifts. We omit a proof, as it uses the same techniques as the proof of Theorem 4.3.
13We also assume h is piece-wise differentiable and exclude those t for which the limit limr→0 h(r−t)r does not exist.
Theorem H.4. Let n be a dimension vector for QL and fix functions h = (h1, . . . , hL) as above. For any W ∈ Rep(n) there exist:
Q ∈ O(n), R = (R1, . . . , RL) ∈ Rep(nred), and U ∈ Rep(n), such that:
1. The matrices R1, . . . , RL−1 are upper triangular.
2. The following equality holds: (W, t) = Q · (R, t) + (U,0).
3. The neural functions defined by (W, t,h) and (R, t,h) coincide: f(W,t,h) = f(R,t,h).
One can use the output of Algorithm 1 to obtain the Q, R, and U appearing in Theorem H.4. Theorem 4.7 also generalizes to the setting of radial neural networks with shifts, using projected gradient descent with respect to the subspace Repint(n)⊕ RL of Rep(n)⊕ RL.

heading: I CATEGORICAL FORMULATION
text: In this appendix, we summarize a category-theoretic approach toward the main results of the paper. While there is no substantial difference in the proofs, the language of category theory provides conceptual clarity that leads to generalizations of these results. References for category theory include Pierce (1991); Dummit & Foote (2003).

heading: I.1 THE CATEGORY OF QUIVER REPRESENTATIONS
text: In this section, we recall the category of representations of a quiver. Background references include Kirillov Jr (2016); Nakajima et al. (1998).
As in Section 3, let Q = (I, E) be a quiver with source and target maps s, t : E → I , and let d : I → Z≥0, i 7→ di be a dimension vector for Q. Recall that a representation of Q with dimension vector d consists of a tuple A = (Ae)e∈E of linear maps, where
Ae : Rds(e) → Rdt(e) . Let A and B be representations of the quiver Q, with dimension vectors d = dim(A) and k = dim(B). A morphism of representations from A to B consists of the data of a linear map
αi : Rdi → Rki , for every i ∈ I , subject to the condition that the following diagram commutes for every e ∈ E:
Rds(e) Ae //
αs(e)
Rdt(e)
αt(e)
Rks(e) Be // Rkt(e)
The resulting categoryR(Q) is known as the category of representations of Q.
I.2 THE CATEGORY I(QL)
In this section, we define a certain subcategory I(QL) of the category R(QL). Its objects are the same as the objects of R(QL), that is, representations of the neural quiver, while morphisms in I(QL) are given by isometries. Let L be a positive integer, and recall the neural quiver QL from Section 3.2:
• // • // • // · · · // • // •
•
77==aagg
As a reminder, the vertices in the top row are indexed from i = 0 to i = L, and we only consider dimension vectors whose value at the bias vertex is equal to 1. So a dimension vector for QL will refer to a tuple n = (n0, n1, . . . , nL). Recall the isomorphism
Rep(QL,n) ' L⊕ i=1 Hom(R1+ni−1 ,Rni)
from Lemma 3.1. We denote a representation of QL as a tuple W = (Wi)Li=1, where each Wi belongs to Hom(R1+ni−1 ,Rni). Let X = (Xi)Li=1 be a representation of QL with dimension vector m. Tracing through the proof of Lemma 3.1 and the definitions in Section I.1, we see that a morphism α : X→W inR(QL) consists of the following data:
• a linear map αi : Rmi → Rni , for i = 0, 1, . . . , L, and • a scalar αb ∈ R at the bias vertex,
making the following diagram commute, for i = 1, . . . , L:
R1+mi−1 Xi //[ αb 0 0 αi−1 ] Rmi αi
R1+ni−1 Wi // Rni
Definition I.1. We define a subcategory I(QL) ofR(QL) as follows. The objects of I(QL) are the same as the objects ofR(QL), that is, representations ofQL. Let X and W be such representations, with dimension vectors m and n, respectively. A morphism α : X → W in R(QL) belongs to I(QL) if the following hold:
• n0 = m0 and α0 is the identity on Rn0 ,
• nL = mL and αL is the identity on RnL ,
• for i = 1, . . . , L−1, the linear map αi : Rmi → Rni is norm-preserving, i.e. |αi(v)| = |v| for all v ∈ Rmi , and
• αb = 1. Remark I.2. A norm-preserving map is called an isometry, which explains the notation I(QL). Lemma I.3. We have:
1. The category I(QL) is a well-defined subcategory ofRL.
2. Let α : X→W be a morphism in I(QL). For i = 0, 1, . . . , L, the linear map αi : Rmi → Rni is injective.
Proof. The claims follow from the facts that (1) the composition of two norm-preserving maps is norm-preserving, and (2) any linear norm-preserving map is injective.
Definition I.4. Let α : X→W be a morphism in I(QL), and let m = dim(X) and n = dim(W) be the dimension vectors. An orthogonal factorization of αi is an element Q = (Q1, . . . , QL−1) of O(n) = O(n1)× · · · ×O(nL−1) such that
αi = Qi ◦ incmi,ni for i = 1, . . . , L− 1. The correction term corresponding to an orthogonal factorization is:
U = W −Q ·X.
The correction term belongs to Rep(QL,n). Remark I.5. Orthogonal factorizations always exist, since any norm-preserving linear map Rm → Rn can be written as the composition Q ◦ incm,n for some orthogonal Q ∈ O(n).
Remark I.6. Suppose α is a morphism in I(QL). For i ∈ {1, . . . , L − 1}, the map αi is an isomorphism if and only if mi = ni. In this case, the choice of Qi is unique and Ui+1 = 0. Conversely, αi is not an isomorphism if and only if mi < ni. In this case, there are O(ni − mi) choices for Qi.
Fix functions hi : R → R for i = 1, . . . , L. Hence we obtain radial functions h(n)i : Rn → Rn for any n ≥ 1. We group the hi into a tuple h = (h1, . . . , hL). Given h, we attach a neural network (and hence a neural function) to every object in I(QL) as in Construction 4.2. Proposition I.7. Let X and W be a representations ofQL. Suppose there is a morphism in I(QL) from X to W. Then the neural functions of the radial neural networks (X,h) and (W,h) coincide:
f(X,h) = f(W,h).
Sketch of proof. The key is to show that, for i = 1, . . . , L, we have:
h (ni) i ◦ W̃i ◦ [ 1 0 0 αi−1 ] = αi ◦ h(mi)i ◦ X̃i
where W̃i and X̃i are the affine map corresponding to Wi and Xi. The first step in the verification of this identity is to choose an orthogonal factorization of α. The rest of the proof proceeds along the same lines as the proof of Equation E.2 in Section E.2.

heading: I.3 PROJECTED GRADIENT DESCENT: SET-UP
text: In this section, we collect notation necessary to state results about projected gradient descent.
Let m = (m0, . . . ,mL) and n = (n0, . . . , nL) be dimension vectors for QL. We write m n if:
• m0 = n0 and mL = nL, • mi ≤ ni for i = 1, . . . , L− 1.
Consequently, if α : X → W is a morphism in I(QL), then the dimension vectors satisfy dim(X) dim(W). For m n, we make the following abbreviations, for i = 0, 1, . . . , L:
inci = incmi,ni : Rmi ↪→ Rni ĩnci = inc1+mi,1+ni : R1+mi ↪→ R1+ni
πi : Rni Rmi π̃i : R1+ni R1+mi
Using these maps, one defines an inclusion ι : Rep(m) ↪→ Rep(n) taking X = (X1, . . . , XL) ∈ Rep(m) to the representation with ι(X)i = inci ◦Xi ◦ π̃i−1. Recall the functions h = (h1, . . . , hL). As in Construction 4.2, these define activation functions( h (n1) 1 , . . . , h (nL) L ) (resp. ( h (m1) 1 , . . . , h (mL) L ) ) for a representation with dimension n (resp. m).
Finally, we fix a batch of training data X = {(xj , yj)} ⊆ Rn0 × RnL = Rm0 × RmL . Using the activation functions defined by h, we have loss functions on Rep(n) and Rep(m) (see Section 3.4):
Ln : Rep(n)→ R Lm : Rep(m)→ R W 7→ ∑ j C(f(W,h)(xj), yj) X 7→ ∑ j C(f(X,h)(xj), yj)
Fixing a learning rate η > 0, there are resulting gradient descent maps on Rep(n) and Rep(m) given by:
γn : Rep(n)→ Rep(n) γm : Rep(m)→ Rep(m) W 7→W − η · ∇WLn X 7→ X− η · ∇XLm
The verification of the following lemma is analogous to the proof of Part 1 of Lemma F.7.
Lemma I.8. We have that Ln ◦ ι = Lm.

heading: I.4 THE INTERPOLATING SPACE
text: We first define a space that interpolates between Rep(m) and Rep(n). The discussion of this section is completely analogous to that in Sections F.1 and F.2.
Definition I.9. Let Repint(m,n) denote the subspace of Rep(n) consisting of those T = (T1, . . . , TL) ∈ Rep(n) such that, for i = 1, . . . , L, we have:
(idni − inci ◦ πi) ◦ Ti ◦ ĩnci−1 ◦ π̃i−1 = 0.
Just as in Section F.1, the space Repint(m,n) consists of representations T = (T1, . . . , TL) ∈ Rep(n) for which the bottom left (ni −mi) × (1 + nredi−1) block of Ti is zero for each i. Consider the maps:
Rep(m) ι2
-- Repint(m,n)
q2
mm
ι1 -- Rep(n)
q1 mm
• The map ι1 : Repint(m,n) ↪→ Rep(n) is the natural inclusion.
• The map q1 : Rep(n) → Repint(m,n) takes T ∈ Rep(n) to the representation whose i-th slot is
Wi − (idni − inci ◦ πi) ◦Wi ◦ ĩnci−1 ◦ π̃i−1.
• The map ι2 : Rep(m) ↪→ Repint(m,n) is defined by taking X to the representation whose i-th coordinate is inci ◦Xi ◦ π̃i−1.
• The map q2 : Repint(n) Rep(m) is defined by taking T to the representation whose i-th slot is πi ◦ Ti ◦ ĩnci−1.
Definition I.10. The projected gradient descent map on Rep(n) with respect to Repint(m,n) and learning rate η > 0 is defined as:
γn : Rep(n)→ Rep(n) W 7→W − η · i1 ◦ q1(∇WL)
We now state the main result of this appendix. We omit a proof, as it follows the same ideas as the proof of Theorem 4.7 given in Section F.3. The main difference is that all appearances of nred must be replaced by m.
Theorem I.11. Let α : X→W be a morphism in I(QL). Let Q be an orthogonal factorization of α and set U = W −Q ·X.
1. The representation Q−1 ·W = X+Q−1 ·U belongs to Repint(m,n).
2. For any k ≥ 0, we have γkn(W) = Q · γkn(Q−1 ·W).
3. For any k ≥ 0, we have
γkn(Q −1 ·W) = γkm(X) +Q−1 ·U.

heading: I.5 RELATION TO ALGORITHM 1
text: In this final section, we relate the general categorical results of this appendix to Algorithm 1. Recall that, in Section 4.2, we associated a reduced dimension vector nred to any dimension vector n of the neural quiver QL. Proposition I.12. Let W be a representation of QL with dimension vector n. Then W admits a morphism in I(QL) from a representation with dimension vector nred.
Proof. We proceed by induction on the number N of i ∈ {1, . . . , L − 1} such that nredi < ni. If N = 0, there is nothing to show. Otherwise, let j be the smallest element of {1, . . . , L − 1} such that nredj < nj . Then n red j = nj−1 + 1. Let Wj = Q ◦ inc1+nj−1,nj ◦ R be a QR decomposition of W , where Q ∈ O(nj) and R is an upper-triangular nj−1 + 1 by nj−1 + 1 matrix. Consider the representation of QL given by:
X = ( W1 , . . . , Wj−1 , R , Wj+1 ◦ [ 1 0 0 Q ] ◦ inc1+nj−1,nj , Wj+2 , . . . ,WL ) Then α = ( idn1 , . . . , idnj−1 , Q ◦ inc1+nj−1,nj , idnj+1 , . . . , idnL) is a morphism form X to W. The dimension vector of X is (nred0 , . . . , n red j−1, n red j , nj+1, . . . , nL) and has one less coordinate that n not equal to nred, so the induction hypothesis applies.
Consequently, Theorems 4.3 and 4.7 can be viewed as a corollaries of Proposition I.7, Theorem I.11, and Proposition I.12. Remark I.13. Let Q, R and U be the outputs of Algorithm 1 applied to a representation W of dimension vector n. Then R defines a representation of dimension vector nred, and Q defines a morphism in I(QL) from R to W. Indeed, for i = 1, . . . , L, Algorithm 1 provides the equality
Wi ◦ [ 1 0 0 Qi−1 ] ◦ ĩnci−1 = Vi = Qi ◦ inci ◦Ri,
(where Q0 = idn0 = inc0 and QL = idnL = incL), which implies that the following diagram commutes:
R1+n red i−1 Ri //[ 1 0 0 Qi−1 ] ◦ ĩnci−1 Rnredi Qi◦inci
R1+ni−1 Wi // Rni

Review summaries:
The authors make use of a theoretical framework for viewing neural networks as representation of quivers to introduce a reparametrization strategy for neural networks with radial activation functions. This reparametrization is based on a QR decomposition, and leads to a lossless compression of the number of parameters by factoring redundant symmetries of the model. A corresponding gradient descent algorithm is derived on the compressed parameter space corresponding to gradient descent in the original space.

This paper provides a framework to study compression of NN models by analyses of symmetry properties of their parameter spaces. For lossless model compression, an algorithm employing QR decomposition for parameter compression is proposed. In the theoretical analyses, the decomposition of the proposed neural quiver is studied. The proposed methods were experimentally analyses in synthetic datasets.

This paper presents a dimensionality reduction technique that can preserve the model of a neural network by leveraging radial symmetry. Consequently, when the required conditions apply, they also show that training the compressed model is equivalent to training the original model on a particular projection of the parameter space.  The first two sections of the paper are very clear and well written, although some definitions would have helped (more on that later), but the sudden change of pace starting at section 3 makes it difficult to follow if you do not have the same background. I believe that I have captured the gist of their results, which seem to be intuitively correct, but I cannot comment on how they were derived. Hence, what follows is a description in my own words and perspective of what they did.  The authors present a method that would reduce the width of each layer of the neural network to the smallest width of any preceding layer, including the input layer. That is possible in radial neural networks because (1) the activation function involves the entire layer as opposed to each neuron individually and (2) the activation function is symmetric along any direction considered.