Model Compression via Symmetries of the Parameter Space

1 INTRODUCTION . Recent work has shown that representation theory , the formal study of symmetry , provides the foundation for various innovative techniques in deep learning ( Cohen & Welling , 2016 ; Kondor & Trivedi , 2018 ; Ravanbakhsh et al. , 2017 ; Cohen & Welling , 2017 ) . Much of this previous work considers symmetries inherent to the input and output spaces , as well as distributions and functions that respect these symmetries . By contrast , in this paper , we expose a broad class of symmetries intrinsic to the parameter space of the neural networks themselves . We use these parameter space symmetries to devise a model compression algorithm that reduces the widths of the hidden layers , and hence the number of parameters . Unlike representation-theoretic techniques in the setting of equivariant neural networks , our methods are applicable to deep learning models with non-symmetric domains and non-equivariant functions , and hence pertain to some degree to all neural networks . Specifically , we formulate a theoretical framework for neural networks inspired by quiver representation theory , a mathematical field with connections to symplectic geometry and Lie theory ( Kirillov Jr , 2016 ; Nakajima et al. , 1998 ) . This approach builds on that of Armenta & Jodoin ( 2020 ) and of Wood & Shawe-Taylor ( 1996 ) , but is arguably simpler and encapsulates larger symmetry groups . Formally , a quiver is another name for a finite directed graph , and a representation of a quiver is the assignment of a vector space to each vertex and a compatible linear map to each edge . Our starting point is to regard the vector space of parameters for a neural network as a representation of a particular quiver , namely , the neural quiver ( Figure 1 ) . The advantage of this viewpoint is that representations of quivers carry rich symmetry groups via change-of-basis transformations ; such operations can be viewed as symmetries of the neural network parameter space . Moreover , these symmetries may be factored out without affecting the feedforward function , making our method a lossless model compression algorithm ( Serra et al. , 2020 ) . Model compression has become critically important as models have grown to billions of parameters ; with compression , enormous models may be reduced and run on smaller systems with faster inference ( Buciluǎ et al. , 2006 ; Cheng et al. , 2017 ; Frankle & Carbin , 2018 ; Zhang et al. , 2018 ) . Whereas many previous approaches to model compression are based on weight pruning , quantization , matrix factorization , or knowledge distillation , similar to Sourek et al . ( 2020 ) , our approach exploits symmetries of neural network parameter spaces . The size of the parameter space symmetry group is determined by properties of the activation functions . We focus on radial activation functions , as in Weiler & Cesa ( 2019 ) ; Sabour et al . ( 2017 ) ; Weiler et al . ( 2018a ; b ) ; these interact favorably with certain QR decompositions , and , consequently , the model compression is significant compared to the more common pointwise ( or ‘ local ’ ) activations . We refer to neural networks with radial activation functions as radial neural networks . Given a radial neural network , our results produce a new network with fewer neurons in each hidden layer and the same feedforward function , and hence the same loss for any batch of training data . Moreover , the value of the loss function after a step of gradient descent applied to the compressed model is the same as the value of the loss function after a step of projected gradient descent applied to the original model . As we explain below , in projected gradient descent , one subtracts a truncation of the gradient , rather than the full gradient . When the compression is significant enough , the compressed model takes less time per epoch to train and reaches local minima faster . To state these results slightly more precisely , recall that the parameters of a neural network with layer widths ( n0 , n1 , . . . , nL ) consist of an ni× ( ni−1+1 ) matrixWi of weights for each layer i , where we include the bias as an extra column . These are grouped into a tuple W = ( Wi ∈ Rni× ( ni−1+1 ) ) i . We define the reduced widths recursively as nredi = min ( ni , n red i−1 + 1 ) for i = 1 , . . . , L− 1 , with nred0 = n0 and n red L = nL . Note that n red i ≤ ni for all i. Theorem 1.1 ( Informal version of Theorems 4.3 and 4.7 ) . Suppose a neural network has L layers with widths ( n0 , . . . , nL ) , parameters W , and radial activation functions . Let fW : Rn0 → RnL be the feedforward function of the network . 1 . There exists an algorithm to produce a reduced radial neural network with layer widths ( n0 , n red 1 , . . . , n red L−1 , nL ) , parameters R , and the same feedforward function fR = fW . 2 . Training fR with gradient descent is an equivalent optimization problem to training fW with projected gradient descent . This theorem can be interpreted as a model compression result : the reduced ( or compressed ) neural network R has the same accuracy as the original neural network W , and there is an explicit relationship between the gradient descent optimization problems for the two neural networks . This result is not just theoretical ; it emerges from a practical and efficient algorithm depending on successive QR decompositions . We describe this procedure below ( Algorithm 1 ) and implement it in Python . To summarize , our contributions are as follows : 1 . A theoretical framework for neural networks based on quiver representation theory ; 2 . A QR decomposition for radial neural networks ; 3 . An implementation of a lossless model compression algorithm for radial neural networks ; 4 . A theorem relating gradient descent optimization of the original and compressed networks . We view this work as a step in the direction of improving learning algorithms by exploiting symmetry inherent to neural network parameter spaces . As such , we expect our framework and results to generalize in several ways , including : ( 1 ) further reductions of the hidden widths , ( 2 ) incorporating certain non-radial activation functions , ( 3 ) encapsulating networks beyond MLPs , such as convolutional , recurrent , and graph neural networks , ( 4 ) integration of regularization techniques . We include a detailed discussion of the limitations of our results , as well as future directions , in Section 6 . 2 RELATED WORK . Quiver Representation Theory and Neural Networks . Armenta & Jodoin ( 2020 ) give an approach to understanding neural networks in terms of quiver representations . Our work generalizes their approach as it ( 1 ) accommodates both pointwise and non-pointwise activation functions , ( 2 ) taps into larger symmetry groups , and ( 3 ) connects more naturally to gradient descent . Jeffreys & Lau ( 2021 ) and Manin & Marcolli ( 2020 ) also place quivers in the context of neural networks ; our approach is inspired by similar algebro-geometric and categorical perspectives , but differs in our emphasis on practical consequences for optimization techniques at the core of machine learning . These works have a number of precursors . One is the study of the “ non-negative homogeneity ” ( also known as “ positive scaling invariance ” ) property of ReLU activation functions ( Dinh et al. , 2017 ; Neyshabur et al. , 2015 ; Meng et al. , 2019 ) , which is a special case of the symmetry studied in this paper . Wood & Shawe-Taylor ( 1996 ) regard layers in a neural network as representations of finite groups and consider only pointwise activation functions ; by contrast , our framework captures Lie groups as well as non-pointwise activations . Our quiver approach to neural networks shares parallels with the “ algebraic neural networks ” of Parada-Mayorga & Ribeiro , and special cases of their formalism amount to representations of quivers over base rings beyond R , such as the ring of polynomials . In a somewhat different data-analytic context , Chindris & Kline ( 2021 ) use quiver representation theory in order to untangle point clouds ; though they do not use neural networks . Equivariant Neural Networks . Previously , representation theory has been used to design neural networks which incorporate symmetry as an inductive bias . A variety of architectures such as G-convolution , steerable CNN , and Clebsch-Gordan networks are constrained by various weightsharing schemes to be equivariant or invariant to various symmetry groups ( Cohen et al. , 2019 ; Weiler & Cesa , 2019 ; Cohen & Welling , 2016 ; Chidester et al. , 2018 ; Kondor & Trivedi , 2018 ; Bao & Song , 2019 ; Worrall et al. , 2017 ; Cohen & Welling , 2017 ; Weiler et al. , 2018b ; Dieleman et al. , 2016 ; Lang & Weiler , 2021 ; Ravanbakhsh et al. , 2017 ) . Our approach , in contrast , does not rely on symmetry of the input domain , output space , or mapping . Rather , our method exploits symmetry of the parameter space and thus applies more generally to domains with no obvious symmetry . From the point of view of model compression , equivariant networks do achieve reduction in the number of trainable parameters through weight-sharing for fixed hidden dimension widths ; however , in practice , they may use larger layer widths and consequently have larger memory requirements than non-equivariant models . Sampling or summing over large symmetry groups may make equivariant models computationally slow as well ( Finzi et al. , 2020 ; Kondor & Trivedi , 2018 ) . Model Compression and Weight Pruning . A major goal in machine learning is to find methods for compressing models in order to reduce the number of trainable parameters , decrease memory usage , or accelerate inference and training ( Cheng et al. , 2017 ) . Our approach toward this goal differs significantly from most existing methods in that it is based on the inherent symmetry of neural network parameter spaces . One prior model compression method is weight pruning , which removes redundant , small , or unnecessary weights from a network with little loss in accuracy ( Han et al. , 2015 ; Blalock et al. , 2020 ; Karnin , 1990 ) . Recent work has shown pruning can be done during training by identifying and removing weights of less relevance and reverting other weights to earlier values ( Frankle & Carbin , 2018 ) . Related work shows effective pruning may be done at initialization ( Lee et al. , 2019 ; Wang et al. , 2020 ) . Gradient-based pruning identifies low saliency weights by estimating the increase in loss resulting from their removal ( LeCun et al. , 1990 ; Hassibi & Stork , 1993 ; Dong et al. , 2017 ; Molchanov et al. , 2016 ) . A complementary approach is quantization , in which the bit depth of weights is decreased ( Wu et al. , 2016 ; Howard et al. , 2017 ; Gong et al. , 2014 ) . Knowledge distillation works by training a small model to mimic the performance of a larger model or ensemble of models ( Buciluǎ et al. , 2006 ; Hinton et al. , 2015 ; Ba & Caruana , 2013 ) . Matrix Factorization methods replace fully connected layers with lower rank or sparse factored tensors ( Cheng et al. , 2015a ; b ; Tai et al. , 2015 ; Lebedev et al. , 2014 ; Rigamonti et al. , 2013 ; Lu et al. , 2017 ) and can often be applied before training . Our method involves a generalized QR decomposition , which is a type of matrix factorization ; however , rather than aim for a rank reduction of linear layers , we leverage this decomposition in order to reduce hidden layer widths via change-of-basis operations on the hidden representations . Closest to our method are lossless compression methods . Serra et al . ( 2021 ; 2020 ) identify and remove stable neurons in ReLU networks . Sourek et al . ( 2020 ) also exploit symmetry in parameter space to remove redundant neuron . However , their symmetries are induced from permutation equivariance ; ours follow from the symmetries of the radial activation .

Summary: This paper presents a dimensionality reduction technique that can preserve the model of a neural network by leveraging radial symmetry. Consequently, when the required conditions apply, they also show that training the compressed model is equivalent to training the original model on a particular projection of the parameter space.  The first two sections of the paper are very clear and well written, although some definitions would have helped (more on that later), but the sudden change of pace starting at section 3 makes it difficult to follow if you do not have the same background. I believe that I have captured the gist of their results, which seem to be intuitively correct, but I cannot comment on how they were derived. Hence, what follows is a description in my own words and perspective of what they did.  The authors present a method that would reduce the width of each layer of the neural network to the smallest width of any preceding layer, including the input layer. That is possible in radial neural networks because (1) the activation function involves the entire layer as opposed to each neuron individually and (2) the activation function is symmetric along any direction considered.