Model Compression via Symmetries of the Parameter Space

1 INTRODUCTION . Recent work has shown that representation theory , the formal study of symmetry , provides the foundation for various innovative techniques in deep learning ( Cohen & Welling , 2016 ; Kondor & Trivedi , 2018 ; Ravanbakhsh et al. , 2017 ; Cohen & Welling , 2017 ) . Much of this previous work considers symmetries inherent to the input and output spaces , as well as distributions and functions that respect these symmetries . By contrast , in this paper , we expose a broad class of symmetries intrinsic to the parameter space of the neural networks themselves . We use these parameter space symmetries to devise a model compression algorithm that reduces the widths of the hidden layers , and hence the number of parameters . Unlike representation-theoretic techniques in the setting of equivariant neural networks , our methods are applicable to deep learning models with non-symmetric domains and non-equivariant functions , and hence pertain to some degree to all neural networks . Specifically , we formulate a theoretical framework for neural networks inspired by quiver representation theory , a mathematical field with connections to symplectic geometry and Lie theory ( Kirillov Jr , 2016 ; Nakajima et al. , 1998 ) . This approach builds on that of Armenta & Jodoin ( 2020 ) and of Wood & Shawe-Taylor ( 1996 ) , but is arguably simpler and encapsulates larger symmetry groups . Formally , a quiver is another name for a finite directed graph , and a representation of a quiver is the assignment of a vector space to each vertex and a compatible linear map to each edge . Our starting point is to regard the vector space of parameters for a neural network as a representation of a particular quiver , namely , the neural quiver ( Figure 1 ) . The advantage of this viewpoint is that representations of quivers carry rich symmetry groups via change-of-basis transformations ; such operations can be viewed as symmetries of the neural network parameter space . Moreover , these symmetries may be factored out without affecting the feedforward function , making our method a lossless model compression algorithm ( Serra et al. , 2020 ) . Model compression has become critically important as models have grown to billions of parameters ; with compression , enormous models may be reduced and run on smaller systems with faster inference ( Buciluǎ et al. , 2006 ; Cheng et al. , 2017 ; Frankle & Carbin , 2018 ; Zhang et al. , 2018 ) . Whereas many previous approaches to model compression are based on weight pruning , quantization , matrix factorization , or knowledge distillation , similar to Sourek et al . ( 2020 ) , our approach exploits symmetries of neural network parameter spaces . The size of the parameter space symmetry group is determined by properties of the activation functions . We focus on radial activation functions , as in Weiler & Cesa ( 2019 ) ; Sabour et al . ( 2017 ) ; Weiler et al . ( 2018a ; b ) ; these interact favorably with certain QR decompositions , and , consequently , the model compression is significant compared to the more common pointwise ( or ‘ local ’ ) activations . We refer to neural networks with radial activation functions as radial neural networks . Given a radial neural network , our results produce a new network with fewer neurons in each hidden layer and the same feedforward function , and hence the same loss for any batch of training data . Moreover , the value of the loss function after a step of gradient descent applied to the compressed model is the same as the value of the loss function after a step of projected gradient descent applied to the original model . As we explain below , in projected gradient descent , one subtracts a truncation of the gradient , rather than the full gradient . When the compression is significant enough , the compressed model takes less time per epoch to train and reaches local minima faster . To state these results slightly more precisely , recall that the parameters of a neural network with layer widths ( n0 , n1 , . . . , nL ) consist of an ni× ( ni−1+1 ) matrixWi of weights for each layer i , where we include the bias as an extra column . These are grouped into a tuple W = ( Wi ∈ Rni× ( ni−1+1 ) ) i . We define the reduced widths recursively as nredi = min ( ni , n red i−1 + 1 ) for i = 1 , . . . , L− 1 , with nred0 = n0 and n red L = nL . Note that n red i ≤ ni for all i. Theorem 1.1 ( Informal version of Theorems 4.3 and 4.7 ) . Suppose a neural network has L layers with widths ( n0 , . . . , nL ) , parameters W , and radial activation functions . Let fW : Rn0 → RnL be the feedforward function of the network . 1 . There exists an algorithm to produce a reduced radial neural network with layer widths ( n0 , n red 1 , . . . , n red L−1 , nL ) , parameters R , and the same feedforward function fR = fW . 2 . Training fR with gradient descent is an equivalent optimization problem to training fW with projected gradient descent . This theorem can be interpreted as a model compression result : the reduced ( or compressed ) neural network R has the same accuracy as the original neural network W , and there is an explicit relationship between the gradient descent optimization problems for the two neural networks . This result is not just theoretical ; it emerges from a practical and efficient algorithm depending on successive QR decompositions . We describe this procedure below ( Algorithm 1 ) and implement it in Python . To summarize , our contributions are as follows : 1 . A theoretical framework for neural networks based on quiver representation theory ; 2 . A QR decomposition for radial neural networks ; 3 . An implementation of a lossless model compression algorithm for radial neural networks ; 4 . A theorem relating gradient descent optimization of the original and compressed networks . We view this work as a step in the direction of improving learning algorithms by exploiting symmetry inherent to neural network parameter spaces . As such , we expect our framework and results to generalize in several ways , including : ( 1 ) further reductions of the hidden widths , ( 2 ) incorporating certain non-radial activation functions , ( 3 ) encapsulating networks beyond MLPs , such as convolutional , recurrent , and graph neural networks , ( 4 ) integration of regularization techniques . We include a detailed discussion of the limitations of our results , as well as future directions , in Section 6 . 2 RELATED WORK . Quiver Representation Theory and Neural Networks . Armenta & Jodoin ( 2020 ) give an approach to understanding neural networks in terms of quiver representations . Our work generalizes their approach as it ( 1 ) accommodates both pointwise and non-pointwise activation functions , ( 2 ) taps into larger symmetry groups , and ( 3 ) connects more naturally to gradient descent . Jeffreys & Lau ( 2021 ) and Manin & Marcolli ( 2020 ) also place quivers in the context of neural networks ; our approach is inspired by similar algebro-geometric and categorical perspectives , but differs in our emphasis on practical consequences for optimization techniques at the core of machine learning . These works have a number of precursors . One is the study of the “ non-negative homogeneity ” ( also known as “ positive scaling invariance ” ) property of ReLU activation functions ( Dinh et al. , 2017 ; Neyshabur et al. , 2015 ; Meng et al. , 2019 ) , which is a special case of the symmetry studied in this paper . Wood & Shawe-Taylor ( 1996 ) regard layers in a neural network as representations of finite groups and consider only pointwise activation functions ; by contrast , our framework captures Lie groups as well as non-pointwise activations . Our quiver approach to neural networks shares parallels with the “ algebraic neural networks ” of Parada-Mayorga & Ribeiro , and special cases of their formalism amount to representations of quivers over base rings beyond R , such as the ring of polynomials . In a somewhat different data-analytic context , Chindris & Kline ( 2021 ) use quiver representation theory in order to untangle point clouds ; though they do not use neural networks . Equivariant Neural Networks . Previously , representation theory has been used to design neural networks which incorporate symmetry as an inductive bias . A variety of architectures such as G-convolution , steerable CNN , and Clebsch-Gordan networks are constrained by various weightsharing schemes to be equivariant or invariant to various symmetry groups ( Cohen et al. , 2019 ; Weiler & Cesa , 2019 ; Cohen & Welling , 2016 ; Chidester et al. , 2018 ; Kondor & Trivedi , 2018 ; Bao & Song , 2019 ; Worrall et al. , 2017 ; Cohen & Welling , 2017 ; Weiler et al. , 2018b ; Dieleman et al. , 2016 ; Lang & Weiler , 2021 ; Ravanbakhsh et al. , 2017 ) . Our approach , in contrast , does not rely on symmetry of the input domain , output space , or mapping . Rather , our method exploits symmetry of the parameter space and thus applies more generally to domains with no obvious symmetry . From the point of view of model compression , equivariant networks do achieve reduction in the number of trainable parameters through weight-sharing for fixed hidden dimension widths ; however , in practice , they may use larger layer widths and consequently have larger memory requirements than non-equivariant models . Sampling or summing over large symmetry groups may make equivariant models computationally slow as well ( Finzi et al. , 2020 ; Kondor & Trivedi , 2018 ) . Model Compression and Weight Pruning . A major goal in machine learning is to find methods for compressing models in order to reduce the number of trainable parameters , decrease memory usage , or accelerate inference and training ( Cheng et al. , 2017 ) . Our approach toward this goal differs significantly from most existing methods in that it is based on the inherent symmetry of neural network parameter spaces . One prior model compression method is weight pruning , which removes redundant , small , or unnecessary weights from a network with little loss in accuracy ( Han et al. , 2015 ; Blalock et al. , 2020 ; Karnin , 1990 ) . Recent work has shown pruning can be done during training by identifying and removing weights of less relevance and reverting other weights to earlier values ( Frankle & Carbin , 2018 ) . Related work shows effective pruning may be done at initialization ( Lee et al. , 2019 ; Wang et al. , 2020 ) . Gradient-based pruning identifies low saliency weights by estimating the increase in loss resulting from their removal ( LeCun et al. , 1990 ; Hassibi & Stork , 1993 ; Dong et al. , 2017 ; Molchanov et al. , 2016 ) . A complementary approach is quantization , in which the bit depth of weights is decreased ( Wu et al. , 2016 ; Howard et al. , 2017 ; Gong et al. , 2014 ) . Knowledge distillation works by training a small model to mimic the performance of a larger model or ensemble of models ( Buciluǎ et al. , 2006 ; Hinton et al. , 2015 ; Ba & Caruana , 2013 ) . Matrix Factorization methods replace fully connected layers with lower rank or sparse factored tensors ( Cheng et al. , 2015a ; b ; Tai et al. , 2015 ; Lebedev et al. , 2014 ; Rigamonti et al. , 2013 ; Lu et al. , 2017 ) and can often be applied before training . Our method involves a generalized QR decomposition , which is a type of matrix factorization ; however , rather than aim for a rank reduction of linear layers , we leverage this decomposition in order to reduce hidden layer widths via change-of-basis operations on the hidden representations . Closest to our method are lossless compression methods . Serra et al . ( 2021 ; 2020 ) identify and remove stable neurons in ReLU networks . Sourek et al . ( 2020 ) also exploit symmetry in parameter space to remove redundant neuron . However , their symmetries are induced from permutation equivariance ; ours follow from the symmetries of the radial activation .

Summary: The authors make use of a theoretical framework for viewing neural networks as representation of quivers to introduce a reparametrization strategy for neural networks with radial activation functions. This reparametrization is based on a QR decomposition, and leads to a lossless compression of the number of parameters by factoring redundant symmetries of the model. A corresponding gradient descent algorithm is derived on the compressed parameter space corresponding to gradient descent in the original space.