The Expressive Power of Deep Neural Networks with Circulant Matrices

1 INTRODUCTION . Recent progress in deep neural networks came at the cost of an important increase of model sizes . Nowadays , state-of-the-art architectures for common tasks such as object recognition typically have tens of millions of parameters ( He et al. , 2016 ) and up to a billion parameters in some cases ( Dean et al. , 2012 ) . Best performing ( ensemble ) models typically combine dozens of such models , and their size can quickly add up to ten or twenty gigabytes . Large models are often more accurate , but training them requires time and large amounts of computational resources . Even when they are trained , they remain difficult to deploy , especially on mobile devices where memory or computational power is limited . In linear algebra , it is common to exploit structural properties of matrices to speedup computations , or reduce memory usage . Cheng et al . ( 2015 ) have applied this principle in the context of deep neural networks , and proposed a network architecture in which large unstructured weight matrices have been replaced with more compact matrices with a circulant structure . Since any n-by-n circulant matrix can be represented in memory using only a vector of dimension n , the change resulted in a drastic reduction of the model size ( from 230MB to 21MB ) . Furthermore , Cheng et al . have shown empirically that their network architecture can be almost as accurate as the original network . Moczulski et al . ( 2015 ) have proposed a more principled approach leveraging a result by Huhtanen & Perämäki ( 2015 ) stating that any matrix A ∈ Cn×n can be decomposed into 2n − 1 diagonal and circulant matrices . They use this result to design Deep diagonal-circulant ReLU networks . However their experiments show good results even with a small number of factors ( down to 2 factors ) , suggesting that Deep diagonal-circulant ReLU networks can achieve good approximation error , even with few factors . In this paper , we bridge the gap between the good empirical results observed by Moczulski et al . ( 2015 ) , and the theoretical approximation capabilities of Deep diagonal-circulant ReLU networks . We prove that Deep diagonal-circulant ReLU networks with bounded width and small depth can approximate any dense neural network . We obtain this result by showing that any matrix A can be decomposed into 4k + 1 diagonal and circulant matrices where k is the rank of the matrix A . In practice , this result is more useful than the one by Huhtanen & Perämäki since one can rely on a low rank SVD decomposition of A while controlling the approximation error . In addition to this theoretical contribution , we also conduct thorough experiments on synthetic and real datasets . In accordance with the theory , our experiments demonstrate that we can easily tradeoff accuracy for model size by adjusting the number of factors in the matrix decomposition . Finally we evaluate the applicability of this approach on state-of-the-art neural network architectures trained for video classification on the Youtube-8m Video dataset ( over 1TB of training data ) . This experiment demonstrates that Deep diagonal-circulant ReLU networks can be used to train more compact neural networks on large scale , real world scenarios . 2 RELATED WORK . A variety of techniques have been proposed to build more compact deep learning models . A first category of techniques aims at compressing a trained network into a smaller model , without compromising the accuracy . For example model distillation ( Hinton et al. , 2015 ) is two step training procedure : first a large model is trained to be as accurate as possible , second a more compact model is trained to approximate the first one . Other approaches have focused on compressing the network by reducing the memory , at the level of individual weights , ( for example , using weight quantization ( Han et al. , 2016 ) or parameter pruning ) or at the level of weight matrices , using low rank decomposition of the original weight matrix ( Sainath et al. , 2013 ) or using sparse representations ( Collins & Kohli , 2014 ; Dai et al. , 2018 ; Liu et al. , 2015 ) . Instead of compressing the network a posteriori , several researchers have focused on designing models that are compact by design . This approach has several benefits , but most importantly , it reduces memory footprint , both required during training and inference . Chen et al . ( 2015 ) have proposed to compress weight matrices by using hashing functions to map several matrix coefficients into the same memory cell . The techniques works well in theory , but suffers from poor performance on modern GPU devices due to irregular access patterns . In their paper , Cheng et al . ( 2015 ) observed that fully connected layers ( which typically occupy 90 % 1 . of the total number of weights ) are often used to perform simple dimensionality reduction operation between layers of different dimension . The idea of replacing large weight matrices from fully connected layers with more compact circulant layers comes from a result by Hinrichs & Vybı́ral ( 2011 ) that have demonstrated that circulant matrices can be use to approximate the Johson-Lindenstrauss transform , often used to perform dimensionality reduction . Building on this result Cheng et al . proposed to replace the weight matrix of a fully connected layer by a circulant matrix initialized with random weights . The resulting models achieve good accuracy , with the random circulant matrix , but even better when the weights of the circulant matrix are trained with the rest of the network using a gradient based optimization algorithm . This suggests that such layers , often perform more than simple random projections , and that more expressive fully connected layers are beneficial to the overall accuracy of the model . Fortunately , more general linear transforms can also be described using circulant matrices or other structured matrices , at the cost of using more of them . Müller-Quade et al . ( 1998 ) and Schmid et al . ( 2000 ) have demonstrated this formally by showing that any matrix can be decomposed into the product of diagonal and circulant matrices , and Moczulski et al . ( 2015 ) have proposed a compact neural network architecture based on this decomposition that exhibit good accuracy in practice . Other researchers have investigated using alternative structures such as Toeplitz ( Sindhwani et al. , 2015 ) , Vandermonde ( Sindhwani et al. , 2015 ) or Fastfood transforms ( Yang et al. , 2015 ) . Despite demonstrating good empirical results , there have been 1In network such as AlexNet , the last 3 fully connected layers use 58M out of the 62M total trainable parameters . little theoretical insight to explain the good approximation capabilities of deep neural networks based on structured matrices . Barron ( 1993 ) presented the universal approximation theorem which states that any neural network with at least 1 hidden layer and sigmoid non linearity can approximate any function . However , the theorem by Barron ( 1993 ) does not bound the width of the neural network and does not consider the training procedure . Since then , substantial theoretical work has been done to evaluate the expressiveness of a neural network as a function of the width ( i.e . the number of neurons ) and the depth of the network Arora et al . ( 2018 ) ; Mhaskar et al . ( 2017 ) ; Lin et al . ( 2017 ) ; Poole et al . ( 2016 ) ; Raghu et al . ( 2016 ) ; Telgarsky ( 2016 ) ; Mhaskar & Poggio ( 2016 ) . In 2000 , Hanin ( 2017 ) have investigated the approximation capabilities of neural networks with ReLU activations and demonstrated that such networks can approximate any function . More recently , Zhao et al . ( 2017 ) have provided a theoretical study of Deep diagonal-circulant ReLU networks and demonstrated that 2-layers networks of unbounded width are universal approximators . However , these results are of limited interest because the networks used in practice are of bounded width . Unfortunately , nothing is known about the theoretical properties of Deep diagonal-circulant ReLU networks in this case . 3 BUILDING COMPACT DEEP NEURAL NETWORKS USING CIRCULANT MATRICES . 3.1 PRELIMINARIES ON CIRCULANT MATRICES . A n-by-n circulant matrix C is a special kind of Toeplitz matrix where each row is a cyclic right shift of the previous one as illustrated below . C = circ ( c ) = c0 cn−1 cn−2 . . . c1 c1 c0 cn−1 c2 c2 c1 c0 c3 ... . . . ... cn−1 cn−2 cn−3 c0 Despite their rigorous structure , circulant matrices are expressive enough to model a variety of linear transforms such as random projections ( Hinrichs & Vybı́ral , 2011 ) and when they are combined together with diagonal matrices , they can be used to represent an arbitrary transform ( Schmid et al. , 2000 ) . Circulant matrices also exhibit several properties that are interesting from a computational perspective . First , a circulant n-by-n matrix C can be represented using only n coefficients . Thus , it is far more compact that a full matrix that requires n2 coefficient . Second , the product between a circulant matrix C and a vector x can be simplified to a simple element-wise product between the vector c and x in the Fourier domain ( which is generally performed efficiently on GPU devices ) . This results in a complexity reduced from O ( n2 ) to O ( nlog ( n ) ) . In their paper , Huhtanen & Perämäki ( 2015 ) have demonstrated that any matrix A ∈ Cn×n can be approximated with an arbitrary precision by a product of circulant and diagonal matrices : Theorem 1 . ( Huhtanen & Perämäki , 2015 ) For any given matrix A ∈ Cn×n , let p be the smallest integer such that A = ∑p i=1DiS i−1 where D1 . . . Dp are diagonal matrices . Then for any > 0 , for any matrix norm ‖·‖ , there exists a sequence of matrices B1 . . . B2n−1 where Bi is a circulant matrix if i is odd , and a diagonal matrix otherwise , such that ‖B1B2 . . . B2n−1 −A‖ < , and where S = circ ( 0 , 1 , 0 , . . . , 0 ) Because of their interesting properties , several researchers have considered circulant matrices as a replacement from full weight matrices inside neural networks . 3.2 THEORETICAL PROPERTIES OF DEEP DIAGONAL-CIRCULANT RELU NETWORKS . There has already been some recent theoretical work on Deep diagonal-circulant ReLU networks , in which 2-layer networks of unbounded width where shown to be universal approximators . These results are of limited interest , because the networks used in practice are of bounded width . Unfortunately , nothing is known about the theoretical properties of Deep diagonal-circulant ReLU networks in this case . In particular , the following questions remained unanswered up to now : Are Deep diagonal-circulant ReLU networks with bounded width universal approximators ? What kind of functions can Deep diagonal-circulant ReLU networks with bounded-width and small depth approximate ? In this section , we first define formally diagonal-circulant ReLU networks , and then provide a theoretical analysis of their approximation capabilities . Definition 1 ( Deep ReLU networks ) . With ReLU ( x ) = max ( 0 , x ) , let fA , b ( x ) = ReLU ( Ax+ b ) for any matrices A ∈ Cn×n , and any b ∈ Rn . A Deep ReLU network is a function fAl , bl ◦ . . . ◦ fA1 , b1 , where A1 . . . Al are arbitrary n× n matrices and b1 . . . bl ∈ Cn and where l and n are the depth and the width of the network respectively . As in Moczulski et al . ( 2015 ) , Deep diagonal-circulant ReLU networks can be defined as follows : Definition 2 ( Deep diagonal-circulant ReLU networks ) . A Deep diagonal-circulant ReLU network is a function fDlCl , bl ◦ . . . ◦ fD1C1 , b1 where D1 . . . Dl ∈ Cn×n are diagonal matrices , C1 . . . Cl ∈ Cn×n are circulant matrices , and where l and n are the depth and the width of the network respectively . To show that bounded-width Deep diagonal-circulant ReLU networks are universal approximators , we first need a proposition relating standard deep neural networks to Deep diagonal-circulant ReLU networks . Proposition 2 . Let N : Rn → Rn be a deep ReLU networks of width n and depth l , and let X ⊂ Rn be a compact set . For any > 0 , there exists a deep diagonal-circulant ReLU network N ′ of width n and of depth ( 2n− 1 ) l such that ‖N ( x ) −N ′ ( x ) ‖2 < for all x ∈ X . We can now state the universal approximation corrolary : Corrolary 1 . Bounded depth Deep diagonal-circulant ReLU networks are universal approximators on any compact set X . Proof . Proposition 2 shows that bounded-width Deep diagonal-circulant ReLU networks can approximate any Deep ReLU network . It has been shown recently in Hanin ( 2017 ) that bounded-width deep ReLU networks are universal approximators . Together , these two results concludes the proof . It is important to remark that Deep diagonal-circulant ReLU networks are not necessarily more compact than Deep ReLU networks . Indeed , consider a n-wide Deep ReLU network with l layers having ln2 weights . The previous corollary tells us that this network can be decomposed in a Deep ReLU networks involving l ( 2n−1 ) matrices , i.e . 2ln ( 2n− 1 ) weights . Despite the lack of theoretical guarantees a number of work provided empirical evidence that bounded width and small depth Deep diagonal-circulant ReLU networks result in good performance ( e.g . Moczulski et al . ( 2015 ) ; Araujo et al . ( 2018 ) ; Cheng et al . ( 2015 ) ) . The following theorem studies the approximation properties of these small depth networks . Proposition 3 . Let N : fAl , bl ◦ . . . ◦ fA1 , b1 be a deep ReLU network of width n and depth l , such that each matrix Ai is of rank ki , where ki divides n. Let X ⊂ Rn be a compact set . For any > 0 , there exists a deep diagonal-circulant ReLU network N ′ of width n and of depth ( ∑n i=1 ( 4ki + 1 ) ) l such that ‖N ( x ) −N ′ ( x ) ‖2 < for all x ∈ X . This result generalizes Proposition 2 , showing that a Deep diagonal-circulant ReLU networks of bounded width and small depth can approximate a deep ReLU network in which the dense matrices are of low rank . Note in the proposition , we require that ki divides n. We conjecture that the proposition holds even without this condition , but we were not able to prove it . Finally , what if we choose to use small depth network to approximate deep ReLU networks where matrices are not of low rank ? To answer this question , we first need to show the negative impact of replacing matrices by their low rank approximators in neural networks : Proposition 4 . Let N = fAl , bl ◦ . . . ◦ fA1 , b1 be a deep ReLU network , where Ai ∈ Cn×n , bi ∈ Cn for all i ∈ [ l ] . Let Āi be the matrix obtained by a SVD approximation of rank k of matrix Ai . Let σi , j is the jth singular value of Ai . Define N̄ = fĀl , bl ◦ . . . ◦ fĀ1 , b1 . Then , for any x ∈ C n , we have ∥∥N ( x ) − N̄ ( x ) ∥∥ ≤ ( σlmax,1−1 ) Rσmax , k σmax,1−1 where R is an upper bound on norm of the output of any layer in N , and σmax , j = maxi σi , j . Basically , this proposition shows that we can approximate matrices in a neural network by low rank matrices , and control the approximation error . In general , the term σlmax,1 could seem large , but in practice , it is likely that most singular values in deep neural network are small in order to avoid divergent behaviors . We can now prove the result on Deep diagonal-circulant ReLU networks : Corrolary 2 . Consider any deep ReLU network N = fAl , bl ◦ . . . ◦ fA1 , b1 of depth l and width n. Let σmax , j = maxi σi , j where σi , j is the jth singular value of Ai . Let X ⊂ Rn be a compact set . For any k dividing n , there exists a deep diagonal-circulant ReLU network N ′ = fDmCm , b′l ◦ . . . ◦ fD1C1 , b′1 of width n and of depth m = 4 ( k + 1 ) n , such that for any x ∈ X , ‖N ( x ) −N ′ ( x ) ‖ < ( σ l max,1−1 ) Rσmax , k σmax,1−1 , where

Summary: The paper proposes using structured matrices, specifically circulant and diagonal matrices, to speed up computation and reduce memory requirements in NNs. The idea has been previously explored by a number of papers, as described in the introduction and related work.  The main contribution of the paper is to do some theoretical analysis, which is interesting but of uncertain impact.