The Expressive Power of Deep Neural Networks with Circulant Matrices

1 INTRODUCTION . Recent progress in deep neural networks came at the cost of an important increase of model sizes . Nowadays , state-of-the-art architectures for common tasks such as object recognition typically have tens of millions of parameters ( He et al. , 2016 ) and up to a billion parameters in some cases ( Dean et al. , 2012 ) . Best performing ( ensemble ) models typically combine dozens of such models , and their size can quickly add up to ten or twenty gigabytes . Large models are often more accurate , but training them requires time and large amounts of computational resources . Even when they are trained , they remain difficult to deploy , especially on mobile devices where memory or computational power is limited . In linear algebra , it is common to exploit structural properties of matrices to speedup computations , or reduce memory usage . Cheng et al . ( 2015 ) have applied this principle in the context of deep neural networks , and proposed a network architecture in which large unstructured weight matrices have been replaced with more compact matrices with a circulant structure . Since any n-by-n circulant matrix can be represented in memory using only a vector of dimension n , the change resulted in a drastic reduction of the model size ( from 230MB to 21MB ) . Furthermore , Cheng et al . have shown empirically that their network architecture can be almost as accurate as the original network . Moczulski et al . ( 2015 ) have proposed a more principled approach leveraging a result by Huhtanen & Perämäki ( 2015 ) stating that any matrix A ∈ Cn×n can be decomposed into 2n − 1 diagonal and circulant matrices . They use this result to design Deep diagonal-circulant ReLU networks . However their experiments show good results even with a small number of factors ( down to 2 factors ) , suggesting that Deep diagonal-circulant ReLU networks can achieve good approximation error , even with few factors . In this paper , we bridge the gap between the good empirical results observed by Moczulski et al . ( 2015 ) , and the theoretical approximation capabilities of Deep diagonal-circulant ReLU networks . We prove that Deep diagonal-circulant ReLU networks with bounded width and small depth can approximate any dense neural network . We obtain this result by showing that any matrix A can be decomposed into 4k + 1 diagonal and circulant matrices where k is the rank of the matrix A . In practice , this result is more useful than the one by Huhtanen & Perämäki since one can rely on a low rank SVD decomposition of A while controlling the approximation error . In addition to this theoretical contribution , we also conduct thorough experiments on synthetic and real datasets . In accordance with the theory , our experiments demonstrate that we can easily tradeoff accuracy for model size by adjusting the number of factors in the matrix decomposition . Finally we evaluate the applicability of this approach on state-of-the-art neural network architectures trained for video classification on the Youtube-8m Video dataset ( over 1TB of training data ) . This experiment demonstrates that Deep diagonal-circulant ReLU networks can be used to train more compact neural networks on large scale , real world scenarios . 2 RELATED WORK . A variety of techniques have been proposed to build more compact deep learning models . A first category of techniques aims at compressing a trained network into a smaller model , without compromising the accuracy . For example model distillation ( Hinton et al. , 2015 ) is two step training procedure : first a large model is trained to be as accurate as possible , second a more compact model is trained to approximate the first one . Other approaches have focused on compressing the network by reducing the memory , at the level of individual weights , ( for example , using weight quantization ( Han et al. , 2016 ) or parameter pruning ) or at the level of weight matrices , using low rank decomposition of the original weight matrix ( Sainath et al. , 2013 ) or using sparse representations ( Collins & Kohli , 2014 ; Dai et al. , 2018 ; Liu et al. , 2015 ) . Instead of compressing the network a posteriori , several researchers have focused on designing models that are compact by design . This approach has several benefits , but most importantly , it reduces memory footprint , both required during training and inference . Chen et al . ( 2015 ) have proposed to compress weight matrices by using hashing functions to map several matrix coefficients into the same memory cell . The techniques works well in theory , but suffers from poor performance on modern GPU devices due to irregular access patterns . In their paper , Cheng et al . ( 2015 ) observed that fully connected layers ( which typically occupy 90 % 1 . of the total number of weights ) are often used to perform simple dimensionality reduction operation between layers of different dimension . The idea of replacing large weight matrices from fully connected layers with more compact circulant layers comes from a result by Hinrichs & Vybı́ral ( 2011 ) that have demonstrated that circulant matrices can be use to approximate the Johson-Lindenstrauss transform , often used to perform dimensionality reduction . Building on this result Cheng et al . proposed to replace the weight matrix of a fully connected layer by a circulant matrix initialized with random weights . The resulting models achieve good accuracy , with the random circulant matrix , but even better when the weights of the circulant matrix are trained with the rest of the network using a gradient based optimization algorithm . This suggests that such layers , often perform more than simple random projections , and that more expressive fully connected layers are beneficial to the overall accuracy of the model . Fortunately , more general linear transforms can also be described using circulant matrices or other structured matrices , at the cost of using more of them . Müller-Quade et al . ( 1998 ) and Schmid et al . ( 2000 ) have demonstrated this formally by showing that any matrix can be decomposed into the product of diagonal and circulant matrices , and Moczulski et al . ( 2015 ) have proposed a compact neural network architecture based on this decomposition that exhibit good accuracy in practice . Other researchers have investigated using alternative structures such as Toeplitz ( Sindhwani et al. , 2015 ) , Vandermonde ( Sindhwani et al. , 2015 ) or Fastfood transforms ( Yang et al. , 2015 ) . Despite demonstrating good empirical results , there have been 1In network such as AlexNet , the last 3 fully connected layers use 58M out of the 62M total trainable parameters . little theoretical insight to explain the good approximation capabilities of deep neural networks based on structured matrices . Barron ( 1993 ) presented the universal approximation theorem which states that any neural network with at least 1 hidden layer and sigmoid non linearity can approximate any function . However , the theorem by Barron ( 1993 ) does not bound the width of the neural network and does not consider the training procedure . Since then , substantial theoretical work has been done to evaluate the expressiveness of a neural network as a function of the width ( i.e . the number of neurons ) and the depth of the network Arora et al . ( 2018 ) ; Mhaskar et al . ( 2017 ) ; Lin et al . ( 2017 ) ; Poole et al . ( 2016 ) ; Raghu et al . ( 2016 ) ; Telgarsky ( 2016 ) ; Mhaskar & Poggio ( 2016 ) . In 2000 , Hanin ( 2017 ) have investigated the approximation capabilities of neural networks with ReLU activations and demonstrated that such networks can approximate any function . More recently , Zhao et al . ( 2017 ) have provided a theoretical study of Deep diagonal-circulant ReLU networks and demonstrated that 2-layers networks of unbounded width are universal approximators . However , these results are of limited interest because the networks used in practice are of bounded width . Unfortunately , nothing is known about the theoretical properties of Deep diagonal-circulant ReLU networks in this case . 3 BUILDING COMPACT DEEP NEURAL NETWORKS USING CIRCULANT MATRICES . 3.1 PRELIMINARIES ON CIRCULANT MATRICES . A n-by-n circulant matrix C is a special kind of Toeplitz matrix where each row is a cyclic right shift of the previous one as illustrated below . C = circ ( c ) = c0 cn−1 cn−2 . . . c1 c1 c0 cn−1 c2 c2 c1 c0 c3 ... . . . ... cn−1 cn−2 cn−3 c0 Despite their rigorous structure , circulant matrices are expressive enough to model a variety of linear transforms such as random projections ( Hinrichs & Vybı́ral , 2011 ) and when they are combined together with diagonal matrices , they can be used to represent an arbitrary transform ( Schmid et al. , 2000 ) . Circulant matrices also exhibit several properties that are interesting from a computational perspective . First , a circulant n-by-n matrix C can be represented using only n coefficients . Thus , it is far more compact that a full matrix that requires n2 coefficient . Second , the product between a circulant matrix C and a vector x can be simplified to a simple element-wise product between the vector c and x in the Fourier domain ( which is generally performed efficiently on GPU devices ) . This results in a complexity reduced from O ( n2 ) to O ( nlog ( n ) ) . In their paper , Huhtanen & Perämäki ( 2015 ) have demonstrated that any matrix A ∈ Cn×n can be approximated with an arbitrary precision by a product of circulant and diagonal matrices : Theorem 1 . ( Huhtanen & Perämäki , 2015 ) For any given matrix A ∈ Cn×n , let p be the smallest integer such that A = ∑p i=1DiS i−1 where D1 . . . Dp are diagonal matrices . Then for any > 0 , for any matrix norm ‖·‖ , there exists a sequence of matrices B1 . . . B2n−1 where Bi is a circulant matrix if i is odd , and a diagonal matrix otherwise , such that ‖B1B2 . . . B2n−1 −A‖ < , and where S = circ ( 0 , 1 , 0 , . . . , 0 ) Because of their interesting properties , several researchers have considered circulant matrices as a replacement from full weight matrices inside neural networks . 3.2 THEORETICAL PROPERTIES OF DEEP DIAGONAL-CIRCULANT RELU NETWORKS . There has already been some recent theoretical work on Deep diagonal-circulant ReLU networks , in which 2-layer networks of unbounded width where shown to be universal approximators . These results are of limited interest , because the networks used in practice are of bounded width . Unfortunately , nothing is known about the theoretical properties of Deep diagonal-circulant ReLU networks in this case . In particular , the following questions remained unanswered up to now : Are Deep diagonal-circulant ReLU networks with bounded width universal approximators ? What kind of functions can Deep diagonal-circulant ReLU networks with bounded-width and small depth approximate ? In this section , we first define formally diagonal-circulant ReLU networks , and then provide a theoretical analysis of their approximation capabilities . Definition 1 ( Deep ReLU networks ) . With ReLU ( x ) = max ( 0 , x ) , let fA , b ( x ) = ReLU ( Ax+ b ) for any matrices A ∈ Cn×n , and any b ∈ Rn . A Deep ReLU network is a function fAl , bl ◦ . . . ◦ fA1 , b1 , where A1 . . . Al are arbitrary n× n matrices and b1 . . . bl ∈ Cn and where l and n are the depth and the width of the network respectively . As in Moczulski et al . ( 2015 ) , Deep diagonal-circulant ReLU networks can be defined as follows : Definition 2 ( Deep diagonal-circulant ReLU networks ) . A Deep diagonal-circulant ReLU network is a function fDlCl , bl ◦ . . . ◦ fD1C1 , b1 where D1 . . . Dl ∈ Cn×n are diagonal matrices , C1 . . . Cl ∈ Cn×n are circulant matrices , and where l and n are the depth and the width of the network respectively . To show that bounded-width Deep diagonal-circulant ReLU networks are universal approximators , we first need a proposition relating standard deep neural networks to Deep diagonal-circulant ReLU networks . Proposition 2 . Let N : Rn → Rn be a deep ReLU networks of width n and depth l , and let X ⊂ Rn be a compact set . For any > 0 , there exists a deep diagonal-circulant ReLU network N ′ of width n and of depth ( 2n− 1 ) l such that ‖N ( x ) −N ′ ( x ) ‖2 < for all x ∈ X . We can now state the universal approximation corrolary : Corrolary 1 . Bounded depth Deep diagonal-circulant ReLU networks are universal approximators on any compact set X . Proof . Proposition 2 shows that bounded-width Deep diagonal-circulant ReLU networks can approximate any Deep ReLU network . It has been shown recently in Hanin ( 2017 ) that bounded-width deep ReLU networks are universal approximators . Together , these two results concludes the proof . It is important to remark that Deep diagonal-circulant ReLU networks are not necessarily more compact than Deep ReLU networks . Indeed , consider a n-wide Deep ReLU network with l layers having ln2 weights . The previous corollary tells us that this network can be decomposed in a Deep ReLU networks involving l ( 2n−1 ) matrices , i.e . 2ln ( 2n− 1 ) weights . Despite the lack of theoretical guarantees a number of work provided empirical evidence that bounded width and small depth Deep diagonal-circulant ReLU networks result in good performance ( e.g . Moczulski et al . ( 2015 ) ; Araujo et al . ( 2018 ) ; Cheng et al . ( 2015 ) ) . The following theorem studies the approximation properties of these small depth networks . Proposition 3 . Let N : fAl , bl ◦ . . . ◦ fA1 , b1 be a deep ReLU network of width n and depth l , such that each matrix Ai is of rank ki , where ki divides n. Let X ⊂ Rn be a compact set . For any > 0 , there exists a deep diagonal-circulant ReLU network N ′ of width n and of depth ( ∑n i=1 ( 4ki + 1 ) ) l such that ‖N ( x ) −N ′ ( x ) ‖2 < for all x ∈ X . This result generalizes Proposition 2 , showing that a Deep diagonal-circulant ReLU networks of bounded width and small depth can approximate a deep ReLU network in which the dense matrices are of low rank . Note in the proposition , we require that ki divides n. We conjecture that the proposition holds even without this condition , but we were not able to prove it . Finally , what if we choose to use small depth network to approximate deep ReLU networks where matrices are not of low rank ? To answer this question , we first need to show the negative impact of replacing matrices by their low rank approximators in neural networks : Proposition 4 . Let N = fAl , bl ◦ . . . ◦ fA1 , b1 be a deep ReLU network , where Ai ∈ Cn×n , bi ∈ Cn for all i ∈ [ l ] . Let Āi be the matrix obtained by a SVD approximation of rank k of matrix Ai . Let σi , j is the jth singular value of Ai . Define N̄ = fĀl , bl ◦ . . . ◦ fĀ1 , b1 . Then , for any x ∈ C n , we have ∥∥N ( x ) − N̄ ( x ) ∥∥ ≤ ( σlmax,1−1 ) Rσmax , k σmax,1−1 where R is an upper bound on norm of the output of any layer in N , and σmax , j = maxi σi , j . Basically , this proposition shows that we can approximate matrices in a neural network by low rank matrices , and control the approximation error . In general , the term σlmax,1 could seem large , but in practice , it is likely that most singular values in deep neural network are small in order to avoid divergent behaviors . We can now prove the result on Deep diagonal-circulant ReLU networks : Corrolary 2 . Consider any deep ReLU network N = fAl , bl ◦ . . . ◦ fA1 , b1 of depth l and width n. Let σmax , j = maxi σi , j where σi , j is the jth singular value of Ai . Let X ⊂ Rn be a compact set . For any k dividing n , there exists a deep diagonal-circulant ReLU network N ′ = fDmCm , b′l ◦ . . . ◦ fD1C1 , b′1 of width n and of depth m = 4 ( k + 1 ) n , such that for any x ∈ X , ‖N ( x ) −N ′ ( x ) ‖ < ( σ l max,1−1 ) Rσmax , k σmax,1−1 , where

Summary: In this paper, the authors prove that bounded width diagonal-circulant ReLU networks (I will call them DC-ReLU henceforth) are universal approximators (this was shown previously without the bounded width condition). They also show that bounded width and small depth DC-ReLUs can approximate deep ReLU nets with row rank parameters matrices. This explains the observed success of such networks. The authors also provide experiments to demonstrate the compression one can achieve without sacrificing accuracy.