A General Computational Framework to Measure the Expressiveness of Complex Networks using a Tight Upper Bound of Linear Regions

1 INTRODUCTION . Deep nerual network ( DNN ) ( LeCun et al. , 2015 ) has obtained great success in many fields such as computer vision , speech recognition and neural language process ( Krizhevsky et al. , 2012 ; Hinton et al. , 2012 ; Devlin et al. , 2018 ; Goodfellow et al. , 2014 ) . However , it has not been completely understood why DNNs can perform well with satisfying generalization on different tasks . Expressiveness is one perspective used to address this open question . More specifically , one can theoretically study expressiveness of DNNs using approximation theory ( Cybenko , 1989 ; Hornik et al. , 1989 ; Hanin , 2019 ; Mhaskar & Poggio , 2016 ; Arora et al. , 2016 ) , or measure the expressiveness of a DNN . While sigmoid or tanh functions are employed as the activation functions in early work of DNNs , rectified linear units ( ReLU ) or other piece-wise linear functions are more popular in nowadays . Yarotsky ( 2017 ) has proved that any DNN with piece-wise linear activation functions can be transformed to a DNN with ReLU . Thus , the study of expressiveness usually focuses on ReLU DNNs . It is known that a ReLU DNN represents a piece-wise linear ( PWL ) function , which can be regarded to have different linear transforms for each region . And with more regions the PWL function is more complex and has stronger expressive ability . Therefore , the number of linear regions is intuitively a meaningful measurement of expressiveness ( Pascanu et al. , 2013 ; Montufar et al. , 2014 ; Raghu et al. , 2017 ; Serra et al. , 2018 ; Hinz & Van de Geer , 2019 ) . A direct measurement of linear regions number is difficult , if not impossible , and thus the upper bound of linear regions number is practically used as a figure of metrics to characterize the expressiveness . Inspired by the computational framework in ( Hinz & Van de Geer , 2019 ) , we improve the upper bound in Serra et al . ( 2018 ) for multilayer perceptrons ( MLPs ) and extend the framework to more complex networks . More importantly , we propose a general approach to construct a more accurate upper bound for almost any type of network . The contributions of this paper are listed as follows . • Through a geometric analysis , we derive a recursive formula for γ , which is a key parameter to construct a tight upper bound . Employing a better initial value , we propose a tighter upper bound for deep fully-connected ReLU networks . In addition , the recursive formula provide a potential to further improve the upper bound given an improved initial value . • Different from Hinz & Van de Geer ( 2019 ) , we not only consider deep fully-connected ReLU networks , but also extend the computational framework to more widely used network architectures , such as skip connections , pooling layers and so on . With the extension , the upper bound of U-Net ( Ronneberger et al. , 2015 ) or other common networks can be computed . By comparing the upper bound of different networks , we show the relation between expressiveness of networks with or without special structures . • Our experiments show that novel network structures enhance the upper bound in most cases . For cases in which the upper bound is almost not enhanced by novel network settings , we explain it by analysing the partition efficiency and the practical number of linear regions . 2 RELATED WORK AND PRELIMINARIES . 2.1 RELATED WORK . There are literature on the linear regions number in the case of ReLU DNNs . Pascanu et al . ( 2013 ) compare the linear regions number of shallow networks by providing a lower bound . Montufar et al . ( 2014 ) give a simple but improved upper bound compared with Pascanu et al . ( 2013 ) . Montúfar ( 2017 ) proposes a even tighter upper bound than Montufar et al . ( 2014 ) . And Raghu et al . ( 2017 ) also prove a similar result which has the same order compared to Montúfar ( 2017 ) . Later , Serra et al . ( 2018 ) propose a tighter upper bound and a method to count the practical number of linear regions . Furthermore , Serra & Ramalingam ( 2018 ) ; Hanin & Rolnick ( 2019a ; b ) explore the properties of the practical number of linear regions . Finally , Hinz & Van de Geer ( 2019 ) employ the form of matrix computation to erect a framework to compute the upper bound , which is a generalization of previous work ( Montufar et al. , 2014 ; Montúfar , 2017 ; Serra et al. , 2018 ) 2.2 NOTATIONS , DEFINITIONS AND PROPERTIES . In this section , we will introduce some definitions and propositions . Since the main computational framework is inspired by Hinz & Van de Geer ( 2019 ) , some notations and definitions are similar . Let us assume a ReLU MLP has the form as follows . f ( x ) = W ( L ) σ ( W ( L−1 ) · · ·σ︸ ︷︷ ︸ ( L−1 ) ( W ( 1 ) x + b ( 1 ) ) · · ·+ b ( L−1 ) ) + b ( L ) ( 1 ) where x ∈ Rn0 , W ( i ) ∈ Rni×ni−1 , b ( i ) ∈ Rni and σ ( x ) = max ( x , 0 ) denoting the ReLU function . W ( i ) is the weights in the ith layer and b ( i ) is the bias vector . f ( x ) can also be written as : h0 ( x ) = x , hi ( x ) = σ ( W ( i ) hi−1 ( x ) + b ( i ) ) , 1 ≤ i < L , ( 2 ) f ( x ) = hL ( x ) = W ( L ) hL−1 ( x ) + b ( L ) ( 3 ) Firstly , we define the linear region in the following way . Definition 1 . For a PWL function f ( x ) : Rn0 → RnL , we define D is a linear region , if D satisfies that : ( a ) D is connected ; ( b ) f is an affine function on D ; ( c ) Any D′ % D , f is not affine on D′ . For a PWL function f , the domain can be partitioned into different linear regions . Let P ( f ) = { Di|Di is a linear region of f , ∀Di 6= Dj , Di ∩ Dj = ∅ } represent all the linear regions of f . We then define the activation pattern of ReLU DNNs as follows . Definition 2 . For any x ∈ Rn0 , we define the activation pattern of x in ith layer shi ( x ) ∈ { 0 , 1 } ni as follows . shi ( x ) j = { 1 , if W ( i ) j , : hi−1 ( x ) + b ( i ) j > 0 0 , if W ( i ) j , : hi−1 ( x ) + b ( i ) j ≤ 0 , for i ∈ { 1 , 2 , . . . , L− 1 } , j ∈ { 1 , 2 , . . . , ni } , where W ( i ) j , : is the j th row of W ( i ) , b ( i ) j is the j th component of b ( i ) . ( Hinz & Van de Geer , 2019 ) For any x , hi ( x ) can be rewritten as hi ( x ) = W ( i ) ( x ) hi−1 ( x ) + b ( i ) ( x ) , where W ( i ) ( x ) is a matrix with some rows of zeros and b ( i ) ( x ) is a vector with some zeros . More precisely , W ( i ) ( x ) j , : = { W ( i ) j , : , if shi ( x ) j = 1 ; 0 , if shi ( x ) j = 0. b ( i ) ( x ) j = { b ( i ) j , if shi ( x ) j = 1 ; 0 , if shi ( x ) j = 0 . ( 4 ) To conveniently represent activation patterns of multi-layer in a MLP , we denote h ( i ) = { h1 , ... , hi } , where hi is defined in Eq.2 , and Sh ( i ) ( x ) = ( sh1 ( x ) , ... , shi ( x ) ) , S ( h ( i ) ) = { Sh ( i ) ( x ) |x ∈ Rn0 } . Given a fixed x , it is easy to prove that hi ( x ) is an affine transform ( i = 1 , 2 , . . . , L ) . Suppose that s ∈ { 0 , 1 } n1 × · · · × { 0 , 1 } nL−1 , h = { h1 , ... , hL−1 } and h ( x ) represents hL−1 ( x ) , if D = { x|Sh ( x ) = s } 6= ∅ , then f is an affine transform in D. And it is easy to prove that there exists a linear region D′ so that D ⊆ D′ . Therefore we have |P ( f ) | ≤ |S ( h ) | . In our computational framework , histogram is a key concept and defined as follows . Definition 3 . Define a histogram v as follows . ( Hinz & Van de Geer , 2019 ) v ∈ V = x ∈ NN| ‖ x‖1 = ∞∑ j=0 xj < ∞ ( 5 ) A histogram is used to represent a discrete distribution of N. For example , the histogram of nonnegative integers G = { 1 , 0 , 1 , 4 , 3 , 2 , 3 , 1 } is ( 1 , 3 , 1 , 2 , 1 ) > . For convenience , let vi = ∑ x∈G 1x=i and the histogram of G is denoted by Hist ( G ) . We can then define an order relation . Definition 4 . For any two histograms v , w , define the order relation as follows . ( Hinz & Van de Geer , 2019 ) v w : ⇔ ∀J ∈ N , ∞∑ j=J vj ≤ ∞∑ j=J wj ( 6 ) It is obvious that any two histograms are not always comparable . But we can define a max operation such that v ( i ) max ( { v ( i ) |i ∈ I } ) where I is an index collection . More precisely : Definition 5 . For a finite index collection I , let VI = { v ( i ) |i ∈ I } , define max operation as follows . max ( VI ) J = maxi∈I ∞∑ j=J v ( i ) j −max i∈I ∞∑ j=J+1 v ( i ) j for J ∈ N ( 7 ) where v ( i ) j is the j th component of the histogram v ( i ) . ( Hinz & Van de Geer , 2019 ) When a region is divided by hyperplanes , the partitioned regions number will be affected by the space dimension which is defined as : Definition 6 . For a connected and convex set D ⊆ Rn , if there exists a set of linear independent vectors { v ( i ) |v ( i ) ∈ Rn , i = 1 , . . . , k , k ≤ n } and a fixed vector c ∈ Rn , s.t . ( a ) any x 6= c ∈ D , x = c + ∑k i=1 aiv ( i ) where { ai } are not all 0 ; ( b ) there exists ai 6= 0 s.t . aiv ( i ) + c ∈ D. Then the space dimension of D is k and denote it as Sd ( D ) = k. The following proposition shows the change of space dimension after an affine transform . Proposition 1 . Suppose D ⊆ Rn is a connected and convex set with space dimension of k ( k ≤ n ) , f is an affine transform with domain of D and can be written as f ( x ) = Ax+b , where A ∈ Rm×n and b ∈ Rm . Then f ( D ) is a connected and convex set and Sd ( f ( D ) ) ≤ min ( k , rank ( A ) ) ( 8 ) The proof of proposition 1 is given in Appendix A.1.1 . Now we can analyze the relationship between the change of space dimension and activation patterns . Let us first consider the 1st layer in an MLP f . W ( 1 ) j , : and b ( 1 ) j construct a hyperplane ( W ( 1 ) j , : x + b ( 1 ) j = 0 ) in Rn0 and this hyperplane divides the whole region of Rn0 into two parts . One part corresponds to successful activation of the jth node in the 1st layer and the other to unsuccessful activation . This can be represented by the jth component of sh1 ( x ) . Therefore all the possible activation pattern sh1 ( x ) is one by one correspondent to the divided regions by n1 hyperplanes of { W ( 1 ) j , : x+b ( 1 ) j = 0 , j = 1 , . . . , n1 } , which are denoted as Hh1 . For any region D divided by Hh1 , we have h1 ( x ) = W ( 1 ) ( x ( 0 ) ) x+b ( 1 ) ( x ( 0 ) ) where x ( 0 ) is any point in D and rank ( W ( 1 ) ( x ( 0 ) ) ) ≤ |s|1 where s is the correspondent activation pattern of D. According to proposition 1 , h1 ( D ) satisfies that Sd ( h1 ( D ) ) ≤ min { n0 , |s|1 } . Similarly , Hh2 divides h1 ( D ) into different parts , and this corresponds to divide D into more sub-regions . In general , every element of S ( h ( i ) ) is correspondent to one of regions that are partitioned by Hh1 , Hh2 , . . . , Hhi . The next two definitions are used to describe the relationship between the change of space dimension and activation patterns . Definition 7 . Define Hsd ( Sh ) as the space dimension histogram of regions partitioned by a ReLU network defined by Eq.1 where h = { h1 , . . . , hL−1 } , i.e . Hsd ( Sh ) = Hist ( { Sd ( h ( D ( s ) ) ) | D ( s ) = { x|Sh ( x ) = s } , s ∈ S ( h ) } ) . ( 9 ) Definition 8 . DefineHd ( Sh ) as the dimension histogram of regions partitioned by a ReLU network defined by Eq.1 where h = { h1 , . . . , hL−1 } , i.e . Hd ( Sh ) = Hist ( { min { n0 , |sh1 |1 , ... , |shL−1 |1 } |s ∈ S ( h ) } ) . ( 10 ) We then have the following proposition . Proposition 2 . Given a ReLU network defined by Eq.1 , let h = { h1 , . . . , hL−1 } , then Hsd ( Sh ) Hd ( Sh ) . The proof of Proposition 2 is given in the Appendix A.1.2 . Proposition 2 shows that space dimension is limited by dimension histogram . This idea is used to compute the upper bound . We can then start to introduce our computational framework . For convenience , we denote RL ( n , n′ ) as one layer of a ReLU MLP with n input nodes and n′ output nodes ( containing one linear transform and one ReLU activation function ) , and define its activation histogram as follows . Definition 9 . Given h ∈ RL ( n , n′ ) , define Ha ( Sh ) as the activation histogram of regions partitioned by h , i.e . Ha ( Sh ) = Hist ( { |s|1 | s ∈ { sh ( x ) | x ∈ R n } } ) . ( 11 ) The activation histogram is similar to the dimension histogram , and it is used to define γ which is a key parameter to construct an upper bound . Definition 10 . If γ′n , n satisfies following conditions : ( a ) ∀n′ ∈ N+ , n ∈ { 0 , . . . , n′ } , max { Ha ( Sh ) |h ∈ RL ( n , n′ ) } γn , n′ ; ( b ) ∀n′ ∈ N+ , n , ñ ∈ { 0 , . . . , n′ } , n ≤ ñ =⇒ γn , n′ γñ , n′ . Then , γn , n′ ( n′ ∈ N+ , n ∈ 1 , ... , n′ ) satisfies the bound condition . ( Hinz & Van de Geer , 2019 ) Here , for h ∈ RL ( 0 , n′ ) , we define H ( Sh ) = e0 = ( 1 , 0 , 0 , . . . ) > . Let Γ be the set of all ( γn , n′ ) n′∈N+ , n∈ { 0 , ... , n′ } that satisfy the bound conditions . When n > n ′ , γn , n′ is defined to be equal to γn′ , n′ since max { Ha ( Sh ) |h ∈ RL ( n , n′ ) } is equal to max { Ha ( Sh ) |h ∈ RL ( n′ , n′ ) } ( Hinz & Van de Geer , 2019 ) . By the definition γn , n′ represents an upper bound of the activation histogram of regions which are derived from n-dimension space partitioned by n′ hyperplanes . According to Proposition 1 , this upper bound is also related to the upper bound of space dimension . Therefore when γn , n′ is tighter the computation of upper bound of linear regions number will be more accurate . The following function is used to describe the relationship between upper bounds of activation histogram and space dimension . Definition 11 . For i∗ ∈ N , define a clipping function cli∗ ( · ) : V → V as follows . ( Hinz & Van de Geer , 2019 ) cli∗ ( v ) i = vi for i < i ∗∑∞ j=i∗ vj for i = i ∗ 0 for i > i∗ ( 12 ) With the definitions and notations above , we can introduce the computational framework to compute the upper bound of linear regions number as follows . Proposition 3 . For a γ ∈ Γ , define the matrix B ( γ ) n′ ∈ N ( n ′+1 ) × ( n′+1 ) as ( B ( γ ) n′ ) i , j = ( clj−1 ( γj−1 , n′ ) ) i−1 , i , j ∈ { 1 , . . . , n ′ + 1 } . Then , the upper bound of linear regions number of an MLP in Eq.1 is∥∥∥B ( γ ) nL−1MnL−2 , nL−1 . . .B ( γ ) n1 Mn0 , n1en0+1∥∥∥ 1 , ( 13 ) where Mn , n′ ∈ R ( n ′+1 ) × ( n+1 ) , ( M ) i , j = δi , min ( j , n′+1 ) . ( Hinz & Van de Geer , 2019 )

Summary: This paper studies the counting of linear regions of a multi-layer ReLU network and gives an upper bound on the number of linear regions that is tighter than existing results. Networks with skip connections and pooling layers are also considered. The authors then compare their bounds for standard multi-layer networks and networks with skip connections/pooling layers and conclude that the latter has certain advantages in expressive power.