Empirical Bounds on Linear Regions of Deep Rectifier Networks

1 INTRODUCTION . Neural networks with piecewise linear activations have become increasingly more common along the past decade , in particular since Nair & Hinton ( 2010 ) and Glorot et al . ( 2011 ) . The simplest and most commonly used among such forms of activation is the Rectifier Linear Unit ( ReLU ) , which outputs the maximum between 0 and its input argument ( Hahnloser et al. , 2000 ; LeCun et al. , 2015 ) . In the functions modeled by these networks , we can associate each part of the domain in which the network corresponds to an affine function with a particular set of units having positive outputs . We say that those are the active units for that part of the domain . Counting these “ pieces ” into which the domain is split , which are often denoted as linear regions or decision regions , is one way to compare the expressiveness of models defined by networks with different configurations or coefficients . The theoretical analysis of the number of input regions in deep learning dates back to at least Bengio ( 2009 ) , and more recently Serra et al . ( 2018 ) have shown empirical evidence that the accuracy of similar rectifier networks can be associated with the number of such regions . From the study of how many linear regions can be defined on such a rectifier network with n ReLUs , we already know that not all configurations – and in some cases none – can reach the ceiling of 2n regions . We have learned that the number of regions may depend on the dimension of the input as well as on the number of layers and how the units are distributed among these layers . On the one hand , it is possible to obtain neural networks where the number of regions is exponential on network depth ( Pascanu et al. , 2014 ; Montúfar et al. , 2014 ) . On the other hand , there is a bottleneck effect by which the width of each layer affects how the regions are partitioned by subsequent layers due to the dimension of the space containing the image of the function , up to the point that shallow networks define the largest number of linear regions if the input dimension exceeds n ( Serra et al. , 2018 ) . The literature on this topic has mainly focused on bounding the maximum number of linear regions . Lower bounds are obtained by constructing networks defining increasingly larger number of linear regions ( Pascanu et al. , 2014 ; Montúfar et al. , 2014 ; Arora et al. , 2018 ; Serra et al. , 2018 ) . Upper bounds are proven using the theory of hyperplane arrangements by Zaslavsky ( 1975 ) along with other analytical insights ( Raghu et al. , 2017 ; Montúfar , 2017 ; Serra et al. , 2018 ) . These bounds are only identical – and thus tight – in the case of one-dimensional inputs ( Serra et al. , 2018 ) . Both of these lines have explored deepening connections with polyhedral theory , but some of these results have also been recently revisited using tropical algebra ( Zhang et al. , 2018 ; Charisopoulos & Maragos , 2018 ) . In addition , Serra et al . ( 2018 ) have shown that the linear regions of a trained network correspond to a set of projected solutions of a Mixed-Integer Linear Program ( MILP ) . Other methods to study neural network expressiveness include universal approximation theory ( Cybenko , 1989 ) , VC dimension ( Bartlett et al. , 1998 ) , and trajectory length ( Raghu et al. , 2017 ) . Different networks can be compared by transforming one network to another with different number of layers or activation functions . For example , it has been shown that any continuous function can be modeled using a single hidden layer of sigmoid activation functions ( Cybenko , 1989 ) . In the context of ReLUs , Lin & Jegelka ( 2018 ) have shown that the popular ResNet architecture ( He et al. , 2016 ) with a single ReLU neuron in every hidden layer can be a universal approximator . Furthermore , Arora et al . ( 2018 ) have shown that a network with single hidden layer of ReLUs can be trained for global optimality with a runtime polynomial in the data size , but exponential in the input dimension . The use of trajectory length for expressiveness is related to linear regions , i.e. , by changing the input along a one dimensional path we study the transition in the linear regions . Certain critical network architectures using leaky ReLUs ( f ( x ) = max ( x , αx ) , α ∈ ( 0 , 1 ) ) are identified to produce connected decision regions ( Nguyen et al. , 2018 ) . In order to avoid such degenerate cases , we need to use sufficiently wide hidden layers . However , this result is mainly applicable for leaky ReLUs and not for the standard ReLUs ( Beise et al. , 2018 ) . Although the number of linear regions has been long conjectured and recently shown to work for comparing similar networks , this metric would only be used in practice if we come up with faster methods to count or reasonably approximate such number . Our approach in this paper consists of introducing empirical upper and lower bounds , both of which based on the weight and bias coefficients of the networks , and thus able to compare networks having the same configuration of layers . In particular , we reframe the problem of determining the potential number of linear regions N of an architecture with that of estimating the representation efficiency η = log2N of a network , which can be interpreted as the minimum number of units to define as many linear regions , thereby providing a more practical and interpretable metric for expressiveness . We present the following contributions : ( i ) We adapt approximate model counting methods for propositional satisfiability ( SAT ) to obtain probabilistic bounds on the number of solutions of MILP formulations , which we use to count regions . Interestingly , these methods are particularly simpler and faster when restricted to lower bounds on the order of magnitude . See results in Figure 2 and algorithm in Section 5 . ( ii ) We refine the best known upper bound by considering the coefficients of the trained network . With such information , we identify that unit activity further contributes to the bottleneck effect caused by narrow layers ( Serra et al. , 2018 ) . Furthermore , we are able to compare networks with the same configuration of layers . See results in Table 1 and theory in Section 4 . ( iii ) We also survey and contribute to the literature on MILP formulations of rectifier networks due to the impact of the formulation on obtaining better empirical bounds . See Section 3 . 2 PRELIMINARIES AND NOTATIONS . In this paper , we consider feedforward Deep Neural Networks ( DNNs ) with Rectifier Linear Unit ( ReLU ) activations . Each network has n0 input variables given by x = [ x1 x2 . . . xn0 ] T with a bounded domain X and m output variables given by y = [ y1 y2 . . . ym ] T . Each hidden layer l = { 1 , 2 , . . . , L } has nl hidden neurons with outputs given by hl = [ hl1 hl2 . . . hlnl ] T . For notation simplicity , we may use h0 for x and hL+1 for y . Let W l be the nl × nl−1 matrix where each row corresponds to the weights of a neuron of layer l. Let bl be the bias vector used to obtain the activation functions of neurons in layer l. The output of unit i in layer l consists of an affine transformation gli = W l ih l−1 + bli to which we apply the ReLU activation h l i = max { 0 , gli } . We may regard the DNN as a piecewise linear function F : Rn0 → Rm that maps the input x ∈ X ⊂ Rn0 to y ∈ Rm . Hence , the domain is partitioned into regions within which F corresponds to an affine function , which we denote as linear regions . Following the same convention as Raghu et al . ( 2017 ) ; Montúfar ( 2017 ) ; Serra et al . ( 2018 ) , we characterize each linear region by the set of units that are active in that domain . For each layer l , let Sl ⊆ { 1 , . . . , nl } be the activation set in which i ∈ Sl if and only if hli > 0 . Let S = ( S1 , . . . , Sl ) be the activation pattern aggregating those activation sets . Consequently , the number of linear regions defined by the DNN corresponds to the number of nonempty sets in x defined by all possible activation patterns . 3 COUNTING AND MILP FORMULATIONS . We can represent each linear region defined by a rectifier network with n hidden units on domain X by a distinct vector in { 0 , 1 } n , where each element denotes if the corresponding unit is active or not . Serra et al . ( 2018 ) have shown that such vector can be embedded into an MILP formulation mapping inputs to outputs of a rectifier network . For a neuron i in layer l , this mapping uses such binary variable zi , the vector hl−1 of inputs coming from layer l − 1 , the variable gli for the value of the affine transformation W lih l−1 + bli , the variable h l i = max { 0 , gli } denoting the output of the unit , and a variable h̄li denoting the output of a complementary fictitious unit h̄ l i = max { 0 , −gli } : W lih l−1 + bli = g l i ( 1 ) gli = h l i − h̄li ( 2 ) hli ≤ H lizli ( 3 ) h̄li ≤ H̄ li ( 1− zli ) ( 4 ) hli ≥ 0 ( 5 ) h̄li ≥ 0 ( 6 ) zli ∈ { 0 , 1 } ( 7 ) For correctness , constants H li and H̄ l i should be as large as h l i and h̄ l i can be . In such case , the value of gli determines if the unit or its fictitious complement is active . Note , however , that constraints ( 1 ) – ( 7 ) allow zli = 1 when g l i = 0 . To count the number of linear regions , Serra et al . ( 2018 ) uses the projection on the binary variables of the solutions where all active units have positive outputs , i.e. , hli > 0 if z l i = 1 , thereby counting the positive solutions with respect to f on the binary variables of max f ( 8 ) s.t . ( 1 ) − ( 7 ) l = 1 , . . . , L ; i = 1 , . . . , nl ( 9 ) f ≤ hli + ( 1− zli ) H li l = 1 , . . . , L ; i = 1 , . . . , nl : H li > 0 ( 10 ) x ∈ X ( 11 ) The solutions of this projection can be enumerated using the one-tree algorithm ( Danna et al. , 2007 ) , in which the branch-and-bound tree used to obtain the optimal solution is further expanded to collect near-optimal solutions up to a given limit . In general , finding a feasible solution to a MILP is NPcomplete ( Cook , 1971 ) and thus optimization is NP-hard . However , Fischetti & Jo ( 2018 ) note that a feasible solution can always be obtained by evaluating any valid input . While that does not directly imply that optimization problems on DNNs are easy , it hints at the possibility of good properties . Several MILP formulations with an equivalent feasible set have been used in the context of network verification to determine the image of the function modeled ( Lomuscio & Maganti , 2017 ; Dutta et al. , 2018 ) and evaluate adversarial perturbations in the domain X ( Cheng et al. , 2017 ; Fischetti & Jo , 2018 ; Tjeng et al. , 2017 ; Xiao et al. , 2018 ) . There are also similar applications relaxing the binary variables as continuous variables in the domain [ 0 , 1 ] or using the linear formulation of a particular linear region ( Bastani et al. , 2016 ; Ehlers , 2017 ; Wong & Kolter , 2018 ) , which can be simply defined using W lih l−1 + bli ≥ 0 for active units and the complement for inactive units . Although equivalent , some authors have explored how these formulations may differ in strength ( Fischetti & Jo , 2018 ; Tjeng et al. , 2017 ; Huchette , 2018 ) . When the binary variables are relaxed as continuous variables in the domain [ 0 , 1 ] , we obtain a linear relaxation that may differ across formulations . We say that an MILP formulation A is stronger than another formulation B if , when projected on a common set of variables , the linear relaxation of A is a proper subset of the linear relaxation ofB . Formulation strength is commonly regarded as a proxy for MILP solver performance . Differences in strength may be due to changes in constants such as H li and H̄ l i , use of additional valid inequalities that remove fractional solutions , or even additional variables defining an extended formulation . For mapping DNNs , we can discuss strength in at least three levels of scope . First , we can consider the strength of the formulation to represent a ReLU activation hli = max { 0 , gli } . Ideally , we want the projection on gli and hli to be the convex outer approximation of all possible combined values of those variables ( Wong & Kolter , 2018 ) , as illustrated in Figure 1 ( a ) and ( b ) , which is in fact the case if the values of H li and H̄ l i are the smallest possible . Lemma 1 . If H li = arg maxgl−1 { gli } ≥ 0 and H̄ li = arg maxgl−1 { −gli } ≥ 0 , then the linear relaxation of ( 2 ) – ( 7 ) defines the convex outer approximation on ( gli , h l i ) . Lemma 1 evidences that , for formulations like the one above , constants for both the maximum and the minimum values of hli are necessary to obtain a strong formulation . The proof can be found in Appendix A . We note that a similar claim without proof is made by Huchette ( 2018 ) . When the domain X of the network is defined by a box , in which case the domain of each input variable xi is an independent continuous interval , then the smallest possible values for H1i and H̄ 1 i can be computed with interval arithmetic by taking element-wise maxima ( Cheng et al. , 2017 ; Serra et al. , 2018 ) . When extended to subsequent layers , however , this approach is prone to overestimate the values for H li and H̄ l i because the output of subsequent layers is not necessarily a box in which the maximum value of each input are independent from each other . More generally , if X is polyhedral , Fischetti & Jo ( 2018 ) and Tjeng et al . ( 2017 ) show that we can obtain the smallest values for these constants by solving a sequence of MILPs on layers l′ = 1 , . . . , L of the form H l ′ i = max g l′ i ( 12 ) s.t . ( 1 ) − ( 7 ) l = 1 , . . . , l′ − 1 ; i = 1 , . . . , nl ( 13 ) x ∈ X ( 14 ) and replacing ( equation 12 ) with max−gl′i to compute H̄ l ′ i . In large rectifier networks , Tjeng et al . ( 2017 ) found that many units are always active because H̄ li < 0 or always inactive because H l i ≤ 0 . At the very least , we can use 0 on either case . In the former case , which they denote as stably active , we can simply replace constraints ( 1 ) - ( 7 ) with hli = g l i . In the latter case , which they denote as stably inactive , we note that the unit can be removed from the formulation without any loss . They denote as unstable the remaining units , which can be active or not depending on their inputs . Second , we can consider the strength of the formulation to represent the mapping of hl−1 to hl on each layer . Huchette ( 2018 ) argues that this additional strengthening may remove certain combinations of hl−1 and hli that can never occur , and has shown that this can be done using an extended formulation following Balas ( 1998 ) . Figure 1 ( c ) describes one such example . However , we observed a slower performance to count linear regions due to the larger number of variables . Huchette ( 2018 ) has also shown that these variables can be projected out , with the resulting formulation having an exponential number of constraints on nl . In the context of finding a single optimal solution , usually not requiring all of them , these constraints can be efficiently generated as needed . Third , we can consider constraints strengthening the formulation across different layers . For example , Huchette ( 2018 ) presents such a family of valid inequalities that resemble those obtained by projecting out the extra variables after convexifying each layer as described above .

Summary: This paper contributes to the study of the number of linear regions in ReLU neural networks. An approximate probabilistic counting algorithm is used to estimate the lower bound of that quantity, whereas an upper bound is derived analytically. The probabilistic counting algorithm is shown to be much more efficient than exact counting, and is adapted from the SAT literature. The new upper bound uses the weights of the network, a new technique compared to previous work on these bounds, and is shown to be sometimes tighter than the older bound.