On the Expected Complexity of Maxout Networks

1 Introduction . We are interested in the functions parametrized by artificial feedforward neural networks with maxout units . Maxout units compute parametric affine functions followed by a fixed multi-argument activation function of the form ( s1 , . . . , sK ) 7→ max { s1 , . . . , sK } and can be regarded as a natural generalization of ReLUs , which have a single-argument activation function s 7→ max { 0 , s } . For any choice of parameters , these networks subdivide their input space into activation regions where different pre-activation features attain the maximum and the computed function is ( affine ) linear . We are concerned with the expected number of activation regions and their volume given probability distributions of parameters , as well as corresponding properties for the decision boundaries in classification tasks . We show that different architectures can attain very different numbers of regions with positive probability , but for parameter distributions for which the conditional densities of bias values and the expected gradients of activation values are bounded , the expected number of regions is at most polynomial in the rank K and the total number of units . Activation regions of neural networks For neural networks with piecewise linear activation functions , the number of activation regions serves as a complexity measure and summary description which has proven useful in the investigation of approximation errors , Lipschitz constants , speed of convergence , implicit biases of parameter optimization , and robustness against adversarial attacks . In particular , Pascanu et al . ( 2013 ) ; Montúfar et al . ( 2014 ) ; Telgarsky ( 2015 , 2016 ) obtained depth separation results showing that deep networks can represent functions with many more linear regions than any of the functions that can be represented by shallow networks with the same number of 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . units or parameters . This implies that certain tasks require enormous shallow networks but can be solved with small deep networks . The geometry of the boundaries between linear regions has been used to study function-preserving transformations of the network weights ( Phuong and Lampert , 2019 ; Serra et al. , 2020 ) and robustness ( Croce et al. , 2019 ; Lee et al. , 2019 ) . Steinwart ( 2019 ) demonstrated empirically that the distribution of regions at initialization can be related to the speed of convergence of gradient descent , and Williams et al . ( 2019 ) ; Jin and Montúfar ( 2020 ) related the density of breakpoints at initialization to the curvature of the solutions after training . The properties of linear regions in relation to training have been recently studied by Zhang and Wu ( 2020 ) . The number of regions has also been utilized to study the eigenvalues of the neural tangent kernel and Lipschitz constants ( Nguyen et al. , 2020 ) . Maximum number of regions Especially the maximum number of linear regions has been studied intensively . In particular , Montúfar ( 2017 ) ; Serra et al . ( 2018 ) improved the upper bounds from Montúfar et al . ( 2014 ) by accounting for output dimension bottlenecks across layers . Hinz and Van de Geer ( 2019 ) introduced a histogram framework for a fine grained analysis of such dimensions in ReLU networks . Based on this , Xie et al . ( 2020 ) ; Hinz ( 2021 ) obtained still tighter upper bounds for ReLU networks . The maximum number of regions has been studied not only for fully connected networks , but also convolutional neural networks ( Xiong et al. , 2020 ) , graph neural networks ( GNNs ) and message passing simplicial networks ( MPSN ) ( Bodnar et al. , 2021 ) . Expected number of regions Although the maximum possible number of regions gives useful complexity bounds and insights into different architectures , in practice one may be more interested in the expected behavior for typical choices of the parameters . The first results on the expected number of regions were obtained by Hanin and Rolnick ( 2019a , b ) for the case of ReLU networks or singleargument piecewise linear activations . They show that if one has a distribution of parameters such that the conditional densities of bias values are bounded and the expected gradients of activation values are bounded , then the expected number of linear regions can be much smaller than the maximum theoretically possible number . Moreover , they obtain bounds for the expected number and volume of lower dimensional linear pieces of the represented functions . These results do not directly apply to the case of maxout units , but we will adapt the proofs to obtain corresponding results . Regions of maxout networks Most previous works focus on ReLUs or single-argument activation functions . In this case , the linear regions of individual layers are described by hyperplane arrangements , which have been investigated since the 19th century ( Steiner , 1826 ; Buck , 1943 ; Zaslavsky , 1975 ) . Hence , the main challenge in these works is the description of compositions of several layers . In contrast , the linear regions of maxout layers are described by complex arrangements that are not so well understood yet . The study of maxout networks poses significant challenges already at the level of individual layers and in fact single units . For maxout networks , the maximum possible number of regions has been studied by Pascanu et al . ( 2013 ) ; Montúfar et al . ( 2014 ) ; Serra et al . ( 2018 ) . Recently , Montúfar et al . ( 2021 ) obtained counting formulas and sharp ( asymptotic ) upper bounds for the number of regions of shallow ( deep ) maxout networks . However , their focus was on the maximum possible value , and not on the generic behavior , which we investigate here . Related notions The activation regions of neural networks can be approached from several perspectives . In particular , the functions represented by networks with piecewise linear activations correspond to so-called tropical rational functions and can be studied from the perspective of tropical geometry ( Zhang et al. , 2018 ; Charisopoulos and Maragos , 2018 ) . In the case of piecewise affine convex nonlinearities , these can be studied in terms of so-called max-affine splines ( Balestriero et al. , 2019 ) . A related but complementary notion of network expressivity is trajectory length , proposed by Raghu et al . ( 2017 ) , which measures transitions between activation patterns along one-dimensional paths on the input space , which also leads to depth separation results . Recent work ( Hanin et al. , 2021 ) shows that ReLU networks preserve expected length . Contributions We obtain the following results for maxout networks . • There are widely different numbers of linear regions that are attained with positive probability over the parameters ( Theorem 7 ) . There is a non-trivial lower bound on the number of linear regions that holds for almost every choice of the parameters ( Theorem 8 ) . These results advance the maximum complexity analysis of Montúfar et al . ( 2021 ) from the perspective of generic parameters . • For common parameter distributions , the expected number of activation regions is polynomial in the number of units ( Theorem 9 ) . Moreover , the expected volume of activation regions of different dimensions is polynomial in the number of units ( Theorem 10 ) . These results correspond to maxout versions of results from Hanin and Rolnick ( 2019b ) and Hanin and Rolnick ( 2019a ) . • For multi-class classifiers , we obtain an upper bound on the expected number of linear pieces ( Theorem 11 ) and the expected volume ( Theorem 12 ) of the decision boundary , along with a lower bound on the expected distance between input points and decision boundaries ( Corollary 13 ) . • We provide an algorithm and implementation for counting the number of linear regions of maxout networks ( Algorithm 1 ) . • We present parameter initialization procedures for maxout networks maximizing the number of regions or normalizing the mean activations across layers ( similar to Glorot and Bengio 2010 ; He et al . 2015 ) , and observe experimentally that these can lead to faster convergence of training . 2 Activation regions of maxout networks . We consider feedforward neural networks with n0 inputs , L hidden layers of widths n1 , . . . , nL , and no skip connections , which implement functions of the form f = ψ ◦ φL ◦ · · · ◦ φ1 . The l-th hidden layer implements a function φl : Rnl−1 → Rnl with output coordinates , i.e . units , given by trainable affine functions followed by a fixed real-valued activation function , and ψ : RnL → RnL+1 is a linear output layer . We denote the total number of hidden units by N = n1 + · · ·+ nL , and index them by z ∈ [ N ] : = { 1 , . . . , N } . The collection of all trainable parameters is denoted by θ . We consider networks with maxout units , introduced by Goodfellow et al . ( 2013 ) . A rank-K maxout unit with n inputs implements a function Rn → R ; x 7→ maxk∈ [ K ] { wk · x+ bk } , where wk ∈ Rn and bk ∈ R , k ∈ [ K ] , are trainable weights and biases . The activation function ( s1 , . . . , sK ) 7→ max { s1 , . . . , sK } can be regarded as a multi-argument generalization of the rectified linear unit ( ReLU ) activation function s 7→ max { 0 , s } . The K arguments of the maximum are called the pre-activation features of the maxout unit . For unit z in a maxout network , we denote ζz , k ( x ; θ ) its k-th pre-activation feature , considered as a function of the input to the network . For any choice of the trainable parameters , the function represented by a maxout network is piecewise linear , meaning it splits the input space into countably many regions over each of which it is linear . Definition 1 ( Linear regions ) . Let f : Rn0 → R be a piecewise linear function . A linear region of f is a maximal connected subset of Rn0 on which f has a constant gradient . We will relate the linear regions of the represented functions to activation regions defined next . Definition 2 ( Activation patterns ) . An activation pattern of a network with N rank-K maxout units is an assignment of a non-empty set Jz ⊆ [ K ] to each unit z ∈ [ N ] . An activation pattern J = ( Jz ) z∈ [ N ] with ∑ z∈ [ N ] ( |Jz| − 1 ) = r is called an r-partial activation pattern . The set of all possible activation patterns is denoted P , and the set of r-partial activation patterns is denoted Pr . An activation sub-pattern is a pattern where we disregard all Jz with |Jz| = 1 . The set of all possible activation sub-patterns is denoted S , and the set of r-partial activation sub-patterns is denoted Sr . Definition 3 ( Activation regions ) . Consider a network N with N maxout units . For any parameter value θ and any activation pattern J , the corresponding activation region is R ( J , θ ) : = { x ∈ Rn0 ∣∣ argmax k∈ [ K ] ζz , k ( x ; θ ) = Jz for each z ∈ [ N ] } . For any r ∈ { 0 , . . . , n0 } we denote the union of r-partial activation regions by XN , r ( θ ) : = ⋃ J∈Pr R ( J ; θ ) . By these definitions , we have a decomposition of the input space as a disjoint union of activation regions , Rn0 = tJ∈PR ( J , θ ) . See Figure 1 . Next we observe that for almost every choice of θ , r-partial activation regions are either empty or relatively open convex polyhedra of co-dimension r. In particular , for almost every choice of the parameters , if r is larger than n0 , the r-partial activation regions are empty . Therefore , in our discussion we only need to consider r up to n0 . Lemma 4 ( r-partial activation regions are relatively open convex polyhedra ) . Consider a maxout network N . Let r ∈ { 0 , . . . , n0 } and J ∈ Pr . Then for any θ , R ( J , θ ) is a relatively open convex polyhedron in Rn0 . For almost every θ , it is either empty or has co-dimension r. The proof of Lemma 4 is given in Appendix A . Next we show that for almost every choice of θ , 0-partial activation regions and linear regions correspond to each other . Lemma 5 ( Activation regions vs linear regions ) . Consider a maxout networkN . The set of parameter values θ for which the represented function has the same gradient on two distinct activation regions is a null set . In particular , for almost every θ , linear regions and activation regions correspond to each other . The proof of Lemma 5 is given in Appendix A . We note that for specific parameters , linear regions can be the union of several activation regions and can be non-convex . Such situation is more common in ReLU networks , whose units can more readily output zero , thereby hiding the activation pattern of the units in the previous layers . To summarize the above observations , for almost every θ , the 0-partial activation regions are n0dimensional open convex polyhedra which agree with the linear regions of the represented function , and for r = 1 , . . . , n0 the r-partial activation regions are co-dimension-r polyhedral pieces of the boundary between linear regions . Next we investigate the number non-empty r-partial activation regions and their volume within given subsets of the input space . We are concerned with their generic numbers , where we use “ generic ” in the standard sense , to refer to a positive Lebesgue measure event .

Summary: EDIT: I have read the authors' response and decided to keep my (positive) score.  The paper studies the problem of expressivity of feedforward neural networks with maxout (multi-argument) activation functions. Maxout units compute parametric affine functions followed by a fixed multi-argument activation function of the form (s1, . . . , sK)→max{s1, . . . , sK}.  A special case of course is ReLU nets, and many prior results have been obtained, showing how the activation patterns and linear regions of the network depend on the architecure of the network, mainly the depth and the width. For example, Telgarsky's "Benefits of depth in neural networks." paper, shows how depth in neural nets can be exponentially more efficient that width in representing certain functions like the triangular wave and compositions with itself.   The question that is raised here however is how do the linear regions of maxout nets look like for "practical" neural nets. Practical here, just means networks initialized with a multitude of different distributions that are common in practice. The importance of this question is motivated by understanding average case performance instead of worst-case performance. This reflects an approach originally put forth by two works of Hanin and Rolnick.  Briefly, the main results of the paper generalize prior results for ReLUs to the case of maxout networks. For example, the main result of the paper is Theorem 9 and 10 that state that for most parameter settings in the network, the expected number of activation regions is polynomial in the number of units. This comes in contrast with Telgarsky's and follow-up works that give exponential bounds for deep ReLU nets.