Towards Lower Bounds on the Depth of ReLU Neural Networks

1 Introduction . A core problem in machine learning or statistical pattern recognition is the estimation of an unknown data distribution with access to i.i.d . samples from the distribution . It is well-known that there is a tension between how much prior information one has about the data distribution and how many samples one needs to solve the problem with high confidence ( or equivalently , how much variance one has in one ’ s estimate ) . This is referred to as the bias-variance trade-off or the bias-complexity trade-off . Neural networks provide a way to turn this bias/complexity knob in a controlled manner that has been studied for decades going back to the idea of a perceptron by Rosenblatt [ 1958 ] . This is done by modifying the architecture of a neural network class of functions , which has two parameters : depth and size . As one increases these parameters , the class of functions becomes more expressive . In terms of the bias-variance trade-off , the “ bias ” decreases as the class of functions becomes more expressive , but the “ variance ” or “ complexity ” increases . So-called universal approximation theorems [ Cybenko , 1989 , Hornik , 1991 , Anthony and Bartlett , 1999 ] show that even with a single hidden layer , i.e. , when the depth of the architecture is the smallest possible value , one can essentially reduce the “ bias ” as much as one desires , by increasing the size of the network or the number of neurons used in the neural network . Nevertheless , it can be advantageous both theoretically and empirically to increase the depth because a substantial reduction in the size can be achieved ; Telgarsky [ 2016a ] , Eldan and Shamir [ 2016 ] , Arora et al . [ 2018 ] is a small sample of recent work in this direction . To get a better quantitative handle on these trade-offs , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . it is important to understand what classes of functions are exactly representable by neural networks with a certain architecture . The precise mathematical statements of universal approximation theorems show that single layer networks can approximate arbitrarily well any continuous function ( under some additional mild hypotheses ) . While this suggests that single layer networks are good enough from a learning perspective , from a mathematical perspective , one can ask the question if the class of functions represented by a single layer is a strict subset of the class of function represented by two or more hidden layers . On the question of size , one can ask for precise bounds on the size of the network of a given depth to represent a certain class of functions . We believe that a better understanding of the function classes exactly represented by different architectures will have implications not just for mathematical foundations , but also algorithmic and statistical learning aspects of neural networks . The task of searching for the “ best ” function in that class can only benefit from a better understanding of the nature of functions in that class . A motivating question behind the results in this paper is to understand the hierarchy of function classes exactly represented by neural networks of increasing depth . We now introduce more precise notation and terminology to set the stage for our investigations . Notation . We write [ n ] : = { 1 , 2 , . . . , n } for the set of natural numbers up to n ( without zero ) and [ n ] 0 : = [ n ] ∪ { 0 } for the same set including zero . For any n ∈ N , let σ : Rn → Rn be the component-wise rectifier function σ ( x ) = ( max { 0 , x1 } , max { 0 , x2 } , . . . , max { 0 , xn } ) . For any number of hidden layers k ∈ N , a ( k + 1 ) -layer feedforward neural network with rectified linear units ( ReLU NN or simply NN ) is given by k affine transformations T ( ` ) : Rn ` −1 → Rn ` , x 7→ A ( ` ) x+ b ( ` ) , for ` ∈ [ k ] , and a linear transformation T ( k+1 ) : Rnk → Rnk+1 , x 7→ A ( k+1 ) x . It is said to compute or represent the function f : Rn0 → Rnk+1 given by f = T ( k+1 ) ◦ σ ◦ T ( k ) ◦ σ ◦ · · · ◦ T ( 2 ) ◦ σ ◦ T ( 1 ) . The matrices A ( ` ) ∈ Rn ` ×n ` −1 are called the weights and the vectors b ( ` ) ∈ Rn ` are the biases of the ` -th layer . The number n ` ∈ N is called the width of the ` -th layer . The maximum width of all hidden layers max ` ∈ [ k ] n ` is called the width of the NN . Further , we say that the NN has depth k + 1 and size ∑k ` =1 n ` . Often , NNs are represented as layered , directed , acyclic graphs where each dimension of each layer ( including input layer ` = 0 and output layer ` = k + 1 ) is one vertex , weights are arc labels , and biases are node labels . Then , the vertices are called neurons . For a given input x = x ( 0 ) ∈ Rn0 , let y ( ` ) : = T ( ` ) ( x ( ` −1 ) ) ∈ Rn ` be the activation vector and x ( ` ) : = σ ( y ` ) ∈ Rn ` the output vector of the ` -th layer . Further , let y : = y ( k+1 ) = f ( x ) be the output of the NN . We also say that the i-th component of each of these vectors is the activation or the output of the i-th neuron in the ` -th layer . For k ∈ N , we define ReLUn ( k ) : = { f : Rn → R | f can be represented by a ( k + 1 ) -layer NN } , CPWLn : = { f : Rn → R | f is continuous and piecewise linear } . By definition , a continuous function f : Rn → R is piecewise linear in case there is a finite set of polyhedra whose union is Rn , and f is affine linear over each such polyhedron . In order to analyze ReLUn ( k ) , we use another function class defined as follows . We call a function g a p-term max function if it can be expressed as maximum of p affine terms , that is , g ( x ) = max { ` 1 ( x ) , . . . , ` p ( x ) } where ` i : Rn → R is affinely linear for i ∈ [ p ] . Based on that , we define MAXn ( p ) : = { f : Rn → R | f is a linear combination of p-term max functions } . If the input dimension n is not important for the context , we sometimes drop the index and use ReLU ( k ) : = ⋃ n∈N ReLUn ( k ) and MAX ( p ) : = ⋃ n∈N MAXn ( p ) instead . Since we deal with polyhedra a lot in this paper , we will use the standard notations convA and coneA for the convex and conic hulls of a set A ⊆ Rn . For an in-depth treatment of polyhedra and ( mixed-integer ) optimization , we refer to Schrijver [ 1986 ] . 1.1 Our Contribution . It is not hard to see that any function expressed by a ReLU network is a continuous and piecewise linear ( CPWL ) function , because one is composing continuous piecewise linear functions together . Based on a result by Wang and Sun [ 2005 ] , Arora et al . [ 2018 ] show that every CPWL function defined on Rn can be represented by a ReLU neural network with dlog2 ( n+ 1 ) e hidden layers . We wish to understand whether one can do better . We believe it is not possible to do better and we pose the following conjecture to better understand the importance of depth in neural networks . Conjecture 1.1 . For any n ∈ N , let k∗ : = dlog2 ( n+ 1 ) e. Then it holds that ReLUn ( 0 ) ( ReLUn ( 1 ) ( · · · ( ReLUn ( k∗ − 1 ) ( ReLUn ( k∗ ) = CPWLn . ( 1 ) Conjecture 1.1 claims that any additional layer up to k∗ hidden layers strictly increases the set of representable functions . This would imply that the construction by Arora et al . [ 2018 , Theorem 2.1 ] is actually depth-minimal . Observe that in order to prove Conjecture 1.1 , it suffices to find a single function f ∈ ReLUn ( k∗ ) \ ReLUn ( k∗ − 1 ) with n = 2k ∗−1 for all k∗ ∈ N. This also implies all remaining strict inclusions ReLUn ( i− 1 ) ( ReLUn ( i ) for i < k∗ since ReLUn ( i− 1 ) = ReLUn ( i ) directly implies that ReLUn ( i− 1 ) = ReLUn ( i′ ) for all i′ ≥ i− 1 . In fact , there is a canonical candidate for such a function , allowing us to reformulate the conjecture as follows . Conjecture 1.2 . For any k ∈ N , n = 2k , the function fn ( x ) = max { 0 , x1 , . . . , xn } can not be represented with k hidden layers . Proposition 1.3 . Conjecture 1.1 and Conjecture 1.2 are equivalent . Proof ( Sketch ) . We argued above that Conjecture 1.2 implies Conjecture 1.1 . For the other direction , one can argue that , if the specific ( n+1 ) -term max function fn can be represented by k hidden layers , then every other ( n+ 1 ) -term max function as well . The claim then follows via a result by [ Wang and Sun , 2005 ] stating that any f ∈ CPWLn can be written as linear combination of ( n+ 1 ) -term max functions . We provide a detailed argument in Appendix A . It is known that Conjecture 1.2 holds for k = 1 [ Mukherjee and Basu , 2017 ] . However , the conjecture remains open for k ≥ 2 . In this paper , we present the following results as partial progress towards resolving this conjecture . In Section 2 , we resolve Conjecture 1.2 for k = 2 , under a natural assumption on the breakpoints of the function represented by any intermediate neuron . We achieve this result by leveraging techniques from mixed-integer programming to analyze the set of functions computable by certain NNs . It is not hard to see that MAX ( 2k ) ⊆ ReLU ( k ) for all k ∈ N [ Arora et al. , 2018 ] , that is , any 2k-term max function ( and linear combinations thereof ) can be expressed with k hidden layers . One might ask whether the converse is true as well , that is , whether the classes MAX ( 2k ) and ReLU ( k ) are actually equal . This would not only provide a neat characterization of ReLU ( k ) , but also prove Conjecture 1.2 without any additional assumption since one can show that max { 0 , x1 , . . . , x2k } is not contained in MAX ( 2k ) . In fact , this is true for k = 1 , that is , a function is computable with one hidden layer if and only if it is a linear combination of 2-term max functions . However , in Section 3 , we show that for k ≥ 2 , the class ReLU ( k ) is a strict superset of MAX ( 2k ) . In this section , the key technical ingredient is the theory of polyhedral complexes associated with CPWL functions . This way , we provide important insights concerning the richness of the class ReLU ( k ) . So far , we have focused on understanding the smallest depth needed to express CPWL functions using neural networks with ReLU activations . In Section 4 , we complement these results by upper bounds on the sizes of the networks needed for expressing arbitrary CPWL functions . In particular , Theorem 4.4 shows that any continuous piecewise linear function with p linear/affine pieces on Rn can be expressed by a network with depth at most O ( log n ) and width at most pO ( n2 ) . We arrive at this result by introducing a novel application of recently established interactions between neural networks and tropical geometry .

Summary: This paper considers the problem of characterizing exact representations for ReLU networks of a given depth, but any width. It was previously shown that the functions represented by ReLU networks of depth logarithmic in dimension is exactly equal to the set of all continuous piecewise linear functions. This work provides some results which suggest that this result is tight. That is, a depth logarithmic in dimension is necessary to exactly represent all piecewise linear functions.   Proposition 1.3 simplifies this conjecture by giving a simple equivalent condition in terms of representing max(0,x_1,\dots,x_n) and proves the conjecture up to dimension 4 under the assumption 2.4 which is unproven. In section 3 it is shown that the set of functions representable by a depth k network is a strict super set of Max(2^k) - i.e, set of all functions which can written as a linear combination of max of 2^k affine functions. This provides further evidence in support of the conjecture.  Section 4 then extends the results of Arora et. al 2018 to provide an upper bound on the width required for a relu network to exactly represent affine function with p pieces. This bound is $p^{O(n^2)}$.