Towards Lower Bounds on the Depth of ReLU Neural Networks

1 Introduction . A core problem in machine learning or statistical pattern recognition is the estimation of an unknown data distribution with access to i.i.d . samples from the distribution . It is well-known that there is a tension between how much prior information one has about the data distribution and how many samples one needs to solve the problem with high confidence ( or equivalently , how much variance one has in one ’ s estimate ) . This is referred to as the bias-variance trade-off or the bias-complexity trade-off . Neural networks provide a way to turn this bias/complexity knob in a controlled manner that has been studied for decades going back to the idea of a perceptron by Rosenblatt [ 1958 ] . This is done by modifying the architecture of a neural network class of functions , which has two parameters : depth and size . As one increases these parameters , the class of functions becomes more expressive . In terms of the bias-variance trade-off , the “ bias ” decreases as the class of functions becomes more expressive , but the “ variance ” or “ complexity ” increases . So-called universal approximation theorems [ Cybenko , 1989 , Hornik , 1991 , Anthony and Bartlett , 1999 ] show that even with a single hidden layer , i.e. , when the depth of the architecture is the smallest possible value , one can essentially reduce the “ bias ” as much as one desires , by increasing the size of the network or the number of neurons used in the neural network . Nevertheless , it can be advantageous both theoretically and empirically to increase the depth because a substantial reduction in the size can be achieved ; Telgarsky [ 2016a ] , Eldan and Shamir [ 2016 ] , Arora et al . [ 2018 ] is a small sample of recent work in this direction . To get a better quantitative handle on these trade-offs , 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . it is important to understand what classes of functions are exactly representable by neural networks with a certain architecture . The precise mathematical statements of universal approximation theorems show that single layer networks can approximate arbitrarily well any continuous function ( under some additional mild hypotheses ) . While this suggests that single layer networks are good enough from a learning perspective , from a mathematical perspective , one can ask the question if the class of functions represented by a single layer is a strict subset of the class of function represented by two or more hidden layers . On the question of size , one can ask for precise bounds on the size of the network of a given depth to represent a certain class of functions . We believe that a better understanding of the function classes exactly represented by different architectures will have implications not just for mathematical foundations , but also algorithmic and statistical learning aspects of neural networks . The task of searching for the “ best ” function in that class can only benefit from a better understanding of the nature of functions in that class . A motivating question behind the results in this paper is to understand the hierarchy of function classes exactly represented by neural networks of increasing depth . We now introduce more precise notation and terminology to set the stage for our investigations . Notation . We write [ n ] : = { 1 , 2 , . . . , n } for the set of natural numbers up to n ( without zero ) and [ n ] 0 : = [ n ] ∪ { 0 } for the same set including zero . For any n ∈ N , let σ : Rn → Rn be the component-wise rectifier function σ ( x ) = ( max { 0 , x1 } , max { 0 , x2 } , . . . , max { 0 , xn } ) . For any number of hidden layers k ∈ N , a ( k + 1 ) -layer feedforward neural network with rectified linear units ( ReLU NN or simply NN ) is given by k affine transformations T ( ` ) : Rn ` −1 → Rn ` , x 7→ A ( ` ) x+ b ( ` ) , for ` ∈ [ k ] , and a linear transformation T ( k+1 ) : Rnk → Rnk+1 , x 7→ A ( k+1 ) x . It is said to compute or represent the function f : Rn0 → Rnk+1 given by f = T ( k+1 ) ◦ σ ◦ T ( k ) ◦ σ ◦ · · · ◦ T ( 2 ) ◦ σ ◦ T ( 1 ) . The matrices A ( ` ) ∈ Rn ` ×n ` −1 are called the weights and the vectors b ( ` ) ∈ Rn ` are the biases of the ` -th layer . The number n ` ∈ N is called the width of the ` -th layer . The maximum width of all hidden layers max ` ∈ [ k ] n ` is called the width of the NN . Further , we say that the NN has depth k + 1 and size ∑k ` =1 n ` . Often , NNs are represented as layered , directed , acyclic graphs where each dimension of each layer ( including input layer ` = 0 and output layer ` = k + 1 ) is one vertex , weights are arc labels , and biases are node labels . Then , the vertices are called neurons . For a given input x = x ( 0 ) ∈ Rn0 , let y ( ` ) : = T ( ` ) ( x ( ` −1 ) ) ∈ Rn ` be the activation vector and x ( ` ) : = σ ( y ` ) ∈ Rn ` the output vector of the ` -th layer . Further , let y : = y ( k+1 ) = f ( x ) be the output of the NN . We also say that the i-th component of each of these vectors is the activation or the output of the i-th neuron in the ` -th layer . For k ∈ N , we define ReLUn ( k ) : = { f : Rn → R | f can be represented by a ( k + 1 ) -layer NN } , CPWLn : = { f : Rn → R | f is continuous and piecewise linear } . By definition , a continuous function f : Rn → R is piecewise linear in case there is a finite set of polyhedra whose union is Rn , and f is affine linear over each such polyhedron . In order to analyze ReLUn ( k ) , we use another function class defined as follows . We call a function g a p-term max function if it can be expressed as maximum of p affine terms , that is , g ( x ) = max { ` 1 ( x ) , . . . , ` p ( x ) } where ` i : Rn → R is affinely linear for i ∈ [ p ] . Based on that , we define MAXn ( p ) : = { f : Rn → R | f is a linear combination of p-term max functions } . If the input dimension n is not important for the context , we sometimes drop the index and use ReLU ( k ) : = ⋃ n∈N ReLUn ( k ) and MAX ( p ) : = ⋃ n∈N MAXn ( p ) instead . Since we deal with polyhedra a lot in this paper , we will use the standard notations convA and coneA for the convex and conic hulls of a set A ⊆ Rn . For an in-depth treatment of polyhedra and ( mixed-integer ) optimization , we refer to Schrijver [ 1986 ] . 1.1 Our Contribution . It is not hard to see that any function expressed by a ReLU network is a continuous and piecewise linear ( CPWL ) function , because one is composing continuous piecewise linear functions together . Based on a result by Wang and Sun [ 2005 ] , Arora et al . [ 2018 ] show that every CPWL function defined on Rn can be represented by a ReLU neural network with dlog2 ( n+ 1 ) e hidden layers . We wish to understand whether one can do better . We believe it is not possible to do better and we pose the following conjecture to better understand the importance of depth in neural networks . Conjecture 1.1 . For any n ∈ N , let k∗ : = dlog2 ( n+ 1 ) e. Then it holds that ReLUn ( 0 ) ( ReLUn ( 1 ) ( · · · ( ReLUn ( k∗ − 1 ) ( ReLUn ( k∗ ) = CPWLn . ( 1 ) Conjecture 1.1 claims that any additional layer up to k∗ hidden layers strictly increases the set of representable functions . This would imply that the construction by Arora et al . [ 2018 , Theorem 2.1 ] is actually depth-minimal . Observe that in order to prove Conjecture 1.1 , it suffices to find a single function f ∈ ReLUn ( k∗ ) \ ReLUn ( k∗ − 1 ) with n = 2k ∗−1 for all k∗ ∈ N. This also implies all remaining strict inclusions ReLUn ( i− 1 ) ( ReLUn ( i ) for i < k∗ since ReLUn ( i− 1 ) = ReLUn ( i ) directly implies that ReLUn ( i− 1 ) = ReLUn ( i′ ) for all i′ ≥ i− 1 . In fact , there is a canonical candidate for such a function , allowing us to reformulate the conjecture as follows . Conjecture 1.2 . For any k ∈ N , n = 2k , the function fn ( x ) = max { 0 , x1 , . . . , xn } can not be represented with k hidden layers . Proposition 1.3 . Conjecture 1.1 and Conjecture 1.2 are equivalent . Proof ( Sketch ) . We argued above that Conjecture 1.2 implies Conjecture 1.1 . For the other direction , one can argue that , if the specific ( n+1 ) -term max function fn can be represented by k hidden layers , then every other ( n+ 1 ) -term max function as well . The claim then follows via a result by [ Wang and Sun , 2005 ] stating that any f ∈ CPWLn can be written as linear combination of ( n+ 1 ) -term max functions . We provide a detailed argument in Appendix A . It is known that Conjecture 1.2 holds for k = 1 [ Mukherjee and Basu , 2017 ] . However , the conjecture remains open for k ≥ 2 . In this paper , we present the following results as partial progress towards resolving this conjecture . In Section 2 , we resolve Conjecture 1.2 for k = 2 , under a natural assumption on the breakpoints of the function represented by any intermediate neuron . We achieve this result by leveraging techniques from mixed-integer programming to analyze the set of functions computable by certain NNs . It is not hard to see that MAX ( 2k ) ⊆ ReLU ( k ) for all k ∈ N [ Arora et al. , 2018 ] , that is , any 2k-term max function ( and linear combinations thereof ) can be expressed with k hidden layers . One might ask whether the converse is true as well , that is , whether the classes MAX ( 2k ) and ReLU ( k ) are actually equal . This would not only provide a neat characterization of ReLU ( k ) , but also prove Conjecture 1.2 without any additional assumption since one can show that max { 0 , x1 , . . . , x2k } is not contained in MAX ( 2k ) . In fact , this is true for k = 1 , that is , a function is computable with one hidden layer if and only if it is a linear combination of 2-term max functions . However , in Section 3 , we show that for k ≥ 2 , the class ReLU ( k ) is a strict superset of MAX ( 2k ) . In this section , the key technical ingredient is the theory of polyhedral complexes associated with CPWL functions . This way , we provide important insights concerning the richness of the class ReLU ( k ) . So far , we have focused on understanding the smallest depth needed to express CPWL functions using neural networks with ReLU activations . In Section 4 , we complement these results by upper bounds on the sizes of the networks needed for expressing arbitrary CPWL functions . In particular , Theorem 4.4 shows that any continuous piecewise linear function with p linear/affine pieces on Rn can be expressed by a network with depth at most O ( log n ) and width at most pO ( n2 ) . We arrive at this result by introducing a novel application of recently established interactions between neural networks and tropical geometry .

Summary: The authors study the class of functions that can be represented by a fully connected neural network with ReLU activations. First, They conjectured that for any $k\in\mathbb{N}$, $n=2^k$, the function $f_n(x)=\max\{0,x_1,\dots,x_n\}$ cannot be represented by a fully connected network with $k$ hidden layers, and prove this conjecture for $k=2$ under some mild assumption. Second, they proved the class of functions that can be represented by a $(k + 1)$-layer NN is strictly larger than the class of functions that are linear combinations of $2^k$-term max functions. Finally, they provided a bound on the width of the NN required to represent continuous piecewise linear functions.