Depth-Width Trade-offs for ReLU Networks via Sharkovsky's Theorem

1 INTRODUCTION . In approximation theory , one typically tries to understand how to best approximate a complicated family of functions using simpler functions as building blocks . For instance , Weierstrass ( 1885 ) proved a general result stating that every continuous function can be uniformly approximated as closely as desired by a polynomial . It wasn ’ t until later that Vitushkin ( 1959 ) gave quantitative bounds between the approximation error and the polynomial ’ s degree . Drifting away from polynomials and given the recent breakthroughs of deep learning in a variety of difficult tasks like image classification , natural language processing , game playing and self-driving cars , researchers have tried to understand the approximation theory that governs neural networks . This question of neural network expressivity , i.e . how architectural properties like the depth , width or the activation units affect the functions it can compute , has been a fundamental ongoing challenge with a rich history . A classical result by Cybenko ( 1989 ) , Hornik et al . ( 1989 ) , Fukushima ( 1980 ) demonstrates the expressive power of neural networks : it states that even two layered neural networks ( using well known activation functions ) can approximate any continuous function on a bounded domain . The caveat is that the size of such networks may be exponential in the dimension of the input , which makes them highly susceptible to overfitting as well as impractical , since one can always add extra layers in their model aiming at increasing the representational power of the neural network . More recently , in a seminal paper by Telgarsky ( 2016 ) , it was shown that there exist functions that can be represented by DNNs , i.e , by some particular choice of weights on their edges ( and for a wide variety of standard activation units in their layers ) , yet can not be approximated by shallow networks unless they are exponentially large . More concretely , he showed that for any positive integer k , there exist neural networks with Θ ( k3 ) layers , Θ ( 1 ) nodes per layer , and Θ ( 1 ) distinct parameters which can not be approximated by networks with O ( k ) layers , unless they have Ω ( 2k ) nodes . At a high level , he uses the number of oscillations present in certain functions as a notion of “ complexity ” that distinguishes between deep and shallow networks ’ representation capabilities via the following three facts : a ) functions with few oscillations poorly approximate functions with many oscillations , b ) functions computed by networks with few layers must have few oscillations and c ) functions computed by networks with many layers can have many oscillations . Our main contribution is a novel connection between the theory of dynamical systems and the representational power of DNNs via the well-studied notion of periodic points , a notion that captures the important notion of fixed points of a continuous function . Definition 1 ( Period ) . We say that a ( continuous ) Lipschitz function f : [ 0 , 1 ] → [ 0 , 1 ] contains a point of period n ≥ 1 if there exists a point x0 ∈ [ 0 , 1 ] such that1 : fn ( x0 ) = x0 and ( point of period n ) fk ( x0 ) 6= x0 , ∀ 1 ≤ k ≤ n− 1 . In particular , all numbers in C = { x0 , f ( x0 ) , f ( f ( x0 ) ) , . . . , fn−1 ( x0 ) } are distinct , each of which is a point of period n and the set C is called a cycle ( or orbit ) of period n. Observe that since f : [ 0 , 1 ] → [ 0 , 1 ] is continuous , it certainly has at least one point of period 1 , which is called a fixed point . For the rest of this paper , we focus on ( continuous ) Lipschitz functions f : [ 0 , 1 ] → [ 0 , 1 ] , unless otherwise stated . Note that the choice of interval [ 0 , 1 ] is for simplicity of our presentation and that our results will hold for any closed interval [ a , b ] . As we observe , points of period 3 are contained in both Telgarsky ( 2016 ) and Schmitt ( 2000 ) constructions and this could as well have been a coincidence , however we show that the existence of periodic points of certain periods are actually one of the reasons explaining why depth is needed to represent functions that contain them ( otherwise exponential width is required ) . Towards this direction , we will make use of a deep result in the literature of iterated dynamical systems , that gave rise to the field of Combinatorial Dynamics ( Alseda et al . ( 2000 ) ) , called Sharkovsky ’ s Theorem ( Sharkovsky ( 1964 ; 1965 ) ) . 1.1 SHARKOVSKY ’ S THEOREM . Consider the set of positive natural numbers N∗ = { 1 , 2 , . . . } and define the following ( decreasing ) ordering . called Sharkovsky ’ s ordering as follows : 3 . 5 . 7 . · · · . ( odd numbers bigger than one ) .2 · 3 . 2 · 5 . 2 · 7 . · · · . ( odd multiples of two but not two ) .22 · 3 . 22 · 5 . 22 · 7 . · · · . ( odd multiples of four but not four ) ... . · · · . 24 . 23 . 22 . 2 . 1 ( powers of two in decreasing order ) . This is a total ordering ; we write l . r or r / l whenever l is to the left of r. Sharkovsky showed that this ordering describes which numbers can be periods for a continuous map on an interval ; allowed periods need to be a suffix of the Sharkovsky ordering : Theorem 1.1 ( Sharkovsky ’ s “ Forcing ” Theorem , Sharkovsky ( 1964 ; 1965 ) ) . Let I be a closed interval and f : I → I be a continuous map . If n is a period for f and n . n′ , then n′ is also a period for f . Remark 1.1 . Note that the number 3 is the maximum period according to Sharkovsky ’ s ordering , so an important corollary is that a function having a point of period 3 , must also have points of any 1As usual , fn ( x0 ) denotes the composition of f with itself n times , evaluated at point x0 . period . This special corollary is a weaker version of Sharkovsky ’ s theorem and was proved some years later2 in a celebrated result by Li & Yorke ( 1975 ) , who coined the term “ chaos ” as used in Mathematics . We conclude the subsection with the definition of a prime period of a function f . Definition 2 ( Prime period ) . A function f has prime period n as long as it has a cycle of period n , but has no cycles with period greater than n according to the Sharkovsky ordering . For example , in the interval [ 0,1 ] , the function f ( x ) = 1 − x has prime period 2 , since f ( f ( x ) ) = 1− ( 1− x ) = x so all points are periodic with period 2 , except the fixed point at 1/2 . Before formally stating our main theorems , we present an illustrative example inspired from Telgarsky ’ s triangle wave construction and we connect it to DNNs ’ sensitivity to weight perturbations and their representational power . 1.2 SENSITIVITY ANALYSIS - A MOTIVATING EXAMPLE . An important ingredient in Telgarsky ’ s proof , was the “ triangular wave ” function ( sometimes referred to as the tent map or sawtooth ) depicted in Figure 1b and given by : t ( x ; 2 ) = 2x , if 0 ≤ x ≤ 12 2 ( 1− x ) , if 12 < x ≤ 1 0 , otherwise He shows that the composition of t ( x ; 2 ) with itself k times ( denoted by tk ( x ; 2 ) ) , will create exponentially ( in k ) many oscillations and as a result he is able to show a separation for the classification error when using a shallow vs a deep neural network as a predictor . Our starting point is the observation that the triangular wave function t ( x ; 2 ) contains points of period 3 , e.g . ( 29 → 4 9 → 8 9 → 2 9 ) . It follows in particular , that t ( x ; 2 ) exhibits Li-Yorke Chaos ( Li & Yorke ( 1975 ) ) in the sense that it contains all periods . The compositions of such functions will look highly complex ( see Figure 2 ) and in fact Telgarsky heavily relied on the highly oscillatory behavior of t ( x ; 2 ) to prove his depth separation result . However , his result doesn ’ t inform us on what would happen if one used a slightly modified version of the triangle wave t ( x ; 2 ) . Observe that since a simple neural network with one hidden layer can represent the function t ( x ; 2 ) , the question is basically equivalent to asking how modifying the weights on the edges of the neural network can affect its representational power ( see Figure 1 ) , hence the title of the current subsection . The main question is can we have a general theory that informs us on when will the function composition be hard to represent and when not ? Our paper ’ s main point is to provide an answer by checking if the function at hand has a simple property , relating to the presence of chaotic behavior . To illustrate our point , consider the generalized triangle wave function t ( x ; µ ) parameterized by µ : t ( x ; µ ) = µx , if 0 ≤ x ≤ 12 µ ( 1− x ) , if 12 < x ≤ 1 0 , otherwise This function parameterized by µ ranges from [ 0 , µ/2 ] and is closely related to the logistic map f ( x ) : = rx ( 1− x ) used in Schmitt ( 2000 ) and exhibits a variety of limiting behaviors : for instance , it converges to a stable fixed point when µ ≤ 1 , it exhibits chaos when µ = 2 etc.3 Instead of µ = 2 , if we set µ = 1 , we get the network depicted in Figure 1c , 1d . 2Due to historical reasons during the late 20th century , the theory of dynamical systems saw a parallel development in the USA and the USSR , hence Sharkovsky ’ s theorem ( 1964 ) remained unknown in the USA , until in 1975 a weaker version was rediscovered by James Yorke and his graduate student Tien-Yien Li , in their celebrated paper called “ Period Three Implies Chaos ” . 3For more , the interested reader can also check https : //en.wikipedia.org/wiki/Logistic_ map . Note that compositions of t ( x ; 1 ) ( created by the same neural network architecture but with slightly different weights ) , behave completely differently since in the µ = 1 case , we will not get a highly oscillatory behavior . This can be seen in Figure 3 . One difference between the two cases is the relative position of the map with the line y = x and this seems to be pointing that fixed points and their generalizations i.e . periodic orbits play an important role when dealing with function compositions . Indeed , despite the wide range of possibilities one can expect by composing such functions , as we show , their behavior can be characterized using tools from dynamical systems ; the exponential growth in complexity ( or lack thereof ) of these compositions can be explained by invoking a fundamental property of these continuous functions on bounded intervals which is the existence ( or not ) of periodic points of certain periods . Similarly , we can argue about changing the parameters of the logistic map which is given by f ( x ; r ) : = rx ( 1 − x ) used in Schmitt ( 2000 ) for sigmoidal networks ( where f ( x ; 4 ) was used ) . The properties of the logistic map are well known and was first studied by Robert May and Mitchell Feigenbaum ( May ( 1976 ) and Feigenbaum ( 1976 ) ) . It is known that as one varies the parameter r , the logistic map gives rise to a plethora of different behaviors , hence the same is true for when one slightly perturbs the weights of a neural net used to represent the map . Please refer to Appendix C for some figures that illuminate these differences in the logistic map .

Summary: The paper studies how the expressive power of NN depends on its depth and width. Sharkovsky's theorem is leveraged to characterize the depth-width tradeoff in the ability of ReLU networks to represent functions with periodic points. A lower bound on the depth necessary to represent periodic functions is also provided. All in all, the paper furthers the understanding on the benefit of deep nets for representing certain function classes.