SUMMARY  This paper studies the expressive power of deep neural networks under various related measures of expressivity.  It discusses how these measures relate to the `trajectory length', which is shown to depend exponentially on the depth of the network, in expectation (at least experimentally, at an intuitive level, or theoretically under certain assumptions).  The paper also emphasises the importance of the weights in the earlier layers of the network, as these have a larger influence on the represented classes of functions, and demonstrates this in an experimental setting.   PROS  The paper further advances on topics related to the expressive power of feedforward neural networks with piecewise linear activation functions, in particular elaborating on the relations between various points of view.   CONS  The paper further advances and elaborates on interesting topics, but to my appraisal it does not contribute significantly new aspects to the discussion.   COMMENTS - The paper is a bit long (especially the appendix) and seems to have been written a bit in a rush.  Overall the main points are presented clearly, but the results and conclusions could be clearer about the assumptions / experimental vs theoretical nature.  The connection to previous works could also be clearer.   - On page 2 one finds the statement ``Furthermore, architectures are often compared via ‘hardcoded’ weight values -- a specific function that can be represented efficiently by one architecture is shown to only be inefficiently approximated by another.''   This is partially true, but it neglects important parts of the discussion conducted in the cited papers.  In particular, the paper [Montufar, Pascanu, Cho, Bengio 2014] discusses not one hard coded function, but classes of functions with a given number of linear regions.  That paper shows that deep networks generically* produce functions with at least a given number of linear regions, while shallow networks never do.  * Generically meaning that, after fixing the number of parameters, any function represented by the network, for parameter values form an open, positive -measure, neighbourhood, belongs to the class of functions which have at least a certain number of linear regions.  In particular, such statements can be directly interpreted in terms of networks with random weights.   - One of the measures for expressivity discussed in the present paper is the number of Dichotomies. In statistical learning theory, this notion is used to define the VC-dimension. In that context, a high value is associated with a high statistical complexity, meaning that picking a good hypothesis requires more data.   - On page 2 one finds the statement ``We discover and prove the underlying reason for this – all three measures are directly proportional to a fourth quantity, trajectory length.''  The expected trajectory length increasing exponentially with depth can be interpreted as the increase (or decrease) in the scale by a composition of the form a*...*a x, which scales the inputs by a^d. Such a scaling by itself certainly is not an underlying cause for an increase in the number of dichotomies or activation patterns or transitions. Here it seems that at least the assumptions on the considered types of trajectories also play an important role.  This is probably related to another observation from page 4: ``if the variance of the bias is comparatively too large... then we no longer see exponential growth.''  OTHER SPECIFIC COMMENTS  In Theorem 1  - Here it would be good to be more specific about ``random neural network'', i.e., fixed connectivity structure with random weights, and also about the kind of one-dimensional trajectory, i.e., finite in length, closed, differentiable almost everywhere, etc.   - The notation ``g \geq O(f)'' used in the theorem reads literally as |g| \geq \leq k |f| for some k>0, for large enough arguments. It could also be read as g being not smaller than some function that is bounded above by f, which holds for instance whenever g\geq 0.  For expressing asymptotic lower bounds one can use the notation \Omega (see
Output: en