Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Summary: The authors examine trajectory growth of deep ReLU neural networks whose weights come from a (random) “sparse-Gaussian”, “sparse-uniform”, and “sparse-discrete” distribution. They give definitions of these distributions in the paper. They do this by extending the proof of Raghu (2017) so that it can handle more general distributions than the standard Gaussian. They also provide some numerical experiments verifying their theories.

Strengths: The paper is well-written and the proofs are clearly explained. I’m grateful that the authors specifically mentioned where their proof deviates from the original and they clearly delineate how their proof method extends Raghu (2017)

Weaknesses: This is an interesting direction, but I do not believe there are enough results to constitute an accept. If the authors are following Raghu (2017), then I would have also liked to see analysis on trained networks as done in Raghu (2017) for example. I also think the title of the paper is too general for the specific results contained in the paper, namely sparsity should at least be mentioned in the title.
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This submission proposes an alternative way to lower bound the trajectory growth through random networks. It generalizes to a variety of weights distributions. For example, the authors showcase their approach on sparse-Gaussian, sparse-uniform, and sparse-discrete-valued random nets and prove that trajectory growth can be exponential in depth with these distributions, with the sparsity appearing in the base of the exponential. 

I give an initial rating of weak accept because (1) the paper is well written and well-organized. (2) the numerical simulation results support the claims and proofs. (3) the investigation on sparsely connected networks seems timely. However, I'm not an expert in this area. It also seems that most derivation and insights are from previous literature Raghu 2017, which makes the contribution of this submission limited. 

I have a question which may be invalid. For Figure 3, the observed expectation matches perfectly with the the lower bound for all three distributions. This seems amazing, have the authors try with other dataset or other settings to do this experiment? Did it always match perfectly? 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: In this paper, how the length of the trajectory in the input space is amplified by a ReLU neural network is analyzed. Specifically, the paper studied the case when the weights and biases are sparse random matrices. Some theoretical lower bounds are derived and are also empirically checked. 

Though the results are interesting, I am slightly lean to the rejection side. The main reason is that the motivation is not strong enough and it makes the entire work somehow incremental.

As described in the paper, the analysis of the trajectory length of NN has been initiated by Raghu et al. (2017). Although Raghu et al. considered the case of densely connected NNs, this work extended the notion to the sparsely connected NNs. I feel the extension is reasonable but it is not a big "jump" from the original work, and the originality is less significant. A good point of the derived results is that they are simple and easy to understand the dependency of the variance of the weights and the sparsity level. However, I am not so excited about these results because I cannot find a practical value from them. For example, (the current form of) Corollary 1 does not tell us how to control the sparsity level \alpha to maintain some accuracy-sparsity tradeoff. 

From the technical side, it is nice the proof is written in line by line. However, it looks everything is done in a normal manner and nothing special happens. Also, the condition of Theorem 2 is unclear. What does the "restriction" of \hat{w}_i mean? Does the condition E[] >= M ||u|| hold for any u and M? 

Finally, I have a simple question about the relationship between the scale of the output and the trajectory length. Let W be a weight and x be an input. When the smallest singular value of W is m, x is amplified at least by m, i.e., ||Wx|| >= m||x|| for any x. This means that (a part of) the trajectory growth is explained by the smallest singular value of the sparse random matrix W. Can you clarify the difference?
review_title: Official Blind Review #2

Meta-review: This article studies the length of one-dimensional trajectories as they are mapped through the layers of a ReLU network, simplifying proof methods and generalising previous results on networks with random weights to cover different classes of weight distributions including sparse ones. It is observed that the behaviour is similar for different distributions, suggesting a type of universality. The reviewers found that the paper is well written and appreciated the clear description of the places where the proofs deviate from previous works. However, they found that the results, although adding interesting observations in the sparse setting, are qualitatively very close to previous works and possibly not substantial enough for publication in ICLR. The revision includes some experiments with trained networks and updates the title to better reflect the contribution. However, the reviewers did not find this convincing enough. The article would benefit from a deeper theory clarifying the observations that have been made so far, and more extensive experiments connecting to practice. 
Decision: Reject