Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper studies a number of interesting phenomena in deep learning by characterizing the linear regions of fully connected ReLU networks. The advantage of FC ReLU networks is that they are piecewise linear, and so the overall function can be understood in terms of these linear regions. The paper characterizes both the break points (boundaries) and slopes of these linear regions, using them to shed light on loss surfaces, generalization, and training dynamics.

The paper has interesting analyses, but I think the main drawback is that the clarity and presentation could be improved.

In particular, while reading the paper, I found myself wanting:
- More discussion of connections to relevant work. There have been a few papers (e.g. https://arxiv.org/abs/1611.01491, https://papers.nips.cc/paper/5422-on-the-number-of-linear-regions-of-deep-neural-networks.pdf, http://proceedings.mlr.press/v80/serra18b/serra18b.pdf) that use linear regions to understand deep networks. Reading this paper, I do not get a good understanding of how the work presented here fits into this larger research context.
- More expository text for particular concepts. A number of results are presented, and it would be helpful to have brief high-level summaries of the main findings of each section after diving through technical details.
- A number of terms are used before they are defined. For example, "roughness" is used on pages 1 and 2 but not defined until page 3. I think adding a preliminaries section with central terms/definitions in the paper clearly laid out could help with the exposition.

Other comments
- Consider defining breakpoints and delta-slopes when they are introduced at the bottom of pg. 1.
- typo on pg. 1: by doing *so* using small widths
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper wants to answer the question what is the value of the neural network’s depth? The author targets at Deep ReLU Network and uses Continuous Piecewise Linear (CWPL) to analyze the network’s parameter distribution. The main contributions of this paper are as follows, (1) For common initializations, this paper proves a deeper model will lead to flatter approximations and better approximation over a broader range of inputs. (2) A deeper model performs better when one optimizes with (Gradient Descent) GD methods. (3) Flat Initialization in the overparameterized regime could explain generalization. They found that the value of depth in deep nets seems less about expressivity, but enable GD to find better solutions.

My decision is Weak Accept, considering the following aspects.
Positive points: (1) The theory seems solid, authors prove breakpoint and delta-slope distribution will influence by the depth of the network. (2) The conclusions of the paper are inspiring, e.g., depth makes it easier for GD to the optimizer.

Negative points: (1) For the experiment, the authors find breakpoints can’t migrate very far from their initial location. I hope the author could explain this phenomenon since it is very crucial to proving the importance of the breakpoint’s initial distribution. (2) Some formulas in the appendix parts are beyond the scope of the page. 

Suggestions: I think that a figure that shows breakpoint and input data distribution together will be very interesting. I want to see the breakpoint’s distribution change as training and its relationship with input data distribution.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: 
This paper proposes a functional characterization to understand the empirical success of deep neural networks. In particular, this paper focuses on the case of deep fully connected univariate ReLU networks, and show that the parameters will result in a Continuous Piecewise Linear (CPWL) approximation to the target function. Moreover, the authors derive the induced distributions of the function space parameters and show that increasing width can reduce the roughness of the initial function.

Besides, this paper analyzes the loss surface in the function space and reveals some relationship between the critical points in the function space and original NN parameter space. Furthermore, a type of gradient descent dynamic in the function space has also been derived.

Many experiments have been conducted to reveal how the expressiveness and optimization performance varies with the neural network width and depth.

Overall, the functional characterization is interesting can potentially help explain the generalization/expressiveness of deep neural networks. However, this paper is not well written and organized, and there are some “??'s” appearing on page 4. The authors should pay more attention to improving the writing and organization of this paper. 

Here are the detailed comments:

The statements of theorems are not clear. For example, Theorems 3 and 4 convey too much information, I believe the authors should simplify the statements to make them more concise.  Moreover, the purpose of these two lemmas is not clear. Do they imply something related to expressiveness?
There should be some discussion in the surrounding text of Theorem 5. Some notations are also missing. For example, what’s $\hat \epsilon(t)$?, what’s $a_i(t)$? Besides, what’s the purpose of Theorem 5. It seems that this Theorem is used in the main part of this paper.
One major drawback of this paper is that it only provides the theoretical analysis for two-layer networks, but the title and claimed contributions are related to deep ReLU networks. I wonder whether the theory can be extended to deep cases?
In Theorem 2, I think He initialization will give you $\sigma_w = \sqrt{2/(H+1)}$ and $\sigma_v = \sqrt{2/k}$, if the output dimension is k. In this way, I am curious whether Theorem 2 can still hold. Besides, what’s the meaning of the so-called “roughness“?
I don’t get the point in Implications of Corollary 1. For example, what do you mean “$f$ has significant curvature in the boundaries?”, why an initialization that allocates more breakpoints to the area where the curvature of $f$ lies can be faster to train? The authors should elaborate more on this.


After reading the authors' response.

Thanks for your response.  I still think the contribution of this paper is not enough as the theoretical analysis may not be able to be generalized to deep networks. Thus I would like to keep my score.
review_title: Official Blind Review #3

Meta-review: This article sets out to study the advantages of depth and overparametrization in neural networks from the perspective of function space, with results on univariate shallow fully connected ReLU networks and some experiments on deep networks. 
The article presents results on the concentration /dispersion of the slope / break point distribution of the functions represented by shallow univariate ReLU networks for parameters from various distributions. The reviewers found that the article contains interesting analysis, but that the presentation could be improved. The revision clarified some aspects and included some experiments illustrating breakpoint distributions in relation to the curvature of some target functions. However, the reviewers did not find this convincing enough, pointing out that the analysis focuses on a very restrictive setting and that that presentation of the article still could be improved. The discussion of implicit regularisation in section 2.4 seems promising, but it would benefit from a clearer motivation, background, and discussion. 
Decision: Reject