Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: This paper addresses the following: how do batch normalization and dropout affect the number of linear regions present in a deep network? It does so by devising a search procedure for enumerating a set of linear inequalities that define the linear region around a particular input. The linear region is defined as the region of input space that activates the same units/nodes in the network. The authors compute these linear regions for three different types of fully connected networks trained with: vanilla (nothing added), batch norm, and dropout. Given these linear regions, the authors studied a number of their properties, such as the radii of inscribed spheres, angles between hyperplanes, and number of unique surrounding regions.

Comments
- This paper enumerates a number of interesting findings, all of which seem to raise intriguing questions about the properties of trained networks. However, after reading the paper, I am left a little unsure of what to make of the results. However, I do not think this is a fault of the paper, instead I enjoyed that this paper raises so many interesting questions. Still, some more discussion and interpretation of the results is perhaps warranted.
- I especially enjoyed the writing, the problem statement and exposition were clear and easy to follow.
- Perhaps the authors could comment, in the discussion, if they think their methods could be extended to networks with smooth nonlinearities (such as tanh), or what aspects of their results are also apply to networks with different nonlinearities.
- I was also curious if the authors could comment on similarities and differences between their findings and this relevant paper (https://arxiv.org/abs/1802.08760) by Novak et al. that empirically computes linear regions for 2D slices through input space.

Minor edits
- After introducing the definition of the insphere (eq 5), it would be helpful to remind the reader that this is for a particular region defined by the set of inequalities C^\*.
- Typo in footnote 2 on page 5: partitioned
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: First, I believe that the acknowledgements in the manuscript give identifying information which could stand in conflict with a double blind review process. I’ll leave it to the area chairs/program chairs to make a decision on this. The following review will be contingent on the fact that the authors did not break the submission rules.

This paper aims to give new insights into deep neural networks by presenting a number of approaches to analyse the linear regions in such networks. The authors define a linear region around a point x* as the intersection between a number of half spaces that are defined through linear approximations of a DNN (tangents) around that point x*. The authors show that points within these regions can be found using convex optimization with a number of linear constraints that are equal or less than the number of nodes in a DNN. In experiments with a fully connected network the authors analyse different properties of these linear regions: (1) How big is the biggest sphere that we can fit in a linear region? (2) How much do the hyperplanes that define a region correlate with each other? (3) How reliably does a linear region represent a single class? And (4), How does a linear region interact with neighbouring regions? In their presentation, the authors focus on comparing these properties between models that were either trained without regularisation, with batch normalisation, or with dropout and with different learning rates. This allows them to draw insightful conclusions about the difference between linear regions in these models. The authors hope that their work will enable new ways of analysing DDNs that will inspire new architectures and optimization techniques.

I vote to accept this paper. The authors present a large array of methods to analyse the linear regions of DNNs. Their insights into the differences of BN and Dropout are useful (figure 1 & 2) and sensible (figure 3). The implications of linear regions on adversarial robustness can have an impact in the future. Because the paper relies on geometrical reasoning, I wished there would be more visualisations that guide the reader.

Here are a number of comments and questions that I have on the manuscript:
- Figure 1 Top: What do the different colour represent in the linear regions plot?
- Section 2.1: maybe add a toy graph that visualises the depth-wise ‘exclusion’ process of feasible “neighbours” of x*?
- Eq. (2) & (3): Explain where these equations come from.
- Sec 3.2, first sentence. The authors claim that inspheres of linear regions are highly relate to the expressivity of DNN. Can they elaborate on that claim? Is this claim a result of their experiments?
- What is the relationship between the number of constraints in eq. (5) and the radius of an insphere? Does the insphere size decrease with more constraints? What implications would that have on deeper networks than the one that was presented?
- Why should distortion be a good measure of the size of a linear region?
- The authors claim that it is expensive to run their approach, and that they will aim to improve speed in the future. Can the authors give a more concrete example of runtimes in their current approach?

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This work presents an array of analytical tools to characterize linear regions of deep neural networks (DNNs). Using the tools the work analyzes the effect of dropout and batch normalization (BN) on the linear regions of trained DNNs; namely, by assessing the properties such as inspheres, orientation of hyperplanes, decision boundaries and relevance of surrounding regions, the authors highlight the differences and similarities of linear regions induced by vanilla SGD as compared to SGD with dropout or BN.   

The paper is clearly written and is easy to follow for the most part. The paper indeed presents a number properties for analyzing the nature of linear regions in DNNs; however it falls short of connecting them with an improvement in the optimization or interpretability of DNNs. 

Even with respect to providing support for general applicability, the work does not go very far: without enough variation in data (not just image benchmarks), tasks and architecture, it is hard to determine if the analysis tools presented in the paper generalize beyond the chosen setup. For instance just the optimization techniques compared in the paper have their own hyperparameters and it is not clear how the results might vary with them. 

I am not sure what to take away from figure 1 since it's only a two-dimensional slice of very high-dimensional input space. Maybe the authors could instead choose an example with a low-dimensional input space for illustration purposes.

Also, how much can be perceived from distributions shown in figure 2, since inradius (Eq. 5) may turnout to be a very coarse representation of linear regions, especially for deeper networks. Can the authors clarify this? Moreover, how would the figures look if we were using a different objective, dataset or architecture?
 
In figure 3, is it not possible to show the average results instead of just one example?
 
I would further like to know how the authors would deal with scalability issues if their analysis were to applied to more realistic (i.e. large) network architectures.
review_title: Official Blind Review #2

Meta-review: This paper studies the properties of regions where a DNN with piecewise linear activations behaves linearly. They develop a variety of techniques to chracterize properties and show how these properties correlate with various parameters of the network architecture and training method.

The reviewers were in consensus on the quality of the paper: The paper is well written and contains a number of insights that would be of broad interest to the deep learning community.

I therefore recommend acceptance.
Decision: Accept (Poster)