Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper introduces concept of "module criticality" to understand the role played by several modules in a model and how this affects the generalization of the models. This is quite an important problem to study as this helps to develop better understanding of the current architectures and potentially reduce their size without suffering the accuracy drop. This is a great theoretical contribution.

The paper studies this per module compared to previous works where the entire architecture is rewounded. The paper also studies this for ResNet models which are more widely/practically used than just the fully connected layers alone. This helps better understand the model as a whole. 

The authors do a good robust experimental study for different network initialization, various CNN models like ResNet18, 34, 101, VGG16 and also FCN. The results demonstrate the module criticality to be a good metric for generalization of models.

Overall, a good paper.
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: This paper introduces a new way to reason about neural network generalization using a module criticality measure. The measure is tangible and intuitive. It leads to some formal bounds on the generalization of deep networks, and is able to better rank trained image classification architectures than previous measures.

I am leaning to accept, as I expect this to be a significant theoretical contribution with several potential practical applications. With a few additional details, this could be a very strong submission:

(1)	Choice of module decomposition. Having each module be a single convolutional or fully-connected layer makes intuitive sense, but is there some theoretical motivation for this choice? If the only requirement for a module is that it includes some linear transformation, in the extreme, a module could consist of a single weight, or the entire network. Would those choices change the generalization bounds or relative criticality across different architectures?
(2)	Scope of experimental results. The ranking results would be much more compelling if they included a broader range of architectures, including more recent models with more branching, e.g., DenseNet. Is there some reason ResNet101 has higher generalization error than 18 and 34? Net. Criticality for ResNets is inversely correlated with the number of layers; is there an explanation for this? Is this true for other very deep models?
(3)	Practical use. To compute the criticality measure, we must train the model; but, if we train the model, we can compute generalization directly. So, what is the practical application of the measure? Is there some way it could be used to save computation? Could it help in the case of a small validation dataset, which we do not want to look at many times during model selection?

Minor typos:
-	Section 2.2: “An stable phenomena”
-	Section 2.3: “…an the…”
-	In appendix: “ResNet101: ResNet34 architectures…”

----------------------------

After rebuttal:

The authors have addressed my concerns, and I've increased my rating. There are still a few points I'd like to see addressed in the final version:

1. The fact that the approach cannot yet be applied to batch normalization is a big practical drawback. Some discussion of approaches you tried, why they didn't work, and possible future directions for overcoming this would be appreciated.

2. Clarify in the paper that the "PAC Bayes" approach used for comparison (Table 1) is a your method, i.e., an ablated version of criticality. As is, someone reading the paper quickly may think all you've done is add an alpha parameter to an existing "PAC Bayes" approach, which does fairly well on its own.

3. Visualizing the experimental tables as scatterplots could make them easier for a reader to interpret.



review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper builds upon the "module criticality" phenomenon and proposes a quantitative approach to measure this at the module and the network level.  A module's criticality is low if when it is switched to its initialization value, the error does not change drastically. 

The paper uses a convex combination of the initial weights and the final weights of a layer/module to define an optimization path to traverse. The authors quantitatively define the module criticality such that it depends on how much closer the weights can get to the initial weights on this path while still being robust to random permutations. The network critically is defined as the sum of the module criticality measure of all the layers. 

Empirical results on CIFAR10 show that the network's criticality is reflective of the generalization performance. For example, increasing resnet depth leads to improved generalization and low criticality. Though intuitively, it is not clear why moving closer to the initial values and thus lower average criticality indicated better generalization. It will be useful ot have a discussion on this issue. Results on other datasets will also be useful.

Overall, the network criticality measure appears a useful tool to predict the generalization performance compared to other measures such as distance from initialization, weight spectrum, and others. 
review_title: Official Blind Review #3

Meta-review: The paper analyses the importance of different DNN modules for generalization performance, explaining why certain architectures may be much better performing than others. All reviewers agree that this is an interesting paper with a novel and important contribution. 
Decision: Accept (Spotlight)