Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper describes a connection between flatness of minima and generalization in deep neural networks. The authors define a concept called "feature-robustness" and show that it is related to flatness. This is derived through a straightforward observation that perturbations in feature space can be recast as perturbations of the model in parameter space. This allows the authors to define a (layerwise) flatness measure for minima in deep networks (this layerwise flatness measure is also invariant to rescalings of the layers in neural networks with positively homogenous activations). The authors combine their notion of feature robustness with epsilon representativeness of a function to connect flatness to generalization. They present a few empirical evaluations on CIFAR10 and MNIST.

I believe this paper is able to once again confirm the relationship between flatness and generalization in an empirical manner with their layerwise measure of flatness. I am not so convinced about the theoretical justification that they claim to provide and thus do not recommend acceptance.

Theory - The key theorem relating generalization and flatness is Theorem 10 which says that if a compositional model is feature robust and the output of the first component is an epsilon-representative for the second component, then the compositional model will generalize. While this is interesting, it is not clear to me that this guarantees generalization for deep neural networks. This result only talks about feature robustness and representativeness for a particular layer. If a deep network has many layers, will the feature robustness layers closer to the input guarantee feature robustness at deeper layers? That might require a further unit operator norm constraint on the layer operator, which is a restriction on the types of weights that can be used. If a sample is epsilon representative at one layer, what is required for the next layer to be epsilon representative for the rest of the deep network? This seems to be a missing step in relating flatness/feature robustness of a layer to the generalization of the whole network.

Another idea that I think arises from Theorem 10 is that the flatness of loss landscapes is important when you have learning problems where the hypothesis class is compositional. While flatness is only spoken of in the case of deep neural networks, can we identify the same phenomenon in other problems? I would encourage the authors to try and identify another model in which the flatness-generalization relationship exists (even empirical evidence would suffice for now). This would strengthen the case for studying flatness and biasing optimization towards flatter solutions in the case of deep networks.

Experiments - This section seems to be pretty rudimentary, I would like to see more results on different kinds of network architectures (VGG? Inception? AlexNet?), more datasets (KMNIST? Fashion MNIST? SVHN?), and possibly more repetitions. At one point the authors mention that they declare a minimum has been reached if the training loss is < 0.07. Atleast on CIFAR10 and MNIST it is possible to achieve training loss <1e-4 so am not sure if the networks that the authors are testing are minima at all (It is important for them to be minima since the flatness measure is only defined at minima). Can the authors also identify more situations other than large batch vs small batch training that would lead them to obtain flatter/sharper minima?

The authors also claim that measuring generalization using test error is flawed, but do not provide details about their method of measuring generalization. I would want to see these details and a more thorough discussion of why measuring generalization through test error is flawed.

While this is an interesting paper, I do not believe it is ready for acceptance at ICLR 2020.


review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: This paper proposes a notion of feature robustness which is invariant with respect to rescaling the weight. The authors discuss the relationship of this notion to generalization.

The definition of feature robustness is interesting and could potentially be useful. However, the paper has the following major issues:

1- Related work: It seems that authors are unaware of the related work in this area. There are many relevant work in this area that connect feature or weight robustness to generalization look at [1,2,3,4] for some examples. I suggest authors to do a comprehensive literature review.

2- Theoretical results: The theoretical results presented in the paper have very limited value. For example, authors fail to really connect their suggested measure to generalization in any meaningful way. Instead they end up decomposing the test error to the sum of their robustness measure and the gap between robustness and test error which is trivial. I suggest authors to look at the literature on PAC-Bayesian and compression-based bounds to connect their suggested measure to generalization.

3- Experiments: The experiments are not really convincing. The empirical results show that the suggested measure can correlate with generalization when training with different batch-sizes. When varying other things, the measure is not really correlated. Therefore, this is not any better than the version suggested by Keskar et. al. Moreover, the experiments are very limited and I suggest authors to look at more controlled setting to verify the relationship of their measure to generalization. Also, when looking at the generalization, it is important to set the stopping criterion based on the cross-entropy instead of number of epochs.

[1] Dziugaite and Roy. "Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data". AAAI, 2017.
[2] Neyshabur et. al. "Exploring Generalization in Deep Learning", NeurIPS 2017.
[3] Arora et. al. "Stronger generalization bounds for deep nets via a compression approach". ICML 2018.
[4] Wei and Ma. "Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentation", NeurIPS 2019.

****************************
After author rebuttals:

Author have added discussion of related work which was missing in the original submission (thanks!). However, the other two issues are still present. On the theoretical side, I think the major issue is that the paper cannot connect the measure to generalization properly and ends up decomposing the test error to the sum of the robustness measure and the gap between test error and the robustness measure which is not informative. However, this would have been still interesting if the measure could go beyond other empirical measures. Unfortunately, the correlation only happens in case of changing batch size (and learning rate which we know is empirically equivalent to changing batch size) and therefore cannot go beyond what is shown in Keskar et al. 2017. Therefore, my evaluation remains the same.



review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper proposes a flatness measure that is invariant to layer-wise reparametrizations in ReLU networks. The notion of feature robustness, which is a notion the paper proposes, connects the flatness measure to generalization error.

This paper should be rejected because it is not well-placed in the literature. Similar notions with the proposed flatness measure have repeatedly appeared in the literature. This paper needs to discuss novel insights.

Major comments:
1) Many studies proposed or mentioned the flatness measures listed in Table 1 [1, 2, 3, 4]. One of the most relevant work will be [1]. It appears that the Fisher-Rao norm [1] has several advantages over the proposed measure in the submitted paper.
A) Fisher-Rao norm is invariant to a broader range of linear transformations.
B) Fisher-Rao norm does not rely on the Hessian, which is more suitable for non-smooth ReLU networks.
Additionally and importantly, the Fisher-Rao norm has a direct connection with the size of input gradients, which has a strong relationship with the feature robustness. It is strongly encouraged to discuss the connections and comparisons with the Fisher-Rao norm.
2) On the connection to the generalization error, Theorem 10 relies on the strong assumption defined in Definition 9. Given the high ability of deep networks to express many functions, assuming that \phi(S) is epsilon-representative seems difficult to justify. This paper should discuss why the assumption is reasonable. Otherwise, it is hard to claim that this paper connected the modified flatness measure to generalization error.

[1] Liang et al. "Fisher-Rao Metric, Geometry, and Complexity of Neural Networks." AISTATS 2019
[2] Achille et al. "Emergence of Invariance and Disentanglement in Deep Representations." JMLR 19 (2018)
[3] Neyshabur et al. "Exploring Generalization in Deep Learning." NeurIPS 2017
[4] Tsuzuku et al. "Normalized Flat Minima: Exploring Scale Invariant Definition of Flat Minima for Neural Networks using PAC-Bayesian Analysis." arXiv:1901.04653

=====

Update:

Thank you for the replies and the clarifications. They did address some of my concerns. However, the theoretical result is limited, and I do not think it provides clear connections of flatness and the local loss landscape. I think this paper is not ready for publication, and I keep my score.
review_title: Official Blind Review #1

Meta-review: The authors propose a notion of feature robustness, provide a straightforward decomposition of risk in terms of this robustness measure, and then provide some empirical evidence for their perspective. Across the board, the reviewers raised issues with missing related work, which the authors then addressed. I will point out that some things the authors say about PAC-Bayes are false. E.g., in the rebuttal the authors say that PAC-Bayes is limited to 0-1 error. It is generally trivial to obtain bounds for bounded loss. For unbounded loss functions, there are bounds based on, e.g., sub gaussian assumptions. 

Despite improvements in connections with related work, reviewers continued to find the theoretical contributions to be marginal. Even the empirical contributions were found to be marginal.
Decision: Reject