Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper conjectures that the so called Jeffreys prior over the parameters of a neural network is the prior leading to the best generalization performance. The authors test this conjecture on an artificial task using a small neural network, and investigate the sensitivity of the results to noise.

I like the general idea of the paper and appreciate the very detailed exposition placing it in the context of other works. In particular, I enjoyed the summary showing the sometimes conflicting evidence for better generalization in either broader or sharper minima, and how it relates to the Jeffreys prior.

However, as I understood the paper, the main claim in page 5 Equation 4 “Thus we conjecture that a correct prior for a model would be:” is an *assertion* that Jeffreys prior is the correct prior to use over the parameter space of neural networks. While it is a possibility, the amount of empirical evidence presented does not (at least to me) provide strong enough justification.

On page 7, you say “This model was a neural network composed of one, 5 neuron, hidden layer which utilized a sigmoid activation function in its hidden layer and a linear activation in its scalar output layer.“, describing your experiment. I don't think this experiment is sufficiently large to convince me.

Furthermore, in Figure 1 values outside the optical cluster at 0.0 appear nonetheless. I am not sure how to judge the amount of spread I see, and what effect they have on the performance of the network.

In general I would like to see experiments on datasets and with architectures that are at least somewhat close to what people use in practice (at least in terms of the size of the task and the capacity of the net). That would give me more confidence that your conjecture is true. While I appreciate your detailed theoretical exposition, I think the amount of empirical evidence you provide is insufficient to back the claims. Considering the explicit instruction to judge papers exceeding 8 pages with a higher standard, I believe that the lack of a greater amount of empirical evidence is a significant deficiency of your otherwise very interesting work.

I encourage you to expand this paper and resubmit to another venue -- I believe it has a great potential.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper argues that the widest minimum in the loss landscape is not the best in terms of generalization. The authors provide theoretical arguments and claim that there exists an optimal width beyond which generalization can be poor. Synthetic simulations are presented to support these claims.

The authors employ Fisher Information to characterize the optimal width or the curvature around the minimum. The fact that the determinant of the Fisher Information Matrix is invariant to parametrization, under certain conditions, serves as the motivation to design an objective Bayesian prior called Jeffrey's prior. 

The motivation and the theoretical arguments are interesting, but the paper lacks in presentation and sufficient empirical evidence is also lacking to get fully convinced by the claims. 

The authors should discuss the architecture design choices used for the synthetic data-generating model. Why are the last 3 layers of the larger model comprise of linear mappings?

Fig 1 is not clear. What does n=23 signify in the caption? More discussion is needed to describe "intersection of the likelihood values", "Difference in update Step" and "density is placed around 0" in section 5.



review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: This paper targets at a deep learning theory contribution based on information geometry. This contribution is tightly based on Zhang et al. (2018) and explains the generalization of deep learning from a Bayesian perspective. The main contribution the authors claimed is an optimal degree of curvature exist which gives the best generalization guarantees, which is in contrast to the commonly perceived "the wider the better".

First of all, the writing (including language etc) is of poor quality, to the extent that the submission is very difficult to read and can be rejected merely based on this, with unusual expressions, missing  punctuations, super long sentenses, and wongly used words. The reviewer won't list example here because they are everywhere.

What is even worse is the conceptral errors and defected derivations. For example, in eq.(1), the authors equate the Fisher information matrix (which is an expected Hessian) to the Hessian matrix, this is subject to conditions which must be clearly given right before/after the equation. As their results are largely based on the correctness of eq.(2), let's examine the derivations in appendix A.1.  In the first equation in A.1, what is the subindex "j"?  "Utilizing Laplace Approximation of the integral": such approximations have conditions that must be clearly stated. It is not clear how one can get the last approximation in page 12 from the previous equations. In summary, their eq.(2) is a loose approximation which is subject to a set of conditions (that are not given), and the derivation is of poor quality.

As a theoreiritical contirbution, the authors did not manage to converge to some simple and clear statements (theorems or equvalent). Instead, the contribution is largely *explanatory*. It is hard to observe anything new, given the poor writing and organization. The first 4 pages are mainly introductions of previous works.

The authors used information geometry and minimum description length to explain the generalization of deep learning. This is a small area. It is hard to miss closely related works by simple searching. Instead, the authors only cited Rissanen (1978). On the other hand, as the authors used the spectrum properties of the Fisher information matrix, there are some recent works by Amari which can be cited.
review_title: Official Blind Review #1

Meta-review: There has been significant discussion in the literature on the effect of the properties of the curvature of minima on generalization in deep learning.  This paper aims to shed some light on that discussion through the lens of theoretical analysis and the use of a Bayesian Jeffrey's prior.  It seems clear that the reviewers appreciated the work and found the analysis insightful.  However, a major issue cited by the reviewers is a lack of compelling empirical evidence that the claims of the paper are true.  The authors run experiments on very small networks and reviewers felt that the results of these experiments were unlikely to extrapolate to large scale modern models and problems.  One reviewer was concerned about the quality of the exposition in terms of the writing and language and care in terminology.  Unfortunately, this paper falls below the bar for acceptance, but it seems likely that stronger empirical results and a careful treatment of the writing would make this a much stronger paper for future submission.
Decision: Reject