Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper introduces “stiffness”, a new metric to characterize generalization in neural networks. Stiffness is a pretty simple concept and is relatively straightforward to compute. The authors evaluate this metric on standard datasets using two relatively small neural networks. On the whole, the paper is written clearly and explains its methodology in simple language.

I have a few observations:
1. The equivalence between equation 2 and equation 3 is mentioned in passing but no explanation is provided. Th equivalence is not clear so I would encourage the authors to provide a short proof.
2.  Since stiffness depends on the gradients obtained on points in the input space, which in turn depends on the loss, why would a practitioner training a neural network turn to stiffness to diagnose overfitting instead of just looking at the values of the training and validation losses? Indeed, the authors themselves say that a network has overfitted when training and validation losses diverge. The paper fails to motivate why stiffness is better than just looking at losses during training. 
3. The authors mention “The train-val stiffness is directly related to generalization, as it corresponds to the amount of improvement on the training set transferring to the improvement of the validation set. ”. Typically, generalization is evaluated on a held out test set so I fail to understand what the authors mean by this statement. We would expect validation error to underestimate test error  so while they are related, train-val stiffness would not necessarily characterize generalization. It would be interesting to see a train-test stiffness graph to test the authors claim.
4. The paper fails to motivate the the utility of the concept of “Dynamical Critical distance”. Since the primary goal of  paper is to understand generalization, I would like the authors to clarify the motivation to study this quantity. What additional insight does this provide with respect to generalization?
5. The term “dynamical critical distance” is not used uniformly. For example, it is mentioned as “dynamical critical scale” in section 3.3 and “dynamical critical length” in section 4.2.
6. While the paper on the whole is written in a clear fashion, I found section 4.4 to be particularly confusing. The authors should consider rewriting that section to make it clearer.

In summary, the concept of stiffness seems to closely follow training and validation losses and any problem diagnosed using stiffness would therefore be also diagnosed via examining the loss values. This along with other concerns mentioned above mean that I cannot recommend this paper for publication.
review_title: Official Blind Review #5

final_score: rating: 3: Weak Reject
review_content: This paper introduces the concept of stiffness: a measure of the change in the loss of sample A due to a gradient step based on sample B. It analyses the expected dynamic for A, B samples from the same and different classes, as well as, samples from the train and test sets.

To better understand the dynamics of optimization in neural networks is an open and important problem and the paper is clearly motivated in this regard. The proposed method is straight forward and I am not aware of a similar method. 

In addition to that, the paper also introduces "dynamical critical length ξ" which is the stiffness of A, B samples based on the cosine similarity of the respective inputs (section 2.4). A linear estimator of when this length becomes 0 is also introduced. Confusingly this is also called the "dynamical critical length ξ" in section 4.2. Later on the term "dynamical scale ξ" and "dynamical critical scale ξ" seem to be used interchangeably. Figure 6 mentions the "critical length χ" on the y-axis which seems to be a typo as no such measure was introduced.

The equivalence between eq. 2 and the two parts of eq. 3 is not obvious. We'd appreciate if the authors would provide a proof of such. 

Overall, the paper is written in a simple language but paragraphs remain surprisingly hard to understand. An example of such is e.g. section 4.4: What do the authors mean by "characteristic distance" between two input points? What is "the typical scale of spatial variation" of a function? etc.

The paper concludes that:

1.) there is a link between generalization and stiffness
2.) stiffness decreases with the onset of overfitting
3.) "general gradient updates with respect to a member of a class help to improve loss on data points in the same class"
4.) "The pattern breaks when the model starts overfitting to the training set, after
which within-class stiffness eventually reaches 0"
5.) This is observed for different models on different datasets
6.) "we observed that the farther the datapoints and the higher the epoch of training, the less
stiffness exists between them on average"
7.) "the higher the learning rate, the smaller the ξ"

Verdict: Reject

The conclusions are self-evident. The paper fails to demonstrate the usefulness of stiffness and most results are expected and provide little to no insights into the optimization dynamics of deep neural networks. In fact, the reasoning in this paper is almost tautological (conclusions 1-6).

E.g. if the A, B samples used to compute stiffness are separately drawn from the train and test set then stiffness is a proxy for the difference between the train error and the test error after another gradient step. The authors then compute stiffness at different points of the optimization procedure and conclude that stiffness decreases when the network starts to overfit. Since overfitting is the point in training where train error and test error diverge it is obvious that this can also be observed with regards to "stiffness". Hence, the reasoning is circular.

Conclusion 7 is slightly different in that it observes that larger learning rates result in smaller ξ which, given the previous paragraph, we can rewrite into the statement "larger learning rates generalise better". This is a well known empirical observation and has been discussed thoroughly (e.g. on connection with flat and sharp minima or learning rate decay schedules). 

Disclaimer: This review was done on short notice. 
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This submission introduces a metric, termed stiffness, to evaluate the generalization capability of neural networks. The metric is novel and straightforward, it measures how stiff a network is by looking at how a small gradient step on one example affects the loss on another example. The authors study several configurations  on three small datasets. They demonstrate that stiffness is a useful concept for diagnosing and characterizing generalization. 

I give an initial rating of weak accept because (1) The paper is well motivated and well written. Studying generalization is important for neural networks. (2) It seems from experiments that stiffness is a useful metric to indicate models' generalization capability. However, I have a few concerns. 

First, the authors study several configurations like train-train, train-val and val-val. However, these configurations are still in-domain analysis, the data distribution is quite similar. It can not support author's claims well. Adding an experiment where domain gap is large will make the submission stronger, such as train-test, cross-dataset or challenging tasks like semantic segmentation. 

Second, the datasets being used are very small. I understand that for theoretically analysis, small datasets are quick to converge and easy to demonstrate. However, this submission focuses on generalization problem during transfer learning. Hence, it needs at least a bigger dataset, like ImageNet, to show it really works. 
review_title: Official Blind Review #1

Meta-review: While there was some support for the ideas presented, the majority of reviewers felt that this submission is not ready for publication at ICLR in its present form.

Concerns raised include lack of sufficient motivation for the approach, and problems with clarity of the exposition.
Decision: Reject