Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors proposed a new concept called "Model Gradient Similarity" (MGS), which can be used to analyze the gradient descent algorithm and the effect of explicit regularizations, and can be used to design new regularization schemes. The main contributions of this paper are as follows:

(1) The authors defined MGS as the dot product between two model-parameter gradient vectors from two different training inputs -- this can be viewed as a kernel. The authors did an analysis based on first-order Taylor decomposition, and derived a first-order approximation of how the model behavior would change after one update of gradient descent. In essence, the (approximate) amount of change is (the negative of) the learning rate, multiplied by the matrix vector product between the MGS kernel matrix and the loss-model gradient vector.

(2) The authors further developed two metrics to summarize the MGS, by using the trace and the determinant of the MGS kernel matrix computed from training data. These can be used to approximate variance and covariance of model gradients on training data, and provide summary statistics of MGS -- the smaller these metrics, the larger the MGS is.

(3) The authors tracked the MGS metrics during training for several regularization schemes, and discovered that all of them to some extent function by trying to inhibit the growth of MGS metrics (i.e., encouraging higher MGS among training samples).

(4) Finally, the authors designed explicit regularization schemes, one based on the trace and the other the determinant of the MGS kernel matrix on training data and showed that (1) both worked well on a toy example, and (2) the trace regularization can be applied to real-world datasets to improve both accuracy and robustness -- for computational reasons only trace is applied to real datasets.
strengths_and_weaknesses: Strengths:

+ The first-order Taylor decomposition analysis on gradient descent to show the effect of MGS on model outputs is very neat. Given that the learning rate is usually small in practice, the change in parameters after one update is generally small in norm, so the first-order analysis generally holds. Great work!

+ The summary metrics to quantify MGS are theoretically grounded and sound.

+ The observation that all explicit regularization schemes inhibit MGS metrics (trace and determinant) growth is also insightful, and sheds light on why regularization works.

+ The proposed MGS trace regularization is simple and effective.

Weaknesses:

There are not many weaknesses. If possible, I would love to see a mathematical analysis of how MGS is linked to regularization schemes such as weight decay -- but this can be future work.

There is one small typo that needs to be fixed:

Equation (2) contains an error after the "approximately equal" sign. It should be f_\theta(x'), rather than f_\theta(x).
questions: I do not have questions -- the paper is very well written and everything is explained clearly.
limitations: The authors have adequately addressed the limitations of their work, namely, the computationally intensive nature of gradient-based regularization schemes, and also in particular, of the proposed MGS determinant regularization.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: the submission proposed to use the trace of the kernel matrix, which is constructed by taking the inner product of a matrix that has gradients of function outputs w.r.t. parameters and the transpose of the matrix itself, to regularise the training of a neural network. The submission through experiments showed that their proposed gradient-based regulariser provided better generalisation than other existing regularisation approaches. 
strengths_and_weaknesses: 1. the title of the submission doesn't seem to reflect the content of the submission. 

The analysis and the proposed regulariser eventually don't convey much information on the similarity of gradients computed on a pair of data samples, since it is purely based on the l2 norm of gradient vectors on individual data samples. Even though the determinant of a matrix utilises the pair-wise similarity, as the submission mentioned as well, the difference between the effect of trace and that of the determinant is subtle.

2. I am not sure how the gradient is used when there are multiple outputs, for example, multi-class classification problems.

For classification problems with more than two output classes, the gradient of the function w.r.t. the parameter on each data sample is a matrix. It doesn't seem clear to me how the proposed approach uses the matrix. 

3. The proposed regulariser is similar to the Lipschitz constraint that has been used in Wasserstein GANs, and other regularisation schemes. 

The idea of Lipschitz constraint is to make sure that a small change in the input data sample only leads to a tiny change in the output, and the proposed approach in this paper, which computes the trace of the kernel matrix, regularises / penalises the norm of a gradient vector, which is essentially regularises the Lipschitz constant of the model locally.

A relatively thorough theoretical comparison can be found here: https://proceedings.neurips.cc/paper/2020/file/8929c70f8d710e412d38da624b21c3c8-Paper.pdf

4. In all experiments, I wonder if any early stopping is applied, and if not, I think early stopping could alleviate demonstrated overfitting issues. 

It is a little wild to me that weight regularisation and dropout performed very poorly, but I think realistically, early stopping is well-adopted on top of various regularisation schemes, and it could reduce the performance between the proposed approach and all other methods.
questions: See above. 
limitations: See above.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a novel view on regularization for neural networks. The core idea focuses on the similarity of model gradients when evaluating data points. The paper proposes a kernel that conveys the influence of one training sample has over the other. Two global metrics, the trace and the determinant, are used as proxy metrics to track the model generalization. They are also used as regularizers by including them in the optimization objective. Experimental results show that tracking and optimizing these metrics leads to models that retain a high generalization, even in the presence of noisy annotations for classification and regression problems.
strengths_and_weaknesses: - *Originality:* 
    - Most of the elements in the paper, in isolation, are not new: the topic of regularization in ANNs has been studied for the past few decades. Tracking or including gradients (or other statistics) of data-batches to add constraints to the optimization process has also been done in the past. The idea of measuring the influence between samples during learning has also been addressed before. (In this regard, I was missing a discussion to the work on influence functions from Koh and Percy [1].) However, the originality from this work comes from the unique way to combine these ideas. In particular, the use of the determinant and the trace as global metrics.

- *Quality*:
    - In general, the main claims of the paper are well supported. A weak point can be found e.g., in the motivation (L39-40) where, the fact that one thing (regularization) can be achieved in multiple ways, doesn’t immediately implies that it is not well understood. One could argue that the more understood a problem gets, the more ways one could find to make it work.

- *Clarity*:
    - MGS was never defined nor tied to any metric or formula. (Saying that it is a “framework” is still vague, and needs a more concrete definition.) Something like “any measure of similarity that expresses the influence that one sample has on the model” will make MGS more tangible. Alternatively, MGS could be represented by the kernel $K_{\theta}$, while $\text{tr}$ and $\text{det}$ are the proposed metrics to track MGS.
    - Plots are small and most axes are not labeled.
    - Introducing MGS regularizers in section 5, while already presenting results based on those in section 4 is rather confusing. I strongly suggest restructuring the development of the paper so that it follows a more sequential timeline.
    - For the abstract, please use `\begin{abstract}...\end{abstract}`.
    - L143: does FCN stand for fully-connected network? Make sure it is explained somewhere to prevent confusions with “fully *convolutional* networks.”

- *Significance:*
    - Experiments using AlexNet (Figure 3): how representative is AlexNet w.r.t. state-of-the-art DL? Can architectural components such as residual connections, batch-normalization or initialization methods (He initialization vs. truncated gaussians) that are not part of AlexNet, affect the outcome of these experiments?
    - For a better overview of the viability of MGS metrics, it is important to report the computational cost (memory and FLOPs) of using MGS compared to dropout, weight-decay, etc.

## References

1. Koh, Pang Wei, and Percy Liang. "Understanding black-box predictions via influence functions." *International conference on machine learning*
. PMLR, 2017.
questions: 1. Figure 2: regularization methods are dependent on hyperparameters. Dropout depends on the probability of dropping out, weight decay is controlled by a small coefficient $\alpha_{\theta}$, etc. Are these results robust to the choice of these hyperparameters?
2. In general: the different regularizations depend on hyperparameters that are proper of each method. Those that are included in the loss (loss grad and weight decay) are modulated by a hyperparameter $\lambda$, similar to the way $\alpha$ is defined for $\widehat{\mathcal{L}}(f(X), Y)$ (equations between lines 175-177). Dropout depends on the probability of an activation being dropped (usually $p=0.5$). What protocol was followed to guarantee that those hyperparameters were optimal? This is particularly relevant for section 6.3 where the other hyperparameters were analyzed.
3. How dependent are these results on the batch size?
4. Corrupted MNIST: Section 6.1 mentions the use of motion blur, but table 1 only talks about flipped labels. What is considered a corruption in this case and what is the motivation behind adding motion blur?
5. Could MGS (i.e., $\text{tr}~K_{\theta}$) be used as an early stopping criterion?
limitations: There is little discussion of the limitations of this work, even though there are a few points that could be addressed:

- Even though the method is well supported by theoretical findings, a discussion on the effects of hyperparameters is lacking. In particular, effects of the batch size or assumptions about the data distribution e.g., the test set exhibits no covariance shift.
- From the empirical results, the model architecture lacks representative (and relevant) components that are often found in modern ANN architectures (see second point on “Significance”).
- A discussion of the experimental hyperparameters (how they were found and how they affect the results) is also missing in the main paper.
- Finally, an account on the computational costs of MGS metrics and the other baselines should be presented.
review_title: null

Meta-review: metareview: This paper is controversial among the reviewers. On the positive side, reviewers like the novelty of the concept, the derivations and the clear presentation. The negative review wonders why the proposed method performs much better than dropout, similarity to Lipschitz constraints, and whether proper early stopping was used. The authors addressed some of the concerns, though the reviewer was not convinced. In the AC's opinion, the objections are sufficiently addressed and are not clear enough for rejection. One reviewer wanted to know more about the computational costs of using this technique and more discussions on the limitations.

The paper proposed a novel and interesting approach to regularization and seems to be a good contribution to the community.
confidence: Less certain
Decision: Accept