Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors establish a connection between sharpness-aware minimization (SAM) and a Bayes objective by using a relaxation of the latter, such that the expected negative-loss is replaced by a tight convex lower bound, which is shown to be equivalent to SAM. Moreover, they propose a Bayesian extension of SAM that improves both generalization and uncertainty estimation.
strength_and_weaknesses: Strength

Using the Fenchel biconjugate to bridge SAM and the Bayes objective is an interesting and insightful idea, and the resulting relaxation of the Bayes objective has shown good empirical performance. Moreover, the authors have conducted a variety of experiments and ablation studies on both toy and real datasets, providing deeper understanding of the proposed method.

Weaknesses
1. While the equivalence between SAM and the relaxed Bayes objective is interesting, it doesn't explain why the relaxed Bayes objective (or SAM) has better generalization performance than the original Bayes objective. In other words, it seems the "maximum loss" performs better than the "expected loss" despite the established connection, the reason of which is unclear.
2. The baselines used in the experiments are too weak. For instance, test accuracies around 95% and 75%, respectively, on CIFAR-10 and CIFAR-100 would be more convincing.
3. The readability of Sec. 2 needs improvement.
clarity,_quality,_novelty_and_reproducibility: This paper is well written in general, but clarity can be improved by simplifying the notation in Sec. 2, or perhaps including a notation table. The proposed method is novel and should be easy to reproduce.

In Eq. (4), f*(u) -> f(u).
summary_of_the_review: This paper provides strong theoretical results, and extensive experimental results. However, some of the main questions remain unanswered, and some of the experimental results are not convincing enough due to weak baselines.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a loss function that looks similar to SAM loss, but is a lower bound of Bayesian objective.
The authors state that the proposed pipeline can be used to obtain reasonable uncertainty estimates for NNs.
strength_and_weaknesses: Strengths:
- Numerical experiments, to some extent, justify the method proposed in the paper.
- The idea is nice and can lead to some insights about how SAM works (but sadly, more on that is in the weakness part)

Weakness:
- Actually, losses (12) and (1) are very different from each other, as in (1) we take the supremum over a neighbourhood and (12) we look at all space of possible values $\varepsilon$. So, the related term $l(m + \varepsilon)$ in (12) can be arbitrarily larger than the corresponding term in (1), making the whole approach inspired by Fenchel biconjugates but not equivalent to it. We also note that in Fenchel biconjuates, we search for a convex analogue for the function we optimize. This makes little sense in general deep learning, as the loss is very far away from convex with many symmetries and local optima observed.
- So, we can't directly optimize (12).
- The accuracy of this approximation is, thus, should be the main concern in the paper. This is not true now. The experiments follow a common pipeline for DL papers comparing to other approaches and making reasonable ablation studies.
- what is the difference between ResNet-20-FRN and ResNet-20? The paper [1] reports similar results for ResNet-20 in Tables 2 and 3 for different variants of SAM for CIFAR10 and CIFAR100. For CIFAR10, they obtain 93.82% accuracy (c.t. bSAM 92.16% in the paper). For CIFAR100 they obtain 71.40% (c.t. bSAM 68.22% in the paper). What is the difference in the protocol that provides such a decrease in quality for your approach compared to the baselines?

Suggestions:
- I suppose that the biconjugate for the Exponential family is worth deeper examination, as it leads to convex problems, and thus, everything should be correct and can lead to interesting theoretical insights and analysis (also, I suppose that you can generalize Theorems to the paper in this more general case). 


[1] Kwon, Jungmin, et al. "Asam: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks." International Conference on Machine Learning. PMLR, 2021.

clarity,_quality,_novelty_and_reproducibility: The paper is clear, given the complexity of the approach at hand, but some details are not given in enough detail.
Also, it would be better to:
- provide an explanation on the evaluation of the uncertainty estimate (maybe, I missed it in the text of the paper apart from ECE)
- direct link the baseline approach to the literature. Now the corresponding paragraph is vague and prevents exact attribution of e.g.
- The main Theorem 2 misses the list of assumptions used during the proof (the activity of the constraint, existence of the stationary point with zero derivatives given the hard constraint $i{\ldots}$. I assume that for the active constraint, the point would be a stationary one.
summary_of_the_review: There is little empirical evidence that the proposed approach is better than other. It has only limited theoretical justification. So, I lean towards the reject the paper in its current state.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper sheds light on the theoretical underpinnings of Sharpness Aware Minimization (SAM) by taking the biconjugation of the PAC-Bayes loss. The result is three-fold:

1. SAM can be understood as an optimal convex relaxation of the PAC-Bayesian loss.
2. This relaxation of the PAC-Bayes loss retains the logical sense sense in which it measures epistemic uncertainty. (The new loss being a looser bound.)
3. The PAC-Bayes relaxation is computed using convex optimization rather than Monte Carlo approximations.

Using the above insight, the authors modify SAM+Adam to fit a posterior mean and variance for Bayesian Neural Networks on standard image benchmarks.
strength_and_weaknesses: SAM is a SOTA optimizer which is poorly understood; any thoughtful story with appropriate empirical validation which explains how it works is a welcome contribution to the community. This paper easily exceeds that bar.  The mathematical details appeared correct and the prose was free from flaws or mistakes.

I only see one weakness in the paper which I share in the interest of constructive feedback and which in no way undermines my enthusiasm for the paper.

I would like to see much more study of the m-sharpness aspect of SAM, perhaps in a future work. M-sharpness was (seemingly?) only mentioned in the appendix. This is unfortunate because:

1. m-sharpness _further_ weakens the bound
2. m-sharpness leads to considerable test improvement.

The fact that weakening the bound improves results undermines the otherwise elegance of the paper's story. That is, we must now square the fact that an optimal convex relation of an otherwise defensible loss (von Neumann-Morgenstern and Savage utility theory / PAC-Bayes) is itself still somehow not optimal.  This is a hard pill to swallow because now that the loss is convex it is harder to lay blame to algorithmic concerns.
clarity,_quality,_novelty_and_reproducibility: As indicated the work was very clearly presented. Empirical findings were compelling and conducted on standard datasets using standard practices and architectures with ResNet sophistication or less. The precise algorithmic changes made to SAM+Adam were made abundantly clear in Algorithm 1.  Motivating biconjugation as an optimal convex relaxation of PAC-Bayes and for purpose of algorithmic development is, as far as I know, novel to that community.

A precise characterization of the algorithm with m-sharpness is missing; this reviewer kindly asks that it be added in at least the appendix.
summary_of_the_review: This reviewer strongly endorses this paper be accepted. The authors present a simple and enlightening connection between (vanilla) SAM and Bayesian methods. We believe such connections are critical toward getting ever closer to understanding how/why modern DNN techniques perform so well and aligning those empirical successes with the theoretical underpinnings of Bayesian methods.

While we believe the lack of attention to m-sharpness was disappointing, we nevertheless recognize this work as a significant step toward better understand SAM and hopefully, ultimately leading to even more significant gains.

We thank the authors for their remarkably well written presentation and their resolve to offer insight into one of the recent empirical success stories, SAM.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Sharpness-aware minimization (SAM) is a learning objective that improves upon maximum likelihood in terms of generalization, calibration and robustness of the resulting models. Intuitively, SAM drives the optimizer toward flatter minima, which are known to be desirable from the Bayes-optimal learning perspective. However, so far SAM has not been shown to directly correspond to any Bayesian learning objectives. In this paper authors derive this connection, showing that SAM is equivalent to minimizing a convex upper bound on a variational Bayesian learning objective. Authors then use this connection to propose an extension of SAM (Bayesian-SAM) that, in addition to the point-estimate of the parameters, estimates the parameter uncertainty, and leads to improved predictive performance and calibration in the numerical experiments.
strength_and_weaknesses: Strengths:
- A novel connection between two important ideas (SAM and Bayesian learning) that required a non-trivial amount of complex mathematical reasoning.
- A well-motivated, novel practical method driven by the theoretical results.
- Strong experimental results on a wide set of numerical benchmarks with a strong set of baseline methods.
- A clear, polished write-up with consistent notation. A precise comparison to the baseline SAM-Adam method with algorithmic differences highlighted line-by-line.
- Experiments to understand the sensitivity of the method to $\rho$, as well as additional ablation studies in the appendix.

Weaknesses:
- The toy experiment could be more informative: have authors considered doing non-linear classification with a NN instead? This might accentuate the differences between uncertainty estimates.
- I would recommend replacing the MNIST datasets in Table 1 with alternative, more challenging datasets. MNIST-based datasets are likely too easy for meaningful comparisons between methods, as indicated by more ambiguous results on these.
- A (brief) discussion on the computation cost of the method and how it compares to SAM/SAM-Adam would be useful.
clarity,_quality,_novelty_and_reproducibility: I rate all four aspects as excellent: the write-up is of high quality and clarity, the theory and the proposed method are novel, and plenty of experimental details are provided in the appendix.
summary_of_the_review: I strongly recommend accepting this paper. It's a rare example of a paper that theoretically connects two important ideas, _and_ uses this connection to derive a novel practical method that works well on real data. The theoretical connection is non-trivial, and the method is presented and evaluated well (albeit the evaluation could be made even stronger, see my suggestions above).
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper builds a beautiful connection between SAM and the Bayes objective, by showing that the former coincides with an optimal lower bound of the latter. The relaxation is constructed by replacing the expected negative-loss by its optimal convex lower bound using Fenchel biconjugate. As said by reviewer, and I echo that in strong agreement, "SAM is a SOTA optimizer which is poorly understood; any thoughtful story with appropriate empirical validation which explains how it works is a welcome contribution to the community". I believe this connection can open doors to better understanding the reasons behind SAM's success. Based on the proposed connection, the authors also introduce a modified version of SAM, namely bSAM, which can produce uncertainty estimates and also sometimes improve the accuracy of SAM. Two reviewers have rated the work 8 (accept) while the other two have rated borderline. I find the response to borderline reviews convincing though. In particular, in one case, the criticisms by the reviewer was due to a confusion about convexity claim which got clarified b the authors. Overall this paper has great contributions and interesting insights, and I recommend accept.


justification_for_why_not_lower_score: The paper has solid theoretical and empirical results simultaneously. Their theoretical result, which relates SAM to an optimal relaxation of Bayes objective is very insightful and may open new doors to understanding how SAM can improve generalization. On the empirical side, the resulted algorithm can not only provide uncertainty estimates for predictions, but sometimes also outperform SAM in terms of test accuracy. The paper is very well written and the ideas well illustrated by multiple graphs.
Decision: Accept: notable-top-5%