Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance
Authors: Haiquan Lu
Xiaotian Liu
Yefan Zhou
Qunli Li
Kurt Keutzer
Michael W. Mahoney
Yujun Yan
Huanrui Yang
Yaoqing Yang
Venue: NeurIPS 2024 poster (2024)
Keywords: Diversity
loss landscape
deep ensemble
Primary area: optimization_for_deep_networks
TL;DR: This paper reveals a trade-off between sharpness and diversity in deep ensembles, both empirically and theoretically, and proposes SharpBalance, a novel ensemble algorithm that achieves an optimal balance between these two crucial metrics.

Abstract: Recent studies on deep ensembles have identified the sharpness of the local minima of individual learners and the diversity of the ensemble members as key factors in improving test-time performance. Building on this, our study investigates the interplay between sharpness and diversity within deep ensembles, illustrating their crucial role in robust generalization to both in-distribution (ID) and out-of-distribution (OOD) data. We discover a trade-off between sharpness and diversity: minimizing the sharpness in the loss landscape tends to diminish the diversity of individual members within the ensemble, adversely affecting the ensemble's improvement. The trade-off is justified through our rigorous theoretical analysis and verified empirically through extensive experiments. To address the issue of reduced diversity, we introduce SharpBalance, a novel training approach that balances sharpness and diversity within ensembles. Theoretically, we show that our training strategy achieves a better sharpness-diversity trade-off. Empirically, we conducted comprehensive evaluations in various data sets (CIFAR-10, CIFAR-100, TinyImageNet) and showed that SharpBalance not only effectively improves the sharpness-diversity trade-off but also significantly improves ensemble performance in ID and OOD scenarios.

Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance
Haiquan Lu Xiaotian Liu2; Yefan Zhou?; Qunli Li
Kurt K Michael W. Mahoney Yujun Yan", Huanrui Yang'. Yaoqing Yang"
1 Nankai University
Dartmouth College
University of California San Diego
4 University of California at Berkeley
International Computer Science Institute
6 Lawrence Berkeley National Laboratory
Abstract
Recent studies on deep ensembles have identified the sharpness of the local minima of individual learners and the diversity of the ensemble members as key factors in improving test-time performance. Building this, our study investigates the interplay between sharpness and diversity within deep ensembles, illustrat- ing their crucial role robust generalization out-of-distribution (OOD) data. We discover trade-off between sharpness and diversity: minimizing the sharpness in the loss landscape tends to diminish the diversity of individual members within the ensemble, adversely affecting the en- semble's improvement. The trade-off justified through our theoretical analysis and verified empirically through extensive experiments. To address the issue of reduced diversity, we introduce SharpBalance, novel training approach that balances sharpness and diversity within ensembles. Theoretically, we show that our training strategy achieves a better Sharpeess-diversig trade-off. Empirically, we conducted comprehensive evaluations in various data sets (CIFAR-10, CIFAR-100, TinylmageNet) and showed that SharpBalance not only effectively improves the sharpness- -rrperss-diversity trade-off, but also significantly improves ensemble performance in ID and OOD scenarios. Our code has been made open-source."
both in-distribution (ID) and
1 Introduction
There has been interest in understanding the properties of neural networks (NNs) and their implications for robust generalization to both in-distribution (ID) and out-of-distribution (OOD) data [Hendrycks and Dietterich, 2019a]. Two properties of particular importance, sharpness (or flatness) [Granziol, 2020, Andriushchenko et al., 2023, Yang al., 2021, Dinh et al., 2017, Yao et al., 2020] and diversity [Laviolette et al., 2017, Fort al., 2019, Yao et al., 2020, Dietterich, 2000, Ortega et al., 2022, Theisen et al., 2023], have been shown to have a significant influence on performance. In context of deep ensembles [Ovadia et al., 2019, Lakshminarayanal Mehrtash et al., 2020, Ganaie et al., 2022], diversity (which measures the variance in output between independently-trrnn -trinncd models) is shown to be critical in enbancing ensemble accuracy. Sharpness, the other hand, quantifies the curvature of local minima and is believed to be empirically correlated with individual model's generalization ability.
al., 2017, Fort et al., 2019,
First four authors contributed equally. httpss//ggtuub.commhaiuuualll /Sarppalaaraaee
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

SharpBalance
Sharpness aware Minimization
Ensemble OOD Acc
SharpBalance (ours)
High diversity Low sharpness
75.5
76.5
77.5
High diversity High sharpness
0.08
Diversity
0.06
Diversity x
0.04
00000000880000oo
Ordinary deep ensemble
www
SAM
Low diversity Low sharpness
0.02
SAM (theary)
2-
Shorpness-Divern Diversity Trade-off
0.00
Shampalunce (theory)
0.5
0.6
0.7
0.8
0.2
0.4
one NN in a tereeemember ensemble.
Sharpness (upper hound)
Sharpness
(a) Overview
(b) Theoretical results
(c) Empirical results
Figure 1: (Sharpness-divesstt) trade-off and SharpBalance), (a) Caricature illustrating the sharpness-diversity trade-off that emerges an ensemble's ' loss landscape induced by the Sharpness- aware Minimization (SAM) optimizer. We propose SharpBalance to address this trade-off. Each black circle represents an individual NN in a three-membe ensemble. The distance between circles represents the diversity between NNs and the ruggedness of the basin represents the sharpness of each NN. (b) Theoretically proving the existence of the sharpness-dive diversssy trade-off and improvement from SharpBalam ance, plotting the analytic representation of sharpness and diversity from Theorem I and Theorem by changing the perturbation radius p SAM. SharpBalance ance achieves a larger di- versity for the same level of sharpness. (c) Empirical results of verifying sharpness-diversity trade-off improvement from SharpBalance. Each marker represents a three-member ResNet18 ensemble trained on CIFAR- 10. Diversity is measured by the variance of individual models' predictions, and sharpness is measured by the adaptive worst-case sharpness, both defined in Section 2.
Recent research on loss landscapes [Yang et al., 2021] highlights that single structural property of the loss landscape is insufficient to fully capture model's generalizability, and it underscores the importance of a joint analysis of sharpness and diversity. Despite significant efforts in studying sharpness and diversity individually, gap persists in understanding their relationship, particularly in the context of ensemble learning. Our work seeks to bridge this gap by investigating ensemble learning through the lens of loss landscapes, with specific focus on the interplay between sharpness and diversity.
cearpness-diversiie trade-off. Our examination of loss landscape structure for ensembling revealed a "trade-off" between the diversity of individual NNs and the sharpness of the local minima to which they converge. This trade-off introduces a potential limitation the achievable performance of the deep ensemble: the test accuracy of individual NN can be improved as the sharpness is reduced, but simultancously reduces diversity, thereby compromising the ensembling improvement (evidence in Section 4.2 and 4.4). This tradc-off is visually summarized in the lower transition branch in Figure la. We also developed theories (in Section to verify the trade-off. The theoretical results characterizing this phenomenon are visualized in Figure 1b, and the experimental observation is presented in Figure lc. In Section 4.2, we also verified the existence of the trade-off by varying the experimental setting to include different datasets and different levels of overparamettriizatio (e.g.. changing model width).
SharpBalance ancce mitigates the trade-off and improves ensembling performance. To address the challenge presented by the sharpness-diversity tradcoff, we propose a novel ensemble training method called SharpBalance. This method aims to simultancously reduce the sharpness of individual NNs and prevent diversity reduction among them, demonstrated in the upper transition branch of Figure la. This method is designed based on our theoretical results, which suggest that training different ensemble members using a loss function that aims reduce sharpness on different subsets of the training data can improve the trade-off between sharpness and diversity. Our theoretical results are summarized Figure 1b. Aligned with theoretical insights, our SharpBalance method lets each ensemble member minimize the sharpness objective exclusively on a subset of training data, termed the charpness-awand set. The sharpness-aware set of each ensemble member diversified by an adaptive strategy based on data-dependent sharpness measures. As shown in Figure 1c, we verify

---

that SharpBalance improves the charpness-dvvrsiit tradeoff' in training the ResNet18 ensemble on CIFARIO. We conducted experiments on three classification datasets show that SharpBalance boosts ensembling performance in ID) and OOD data.
Our contributions are summarized follows:
Comprehensive identification of the sharpness-diversio trade-off: This work provides a thor- ough examination of the phenomenon sharpness-diversity trade-off' where reducing the sharpness of individual models can decrease diversity between models within an ensemble. We demonstrate this effect through extensive experiments across various settings, using different sharpness and diversity measures, as well as different model capacities. Our findings show that this trade-off can negatively affect the ensemble improvements.
We prove the existence of the trade-off under a novel theoretical framework based on rigorous analysis of sharpness-aware training objectives [Foret et al., 2021, Behdin and Mazumder, 2023]. Our analysis borrows tools from analyzing Wishart moments [Bishop et al., 2018], and characterizes the exact dynamics of training, bias- bias-variance tradeoff, and the upper and lower bounds of sharpness. Notably, our novel theoretical analysis generalizes existing analysis to ensemble members trained with different data, which is the key to analyzing our own training method SharpBalance.
Novel theory:
Effective approach: To mitigate the sharpness-diversity trade-off, we introduce SharpBalance, an ensemble training approach. Our theoretical framework demonstrates that CharpBalance provably achieves improvements the sharpness-diversity trade-off by reducing sharpness while mitigating the decrease in diversity. Empirically, we confirm this improvement and demonstrate that SharpBalan ance enhances overall ensemble performance, outperforming baseline methods in CIFAR-I0, CIFAR-100 IKrizhevsky, 2009), TinylmageNet [Le and Yang, 2015], and their corrupted versions to assess OOD performance.
We provide more detailed discussion on related work in Appendix B.
2 Background
Preliminaries. We use a NN denoted as fe RR + Rd where 0 E R2 denotes the trainable parameters. The training dataset comprises data-label pairs D = {I y1) ..... (In, yn)}. The training loss of NN fe over a dataset D can be defined as (D = Σ E. (fo (x,) : y:). Here E(-) is a loss function, which, for instance, can be the cross entropy loss or l2 loss. We construct a deep ensemble consisting of m distinct NNs .e...... Jo.. For classification tasks, the ensemble's output is derived by averaging the predicted logits of these individual networks. We use flat ensemble to mean the deep ensemble in which each ensemble member is trained using a sharpness-aware optimization method [Foret et al., 2021]. differentiating it from other ensemble approaches.
Diversity metries. Distinct measures of diversity have been proposed in the literature [Laviolette et al., 2017, Fort et al., 2019, Dietterich, 2000, Baek et al., 2022, Ortega et al., 2022, Theisen al., 2023], and they are primarily calculated using the predictions made by individual models. Ortega et al. [2022] define diversity D to be the variance of model outputs averaged over the data-gener distribution, which we adopt in the theoretical analysis:
ID (()) (Vr((afoo
In our experiments, diversity is measured using variance defined above, as well as two other widely used metrics in ensemble learning, namely Disagreement Error Ratio (DER) [Theisen et al., 2023] defined in equation (2), and KI. divergence [Kullback and Leibler, 1951] defined in equation (11) in the appendices. We show in Section 4.2 that our main claim generalizes these three metrics in characterizing the diversity between members within an ensemble. Specifically, denote P as the distribution of model weights 0 after training. Then, the DER is defined as
DER Eg.00 ~p Dis [is )o
(2)
where Dis((JosJo') the prediction disagreement [Masegosa, 2020, Mukhoti et al., 2021, Jiang et al., 2022] between two classifier fe, fo, and E(fe) is the prediction error.
3

---

Sharpness Metric. In accordance with the definition proposed by Foret et al. [2021]. we char- acterize the first-order sharpness of model as the worst-case perturbation within radius of po- Mathematically, the sharpness x of a model f is expressed as follows:
K ((0 po)
max I
Lp(e + +) C
Empirically, we measure the sharpness of the NN via the adaptive worst-case sharpness [Kwon et al., 2021, Andriushchenko al., 2023]. The adaptive worst-case sharpness captures how much the loss can increase within the perturbation radius of 0:
max I|T | <|l=Spo
LD 0 + E) - LD(O),
where 0 = [o1...., Ot]. and Te diag (I ..... (0:)). sharpness "scale-frree,, that is, such that scaling operations on that do not alter NN predictions will not impact the sharpness measure.
Tg is a normalization operator to make
Ensembling. We characterize the effectiveness of ensembling by the metric called ensemble im- provement rate (EIR) [Theisen et al., 2023]. which is defined as the ensembling improvement over the average performance of single models. Let Eens denote the test error an ensemble; the EIR then defined as follows:
EIR Ea-p(E(So) - Eens
(4)
Eg-plE(fo
Sharpness Aware Minimization (SAM). SAM [Foret et al., 2021] ]] has been shown to be an effective
method for improving the generalization of NNs by reducing the sharpness of local minima.
essentially functions by penalizing the maximum loss within specified radius p of the current parameter 0, The training objective of SAM is to minimize the following loss function:
CD
CAMM =
max (D +) 2
(5)
where À is the hyperparameter of standard &g regularization term.
3 Theoretical Analysis of Sharpness-diversit Trade-off
This section theoretically analyzes the sharpness-diversity trade-off. The diversity among individual models is quantified using equation (1). The first theorem establishes the existence a trade-off between sharpness and diversity. The second theorem demonstrates that training models with only a subset of data samples leads to a more favorable trade-off between these two metrics.
Sharpness and Diversity of SAM. Assume the training data matrix A E R and test data matrix E The die are random with entries drawn from Gaussian ((00 J, I). Suppose the model weight at the O-th time step, Oo. is initialized randomly such that [[Oo] 0 and E[0,0%] I and updated with quadratic optimization objective through SAM. The learned weight matrix after & time steps is denoted as 0g. Let 0* be the teacher model (i.e., ground-truth model) such that AØ = ((A) and TØ* ((T), where (D) the label vector for data matrix D. Given perturbation radius po. the sharpness of a model after k iteration under the random matrix assumption is defined as
x(Ox: Po) = EA[, max f (Eo, [0&] + E; A) (Eo, [0x] A)],
which is the expected fluctuation of the model output after perturbation over the data distribution. For simplicity, we denote x( x(0AMM po) = x 9AM for the rest of the paper. We derive an explicit formulation of diversity and upper and lower bounds of sharpness for models optimized with SAM in Theorem 1. Detailed proof can be found in Appendix C.
Theorem 1 (Diversity and Sharpness of SAM), Let Oy be initialized randomly such that [[ O and E, E,0,0[] Iº1. Suppose
0% OSAM is the model weight after k iterations of training with SAM on A E Xde and evaluated T Xdu,, Let be the step size, p be the perturbation radius in SAM and be the radius for measuring sharpness SAM Then
D(0 =2kk,, 0)o²
Pd
2
dm -1)
Thr
+ POV
/ ((2k, 12 G SNEAM
SAM <
P
2
1 d
Btr
+1)
Pav (22k, 00

---

where
((i.)) : == +
Σ
kn kilk?!kss (-1)) 1)) box p
i!
Ftrr
m
m
din
m--
O + 1/dm))
dn
Fhu
Ar to ===
G
o( 20(2k.2)37210-111 112 and m = 62 + 2k3 + j. Nm. = (TL-)) is the Narayana number.
To provide a clearer understanding of the relationship between sharpness and diversity, Figure 2 presents a trade-off curve between these two metries. The estimated sharpness and diversity are displayed on the x: and y axes, respectively. Each point in the plot corresponds to a model trained using SAM with different p value, showcasing the outcome of varying perturbation radius. In these experiments, we evaluated the sharpness and diversity of the models empirically and compared them to the estimates obtained using Theorem 1. The soundness of Theorem I and tightness of the derived bounds are further supported by empirical evidence, as depicted in Figure 2. Further verification results supporting our theoretical analysis are provided in Appendix C.3
Training with Data Subsets. Assume A is par- titioned into S horizontal submatrices, such that A = [A ...... AS) We show in Theorem 2 a similar analysis of the sharpness and diversity of ensembles for which each model is trained with submatrix. Under this setting, we first selected a subset of data A, uniformly at random and then train the model with the selected subset with SAM.
Lower bound of sharpness (x-axis) Upper bound of sharpness (x-axis)
1000
Simulated
p.5.
Diversily
Lower Bound on Sharpmess
p=2.5e-1
10
Theorem 2 (Diversity and Sharpness when Models are Trained on Subsets). Suppose the training data matrix A is partitioned into S horizontal submatrices. Let Oo be initialized randomly such that E[Oo] = 0 and [[000]] = I Let SharpBal be the model weight trained with for l iterations on the sub- matrix A, R 3 xdu, selected uniformly random, and evaluated on test data T E xda, Let be the step size, p be the perturbation radius in SAH. po be the radius for measuring sharpness Kkk ny. Then r "Sha"
0
Upper Bound on Sharpness
<-p-0.1
10
100
Sharpness
Figure (Theoretical vs. Simulated sharpness-diversit trade-off). This figure illustrates the relationship between sharpness (upper and lower bounds) and diversity as predicted by Thereom 1 and as observed in simulations. Note that the upper and lower bounds correspond
SAM and
the sharpness values plotted along the x-axis, with the upper bound positioned to the right and the lower bound to the left. Also, note that the bounds provided are for the expected sharpness, which means that random fluctuations can cause the sim- ulation results to move beyond these bounds.
(10 SharpBal (2k, 0)o²
S-1 (0',2k, 0) o1(k,,))) |
+ S
and
SharpBal Kik
(pv) <
Par
+1
Pn
/0 S
2
V dm
where
C
2)0
+ +1L-1195551102222200225----11111 1) ()(1+))(( 1)(S - 2)0 210) 1)(S 2)k²(2k,
3
+ +3rS(s 1)(S 2) (k, 0)00 (k: 1)+ 1---PS(S1111 1)(S - -2)(S 3k,
(((i.) 1,=0 Σ
i!
to +ks pks
Tue Sdin
m
m
Sdh
m-
0(1
1 din
Nmils
1 =g== ko !kg!
Ahr
where m = k2 + 2kg + j. Nm, = (( is the Narayana number.
The proof of Theorem 2 provided in Appendix C.2, Similar experimental validations are conducted to verify Theorem 2, with results also presented in Appendix C. The main insight from Theorem 2 is
5

---

that training models on a randomly selected data subset offers a better trade-off between sharpness and diversity compared training on the complete dataset. This idea further illustrated in Figure lb, where we compare the sharpness upper bound and diversity of models trained on the full dataset (labeled as SAM) and those trained on subsets (labeled as SharpBalance), The results demonstrate that SharpBalare annce achieves more favorable trade-off. For given level of sharpness, deep ensembles with models trained on subsets of the data exhibit higher diversity compared to those trained on the entire dataset. This indicates that minimizing sharpness on randomly sampled data subsets for each model within the ensemble promotes the diversity among the models, thereby enhancing the caarpness-divesiie trade-off.
4 Experiments
In this section, we describe experiments. In particular, following Section 4.1 where we describe our experimental setup. in Section 4.2, we provide an empirical evaluation across various datasets explore the trade-off between sharpness and diversity. We also examine how this trade-off changes with different levels of overparameterization. Then, in Section 4.3 and 4.4, we elaborate the SharpBalance algorithm and compare its performance with baseline methods.
4.1 Experimental setup
Here, we describe the experiment setup for Section 4.2. Each ensemble member is trained individually using SAM with a consistent perturbation radius p. as defined in equation (5). We adjust across different ensembles to achieve varying levels minimized sharpness. Sharpness for each NN
was measured using the adaptive worst-case sharpness metric, defined in equation (3). The sharpness measurement was done on the training set, using 100 batches of size The diversity between NNs is measured using DER defined in equation (2). The diversity between ensemble members tested on OOD data. We evaluated this trade-off using a variety of image classification datasets, including CIFAR- 10, CIFAR- 100 [Krizhevsky, 20091, TinylmageNet [Le and Yang, 2015], and their corrupted versions [Hendrycks and Dietterich, 2019b]. For the setup of Section 4.4, we used the same datasets and architecture. The hyperparameters of the baseline methods has been carefully tuned. The hyperparameters for conducting the experiments are detailed in Appendix D.
Ensemble Improvement Rate
Ensemble Improvement Rate
Ensemble Improvement Rate
Diversity (Variance)
0.03
0.04
0.05
0.03
0.04
0.05
0.03
0.04
0.05
Diversity (DER)
Diversity (KL)
0.4
0.6
4-
0.2
2-
0.4
0.2
0.4
0.6
0.2
0.4
0.6
0.2
0.4
0.6
Sharpness
Sharpness
Sharpness
(a) Measuring diversity via Variance
(b) Mcasuring diversity via DER
(c) Measuring diversity via KL
Figure 3: (Varying diversity measure in empirical study). Three different metrics are employed to measure the diversity of individual models within an ensemble, i.e., Variance in equation (1), DER in equation (2), and KL divergence in equation (11). The results of the three metrics show consistent trends, demonstrating the sharpness-diversee trade-off: lower sharpness is correlated with lower diversity. The experiment conducted by training three-member ResNet18 ensemble on CIFAR10.
4.2 Empirical validation of Sharpness-diversity trade-off
We provide empirical observation to validate and explore the sharpness- charpness-diversity trade-off. Figure presents the validation of observing the trade-off phenomenon on training ResNet18 ensembles on CIFARIO 10 applying three different metries to measure the diversity. The results demonstrate that this trade-off phenomenon generalizes to the three diversity metrics defined in Section 2. Figure 4 presents the validation on three different datasets. In the following empirical study, DER will be the primary metric for measuring diversity of models.

---

Ensemble Improvement Rate
Ensemble Improvement Rale
Ensemble Improvement Rate
0.03
0.04
0.05
0.04
0.05
0.9
0.035
0.040
0.045
Diversity
0.6
Diversity
Diversity
0.7
0.8
Ensemble improvement decrease while diversity decresses.
Ensemble improvement decrcase while
Ensembll improvement decrease while diversity decrenses
0.6
0.4
diversity decreases
0.7
0.2
0.4
0.6
2.0
2.5
3.0
Sharpness
Sharpness
Sharpness
Individual OOD Acc
Individual OOD Acc
Individual OOD Acc
74.75
75.00
75.25
75.50
0.8
48.0
49.0
49.5
0.9
0.27
0.28
Diversity
0.6
Diversity
Diversity
0.7
0.8
Individual performance increissc while sharpness decreases
0.6
Indivxlual perfornance increase whill sharpness decrcases
Individual performance increase while shapposs decrenses
0.4
0.7
0.2
0.4
0.6
0.5
1.0
2.0
2.5
3.0
Sharpness
Sharpness
Sharpness
(a) CIFAR-10
(b) CIFAR-100
(c) TinyImageNet
Figure 4: (Empirical observations of sharpness-diversity trade-off). The identified trade-off
shows that while reducing sharpness enhances individual model performance, it concurrently lowers diversity and thus diminishes the ensemble improvement rate. First row: the color encoding represents the ensemble improvement rate (FIR) defined in equation (4), from red to blue means ensembling improvement decreases. Second row: the color encoding represents the individual ensemble member's OOD accuracy, from blue to red means individual performance becomes better. Each marker represents a three-member ResNet18 ensemble trained with SAM with a different perturbation radius.
Experimental results obtained with the other two metrics are available in Appendix E. The three sets of results first verify that minimizing individual member's sharpness indeed reduces diversity. This confirmed by the consistent trends of markers moving from upper right to lower left. Second, the first row of Figure 4 shows that an ensemble with decreased diversity (lower in y-axis) shows lower ensemble improvement rate (from red to blue), highlighting the negative impact of this trade-off. Lastly, the second row shows when the sharpness of the individual model is reduced (lower in ær-axis, the individual model's OOD accuracy is improved (from blue to red), demonstrating the benefits of minimizing sharpness. We verify the robustness of the phenomenon by measuring the sharpness and diversity using different metries in Appendix E.
Figure illustrates the trade-off curves as the overparameterivation level of the model is adjusted by changing width or sparsity (introduced using model pruning). This visualization confirms that the trade-off is a consistent phenomenon across models of different sizes, and the ensemble provides less improvement (blue color) at the lower left end of each trade-off curve. It also highlights that models with smaller or sparser configurations show a more significant trade-off effect, as evidenced by the steeper slopes and higher coefficient values of the linear fitting curves. As sparse ensembles are now being used to demonstrate the benefits of ensembling for efficient models [Liu et al., 2022, Diffenderfer et al., 2021, Whitaker and Whitley, 2022, Kobayashi et al., 2022, Zimmer et al., 2024], addressing the conflict between sharpness and diversity becomes particularly crucial.
4.3 Our SharpBalame ance method
Here, we describe the design and implementation of our main method, SharpBalance. Figure 6 provides an overview. Our approach is motivated by the theoretical analysis in Section 3, which suggests that having each ensemble member minimize sharpness on diverse subsets of the data can lead to a better trade-off between sharpness and diversity. SharpBalance aims to achieve the optimal balance by applying SAM to carefully selected subset of the data, while performing standard optimization on the remaining samples. More specifically, for each ensemble member NN fo., our method divides the entire training dataset D into two distinct subsets: sharpness-ave set D SAM and normal set Di Nommal The model is trained to optimize the sharpness reduction objective on D SAM
7

---

Ensemble Improvement Rate
Ensemble Improvement Rate
0.8
0.04
0.06
0.07
0.8
0.04
0.05
0.06
0.07
Walth-134
Width-44
Spansity-
Sparatt--...
0.7
Walth-16
WiddeeR
0.7
Spanityy0. 95
Spureity 29
Diversity
0.6
$-0.28
Diversity
0.6
p-0.33
0.5
0.5 +
B-0.55
5-0 58
7-0.50
0.4
pro.54
0.4
8-0.4
B-0.47
0.1
0.2
0.4
0.8
1.6
3.2
0.2
0.4
0.8
1.6
3.2
Sharpness
Sharpness
(a) Varying model width
(b) Varying model sparsity
Figure (Sharpnes-diieessity trade-off in models varying overparameterization levels). Different types of markers represent models with varying degrees of overparameterization. determined by changing the model width (a) or sparsity (b). Each marker represents a three-membin ensemble trained with SAM with a different perturbation radius. The / reflects the rate of decline in the trade-off curve, calculated via applying linear fitting over the ensembles at each level of overparameterization. A higher 3 points to steeper decline in the trade-off. Ensembles with narrower widths or increased sparsity display more pronounced trade-off effects. The model used in ResNet18 and the dataset is CIFAR-10.
while optimizes the normal training objective on Diormal- These training objectives are denoted as CSAM (0,) and CD Da (0.), respectively. The D'AM is selected by an adaptive strategy from the whole dataset D: is composed of the union of samples that deemed "sharp" by all other members of the ensemble except the i-th. Specifically, for each model, we pick the subset of data samples with sharpness." Then, take the union of all such subsets expect
the top-k% highest peer-daa-sammpn the i-th for creating the subset DAMM This partition data samples can be efficiently computed in parallel there is no sequential dependency the training of the ensemble members. However, SharpBalanc Banceee can be easily adapted for sequential training if memory constraints permit training only model time.
Ensemble
Dataset D ((0), (1p...)......e.. M..)
immmiiation
objective
CD
(0c)
fo.
sharpness roduction objective
C SAM (0,) Di
DAAM Dartmal
Top
fe..1
Top- 1%
fa,
Union of Top- 8%
par-data-sample sharpness
Figure 6: (System diagram of SharpBalance). Each ensemble member fa, optimizes the sharpness reduction objective on subset DSSM and the normal training objective on ' D'SAM is formed by selecting data samples from D that significantly affect the loss landscape sharpness of other ensemble members.
Per-data-sample sharpness. This metric is designed efficiently assess the sharpness of model for individual data samples. For each data point ( sharpness is quantified using the Fisher Information Matrix (FIM), which is expressed as V Vol(fo(zz) y, (o
)T Following well-established approach [Bottou al., 2018), we approximate the trace of the FIM by computing the squared & norm of the gradient: [Vol(f((a) worst-case sharpness, trace of the Hessian, or Hessian eigenvalues, are computationally slightly more expensive to approximate [Yao et al., 2020, 2021]. but are expected to lead to similar results.
l1 Other common sharpness metrics, such as

---

4.4 Empirical evaluation of SharpBalance
Average of individual performance
Ensembling performance
Tost Accuney
52-
Ancunnyy
30-
50
48
Deep Esscmble
SAM
Sharpladance
Deep Enacmblle
SAM
SharpBallnce
B
Emeenble
SAM
StarpBallazee
(a) CIFARIO-C
(b) CIFARIOO-C
(c) Tiny
96.5
8n
That Arouncy
66.0
Tox Accuraty
2028
70-
R5.5
Tox 68
28-
Derp Ennemble
SAM
Shrppwiance
Deep Ennemblee
SAM
Sharpllabmne
Deep Froembie
SAM Sharplillene
(d) CIFARIO
(e) CIFAR100
(f) TinyImageNet
SharpBalance improves the overall ensembling performance and mitigates the reduced ensembling improvement caused by sharpness-diversse trade-off). The three-member ResNet18 8 ensemble is trained with different methods on three datasets. The first row reports the OOD accuracy and the second row reports the ID accuracy. The lower part of each bar with the diagonal lines represents the individual model performance. The upper part of each bar represents the ensembling improvement. The results are reported by averaging three ensembles, and each ensemble is comprised of three models.
Figure 7: (Main results:
We evaluate SharpBalance by benchmarking it against both a standard Deep Ensemble, trained using SGD, and Deep Ensemble enhanced with SAM. The results are presented in Figure for CIFAR-10, CIFAR-100, and TinylmageNet. The comparison between the middle and left bars shows that SAM enhances individual model performance by reducing sharpness. However, this reduction in sharpmess also diminishes the overall ensemble effectiivnness by lowering diversity, exemplifying the seappeess-diversii trade-off discussed in Section 4.2. Further comparison between the right and middle bars shows that SharpBalance maintains or improves individual performance while improving ensemble effectiveness.
We also evaluate SharpBalance on different ensemble sizes. As shown in Figure 8, SharpBalance demonstrates more pronounced empirical improvements the number of ensemble models increases. The accuracy difference between SharpBalance ancee and the baseline methods becomes more significant, especially on corrupted data. Specifically, SharpBalance outperforms the baselines by up to 1.30% when ensembling 5 models on CIFAR 100-C dataset.
To further evaluate SharpBalance, we provide corroborating results in Appendix F, which includes:
We evaluate SharpBalance on different severity of the corruption on CIFARIO-C, CIFAR100-C and Imae-em SharpBallhaaec increasingly outperforms the baselines the severity of the corruption increases. We also evaluate the proposed method using uncertainty metrics such as negative log-likelihood and expected calibration error.
We further evaluate SharpBalance on other model architectures and tasks, such WideResNet, ViT, and ALBERT [Lan et al., 2020] on language tasks.
We compare our method of measuring sharpness with another method of measuring the curvature of the loss around a data point [Garg and Roy, 2023] and show the strong correlation between these two methods.
We further compare SharpBalanc ance with ensemble baseline EoA [Arpit et al., 2022], an improved version of SAM (for which individual models in ensemble are trained with different values) and GSAM [Zhuang et al., 2022]. Results show that SharpBalance can significantly outperform the baselines.

---

Deep ensemble SAM
SharpBalance
CIFRRIO
COFAR
CIFARIO-C
CIFAR
96.50
Acqury
96. 96.25
Accurny
80-
Accura
Accuryy
52
96.00
3
4
6
2
4
5
3
4
6
3
4
6
Number of models
Number m'modlls
Number models
Number m'mddlls
CIFARIO
CIFARI00
CIFARIO-C
100-C
0.02
0.03
0.06
EIR
0.006
EIR
EIR
LIK
0.02
0.04
0.004
0.01
2 3
4
5
2
5 Number of models
3
Number of models
Number models
Number models
Figure 8: SharpBalance achieves more pronounced improvement when increasing the number of ensembling models. "EIR" represents the ensemble inprovement rate, which is defined in Section 2, the larger the better. z-axis represents the number of individual models in one ensemble.
We demonstrate that, compared to training deep ensemble with SAM, our method adds only minimal computational cost. The extra time complexity is dominated by the computation of Fisher trace for evaluating per-sample sharpness, which empirically increases the training time by 1%.
Conclusion
Our theoretical and empirical analyses demonstrate the existence of a sharpness-diversit trade-off when cuuppess-minimization training methods are applied deep ensembles. This leads two main insights that are relevant for improving model performance. First, reducing the sharpness in individual models proves to be beneficial in enhancing the performance of the ensemble as a whole. Second, the accompanying reduction in diversity suggests that popular ensembling methods have limitations, and also highlights the potential for more sophisticated designs that promote diversity among models with lower sharpness. These results are particularly timely, given recent theoretical work on characterizing ensemble improvement [Theisen et al., 2023]. response to these findings, we have proposed SharpBal ance, which "diagnoses" the training data by evaluating the sharpness of each sample and then fine-tunes the training of individual models to focus on a diverse subset of the sharpest training data samples. This targeted approach helps maintain diversity among models while also reducing their individual sharpness. Extensive evaluations indicate that SharpBalance not only improves the sharpness -iversssiy trade-off but also delivers superior OOD performance for both dense and sparse models across various datasets and architectures when compared to other ensembling approaches.
Limitations. One limitation of the study is that our theoretical analysis in Section relies on the assumption that the data matrices A, T follow a Gaussian distribution and assumed the optimization objective to be quadratic, which may not always hold in practice. Despite the potentially strong assumptions, empirical findings in Section 4 show that the conclusions remain robust in real-world datasets with various model architectures. This suggests the insights discovered in our study are applicable to a wider range of real-world scenarios, beyond just those strictly adhering to the Gaussian assumption. Nevertheless, future research could explore how such assumptions can be relaxed and extend the theoretical analysis weaker condition.
Acknowledgements. Michael W. Mahoney would like to acknowledge the UC Berkeley CLTC, ARO, IARPA (contract W91 INF20C0035). NSF, and ONR for providing partial support of this work. Kurt Keutzer would like to acknowledge support from Berkeley Deep Drive. Yaoqing Yang would like to acknowledge support from DOE under Award Number DE-SC0025584, DARPA under Agreement
10

---

number HR001 12490441, and Dartmouth College. Our conclusions do not necessarily reflect the position or the policy of our sponsors, and no official endorsement should be inferred.

---

References
Maksym Andriushchenko, Francesco Croce, Maximilian Mueller, Matthias Hein, and Nicolas Flam- marion. A modern look at the relationship between sharpness and generalization. In International Conference on Machine Learning, 2023.
Devansh Arpit, Huan Wang, Yingho Zhou, and Caiming Xiong. Ensemble of averages: Improv- ing model selection and boosting performance in domain generalization. Advances in Neural Information Processing Systems, 2022.
Christina Baek, Yiding Jiang, Aditi Raghunathan, and J Zico Kolter. Agreement-on-thh-line: Predict- ing the performance of neural networks under distribution shift. Advances in Neural Information Processing Systems, 35:19274-1928) 2022.
Kayhan Behdin and Rahul Mazumder. On statistical properties of sharpness- minimization: Provable guarantees. arXiv preprint arXiv:2302. 2023.
Adrian N Bishop, Pierre Del Moral, Angèle Niclas, al. An introduction wishart matrix moments. Foundations and Trends@ in Machine Learning, 11(2)97-2118, 2018.
Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM review, 60(2):223331, 2018.
Leo Breiman. Random forests. Machine learning, 45(1):5-32, 2001.
Anh Bui, Vy Vo, Tung Pham, Dinh Phung, and Trung Le. Diversity agnostic ensemble of sharpness minimizers. arXiv preprint arXiv:2403. 13204, 2024.
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34:22405 1--22405-22418, 2021.
Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785-794, 2016.
Thomas Dietterich. Ensemble methods in machine learning. In International workshop on multiple classifier systems, pages 1-15. Springer, 2000.
James Diffenderfer, Brian Bartoldson, Shreya Chaganti, Jize Zhang, and Bhavya Kailkhura. A winning hand: Compressing deep networks can improve out-of-distribution robustness. Advances in nearal information processing systems, 34:664-676, 2021.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pages 1019-1028. PMLR, 2017.
Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent Tan. Efficient sharpness-awar minimization for improved training of neural networks, In International Conference on Learning Representations, 2024.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
Stanislav Fort, Huiyi Hu, and Balaji Lakshninarayanan. Deep ensembles: A loss landscape perspec- tive. arXiv preprint arXiv:1912.02757, 2019.
Yoav Freund. Boosting weak learning algorithm by majority. Information and computation, 121(2): 256-285 1995.
Yoav Freund and Robert E Schapire. A application to boosting. Journal of computer and system sciences, 55(1):119-155 1997.
generalization of on-line learning and an
Mudasir A Ganaie, Minghui Hu, AK Malik, Tanveer, and Suganthan. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence, 115:105151, 2022.
12

---

Xiang Gao, Meera Sitharam. and Adrian E. Roitberg. Bounds on the jensen gap, and implica- tions for mean concentrated distributions. The Australian Journal of Mathematical Analysis and Applications, 2019.
Isha Garg and Kaushik Roy. Samples with low loss curvature improve data efficiency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20290-20300 2023.
Diego Granziol. Flatness is false friend. arXiv preprint arXiv:2006.0909) 2020..
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019a.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In International Conference on Learning Representations, 2019b.
Geoffrey E. Hinton and Drew Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the Sixth Annual Conference on Computational Learning Theory, 1993.
Sepp Hochreiter and Jürgen Schmidhuber. Flat Minima. Neural Computation, 9(1):1-42, 1997.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. In 34th Conference Uncertainty in Artificial Intelligence 2018, UAI 2018, pages 876-885. Association For Uncertainty in Artificial Intelligence (AUAI), 2018
Weisen Jiang, Hansi Yang. Yu Zhang, and James Kwok. An adaptive policy employ sharpness- aware minimization. In International Conference on Learning Representations, 2023.
Yiding Jiang, Vaishnavh Nagarajan, Christina Back, and Zico Kolter. Assessing generalization of SGD via disagreement. In International Conference on Learning Representations, 2022.
Jean Kaddour, Linqing Liu, Ricardo Silva, and Matt J Kusner. When do flat minima optimizers work? Advances in Neural Information Processing Systems, 35:16577-16595, 2022.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations, 2016.
Sosuke Kobayashi, Shun Kiyono, Jun Suzuki, and Kentaro Inui. Diverse lottery tickets boost ensemble from single pretrained model. In Proceedings of BigScience Episode #5 - Workshop on Challenges & Perspectives in Creating Large Language Models, 2022.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
S. Kullback and R. A. Leibler. On Information and Sufficiency, The Annals of Mathematical Statistics, 22(1):79 86, 1951. doi: 10.1214/aoms/ wtteooo///17770010 177729694.
Jungmin Kwon, Jcongseop Kim, Hyunsco Park, and In Kwon Choi. Asam: Adaptive sharpness-awar aware minimization for scale-invariant learning of deep neural networks. In International Conference Machine Learning, pages 5905-5914. PMLR, 2021.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: lite bert for self-supervised learning of language representations. International Conference Learning Representations, 2020.
13

---

François Laviolette, Emilie Morvant, Liva Ralaivola, and Jean-Francis Roy. Risk upper bounds for general ensemble methods with an application multiclass classification. Neurocomputing. 219: 15-25, 2017.
Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015.
Yoonho Lee, Huaxiu Yao, and Chelsea Finn. Diversify and disambiguate: Learning from under- specified data. In ICML 2022: Workshop on Spurious Correlations, Invariance and Stability, 2022.
Shiwei Liu, Tianlong Chen, Zahra Atashgahi, Xiaohan Chen, Ghada Sokar, Elena Mocanu, Mykola Pechenizkiy, Zhangyang Wang, and Decebal Constantin Mocanu. Deep ensembling with no overhead for either training or testing: The all-round blessings of dynamic sparsity. In International Conference on Learning Representations, 2022.
Andres Masegosa. Learning under model misspecification: methods. Advances in Neural Information Processing Systems, 33:5479-5491, 2020.
Applications to variational and ensemble
Alireza Mehrtash, Purang Abolmaesumi, Polina Golland, Tina Kapur, Demian Wassermann, and William Wells. Pep: Parameter ensembling by perturbation. Advances in neural information processing systems, 33:8895-8906 2020.
Jishnu Mukhoti, Andreas Kirsch, Joost van Amersfoort, Philip HS Torr, and Yarin Gal. Deterministic neural networks with appropriate inductive biases capture epistemic and aleatoric uncertainty. arXiv preprint arXiv:2102.11 1158,, 2021.
Behnam Neyshabur, Srinadh Bhojanapalli,, and Nathan Srebro. A PAC-bayesian approach to opectrally-normalizee margin bounds for neural networks. In International Conference Learning Representations, 2018
Luis A Ortega, Rafael Cabañas, and Andres Masegosa. Diversity and generalization in neural network ensembles. In International Conference Artificial Intelligence and Statistics, pages 11720-11743. PMLR, 2022.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, David Sculley, Sebastian Nowozin, Joshua Dillon, Balaji Lakshminarayanan, and Jasper Snoek. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems, 32, 2019.
Tianyu Pang. Kun Xu, Chao Du, Ning Chen, and Jun Zhu, Improving adversarial robustness via promoting ensemble diversity. In International Conference on Machine Learning, pages 4970 4979. PMLR, 2019.
Jack Parker-Holder, Luke Metz, Cinjon Resnick, Hengyuan Hu, Adam Lerer, Alistair Letcher, Alexander Peysakhovich, Aldo Pacchiano, and Jakob Foerster. Ridge rider: Finding diverse solutions by following eigenvectors of the hessian. Advances in Neural Information Processing Systems, 33:753-765, 2020.
Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. Diverse weight averaging for out-of-distribution generalization. Advances in Neural Information Processing Systems, 15:108811-088 2022.
Ryan Theisen, Hyunsuk Kim, Yaoqing Yang, Liam Hodgkinson, and Michael W Mahoney. When are ensembles really effective? Advances in neural information processing systems, 2023.
Giorgio Valentini and Thomas G Dietterich. Low bias bagged support vector machines. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pages 752-759, 2003.
Roman Vershynin. High-DDmensional Probability: An Introduction with Applications Data Science. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2018
Tim Whitaker and Darrell Whitley. Prune and tune ensembles: low-cost ensemble learning with sparse independent subnetworks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 8638-8646, 2022.
14

---

Yaoqing Yang, Liam Hodgkinson, Ryan Theisen, Joe Zou, Joseph E Gonzalez, Kannan Ramchandran, and Michael W Mahoney. Taxonomizing local versus global structure in neural network loss landscapes. Advances in Neural Information Processing Systems, 34:18722-18873, 2021.
Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W Mahoney. Hessian-based analysis of large batch training and robustness to adversaries. Advances Neural Information Processing Systems, 31, 2018.
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data), pages 581-590. IEEE, 2020.
Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and Michael Mahoney. Adahessian: An adaptive second order optimizer for machine learning, In proceedings of the AAAI conference on artificial intelligence, volume 35, pages 10665-10673, 2021.
Juntang Zhuang. Boqing Gong. Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha C Dvornek, James Duncan, Ting Liu, al. Surrogate gap minimization improves sharpness-awan training. In International Conference on Learning Representations, 2022.
Max Zimmer, Christoph Spiegel, and Sebastian Pokutta. Sparse model soups: A recipe for improved pruning via model averaging. In International Conference on Learning Representations, 2024.
15

---

Appendix
A Impact Statement
This paper uncovers trade-off between sharpness and diversity in deep ensembles and introduces a novel training strategy to achieve an optimal balance between these two crucial metrics. While the proposed method could potentially be misused for malicious purposes, itself does not pose any direct negative societal impact. More importantly, this research advances the field of ensemble learning and contribute the development of more reliable deep ensemble models. These ad advancements consequently result in enhanced robustness when dealing with OOD data and enable the quantification of uncertainty, thereby strengthening the reliability and applicability of deep learning systems in real-world scenarios.
believe that the study
B Related work
Ensembling. Diversity is one of the major factors that contribute to the success of the ensembling method. Popular ensemble techniques have been developed for tree-type individual learners, which are known to have a high variance. This is evident such in [Breiman, 2001, Chen and Guestrin, 2016, Freund, 1995, Freund and Schapire, 1997]. In contrast, more stable algorithms, such as support vector machines (SVM) type learners, are less commonly used for ensembles, unless they are tuned to low-bias, high-variance regime, as explored in [Valentini and Dietterich, 2003]. When comes to diversity and ensembling, NNs are known to exhibit properties different than traditional models, e.g., as described in recent theoretical and empirical work on loss landscapes and emsemble improvement [Theisen et al., 2023, Yang al., 2021]. Therefore, ensembling techniques that work well for traditional models (e.g.. tree-type models) often underperform the simple yet efficient deep ensembles method [Fort al., 2019, Ortega et al., 2022] that uses the independent initialization and optimization. Previous literature has explored various new methods to learn diverse NNs [Lee al., 2022, Rame et al., 2022, Pang al., 2019, Parker-Holder et al., 2020]. Our work is different from previous work in that we study flat ensembles obtained from sharpness-awee especially focusing on diversifying flat ensembles by reducing the overlap between sharpness-awe aware data subsets. While our work demonstrates significant improvements in OOD generalization, it is known that (in some cases, see also Theisen et al. [2023]) deep ensembling a simple, yet effective method to improve OOD performance [Diffenderfer et al., 2021]. Therefore, we compare the OOD performance of SharpBalance to deep ensembles.
training methods,
Sharpness and generalization. A large body of work has studied the relationship between the sharpness (or flatness) of minima and the generalizability of models [Hochreiter and Schmidhuber, 1997, Hinton and van Camp. 1993, Keskar et al., 2016, Neyshabur et al., 2018, Yang et al., 2021, Kaddour al., 2022, Yao al., 2021, 2020]. Works such as those by [Hochreiter and Schmidhuber. 1997] and [Hinton and van Camp, 1993] use Bayesian learning and minimum description length to explain why we should train models flat minima. [Keskar al., 2016] introduces sharpness-bart metric, demonstrating how large-batch training can skew NNs towards sharp local minima, adversely affecting generalization. In addition, [Neyshabur al., 2018) uses a PAC-Bayesian framework to prove bounds on generalization, which can be interpreted as the relationship between sharpness and test accuracy. Furthermore, [Cha et al., 2021] presents a theoretical exploration of the link between the sharpness of minima and OOD generalization.
Motivated by the good generalization property of flat minima, variants of sharpness-guided opti- mization techniques have been proposed [Yao et al., 2018, 2021, Du et al., 2024, Jiang et al., 2023), including sharpness- aware minimization [Foret et al., 2021]. The DiWA method [Rame al.. 2022] observed that SAM can decrease the diversity of models in the context of weight averaging (WA) [Izmallov al., 2018]. However, WA imposes constraints on diffeernnt models, requiring them to share the same initialization and stay close to each other in the parameter space. In contrast, our work focuses on deep ensembles that do not pose additional constraints on the training trajectories of individual ensemble members. Previous work by [Behdin and Mazumder, 2023] provided theoreti- cal characterization of important statistical properties for kernel regression models and single-layer ReLU networks, optimized using SAM noisy datasets. Our theoretical analysis borrows ideas from [Behdin and Mazumder, 2023] and extends the analysis using random matrix theory. DASH was proposed in [Bui al., 2024] minimize the generalization loss by adding KL divergence constraint
16

---

on the output logits of ensemble members. The authors believe that the decrease in diversity is a result of models being initialized closely and updated with the same direction. In contrast, SharpBalanc ance trade-off is ubiquitous across various settings and provides
observed that the sharpness-diversie a rigorous theoretical quantification that characterizes the interplay of the two metrics. Compare to DASH, SharpBalane ance provably achieves improved performance and is simple, effective, and computationally cheap to implement.
C Proof of Theorems in Section 3
Recall that SAM updates the model weights, ignoring the normalization constant and regularization, through the following recursive rule
k+1 0 S.M - V (0
(0SA + PVI
We first show an unrolling of the iterative optimization on a quadratic objective.
Theorem 3 (Unrolling SAM). Let 0" be the teacher model. Let Oo be randomly initialized and
updated with SAM to solve quadratic objective L (0) (0 07 A 0²). Then,
SAAM 8+1
T
Σ
B
0- +B B 0o:
2=
where B = A -
Proof. The gradient of the objective f given by VS(0)- AT 0*). Therefore,
0 SAAM
+ PV SAM (I+ PA T A)O
SAA -PA" -AAAA' AO".
With SAM update,
SSAM
R SAM nV f CO + PV
R++
SAAM MATA A (O PVV(OSAMM - 0*)
OSAM TAT T' (II+ AJ
PA T A0' - -0")
= (I A (P(AT))) Of OSAM ( ((AT+p(AAA)]]] 0r
17 Σ B' (' 0" + B
1=0
where the last equation is obtained by recursively unrolling the weight by previous updates.
Theorem 3 offers valuable tool to analyze the statistical behavior of the models optimized by SAM. However, one more ingredient is required arrive at the interesting conclusions claimed in Section 3, the random matrix theory. Recall that the data matrix A E x is random with entries drawn from Gaussian (000, I/dm). As result, entries in A A follows the Wishart distribution and according to Corollary 3.3 Bishop et al. [2018), for 2 1,
E (ATA)
Ner
din
111+1/dm)Ne.
(6)
d
where Ni., = ( the Narayana number. With the help of this Corollary, we now prove proposition on the expectation of B
Proposition 1 (Expectation of Wishart Moments). Let i, be non-negative integers, then EA AAT (ii,))
where
o(i.j) =0 +
Σ
k ki!k!!!?! (-7) 12 tot
Flar
Σ 1=1
dgn
m--
O + 1/d,g) Nm..
dm
Flr
As tha
and 122 + 2kg + j.
17

---

Proof. By Multinomial Theorem,
('(AAA)=
E
Ih:
(ATA)
ky !ki!! +ky
i!
(A A)
ky !k!
Let m lig + 2kgg + j and taking the expectation with equation (6) gives
i!
[A
Σ
( (-1)
Katkk p AA [AA
A
k
on lkg!li!!
Σ
kn lilgg (-n) kot pks
Thr
din
m--
O(1 + N/
din
Thr
h1 ksz
j 0, then there is case when 12 Therefore,
1os = 0, and the expectation of (ATA) simply becomes I.
BA A A|A = 15 +
E
( (-1) tz
pks
Thr din
din
m--
O(1 + 1/dmn) N
in !i
fhr
Aa Aht ky=
= ((i,))
C.1
Proof of Theorem
In this subsection, show proof for Theorem 1.
Proof. Apply Singular Value Decomposition (SVD) to obtain A VEUT and AT A US Let D x² By Theorem 3,
k- Σ
OSAM
B
+ +BOO
k
A - M(AAAAP) (ATA 0" + B
U(I 1/D D PD3 +B*0
i=0
- diag
Σ (1 Fdl nppd)² (d; + pdf)
UT + BB
1=0
j=1
[ (1 nd modd))
U diag
(dl + pd?))
UT +BO
nd, + gpdg
U [I (I- 1,D - mpD UT + (I- Ap(A T A)) * 0o
0" + -[I (I-7ATA - np(AT -0).
18

---

As result, Ee, [0 = -I- NATA A ()(A))))) 0" - - By definition,
The 1 = EAT
[Ba If SAM. T.) - Di 1
EA,T((E,, [0
0 T TEo [ 0¹)
EEl(Eo, 10 SAAM
0" y ET (E,, 1 [ 0*)
The EA (00) B 8²200 0*)
din
The 0f 2k: 0)100"12
din
The E AMM
(T)
8 9o
yi
i=1
EA, 10
-g
EAA..... 100 (00-0))
BB - 0o)
EAT (B")"BTTTBB' RA.T. 10 1 BBTTTBBB
The 6(2k. 0)1 n 0)o²
dn
Since ET(TTT] =
The and [[ I Hence, dg
(O0A Var (f(0 T))
(n, (r nhe Bias (0$AM)) 6(2k, 0)02.
The
Recall that given perturbation radius Po. the sharpness is defined as
<(0z)=
EA max f (Te, [0x] + f (To,, [0x))]- ||
We first compute
1
(Ea. [0. SASAN] + + AE
(Ea. [0 + 8T A" A [0SAM] + - -0))
2
1
(E A² -
2
ST AT AB + 1 (0"
(7)
2
Similarly,
f (Eo, [0gAM] AA = (O A"
(8)
2
Let Amin be the least eigenvalue of A ATA. A. By subtracting equation (7) with equation (8), we have
SAM Ki
EAI max SPl
T AT AT
2
EAI. max
Amin |UTE|12 - - A0
EA max
Amin Amiin|U"' -
u=Uv
EA max
Amin || 1 ||
min E eBBAATAO
Vill no 2
EA Amin o + Po A [2]]
The smallest singular value Amin of random X dm matrix A can be bounded by the following inequality on the smallest singular value Tmin (A) by Vershynin [2018], assuming fhe 2 din. then almost surely
19

---

EA [amin (A)] 2
Thr
1.
din
2
Therefore, Ea[mmi]] 2 E l min (A))² 2
1)
Now we show lower bound on
EA po B A A ll2. By Gao et al. [2019], the Jensen gap (E(Z))1/2-B)) (Z) is upper bounded
by 2
Varl 22) when Z is non-negative and E / 1. Notice that
EAlpo B A A | pa ( (A A))0*) "9,
and we let Z = (0*) (0*)²==
B 2k
A A) 0- Then E [2] - 2) 0 11 and
Var[Z] = (0(4k, 4) 6 2)²) 1 11
By normalizing Z and applying the Jensen gap upperbound, we have
6(4k; 4) ((2k, 2)² 20(2k, 2)3//11001111
| B A0 [2] 2 Po o(k, 2)
As result,
SAM
P
(V
ThE pon 22
1)
6(4k, 4) ((2k, 2) 26(2k, 2)3/2 0+
2
din
The derivation of the upper bound follows from similar proof, ignoring the Jensen gap.
C.2 Proof of Theorem
Below we show a proof of Theorem 2.
Proof. We apply SVD A, to obtain A, V.E.U" and AT U.E.U? Let D. = Σg and B. --=InAA A. np(ATA,) By Theorem and a similar derivation in the proof of Theorem 1,
SharpBal
Σ
B (A ((ATA.))) 0" +B.00
5=0
0" + (I - nAA A, - -p(p )²) (0. 0¹).
As result, Ea,. Sharpbal
= E. B = 0" - Σ BBO
Applying Proposition 1, we have
EA B [A] A A = (1
where
11=0
Σ
(-n) Ao to phs
Thu Sclm
m
Sdin
m-1
Nm.
kn lkg
Dar
kn +oy
Then.
The Biass = EA,T(
EEoo. 10
Sharpball
- )0²)² T'
TTT
EBo.. 10 @0harpbbll
- 0001
The EAIl
B
Σ
B
= dm
S
5 s==1
s=1
S
The
EA
B Σ
B 0
d S2
The
(0'(2k, 0) (s ()' 0)2) 1011.
S
20

---

The last equality is the result of applying EA B (i(, 0) with different combinations of B,, B counting multiplicity. Similarly,
The Error 0% Sharphal) = EA,
" + E, 10"
The Φ (2k, 0) 112 + new
(2k,0)o* m
dmn
Therefore,
Var SharpBall T))
1
The 10 - The Bias (00
The
=w' (2k: -2'(2k, 0)0² + 5-1 (d 0 (k, 0)² 10 12.
S
When the model trained on the submatrix, the sharpness of model 0% SharpBol is defined as
SaarpBal Kg
= EEAl max f [Eee..e Sp
0 SharpBall +e; - f (Boo.s 0 SherpBall
; I
From a similar analysis of the proof for Theorem 1.
Kk
SharpBal
Pō
2
V
Per
din
+
2 +
Po EAll Σ B A0
S
and with r
The Sd.
EAll Σ =EA(())))))())))))))) A
Σ
B k
Σ
B
A 10
1/2
(0))
A
B
B tk
A A
1=1
(SQ + ()
1)0 (2k, 1) 2S 1) (k: 2) (k, 0)
+ +r(1 -))0(( 0) + 2S(S - 1)0 (k. 1) (k;
3
2 r( +((++))(S 1)(5 - (k,0)
3 5 -1(S- 2)0' 0)
2
+ +355(S 1)(S (k, 1)0'(k,0)) (k. 0)
+ +rSS(S 1)(5 (S - 3) (k: 1/2 1
The last equality is the result of applying EA |A (A" A.]] (i, with different combinations of
B.. By, AJ Ay. and A/AA, A4, counting multiplicity and the fact that (AI(A A.) r(1 + r)I.. In conclusion,
fók
SharpBal p8
<
2
( /
dm
Tur
dm
+1)
+
Po VC s
where
0 2) + 2rS(S (2 +r(1 +r)S($ - 1) (24, + - (k; (k;
2)0 (k,
+ ( 1)(S 1)) 3 3,2 1)(S 2))'(2k,,0)
- - - 2)(S - 3)6 0)2. + 3rS(S - 1)(S 2)² 1)
The claims Theorem 2 further supported by the experimental validations with results presented in Figure 9.
21

---

Upper bound of sharpoess (K-xiis)
Upper bound of shurpness axis)
Simulated
Simulated
100
Diversity
Diversity
0.06
Upper Bound on Sharpness
Upper Bound on Sharpness
0.04
p-0.4
k-15
0.03
1.0
10.0
0.4
0.5
0.6
0.8 0.9
Sharpness
Sharpness
(a) Varying perturbation radius
(b) Varying number of training iterations k
Figure 9: (Theoretical vs. Simulated sharpness-diversity trade-off in SharpBalance) This figure illustrates the relationship between sharpness(upper bound) and diversity as predicted by Theorem and as observed in simulations under different configurations. (a) validates our theoretical results by varying the perturbation radius p from 1.0 to 0.4. (b) validates the derivation by varying number of iterations k from to 15. These results demonstrate the soundness of our derivation across range of parameters.
C.3 Empirical Verification of Theorem and
To demonstrate the robustness and tightness of the bounds presented in Theorem I, provide verification results across a range of parameter configurations. Interestingly, the observed model behaviors closely align with the upper bound derived in Theorem highlighting the effectiveness of our theoretical analysis in capturing the underlying dynamics of the ensemble. Figure 10 illustrates specific combination of i and with from
these results, with each sub-fight sub-figure corresponding range 0.5 to 0.3. In these experiment, we generated 50 random data matrices A of size 3000 x 150 and test data T of size 1000 150. For each random dataset, initialized 50 random model weights 0o and collected the expected statistics interest after training. To measure the sharpness Aof .AAM we employed projected gradient ascent find the optimal perturbation, using a step size of 0.01 and maximum of 50 steps. Similar experiments are performed to verify the derivations in Theorem 2 with results presented in Figure 11, with the number partitions 10.
D Hyperparamter setting
D.1 Datasets
We first evaluate image classification datasets CIFAR-10 and CIFAR-100. The corresponding OOD robustness is evaluated on CIFAR- and CIFAR-100C [Hendrycks and Dietterich, 2019b]. The experiments carried out on ResNet18 [He et al., 2016]. We use batch size of 128, momentum of 0.9, and weight decay of 0.0005 for model training. TinyImageN classification dataset consisting of 100K images for training and 10K images for in-distribution testing. We evaluate ensemble's OOD robustness on TinylmageNerC [Hendrycks and Dietterich, 2019b].
is an image
D.2 Hyperparamter setting for empirical sharpness-diversity trade-off
Here, we provide the hyperparameter for the experiments in Section 4.2. When using adaptive worst-case sharpness for sharpness measurement, the size of neighborhood 7 defined in equation (3) needed to specified, we use a 7 of 0.5 for the results Figure and Figure 5. Additionally, when training NNs in the ensemble, we change the perturbation radius of SAM so that we can study the trade-off. The range of p for the results in Figure is f0.01, 0.02, 0.03, 0.04, 0.05, 0.2, 0.3). the range of for the results in Figure 5 is {0.01, 0.015, 0.02, 0.025, 0.03, 0.05, 0.1, 0.2, 0.3, 0.4).
22

---

Lower bound of Sheepess (buad))
Lppe bound of auappo Geavia
3x
109
4x10"
10
Divenity
1x10
Doversty
Divenity
2x10
6x10
10'
4x100
10
3 10
4x 10 Sharpness
6 10
6x 10
10
10'
2 10 Sharpeess
3 104 10"
Starpness
(a) 2, 1j 0.1
(b) 17
0.05
(c) l: 2, 0.01
Simulated
Lewer bound chorpeess (tvarts) Upper hound of shrppos (s-asid)
10"
10*
Diversity
Diversity
Diversity
10'
10"
10
10"
10
16
10
10 Sharpness
Sharpness
Sharpness
(d) 4, 1
(c) 1, = 0.05
(f) k: =
Simolated
Lewer hound of Harpoos (t-atis)
Upper sharpoii (w-ais)
10"
Diversity
20
Diversity
10
Diversity
10*
10
10
10²
10
10
10
10
10"
10
10
Sharpness
Sharpesss
Sharpeess
(g) k = 8, 0.1
(h) k 8, 1 0.05
(i) k: 8, 0.01
Figure 10: (Theoretical vs. Simulated sharpness-diverity trade-off in SAM). This figure compares the sharpness and diversity as predicted by Theorem and observed in simulations under various parameter configurations. Results demonstrates the robustness our theoretical analysis and tightness of the derived sharpness upper bound.
D.3 Hyperparamter setting for SharpBalance
Hyperparameter setting on CIFAR10/100. For experiments on CIFAR10/100, we train an NN from scratch with basic data augmentations, including random cropping, padding by four pixels, and random horizontal flipping. We use a batch size of 128, a momentum of 0.9, and a weight decay of 0.0005. For deep ensemble, train each model for 200 epochs.
In addition, we use 10% of the training set as the validation set for selecting p and based on ensemble's performance. We make grid search for p over (0.01, 0.02, 0.05, 0.1, 0.2, 0.5). For SharpBalance, we use the same as SAM and search & over (0.2, 0.3, 0.4, 0.5, 0.6). Ta is another hyperparameter introduced by SharpBalance, a Ta of 100 and 150 respectively when training dense and sparse models on CIFAR100. See Table I for the optimal and A after grid search.
use Ta of 10 for all experiments on CIFAR 10,
Hyperparameter setting on Tiny Net. For experiments on TinyImageNet, we adopt basic data augmentations, including random cropping, padding by four pixels, and random horizontal flipping. We train each model for 200 epochs. We use batch size of 128, a momentum of 0.9, a weight decay of 5e-4, Ta of 100, an initial learning rate of 0.1, and decay it with factor of 10 at Epoch 100 and 150. We search and the same range as what we on CIFARION 10/100. See Table for the optimal p and & after grid search.
23

---

Upper bound of scaapppc (1aaid)
688
10
R............ R........
Diversiyy
10
Divenity
10
Diversity
10
10'
10a
10"
10
10
10
10
10
Shappesss
Sharpness
Sharpeess
(a) 4, 1j = 0.5
(b) 4, 0.3
(c) h: = 4,7 0.1
Simolated
1pper hound shurpposs (s-anis)
10
10
10
Diversity
10"
Diversity
Divensty
10
10
16
10a
10
10
10'
10
10
10'
10
10'
10
Sharpness
Sharposss
Sharpoess
(d) k 8, 1
0.5
(e) A: 8, 0.3
(f) k 8, = 0.1
Simulated
Upper sharpocis (wasis)
10"
10a
10
Divensity
Diversity
Diversity
10
10¹
10"
10
10
10a
10
10
10
10
10
10
10
10
10
Sharpness
Starpness
Sharpness
(g)
12, =
(h) 12, 0.3
(i) k: 12
Figure 11: (Theoretical vs. Simulated sharpness-diversie trade-off in SharpBalance), This figure compares the sharpness and diversity as predicted by Theorem and as observed in simulations under various parameter configurations. The observed model behaviors align closely with our derived upper bounds.
Dataset
Model
Method
P
ResNet18
Deep Ensemble
CIFARIO
ResNet18
Deep Ensemble+SAM
0.2
ResNet18
SharpBalance
0.2
0.4
100
ResNet18
Deep Ensemble
CIFAR100
ResNet18
Deep Ensemble+SAM
0.2
ResNet18
CharpBalance
0.2
0.5
100
ResNet18
Deep Ensemble
ImageNt
ResNet18
Deep Ensemble+SAM
0.2
ResNet18
SharpBalance
0.2
0.3
100
Table 1: Hyperparamter setting for results in Section 4.4, we report the optimal p and after grid search. Each result in Figure 7 is averaged over three ensembles, which corresponds to 9 random seeds, the random seeds we use are {13, 17, 27, 113, 117, 127, 43, 59, 223).
E Ablation studies on loss landscape metrics
In this section, we show that the sharpness-diversin trade-off generalizes to different measurements of sharpness and diversity. The results are presented in Figure 12.

---

Ensemble OOD Accuracy
Ensemble OOD Accuracy
Ensemble OOD Accuracy
0.75
50.5
51.0
$1.5
0.75
50.5
51.0
51.5
50.5
51.0
$1.5
0.17
B: 0.23
0.8
B: 0.52
0.70
0.70
0.7-
DER
DER
0.65
0.65
0.6-
0.60
0.5
0.3
0.4
0.5
0.6
0.7
0.4
0.5
0.6
0.8
1.0
Adaptive P. wowoss-case casss---sseeee sharpness
Adaptive average case sharpness
Adaptive worst case sharpness
(a) Diff sharpness (CIFAR-100)
(b) Diff sharpness (CIFAR-100)
(c)
Diff' diversity (CIFAR-100)
Ensemble OOD A Accuracy
Ensemble OOD Accuracy
Ensemble OOD Accuracy
0.9
30.0
30.5
31.0
30.0
30,5
31.0
30.0
30.5
31.0
0.22
B: 0.32
B: 0.98
1.3
1.1
DBB
0.8
DER
0.8
0.9
0.8
0.7
0.7
1.8 Adaptive Euworst case sharpness
1.0
1.2
1.4
1.6
3.0
3.5
4.0
4.5
2.0
2.5
Adaptive average case sharpness
Adaptive worst case sharpness
(d) Diff sharpness (TIN)
(c) Diff sharpness (TIN)
(f) Diff diversity (TIN)
Figure 12: (Ablation study of varying sharpness and diversity metrics to corroborate existence of sharpness-iiversity trade-off). (a)(d) Varying sharpness metric by using the adaptive foo worst- case sharpness. (b)()) Varying sharpness metric by using the adaptive l average case sharpness. (c)(f) Varying diversity metric by using the KL divergence. The sharpness-diversity trade-off is still observed in all the settings. The --axis and y-axis are in log scale. The notation / stands for the slope of the linear regression function fitted on all the ensembles trained by SAM.
Sharpness metric. In the main paper, we use adaptive worst-case sharpness defined in equation (3), the parameter neighborhood is bounded by &a norm. In this section, we consider two more sharpness metrics [K [won et al., 2021, Andriushchenko et al., 2023]: adaptive worst-case sharpness with the parameter neighborhood bounded by loo norm (referred to as adaptive &oo worst-case sharpness); and adaptive average case sharpness bounded by & norm (termed average case sharpness). The adaptive too worst-case sharpness defined as:
max 11 Spo
( + + (O)
(9)
The average case sharpness is defined as:
E ee dlng(Tb?))
-()())
(10)
where Po is the neighborhood size of current parameter 0. To is normalization operator that ensures the sharpness measure is invariant with respect to the re-scaling operation of the parameter. The results, illustrated in Figures 12, corroborate our observation of trade-off between sharpness and diversity,
Diversity metric. We consider Kullback- (KL) Divergence [Kullback and Leibler, 1951] an alternative diversity metric, which is also widely used in previous literature to gauge the diversity of two ensemble members [Fort al., 2019, Liu et al., 2022]. Specifically, the KL-divergence between the outputs of two ensemble members given a data sample (æ, y) is defined as:
KL (Je, (x), fa) (z)) Eje, (z) [log fe, (x) log fe, (x)).
(11)
We measure the KL divergence on each data sample in the test data and then average the measured KL divergence. The results for KL.-divergence are shown in Figure 12, which demonstrate the trade-off remains consistent for different diversity metrics.
25

---

More results
F.1 Evaluation different corruption severity
SharpBalance 's main advantage lies in OOD scenarios. As shown in Table 2-4, SharpBalance consistently outperforms the baselines on different levels of corruption.
Table 2: Results of different severity levels on CIFARIO-C.
Corruption Severity
4
Deep ensemble
88.90
83.67
77.56
70.37
58.63
Deep ensemble+SAM
89.44
84.24
78.16
71.04
58.77
SharpBalance
89.75 (+0.31)
84.80 (+0.56)
78.98 (+0.82)
72.25 (+1.21)
60.78 (+2.01)
Table 3: Results of different severity levels on CIFARI00-C.
Corruption Severity
4
Deep ensemble
65.78
57.77
51.30
44.33
34.16
Deep Cnsemble+SAM
66.39
58.47
51.89
44.90
34.81
SharpBalance
67.23 (+0.84)
59.53 (+1.06)
53.14 (+1.255)
46.19 (+1.29)
36,20 (+1.39)
Table 4: Results of different severity levels on Imny-lmageNet-C.
Corruption Severity
3
4
5
Deep ensemble
43.62
36.65
28.96
22.08
16.86
Deep ensemble+SAM
45.20
38.04
30.19
22.98
17.71
SharpBalance
46.48 (+1.28)
39.53 (+1.49)
31.70 (+1.51) 24.27 (+1.29)
18.69 (+0.98)
F.2
Evaluation different model architectures
We extend the evaluations on more architectures such as WideResNet (WRN), VIT, and ALBERT. Here describe the experimental setup. For vision tasks with WRN, trained the ensemble members from scratch on CIFAR-10 10 and CIFAR-100. For vision tasks with transformers. we constructed the three-member ensemble by fine-tuning the pre-trained ViT-T/16 model the CIFAR- 100 dataset, evaluated on in-distribution and CIFAR 100-C test sets. For language tasks, we constructed the three-member ensemble by fine-tuning the pre-trained ALBERT-Base model Microsoft Research Paraphrase Corpus (MRPC) dataset and evaluated the performance on its validation set. The hyperparameter search and setup are the same as in Appendix D.3.
These results in Figure 13 and Table confirm that SharpBal ance consistently boosts both ID and OOD) performance across the models and datasets studied.
VIT-T/16
ALBERT-B CIFAR
Method
CIFARIO 100
MRPC
Deep ensemble
88.34
66.64
89.50
Deep ensemble SAM
88.48
66.89
89.89
SharpBalance
88.68
67.21
90.11
Table 5: (Additional experiments on Transforner-architecture)) The ensemble test accuracy is reported and each ensemble comprises three members. The observation is consistent with the residual network results in the main paper: SAM improves the Deep Ensemble, and SharpBalance outperforms both two baselines.
26

---

Average of individual performance
Ensembling performance
77
76.61
52
Test Accuracy
75.58
+1,24
Test Accuracy
51
10
51,06
+3 3,26
75,06
50-
12.64
75
49
48.42
48.65
+4,00
74
+2.03
47
73
73.03
46
46.15
Deep Ensemble
SAM
SharpBalance
Deep Ensemble
SAM
SharpBalance
(a) CIFARIO-C
(b) CIFARI00-C
96.0
95.82
80
79.84
80.01
Test Accuracy
95.5
95,31
95.36
95.65
+0.43
Test Accuracy
78.88
+2.46
+2,62
78-
77.39
95.0
+3.73
+0,00
94.5
94,41
76
75.15
Deep Ensemble
SAM
SharpBalance
Deep Ensemble
SAM
SharpBalance
(c) CIFARIO
(d) CIFAR100
The three-member WRN-40-2 ensemble is trained with different methods two datasets. The first row reports the OOD accuracy and the second row reports the ID accuracy. The lower part of each bar with the diagonal lines represents the individual model performance. The upper part of each bar represents the ensembling improvement. The results are reported by averaging three ensembles, and each ensemble is comprised of three models.
Figure 13:
F.3 Sharpness-aware set: hard easy examples
SharpBalance ance aims to achieve the optimal balance by applying SAM to a carefully selected subset of the data while performing standard optimization the remaining samples. In work, sharpness is determined by the curvature of the loss around the model's weights, whereas [Garg and Roy, 2023] determines it based the curvature of the loss around data point. In Figure 14, rank 1000 samples using both metrics and found strong correlation between these two.
Rank correlation:..2
sharpness(fishe trace)
5000
4000
3000
2000
1000
0
2
loss curvature around input data
Figure 14: Rank correlation between fisher trace and loss curvature around input data
27

---

F.4 Comparison with more baselines
We compare SharpBalance with stronger ensemble method EoA [Arpit et al., 2022] and stronger SAM methods. We carefully tuned the hyperparameters for EoA. EoA fine-tuned pre-trained model; and in our paper, all models are trained from scratch. We compare SharpBalance with another SAM baseline: SAM +, where three individual models are trained with different p values, e.g., 0.05, 0.1, and 0.2, respectively. From Table 6, SharpBal ance outperforms these two baselines both in-distribution and OOD generalization.
In Table 7, we combine GSAM [Zhuang et al., 2022] with Deep Ensemble as new baseline method "Deep Ensemble + GSAM", and incorporate the GSAM into our method SharpBalance. The results show that the new baseline with GSAM outperforms the original baseline in ID and OOD performance but still underperforms SharpBalance (w/ SAM). Furthermore, we enhance SharpBalance by replacing the SAM with GSAM, which leads better ID performance.
Dataset
Method
ACC
cACC
SAM +
96.03
76.29
CIFARIO
EoA
95.55
75.57
SharpBalance
96. (+0.15)
77.32 (+1.03)
SAM
79.67
51.28
CIFAR100
EoA
79.53
51.45
SharpBalance
79.84 (+0.17)
52.46 (+1.01)
Table 6: SharpBalance outperforms EoA and SAM + both in-distribution and OOD generalization on CIFARIO and CIFAR100.
Method
CIFARI00 Ace
CIFAR100-C Ace
Deep Ensemble
79.28
50.67
Deep Ensemble SAM
79.50
51.28
SharpBal ance (SAM)
79.84
52.46
Deep Ensemble GSAM SharpBalance (GSAM)
79.74
51.37
80.01
51.92
Table 7: (Comparing our method SharpBalance with stronger SAM baseline). The ensemble test accuracy is reported and each ensemble comprises three members. GSAM improves the original baseline method with SAM and SharpBalance. The model is ResNet18.
Deep ensemble
Deep ensemble + SAM
SharpBalance
2.3
0.12
CF
NLL
2.2
0.10
2.1
0.08
2
4 Number of models
4
Number of models
(a) Expected Calibration Error (BCE)
(b) Negative Log Likelihood (NLL)
Figure 15: (Uncertainty metrics on CIFARI00-C), "ECE" represents expected calibration error, and "NLL" represents negative log-likelihood. Both metrics are lower the better. The model architecture ResNet-18. The uncertainty metrics demonstrate the superior performance SharpBalance. r-axis represents the number of individual models in ensemble.
28

---

E5 Evaluation on uncertainty metrics
In Figure 15, we present the results of uncertainty metrics, i.e., negative lig-liielhood and expected calibration error. These uncertainty metrics exhibit trends similar to the accuracy metrics: "Deep Ensemble SAM" outperforms "Deep Ensemble", and our method outperforms both baselines. The experiments were conducted using ResNet-18 8 on CIFAR100, with metrics reported on corrupted datasets. Additionally, we observe that both metrics improve as the number of ensemble members increases for all three methods.
G Experiments Compute Resources
All codes are implemented in Py PyTorch, and the experiments are conducted on 3 Nvidia Quadro RTX 6000 GPUs for training an ensemble of models. Compared to SAM, our method adds minimal computational cost. The extra time comes from using Fisher trace to compute the per-sample sharpness. Therefore, computing the per-sample sharpness requires one single forward pass and one backward pass. We report the additional training cost in Table 8. SharpBal ance only increases the training time by %: 0.83 (84.48 0.83) x 100% 1%.
Additional training cost
Total training cost
0.83 min
84.48 min
Table 8: Additional training cost introduced by SharpBalance. We train ResNet18 on CIFARIO for 200 epochs.
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Main claims made in the abstract and introduction accurately reflect the scope and contribution of the paper, supported by our theoretical and experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work discussed in Section 5,
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
30

---

Justification: Proofs of all theoretical results are provided in Appendix C and the assumptions are clearly stated in the theorems. Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The paper fully discloses all relevant information including detailed descrip- tion of the algorithms, datasets, experimental set in Section 4 and typeraarameters in Appendix to reproduce main experimental results claimed the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
31

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The implementation can be found through the anonymous github repository and the zip file uploaded as the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
www.emissssssiiiiiiiieooo (iPolllyy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //npss wwtppblcccc/////cccccccccccc..lt wwwpc//gg///aeieeeocooeeooss
wwwsssssssiigiyooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The experimental settings with hyperparameters are provided in both Section 4 and Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The results are accompanied by error bars, for the experiments in Section 4.4 that support the main claims of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
32

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The information on the computer resources for experiments can be found in Appendix G.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics h httpppiicc/thicsuiieeeee..
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The impact statement of the study can be found in Appendix A.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
33

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The original papers that produced the code dataset appropriately cited in this work.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
34

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
35

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "haBGktuqCH", "reviewer": "Reviewer_8MB6", "summary": "Ensemble methods and sharpness-aware optimization techniques are well-known strategies for improving generalization. This work identifies a trade-off between sharpness and diversity, observing that reducing sharpness can diminish diversity and harm ensemble performance. Through theoretical and empirical analysis of this sharpness-diversity trade-off, the authors present SharpBalance, an algorithm for training ensembles with sharpness-aware solutions without sacrificing diversity. Evaluation results on CIFAR-10/100, TinyImageNet, and their corrupted variants confirm the effectiveness of SharpBalance.", "questions": "- It appears that the current emphasis is on logit-ensemble (lines 82-83). Does the same rationale apply when ensembling categorical probabilities (i.e., probability-ensemble)?\n- In the proposed SharpBalance algorithm, it seems that the training data and objective for the i-th member are defined using other members (such as members i+1, i+2, as illustrated in the figure). Does this imply that in practice, each member is trained sequentially?", "limitations": "Section 5 addressed the limitations.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Ensemble methods and sharpness-aware optimization techniques are both prominent approaches for improving generalization. The aim of this work, which combines these two approaches, is well-motivated.\n- While the theoretical analysis uses the variance metric to indicate diversity, the experimental results show consistent trends across different diversity metrics. It suggests that the proposed analysis is widely applicable to the general concept of diversity.\n- Extensive empirical results effectively validate the theoretical analysis. The summary plots of the results are generally highly readable.", "weaknesses": "- The evaluation results are centered exclusively on classification accuracy; since ensembling usually highlights both predictive accuracy and uncertainty, relying solely on accuracy to assess overall performance is insufficient. \n- Specifically, for the corrupted CIFAR benchmark, uncertainty metrics like negative log-likelihood or expected calibration error are more important than test accuracy, but these aspects are not currently considered.\n- It seems that all experiments were conducted exclusively with residual networks. It is essential to verify if the proposed analysis and algorithm are applicable to other architecture families as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weakness 1 and 2\nIn Figure 14 of the rebuttal PDF, we present the results for negative log-likelihood and expected calibration error. These uncertainty metrics exhibit trends similar to the accuracy metrics reported in the main paper: \"Deep Ensemble + SAM\" outperforms \"Deep Ensemble\", and our method outperforms both baselines. The experiments were conducted using ResNet-18 on CIFAR100, with metrics reported on corrupted datasets. Additionally, we observe that both metrics improve as the number of ensemble members increases for all three methods.\n\n## Weakness 3\nWe provide new experiments on transformer models on vision and language tasks. The results are shown in Table 8 of the rebuttal PDF. We show that \"Deep Ensemble + SAM\" outperforms the Deep Ensemble while SharpBalance still outperforms both baselines. This observation is consistent with the residual network results in Figure 7 of the submitted paper. \n\nHere we describe the experimental setup. For vision tasks, we constructed the three-member ensemble by fine-tuning the pre-trained ViT-T/16 model on CIFAR100 dataset, evaluated on in-distribution and CIFAR100-C test set. For language tasks, we constructed the three-member ensemble by fine-tuning the pre-trained ALBERT-Base model on Microsoft Research Paraphrase Corpus (MRPC) dataset and evaluated the performance on its validation set. The hyperparameter search and setup are the same as in Appendix D.3 of the submitted paper.\n\n## Question 1\nYes, the rationale behind SharpBalance also applies when ensembling categorical probabilities. Studies in [1-3] reveal that the correlation between the output probabilities of probability-ensemble can significantly affect the classification error rate and uncertainty quantification. Therefore, obtaining more diverse ensemble members following the idea of SharpBalance is certainly beneficial to probability-ensembles.\n\nWe conducted additional experiments to verify this insight, with new results shown in Figure 15 of the rebuttal PDF. Results show that SharpBalance outperforms both baseline methods, including Deep Ensemble and \"Deep Ensemble + SAM\".\n\n## Question 2\nSharpBalance trains each ensemble member in parallel, distinct from classical boosting strategies. In particular, the $i$-th member's objective is computed based on the current status of the other members, and hence, there is no sequential dependency on the training of the individual models. In practice, the sharp-aware subsets for each model are selected at a synchronized time step that happens only once in the training process. \n\n**Reference**\n\n[1] Ryabinin et al. Scaling Ensemble Distribution Distillation to Many Classes with Proxy Targets.\n\n[2] Brown et al. Managing Diversity in Regression Ensembles. \n\n[3] Brocker et al. From ensemble forecasts to predictive distribution functions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "08VW5TVNbk", "reviewer": "Reviewer_jD9M", "summary": "The paper proposes SharpBalance, that is a method aiming to investigate the relationship between sharpness and diversity for deep ensembles.", "questions": "- Regarding the new discovery contribution. As I previously stated on the weakness section were the authors aware of the paper “Diversity-Aware Agnostic Ensemble of Sharpness Minimizers. Could the authors present the main differences between this method and SharpBalance.", "limitations": "- The proposed method is not a significant improvement for ID scenarios as the authors claim. I would tone that down over the whole text. Maybe clearly state that SharpBalance is indeed superior to OOD scenarios and competitive on ID settings.  \n\n- I would not claim the phenomenon called sharpness-diversity trade-off is a discovery, paper [1] is addressing the same phenomena and it was publicly available on arxiv before submission and on openreview since the beginning of the year.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- SharpBalance looks quite effective for the out-of-distribution setting. The goal of balancing sharpness and diversity within ensembles is an important idea.  \n- Great theoretical analysis", "weaknesses": "- The authors are aware of the paper called “Diversity-Aware Agnostic Ensemble of Sharpness Minimizers” [1], the idea is quite like the proposed paper, they aim to investigate the relations between sharpness and diversity on ensemble learning. I suggest the authors to discuss the main differences between both.    \n\n[1] Anh Bui, Vy Vo, Tung Pham, Dinh Phung and Trung Le, Diversity-Aware Agnostic Ensemble of Sharpness Minimizers, arXiv:2403.13204. \n\n- Regarding the baselines the authors only compare SharpBalance with SAM. Nevertheless, newer, and stronger baselines like GSAM [2] and OBF [3] should also be benchmarked since they are the current state-of-the-art. \n\n[2] Zhuang, J., Gong, B., Yuan, L., Cui, Y., Adam, H., Dvornek, N., Tatikonda, S., Duncan, J., and Liu, T. Surrogate gap minimization improves sharpness-aware training. arXiv preprint arXiv:2203.08065, 2022. \n\n[3] Vani, A; Tung, F; Oliveira G; Sharifi H. Forget Sharpness: Perturbed Forgetting of Model Biases Within SAM Dynamics, International Conference on Machine Learning (ICML) 2024.  \n\n- Another point to improve are the datasets. I strongly suggest the authors to benchmark with at least a couple large scale datasets. Options are ImageNet-V1 [4] for training and ImageNet-Real [5] and ImageNet-V2 [6] for testing, ImageNet-R [7] for out-of-distribution robustness benchmark and ImageNet-Sketch [8].  \n\n[4] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. ImageNet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255. IEEE, 2009 \n\n[5] Beyer, L., He ́naff, O. J., Kolesnikov, A., Zhai, X., and Oord, A. v. d. Are we done with imagenet? arXiv preprint arXiv:2006.07159, 2020. \n\n[6] Recht, B., Roelofs, R., Schmidt, L., and Shankar, V. Do imagenet classifiers generalize to imagenet? In Interna- tional conference on \nmachine learning, pp. 5389–5400. PMLR, 2019. \n\n[7] Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: A critical analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8340–8349, 2021. \n\n[8] Wang, H., Ge, S., Lipton, Z., and Xing, E. P. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pp. 10506–10518, 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weakness1 and Question\nWe present three key distinctions between DASH in [1] and SharpBalance. First, SharpBalance offers a comprehensive identification and rigorous analysis of the sharpness-diversity trade-off phenomenon. Second, our novel theoretical approach using random matrix theory provides precise quantifications and reveals the relationship between sharpness and diversity. Finally, our theory-motivated algorithm provably achieves greater diversity while maintaining the same level of sharpness, leading to an improved ensemble performance.\n\n1.  **(Comprehensive identification and analysis of the sharpness-diversity trade-off).** SharpBalance provides a thorough examination of the sharpness-diversity trade-off through extensive experiments across various settings, including different sharpness/diversity measures and different model capacities, and rigorous theoretical analysis with random matrix theory. Our findings theoretically proved the existence of such a trade-off and empirically demonstrated the universality of this phenomenon through extensive experiments. DASH proposed in [1], in contrast, offers an intuitive explanation of why reducing sharpness might decrease diversity. The explanation is that the decrease in diversity is a result of models being initialized closely and updated with the same mini-batch of data.\n2.  **(Tight quantification using random matrix theory).** SharpBalance develops a general theoretical framework using random matrix theory (RMT) to explain the interplay between sharpness and diversity in deep ensembles which allows us to derive tight quantifications for these two important metrics, providing a more accurate characterization of the training dynamics of ensemble models. We used these quantification results to characterize the relationship between sharpness and diversity, further validating the correctness of our empirical observation. DASH, on the other hand, provides an upper bound on generalization error with the sharpness of both base learners and the ensemble.\n3. **(Theory-based algorithm provably achieves improved performance).** We propose SharpBalance to balance sharpness and diversity within ensembles and theoretically show that the method achieves an improved trade-off between the two metrics. Empirical validations suggest our method indeed enhances the trade-off and therefore improves ensemble performance. The algorithm selects a sharpness-aware subset of data for each individual model to train on the sharpness objective and is simple, effective, and computationally cheap to implement. In contrast, DASH adds a KL divergence constraint on the output logits, which is different from our method which uses distinct subsets of data to train individual models. While their method will introduce diversity, our method is *theoretically guaranteed* to improve diversity. Furthermore, we have extensive experiments to demonstrate the improved diversity of SharpBalance. The two methods are in fact orthogonal and can be seen as complements of each other for promoting diversity.\n\n## Weakness 2\nWe conducted new experiments to include the stronger SAM method GSAM. The results are shown in Table 7 of the rebuttal PDF. We combine GSAM with Deep Ensemble as a new baseline method \"Deep Ensemble + GSAM\" and incorporate the GSAM into our method SharpBalance. The results show that the new baseline with GSAM outperforms the original baseline in ID and OOD performance but still underperforms SharpBalance (w/ SAM). Furthermore, we enhance SharpBalance by replacing the SAM with GSAM, which leads to better ID performance. The hyperparameter search and setup are consistent with Appendix D.3 of the submitted paper.\n\n## Limitation 1 and 2 \nWe thank the reviewer for the suggestions. In the revised draft, we will provide more precise statements on SharpBalance's performance in ID and OOD settings and clarify the contributions of [1] and our work, as suggested.\n\n**Reference**\n\n[1] Bui et al. Diversity-Aware Agnostic Ensemble of Sharpness Minimizers", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pgwQPVcvcI", "reviewer": "Reviewer_TwFd", "summary": "This paper investigates the sharpness and diversity within deep ensembles. Specifically, it identifies the trade-off phenomenon between sharpness and diversity with both theoretical and empirical evidence. Additionally, it proposes a method called SharpBalance, which trains individuals using selective 'sharp' subsets. Conducted experiments have demonstrated the effectiveness of the proposed SharpBalance when applied to deep ensembles.", "questions": "Please refer to the **Weaknesses**.", "limitations": "The authors have provided **Limitations** section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "There are several strengths in this paper:\n\n- The exploration of sharpness and diversity in deep ensembles is both interesting and novel.\n\n- Sufficient theoretical and empirical evidence has been provided for validation.\n\n- The proposed method is simple, effective, and accompanied by code for verification.", "weaknesses": "However, I still have the following concerns:\n\n - The evaluation seems a bit weak. The authors should consider comparing with more ensemble baselines.\n\n - What is the scale of $D_{SAM}^i$ and how does it change during training? Providing some details on this would help in understanding the proposed method.\n\n - Refer to Line 166: How do the authors train individuals with the full datasets? Are these individuals trained with different initializations?\n\n - (Optional) As described, the model's generalization is not merely correlated with sharpness, which aligns with some recent advanced SAM variants. Thus, integrating these advanced variants [1][2] with SharpBal would be more beneficial for studying the trade-off between sharpness and diversity.\n\nReferences:\n\n[1] Random Sharpness-Aware Minimization. In NeurIPS 2022.\n\n[2] Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization. In CVPR 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weakness 1\nIn addition to the main experiments, we compared SharpBalance with ensemble baselines, including those in the appendix and new experiments in the rebuttal PDF.\n1. **Ensemble with models trained with different hyperparameters.** In Appendix F.4, we compared with the \"SAM+\" baseline, which forms an ensemble using three models trained with different SAM perturbation ratios (0.05, 0.1, and 0.2).\n2. **Ensemble of moving averages (EoA) method [1].** In Appendix F.4, we also compared SharpBalance with EoA, a strong baseline that uses an efficient model averaging protocol.\n3. **Ensemble with other strong SAM optimizer GSAM [2].** In Table 7 of the rebuttal PDF, we presented new experiments with the \"Deep Ensemble + GSAM\" baseline, which replaces the SAM optimizer with GSAM.\nThe results show that SharpBalance outperforms all these baseline methods. \n\n## Weakness 2\nThe scale of $D_{SAM}^i$ is determined by the hyperparameter $k$. For each model, we choose the subset of data samples with the highest \"per-data-sample sharpness\" in accordance with the definition provided in Section 4.3 of the submitted paper. The subset $D_{SAM}^i$ is then formed by taking the union of all such subsets except the $i$-th. For instance, in the case of a three-member ensemble trained on CIFAR10, we set $k$ as 50, and the scale of $D_{SAM}^i$ is about 70% of the training set. $D_{SAM}^i$ is determined at a specific training step and remains constant for the $i$-th model until the end of the training, implying that its scale remains unchanged.\n\n## Weakness 3\nWe train each individual model with different random initializations and different data orderings, controlled by distinct random seeds.\n\n**Reference**\n\n[1] Arpit et al. Ensemble of averages: Improving model selection and boosting performance in domain generalization. \n\n[2] Zhuang et al. Surrogate gap minimization improves sharpness-aware training.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tQ5dnyp6pr", "reviewer": "Reviewer_b225", "summary": "This paper presents introduces a training approach for ensemble learning called SharpBalance to balance sharpness and diversity within ensembles. This paper shows theoretically that SharpBalance achieves a better sharpness-diversity trade-off.", "questions": "1. Why does SharpBalance seem more effective on corrupted data?\n2. Do models in an ensemble converge on the same local minima?", "limitations": "Limitations are adequately addressed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Ensemble learning is an important research direction.\n2. Understanding of sharpness and diversity within deep ensembles is important for the study of generalization to both in-distribution and out-of-distribution data.\n3. The paper is technically sound.", "weaknesses": "1. Since SharpBalance focuses \"on a diverse subset of the sharpest training data samples\", it may not apply in small datasets where available data is already sparse.\n2. Empirical improvement over existing methods is marginal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Weakness 1\nWe conducted an additional experiment to verify the effectiveness of the proposed method on small datasets, with results shown in Table 9 of the rebuttal PDF. The small datasets were generated by randomly subsampling the training set with ratios of 0.3 and 0.5. The experiments used a three-member ResNet18 ensemble on CIFAR10. The results demonstrate that SharpBalance maintains its performance advantage on small datasets compared to the two baseline methods. The hyperparameter search and setup are consistent with Appendix D.3 of the submitted paper.\n\n## Weakness 2\nWe clarify that SharpBalance demonstrates more pronounced empirical improvements as the number of ensemble models increases. In Figure 16 of the rebuttal PDF, we present additional results showing the impact of increasing the number of models in the ensemble. The accuracy difference between SharpBalance and the baseline methods becomes more significant, especially on corrupted data. Specifically, SharpBalance outperforms the baseline by up to 1.30% when ensembling 5 models on CIFAR100-C.\n\n## Question 1\nEnsembles trained with SharpBalance achieve greater diversity among individual models while maintaining their individual predictive power. The enhanced diversity allows the ensemble model to perform well under distribution shifts in the noisy dataset. This is because, in a diverse ensemble, each individual model may capture more distinct features of the data distribution and effectively mitigate the impact of corrupted data [1-3]. Also, research in [4] suggests that the high diversity of the features learned by the model promotes transferability to OOD data. On the other hand, the improved sharpness-diversity tradeoff also reduces the sharpness of the overall ensemble model, which reduces the impact of overfitting. On corrupted datasets, overfitting is more harmful to the ensemble generalization performance due to the presence of noisy features, which makes SharpBalance more effective over baseline methods.\n\n## Question 2\nRigorously defining \"local minimum\" in light of mode connectivity [5] can be tricky. Here we only provide an intuitive answer using commonly acknowledged statements in this field, which may be imprecise under certain conditions. In general, models in an ensemble are unlikely to converge to the same local minima. Studies in [6] show when models are randomly initialized, the training trajectories of these models tend to explore diverse modes (minima) in the loss landscape and as a result, do not converge to the same local minimum. In particular, ensemble members trained by SharpBalance are unlikely to converge to the same local minimum either. This is implied by the increased diversity of the individual model's output as models residing in the same local minimum are more likely to output similar logits resulting in a low diversity among ensemble members. Notice that this explanation is based on the common intuition that a loss landscape indeed contains many minima. If, in the overparameterized case where all local minima are connected through low-loss paths [5], this statement has to be taken with a grain of salt.\n\n**Reference**\n\n[1] Abe et al. Pathologies of Predictive Diversity in Deep Ensembles.\n\n[2] Stickland et al. Diverse Ensembles Improve Calibration.\n\n[3] Kumar et al. Calibrated ensembles can mitigate accuracy tradeoffs under distribution shift.\n\n[4] Nayman et al. Diverse Imagenet Models Transfer Better.\n\n[5] Garipov et al. Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs.\n\n[6] Fort et al. Deep Ensembles: A Loss Landscape Perspective.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We want to thank all the reviewers for the constructive feedback, which helps us improve our paper. Please refer to the attached PDF for our new experiments and see below for our responses to each comment.

Meta-review: While none of the reviewers expressed strong enthusiasm for the paper, they agree that it presents a solid framework offering a theory on the sharpness-diversity tradeoff, along with an algorithm to enhance the diversity of flat ensembles. The topic is very important to the ensemble learning and Bayesian deep learning communities, and I believe the proposed theory on the tradeoff would be a valuable contribution to the field. Although there are some valid concerns about the experiments, I believe these do not outweigh the merits of the work.
Decision: Accept (poster)