Ex Uno Pluria: Insights on Ensembling in Low Precision Number Systems
Authors: Giung Nam
Juho Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: ensemble methods
low precision number systems
Primary area: other
TL;DR: We present a novel insight on ensemble within low precision number systems, especially for large models.

Abstract: While ensembling deep neural networks has shown promise in improving generalization performance, scaling current ensemble methods for large models remains challenging. Given that recent progress in deep learning is largely driven by the scale, exemplified by the widespread adoption of large-scale neural network architectures, scalability emerges an increasingly critical issue for machine learning algorithms in the era of large-scale models. In this work, we first showcase the potential of low precision ensembling, where ensemble members are derived from a single model within low precision number systems in a training-free manner. Our empirical analysis demonstrates the effectiveness of our proposed low precision ensembling method compared to existing ensemble approaches.

Ex Uno Pluria: Insights on Ensembling in Low Precision Number Systems
Giung Nam Kim Jaechul Graduate School of AI KAIST, Daejeon, South Korea giungākaist. ac
Juho Lee
Kim Jaechul Graduate School of Al KAIST, Daejeon, South Korea juholee@kaist. ac.kr kr
Abstract
While ensembling deep neural networks has shown promise in improving general- ization performance, scaling current ensemble methods for large models remains challenging. Given that recent progress in deep learning is largely driven by the scale, exemplified by the widespread adoption of large-scale neural network ar- chitectures, scalability emerges increasingly critical issue for machine learning algorithms in the era of large-scale models. In this work, we first showcase the po- tential of low precision ensembling, where ensemble members single model within low precision number systems in a training-free manner. Our empirical analysis demonstrates the effectiveness of our proposed low precision ensembling method compared to existing ensemble approaches.
derived from
1 Introduction
In computer science, it is de facto standard to represent continuous real numbers using finite pre- cision number systems. While many applications rely on precision formats like FP32 FP64, the deep learning community is increasingly turning to 16-bit floating-point formats such as FP16 (Mi- cikevicius al., 2018) or BF16 (Dean et al., 2012) reduce memory usage during training. More recently, researchers are further exploring low precision optimization, aiming to utilize fewer bits (8 bits less) represent weights, activations, and gradients throughout the training process (Gupta et al., 2015; Li et al., 2017; Sun et al., 2020; Wortsman et al., 2023).
While low precision number systems aid in training deep neural networks, they also ben- eficial for reducing inference costs in roal-world deployments of such models (Jacob et al., 2018). In particular, recent advancements in large language models (Brown al., 2020; Touvron et al., 2023) containing billions of parameters have triggered active exploration of post-training quantiza- tion techniques, for deploying pre-trained large language models on hardware with limited memory resources. This exploration encompasses quantizing both weights and activations (Dettmers et al., 2022: Yao al., 2022), as well quantizing weights only (Frantar et al., 2023; Lin et al., 2024).
Originally, the goal of post-training quantization is to specify the best solution represented in low precision number systems, aiming reduce discrepancies from the original high-precision weights, like perturbations in weight values or increases in loss functions (Nagel et al., 2020). On the other hand, the presence of numerous distinct yet high-performing models within a single basin loss landscapes (Sadrtdinov et al., 2023; Lion et al., 2024) evokes a Bayesian concept of marginalization instead of optimization, which involves utilizing multiple solutions rather than relying solely one solution (Wilson and Izmailov, 2020).
Hinging this insight, suggest building ensembles within low precision number systems, as illustrated in Fig. 1. depicts a proof-of- of-ecccp--ccpttt method for ensemble construction using stochastic rounding, technique commonly used in low precision training to address the issue of rounding
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(b)
(d0
Figure 1: Concepts of low precision ensembling. It shows two-dimensional schematic, where the x and axes represent the neural network weights, while the contours above visualize the loss surface. (a) Let the pre-trained weights, denoted by a yellow star-shaped marker (1²), be positioned within basin on the loss landscape. In general, (b) post-training quantization methods introduce lower precision number systems, and then (c) choose one candidate from the system, such as the nearest one. (d) However, there are many other highly effective models available, that can contribute to ensemble predictions.
weight updates below the minimum precision threshold to zero (Gupta et al., 2015). In our approach, stochastic rounding employed for ensembling rather than optimization. Indeed, our experimental findings in Section 4 validate that low precision ensembling improves the downstream performance of pre-trained large models without any further training on downstream data.
Confirming the potential of low precision ensembling for pre-trained models, we extend our inves- tigation through comparative study with existing methods involving ensembling. Specifically, examine Bayesian approaches that approximate a Gaussian posterior over the loss landscape (Mad- dox al., 2019; Shen et al., 2024), and sampling techniques that collect model copies from the update trajectory (Huang et al., 2017: Zhang et al., 2020). Our experimental results in Section show that low precision ensembling successfully gathers diverse ensemble members contributing the final ensemble performance within the low precision number system.
The main contributions of our work can be outlined as follows:
Introducing low precision number systems inevitably results in quantization errors, usually seen as a flaw to be corrected in neural network quantization. Our work presents novel viewpoint: these errors can be utilized as source to improve ensemble diversity. Expand- ing our comprehension of diversity, we suggest simple yet powerful approach to con- structing ensembles called Low Precision Ensembling with Bernoulli Stochastic Rounding (LPE-BSR), particularly advantageous for large models.
The proposed LPE-BSR involves assembling ensemble members with low precision num- ber systems, effectively addressing notable challenge associated with memory costs inher- ent in ensemble methods. In this regard, our work holds promise for utilizing low precision number systems construct ensembles of large models, offering a potential solution for the scalability issue faced by the Bayesian deep learning community in the era of large-scale models (Papamarkou al., 2024).
2 Ensemble methods in modern transfer learning scenarios
Ensembling neural networks a long-established idea in machine learning (Hansen and Salamon, 1990), where the underlying design principle create strong hypothesis that effectively explains the data by combining a set of weak hypotheses (Kearns, 1988). Notably, even after the transition into the deep learning era, ensemble methods continue serve as straightforward yet powerful strategy for boosting the performance of machine learning algorithms involving deep neural net- works (Krizhevsky et al., 2012; Ciresan al., 2012). However, the operational principles of such deep ensembles, i.e., ensembles composed of deep neural networks, deviate from those of classical statistical models and remain not fully comprehended. For instance, both Lee et al. (2015) and Nixon et al. (2020) validated that bugging (Breiman, 1996), built the theoretically well-motivated boot- strap method (Efron, 1992), does not offer any benefits over the simplest deep ensembles consisting of models obtained from multiple training runs with different random seeds.

---

Empirically, it is well-known that we need nothing more than employing different initializations for each ensemble member construct perf-eerrraacee deep ensembles (Lakshminarayanaal et al., 2017). Fort et al. (2019) delved deeper into this and emphasized the significant role played by highly non-convex loss function of deep neural networks, where varying initializations for stochas- tic optimization yield different functional modes. It also aligns with the Bayesian perspective pro- vided by Wilson and Izmailov (2020), which suggests that deep ensembles are involved in approx- imating multimodal posterior distribution to the Bayesian model averaging. To sum up, while the operational principle of deep ensembles may differ from classical ensembles, the underlying idea re- garding diversity remains constant (Krogh and Vedelsby, 1994; Ortega al., 2022), i.e., ensembles demonstrate improved performance when their individual members offer diverse predictions.
However, the simple strategy mentioned earlier, which aims to cover multiple modes on the loss landscape by starting from different initializations to achieve ensemble diversity (Fort et al., 2019; Wilson and Izmailov, 2020), faces challenges in modern transfer learning scenarios. It arises from the typical situation that there is only one pre-trained model available for fine-tuning: due to the con- siderable cost for pre-training of large models, model providers usually do not distribute multiple model copies. In such cases, fine-tuned solutions originating from the same pre-trained weights often inhabit the same pre-trained basin, leading to restricted exploration within the loss land- scape (Neyshabur et al., 2020; Mustafa et al., 2020). Consequently, our attention should be directed towards addressing the local structure around the pre-trained basin in modern transfer learning sce- narios (Wottsman et al., 2022; Sadrtdinov et al., 2023; Lee et al., 2024), rather than the global multimodal structure of the loss landscape (Fort et al., 2019; Wilson and Izmailov, 2020).
3 Preliminaries
Finite precision number systems. Computers use binary sequences for data encoding, with FP32 serving as the primary finite precision number system employed to represent real numbers. Given its coverage from approximately 10-35 10a8 with a resolution of 10-7, we consider the FP32 system as the continuous set of real numbers TR. Moreover, we describe INT-B systems, commoly utilized low precision number systems for neural network quantization:
Wabanax 28-1-1 : m E (-2"-1 + +1, +....., O. 2- 1}}
F FINT-B=
(m x
(1)
where the system can represent real numbers in the range [aaamaaax Wabanax] with a resolution of Wabemax/ (2 (2B-1 1), and the integer m can be encoded using B bits. This is the simplest form of integer quantization, with possible variations like a zero offset or non-uniform resolution (Gholami et al., 2021; Yvinec et al., 2023). Unless otherwise specified, our experimental results employ this basic form of symmetric uniform quantization with B 5 for simplicity. We also use per-channel granularity, sharing the number systems among the output channels of each linear layer.
Rounding rules. Let F c R be finite precision number system. For any w e I. there are two rounding options in practice: [w] max ( t0 E F: ④ S w) and [w] min (Ø ε F: w 2 w). The rounding-lo-neure (RTN) scheme deterministically selects the closest one, i.e., the Bernoulli stochastic rounding (BSR) scheme randomly chooses one of them, i.e.,
[w], whereas
w = [w] with probability |w] - ww, or Φ = [w] with probability w [w]
(2)
Observing empirically that BSR sometimes produce superior results compared to RTN motivates the neural network quantization community to explore more sophisticated rounding schemes (Nagel et al., 2020; Lee et al., 2023). On the other hand, recognizing the presence of multiple competitive solutions, we consider leveraging them for low precision ensembling.
Ensemble methods. In ensemble methods for classification problems, the final prediction during testing is obtained by combining S predictions:
g _ p(ylar, wx), S --1
p(yxx)
(3)
where y is a class label, x is an imput, and p(y/2x, w.) is the categorical probabilities predicted by the th member, which is neural network model with parameters w.. These w, could either max- imum posteriori (MAP) solutions obtained through multiple stochastic optimizations (Lakshmi- narayanan et al., 2017), or they might be intermediate checkpoints from a single training run (Huang et al., 2017; Garipov et al., 2018).
3

---

Table 1: Motivating results for low precision ensembling of pre-trained ViT models. Negative log- oggllikelhhood (NLL), classification error (ERR), and ensemble ambiguity (AMB) for rounding- to- nearest (RTN) and low precision ensembling with Bernoulli stochastic rounding (LPE-BSR) derived from the publicly available pre-trained ImageNet model (1²). Blue highlights the areas where LPE- BSR excels, particularly in larger models and lower precision settings.
VIT-T/16 (6M)
ViT-S/16 (22M)
VIT-B/16 (S7M)
ViT-L/16 (307M)
Method
System
NLL
ERR
AMB
NLL
ERR
AMB
NLL
ERR
AMB
NLL
ERR
AMB
# Pre-trained
FP32
.922
243
.667
.687
.182
.639
.165
RTN
INT-6
.948
.247
.671
.185
.687
.182
.639
.165
INT-4
1.23
.315
.822
.647
.167
LPF-BSR
INT-6
.932
.245
.024
.665
.185
.014
.681
.181
.006
.632
.164
.003
INT-4
1.30
.489
.211
.268
.088
.037
In the Bayesian framework, neural network weights w are treated as random variables, and w, are seen as Monte Carlo samples employed approximate the posterior distribution. More pre- cisely, Eq. 3 can be seen as a simple Monte Carlo method to approximate the posterior with a set of point masses, where the locations are given by samples from another upproximate posterior q, i.e. pl () Σ._1 5(w w.)/S, ws q(wr, where DD denotes the data and is the Dirac delta function (Wilson and Izmailov, 2020). A common practice is to use Gaussian approximation ((w) -w: ur Σ) to generate samples w, ((w) in a tractable manner (Maddox et al., 2019; Shen et al., 2024), and then approximate the predictive distribution by computing
1 8 E S E(ul,, ws). w Ws (w)).
p(ylx)
(4)
4 An empirical study of low precision ensembling
We present simple yet effective low precision ensemble construction strategy, Low Precision En- sembling with Bernoulii Stochastic Rounding (LPE-BSR), which computes Eq. 4 using
D ((w) ()), ((w())= >u (6.0 - (1) + + (1) ((w - (20(1)).
(5)
6-1
for i I,...,.. D. Here, D denotes the number of neural network parameters, where each per- channel parameter group shares the same low precision number system, as explained in Section 3. Using the rounding operations [-] and |-] defined within each system, the probability is determined by X; = [w;] Wi, as in Eq. 2. Certainly, the proposed LPE-BSR is not Bayesian, and we have simply expressed in the form of Eq. for the sake of notational simplicity.
4.1 Motivation: training-free free ensemble construction of large ViT models
We begin with motivating experiments using the publicly available series of pre-trained vision trans- former models (VIT; Dosovitskiy et al., 2021). Detailed information about each model can be found in Appendix A. Table summarizes the evaluation results on subset of the ImageNet validation split, along with the parameter count for each model. In this context, the pre-trained model corre- sponds to the star-shapred marker (œz) in Fig. 1, with RTN using the nearest value in low precision number systems as shown in Fig. I(c), and LPE-BSR forming ensemble by selecting S = 10 nearby samples as illustrated in Fig. ((d).
Table 1 provides the following key findings: 1) Larger models experience less performance degrada- tion when reducing the precision of numerical systems. More precisely, in the RTN results, the clas- sification eтror increases from .243 + .247 .311 when transitioning from FP32 + INT-6 + INT-4 at ViT-T/16, whereas at ViT-L/16, shifts from .165 + .165 .167. 2) Lower precision systems introduce diversity among samples in LPE-BSR. Specifically, ensemble ambiguity is the metric for quantifying the ensemble diversity (to be defined in Section 4.3), and in all models, the ensemble ambiguity increases when transitioning from INT-6 INT-4.

---

SGD
SWAG
IVON
MAP
MAP
MAP
0.490
LPE-85R
0.490
0.490
NLI
LPE BSR
LPE BSR
0.480
0.480
0.480
0.470
0.470
0.470
20
15
20
15
Ensemble size
Ensemble size
Ensemble size
Figure 2: Comparing low precision ensembling to Bayesian methods. Negative log- lieeliiel for Bayesian model averaging using an approximate Gaussian posterior derived from SWAG or IVON (BMA, shown in orange) and low precision ensembling with Bernoulli stochastic rounding centered around the MAP solution obtained by each optimizer (LPE-BSR, shown in green).
Importantly, these findings are in line with the core principle of ensemble methods being effective: a key condition for an ensemble of classifiers to outperform any of its individual members when the classifiers are both accurate and diverse (Dietterich, 2000). When introducing the low precision number system, 2) suggests that diversity can be achieved by leveraging the quantization error inher- ent in this process, while 1) emphasizes that larger models maintain accurate individual performance throughout this process. With this compelling motivation for low precision ensembling established, proceed compare it with existing ensemble methods.
4.2 Comparative study to Bayesian methods
Our initial investigation into the proposed LPE-BSR aims assess the effectiveness of collecting ensemble members within the discrete space defined by the low precision number system. While using fewer bits to represent samples would certainly reduce ensemble costs, there is a concern that we might overlook potentially good ensemble candidates outside this discrete space. To address this, conduct comparative study with two Bayesian deep learning methods: improved variational online newton (IVON; Shen et al., 2024) and stochastic weight averaging Gaussian (SWAG; Maddox al., 2019). Both methods samples drawn from an approximate Gaussian posterior in the continuous weight space and perform ensembling in a Bayesian manner.
To sum up our experiment, we first fine-tune the zero-shot CLIP-ViT ViT model (Radford et al., 2021) on the ImageNet training split to obtain the MAP solution WMAP- For LPE-BSR, this fine-tuning can employ any of the SGD, SWAG, or IVON optimizers. We then define q according to Eq. 5 using "MAP and compute Eq. 4 with samples. For SWAG and IVON, ((w) N(w; μa: Σ) is defined using the f WMAP and Σ obtained during their respective optimization processes, followed by the computation of Eq. 4. Consequently, LPE-BSR samples neighboring points of WHAP within the discrete space defined by the low precision number system, whereas SWAG and IVON sample nearby points of WMAP in the continuous space with Gaussian noise added. For more details on SWAG and IVON, including their hyperparameters, please refer to Appendix E.
Fig. 2 presents the outcomes our experiments conducted with CLT--VYT-L14, In our experimental setup, both SWAG and IVON successfully estimated both the MAP mean and the covariance matrix, resulting in a lower negative log-likelihood compared to MAP through Bayesian model averaging. as illustrated in the second and third subplots of Fig. 2. Remarkably, our IPF-BBR, derived from the MAP solution obtained by the SGD optimizer, achieves competitive results with Bayesian model averaging through SWAG IVON. Moreover, when using the same MAP solution obtained with the SWAG or IVON optimizer for a fair comparison, it even outperforms these methods.
From numerical integration perspective (Wilson and Izmailov, 2020; Wilson, 2021), the condi- tions for successful approximate Bayesian inference in deep learning are very similar to those for successful ensembling, as discussed in Section 4.1 with reference to Dietterich (2000). Specifically, entails 1) finding typical points the posterior that represent regions of substantial mass (cf. accurate); and (ii) ensuring a diverse set of points to give rise to different functions (cf. diverse). Consequently, we proceed to conduct comparative analysis of these two factors concerning our L.P--BRR method and the Bayesian methods we considered, SWAG and IVON.
5

---

Table 2:
Results for low precision ensembling of fine-tuned models. We compute (a) average loss. (b) ambiguity, and (c) ensemble loss for diversity analysis, along with evaluation metrics to assess overall performance. Both BMA and LPE-BSR samples are centered around & within each group (MAP in this context), which are separated by horizontal lines.
Diversity analysis
Evaluation metrics
Optimizer
Method
System
(a)
(b)
(c)
NLL
ERR
ECE
SWAG
* MAP
FP32
.488.003
4882.000
.882.002
.1372001
.0342001
BMA
FP32
.4982.002
.015..00
.881.001
.477..002
.1361.001
.021.001
LPE-BSR
INT-6
.492.003
.007.000
.885..003
.483..002
.1361.001
.0311001
INT-5
.577.002
.0261.00 000
..81
.473..002
.1361.001
.021..01
INT-4
.643.0011
.12
.5142:003
.513.0022
.1462.001
.027.000
IVON
* MAP
FP32
490a.002
.490..002
.489..001
.1362.001
.037..01
BMA
FP32
.503.001
.001
.483.0001
.475.001
.135a.000
.0262.00
LPE-BSR
INT-6
.492..001
.006..000
.866.001
.883.001
.135s.000
.033+001
INT-5
.507.001
.025a.000
.8.10001
.472.000
.135..000
.0230.00
INT-4
6422.003
.131
.512+.002
.509..001
.145a.001
.0262.011
SGD
MAP
FP32
4922.002
.92+.002
.492.0002
.138
0035+001
LPE-BSR
INT-6
.495..001
.006
.888.001
.485a.001
.138=.000
.030.000
LPE-BSR
INT-5
.513.0001
.057+.001
.456.000
.477..001
.1372.000
.020=001
LPE-BSR
INT-4
.663s.003
.120x.00
.544.002
.526=.001
.150s.001
.029
1.5
0.5
0.47
0.5
0.05
0.07
0.0
0.3
14
0.0
0.5
0.40
0.5
0.00
0.00
Figure 3: Comparison between IVON and LPE-BSR samples. Radial landscape plots visualize plane subspace defined by three points: the MAP obtained by IVON (depicted yellow star t)), samples in BMA and LPE-BSR procedures (represented by blue and red circle markers o).
4.3 Diversity analysis of ensemble methods
Quantitative assessment of ensemble diversity is an essential metric for evaluating ensemble meth- ods. To this end, we adopt the generalized ambiguity decomposition for cross-entropy loss presented by Wood et al. (2023), which can be easily measured using logit ensembling instead of probabil- ity ensembling. It should be noted that logit ensembling is solely utilized for diversity analysis, whereas probability ensembling is used for all other experimental results to ensure fair compari- son with Bayesian methods that compute the categorical predictions using the BMA integral. Please refer to Appendix B for more details on our diversity analysis.
Table 2 presents our experimental outcomes for the generalized ambiguity decomposition (cf. 'Di- versity analysis"), ) along with the final ensemble perfomance (cf. 'Evaluation metrics"). Lowering the precision of numerical systems naturally amplifies quantization error, as evidenced by the re- sults in the INT-B rows, where reducing B results in higher (a) average loss. However, this also coincides with an increase ensemble diversity, with smaller values resulting in greater (b) am- biguity. Consequently, the superior (c) ensemble loss at an appropriate precision level (e.g.. B 5) highlights the fundamental principle of the low precision ensembling: il dues not merely perceive quantization error problematic, but rather utilizes it to obtain ensemble diversity. Consequently, the proposed LPE-BBRR yields improvements in evaluation metries, as depicted in Table Definitions for each metric can be found in Appendix B.

---

1.5
0.63
1.5
025
15-
0.16
1.0
1,0
1,0
0.49
0.5
0.11
0.08
-4.
0.36
-0.5
0.00
-0.5
0.00
-2.5
0.0
0.5
1.0
1.5
-0.5
0.0
0.5
1.0
1.5
-0.5
0.0
0.5
1.0
1.5
Figure 4: Comparison between snapshot and LPE-BSR samples. Radial landscape plots visual- ize a plane subspace defined by three points: the first and second snapshot samples obtained by SSE (represented by yellow and blue star-shaped marker t²), and LPE-BSR sample derived from the first snapshot (depicted as red circle o).
Drawing inspiration from Fort al. (2019), we further provide the radial landscape plot depicting plane within weight space containing three points (shown as yellow, blue, and orange markers). The z-values for each subplot represent the negatíve oggliikelihood (displayed on the left in a magma colormap), the function differences from the blue model (shown in the center in a blue colormap), and the red model (presented on the right in red colormap). Namely, the first subplot indicates the placement of each model within the loss landscape, while the subsequent two subplots illustrate the extent which they differ from each other.
Fig. 3 depicts radial landscape plots comparing IVON and LPE-BSR samples, showing their parallel roles in ensemble construction. Both show slightly higher individual negative log-likelihoods, shown by the circle markers, compared to the MAP denoted by a star-shaped marker in the first subplot, while also offering diverse function outputs as demonstrated in the subsequent subplots. Ultimately, LPEE-BR can identify samples in a low precision number system that qualitatively resemble the high-quality posterior samples provided by IVON, which leverages Hessian estimate information to compute the covariance of the approximate Gaussian posterior.
4.4 Combining with fast ensembling methods
A key advantage of LPE-BSR is its ability to gather ensemble members without requiring any back- ward computation. This feature, which eliminates the need for training, aligns with fast ensembling techniques aimed at enhancing the training efficiency of ensemble construction processes (Huang al., 2017; Garipov et al., 2018; Benton et al., 2021). Consequently, we conduct empirical analysis to further investigate this alignment with snapshot ensembling (SSE; Huang et al., 2017), well as cyclical stochastic gradient Langevin dynamics (CSGLD; Zhang et al., 2020), a closely related Bayesian posterior sampling algorithm. Both methods involve collecting snapshot samples the loss landscape around WMAP using a cyclical learning rate schedule. For more details, including hyperparameters, please refer Appendix E.
We first verify whether LPE-BSR can generate an ensemble component distinct from SSE snapshots using radial landscape analysis. Fig. 4 illustrates a plane subspace containing the first and second SSE snapshots (displayed as star-shaped markers) and the LPE-BSR sample obtained from the first snapshot (represented by circle marker). Indeed, LPE-BSR provided novel sample that could contribute the ensemble; it is clearly diverse from the existing snapshots, as shown the second and third subplots, while achieving reasonably low individual negative log-likelihoods, the first subplot. By using such LPE-BSR samples along with SSE snapshots to build an ensemble, reduce the cost of achieving target performance in fast ensembling methods or attain better results with the same training budgets.
shown in
However, while fast ensembling techniques usually prioritize evaluating training budgets, particu- larly the number of backward passes as seen in the literature (Huang et al., 2017; Garipov et al., 2018: Zhang et al., 2020), we also consider memory budgets in our analysis the total number of bits required to represent the entire ensemble model, which grows with the addition of ensemble members in fast ensembling. From this perspective, we devised method to eliminate heavy SSE snapshots with high precision from the final ensemble; as a result, each original SSE snapshot is re- placed by 5 5 LPE-BSR samples. This policy also aligns with our reserach objective of exploring
7

---

SSE LPE-85R
LPPE-5RR
0.030
0.480
0.480
0.025
0.025
NLL
NLL
0.470
0.470
0.020
0.020
0.015
0.460
0.460
200
300
500
1000
300
500
1000
Training budgets (x10*)
Memory budgets (x10F)
Training budgets (x]0*)
Memory budgets (x10P)
SGLD
SGLD LPE-BSR
SGLD
SGLD + LPE-BSR
0.490
0.490
0.480
NLL
0.480
0.025
0.025
0.020
0.020
0.0
0.470
0.470
0.015
000
0.460
200
300
500
1000
200
300
500
1000
Training budgets (x100)
Memory budgets (x10*)
Training budgets (x10')
Memorr budgets (x10*)
Figure 5: Combining with fast ensembling methods. Negative log-likelihood and expected cali- bration error for fast ensembling methods, SSE and CSGLD, in terms of training budgets, i.e., the number of backward passes, and memory budgets, i.e., the total number of bits for representing en- semble. Top: Results with SSE. Bottom: Results with CSGLD.
1.0
1.3
0.5
0.99
0.5
0.11
0.5
0.10
0.0
0.0
0.9
0.88
0.00
0.00
Figure 6: Radial landscapes for zero-shot CLIP- model. Radial landscape plots vi- suaslize plane subspace defined by three points: a pre-trained model (depicted as yellow star- shaped marker xt) and two LPE-BST BSR samples derived from the pre-trained weights (represented by blue and red circle markers o).
ensembling in low precision number systems. Consequently, by forming the ensemble exclusively with LPE-BSR samples in the low precision system, attained better outcomes compared to SSE regarding both training budgets and memory budgets, as shown in Fig. 5.
4.5 Training-fre ensemble construction of pre-trained large models
Our investigation so far, involving deep neural networks up to 300M in size, substantiates the ef- ficacy of the proposed LPE-BSR methodology. In this section, we further extend our validation to confirm that LPE-BSR enables effective ensemble construction without training, models. To this end, in addition to 300M-scale CLIP-ViT-L/IE model (Radford et al., 2021), we em- model (Cherti al., 2023) and an 8B-scale LLaMa model (Touvron
in larger
ploy IB-scale CLIP-ViT-OR et al., 2023) in a zero-shot manner. Appendix A provides public links for each model.
We first present radial landscape analysis of the CLIP-ViT-L/I4 model in Fig. 6. As we analyzed previously in Section 4.3, we can confirm that the conditions for effective ensemble construction are met here as well, i.e., ensemble members exhibit slightly higher individual negative log-likelihoods (circle markers) compared to the pre-trained model (star- marker) the first subplot, and they also offer diverse function outputs, as shown in the subsequent subplots. As depicted in the leftmost.

---

CLIP-VIT-L/14 (304M)
CLIP-VIT-G/14 (1B)
LLaMa-3 (8B)
0,980
Pro Pre-trained
Fro Pro-trained LPE-BSR
Pro tro-trained
LPE-BSR
1.025
LPE-BSR
0.960
0.940
0.940
0.975
20
20
15
Ensemble size
Ensemble size
Ensemble size
CLIP-VIT-L/14 (304M)
CLIP-VIT-G/14 (1B)
LLaMa-3 (8B)
0.980
Pre-traince LPE-B5R
pro-trained LPE-85R
0.950
1.025
LPE-BSR
NLI
0.960
0.940
1.000
0.940
0.930
0.975
100
200
500
1000
1500
2000
4000
6000
Momory budgets (x10*)
Momory budgets (x11")
Memory budgets (x10*)
Figure 7: Constructing low precision ensemble of large models. Negative log- liglliiel for pre- trained models (Pre-trained, shown in blue) and low precision ensembling with Bernoulli stochastic rounding centered around the pre-trained model (LPE-BSR, shown in green). The evaluation was conducted on ImageNet for CLIP models and on MMLU for LLaMa-3 in zero-shot setting. Top: When the x-axis represents the ensemble size. Bottom: When the x-axis represents memory bud- gets, i.e., the total number of bits for representing ensemble.
Table 3: Results for low precision ensembling of pre-trained models. We compute (a) average loss, (b) ambiguity, and (c) ensemble loss for diversity analysis, along with evaluation metrics to assess overall performance. Our LPE-BSR samples are centered around & within each group (pre- trained model in this context), which are separated by horizontal lines.
Diversity analysis
Evaluation metrics
Model
# Params
Method
System
(a)
(b)
(c)
NLL
ERR
ECE
CLIP-VIT-L/14
0.30B
& Pre-trained LPE-BSR
FP32
.488
.948
.988
.251
.049
INT-5
,993
.053
.940
.929
.250
.028
CLIP-VIT-G/14
LOIB
Pre-trained LPE-BSR
FP32
.922
.942
.942
.206
.095
INT-5
.955
.013
.941
.927
.206
.089
LLaMa-3
8.03B
* Pre-trained
FP32
1.03
1.03
1.03
.361
.160
LPE-BSR
INT-5
1.07
.049
1.02
.923
.364
.087
subplot of Fig. 7 it leads to achieving lower negative log-likelihood through LPE-BSR compared to the pre-trained model without the need for additional training.
Fig. 7 demonstrates that LPE-BSR consistently improves upon the pre-trained checkpoint, even for larger models such as CLIP-ViT-G/14 and LLAMa-3. The subplots at the top of Fig. 7 demonstrate that the performance of LPEE-BSR improves with increasing ensemble size, while the subplots at the bottom show that LPE-BSR occupies the preferred lower-left region of the trade-off plots for mem- ory budgets and performance, surpassing the pre-trained checkpoint. Table 3 offers more detailed results for an ensemble size of S 20, including diversity analysis and evaluation results, further confirming the effectiveness of our proposed LPE-BSR for models with billions of parameters.
5 Related Work
The field of Bayesian deep learning provides the most relevant research for our low precision en- sembling strategy; Ferianc et al. (2021) integrated quantization-avwe training (Jacob et al., 2018) into established Bayesian deep learning methods; Zhang et al. (2022) introduced a technique for implementing stochastic gradient Langevin dynamics (Welling and Teh, 2011) with reduced preci- sion. However, our work differs significantly from theirs in two key aspects: 1) They focused on

---

training from scratch, which deviates somewhat from the prevalent practice of utilizing praitedined large models. 2) They employed small-scale models; the largest model they considered, ResNet- 18 with 11 million parameters, falls outside our scope as discussed in Section 4.1, as we are interested in larger scales. Nonetheless, the interest employing low precision ensembling Bayesian deep learning holds significant promise. Our demonstration of its feasibility for large models constitutes a meaningful advancement for the Bayesian deep learning community (Papamarkou et al., 2024).
6 Conclusion
We provided a novel insight on ensembling within low precision number systems. While conven- tional wisdom perceives quantization errors stemming from representing neural network weights in low precision as obstacles, we introduced an innovative viewpoint suggesting that these enrors could serve as a source of ensemble diversity. Our empirical results demonstrated that low precision weights obtained through stochastic rounding of pre-trained weights could effectively form ensem- bles and improve uncertainty estimates and calibration, especially for large models. Considering the growing scale of models in recent trends reduces the appeal of ensemble methods due to their inherent scalability issue, where memory costs increase with the number of ensemble components, our exploration of low precision ensembling lays the foundation for developing efficient ensemble methods in the era dominated by large models.
Limitations and future directions. At present, our investigations have centered on the simplest form of low precision number system, known as the symmetric uniform quantization scheme. Simi- lar to the quest in neural network quantization for systems that yield better quantized solutions (e.g., Yvinec et al.. 2023; Dettmers et al., 2024), the search for systems conducive to more effective low precision ensembling presents an intriguing avenue for future research. Furthermore, we used fake quantization across all experiments for research purposes, which prevented us from benchmarking the latency of the low precision ensemble due to limited access to specialized hardware and soft- ware for accelerating the inference speed of quantized models. Nonetheless, as our work relies on the standard symmetric uniform quantization scheme, remains compatible with ongoing and fu- ture advancements in neural network quantization. Developing practical components such as custom CUDA kernels tailored to low precision ensembles would also be a promising future direction.
Broader impacts. Our method advocates for the utilization of large models, which could potentially raise ethical concerns (e.g.. Weidinger et al., 2021). However, is important to note that this work primarily focuses on analytical aspects and does not inherently entail significant ethical risks.
Acknowledgement
This work was partly supported by Institute of Information & communications Technology Planning & (II)))tion(IITPP) grant funded by the Korea govemment(MSII)
Artifi-
cial Intelligence Graduate School (oogram(KAIST) AI Technologies to Inexpensively Conform to Evolving Policy on Ethics, No.RS-2024 -02-2224000009270 Global AI Frontier Lab), and National Research Foundation of Korea(NRF) grant funded by the Ko- SRRE=202/M355D002000
No.2022-0-00188 84, Development and Study of
This material based upon work supported by the Google Cloud Research Credits program with the award GCP1998090) and Cloud TPUs from Google's TPU Research Cloud (TRC).
rea goVIramen(MMSIT)
References
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wat- tenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learn- ing on heterogeneous systems, 2015. URL https: //t:://wwwww /www. fansorflow org/.
10

---

Gregory Benton, Wesley Maddox, Sanae Lotfi, and Andrew Gordon Gordon Wilson. Loss surface simplexes for mode connecting volumes and fast ensembling. In International Conference on Machine Learning (ICML), 2021.
Charles Blundell, Julien Cornebise, Koray Kavukeuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference on Machine Learning (ICMI), 2015.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milnn and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//github. com/ cttp:/ogle/////e/o/oooo google/
Lco Breiman. Bagging predictors. Machine learning, 24:123-140, 1996.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive aanguage-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Dan Ciresan, Alessandro Giusti, Luca Gambardella, and Jürgen Schmidhuber. Deep neural networks segment neuronal membranes in electron microscopy images. In Advances in Neural Information Processing Systems (NIPS), 2012.
Jeffrey Dean, Greg Corrado, Rajat Monga, Kai Chen, Matthieu Devin, Mark Mao, Marc' aureloo Ranzato, Andrew Senior, Paul Tucker, Ke Yang, et al. Large scale distributed deep networks. In Advances in Neural Information Processing Systems (NIPS), 2012.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Gpt3. int8 O: 8-bit matrix multiplication for transformers scale. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lms.. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multi- ple classifier systems, pages 1-15. Springer, 2000.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
Bradley Efron. Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics: Methodology and distribution, pages 569-593. Springer, 1992.
Martin Ferianc, Partha Maji. Matthew Mattina, and Miguel Rodrigues. On the effects of quantisation on model uncertainty in bayesian neural networks. In Uncertainty in Artificial Intelligence (UAI), 2021.
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep ensembles: A loss landscape per- spective. arXiv preprint arXiv: 1912.02577, 2019.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for
generative pre-trained transformers. In International Conference on Learning Representations (ICLR), 2023.
Yarin Gal and Zoubin Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning (ICML), 2016.
11

---

Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103.136300 2021.
Alex Graves. Practical variational inference for neural networks. In Advances Neural Information Processing Systems (NIPS), 2011.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Conference Muchine Leurning (ICML), 2015.
Lars Kai Hansen and Peter Salamon. Neural network ensembles. IEEE transactions on pattern analysis and machine intelligence, 12(10):993-001, 1990.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021.
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hopcroft, and Kilian Q. Weinberger. Snapshot ensembles: Train get for free. In International Conference Learning Represen- tations (ICLR), 2017.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integerariihmneticonnl inference. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Putlern Recognition (CVPR), 2018.
Michael Kearns. Thoughts on hypothesis boosting. Unpublished manuscript, 1988.
Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems (NIPS), 2012.
Anders Krogh and Jesper Vedelsby. Neural network ensembles. cross validation, and active learning. In Advances Neural Information Processing Systems (NIPS), 1994.
Ananya Kumar, Ruoqi Shen, Sebastien Bubeck, and Suriya Gunasekar. How to fine-tune vision models with SGD. In International Conference on Learning Representations (ICLR), 2024.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems (NIPS), 2017.
Hyungi Lee, Giung Nam, Edwin Fong, and Juho Lee. Enhancing transfer learning with flexible non- parametric posterior sampling. In International Conference on Learning Representations (ICLR), 2024.
Jung Hyun Lee, Jeonghoon Kim, Se Jung Kwon, and Dongsoo Lee. Flexround: Learnable rounding based on element-wise division for post-training quantization. In International Conference on Machine Learning (ICML), 2023.
Stefan Lee, Senthil Purushwalkam, Michael Cogswell, David Crandall, and Dhruv Batra. Why m heads are better than one: Training a diverse ensemble of deep networks. arXiv preprint arXiv: 1.063 14, 2015.
Hao Li, Soham De, Zheng Xu, Christoph Studer, Hanan Samet, and Tom Goldstein. Training quantized nets: deeper understanding. Advances in Neural Information Processing Systems (NIPS), 2017.
12

---

Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-awarc weight quantization for IIm compression and acceleration. In MySys, 2024.
Kai Lion, Lorenzo Noci, Thomas Hofmann, and Gregor Bachmann. How good is a single basin? In International Conference on Artificial Intelligence and Statistics (AISTATS), 2024.
Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson.
A simple baseline for bayesian uncertainty in deep learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. In International Conference on Learning Representations (ICLR), 2018.
Basil Mustafa, Carlos Riquelme, Joan Puigcerver, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Deep ensembles for low-data transfer learning. arXiv preprint arXiv:2010.06866 2020.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort, Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning (ICML), 2020.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learn- ing? In Advances in Neural Information Processing Systems (NeurlPS), 2020.
Jeremy Nixon, Balaji Lakshminarayanan, and Dustin Tran. Why are bootstrapped deep ensembles not better? In "I Can't i Believe It's Not Better!" NeurlPS 2020 workshop. 2020.
Luis A Ortega, Rafael Cabañas, and Andres Masegosa. Diversity and generalization in neural net- work ensembles. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2022.
Kazuki Osawa, Siddharth Swaroop, Mohammad Emtiyaz E Khan, Anirudh Jain, Runa Eschenhagen, Richard E Turner, and Rio Yokota. Practical deep learning with bayesian principles. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated proba- bilities using bayesian binning. In AAAI Conference Artificial Intelligence (AAAI), 2015.
Theodore Papamarkou, Maria Skoularidou, Konstantina Palla, Laurence Aitchison, Julyan Arbel, David Dunson, Maurizio Filippone, Vincent Fortuin, Philipp Hennig, Aliaksandr Hubin, et al. Bayesian deep learning is needed in the age of large-scale ai. In International Confer- ence on Machine Learning (ICML), 2024.
Position:
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning (ICML), 2021.
Herbert Robbins and Sutton Monro. A stochastic approximation method. The Annals of Mathemat- ical Statistics, pages 400-400 407, 1951.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJVV), 115(3), 2015.
Ildus Sadrtdinov, Dmitrii Pozdeev, Dmitry P. Vetrov, and Ekaterina Lobacheva. To stay or not to stay in the pre-train basin: Insights ensembling in transfer learning. In Advances in Neural Information Processing Systems (NeurIPS). 2023.
Yuesong Shen, Nico Daheim, Bai Cong, Peter Nickl, Gian Maria Marconi, Clement Bazan, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan, et al. Variational learning is effective for large deep networks. In International Conference on Machine Learning (ICML.), 2024.
13

---

Xiao Sun, Naigang Wang, Chia-Ye Yu Chen, Jiamin Ni, Ankur Agrawal, Xiaodong Cui, Swagath Venkataramani, Kaoutar El Maghraoui, Vijayalakshmi Viji Srinivasan, and Kailash Gopalakr- ishnan. Ultra-low precision 4-bit training of deep neural networks. In Advances in Neural Infor- mation Processing Systems (NeurIPS), 2020.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, al. Ethical and social risks of harm from language models. arXiv preprini arXiv:2
2021.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In International Conference on Machine Learning (ICML). 2011.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In International Conference Learning Representations (ICLR), 2020.
Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. In Advances in Neural Information Processing Systems (NeurIPS). 2020.
Andrew Gordon Wilson. Deep ensembles approximate bayesian inference, Oct 2021. URL https://ccint http://ccmmm....... www.nuuuuuuuuuuu. anndeeugg/ddppessebbbbe
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. As- sociation for Computational Linguistics. URL https: / /www. aclweb. org/ anthology/2020 emnlp-demos- 6.
Danny Wood, Tingting Mu, Andrew M Webb, Henry WJ Reeve, Mikel Lujan, and Gavin Brown. A unified theory of diversity in ensemble learning. Journal of Machine Learning Research (JMLR), 24(359):1-49 2023.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. International Conference on Machine Learning (ICML), 2022.
Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt. Stable and low-precision training for large-scale vision-language models. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
Edouard Yvinec, Amaud Dapogny, Matthieu Cord, and Kevin Bailly. Powerquant: Automorphism search for non-uniform quantization. In International Conference on Learning Representations (ICLR), 2023.
Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical stochastic gradient meme for bayesian deep learning. In International Conference on Learning Representations (ICLR), 2020.
Ruqi Zhang, Andrew Gordon Wilson, and Christopher De Sa. Low-precision stochastic gradient langevin dynamics. In International Conference on Machine Learning (ICML), 2022.
14

---

A Models and datasets
The pre-trained weights utilized in experiments listed below. We refer readers the respec- tive papers fdetails on each model: ViT (Dosovitskiy et al., 2021), CLIP (Radford et al., 2021; Cherti al., 2023), and LLaMa (Touvron al., 2023). In our CLIP experiments, obtained zero-shot head weights by following the standard procedure outlined in the official code base'.
VIT-T/16: http://wwggg.g........////aa//aa
ViT-S/16: https//wgggiiicccccoovviiivviiia
ViT-B/16: http://aaggiiggeeoooooooog/ootap
ViT-L/16: http:///wnggiggirroooooooooooeeg
CLIP-ViT-L/4: http:///wwgggggggeccc/o///////ea
CLIP-ViT-G/14: http://itrgaaaaacccccoooooooooot
LLLaMa3-8B: http::/wwwiiiiio...ccccccooollaa
We employed two datasets for our experiments: ImageNet (Russakovsky al., 2015) for VIT and CLIP- VIT models, and MMLU (Hendrycks et al., 2021) for LLaMa. The evaluation MMLU was conducted using the template provided in the official repository", and the computation was based on a micro-average.
B Evaluation metrics
ground truth label for the 5th data point, where the total number of data points is N. K} denote the Let P; é [0, 1]K represent the predicted categorical probabilities and E (1....,K)
NLL. The negative likeliiod (NLL) of categorical distribution, also known cross-entropy loss, is a fundamental metric for evaluating the performance of a classification model:
N
NLL
log (v)
(6)
ERR. Another primary metric commonly used to assess the performance of a classification model the classification error (ERR), also referred to as the 0-1 loss:
1
N
(1))
ERR
+ arg max P
N
where [.] denotes the Iverson bracket.
ECE, The common choice for measuring calibration in machine learning is the expected calibration
error (ECE), particularly its empirical variant with binning (Pakdaman Nacini et al., 2015):
|Byl Jaco (Be(BB conf(
ECE
(8)
N
(k) where By denotes the 5th bin comprising |Byl data points whose prediction confidence max; P; falls within the interval ((j- 1))J_.jJ. Here, (B;) is the classification accuracy of Bj. and conf(B,) (B,) is the average confidence within By. As result, it computes weighted average of cali- bration gaps, which are the differences between accuracy and confidence, across bins. Throughout our experiments, we employed J 15 bins for ECE computation.
Ensemble ambiguity, Let æ., R be categorical logits predicted by the model for the 7th data point. The generalized ambiguity decomposition can be written
AMB ambiguity
- 3
Σ log (2.,))))
log o
2
N
s-1
(a) avarage loss
(c) ensemble loss
htttps//gihuuh.com/ppaaaa "hapss//github..o//eennnrtttett
15

---

Table 4: Motivating results for low precision ensembling of pre-trained ViT models. Negative log- ligllike (NLL), classification error (ERR), and ensemble ambiguity (AMB) ) for rounding-to to- nearest (RTN) and low precision ensembling with Bernoulli stochastic rounding (LPE-BSR) derived from the publicly available pre-trained ImageNet model (#)).
ViT-T/I6 (6M)
ViT-S/16 (22M)
Method
System
NLL
ERR
AMB
NLL
ERR
AMB
x Pre-trained
FP32
.932
.23
.667
.185
RTN
INT-6
.948.001
.247=001
..01
.185+.001
INT-4
1.23+001
.315=001
.822+0001
.218+.001
LPE-BSR
INT-6
.932.001
.245.001
.024..011
.6652.001
.1852.001
.014..011
INT-4
1.30=001
.298-001
.489
.821:001
2211:0001
.268.0001
ViT-B/16 (87M)
ViT-L/16 (307M)
Method
System
NLL
ERR
AMB
NLL
ERR
AMB
x Pre-trained
FP32
.687
.182
.639
.165
RTN
INT-6
0087=000
.182=000
.6392.00
.165+.000
INT-
.716=001
18840001
.647+0001
.167+.000
LPE-BSR
INT-6
.681
.181=001
.006..000
.632+.001
.644.000
.003..000
INT-4
.648.001
.175.000
.088.0001
.600+.00
.1 160+.001
.037
SGD
SWAG
IVON
MAP
MAP
MAP
0.490
0.490
0.490
LPE-BSR
LPE-85R
0.480
0,480
0.470
0.470
0.470
100
200
300
500
1000
1500
2000
500
1000
1500
2000
Memory budgets (x11F))
Memory budgett (x110f)
Memory budgets (x110)
Figure 8: Comparing low precision ensembling to Bayesian methods. Negative log-likelihode for Bayesian model averaging using an approximate Gaussian posterior derived from SWAG or IVON (BMA, shown in orange) and low precision ensembling with Bernoulli stochastic rounding centered around the MAP solution obtained by each optimizer (LPE-BSR, shown in green).
where o denotes softmax function that maps categorical logits into probabilities. is worth noting that logit ensembling (c) is essentially the same as computing normalized geometric mean for categorical probabilities. For further information, please see Wood et al. (2023).
C
Additional results
C.1 Motivating results with error bars
Table 4 is an extended version of Table 1, including standard deviations across four trials.
C.2
Comparative results with Bayesian methods in terms of memory budgets
Fig. is modified version of Fig. 2, where the x-axis has been changed from ensemble size to memory budget, defined as the total number of bits used to represent the ensembles. can be interpreted as trade-off plots between memory budgets and performance. Compared to SWAG and IVON, which represent ensemble members in the FP32 system, LPE-BSR occupies the preferred lower-left region, where ensemble members represented in an INT-5 system.
16

---

SSE
-8-
DE
MuLIVON MURIVON LPE4SR
SSE
-8- DE
NuLIVON NutiVON LPE4SR
SSE
LPE
- DE LPE-B5R
+
SSE LPE-BRR
- DE LIE-BSR
0.440
0.010
0.030
0.480
0.480
0.025
0.025
0.020
0.460
0.460
0.015
0.015
200
400
600
800
2000
A000
200
400
600
800
2000
4000
budgets (x:30")
Hemory budgets (x1of)
Training budgets
budgats
Figure 9: Combining with ensembling methods. Negative log-likelihood and expected calibra- tion error for ensembling methods, SSE, DE, and MultilVON, in terms of training budgets, i.c., the number backward passes, and memory budgets, i.e., the total number of bits for representing en- semble. Here, DE represents an ensemble of multiple Adam solutions, while MultilVON represents an ensemble of multiple IVON solutions.
Table 5: Results for low precision ensembling of fine-tuned models. We compute (a) average loss, (b) ambiguity, and (c) ensemble loss for diversity analysis, along with evaluation metrics to assess overall performance. The results are presented in ascending order of memory budgets, i.e., the total number of bits for representing ensemble. The number in parentheses after each method indicates the ensemble size.
Diversity analysis
Evaluation metrics
Method
System
(a)
(b)
(c)
NLL
ERR
ECE
Memory budgets (x10*)
LPE-BSR (4)
INT-5
.513
.025
.488
.481
.138
.021
62.9
LPE-BSR (6)
INT-5
.513
.028
.485
.477
.137
.020
94.4
& (1)
FP32
.487
.487
.487
.136
.035
97.3
BE (4)
FP32
.492
.006
.486
.480
.137
.032
97.4
LPE-BSR (8)
INT-5
.513
.029
.483
.475
.137
.019
126.
DE (4)
FP32
.488
.020
.668
.462
.132
.026
389.
C33 Further comparisons with non-Bayesian ensembles
While the proposed LPE-BSR method does not involve fine-tuning, though do include fine- tuning experiments to compare LPE-BSR's ensemble quality with Bayesian methods in Section 4.2, a comparative study with other non-Bayesian ensemble techniques like deep ensembles (DE; Lak- shminarayanan et al., 2017) and batch ensembles (BE; Wen et al., 2020) would be valuable.
Table 5 summarizes
experimental results using the Adam optimizer (Kingma and Ba, 2015). We observed that in LPE- BSR, (a) each ensemble member had relatively lower performance (= 0.513). However, (b) due to high ensemble diversity (2 0.025), (c) there
significant improvement in the final ensemble performance. Consequently, it achieves performance comparable to BE, another memory-efficient method available in fine-tuning scenarios. In BE, ensemble members similarly centered around one solution, with members derived from shared weights by multiplying rank-one matrices, while LPE-BSR members derived from center weights using stochastic rounding. This comparison with BE, well-known memory-efficiee ensembling strategy, highlights the potential of low precision ensembling with LPE-BSR.
C.4 Further comparisons with other training-free baselines
Our LPE-BSR method forms low precision ensemble from given checkpoint in training- free manner, as presented in Section 4. We further conduct additional comparative experiments with two baselines: 1) Gaussian, which builds ensemble by adding Gaussian noise with fixed variance to the pre-trained weights; and 2) Monte Carlo Dropout (MCD; Gal and Ghahramani, 2016), which constructs an ensemble by applying the dropout technique during inference. MCD is particularly relevant as it uses a q( (w) form similar to Eq. 5 of LPE- BSR, employing 6(0) and 5(w).
Table 6 summarizes the results for CLIP-ViT-L/14 with ensemble size of S 20. It clearly shows that while both the Gaussian and MCD baselines can perform ensembling training-free manner with appropriately tuned noise scales- specifsaeel the variance of Gaussian noise for the Gaussian
17

---

Table 6: Comparative results for training-free ensembles. The training-free ensemble methods, including Gaussian, MCD, and our proposed LPE-BSR, collect ensemble members centered around d (pre-trained CLIP-ViT-L/14 model in this context). Here, a² denotes the variance of Gaussian noise in the Gaussian baseline, and p refers to the drop probability in the MCD baseline.
Method
System
NLL
ERR
ECE
* Pre-trained
FP32
.948
.251
.049
Gaussian (o² = 0.0001)
FP32
.448
.251
.049
Gaussian (o² = 0.0002)
FP32
.948
.251
.048
Gaussian (o² 0.0004)
FP32
.946
.250
.046
Gaussian (o² = 0.0008)
FP32
941
.250
.043
Gaussian (o² = 0.0016)
FP32
.934
.250
.031
Gaussian (o² = 0.0032)
FP32
.981
.644
.038
MCD (p 0.001)
FP32
946
.251
.047
MCD (p 0.002)
FP32
946
.211
.047
MCD (p 0.004)
FP32
.94
.250
.046
MCD (p 0.008)
FP32
.440
.250
.044
MCD (P 0.016)
FP32
.38
.251
.041
MCD (P 0.032)
FP32
.944
.55
.034
LPE-BSR
INT-5
.929
.250
.028
baseline and the drop probability for the MCD baseline- our proposed LPE-BSR outperforms them. worth noting that LPE-BSR is more memory-effiiient, as each of its ensemble members uses INT-5, compared FP32 used by the baseline methods. Therefore, LPE-BSR only achieves better performance but also does so with reduced memory usage.
D Experimental details
We built our experimental code using JAX (Bradbury et al., 2018) and Transformers (Wolf al., 2020), both licensed under Apache-2.0." We conducted experiments using TPUv2/v3/v4 cores, with flexibility in selecting the cores based on the memory requirements of each experiment. The code available at http:///tuub.oom//siiinnu///////
E
Optimization and sampling methods
The optimization process for CLIP-ViT-L/14 models concludes after 100,000 iterations with a mini- batch size of 64, employing cosine decaying learning rate schedule. In the experiments described in the main text, we employed the following optimizers: SGD, IVON, and SWAG. All hyperparameter tuning was conducted using a development created by taking 1% of the training dataset, i.c., trrining [99%:] in TensorFlow Datasets (Abadi et al., 2015).
The zero-shot head weights in any of the experiments. We also froze the embedding layer to enable basic SGD update rules to function with the transformer architecture, described by Kumar al. (2024).
kept entirely fixed, meaning they were not fine-tuned quantized
SGD. Stochastic gradient descent (SGD) is foundational stochastic optimization algorithm in ma- chine learning (Robbins and Monro, 1951). Although it usually deemed ineffective for transformer architectures, recent findings by Kumar al. (2024) has shown that a simple pure SGD, with Adam (Kingma and Ba, 2015). The hyperparameter settings for the SGD optimizer utilized in experiments are summarized in Table 7.
-freeiinnnnn without momentum, produce results competitive
the embedding layer-enalee
IVON, Efforts develop variational methods for implementing Bayesian inference on neural net- work models have continued over time (Graves, 2011; Blundell et al., 2015). However, these at-
18

---

Table Hyperparameters in SGD and IVON.
Method
Hyperparameter
Search space
SGD
Base learning rate du 0.003
{0.03, 0.01, 0.003, 0.001}
l regularization å 10-3
IVON
Base learning rate du 0.003
{0.03, 0,0,1 0.001)
Effective sample size 1268355
l2 regularization 8 10-4
{10-2 10 1
Gradient momentum 31 = 0.9
Hessian momentum 32 0.99999
0.9999,. (.9.9999,099999999
Hessian initialization hgg
Clip radius 10
(10 2103, 10
tempts have frequently proven ineffective in practical scenarios, even for moderately-sized prob- lems (Osawa al., 2019). Recently, the Improved Variational Online Newton (IVON) algorithm, introduced by Shen et al. (2024), has facilitated variational learning for large models with update rule closely resembling that Adam (Kingma and Ba, 2015). In essence, by modifying the update rule of the second momentum in Adam, IVON estimates diagonal covariance of an approximate Gaussian posterior. The hyperparameters employed for the IVON optimizer in our experiments are presented in Table 7; the notations adhere to those described in Shen al. (2024).
SWAG. Besides variational learning, another notable approach for obtaining an approximate Gaus- sian posterior is Stochastic Weight Averaging Gaussian (SWAG; Maddox et al., 2019). Essentially, SWAG entails collecting samples throughout the SGD update process and calculating their sample mean and covariance to approximate the Gaussian posterior. For SWAG, we initially applied SGD updates with a cosine decay schedule until reaching a non-zero constant learning rate over 80,000 iterations. Subsequently, we switched to constant learning rate SGD updates, collecting samples every 1,000 iterations. The constant learning rate is determined by multiplying the base learning rate by decaying factor AswaG (1.0, 0.5, 0.2, 0.1). and the final hyperparameter configuration was ag 0.003 and ÁSWAG = 0.2. Ultimately, SWAG approximates a Gaussian posterior with non-diagonal covariance matrix with rank of 10 from these 20 samples.
SSE and CSGLD. When employing the SGD optimizer, the only difference between Snapshot En- sembling (SSE; Huang et al., 2017) and Cyclical Stochastic Gradient Langevin Dynamics (CSGLD; Zhang et al., 2020) lies in the incorporation of Gaussian noise in the update rule; in CSGLD, noise term derived from stochastic gradient Langevin dynamics (Welling and Teh, 2011) is added at each itcration. In our experiments on fast ensembling, we initialized both SSE and SGLD using the outcomes obtained from training with SGD for 100,000 iterations. Employing cosine-dccaying learning rate schedule with a cycle duration of 20,000 iterations, we iterated this schedule ten times to produce total of ten snapshots.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: This paper focuses on low precision ensembling, clearly outlined in both the abstract and introduction.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The conclusion section discusses limitations well future directions.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This work is more experimental than theoretical, and does not include theo- retical assumptions or proofs.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data provided not)?
Answer: [Yes]
Justification: The appendix contains experimental details, and the main text appropriately directs readers the appendix.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: As supplementary material, we provide the code for certain experiments. The full code will be furnished later
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The appendix contains experimental details, and the main text appropriately directs readers to the appendix.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [es]
Justification: The results with + indicate the average and standard deviation over four mea- surements. The standard deviation is also shaded in the figure.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
20

---

Answer:
[Yes]
Justification: The experimental details in the appendix include information regarding the hardware used.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http::////eeuisppeeeee https////p/p///ttccccc.oooeeeeee
Answer: [Yes]
Justification: This study adheres to code of ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The conclusion section discusses broader impacts.
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We acknowledge and credit previous works employed in our research.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
21


Reviews:
[{"review_id": "gKPNhhS7Qi", "reviewer": "Reviewer_9d5B", "summary": "The paper proposes that ensembles of quantized low-precision instances of large models outperform the source models on image classification and MMLU tasks. The low precision models are generated using Bernoulli stochastic rounding. The authors support their claims by presenting empirical results for several models, and show improvements over bayesian ensembling methods on negative log likelihood, ensemble diversity and error rate.", "questions": "1. Fig. 2 shows that SWAG does not improve with increasing ensembl sizes. This is a bit counter-intuitive. I may have misunderstood the figure, and invite some clarification if so.\n\n2. Are the LLAMA results presented on MMLU 5-shot or 0-shot?", "limitations": "The authors discuss some limitations in the paper. However, I suggest adding the actual compute requirements for the ensemble of low precision models.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of using ensembles of low precision derivatives of existing pretrained large models is inspired. Recent works on linear mode ensembling and model soups have shown that trained parameters of a neural network, when perturbed often exhibit good ensembling properties. Using a quantization technique to reduce compute requirements of an ensemble while improving performance is an interesting approach.\n\n2. The empirical results show improved performance for large model (ViT-B/16, Llama3) ensembles, and show good ensemble diversity.\n\n3. The paper is well-written and easy to follow.", "weaknesses": "1. The paper presents NLL comparisons with SWAG-Diag and IVON. It would be great to see accuracy numbers as well to understand the actual performance differences as all the results are presented on classification tasks. In addition, Table. 1 should have error estimates for LPE-BSR given that it relies on randomness. Also given that LPE-BSR is not really a Bayesian method, it would be great to also compare with other ensembling approaches like Model Soups to see where it lands on the pareto curve of compute-vs-accuracy. Also, the SWAG algorithm shows significant performance differences between SWAG, and SWAG-diag. While it may not be possible to compare for all the presented models, an exemplar comparison would also help support the conclusions presented in the paper.\n\n2. It would also be great to see some additional analysis on the actual compute savings incurred using LPE-BSR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are pleased with the positive feedback that highlights our work as both inspired and interesting. We hope the responses provided below address any remaining concerns. Please let us know if there are any further issues. Thank you for your valuable comments!\n\n---\n\n> The paper presents NLL comparisons with SWAG-Diag and IVON. It would be great to see accuracy numbers as well to understand the actual performance differences as all the results are presented on classification tasks.\n\nThe ERR values presented in our paper represent classification error, computed as one minus the classification accuracy (i.e., ERR = 1.0 - ACC). We opted to report classification error instead of classification accuracy to ensure that our evaluation metrics (i.e., NLL, ERR, ECE) align with the “lower is better.” However, if using ACC enhances readability, we are willing to make that change in the final camera-ready version.\n\n> In addition, Table. 1 should have error estimates for LPE-BSR given that it relies on randomness.\n\nWe empirically checked that the variability in LPE-BSR performance due to randomness is minimal (≤ 0.002). This is likely due to our experimental setup utilizing the same pre-trained model. We also agree that error bars should be included whenever possible, regardless of how small they are, and the camera-ready version will include results with error bars in the appendix. Thanks for the constructive feedback!\n\n> Also given that LPE-BSR is not really a Bayesian method, it would be great to also compare with other ensembling approaches like Model Soups to see where it lands on the pareto curve of compute-vs-accuracy.\n\nModel Soups (Wortsman et al., 2022) involves averaging the weights of models fine-tuned with different hyperparameters from the same pre-trained model. Thus, it is distinct from our LPE-BSR method, which does not involve any fine-tuning. Model Soups boosts the performance of the final fine-tuned checkpoint by averaging multiple fine-tuned checkpoints, whereas LPE-BSR works directly on a single provided checkpoint. Consequently, our LPE-BSR method can also be applied to a checkpoint obtained through Soup. Given that LPE-BSR generates low precision ensemble members around the provided checkpoint (such as MAP or Soup here), it is anticipated that LPE-BSR will achieve better performance with a more advanced checkpoint, like one from Soup with an NLL of .477. Notably, LPE-BSR consistently improves performance whether the given checkpoint comes from MAP or Soup, showcasing the flexibility of the proposed low precision ensembling method.\n\n| Method | NLL / ERR / ECE | $\\Delta$ |\n| :- | :- | :- |\n| MAP (Adam) | .487 / .136 / .035 | - |\n| w/ LPE-BSR | __.469__ / __.135__ / __.019__ | __.018__ / __.001__ / __.016__ |\n| Soup (Adam) | .477 / .133 / .023 | - |\n| w/ LPE-BSR | __.465__ / __.132__ / __.017__ | __.012__ / __.001__ / __.006__ |\n\nIn addition, while weight averaging is straightforward with standard FP32 weights, it becomes more complex with low precision number systems because the averaged points do not align with the low precision system grid. More importantly, regardless of the low precision grid, averaging the weights of LPE-BSR ensemble members results in the original central FP32 weights, similar to how averaging samples in SWAG and IVON results in the original Gaussian mean. We will clarify this point further in the final version of our paper.\n\n> Also, the SWAG algorithm shows significant performance differences between SWAG, and SWAG-diag. While it may not be possible to compare for all the presented models, an exemplar comparison would also help support the conclusions presented in the paper.\n\n> Fig. 2 shows that SWAG does not improve with increasing ensembl sizes. This is a bit counter-intuitive. I may have misunderstood the figure, and invite some clarification if so.\n\nWe are happy to share the improved SWAG baseline results in the General Response. We hope it resolves your concerns and appreciate your constructive feedback.\n\n> Are the LLAMA results presented on MMLU 5-shot or 0-shot?\n\nThe MMLU results pertain to the 0-shot case, and we realized this detail is only mentioned in the caption of Figure 7. We will update Appendix A to include this information. Thanks for bringing it to our attention!\n\n> It would also be great to see some additional analysis on the actual compute savings incurred using LPE-BSR.\n\nAs advised, we have expanded the memory budgets plot from Section 4.4 to cover the experiments in Sections 4.2 and 4.5. Please refer to the figures in the attached PDF, which clearly illustrate the compute savings achieved with LPE-BSR, for further details. Thank you for the suggestion!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BjsWlxqk1H", "reviewer": "Reviewer_yx79", "summary": "This paper addresses the scalability challenge in ensembling deep neural networks for large models by introducing a novel low precision ensembling method. The approach generates an ensemble of models from a single model using low precision number systems in a training-free manner. Empirical analysis shows that this method achieves comparable or superior generalization performance. The findings suggest that low precision ensembling is a promising solution to enhance generalization performance while addressing the scalability issues inherent in traditional ensembling methods.", "questions": "- How does the diversity of low precision ensembles compare with other popular ensembling techniques like deep ensemble and batch ensemble? \n- It would also be interesting to benchmark the performance of the proposed method in terms of computation cost (in addition to the memory budgets shown in Figure 5).\n\n[1] \"BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning\"\n\n[2] \"Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles\"", "limitations": "Yes, the paper addresses the limitations of the work.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper tackles an important problem on how to efficiently obtain an ensemble of models.\n- The paper is well written and easy to follow. \n- The proposed method is technically sound. \n- The experimental results suggest that the proposed method is effective.", "weaknesses": "- Firstly, I think the paper is lacking some novelty. While interesting, it is not very surprising to me that an ensemble of randomly sampled low precision models can improve the generalization performance and uncertainty calibration. There have been numerous works of similar flavor. \n- I think the paper lacks some important benchmark comparisons. For instance, how does the proposed method compare to other sampling based ensembling approaches like dropout ensemble [1] and batchnorm ensemble [2]? \n\n[1] \"Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning\"\n\n[2] \"Bayesian Uncertainty Estimation for Batch Normalized Deep Networks\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the significance of the problem we addressed, the clarity of our writing, the technical robustness of our proposed method, and the effectiveness of our experimental results. We have addressed your comments below and believe that incorporating these revisions into the final version will significantly enhance the contribution of our work. If you have any further concerns, please let us know. Otherwise, we kindly ask that you adjust your assessment accordingly. We appreciate your valuable feedback once again!\n\n---\n\n> Firstly, I think the paper is lacking some novelty. While interesting, it is not very surprising to me that an ensemble of randomly sampled low precision models can improve the generalization performance and uncertainty calibration. There have been numerous works of similar flavor.\n\nWe respectfully disagree with the novelty claim and would like to quote Michael Black’s remark: \"If it is easy to explain and obvious in hindsight, this in no way diminishes the creativity (and novelty) of the idea.\" To our knowledge, the concept of employing quantization erros to obtain ensemble diversity has not been widely explored. Could you please provide specific examples of the \"numerous works of similar flavor\" you mentioned?\n\n> I think the paper lacks some important benchmark comparisons. For instance, how does the proposed method compare to other sampling based ensembling approaches like dropout ensemble [1] and batchnorm ensemble [2]?\n\nThank you for your insightful comment! Both MCDropout (Gal and Ghahramani, 2016) and MCBatchNorm (Teye et al., 2018) leverage dropout and batch normalization techniques--typically used during training--at the inference stage to build ensembles. These methods indeed enable training-free ensemble construction, making them valuable for comparison with our proposed method. However, since modern transformer architectures typically do not include batch normalization layers, we concentrated our additional experiments on MCDropout. MCDropout is particularly relevant as it uses a $q(w)$ form similar to Eq. (5) of LPE-BSR, employing $\\delta(0)$ and $\\delta(w)$. The General Response provides additional comparative results with the MCDropout baseline and the Gaussian baseline mentioned by another reviewer. The results confirm that LPE-BSR outperforms both of these baselines. Thanks again for your constructive suggestion!\n\n> How does the diversity of low precision ensembles compare with other popular ensembling techniques like deep ensemble and batch ensemble?\n\nWhile our LPE-BSR method does not involve fine-tuning (though we do include fine-tuning experiments to compare LPE-BSR’s ensemble quality with Bayesian methods in our paper), a comparative study with other ensemble techniques like deep ensembles (DE) and batch ensembles (BE), as you suggested, would be valuable. Below are the results using the Adam optimizer; the number of ensemble members for each method is indicated in parentheses; the metrics (a), (b), and (c) correspond to those discussed in the diversity analysis of our paper; (a) average loss, (b) ambiguity (i.e., ensemble diversity), and (c) ensemble loss.\n\n| Method      | (a) / (b) / (c)    | NLL / ERR / ECE    | Memory Budget (x 1e8)\n| :- | :- | :- | :- |\n| DE (4)      | .488 / .020 / .468 | __.462__ / __.132__ / .026 | 389.\n| BE (4)      | .492 / .006 / .486 | .480 / __.137__ / .032 | 97.4\n| LPE-BSR (4) | .513 / .025 / .488 | .481 / .138 / .021 | __62.9__\n| LPE-BSR (6) | .513 / .028 / .485 | .477 / __.137__ / __.020__ | __94.4__\n| LPE-BSR (8) | .513 / .029 / .483 | __.475__ / __.137__ / __.019__ | 126.\n\nIn our experiments using the Adam optimizer, we found that in LPE-BSR, (a) each ensemble member had relatively lower performance (NLL = 0.513). However, (b) due to high ensemble diversity (ambiguity ≥ 0.025), (c) there was a significant improvement in the final ensemble performance. Consequently, it achieves performance comparable to another memory-efficient method available in fine-tuning scenarios, BE. In BE, ensemble members are similarly centered around one solution, with BE members derived from shared weights by multiplying rank-one matrices, and LPE-BSR members derived from the center weights using stochastic rounding. This comparison with BE, a well-known memory-efficient ensembling strategy, highlights the potential of low precision ensembling with LPE-BSR.\n\nFurthermore, it is worth noting that LPE-BSR forms low precision ensembles without requiring additional training on any given checkpoint. This suggests that, similar to the enhancements shown in fast ensembling experiments, deep ensembles can also achieve further improvements with LPE-BSR. Figure 12 in the attached PDF illustrates this point clearly.\n\n> It would also be interesting to benchmark the performance of the proposed method in terms of computation cost (in addition to the memory budgets shown in Figure 5).\n\nWe understand that you are addressing computation cost in terms of time complexity, such as latency, rather than space complexity like memory budgets. Unfortunately, quantized models typically have slower inference speeds without specialized hardware and software. As we do not have access to such hardware, we employed fake quantization for research purposes, which makes direct benchmarking challenging. However, since our work is based on the standard symmetric uniform quantization scheme, it aligns with current and future advancements in neural network quantization. We also want to highlight that our contribution, which showcases the potential of low-precision ensembling in large modern models, remains significant and relevant. We will include this point in the limitations section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V2KGnAcciH", "reviewer": "Reviewer_bKfp", "summary": "This paper presents a new way to generate an ensemble of models without the need for training multiple times and with the extra advantage of using low-precision representation which inherently saves memory.\nThe idea is to build an ensemble of models starting from stochastic variations of a single model. Those variations are based on a stochastic rounding scheme that samples either the ciel or the floor of a real number proportionally to the distance to the two rounded values (called Bernoulli stochastic rounding).\nResults show that, especially for large models (see Tab.1), this technique provides accuracies that are better than a single model with the advantage of a quantized representation that has a lower memory footprint.", "questions": "See the questions above associated to weaknesses.\n\nAdditional questions:\n\n\\- In tab.2 Why not showing all configs for SGD?\n\n\\- Fig.3, what is the difference among the three subfigures?", "limitations": "Some of the possible limitations of the approach are not analysed in the paper.\nFor instance, the proposed approach is not contrasted with other common ensembling techniques. Instead it is added to them. In this way, it is difficult to know the real efficacy of the approach.\nAlso, I did not see the performance of the approach for weight averaging. If the approach does not work well for weight averaging, it should be mentioned.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed idea is interesting: how to use the noise generated by quantisation as a way to generate diversity for ensembling.\n\n- The presentation of the paper is normally good although the experiments are a bit misleading (see below)", "weaknesses": "\\- The proposed idea is interesting and I think it has some potential. However, I think that the authors did not analyse and compare in a fair way the drawbacks of the approach. Below some points.\n\n\\- Compare with Deep ensembling. The method is interesting because it does not require to retrain the model. However, authors should compare with a real ensembling of multiple models. \n\n\\- Compare with others without adding its contribution. If the pre-training of large models is too expensive to be evaluated multiple times, the authors should compare with Bayesian and fast ensembling methods. Thus, instead of just adding the approach on top of SWAG, IVON and SSE, it would be interesting to see comparative results of each technique independently. For instance in Tab.2 we can see that in terms of error, the proposed approach (LPE-BSR) trained with normal SGD (error=.137) is inferior to IVON (error=.135) and comparable to SWAG (error=.137). \n\n\\- Weight averaging. Another important point when working with pre-trained models is weight averaging. As the different ensembles come from the same pre-training, normally they are aligned and the ensemble can be approximated with weight averaging. This allows to gain in accuracy with a single inference. What is the performance of the proposed approach with weight averaging?\n\n\\- Another possible baseline to compare with is to build an ensemble by adding Gaussian noise with a fixed variance for all values tuned (for ensembling) on a validation set. Although very simple, that approach can be quite effective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the potential and interest in our idea. We appreciate your detailed and constructive feedback. We have addressed your comments below and are confident that incorporating these revisions into the final version will significantly strengthen our work. If you have any additional concerns, please let us know. Otherwise, we kindly ask that you revise your assessment accordingly. Thank you again for your valuable feedback!\n\n---\n\n> Another possible baseline to compare with is to build an ensemble by adding Gaussian noise with a fixed variance for all values tuned (for ensembling) on a validation set. Although very simple, that approach can be quite effective.\n\nWe recognized that the mentioned baseline offers a simple approach for training-free ensemble construction. As a result, we included additional comparative experiments with the Gaussian baseline in the General Response, along with the MCDropout baseline mentioned by another reviewer. The results validate that LPE-BSR outperforms these baselines. Thank you for your constructive suggestion!\n\n> Compare with others without adding its contribution. (...)\n\nOur LPE-BSR method forms a low precision ensemble from a given checkpoint in a training-free manner, making it dependent on existing checkpoint. In our paper, we conducted a comparative study using fine-tuned checkpoints from SWAG, IVON, and SSE. Therefore, we respectfully disagree with the claim that SGD plus LPE-BSR should be compared to the MAP of IVON and SWAG; both IVON and SWAG are simply optimizers with additional features that provide an approximate Gaussian posterior. Specifically, SWAG is an enhanced version of SGD with Polyak-Ruppert averaging (cf. Izmailov et al., 2018), and IVON is a variation of Adam interpreted through variational learning (cf. Shen et al., 2024). As a result, it is entirely possible for the most basic optimizer, SGD without momentum (NLL = .492), to perform worse than the more advanced optimizers, SWAG and IVON (NLL = .488 and .489), and there is no reason why LPE-BSR should be limited to use with SGD. Importantly, LPE-BSR consistently improves performance over the given checkpoint (MAP), with improvements even surpassing those seen with BMA in IVON and SWAG ($\\Delta$):.\n\n| Optimizer | Method  | NLL / ERR / ECE    | $\\Delta$           |\n| :-        | :-      | :-                 | :-                 |\n| SGD       | MAP     | .492 / .138 / .035 | -                  |\n|           | LPE-BSR | .477 / .137 / .020 | .015 / .001 / .015 |\n| Adam      | MAP     | .487 / .136 / .035 | -                  |\n|           | LPE-BSR | __.469__ / __.135__ / __.019__ | .018 / .001 / .016 |\n| SWAG      | MAP     | .488 / .137 / .034 | -                  |\n|           | BMA     | .479 / .136 / .027 | .009 / .001 / .007 |\n|           | LPE-BSR | .473 / .136 / .021 | .015 / .001 / .013 |\n| IVON      | MAP     | .489 / .136 / .037 | -                  |\n|           | BMA     | .475 / __.135__ / .026 | .014 / .001 / .011 |\n|           | LPE-BSR | .472 / __.135__ / .023 | .017 / .001 / .014 |\n\nWe have further included results using Adam as well as SGD in the table for readers who may not be familiar with the SWAG and IVON optimizers. Given that LPE-BSR generates low precision ensemble members around the provided checkpoint (MAP), it is anticipated that LPE-BSR will perform better with a more advanced optimizer, such as Adam with an NLL of 0.487. Again, the main point is that LPE-BSR reliably produces a low precision ensemble that exceeds the performance of the original FP32 solution without additional training, achieving quality on par with leading Bayesian methods like SWAG and IVON.\n\n> Compare with Deep ensembling. (...)\n\nFollowing the suggestions from Reviewers yx79 and bKfp, we compared LPE-BSR with deep ensembles (DE) and batch ensembles (BE). While LPE-BSR does not involve fine-tuning, these comparisons are valuable, similar to those with the BMA approach. Due to character limits, please see our response to Reviewer yx79. These comparisons show that LPE-BSR is a memory-efficient method achieving performance levels bounded by DE, similar to BE, but without requiring fine-tuning. Thank you for the constructive feedback!\n\n> Weight averaging. (...)\n\nIn short, averaging the weights of LPE-BSR ensemble members results in the original central FP32 weights, similar to how averaging samples in SWAG and IVON results in the original Gaussian mean. Due to space constraints, please refer to our response regarding Model Soups to Reviewer 9d5B for more details.\n\n> Figure 3, what is the difference among the three subfigures?\n\nDetails about radial basis plots are provided in lines 210-216. In short, the first subplot illustrates where each model is positioned within the loss landscape, while the next two subplots show how much the models are diverse from one another. Reviewer svQu also raised similar concerns regarding the clarity of the radial basis plots. Consequently, we intend to enhance the captions and labels in the camera-ready version. We appreciate your feedback on this matter.\n\n> Table 2, why not showing all configs for SGD?\n\nThank you for bringing that to our attention. We initially removed the content to save vertical space and did not reinclude it. The missing results for SGD presented below, which show trends consistent with other results, will be added to the camera-ready version. We appreciate your thorough feedback!\n\n| System | (a)  | (b)  | (c)  | NLL  | ERR  | ECE  |\n| :-     | :-   | :-   | :-   | :-   | :-   | :-   |\n| INT-6  | .495 | .063 | .488 | .485 | .138 | .030 |\n| INT-5  | .513 | .057 | .456 | __.477__ | __.137__ | __.020__ |\n| INT-4  | .663 | .120 | .544 | .526 | .150 | .029 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2z5PtNYPhj", "reviewer": "Reviewer_svQu", "summary": "This paper addresses the problem of ensembling in low precision number systems, where quantized models act as members of the ensemble. The authors suggest that quantization errors can be leveraged to enhance ensemble diversity. Based on this concept, they propose a method called LPE-BSR. Through extensive experiments and analyses, the authors validate the effectiveness of their approach.", "questions": "1. In line 174, what does $w_\\text{MAP}$ mean? Is it a typo for $w^*_\\text{MAP}$?\n2. In line 179, the paper concludes that \"it encountered difficulties in accurately estimating the diagonal covariance matrix.\" However, this conclusion is not clearly supported by the results presented in Figure 2. Please provide more evidence or clarification for this statement.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and clearly written.\n2. The authors perform extensive experiments and analyses regarding ensembling in low precision number systems.", "weaknesses": "1. The paper relies on existing techniques for both ensembling and quantization. It primarily presents an empirical study on the combination of these two approaches rather than introducing a new method.\n2. The practical utility of the proposed LPE-BSR method is somewhat constrained. Large models typically incur high inference costs, and ensembling them—even with low precision systems—can amplify these costs, particularly with ensemble sizes of 10 or 20, as discussed in the paper. Furthermore, according to the experimental results in Table 3, the performance improvement of the quantized ensemble over the full precision model is marginal. As an empirical study, the authors should better justify why ensembling in low precision number systems is a meaningful objective.\n3. The related work section does not provide enough context or comparison with existing research. It would be beneficial to include a more detailed literature review.\n4. The subfigures in Figures 3, 4, and 6 are difficult to interpret. Adding captions or labels for each subfigure would improve clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review. We hope our response below addresses any remaining concerns. If you have any further questions, please let us know. Otherwise, we kindly request that you reconsider your assessment accordingly. Thank you again for your valuable feedback!\n\n---\n\n> In line 174, what does $w_{\\text{MAP}}$ mean? Is it a typo for $w_{\\text{MAP}}^\\ast$?\n\nApologies for the confusion caused by the typos. We have identified that asterisks were missing in lines 174 and 232. Thank you for your meticulous review!\n\n> In line 179, the paper concludes that \"it encountered difficulties in accurately estimating the diagonal covariance matrix.\" However, this conclusion is not clearly supported by the results presented in Figure 2. Please provide more evidence or clarification for this statement.\n\nWe initially concluded this because the SWAG-Diag baseline seemed ineffective at performing BMA. Without knowing the true posterior, we assessed the quality of the approximate Gaussian posterior through MAP and BMA performance. Good MAP performance suggested the mean was acceptable, but poor BMA indicated a problem with the variance estimate. However, after conducting additional experiments, we improved the SWAG baseline results, as detailed in the General Response. This resolves the issue with the SWAG baseline, which will be updated in the camera-ready version. Thank you for raising concerns about the SWAG results.\n\n> The paper relies on existing techniques for both ensembling and quantization. It primarily presents an empirical study on the combination of these two approaches rather than introducing a new method.\n\nOur main contribution is offering a fresh perspective by using quantization errors--often seen as challenges in quantization research--as a source of diversity in ensemble methods. Consequently, although each individual approach (quantization and ensemble) may not be novel by itself, we believe showcasing the potential of low-precision ensembles for modern large-scale models represents a novel and significant contribution.\n\n> The practical utility of the proposed LPE-BSR method is somewhat constrained. Large models typically incur high inference costs, and ensembling them—even with low precision systems—can amplify these costs, particularly with ensemble sizes of 10 or 20, as discussed in the paper. Furthermore, according to the experimental results in Table 3, the performance improvement of the quantized ensemble over the full precision model is marginal. As an empirical study, the authors should better justify why ensembling in low precision number systems is a meaningful objective.\n\nAs you noted, the practicality of ensembling large models is constrained, which we believe explains the decreased preference for ensemble methods in the current era of large models. Our research addresses this by demonstrating that scalability issues with ensemble techniques can potentially be mitigated through the use of low precision number systems, particularly for modern large models. In doing so, we aim to rekindle interest in ensemble methods within the context of current large model trends, as mentioned in lines 289-293 of the conclusion section. We hope that our work on LPE-BSR will be seen as a pioneering effort in showcasing the potential of low precision ensembles.\n\nBesides, following Reviewer 9d5B's recommendation, we have extended the memory budgets plot from Section 4.4 to include the experiments from Sections 4.2 and 4.5. The figures in the attached PDF demonstrate that our LPE-BSR approach is located in the lower-left corner, signifying that it achieves lower NLL while being more memory-efficient. Even with a smaller ensemble size of five, it outperforms both the BMA baselines and the original checkpoint.\n\n> The related work section does not provide enough context or comparison with existing research. It would be beneficial to include a more detailed literature review.\n\nThank you for your constructive comment, which strengthens our paper! We have reviewed relevant studies on low precision ensembling, especially those involving Bayesina methods with quantization-aware training, but have struggled to find further research on this topic. Any recommendations for further literature would be immensely helpful. Meanwhile, we will revise the paper to address your feedback by expanding the related work section to include additional paragraphs on ensemble methods and quantization.\n\n> The subfigures in Figures 3, 4, and 6 are difficult to interpret. Adding captions or labels for each subfigure would improve clarity.\n\nEven though lines 210-216 of the main text explain the radial landscape plots, it appears that Reviewer bKfp also expressed similar concerns about Figure 3. Including captions and labels for each figure, as you recommended, should improve readability in the camera-ready version. We appreciate your constructive feedback!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Global Response

First and foremost, we would like to thank all the reviewers for their time and effort in reviewing our paper. We are pleased to note that all the reviewers agreed our paper is of high quality. In particular, they noted that it tackles an important problem (yx79), is well-written and easy to follow (9d5B, yx79), has a generally good presentation (bKfp), is well-organized and clearly written (svQu), and includes extensive experiments and analyses (svQu). We are also delighted that our idea and methodology received favorable feedback, being described as inspired (9d5B), interesting (9d5B, bKfp), technically sound and effective (yx79).

While we will provide individual responses to each reviewer’s comments, this global response aims to address the major concerns raised by all reviewers. Due to the NeurIPS policy, we are unable to revise the paper or supplementary materials during the author response period. However, we assure that all points addressed during this period will be included in the camera-ready version.

---

## Further comparisons with other training-free baselines (bKfp, yx79)

Reviewers bKfp and yx79 emphasized the need for additional baselines. We acknowledge that incorporating Gaussian noise and Monte Carlo Dropout, as mentioned by the reviewers, can also be used for training-free ensemble construction. Accordingly, we have conducted an additional comparative study with CLIP-ViT-L/14, building upon the results presented in Table 3.

| Method      | NLL / ERR / ECE    | $\Delta$           |
| :-          | :-                 | :-                 |
| Pre-trained | .948 / .251 / .049 | -                  |
| MCDropout   | .938 / .251 / .041 | .010 / .000 / .008 |
| Gaussian    | .934 / __.250__ / .031 | .014 / __.001__ / .018 |
| LPE-BSR     | __.929__ / __.250__ / __.028__ | __.019__ / __.001__ / __.021__ |

The experimental results show that while both Gaussian and MCDropout baselines can also perform ensembling in a training-free manner, LPE-BSR achieves better performance. It is worth noting that LPE-BSR is more memory-efficient since each of its ensemble members uses INT-5, compared to FP32 used by the baseline methods. Therefore, LPE-BSR not only achieves better performance but also does so with reduced memory usage. We believe that updating the camera-ready version with these comparative results will enhance the discussion on training-free ensemble construction.

## Improving SWAG results (9d5B, svQu)

Reviewers 9d5B and svQu expressed concerns about the SWAG-Diag baseline results. After conducting additional experiments, we achieved improved baseline results for SWAG. Firstly, we found that introducing a scale hyperparameter to the variance matrix enhanced the SWAG-Diag results (cf. Appendix D.3 of the SWAG paper). The initially observed small variance matrix was likely due to our use of a small learning rate with the SGD optimizer without momentum. Secondly, we obtained SWAG results using a non-diagonal covariance matrix with a rank of ten. As Reviewer 9d5B pointed out, SWAG indeed outperforms SWAG-Diag.

| Method               | NLL / ERR / ECE    | $\Delta$           |
| :-                   | :-                 | :-                 |
| MAP (SWA)            | .488 / .137 / .034 | -                  |
| SWAG-Diag (previous) | .487 / .137 / .034 | .001 / .000 / .000 |
| SWAG-Diag (improved) | .479 / __.136__ / .027 | .009 / __.001__ / .007 |
| SWAG (rank=10)       | .477 / __.136__ / __.021__ | .011 / __.001__ / __.013__ |
| LPE-BSR              | __.473__ / __.136__ / __.021__ | __.015__ / __.001__ / __.013__ |

However, LPE-BSR achieves performance on par with SWAG while being more memory-efficient, emphasizing the potential of low precision ensembling. We believe these improved baseline results further underscore our contribution, showing that LPE-BSR generates low precision ensemble members that are comparable to the posterior samples from leading Bayesian methods like SWAG and IVON.

## Updated figures

We are providing the updated figures as a PDF in line with NeurIPS policy. Specifically:
- Figure 9 is a revised version of the original Figure 2, featuring an improved SWAG baseline and including the previously missing error bars for BMA.
- Figure 10 shows the memory budgets for ensembling compared to the MAP checkpoint and BMA baselines.
- Figure 11 also presents the memory budgets for ensembling in relation to the pre-trained checkpoint. We have corrected the pre-trained results for CLIP-ViT-G/14 from .948 to .942; Figure 7 will be updated accordingly. LPE-BSR effectively lowers the memory budgets compared to baselines.
- Figure 12 builds on Figure 5 by introducing more advanced ensembling methods rather than fast ensembling techniques like SSE. DE represents an ensemble of multiple Adam solutions, while MultiIVON represents an ensemble of multiple IVON solutions. LPE-BSR consistently improves upon these ensemble methods.

Meta-review: As stated by the Authors, the main contribution of the paper is offering a fresh perspective by using quantization errors--often seen as challenges in quantization research--as a source of diversity in ensemble methods. 

This approach has a particular significance in the era of LLMs. Ensembling is known to improve robustness and the proposed approach makes ensembling more accessible for LLMs due to the reduce memory requirement.

The paper attracted diverging reviews, with two reviewers voting for acceptance (6,5) and two voting for rejection (4,4). It is a borderline paper and the decision will naturally be based more on the highlighted weaknesses and strengths, and less on the absolute score.

Two reviewers emphasized the lack of novelty, which I respectfully disagree with. If the previous work does not disclose the proposed method, the fact it is simple is actually more of a strength of the work rather than a limitation. I would also argue that the results suggest that quantization of the model has interesting and nontrivial effects on its robusstness. It would be interesting to study how these different ensemble members differ, e.g. in terms of which examples they memorize from the original dataset. 

A very fair criticism was raised that the paper lacked comparisons to methods such as SWAG, Deep Ensemble or Batch Ensemble. This I think was convincingly addressed in the rebuttal phase.

All in all, I am recommending acceptance. I believe the method might have an impact, both for applied deep learning and development of new methods.
Decision: Accept (poster)