Ex Uno Pluria: Insights on Ensembling in Low Precision Number Systems

Authors: giung  nam
juho  lee
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 19:23:05

Abstract: While ensembling deep neural networks has shown promise in improving generalization performance, scaling current ensemble methods for large models remains challenging. Given that recent progress in deep learning is largely driven by the scale, exemplified by the widespread adoption of large-scale neural network architectures, scalability emerges an increasingly critical issue for machine learning algorithms in the era of large-scale models. In this work, we first showcase the potential of low precision ensembling, where ensemble members are derived from a single model within low precision number systems in a training-free manner. Our empirical analysis demonstrates the effectiveness of our proposed low precision ensembling method compared to existing ensemble approaches.

Main paper: 
 introduction: 
 In computer science, it is a de facto standard to represent continuous real numbers using finite precision number systems. While many applications rely on precision formats like FP32 or FP64, the deep learning community is increasingly turning to 16-bit floating-point formats such as FP16 (Micikevicius et al., 2018) or BF16 (Dean et al., 2012) to reduce memory usage during training. More recently, researchers are further exploring low precision optimization, aiming to utilize fewer bits (8 bits or less) to represent weights, activations, and gradients throughout the training process (Gupta et al., 2015; Li et al., 2017; Sun et al., 2020; Wortsman et al., 2023) .While low precision number systems can aid in training deep neural networks, they are also beneficial for reducing inference costs in real-world deployments of such models (Jacob et al., 2018) . In particular, recent advancements in large language models (Brown et al., 2020; Touvron et al., 2023) containing billions of parameters have triggered active exploration of post-training quantization techniques, for deploying pre-trained large language models on hardware with limited memory resources. This exploration encompasses quantizing both weights and activations (Dettmers et al., 2022; Yao et al., 2022) , as well as quantizing weights only (Frantar et al., 2023; Lin et al., 2024) .Originally, the goal of post-training quantization is to specify the best solution represented in low precision number systems, aiming to reduce discrepancies from the original high-precision weights, like perturbations in weight values or increases in loss functions (Nagel et al., 2020) . On the other hand, the presence of numerous distinct yet high-performing models within a single basin on loss landscapes (Sadrtdinov et al., 2023; Lion et al., 2024) evokes a Bayesian concept of marginalization instead of optimization, which involves utilizing multiple solutions rather than relying solely on one solution (Wilson and Izmailov, 2020) .Hinging on this insight, we suggest building ensembles within low precision number systems, as illustrated in Fig. 1 . It depicts a proof-of-concept method for ensemble construction using stochastic rounding, a technique commonly used in low precision training to address the issue of rounding Figure 1 : Concepts of low precision ensembling. It shows a two-dimensional schematic, where the x and y axes represent the neural network weights, while the contours above visualize the loss surface. (a) Let the pre-trained weights, denoted by a yellow star-shaped marker (☆), be positioned within a basin on the loss landscape. In general, (b) post-training quantization methods introduce lower precision number systems, and then (c) choose one candidate from the system, such as the nearest one. (d) However, there are many other highly effective models available, that can contribute to ensemble predictions.weight updates below the minimum precision threshold to zero (Gupta et al., 2015) . In our approach, stochastic rounding is employed for ensembling rather than optimization. Indeed, our experimental findings in Section 4 validate that low precision ensembling improves the downstream performance of pre-trained large models without any further training on downstream data.Confirming the potential of low precision ensembling for pre-trained models, we extend our investigation through a comparative study with existing methods involving ensembling. Specifically, we examine Bayesian approaches that approximate a Gaussian posterior over the loss landscape (Maddox et al., 2019; Shen et al., 2024) , and sampling techniques that collect model copies from the update trajectory (Huang et al., 2017; Zhang et al., 2020) . Our experimental results in Section 4 show that low precision ensembling successfully gathers diverse ensemble members contributing the final ensemble performance within the low precision number system.The main contributions of our work can be outlined as follows:• Introducing low precision number systems inevitably results in quantization errors, usually seen as a flaw to be corrected in neural network quantization. Our work presents a novel viewpoint: these errors can be utilized as a source to improve ensemble diversity. Expanding on our comprehension of diversity, we suggest a simple yet powerful approach to constructing ensembles called Low Precision Ensembling with Bernoulli Stochastic Rounding (LPE-BSR), particularly advantageous for large models. • The proposed LPE-BSR involves assembling ensemble members with low precision number systems, effectively addressing a notable challenge associated with memory costs inherent in ensemble methods. In this regard, our work holds promise for utilizing low precision number systems to construct ensembles of large models, offering a potential solution for the scalability issue faced by the Bayesian deep learning community in the era of large-scale models (Papamarkou et al., 2024) . 
ensemble methods in modern transfer learning scenarios: 
 Ensembling neural networks is a long-established idea in machine learning (Hansen and Salamon, 1990) , where the underlying design principle is to create a strong hypothesis that effectively explains the data by combining a set of weak hypotheses (Kearns, 1988) . Notably, even after the transition into the deep learning era, ensemble methods continue to serve as a straightforward yet powerful strategy for boosting the performance of machine learning algorithms involving deep neural networks (Krizhevsky et al., 2012; Ciresan et al., 2012) . However, the operational principles of such deep ensembles, i.e., ensembles composed of deep neural networks, deviate from those of classical statistical models and remain not fully comprehended. For instance, both Lee et al. (2015) and Nixon et al. (2020) validated that bagging (Breiman, 1996) , built on the theoretically well-motivated bootstrap method (Efron, 1992) , does not offer any benefits over the simplest deep ensembles consisting of models obtained from multiple training runs with different random seeds.Empirically, it is well-known that we need nothing more than employing different initializations for each ensemble member to construct high-performance deep ensembles (Lakshminarayanan et al., 2017) . Fort et al. (2019) delved deeper into this and emphasized the significant role played by a highly non-convex loss function of deep neural networks, where varying initializations for stochastic optimization yield different functional modes. It also aligns with the Bayesian perspective provided by Wilson and Izmailov (2020) , which suggests that deep ensembles are involved in approximating multimodal posterior distribution to the Bayesian model averaging. To sum up, while the operational principle of deep ensembles may differ from classical ensembles, the underlying idea regarding diversity remains constant (Krogh and Vedelsby, 1994; Ortega et al., 2022) , i.e., ensembles demonstrate improved performance when their individual members offer diverse predictions.However, the simple strategy mentioned earlier, which aims to cover multiple modes on the loss landscape by starting from different initializations to achieve ensemble diversity (Fort et al., 2019; Wilson and Izmailov, 2020) , faces challenges in modern transfer learning scenarios. It arises from the typical situation that there is only one pre-trained model available for fine-tuning; due to the considerable cost for pre-training of large models, model providers usually do not distribute multiple model copies. In such cases, fine-tuned solutions originating from the same pre-trained weights often inhabit the same pre-trained basin, leading to restricted exploration within the loss landscape (Neyshabur et al., 2020; Mustafa et al., 2020) . Consequently, our attention should be directed towards addressing the local structure around the pre-trained basin in modern transfer learning scenarios (Wortsman et al., 2022; Sadrtdinov et al., 2023; Lee et al., 2024) , rather than the global multimodal structure of the loss landscape (Fort et al., 2019; Wilson and Izmailov, 2020) . 
preliminaries: 
 Finite precision number systems. Computers use binary sequences for data encoding, with FP32 serving as the primary finite precision number system employed to represent real numbers. Given its coverage from approximately 10 -38 to 10 38 with a resolution of 10 -7 , we consider the FP32 system as the continuous set of real numbers R. Moreover, we describe INT-B systems, commoly utilized low precision number systems for neural network quantization:EQUATIONwhere the system can represent real numbers in the range [-w absmax , w absmax ] with a resolution of w absmax /(2 B-1 -1), and the integer m can be encoded using B bits. This is the simplest form of integer quantization, with possible variations like a zero offset or non-uniform resolution (Gholami et al., 2021; Yvinec et al., 2023) . Unless otherwise specified, our experimental results employ this basic form of symmetric uniform quantization with B = 5 for simplicity. We also use per-channel granularity, sharing the number systems among the output channels of each linear layer.Rounding rules. Let F ⊂ R be a finite precision number system. For any w ∈ R, there are two rounding options in practice: ⌊w⌋ = max { ŵ ∈ F ∶ ŵ ≤ w} and ⌈w⌉ = min { ŵ ∈ F ∶ ŵ ≥ w}. The rounding-to-nearest (RTN) scheme deterministically selects the closest one, i.e., ŵ = ⌊w⌉, whereas the Bernoulli stochastic rounding (BSR) scheme randomly chooses one of them, i.e., ŵ = ⌊w⌋ with probability ⌈w⌉ -w, or ŵ = ⌈w⌉ with probability w -⌊w⌋.(2) Observing empirically that BSR sometimes produce superior results compared to RTN motivates the neural network quantization community to explore more sophisticated rounding schemes (Nagel et al., 2020; Lee et al., 2023) . On the other hand, recognizing the presence of multiple competitive solutions, we consider leveraging them for low precision ensembling.Ensemble methods. In ensemble methods for classification problems, the final prediction during testing is obtained by combining S predictions:EQUATIONwhere y is a class label, x is an input, and p(y|x, w s ) is the categorical probabilities predicted by the s th member, which is a neural network model with parameters w s . These w s could either be maximum a posteriori (MAP) solutions obtained through multiple stochastic optimizations (Lakshminarayanan et al., 2017) , or they might be intermediate checkpoints from a single training run (Huang et al., 2017; Garipov et al., 2018) . In the Bayesian framework, neural network weights w are treated as random variables, and w s are seen as Monte Carlo samples employed to approximate the posterior distribution. More precisely, Eq. 3 can be seen as a simple Monte Carlo method to approximate the posterior with a set of point masses, where the locations are given by samples from another approximate posterior q, i.e., p(w|D) ≈ ∑ S s=1 δ(ww s )/S, w s ∼ q(w), where D denotes the data and δ is the Dirac delta function (Wilson and Izmailov, 2020) . A common practice is to use a Gaussian approximation q(w) = N (w; µ, Σ) to generate samples w s ∼ q(w) in a tractable manner (Maddox et al., 2019; Shen et al., 2024) , and then approximate the predictive distribution by computingEQUATION 
an empirical study of low precision ensembling: 
 We present a simple yet effective low precision ensemble construction strategy, Low Precision Ensembling with Bernoulli Stochastic Rounding (LPE-BSR), which computes Eq. 4 usingEQUATIONfor i = 1, . . . , D. Here, D denotes the number of neural network parameters, where each perchannel parameter group shares the same low precision number system, as explained in Section 3. Using the rounding operations ⌊⋅⌋ and ⌈⋅⌉ defined within each system, the probability is determined by λ i = ⌈w i ⌉ -w i , as in Eq. 2. Certainly, the proposed LPE-BSR is not Bayesian, and we have simply expressed it in the form of Eq. 4 for the sake of notational simplicity.We begin with motivating experiments using the publicly available series of pre-trained vision transformer models (ViT; Dosovitskiy et al., 2021) . Detailed information about each model can be found in Appendix A. Table 1 summarizes the evaluation results on a subset of the ImageNet validation split, along with the parameter count for each model. In this context, the pre-trained model corresponds to the star-shapred marker (☆) in Fig. 1 , with RTN using the nearest value in low precision number systems as shown in Fig. 1 (c), and LPE-BSR forming an ensemble by selecting S = 10 nearby samples as illustrated in Fig. 1(d) .Table 1 provides the following key findings: 1) Larger models experience less performance degradation when reducing the precision of numerical systems. More precisely, in the RTN results, the classification error increases from .243 → .247 → .315 when transitioning from FP32 → INT-6 → INT-4 at ViT-T/16, whereas at ViT-L/16, it shifts from .165 → .165 → .167. 2) Lower precision systems introduce diversity among samples in LPE-BSR. Specifically, ensemble ambiguity is the metric for quantifying the ensemble diversity (to be defined in Section 4.3), and in all models, the ensemble ambiguity increases when transitioning from INT-6 → INT-4. Importantly, these findings are in line with the core principle of ensemble methods being effective: a key condition for an ensemble of classifiers to outperform any of its individual members is when the classifiers are both accurate and diverse (Dietterich, 2000) . When introducing the low precision number system, 2) suggests that diversity can be achieved by leveraging the quantization error inherent in this process, while 1) emphasizes that larger models maintain accurate individual performance throughout this process. With this compelling motivation for low precision ensembling established, we now proceed to compare it with existing ensemble methods.Our initial investigation into the proposed LPE-BSR aims to assess the effectiveness of collecting ensemble members within the discrete space defined by the low precision number system. While using fewer bits to represent samples would certainly reduce ensemble costs, there is a concern that we might overlook potentially good ensemble candidates outside this discrete space. To address this, we conduct a comparative study with two Bayesian deep learning methods: improved variational online newton (IVON; Shen et al., 2024) and stochastic weight averaging Gaussian (SWAG; Maddox et al., 2019) . Both methods use samples drawn from an approximate Gaussian posterior in the continuous weight space and perform ensembling in a Bayesian manner.To sum up our experiment, we first fine-tune the zero-shot CLIP-ViT model (Radford et al., 2021) on the ImageNet training split to obtain the MAP solution w * MAP . For LPE-BSR, this fine-tuning can employ any of the SGD, SWAG, or IVON optimizers. We then define q according to Eq. 5 using w * MAP and compute Eq. 4 with S samples. For SWAG and IVON, q(w) = N (w; µ, Σ) is defined using the µ = w * MAP and Σ obtained during their respective optimization processes, followed by the computation of Eq. 4. Consequently, LPE-BSR samples neighboring points of w * MAP within the discrete space defined by the low precision number system, whereas SWAG and IVON sample nearby points of w MAP in the continuous space with Gaussian noise added. For more details on SWAG and IVON, including their hyperparameters, please refer to Appendix E. Fig. 2 presents the outcomes of our experiments conducted with CLIP-ViT-L/14. In our experimental setup, both SWAG and IVON successfully estimated both the MAP mean and the covariance matrix, resulting in a lower negative log-likelihood compared to MAP through Bayesian model averaging, as illustrated in the second and third subplots of Fig. 2 . Remarkably, our LPE-BSR, derived from the MAP solution obtained by the SGD optimizer, achieves competitive results with Bayesian model averaging through SWAG or IVON. Moreover, when using the same MAP solution obtained with the SWAG or IVON optimizer for a fair comparison, it even outperforms these methods.From a numerical integration perspective (Wilson and Izmailov, 2020; Wilson, 2021) , the conditions for successful approximate Bayesian inference in deep learning are very similar to those for successful ensembling, as discussed in Section 4.1 with reference to Dietterich (2000) . Specifically, it entails 1) finding typical points in the posterior that represent regions of substantial mass (cf. accurate); and (ii) ensuring a diverse set of points to give rise to different functions (cf. diverse). Consequently, we proceed to conduct a comparative analysis of these two factors concerning our LPE-BSR method and the Bayesian methods we considered, SWAG and IVON. Quantitative assessment of ensemble diversity is an essential metric for evaluating ensemble methods. To this end, we adopt the generalized ambiguity decomposition for cross-entropy loss presented by Wood et al. (2023) , which can be easily measured using logit ensembling instead of probability ensembling. It should be noted that logit ensembling is solely utilized for diversity analysis, whereas probability ensembling is used for all other experimental results to ensure a fair comparison with Bayesian methods that compute the categorical predictions using the BMA integral. Please refer to Appendix B for more details on our diversity analysis.Table 2 presents our experimental outcomes for the generalized ambiguity decomposition (cf. 'Diversity analysis'), along with the final ensemble perfomance (cf. 'Evaluation metrics'). Lowering the precision of numerical systems naturally amplifies quantization error, as evidenced by the results in the INT-B rows, where reducing B results in higher (a) average loss. However, this also coincides with an increase in ensemble diversity, with smaller B values resulting in greater (b) ambiguity. Consequently, the superior (c) ensemble loss at an appropriate precision level (e.g., B = 5) highlights the fundamental principle of the low precision ensembling: it does not merely perceive quantization error problematic, but rather utilizes it to obtain ensemble diversity. Consequently, the proposed LPE-BSR yields improvements in evaluation metrics, as depicted in Table 2 . Definitions for each metric can be found in Appendix B. Drawing inspiration from Fort et al. (2019) , we further provide the radial landscape plot depicting a plane within weight space containing three points (shown as yellow, blue, and orange markers). The z-values for each subplot represent the negative log-likelihood (displayed on the left in a magma colormap), the function differences from the blue model (shown in the center in a blue colormap), and the red model (presented on the right in a red colormap). Namely, the first subplot indicates the placement of each model within the loss landscape, while the subsequent two subplots illustrate the extent to which they differ from each other. Fig. 3 depicts radial landscape plots comparing IVON and LPE-BSR samples, showing their parallel roles in ensemble construction. Both show slightly higher individual negative log-likelihoods, shown by the circle markers, compared to the MAP denoted by a star-shaped marker in the first subplot, while also offering diverse function outputs as demonstrated in the subsequent subplots. Ultimately, LPE-BSR can identify samples in a low precision number system that qualitatively resemble the high-quality posterior samples provided by IVON, which leverages Hessian estimate information to compute the covariance of the approximate Gaussian posterior.A key advantage of LPE-BSR is its ability to gather ensemble members without requiring any backward computation. This feature, which eliminates the need for training, aligns with fast ensembling techniques aimed at enhancing the training efficiency of ensemble construction processes (Huang et al., 2017; Garipov et al., 2018; Benton et al., 2021) . Consequently, we conduct empirical analysis to further investigate this alignment with snapshot ensembling (SSE; Huang et al., 2017) , as well as cyclical stochastic gradient Langevin dynamics (CSGLD; Zhang et al., 2020) , a closely related Bayesian posterior sampling algorithm. Both methods involve collecting snapshot samples on the loss landscape around w MAP using a cyclical learning rate schedule. For more details, including hyperparameters, please refer to Appendix E.We first verify whether LPE-BSR can generate an ensemble component distinct from SSE snapshots using radial landscape analysis. Fig. 4 illustrates a plane subspace containing the first and second SSE snapshots (displayed as star-shaped markers) and the LPE-BSR sample obtained from the first snapshot (represented by a circle marker). Indeed, LPE-BSR provided a novel sample that could contribute to the ensemble; it is clearly diverse from the existing snapshots, as shown in the second and third subplots, while achieving reasonably low individual negative log-likelihoods, as shown in the first subplot. By using such LPE-BSR samples along with SSE snapshots to build an ensemble, we can reduce the cost of achieving target performance in fast ensembling methods or attain better results with the same training budgets. However, while fast ensembling techniques usually prioritize evaluating training budgets, particularly the number of backward passes as seen in the literature (Huang et al., 2017; Garipov et al., 2018; Zhang et al., 2020) , we also consider memory budgets in our analysis-the total number of bits required to represent the entire ensemble model, which grows with the addition of ensemble members in fast ensembling. From this perspective, we devised a method to eliminate heavy SSE snapshots with high precision from the final ensemble; as a result, each original SSE snapshot is replaced by S = 5 LPE-BSR samples. This policy also aligns with our reserach objective of exploring ensembling in low precision number systems. Consequently, by forming the ensemble exclusively with LPE-BSR samples in the low precision system, we attained better outcomes compared to SSE regarding both training budgets and memory budgets, as shown in Fig. 5 .Our investigation so far, involving deep neural networks up to 300M in size, substantiates the efficacy of the proposed LPE-BSR methodology. In this section, we further extend our validation to confirm that LPE-BSR enables effective ensemble construction without training, even in larger models. To this end, in addition to 300M-scale CLIP-ViT-L/14 model (Radford et al., 2021) , we employ a 1B-scale CLIP-ViT-G/14 model (Cherti et al., 2023) and an 8B-scale LLaMa model (Touvron et al., 2023) in a zero-shot manner. Appendix A provides public links for each model.We first present a radial landscape analysis of the CLIP-ViT-L/14 model in Fig. 6 . As we analyzed previously in Section 4.3, we can confirm that the conditions for effective ensemble construction are met here as well, i.e., ensemble members exhibit slightly higher individual negative log-likelihoods (circle markers) compared to the pre-trained model (star-shaped marker) in the first subplot, and they also offer diverse function outputs, as shown in the subsequent subplots. As depicted in the leftmost When the x-axis represents the ensemble size. Bottom: When the x-axis represents memory budgets, i.e., the total number of bits for representing ensemble.Table 3 : Results for low precision ensembling of pre-trained models. We compute (a) average loss, (b) ambiguity, and (c) ensemble loss for diversity analysis, along with evaluation metrics to assess overall performance. Our LPE-BSR samples are centered around ☆ within each group (pretrained model in this context), which are separated by horizontal lines. subplot of Fig. 7 it leads to achieving lower negative log-likelihood through LPE-BSR compared to the pre-trained model without the need for additional training.Fig. 7 demonstrates that LPE-BSR consistently improves upon the pre-trained checkpoint, even for larger models such as CLIP-ViT-G/14 and LLaMa-3. The subplots at the top of Fig. 7 demonstrate that the performance of LPE-BSR improves with increasing ensemble size, while the subplots at the bottom show that LPE-BSR occupies the preferred lower-left region of the trade-off plots for memory budgets and performance, surpassing the pre-trained checkpoint. Table 3 offers more detailed results for an ensemble size of S = 20, including diversity analysis and evaluation results, further confirming the effectiveness of our proposed LPE-BSR for models with billions of parameters. 
related work: 
 The field of Bayesian deep learning provides the most relevant research for our low precision ensembling strategy; Ferianc et al. (2021) integrated quantization-aware training (Jacob et al., 2018) into established Bayesian deep learning methods; Zhang et al. (2022) introduced a technique for implementing stochastic gradient Langevin dynamics (Welling and Teh, 2011) with reduced precision. However, our work differs significantly from theirs in two key aspects: 1) They focused on training from scratch, which deviates somewhat from the prevalent practice of utilizing pre-trained large models. 2) They employed small-scale models; the largest model they considered, ResNet-18 with 11 million parameters, falls outside our scope as discussed in Section 4.1, as we are interested in larger scales. Nonetheless, the interest in employing low precision ensembling in Bayesian deep learning holds significant promise. Our demonstration of its feasibility for large models constitutes a meaningful advancement for the Bayesian deep learning community (Papamarkou et al., 2024) . 
conclusion: 
 We provided a novel insight on ensembling within low precision number systems. While conventional wisdom perceives quantization errors stemming from representing neural network weights in low precision as obstacles, we introduced an innovative viewpoint suggesting that these errors could serve as a source of ensemble diversity. Our empirical results demonstrated that low precision weights obtained through stochastic rounding of pre-trained weights could effectively form ensembles and improve uncertainty estimates and calibration, especially for large models. Considering the growing scale of models in recent trends reduces the appeal of ensemble methods due to their inherent scalability issue, where memory costs increase with the number of ensemble components, our exploration of low precision ensembling lays the foundation for developing efficient ensemble methods in the era dominated by large models.Limitations and future directions. At present, our investigations have centered on the simplest form of low precision number system, known as the symmetric uniform quantization scheme. Similar to the quest in neural network quantization for systems that yield better quantized solutions (e.g., Yvinec et al., 2023; Dettmers et al., 2024) , the search for systems conducive to more effective low precision ensembling presents an intriguing avenue for future research. Furthermore, we used fake quantization across all experiments for research purposes, which prevented us from benchmarking the latency of the low precision ensemble due to limited access to specialized hardware and software for accelerating the inference speed of quantized models. Nonetheless, as our work relies on the standard symmetric uniform quantization scheme, it remains compatible with ongoing and future advancements in neural network quantization. Developing practical components such as custom CUDA kernels tailored to low precision ensembles would also be a promising future direction.Broader impacts. Our method advocates for the utilization of large models, which could potentially raise ethical concerns (e.g., Weidinger et al., 2021) . However, it is important to note that this work primarily focuses on analytical aspects and does not inherently entail significant ethical risks. 
 
 Appendix: The pre-trained weights utilized in our experiments are listed below. We refer readers to the respective papers fdetails on each model: ViT (Dosovitskiy et al., 2021) , CLIP (Radford et al., 2021; Cherti et al., 2023) , and LLaMa (Touvron et al., 2023) . In our CLIP experiments, we obtained zero-shot head weights by following the standard procedure outlined in the official code base1 .• ViT-T/16: https://huggingface.co/WinKawaks/vit-tiny-patch16-224/tree/main • ViT-S/16: https://huggingface.co/WinKawaks/vit-small-patch16-224• ViT-B/16: https://huggingface.co/google/vit-base-patch16-224• ViT-L/16: https://huggingface.co/google/vit-large-patch16-224• CLIP-ViT-L/14: https://huggingface.co/openai/clip-vit-large-patch14• CLIP-ViT-G/14: https://huggingface.co/laion/CLIP-ViT-bigG-14-laion2B-39B-b160k• LLaMa-3-8B: https://huggingface.co/meta-llama/Meta-Llama-3-8B-InstructWe employed two datasets for our experiments: ImageNet (Russakovsky et al., 2015) for ViT and CLIP-ViT models, and MMLU (Hendrycks et al., 2021) for LLaMa. The evaluation of MMLU was conducted using the template provided in the official repository2 , and the computation was based on a micro-average.Let p i ∈ [0, 1] K represent the predicted categorical probabilities and y i ∈ {1, . . . , K} denote the ground truth label for the i th data point, where the total number of data points is N .NLL. The negative log-likelihood (NLL) of a categorical distribution, also known as cross-entropy loss, is a fundamental metric for evaluating the performance of a classification model:EQUATIONERR. Another primary metric commonly used to assess the performance of a classification model is the classification error (ERR), also referred to as the 0-1 loss:EQUATIONwhere [⋅] denotes the Iverson bracket.ECE. The common choice for measuring calibration in machine learning is the expected calibration error (ECE), particularly its empirical variant with binning (Pakdaman Naeini et al., 2015) :EQUATIONwhere B j denotes the j th bin comprising |B j | data points whose prediction confidence max k p (k) i falls within the interval ((j -1)/J, j/J]. Here, acc(B j ) is the classification accuracy of B j , and conf(B j ) is the average confidence within B j . As a result, it computes a weighted average of calibration gaps, which are the differences between accuracy and confidence, across bins. Throughout our experiments, we employed J = 15 bins for ECE computation.Ensemble ambiguity. Let z s,i ∈ R K be categorical logits predicted by the s th model for the i th data point. The generalized ambiguity decomposition can be written as where σ denotes a softmax function that maps categorical logits into probabilities. It is worth noting that logit ensembling in (c) is essentially the same as computing a a normalized geometric mean for categorical probabilities. For further information, please see Wood et al. (2023) . EQUATIONFig. 8 is a modified version of Fig. 2 , where the x-axis has been changed from ensemble size to memory budget, defined as the total number of bits used to represent the ensembles. It can be interpreted as trade-off plots between memory budgets and performance. Compared to SWAG and IVON, which represent ensemble members in the FP32 system, LPE-BSR occupies the preferred lower-left region, where ensemble members are represented in an INT-5 system. Wen et al., 2020) would be valuable.Table 5 summarizes our experimental results using the Adam optimizer (Kingma and Ba, 2015) . We observed that in LPE-BSR, (a) each ensemble member had relatively lower performance (= 0.513). However, (b) due to high ensemble diversity (≥ 0.025), (c) there was a significant improvement in the final ensemble performance. Consequently, it achieves performance comparable to BE, another memory-efficient method available in fine-tuning scenarios. In BE, ensemble members are similarly centered around one solution, with members derived from shared weights by multiplying rank-one matrices, while LPE-BSR members are derived from center weights using stochastic rounding. This comparison with BE, a well-known memory-efficient ensembling strategy, highlights the potential of low precision ensembling with LPE-BSR.Our LPE-BSR method forms a low precision ensemble from a given checkpoint in a training-free manner, as presented in Section 4. We further conduct additional comparative experiments with two baselines: 1) Gaussian, which builds an ensemble by adding Gaussian noise with fixed variance to the pre-trained weights; and 2) Monte Carlo Dropout (MCD; Gal and Ghahramani, 2016) , which constructs an ensemble by applying the dropout technique during inference. MCD is particularly relevant as it uses a q(w) form similar to Eq. 5 of LPE-BSR, employing δ(0) and δ(w).Table 6 summarizes the results for CLIP-ViT-L/14 with an ensemble size of S = 20. It clearly shows that while both the Gaussian and MCD baselines can perform ensembling in a training-free manner with appropriately tuned noise scales-specifically, the variance of Gaussian noise for the Gaussian baseline and the drop probability for the MCD baseline-our proposed LPE-BSR outperforms them.It is worth noting that LPE-BSR is more memory-efficient, as each of its ensemble members uses INT-5, compared to FP32 used by the baseline methods. Therefore, LPE-BSR not only achieves better performance but also does so with reduced memory usage.We built our experimental code using JAX (Bradbury et al., 2018) and Transformers (Wolf et al., 2020) , both licensed under Apache-2.0.3 We conducted experiments using TPUv2/v3/v4 cores, with flexibility in selecting the cores based on the memory requirements of each experiment. The code is available at https://github.com/cs-giung/lpe-bsr.The optimization process for CLIP-ViT-L/14 models concludes after 100,000 iterations with a minibatch size of 64, employing a cosine decaying learning rate schedule. In the experiments described in the main text, we employed the following optimizers: SGD, IVON, and SWAG. All hyperparameter tuning was conducted using a development set created by taking 1% of the training dataset, i.e., 'training[99%:]' in TensorFlow Datasets (Abadi et al., 2015) .The zero-shot head weights were kept entirely fixed, meaning they were not fine-tuned or quantized in any of the experiments. We also froze the embedding layer to enable basic SGD update rules to function with the transformer architecture, as described by Kumar et al. (2024) .SGD. Stochastic gradient descent (SGD) is a foundational stochastic optimization algorithm in machine learning (Robbins and Monro, 1951) . Although it is usually deemed ineffective for transformer architectures, recent findings by Kumar et al. (2024) has shown that a simple modification-freezing the embedding layer-enables pure SGD, even without momentum, to produce results competitive with Adam (Kingma and Ba, 2015) . The hyperparameter settings for the SGD optimizer utilized in our experiments are summarized in Table 7 .IVON. Efforts to develop variational methods for implementing Bayesian inference on neural network models have continued over time (Graves, 2011; Blundell et al., 2015) . However, these at-https://github.com/openai/CLIPhttps://github.com/hendrycks/testhttps://www.apache.org/licenses/LICENSE-2.0 


Decision: As stated by the Authors, the main contribution of the paper is offering a fresh perspective by using quantization errors--often seen as challenges in quantization research--as a source of diversity in ensemble methods. 

This approach has a particular significance in the era of LLMs. Ensembling is known to improve robustness and the proposed approach makes ensembling more accessible for LLMs due to the reduce memory requirement.

The paper attracted diverging reviews, with two reviewers voting for acceptance (6,5) and two voting for rejection (4,4). It is a borderline paper and the decision will naturally be based more on the highlighted weaknesses and strengths, and less on the absolute score.

Two reviewers emphasized the lack of novelty, which I respectfully disagree with. If the previous work does not disclose the proposed method, the fact it is simple is actually more of a strength of the work rather than a limitation. I would also argue that the results suggest that quantization of the model has interesting and nontrivial effects on its robusstness. It would be interesting to study how these different ensemble members differ, e.g. in terms of which examples they memorize from the original dataset. 

A very fair criticism was raised that the paper lacked comparisons to methods such as SWAG, Deep Ensemble or Batch Ensemble. This I think was convincingly addressed in the rebuttal phase.

All in all, I am recommending acceptance. I believe the method might have an impact, both for applied deep learning and development of new methods.

Reviews: review: paper_summary: paper_summary: The paper proposes that ensembles of quantized low-precision instances of large models outperform the source models on image classification and MMLU tasks. The low precision models are generated using Bernoulli stochastic rounding. The authors support their claims by presenting empirical results for several models, and show improvements over bayesian ensembling methods on negative log likelihood, ensemble diversity and error rate.
strength_weakness: strength_weakness: 1. The paper presents NLL comparisons with SWAG-Diag and IVON. It would be great to see accuracy numbers as well to understand the actual performance differences as all the results are presented on classification tasks. In addition, Table. 1 should have error estimates for LPE-BSR given that it relies on randomness. Also given that LPE-BSR is not really a Bayesian method, it would be great to also compare with other ensembling approaches like Model Soups to see where it lands on the pareto curve of compute-vs-accuracy. Also, the SWAG algorithm shows significant performance differences between SWAG, and SWAG-diag. While it may not be possible to compare for all the presented models, an exemplar comparison would also help support the conclusions presented in the paper.

2. It would also be great to see some additional analysis on the actual compute savings incurred using LPE-BSR.
questions: questions: 1. Fig. 2 shows that SWAG does not improve with increasing ensembl sizes. This is a bit counter-intuitive. I may have misunderstood the figure, and invite some clarification if so.

2. Are the LLAMA results presented on MMLU 5-shot or 0-shot?
limitations: limitations: The authors discuss some limitations in the paper. However, I suggest adding the actual compute requirements for the ensemble of low precision models.
paper_summary: paper_summary: This paper addresses the scalability challenge in ensembling deep neural networks for large models by introducing a novel low precision ensembling method. The approach generates an ensemble of models from a single model using low precision number systems in a training-free manner. Empirical analysis shows that this method achieves comparable or superior generalization performance. The findings suggest that low precision ensembling is a promising solution to enhance generalization performance while addressing the scalability issues inherent in traditional ensembling methods.
strength_weakness: strength_weakness: - Firstly, I think the paper is lacking some novelty. While interesting, it is not very surprising to me that an ensemble of randomly sampled low precision models can improve the generalization performance and uncertainty calibration. There have been numerous works of similar flavor. 
- I think the paper lacks some important benchmark comparisons. For instance, how does the proposed method compare to other sampling based ensembling approaches like dropout ensemble [1] and batchnorm ensemble [2]? 

[1] "Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning"

[2] "Bayesian Uncertainty Estimation for Batch Normalized Deep Networks"
questions: questions: - How does the diversity of low precision ensembles compare with other popular ensembling techniques like deep ensemble and batch ensemble? 
- It would also be interesting to benchmark the performance of the proposed method in terms of computation cost (in addition to the memory budgets shown in Figure 5).

[1] "BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning"

[2] "Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles"
limitations: limitations: Yes, the paper addresses the limitations of the work.
paper_summary: paper_summary: This paper presents a new way to generate an ensemble of models without the need for training multiple times and with the extra advantage of using low-precision representation which inherently saves memory.
The idea is to build an ensemble of models starting from stochastic variations of a single model. Those variations are based on a stochastic rounding scheme that samples either the ciel or the floor of a real number proportionally to the distance to the two rounded values (called Bernoulli stochastic rounding).
Results show that, especially for large models (see Tab.1), this technique provides accuracies that are better than a single model with the advantage of a quantized representation that has a lower memory footprint.
strength_weakness: strength_weakness: \- The proposed idea is interesting and I think it has some potential. However, I think that the authors did not analyse and compare in a fair way the drawbacks of the approach. Below some points.

\- Compare with Deep ensembling. The method is interesting because it does not require to retrain the model. However, authors should compare with a real ensembling of multiple models. 

\- Compare with others without adding its contribution. If the pre-training of large models is too expensive to be evaluated multiple times, the authors should compare with Bayesian and fast ensembling methods. Thus, instead of just adding the approach on top of SWAG, IVON and SSE, it would be interesting to see comparative results of each technique independently. For instance in Tab.2 we can see that in terms of error, the proposed approach (LPE-BSR) trained with normal SGD (error=.137) is inferior to IVON (error=.135) and comparable to SWAG (error=.137). 

\- Weight averaging. Another important point when working with pre-trained models is weight averaging. As the different ensembles come from the same pre-training, normally they are aligned and the ensemble can be approximated with weight averaging. This allows to gain in accuracy with a single inference. What is the performance of the proposed approach with weight averaging?

\- Another possible baseline to compare with is to build an ensemble by adding Gaussian noise with a fixed variance for all values tuned (for ensembling) on a validation set. Although very simple, that approach can be quite effective.
questions: questions: See the questions above associated to weaknesses.

Additional questions:

\- In tab.2 Why not showing all configs for SGD?

\- Fig.3, what is the difference among the three subfigures?
limitations: limitations: Some of the possible limitations of the approach are not analysed in the paper.
For instance, the proposed approach is not contrasted with other common ensembling techniques. Instead it is added to them. In this way, it is difficult to know the real efficacy of the approach.
Also, I did not see the performance of the approach for weight averaging. If the approach does not work well for weight averaging, it should be mentioned.
paper_summary: paper_summary: This paper addresses the problem of ensembling in low precision number systems, where quantized models act as members of the ensemble. The authors suggest that quantization errors can be leveraged to enhance ensemble diversity. Based on this concept, they propose a method called LPE-BSR. Through extensive experiments and analyses, the authors validate the effectiveness of their approach.
strength_weakness: strength_weakness: 1. The paper relies on existing techniques for both ensembling and quantization. It primarily presents an empirical study on the combination of these two approaches rather than introducing a new method.
2. The practical utility of the proposed LPE-BSR method is somewhat constrained. Large models typically incur high inference costs, and ensembling them—even with low precision systems—can amplify these costs, particularly with ensemble sizes of 10 or 20, as discussed in the paper. Furthermore, according to the experimental results in Table 3, the performance improvement of the quantized ensemble over the full precision model is marginal. As an empirical study, the authors should better justify why ensembling in low precision number systems is a meaningful objective.
3. The related work section does not provide enough context or comparison with existing research. It would be beneficial to include a more detailed literature review.
4. The subfigures in Figures 3, 4, and 6 are difficult to interpret. Adding captions or labels for each subfigure would improve clarity.
questions: questions: 1. In line 174, what does $w_\text{MAP}$ mean? Is it a typo for $w^*_\text{MAP}$?
2. In line 179, the paper concludes that "it encountered difficulties in accurately estimating the diagonal covariance matrix." However, this conclusion is not clearly supported by the results presented in Figure 2. Please provide more evidence or clarification for this statement.
limitations: limitations: Yes.

Comments: title: details about experiments with MCDropout and Gaussian noise
Thanks for the response
comment: For the latency issue, as noted in our response to Reviewer svQu, it can be effectively managed through parallelization and techniques from the quantization field. For example, Kim et al. (2024) demonstrate improved latency with quantized models, with the RTN method we utilized showing the greatest speedup. Regarding the novelty aspect, we would appreciate it if you could provide the relevant works of a similar flavor you mentioned, as they would greatly assist in further refining our research. Thank you again for reviewing our paper!

---
Kim et al. (2024), SqueezeLLM: Dense-and-Sparse Quantization.
Our approach indicates that forming a low-precision ensemble from a pre-trained model can improve performance over the original. While there is some debate about whether a quantized version of a larger pre-trained model should be considered--since such models are not always available--we want to emphasize that our work, which highlights the potential of low-precision ensembling for large modern models, remains significant and relevant. Thank you again for taking the time to review our paper!
I would like to thank the authors for providing a careful feedback on my comments and additional experiments to demonstrate the effectiveness of the proposed method. Most of my concerns regarding comparisons with other benchmarks have been resolved. However, I still share the same reservation as reviewer svQu in terms of the novelty of the paper (that the proposed method is "an empirical study on the combination of these two approaches") and the "inference latency" (that the proposed method only addresses the memory cost issue but not inference latency). While I understand that it can be hard to do a thorough empirical investigation without specialized hardwares, qualitative discussions on the potential benefits the proposed quantization can bring is arguably much less convincing. As such, I would like to retain my original score.
I appreciate the authors' prompt response. As mentioned, the LPE-BSR (INT-5) ensemble model does outperform a single FP32 model in terms of both memory footprint and performance. However, if we were to simply adopt a model with more parameters and better performance, applying standard quantization, it might lead to a single dense model with similar memory requirements while performing better than the ensemble. Additionally, dense models typically benefit from a higher level of parallelism in practice compared to an ensemble approach. Therefore, the practical effectiveness of the proposed method is not sufficiently convincing for me. Given this, I will maintain my initial score.
We acknowledge the reviewer’s concerns about the inherent memory and speed costs of ensemble methods. However, we want to highlight that our low-precision ensembling approach can significantly alleviate these issues through the use of quantized models.

One of the key advantages of our proposed low-precision ensembling method is that the reduced size of each quantized model makes parallel execution more feasible. Unlike conventional ensembles of full-precision models, which often struggle with parallelization due to their large memory footprint and must be executed sequentially, our approach allows for efficient parallelization. For example, our LPE-BSR (INT-5) ensemble, consisting of five models, requires less memory than a single FP32 model. This means that in environments where full-precision FP32 models are deployable, there is sufficient memory to support the parallel execution of our LPE ensemble.

While parallelization can significantly boost efficiency, we acknowledge that it may not entirely eliminate the latency issues associated with using multiple models in an ensemble. To better address these latency challenges, it is essential to utilize specialized hardware and software optimized for accelerating the inference of quantized models. Although we used fake quantization for our research due to the lack of access to such hardware, our method follows the standard symmetric uniform quantization scheme, which is compatible with both existing and emerging advancements in neural network quantization. By integrating cutting-edge techniques from the quantization community, we can further improve the inference speed of parallelized ensembles and more effectively address latency concerns.

We assure you that all these points will be thoroughly addressed in the final manuscript. We welcome any additional comments and suggestions, and we appreciate your constructive feedback and the opportunity to enhance our work.

Sincerely,  
The authors of Submission 19991
Thank you for the response. Most of my questions have been addressed. However, regarding the inference costs, memory budget is not the only factor need to be take into consideration. The inference latency should also be compared, since the ensemble of plenty of models would greatly increase the computational complexity.
We appreciate Reviewer bKfp's additional question about the hyperparameters of MCDropout and Gaussian baselines and are happy to provide more detailed results:

| Method               | NLL / ERR / ECE    |
| :-                   | :-                 |
| Pre-trained          | .948 / .251 / .049 |
||
| LPE-BSR (INT-5)      | __.929__ / __.250__ / __.028__ |
| MCDropout (p=0.001)  | .946 / .251 / .047 |
| MCDropout (p=0.002)  | .946 / .251 / .047 |
| MCDropout (p=0.004)  | .944 / __.250__ / .046 |
| MCDropout (p=0.008)  | .940 / __.250__ / .044 |
| __MCDropout (p=0.016)__  | __.938__ / .251 / .041 |
| MCDropout (p=0.032)  | .944 / .255 / __.034__ |
||
| LPE-BSR (INT-5)      | __.929__ / __.250__ / __.028__ |
| Gaussian (σ^2=0.0001)  | .948 / .251 / .049 |
| Gaussian (σ^2=0.0002)  | .948 / .251 / .048 |
| Gaussian (σ^2=0.0004)  | .946 / __.250__ / .046 |
| Gaussian (σ^2=0.0008)  | .941 / __.250__ / .043 |
| __Gaussian (σ^2=0.0016)__  | __.934__ / __.250__ / __.031__ |
| Gaussian (σ^2=0.0032)  | .981 / .264 / .038 |

Here, p refers to the drop probability in MCDropout, and σ^2 denotes the variance of Gaussian noise. The previously reported values were obtained by identifying p or σ^2 values that are optimal--not too small or too large--where the ensemble benefit is greatest. We assure to further clarify this point and include it in the final manuscript.

Sincerely,  
The authors of Submission 19991
We are pleased to know that you found our additional clarifications and results helpful. We will definitely address all reviewer concerns in the final manuscript. Thank you once again for your constructive and supportive feedback!

Sincerely,  
The authors of Submission 19991
Thank you for your additional experiments. Could you please provide more details about the hyper-parameters of MCDropout and Gaussian noise used to build the ensemble?
Which hyper-parameters did you consider? Which values? 
Did you select the optimal hyperparameters? Did you put as much effort in finding the optimal hyper-parameters as in your method?
Thanks
I thank the authors for their clarifications and for fixing the issues with SWAG results. I also find the new memory budget plots useful in actually conveying the tradeoffs of using the proposed approach versus model soups, and finetuning. I am going to keep my positive score.