Training Binary Neural Networks via Gaussian Variational Inference and Low-Rank Semidefinite Programming
Authors: Lorenzo Orecchia
Jiawei Hu
Xue He
Wang Zhe Mark
Xulei Yang
Min Wu
Xue Geng
Venue: NeurIPS 2024 poster (2024)
Keywords: Binarized Neural Networks; Variational Inference; Semi-definite Programming;
Primary area: optimization_for_deep_networks

Abstract: Current methods for training Binarized Neural Networks (BNNs) heavily rely on the heuristic straight-through estimator (STE), which crucially enables the application of SGD-based optimizers to the combinatorial training problem. Although the STE heuristics and their variants have led to significant improvements in BNN performance, their theoretical underpinnings remain unclear and relatively understudied. In this paper, we propose a theoretically motivated optimization framework for BNN training based on Gaussian variational inference. In its simplest form, our approach yields a non-convex linear programming formulation whose variables and associated gradients motivate the use of latent weights and STE gradients. More importantly, our framework allows us to  formulate  semidefinite programming (SDP) relaxations to the BNN training task. Such formulations are able to explicitly models pairwise correlations between weights during training, leading to a more accurate optimization characterization of the training problem. As the size of such formulations grows quadratically in the number of weights, quickly becoming intractable for large networks, we apply the Burer-Monteiro approach and only optimize over linear-size low-rank SDP solutions. Our empirical evaluation on CIFAR-10, CIFAR-100, Tiny-ImageNet and ImageNet datasets shows our method consistently outperforming all state-of-the-art algorithms for training BNNs.

Training Binary Neural Networks via Gaussian Variational Inference and Low-Rank Semidefinite Programming
Lorenzo Orecchia University of Chicago
Jiawei Hu Georgia Institute of Technology
Xue He Northeastern University of China
Zhe Wang PRR, S
Xulei Yang PRR. A
Min Wu PRR A
Xue Geng PRR A
Abstract
Improving the training of Binarized Neural Networks (BNNs) is longstanding challenge whose outcome can significantly affect our ability to deploy deep learning ubiquitously. Current methods heavily rely latent weights and the heuristic straight-through estimator (STE), which enable the application of SGD-based optimizers the combinatorial training problem, but remain theoretically poorly understond. In this paper, we propose an optimization framework for BNN training based on Gaussian variational inference. Our approach yields non-convex linear programming formulation that theoretically motivates the use of latent weights, STE and weight clipping. More importantly, it allows us to go beyond latent weights to formulate and solve low-rank semidefinite programming (SDP) relaxations that explicitly model and learn pairwise correlations between weights during training. resulting in improved accuracy. Our empirical evaluation on CIFAR-10 -00, CIFAR-100, Tiny-ImageNet and ImageNet datasets shows our method consistently outperforms all state-of-the-arr algorithms for training BNNs.
Introduction
The advent of deep learning has revolutionized the field of machine learning and enabled stunning technological advances in numerous application areas, including computer vision [He et al. 2017]. speech recognition [Baevski et al.] [2200] and natural language processing (Devlin et al., 2018). Despite these achievements, the broader application of deep learning is impeded by high compu- tational demands, requiring the advanced hardware and energy consumption typically reserved for supercomputers [Thompoon et al. 2022], both for training and inference. This barrier is particularly formidable when deep learning is deployed on aasource-constraine
devices, like smartphones and loT devices, where limitations in memory, processing power and energy critical Sze et al. 2017).
To address these issues, Muller and Indiveri [2015) and Courbariaux et al. [015b] noted that neural networks could provide the same level of performance while restricting the precision of the representation of parameters To reap the full benefits of quantization, [Courbariaux et al. 2015b]] 2016) and Kim and Smaragdis (0166) independently introduced binarized neural networks (BNNs), which use I-bit representations for each weight, directly leading to 32-fold reduction in model size compared single-precision weights, When activation further binarized (Hubara et al. 2016), the multiplication and addition operations can be replaced by much faster and cheaper XNOR and popcount operations, resulting in
small number of bits, effectively quantizing the space of weights.
"Correponndigg authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

significant cost reduction in both memory and computation. Despite their compelling computational advantages, the problem of constructing and training tegh-penfoiing BNNs still open, as current approaches still yield severe accuracy loss compared to their high-precision counterparts [Rastegari
jet al. 2016
Liu et al. 2018).
While substantial effort has been aimed constructing larger and more effective BNN architectures [Umuroglu et al. [017 Tang et al. 2017 Liu et al.) [2200] Martinez et al. 2020 Shen et al. 2020). training BNNS has remained significant challenge, as the binary constraints yield an intrinsically combinatorial optimization problem, which is a poor fit for traditional continuous optimizers like SGD and Adam. For a domain set X' and a label set V/ let f X' x IR"" + y represent a neural network with an n-dimensional weight vector. Denote by yæ the true label of instance x and by L the smooth loss function. The BNN training problem can then be formulated as the following stochastic optimization problem over the hypercube of binary weights w:
BNN training
min Ex [L 2i)), )m, w (1
(1)
where E,z denotes the expectation over x. uniformly distributed over X. This computational problem adds to the challenge of non-convexity, the additional obstacle of combinatorial feasible set, making continuous gradient queries potentially uninformative. Moreover, as the training must be carried out over a BNN architcture, we face the additional restriction that the gradient V Lffxx, uδ), Uve) of the loss can only be evaluated at binary weights v E (=13". This further limits our capability to explore the loss landscape.
Latent Weights Currently, most BNN-training procedures make use of latent real weights w E IR" [Courbariaux et al., 2015a 2016) to maintain the state of an iterative training algorithm and guide the optimization process. Latent weights are rounded to binary weights vù = round(w) (w) via a potentially stochastic function round R" - {+1]" " to compute forward and backward passes over the BNN architecture. The most common choice of round function is simply the deterministic sign function applied to each latent weight. Unfortunately, any non-trivial round function is discontinuous and cannot be differentiable over R", so that it is not possible to evaluate the true gradient of the loss function with respect to the latent weights w. The main practical solution to this problem has been to simply ignore the round function in the prok-propanation of the gradient, yielding the trriigh-ttrougee estimator (STE) [engiio et al. 2013
Le et al. 2022
V (x, round (m)), yn) R V [f(( 1)), ym (ou
(STE)
While there is no theoretical assurance that the STE is a valid proxy for the gradient Yin et al. 2019], the STE and its variants [Le et al., 2022 Wu et al. 2023) have proved remarkably effective in practice, particularly in combination with weight clipping [Alizadeh et al. [2018) Merolla et al. 2016), by which latent weights are constrained to fixed range around the origin.
Theoretical Interpretations of Latent- Latent-Weights Weights Methods Latent weights have usually been in- terpreted as fractional upproximations of the true binary weights [Anderson and Berg 2017). The influential work of Helwegen et al. [2019] instead proposes to view latent weights as a measurement of the algorithm's 's confidence in the binary weight taking on certain sign. With this intuition. Ajanthan al. 2019 and Meng et al. 2020) have suggested a more formal interpretation of latent weights based on the field approximation from variational inference [Wainwright et al.) 2008). In this setup, which we review in Section 3 the latent weights are the mean parameters of an exponential family of probability distributions over binary weights. However, none of these works are able to fully justify the use of the STE and weight clipping in latght- methods.
Our contributions In this paper, we provide a general optimization framework for BNN training based on Gaussian variational inference, refinement of the mield approximation used by Ajan than et al. [019] and Meng et al. [2020). Our framework allows us to generalize the notion of latent weights as mean parameters in order to introduce new variables modeling the covariances of the weights. As a result, our optimization formulation is able to capture more intricate dependencies among weights and exploit them to learn better solutions during the training phase. We believe the mere statement of our formulation Section 3.T to be a significant contribution of our work, which will hopefully lead to further study of Gaussian variational methods for neural network training.
2

---

we show that the general case of our framework, i.e., the problem of learning an optimal Gaussian distribution over weights in IR" can be cast non-convex semi-definite program (SDP) over the weights mean vector j E IR and covariance matrix Σ E TR"x.. Because of the large number of weights n in typical applications, we do not attempt to maintain full-rank covariance matrices, but only consider low-rank approximate SDP solutions, following the approach championed by Burer and Monteiro [2005) for linear SDP programs, The resulting algorithm (Algorithm 7 is the main contribution of this paper. We name our method the Variational Inference Semidefinite Programming Algorithm (VISPA).
In Section 3.2
In Section 3.3 we demonstrate that a simpler instantiation of our framework yields non-convex program, whose solution by gradient descent naturally recovers the use of the STE and weight clipping in weights methods. Finally, in Section 4 we present a thorough experimental evaluation of VISPA against large number of BNN training procedures in the literature over four standard benchmark datasets. We find that VISPA almost always improves the state-of-the-art accuracy of BNN training, in some cases dramatically. For instance, Top-1 accuracy with AlexNet on ImageNet with fully binarized weights and activations increases by more than 3% compared to the -fte-of-the-art method (see Table 3). Through an ablation study, we also show that the SDP component of our algorithm is crucial in realizing the observed empirical advantage. Code for our algorithm and vi
experimental evaluation can be found at hittt:s::. h //gg:///bbtt
Limitations and Open Problems We focus our first presentation of VISPA on vision-based applica- tions because of the availability of well-studied binarized architectures and well-establiseed baselines, which isolate the performance of our method more closely. Indeed, in the case of transformers, there is not yet agreement on the best binarized architecture, due to the difficulty of binarizing activations in softmax layers. Only recently, researchers have made progress in bypassing this obstacle He et al.) 2002). This also contributes to the scarcity of baselines and the absence of a standard benchmark. In Section we discuss other limitations of the current work and opportunities for future extensions.
2 Related Work
Courbariaux et al.
[2015a] introduced the use of the STE for training BNNs. Since then, researchers
have put forward many variants to this idea, such as adaptive versions of the STE [Le et al.) 2022 Wu
al.
2023 Qin al. 2023], and a number of extensions, e.g., to non-binary quantization Huh et al.
2023
Liu et al. [2024] and sparsity- network designs |Vanderschueren and De Vleeschouwer
2023
All of these variations are in principle applicable within our optimization framework.
A different line of work investigates alternatives to the STE, with two approaches standing out. Modifications to the gradient estimator include using piecewise polynomial functions (BiRealnet LLiu
ct al. 2018)) and dynamic gradient estimators (IR-Net [Qin et al. 2020). RBNN (Lin et al.) 2020))
The other approach designs separate frameworks for discrete back proopagatiionn (PCNN [uu et al.
2019a]. BiPer BiPer /Vargas et 2024). ReCU [Xu et al. 20. ReActNet (Liu et al. 2020)). ). Among this
latter class, the aforementioned work of Helwegen et al. [2019], followed by several variants (Suarez)
Ramirez et al. 2021 Shan et al., 2023). proposes a novel Binary Optimizer (Bop) that maintains a
binary solution and accumulates gradients to determine when to flip a bit. The only -NNN-iaiiiig
method based on variational inference, BayesBiNN by |Meng et al. [2020]. effectively combines the STE and Bop in a principled way. The work of Ajaanhann et al. 2019]] which is also based on variational inference, only deals with network quantization and does not perform training over BNN architecture. Of particular relevance to this paper is the LNS algorithm Han et al. (2020), who also notice that simply binarizing each latent weight independently does not fully explore the relationship between neurons and may not lead to the optimal solution. They propose to train a custom binarization function via supervision noise learning, but do not explicitly model correlations between weights via new variables. Our experimental evaluation compares our algorithm with all the methods just described, showing the superior accuracy of our technique in practice.
Since the seminal work of Goemans and Williamson [1995], semidefinite programming [aanden berghe and Boyd, [996] has become fundamental tool in the design of approximation algorithms for combinatorial optimization problems. Recently, its application to network quantization has been studied by Bartan and Pilanci (2021]. They construct a tight SDP relaxation for training two-layer quantized neural network. Crucially, their method does not run the training on the quantized archi- tecture, i.c., network gradients are evaluated at non-quantized weight settings, and are limited to
3

---

small-scale, shallow networks. However, we believe their idea provides valuable theoretical evidence in favor of the deployment of SDP techniques a larger scale, as is done our work.
3 Variational Inference Approach to BNN Training
We start by quickly reviewing the variational inference approach to BNN training before introducing our novel contribution in Section 3.1 common approach to the construction of approximation
algorithms for intractable combinatorial optimization tasks [Vazirami, 2010 Barak and Steurer 2024
consider relaxed, regularized formulations over a subset of the simplex A, {p [=1]-
Rzo. Σ f Pú = 11 of probability distributions over the hypercube {=1}":
PPTp = min Ew-p.. [L(((x, w)) yw)l - X H()), pEP
(2)
where i ~ p indicates that the random variable w is distributed according to the distribution and H denotes the entropy function. When the regularization parameter À 2 0 is strictly positive, the regularization term -A И is known to encourage generalization. When P equals the set of all probability distributions over { {+1]", this relaxation renders the loss term linear in the distribution p. but requires an exponential-size representation, hence maintaining the computational hardness of the problem. However, this formulation allows approaches, such as Monte Carlo Markov Chain [Gamerman and Lopes) 2006) and variational inference methods (Wainwright et al. 2008). In partieular, the latter approach suggests restricting the feasible space of Problem 2 to a computationally tractable class of distributions P, such as an exponential family, in order obtain a more compact parametrization of a space of probabilities. We can then attempt to find an approximately optimal solution to the resulting non-convex optimization problem via gradient descent over the distribution parameters.
reason more directly about stochastic
Previous works by Ajanthan et al. [019]] and Meng et al. [2020) provide a more rigorous justification for the STE step by deploying this variational inference blueprint in the form of the well-known mean-field approximation (Friedli and Velenik 2001 |Sayaaa 2015). Specifically, they restrict P to product of [=:1)-Berooull distributions, for each coordmaate, where p. equals the probability that w equals 1. The distribution of the variable w. can then be re-parametrized in terms of its mean fa as, for all
1+ Mi
w Bernoulli(p:). P.
μ1 E
(PBeroulli)
Notice that Problem 2 with P = PBernoulli is still relaxation to the original Problem T as the extreme values of μ yield deterministic weight choices. At this point, Ajanthan et al.) [2019] and Meng et al. (2020) then argue that the vector of mean parameter μ constitutes the right choice of latent weights for the BNN training problem. In this way, the fixed range +1]" of explains the use of weight clipping. However, the main challenge with this approach is the estimation of the gradient V v=luu--ppr)) . [f w). ) of the expected loss with respect to the mean parameters μ. In their case, one can only establish the following general form [Williams 1992]:
V vp (L [L(((x, w). ye = Eww-p(r).
w) ya) y²) log p(w)).
This expression leads estimating the gradient via sampling from p and evaluating the loss function, but fails to take advantage of the differentiability of L and fails to reproduce the STE. Indeed, this sethack forces Meng et al. (2200 to use a smooth proxy to the sign function to recover an approximation of the STE. The work of AAaanthan et al. [2099 only performs binary compression and does not rely on the STE.
3.1 BNN Training via Gaussian Variational Inference
In this section, we describe our novel application of Gaussian variational inference, which has long been recognized the most practical refinement of the mean-field approach [Giordano et al. 2015), to BNN training. Specifically, we consider optimizing Problem 2 over the class Poomr of correlated multivariate Gaussian distributions over IR""" including degenerate Gaussian distributions with rank-deficient covariance matrices:
w ~ ~N(,,, E),
P E 1-,,1|", Σ> 0,
(Poore)
Vi [n], Σ. + 1 =1
4

---

Algorithm BNN Training Algorithm VISPA
Input:
Loss Function (x w)yya): ym), Batch Size M
Parameter: Embedding Dimension K, Weight Mean Vector Initialization E R", Weight Deviation Matrix Z E Ra*k Step Length a, Momentum Coefficient B. Number of Epochs
Output: Learned Weight Mean Vectors IR" and Weight Deviation Matrix Z E B
1: while number of epochs less than T do
2: Sample RR from (0, I);
> Sample rounding vector
3: Sample mini-batch (I1, Yey ....., (a Væu
4: μ Zr:
Sample Gaussian weights
5: i sign(w))
D Round to binary
6:
M
g = M
Va [[ Va.... Va......uuua Va. Nu=ai
( Estimate gradient of vector embedding)
m=
7:
By = 8u + (1 - (1-8)gg
> Update velocity for
8:
Z, 3Z, + (1 (gr
Update velocity for Z
9:
14 apti;
Update weight mean vector with momentum Update weight deviation vector with momentum > Compute normalization factor > Normalize weight mean vector
= aZzv
11:
For all i [n]. Yi =1² (ZZZ)))
12: For all i E [n] pe; = Ji.
13: For all i E [n]:
b. Normalize weight variance vector
14:
end while
15:
return μr, Z
The joint constraints on mean and covariance ensure that the second moments E[w[[] equal 1, matching those of a distribution over [=1]". The resulting non-convex semidefinite program is also a valid relaxation of Problem I as setting Σ yields μa E {+13"
At first, the relaxation of the sample space of Poore from {+1} to R" may seem problematic, as sampling now fails yield the desired binary weights. However, the use of multivariate Gaussians as tractable proxies for the discrete probability distributions in PBrrnoulli has a long history in approxi- mation algorithms, particularly in the context of semidefinite programming relaxations [Gooeman) and Williamson, 1995] Alon and Naor [006]. Indeed, the celebrated Grothendieck 's inequal- ity [Grothendieck] [953] shows that Bernoulll can be effectively relaxed Poorr. with only multiplicative constant loss, when optimizing the expectation of a quadratic polynomial. Unfortu- nately, the corresponding rounding procedure from a sample w ~ P.P E Peorr binary vector vi is fairly complex, as it requires taking large tensor powers of the entries of the covariance of [Alon and Naor 2006). We opt instead for the more straightforward hyperplane rounding [Goemans and Williamson 1995]. which takes the simple form v sign(u w). recovering the standard sign-baus based rounding. In this case, the approximation guarantee only holds for quadratic polynomials with non-negative coefficients. This still provides sufficient theoretical motivation for our method and enables the higher performance of our algorithms, as practical results Section demonstrate.
3.2 Solving the SDP over Low-Rank Covariances
In this subsection, we describe VISPA, our algorithm for solving the SDP formulation of Problem over P = Poorr. Inspired by the previous discussion on hyperplane rounding, we let À go to 0, so that no entropy regularization is performed, but the SDP formulation captures more closely the original Problem 2. It is crucial to notice that we cannot hope to maintain general covariance matrix E x Σ ž 0, as this requires storing 32(n²) matrix entries, which infeasible for the large number of weights (n >> 10") in practical BNN architectures. Fortunately, this is typical issue with large-scale SDPs (Yurtsever al. 2021], which can tackled by restricting our attention to low-rank covariance solutions, as first suggested by Burer and Monteiro [2005). Following this setup, for a rank parameter E N, our algorithm maintains a vector of means l E RR" and a square root Paanaarammnn Z gaxk of the covariance Σ- Z T which is now of rank at most K. We call Z a weight
5

---

deviation matrix, as it describes the typical deviation from the mean μ along the subspace identified by the image of Σ. Pseudocode for the resulting algorithm VISPA given Algorithm [ Here, we immediately see an advantage of the parametrization by the square root Z of E rather than by Σ itself: it facilitates sampling from the underlying Gaussian distribution, as a sample ur can be easily taken by rescaling standard K K-dimensional Gaussian (line 4 of Algorithm 1).
In contrast with classical applications of semidefinite programming, the objective function for the SDP of interest is nonlinear and non-convex, so that we must rely on gradient descent to solve the program to local optimality. The following theorem, proved in the Appendix, shows that the gradients of the expected loss with respect to the parameters 4l and Z take on a particularly simple form, which easy to estimate stochastically. This result, which heavily relies on the Gaussianity of the weights w, implicitly solves the challenge encountered by MMnn et al. [2020] in previous work.
Theorem 1. For a random variable w NNe, ZZ"), with μ E I and Z E F we have:
yn)) Ew, V If w) M=)l: V E [Lf w)
V 2 Er
ym)) = Er, V (f w) Uo )r" www
where r ~ (0,,
The approximation provided by hyperplane rounding justifies replacing with sign(w) in the right-hand side of the gradient expressions in the theorem. The mini-batch stochastic gradient descent step with momentum then takes the form of lines 6-10 in Algorithm | with lines 7-8 regulating the momentum. The moment-matching constraints are enforced by the projection steps of lines 11-13. Finally, at the inference stage, the output mean vector je and deviation matrix Z are used to sample binary weights via sign rounding. In Section 4
we carry out a comprehensive evaluation of the accuracy of Algorithm [ against tfate-of-the-ar methods for BNN training.
Running Time of VISPA As in Algorithm D we let n be the number of weight parameters and K the embedding dimension. Additionally, denote by M the batch size. The running time of one iteration of VISPA is O(Mn + nK), where the first term stems from propagating each example through the neural network and the second term comes from updating the weight mean vector and the weight deviation matrix Z.. In comparison, other ofte-offthheart BNN training approaches typically just require (MM) time. In most cases, we have that K << M, so that the increased cost due to the nK term is negligible fraction of the total running time, most time is spent performing forward- and propaappaantio through the neural network.
Memory consumption of VISPA The main potential limitation of VISPA on resource-constrained devices is the increase in memory usage due having maintain the covariance variable Z. Indeed, VISPA requires n (K + 1) memory for storing relevant variables, compared to just n for other methods. The total significance of this increase depends on the batch size and the size of the examples. For instance, in the case of ResNet 8, there are roughly n = 9 10th weights while, for batch size 100, the total size of data batch is 15- 10S. Hence, standard methods yield total memory usage of 24 10 In contrast, choosing K = 1 in VISPA will lead to a usage of 33 10 37.5% increase. This increase will typically get smaller as we consider models trained on larger images. practice, we often observe even smaller increases, as our estimate does not include memory usage due to back- poce-propagation, which can very large, e-g., in the case of residual connections. mitigation strategy is to store our variable at a lower precision. Given that this variable is only accessed via multiplication with Gaussian noise, we believe that this will not change the behavior of our algorithm.
possible
3.3 Diagonal Covariances and New Interpretation of Latent- Weiehtss Methods
Now, we consider the simple case in which the covariance matrix Σ is multiple o² I of the identity, so that the underlying weights w are independent Gaussian random variables with mean μ. We denote the associated class of distributions by Pindep-
~ NN 1 E 11,
(P (Pindep)

---

By applying Theorem T in conjunction with hyperplane rounding, the mini-batch stochastic gradient descent step for Problem 2 with P Pindep takes the following form:
M
VEN, (8)
(( yarm)
e ((++)) - clip((u -ag ag!")
(3)
M
for choice of step length O, sample mini-batch (I11, May)..... (MM,, Mm and sample w(( from Pindep with mean parameter f,()) The clip function restricts the argument the range [-1, +1] Surprisingly, this derivation recovers both the use of the STE and weight clipping, as well as that of the sign function for rounding. In Section 4.1] we compare the deterministic version of this algorithm
The
(a = 0) with Algorithm I to show the accuracy gain due to the use of the deviation matrix Z.
4 Experiments
Datasets We evaluate the performance of various methods on four datasets: CIFAR- 10 Krizhevsky
et al. 2009]. CIFAR- 100 [Krizhevsky et al. 2009], Tiny-ImageNer (Le and Yang 2015 and Im-
ageNet [Deng et al. 2009]. CIFAR- consists of 50k training samples and 10k testing images with 10 classes, while CIFAR-100 consists of 50k training samples and 10k testing images with 100 non-overlapping classes.
a subset of ImageNet with 100k images and 200 classes. ImageNet contains 1.28 million training samples and 50k testing images for 1000 classes.
Implementation Details We implement VISPA using PyTorch (Paszke et al. 2019 and run on single NVIDIA A100 with 40GB GPU memory per GPU card. On CIFAR-I 10, SIFAR-100 and Tiny-ImageNet, models with only binarized weights (denoted as IW32A) are trained for 500 epochs following (Le et al.) [022), using a batch size of 256, an initial learning rate of 1. and weight decay of 5e 4, with covariance rank K' set to For models with both binarized weights and activations (denoted as 1W IW1A), the training epochs is 600 following [Xu et al. 2021b) the initial learning rate set to 0.5, and the weight decay is reduced to le 5. with K set to We run runs to report the mean and standard deviation. All experiments are conducted on a single GPU card.
On ImageNet, we train AlexNet Krizhevsky et al. 2016) for 200 epochs following Xu el al.) 2021b). with a batch size of 1024, and standard pre- processing with random flips and resize in He et al. 2016 Models with IW32A are trained using an initial learming rate of 0.1, and a weight decay of 5e 5, with K set at 4. For models with IWIA, the initial learning rate is set at 0.5, and the weight decay is reduced le 5, with K set at 2. All experiments are conducted on four GPU cards.
2012) for 100 epochs, and ResNet18 (He et al.)
All runs utilize cosine annealing learning rate schedule with a 5-epoch warm-up optimize training. To accelerate convergence, we employ the momentum technique [Sutskever et al.. 2013), setting the momentum coefficient
0.9. To capture the correlation among weights, at the inference stage, we perform 40 samples and average the results to obtain the final prediction. The mean weights μ and the deviation matrix Z are initialized using a Xavier normal distribution (Glorot and Bengio
2010] with mean of 0 and a standard deviation as * V fin_in-fan_our where fan in is the number of input units, fan_ San_outii is the number of output units and s is the scaling factor. We empirically set 10 for and s 1 for u. For detailed study of the impact of the initialization of Z, please refer to the Appendix A.2
Comparison CIFAR-10, CIFAR-100 and Tmay-Im We evaluate the performance of the IWIA and IW32A settings on CIFAR-10, CIFAR- and Tiny-ImageNet. For the IWIA setting,
we compare various methods, including ReSTE (Wu et al.,
2023] and DIR-Net [Qin al. 2023]
using the commonly employed VGG-Small (Zhang et al.
2018 and ResNet18 [He et al. 2016
architectures in [Qin et al. 2020 Wu et al. 2023 Xu et al.
202Tb Lin al. 2022 Table shows
the result. Our method achieves the highest accuracy with 92.77 92.7% on VGG-Small and matches the top performance of 92. 8% on ResNet18. While our approach only shows marginal improvements over recent methods like ReSTE, the results demonstrates its efficiency in optimizing binarized neural networks. For the IW3A setting, we compare various approaches. including AdaSTE [Le et al. 2022] and BayesBiNN MMeg et al.] 2020]. using the commonly employed VGG16 [Simonyan and Zisserman 2014) and ResNetT8 [He et al., 2016) architectures in [Le et al.] 2022, Ajanthan et al. 2019]. Table 2 presents the result. Our proposed method achieves the highest accuracy across all
7

---

Table 1: Performance comparison by testing aceuracy (%) on CIFAR-10 using VGG-Small and ResNet18 architectures with binarized activations and weights.
Methods
"GG-Small
ResNet18
IR-Net Qin et al. 2020
90.4
91.5
XD-
2022
90.8
92.5
RBNN Lin et al 2020
91.3
92.2
ReCU Xu el
2021b
92.2
92.8
ICRRBNN Shang ci
2022a
91.8
FDA-BNN Xü et al
2021a
92.5
RBNN + CMIM IShang ct al
2022b
92.2
92.8
SiMaN Lin et 2022
92.5
92.5
ReSTE Wu al 202.5
92.6
92.6
DIR Net Qui al,
2023
91.1 0.1
92.8 + 0.1
VISPA (Ours)
92.7 0.1
92.8 + 0.2
Table 2: Performance comparison by testing accuracy (%) of various approaches on CIFAR- 10, CIFAR- 100, and Tiny-Image Anny-mmgeNet across VGG16 and ResNet18 architectures with binarized weights only. (1) indicates that results are obtained from the numbers reported by Ajanthan et al. [202]] indicates that results are obtained from the numbers reported by Le et al. [2022].
CIFAR-100
Tiay-ImageNet
VGG16
ResNet18
VGG16
ResNet18
ResNet18
BinaryC ConaarCCnnec1.io (1) Courbbriaux et
2015a
89.04
91.64
59.13
72.14
49.65
ProxQuant (f)
90.11
92.32
55,10
MDS-sottmaas 11511111 Appmiba et 2021
91.30
93.28
63.97
72.18
51.81
MDS-tanh-s (t) iintnaa
91.53
61.69
52.32
PMF Ajanihhm el 2019
91.40
93.24
64.71
71.56
51.52
BayesBlNNN (MDe
2020
0.07
70.33 +
54.22
AdsTE (+) Le) Le at 2022
92.37 + 0.09
94.11 = 0.08
69.28 1: 0.17
75.03 + 0.35
54.92
(Ours)
95.05
0.10
77.05
datasets and architectures, outperforming existing state-of-the-art techniques. Specifically, on more complex datasets, our method shows significant improvements. For CIFAR-I00, our method reaches 72.09% on VGG16, outperforming the best baseline, AdaSTE, by 2.81 %. For Tiny-ImageNet, our approach achieves 58.5 58.98% ResNet18, which is 4.06% higher than AdaSTE. These results highlight the effectiveness of our method in handling more complex datasets.
Table 3: Performance comparison by testing accuracy various methods on ImageNet dataset AlexNet. W/A denotes the bit-width of weights and activations.
AlexNet
Methods
W/A
Topl (%)
Top5 (%)
BinaryNet Hubara et al.
2016
1/1
41.2
65.6
NeOR-Net Rastegan
2016
1/1
44.2
69.2
Bop Helwegen et 2019
1/1
45.9
70.0
Bop2ndOrder Suarez-Kamirezz et al.
2021
1/1
46.9
70.9
INS Han et al. 2020
1/1
44.4
FDA-BNN Xu el al.
2021a
1/1
46.2
69.7
Quantization networks
Yang al.
2019
1/1
47.9
72.5
BNN-DL Ding et al.
2019
1/1
47.8
71.5
VISPA (Ours)
1/1
51.1
75.0
BinaryConnest Courbariaux al. 2015a
1/32
35.4
61.0
DoReFa Zhu et al. 2016
1/32
$3.9
76.3
XNOR-Net Rastegan et
2016
1/32
56.8
79.4
ADMM Leng et al. 2018
1/32
57.0
79.7
Quantization networks Yang et al
2019
1/32
58.8
81.7
VISPA (Ours)
1/32
59.4
81.1
Comparison on ImageNet To evaluate the performance of our proposed binarized neural network method, we compare list of SOTA methods including ReBNN [Xu et al., [023). ReSTE [Wul et al. 2023), DIR-Net [Qin et al. 2023] and BiPer Vaaraa et al. 2024 on the ImageNet dataset using AlexNet and ResNet18 architectures. For AlexNet, we used the standard architecture without binarizing the first and last layers, adapting it for 1WIA and WW3A configurations. For ResNer 88, we employed the BiRealNet architecture for IWIA, as described in [Xu al., 2022b) Qin al.)
2020). For IW2A, we used the original ResNet18 architecture, following common practices in

---

Table 4: Performance comparison by testing accuracy of optimizers on ImageNet dataset across
ResNet18 architectures. W/A denotes the bit-width of weights and activations.
ResNet18
Methods
W/A
Topl (%)
Top5 (%)
Bop Helwegen
2019
1/1
54.2
77.2
Bi-RealNce et al.
2018
1/1
56.4
79.5
IR-Net Qin ct al.
2020
1/1
58.1
80.0
BONN Gu ct al.
20196
1/1
59.3
81.6
LCR-BNN Shang ct al.
2022a
1/1
59.6
81.6
SiBNN Wang
2020
1/1
59.7
81.8
SiMaN Lin al. 2022
1/1
60.1
82.3
md-tanfis) Ajanthan ct al.
2021
1/1
60.3
82.3
EqualBits et al.
2022
1/1
60.4
82.9
DIR-Net ct
2023
1/1 60.4
81.9
ReSTE Wu
202.1
1/1
60.9
82.6
ReCU Xu et al.
2021b
1/1
61.0
82.6
BiPer Vargas al.
2024
1/1
61.4
83.1
ReBNN Xu et al.
2023
1/1
61.6
83.4
VISPA (Ours)
I/1
62.1
83.4
XNOR-Net Rastegari et al.
2016
1/32
60.8
83.0
HWGQ Cai et al. 2017
1/32
61.3
83.2
ADMM Leng
2018
1/32
64.8
86.2
IR-Net Qun al. 2020
1/32
66.5
86.8
Quantizatron networks
Yang et al.
2019
1/32
66.5
87.3
LCR-BNN Shang ct al.
2022a
1/32
66.9
86.4
ReSTE Wu al.
2023
1/32
67.4
87.2
DIR-Ner(Qin
2023
1/32
67.5
87.9
VISPA (Ours)
1/32
68.2
87.8
Table 5: Performance comparison by lesting accuracy (%) with and without Z across CIFAR- 10, CIFAR-100, and Tmag-ImageNet on VGGI6 and ResNet18.
CIFAR-10
CIFAR-100
Tmag-emageNet
Methods
VGG16
ResNet18
VGG16
ResNet18
ResNet18
VISPA 7. (Ours)
92.85
0.09
95.03
0.08
70.12 = 0.10
76.23
0.36
56.73 + 0.33
VISPA (Ours)
93.25
0.11
95 05 0.10
72.09 0.17
77.05 0.41
58. : 0.28
related literature [Qin et al.
2020 Shang al. 2022a] Additionally, for ResNet1 kept the first,
last, and down-sampling layers in full precision.
Table 3 and Table present a performance comparison of various binarized neural network methods on the ImageNet dataset using AlexNet and ResNet18 architectures separately. For AlexNet, our method achieves ttate-of-the-are performance with Top-1 accuracy of 51.1%, surpassing previous best results from Quantization Networks Yang et al.] [209]] by 3.2% in Top-1 accuracy. Similarly, method outperforms all others with Top-1 accuracy of 59.4% and Top-5 accuracy of 81.1 1.15 1.1%, demonstrating a significant improvement over the next best method, Quantization Networks (Yang et al. 2019).
in the 1 IW32A configuration,
For ResNet18, 8, our method again sets new benchmarks with a Top-1 accuracy of 62.1%, which 0.5% higher in Top-1 accuracy compared to the best previous method, ReBNN [u et al. 20033. In the IW32A configuration, our method achieves the highest Top- accuracy of 68.2% and a Top-5 accuracy of 87.8%. indicating its robustness and superior performance. These results highlight once again the effectiveness of our approach in improving the accuracy of BNNs across different architectures and configurations complex datasets like ImageNet.
4.1 Ablation Studies
investigate the significance of maintaining the correlation between weights, we compare VISPA with the simpler, correlation- algorithm of Equation with on CIFAR-10, CIFAR-100, and Tiny- with the IW32A setting. Table Table provides the experimental results. It can be seen that VISPA consistently performs better across all configurations,
Impact of Deviation Matrix

---

Figure 1: Impact of K model accuracy. The table shows the mean of testing accuracy and standard deviation for different K across models and datasets. Darker colors indicate higher accuracy.
Configurations
K=1
K=2
K=4
K=6
K=8
K=10
ResNet18 CIFAR-10 10 (IWIA)
92.53 0.11
92.62
0.13
92.77
0.17
92.86 0.07
92.83
0.19
93.01 0.07
VGG-Small + CIFAR-10 (1W1A)
92.72 x 0.09
92.73 = 0.13
92.67 = 0.14
92.62 x 0.12
92.55 # 0.05
92.23 x 0.09
ResNetla CIFAR-10 (IW32A)
95.01 0.13
95.03
0.18
95.02 0.10
95.06 0.12
95.05 0.10
95 30.00 0.14
ResNet18 + CIFAR-100 (IW32A)
76.45 = 0.15
76.62 = 0.12
77.04 = 0.34
76.66 = 0.24
77.05 x 0.41
76.87 x 0.40
VGG16 CIFAR-10 (IW32A)
93.02 0.20
93.21
0.14
93.24 0.09
93.34 0.19
93.25 0.11
93.30 0.10
VGG16 + CIFAR-100 (1W32A)
71.09 0.27
71.18 = 0.11
71.83 = 0.32
72.16 0.10
72.09 = 0.17
72.26 0.18
especially on the more complex dataset Tiny-ImageN where the accuracy increases from 56.73 % to 58.89% for ResNet18. This suggests that the deviation matrix Z might be particularly useful in scenarios with higher number of classes and potentially more complex patterns.
Impact of Covariance Rank K We investigate the impact of the covariance rank K on the accuracy of various datasets and models. Experiments are conducted on CIFAR-10 10 and CIFAR- 100 datasets using configurations of WW3A and IWIA A across architectures including VGG16, VGG-Small, and ResNet1 8, by setting different values of K and performing 5 runs. Figure 1 presents the results. The data reveals that ResNet1 8 and VGG16 generally benefit from increasing K values, with ResNet18 + CIFAR-10 (IWIA IA) and VGG16 CIFAR- 100 (IW32A) peaking at K = 10. The accuracy of R on CIFAR-10 10 (IWIA) increases with higher K values, from 92.53 + 0.11% at K = to 93.01 + 0.07% at K = 10, indicating improved performance and stability. Conversely, VGG- Small on CIFAR- 10 (IW1A) experiences a slight decline in accuracy with higher K. ResNet18 on CIFAR- 100 (1 W322) shows the most variability, with accuracy peaking at 77.05 + 0.41% for K = 8, suggesting an unpredictable impact of K. These results highlight that higher values generally improve accuracy and stability for most models, particularly for ResNet18 and VGG16, but the benefits may vary depending on the specific model and dataset configuration. We have two hypotheses for this lack of conclusive evidence: i) higher values of K yield larger feasible space and require a longer time to converge; ii) higher values of yield a more complex distribution from which it is more expensive to sample good weights at the inference stage.
5 Limitations and Open Problems
Our contributions naturally open a number of directions for further research. On the theoretical side, it would be interesting to efficiently implement the rounding suggested by Grothendieck's inequality. Similarly, we believe that the gradient descent approach of Burer-Monteing can be formally analyzed as gradient flow over the Bures- wasserseisn (BW) manifold of Gaussian distributions, which has recently been applied successfully in the context of variational inference [Lambert et al. [022]] to show that our method, with proper choice of step size, converges to a stationary point of Problem in the BW geometry.
On the experimental side, the inference stage, our method currently draws and rounds 40 samples from the computed Gaussian distribution and averages the results. It is an active area of focus to further reduce the impact of this procedure inference time. In preliminary results, find that the process may be sped by using a smaller number of correlated samples via Gaussian quadrature. In this case, 2K + 1 samples would suffice, which is as small as 3 for the K version of our method. Here we note that the prime runtime concern is to reduce the cost of training, which we achieve by binarizing both weights and activations while outperforming competitors.
Finally, we believe that the application of our method to the binarization of transformer architec- tures [He et al. 2023 Zhang et al. 2024] and to the general quantization settings, where the weights can take on more than two values, could have substantial practical consequences.
Acknowledgments
This research is supported by the Agency for Science, Technology, and Research (A*STAR) under its MTC Programmatic Fund M2317b0021. And this work partially supported by the National Natural Science Foundation of China under Grant (Nos. 62272093, 62137001).
10

---

References
Thalaiyasingam Ajanthan, Puneet Dokania, Richard Hartley, and Philip Torr. Proximal Mean-Field for Neural Network Quantization. In 2019 IEEE/CVF International Conference Computer Vision (ICCV), pages 4870-4879, October 2019.
Thalaiyasingam Ajanthan, Kartik Gupta, Philip Torr, Richad Hartley, and Puneet Dokania. Mirror descent view for neural network quantization. In International conference on artificial intelligence and statistics, pages 2809-2817. PMLR, 2021.
Milad Alizadeh, Javier Fernández- Marqueé,, Nicholas Lane, and Yarin Gal. An Empirical study of Binary Neural Networks' Optimisation. In International Conference on Learning Representations, September 2018
Noga Alon and Assaf Naor. Approximating the Cut-Norm via Grothendieck's Inequality. SIAM
Journal on Computing, 35(4):787-803, January 2006. ISSN 0097-5397. doi: 10.1137/
50097539704441629,
Alexander G. Anderson and Cory P. Berg. The High-Dimensional Geometry of Binary Neural Networks, May 2017.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 11112244-14660 2020.
Yu Bai, Yu-Xiang Wang, and Edo Liberty. Proxquant: Quantized neural networks via proximal operators, arXiv preprint arXiv:1810.0086: 2018.
Boaz Barak and David Steurer. Proofs, beliefs and algorithms through the lens of Sum of Squares. https//uwww..rr.rooiii//eedeeeee html, 2024.
Burak Bartan and Mert Pilanci. Training Quantized Neural Networks Global Optimality via Semidefinite Programming. In Proceedings of the 38th International Conference on Machine Learning, pages 694-704 PMLR, July 2021.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint /308.334 2013.
Samuel Burer and Renato D.C. Monteiro. Local Minima and Convergence in Low-Rank Semidefinite Programming. Mathematical Programming. 103(3):427-444, July 2005. ISSN 1436-4646. doi:
10.10077/10100--0444556---
Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half-wave gaussian quantization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5918-5926, 2017.
Matthieu Courbariaux, Yoshua Bengio, and Jean- -Perre David. Binaryconnect: Training deep neural networks with binary weights during propagations. Advances in neural information processing systems, 28, 5a.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Training deep neural networks with low precision multiplications, September 2015b.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran EI- Vani,, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ or-1, arXiv preprint arXiv: 0600..022....0 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scall hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. lece, 2009.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805. 2018.
11

---

Ruizhou Ding, Ting- Wu Chin, Zeye Liu, and Diana Marculescu. Regularizing activation distribution for training binarized deep networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11408-11417, 2019.
Sacha Friedli and Yvan Velenik. Statistical Mechanics of Lattice Systems: Concrete Mathematical Introduction, Cambridge University Press, 1 edition, November 2017. ISBN 978-1-107-18482-44
978-1-316-88260-3 doi: 10. 10.1017/97111168222
Dani Gamerman and Hedibert Lopes. Markov Chain Monte Carlo. Chapman and Hal//CRC, Boca Raton, 2nd edition edition, May 2006. ISBN 778-1-58488-587-4
Ryan Giordano, Tamara Broderick, and Michael Jordan. Linear response methods for accurate covariance estimates from mean field variational bayes. Advances in neural information processing systems, 28, 2015.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages
JMLR Workshop and Conference Proceedings, 2010.
Michel X Goemans and David P Williamson. Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of the ACM (JACM), 42(6): 1115-1145, 1995.
A. Grothendieck. Résumé de la théorie métrique des produits tensoriels topologiques. Bol. Soc. Mat. São Paulo, 8:1-79, 1953.
Jiaxin Gu, Ce Li, Baochang Zhang, Jungong Han, Xianbin Cao, Jianzhuang Liu, and David Doer- mann. Projection convolutional neural networks for 1-bit cans via discrete back propagation. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 8344-8351, 2019a.
Jiaxin Gu, Junhe Zhao, Xiaolong Jiang, Baochang Zhang, Jianzhuang Liu, Guodong Guo, and Rongrong Ji. Bayesian optimized 1-bit cnns. In Proceedings of the IEEE/CVF international conference on computer vision, pages -900-49111 2019b.
Kai Han, Yunhe Wang, Yixing Xu, Chunjing Xu, Enhua Wu, and Chang Xu. Training binary neural networks through learning with noisy supervision. In International conference on machine learning, pages 4017-4026. PMLR, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
Yefei He, Zhenyu Lou, Luoming Zhang, Jing Liu, Weijia Wu, Hong Zhou, and Bohan Zhuang. Bivit: Extremely compressed binary vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6651-5663. 2023.
Koen Helwegen, James Widdicombe, Lukas Geiger, Zechun Liu, Kwang-Ting Cheng, and Roe- land Nusselder. Latent weights do not exist: Rethinking binarized neural network optimization. Advances in neural information processing systems, 32, 2019.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran Yaniv, and Yoshua Bengio. Binarized Neural Networks. In Advances in Neural Information Processing Systems, volume 29, 2016.
Minyoung Huh, Brian Cheung, Pulkit Agrawal, and Phillip Isola. Straightening out the straight- through estimator: Overcoming optimization challenges in vector quantized networks. In Interna- tional Conference on Machine Learning, pages 14096- 14113. PMLR, 2023.
Minje Kim and Paris Smaragdis. Bitwise neural networks. arXiv preprint arXiv: arXiv:160I.nm 1601.0007,, 2016.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
12

---

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 25, 2012.
Mare Lambert, Sinho Chewi, Francis Bach, Silvère Bonnabel, and Philippe Rigollet. Variational inference via wasserstein gradient flows. Advances in Neural Information Processing Systems, 2022.
Huu Le, Rasmus Kjær Høier, Che-Tsung Lin, and Christopher Zach. Adaste: An adaptive straight- through estimator to train binary neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 460 469, 2022.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7):3, 2015.
Cong Leng, Zesheng Dou, Hao Li, Shenghuo Zhu, and Rong Jin. Extremely low bit neural net- work: Squeeze the last bit out with admm. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Yunqiang Li, Silvia-L Laura Pintea, and Jan van Gemert. Equal bits: Enforcing equally distributed binary network weights. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 1491-1499, 2022.
Mingbao Lin, Rongrong Ji, Zihan Xu, Baochang Zhang, Yan Wang, Yongjian Wu, Feiyue Huang, and Chia- Wen Lin. Rotated binary neural network. Advances in neural information processing systems, 33:7474-7485, 2020.
Mingbao Lin, Rongrong Ji, Zihan Xu, Baochang Zhang, Fei Chao, Chia-Wen Lin, and Ling Shao. Siman: Sign-to-magnitude network binarization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):6277-6288. 2022.
Liyuan Liu, Chengyu Dong, Xiaodong Liu, Bin Yu, and Jianfeng Gao. Bridging discrete and backpropagation: Straight-through and beyond. Advances in Neural Information Processing Systems, 36, 2024.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cnns with improved representational capability and advanced training algorithm. In Proceedings of the European conference on computer vision (ECCV), pages 722-737, 2018.
Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang- Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. In Computer EssionEECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 143-159. Springer, 2020.
Brais Martinez, Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos. Training binary neural networks with real-to-binary convolutions. arXiv preprint arXiv:2003.11535. 2020.
Xiangming Meng, Roman Bachmann, and Mohammad Emtiyaz Khan. Training Binary Neural Networks using the Bayesian Learning Rule. In Proceedings of the 37th International Conference on Machine Learning, pages 6852-6861. PMLR, November 2020.
Paul Merolla, Rathinakumar Appuswamy, John Arthur, Steve Esser. and Dharmendra Modha. Deep neural networks are robust to weight binarization and other non-linear distortions, June 2016.
Lorenz K. Muller and Giacomo Indiveri. Rounding Methods for Neural Networks with Low Resolu- tion Synaptic Weights, April 2015.
Adam Paszke, Sam Gross. Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, nigh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 2250- 2250-2259, 2020.
13

---

Haotong Qin, Xiangguo Zhang, Ruihao Gong, Yifu Ding, Yi Xu, and Xianglong Liu. Distribution- sensitive information retention for accurate binary neural network. International Journal of Computer Vision, 131(1):26-47) 2023.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks, August 2016.
Hiroki Sayama. Introduction to the Modeling and Analysis of Complex Systems. Open SUNY Textbooks. Published by Open SUNY Textbooks, Milne Library, State University of New York Geneseo, Geneseo, NY, 2015. ISBN 978-1-942341-06
Gu Shan, Zhang Guoyin, Jia Chengwei, and Wu Yanxia. Sgdat: An optimization method for binary neural networks. Neurocomputingg page 126431, 2023.
Yuzhang Shang, Dan Xu, Bin Duan, Ziliang Zong, Liqiang Nie, and Yan Yan. Lipschitz continuity retained binary neural network. In European conference computer vision, pages 603-619 Springer, 2022a.
Yuzhang Shang, Dan Xu, Ziliang Zong, Liqiang Nie, and Yan Yan. Network binarization via contrastive learning. In European Conference on Computer Vision, pages 586-602. Springer, 2022b.
Mingzhu Shen, Xianglong Liu, Ruihao Gong, and Kai Han. Balanced binary neural networks with gated residual. In ICASSP 2020- 2020 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pages 4197-4201. IEEE, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Cuauhtemoc Daniel Ramirez, Miguel Gonzalez-Mendozi Leonardo Chang, Gilberto Ochoa- Vega. A bop and beyond: second order optimizer for binarized
Ruiz, and Mario Alberto Duran-Vega. neural networks. In Proceedings of the IEÉE/CVF Conference on Computer Vision and Pattern Recognition, pages 1273-1281, 2021.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance initialization and momentum in deep learning. In International conference on machine learning, pages 1 1139- 1147. PMLR, 2013.
Vivienne Sze, Yu-Hsin Chen, Tien-Ju Yang, and Joel S Emer. Efficient processing of deep neural networks: A tutorial and survey. Proceedings of the IEEE, 105(12):229--2329 2017.
Wei Tang, Gang Hua, and Liang Wang, How to train compact binary neural network with high accuracy? In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Neil C. Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel F. Manso. The Computational Limits of Deep Learning, July 2022.
Yaman Umuroglu, Nicholas J Fraser, Giulio Gambardella, Michaela Blott, Philip Leong, Magnus Jahre, and Kees Vissers. Finn: A framework for fast, scalable binarized neural network inference. In Proceedings of the 2017 ACM/SIGDA international symposium on field prograaagaee gate arrays, pages 65-74, 2017.
Lieven Vandenberghe and Stephen Boyd. Semidefinite programming. SIAM Review, 38(1):49-95. 1996. doi: 10. 10.1137/103800.
Antoine Vanderschueren and Christophe De Vleeschouwer. Are straight-through gradients and soft- thresholding all you need for sparse training? In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3808-3817, 2023.
Edwin Vargas, Claudia Correa, Carlos Hinojosa, and Henry Arguello. Biper: Binary neural networks using a periodic function. arXiv preprint arXiv:2404.0 0.20..... 2024.
Vijay V. V. Vazirani. Approximation Algorithms. Springer, Berlin Heidelberg, December 2010. ISBN 978-3-642-08466--0
14

---

Martin Wainwright, Michael I Jordan, et al. Graphical models, exponential families, and variational inference. Foundations and Trends@ in Machine Learning. 1(1-2):1-305, 2008.
Peisong Wang, Xiangyu He, Gang Li, Tianli Zhao, and Jian Cheng. Sparsity-inducing binarized neural networks. In Proceedings of the AAAI conference artificial intelligence, volume 34, pages 12192-12199, 2020.
Ronald J. Williams.
Simple statistical gradient-following algorithms for connectionist rein- forcement learning. Machine Learning, 8(3):229 May 1992. ISSN 1573-0565. doi:
10.1007/BF00992666
Xiao-Ming Wu, Dian Zheng, Zuhao Liu, and Wei-Shi Zheng. Estimator meets equilibrium perspective: A rectified straight through estimator for binary neural networks training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17055-17064 2023.
Sheng Xu, Yanjing Li, Teli Ma, Minghao Lin, Hao Dong, Baochang Zhang, Peng Gao, and Jinhu Lu. Resilient binary neural network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 10620-10628, 2023.
Yixing Xu, Kai Han, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang. Learning frequency domain approximation for binary neural networks. Advances in Neural Information Processing Systems, 34:25553- 25565, 2021a.
Zihan Xu, Mingbao Lin, Jianzhuang Liu, Jie Chen, Ling Shao, Yue Gao, Yonghong Tian, and Rongrong Ji. Recu: Reviving the dead weights in binary neural networks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5198-5208, 2021b.
Ping Xue, Yang Lu, Jingfei Chang, Xing Wei, and Zhen Wei. Self-distribution binary neural networks.
Applied Intelligence, 13870 2022.
Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and Xian-sheng Hua. Quantization networks. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 7308-7316, 2019.
Penghang Yin, Jiancheng Lyu, Shuai Zhang, Stanley Osher, Yingyong Qi, and Jack Xin. Under- standing straight-through estimator in training activation quantized neural nets. arXiv preprint arXiv: 1903.00900000 2019.
Alp Yurtsever, Joel A. Tropp. Olivier Fercoq, Madeleine Udell, and Volkan Cevher. Scalable Semidefinite Programming, SIAM Journal on Mathematics of Data Science, 3(1):171-200, January 2021. doi: 10.1137/1M1105004
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European conference on computer vision (ECCV), pages 365-382, 2018.
Yichi Zhang, Ankush Garg, Yuan Cao, Lukasz Lew, Behrooz Ghorbani, Zhiru Zhang, and Orhan Firat. Binarized neural machine translation. Advances in Neural Information Processing Systems, 36, 2024.
Chenzhuo Zhu, Song Han, Huizi Mao, and William Dally. Trained ternary quantization. arXiv preprint arXiv: 2016.
15

---

A Appendix
A.1 Proofs
Proof of Theorem Both proofs are based on the change of variable:
w μl + Zr
where r (0, Ik). For the gradient with respect the mean vector μa, have:
V We (Lf( w) ye)) Ex / (2T) (27)-K/2 V VuL(f(Z, Zr), yo) 2/2
Ew. Lf w) Uz))
For the gradient with respect the deviation matrix Z, have:
Vz Vz [Lf((x w) Uo)) Es
(2T) V2L(f(2, Zr), yu) e- 1r13
RKK
(2T) K/2 1 WL(L(f( w), yer) rd w= 2+r e /21111//dr
Eg
Br
V (
yo vr"" 1
A.2 Impact of the Initialization of Z on the Performance
We draw Z from the Xavier Normal Initialization GGooot and Bengio 2010 using normal distribu- tion with mean of 0 and a standard deviation given by
2
(4)
fan in + fan_out
where fan in is the number of input units, fan_ou is the number of output units and is the scaling factor. To investigate the impact of to the accuracy, we conduct experiments on CIFAR- 10 and CIFAR-I00 dataset with the commonly deployed architectures VGG-17 and ResNet18 for the IW32A setting by setting the covariance rank K at 8 with various values, specifically at 5, 10, 15, 20, 25 by five runs. Figure shows the result. We observe that the accuracy of the models typically peaks at = 5 or s = 10. Besides, the standard deviation is relatively low across all values, indicating consistent performance. Finally, we set 10 across all experiments in Section
Figure 2: Impact of the initailization of Z model accuracy in terms of the hyper-parament The table shows the mean accuracy and standard deviation for different values across models and datasets. Darker colors indicate higher accuracy values. To help reviewers easily see differences and peak values, we prefer visualizing in table rather than in figure. On small datasets like CIFAR, visualizing differences figures is difficult due minimal differences. For example, the experimental results of CIFAR-10 10 + VGG-16 (I W32A) only has 0.1 % difference between = 5 and s = 25.
Configurations
s=1
==5
s=10
1=15
$=20
$=25
VGG-16 + CI 11000001000 (I322A)
1,48 + 0.08
72.27 = 0.17
72.09
0.17
71.96 $
0.14
71.62
0.29 71.28 + 0.22
ResNet18 + CIFAR-100 (1W32A)
70.93 = 0.14
76.64 = 0.15
77.05 = 0.4
16.92
76.40
0.42
76.63
2 0.25
VGG-16 + CIFAR-10 (1W32A)
87.94 = 0.21
93.23 = 0.09
93.25 0.1
93.23
0.00
93.22
= 0.08
93.13
# 0.08
ResNet18 CIFAR-10 (1W32A)
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and the "Our contributions" paragraph of the introduction clearly state the claims of the paper and the context to which they apply.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: These are discussed in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes].
17

---

Justification:
Proof' appear in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We have provided all implementation details in Section 4 We have released a limited version of the code in the supplementary material, including training code and pre-trained models. A full version will be released upon acceptance.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: For reviewers verify our results, we have released a limited version of the code in the supplementary material, including training code and pre-trained models. A full version will be released upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
mublic/guides PoodeSubmissioooolieee
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/pppp wttpccccccpccpcdeeeeeeeeceepeeee
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We have provided all implementation details in Section 4 The full code for our algorithm and the experimental evaluation will be released upon acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For the small datasets CIFAR-10. 10, CIFAR-100, and Liny-ImageNet con- ducted runs to report the mean and standard deviation. For the very large-scale dataset ImageNet, following common practice in the literature, we performed single run report the result.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have included all pertinent information on the computer resources in the experimental section, Section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics / httpa/rrwppp.....aaaacccccccctea
Answer: [Yes]
Justification:
We follow the NeurIPS code Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work consists of foundational research with no direct societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper focuses on fundamental research without posing such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have cited the original papers that produced the codes.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "Mm54A1aXnC", "reviewer": "Reviewer_Bkmh", "summary": "This paper presents an optimization framework for Binarized Neural Network (BNN) training using Gaussian variational inference, resulting in a low-rank semidefinite programming (SDP) formulation. The authors propose the Variational Inference Semidefinite Programming Algorithm (VISPA) to improve accuracy by modeling and learning pairwise correlations between weights. The empirical evaluation demonstrates that VISPA outperforms state-of-the-art algorithms on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet datasets.", "questions": "Can you provide more details on the computational complexity and runtime of VISPA compared to other state-of-the-art BNN training methods?\n\nHow does VISPA perform on other types of neural networks, such as transformer architectures? \nCan you elaborate on the initialization strategy for the weight deviation matrix 𝑍 and its impact on the training stability and final performance?\n\nWhat are the potential limitations of the proposed method when applied to resource-constrained devices, and how can these be mitigated?\n\nCould you provide more insights into the impact of different hyperparameter settings, especially the covariance rank 𝐾, on the performance of VISPA?", "limitations": "N.A.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper introduces a novel optimization framework based on Gaussian variational inference for BNN training, which goes beyond latent weights and leverages low-rank semidefinite programming relaxations.\n\nThe paper provides a theoretical motivation for the use of latent weights, STE, and weight clipping. It also presents a new interpretation of latent weights methods within the proposed optimization framework.\n\nThe empirical evaluation on multiple benchmark datasets shows consistent and significant performance improvements over existing state-of-the-art BNN training methods.", "weaknesses": "While the method shows excellent results on the tested datasets, the paper could benefit from a discussion on the generalization capabilities of the proposed approach to other types of neural networks or applications.\n\nThe performance of VISPA may be sensitive to the choice of hyperparameters, such as the covariance rank \n𝐾, which is not extensively explored in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their assessment and questions. We respond to each point inline below. We encourage the reviewer to ask any follow-up question and to comment on our responses. We hope that they will consider raising their score at the end of our exchange.\n\n>  How does VISPA perform on other types of neural networks, such as transformer architectures? \n\n__See FAQ 1 in the author rebuttal.__ We will include this discussion in Section 5 (Limitations and Open Problems) in the final version. \n\nWe have performed __preliminary experiments of VISPA on the transformer architecture Swin-T[3] using ImageNet__. We binarized the weights of the linear layers except for the qkv layer before the softmax. The experiment was performed with a learning rate of 0.1, a weight decay of 1e-5, and an epoch size of 150, without fine-grained hyper-parameter tuning. Results are shown in the __table within the PDF attached to the author's rebuttal__.\n\nWe have identified 3 methods to which to compare VISPA's performance : __BiViT[1], BiT[2] and BiBERT[4]__. As summarized in the table, such methods differ in fundamental ways from our application on VISPA, making an immediate comparison somewhat tenuous. All of them use Knowledge Distillation to enhance the performance of binarized transformer architectures, putting our method at a disadvantage. Furthermore, our strongest competitor BiViT uses a special handling of the softmax operation, facilitating its task. On the other hand, our implementation of VISPA does not binarize QKV layers, gaining an advantage there. With these caveats, we find that __our method outperforms all competitors in terms of top-1% test accuracy, sometimes decisively__. We believe that this result strongly suggests that __the performance improvements, shown in this paper for vision-based transforms, will translate successfully to transformer__. A more detailed study would involve different architectural choices on the binarized transformer, which are beyond the scope of our submission.\n\n[1] He, Y., et al. \"BiViT: Extremely Compressed Binary Vision Transformers.\" ICCV, 2023.  \n[2] Liu, Z., et al. \"BiT: Robustly Binarized Multi-distilled Transformer.\" NIPS, 2022.  \n[3] Liu, Z., et al. \"Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.\" ICCV, 2021.  \n[4] Qin, H., et al. BiBERT: Accurate Fully Binarized BERT. ICLR, 2022.  \n\n> Could you provide more insights into the impact of different hyperparameter settings, especially the covariance rank 𝐾, on the performance of VISPA?\n\n__See FAQ 2 in the author rebuttal.__\n\n> Can you provide more details on the computational complexity and runtime of VISPA compared to other state-of-the-art BNN training methods?\n\nAs shown in Algorithm 1, let $M$ represent the batch size, $n$ the number of weight parameters, and $K$ the embedding dimension. The computational complexity of the VISPA approach is $O(Mn+nK)$. In comparison, other state-of-the-art BNN training approaches typically require $O(Mn)$. In most cases $M >> K,$ so that the increased cost due to the nK term is a negligible fraction of the total running time, as __most time is spent performing forward- and back-propagation through the neural network.__ \n\n> Can you elaborate on the initialization strategy for the weight deviation matrix 𝑍 and its impact on the training stability and final performance?\n\nIn Section 4 (Experiments), we discussed that the weight deviation matrix $Z$ is initialized using the Xavier normal distribution, with a mean of 0 and a standard deviation of $s * \\sqrt{\\frac{2}{\\text{fan\\\\_in} + \\text{fan\\\\_out}}}$. Here, fan_in represents the number of input units, fan_out the number of output units, and s is a scaling factor parameter. Xavier initialization has been empirically shown to work well in practice across a variety of neural network architectures (e.g., CNNs, ResNets, GANs, and Transformers). It is based on a theoretical analysis that ensures the variance of the input to each neuron is similar to the variance of the output. This choice balances the scale of weights and helps maintain stable gradient magnitudes [1]. \nFor details on the impact of $Z$'s initialization on training stability and performance, see Appendix A.2. There we show that VISPA achieves __consistent performance across a range of choices of scaling factor s.__\n\n[1] Xavier Glorot and Yoshua Bengio, Understanding the difficulty of training deep feedforward neural networks, AISTATS, 2010. \n\n> What are the potential limitations of the proposed method when applied to resource-constrained devices, and how can these be mitigated?\n\nThe main potential limitation of VISPA on resource-constrained device is the increase in memory usage due to having to maintain the covariance variable $Z.$ If $n$ is the number of weights, our memory  for storing relevant variables is at most $(K+1) n$ compared to $n$ for other methods. The total significance of this increase depends on the batch size and the size of the examples. For instance, in the case of ResNet18, there are roughly $9 \\cdot 10^6$ weights while, for batch size=100, the total size of a data batch is $15 \\cdot 10^6$, for a total memory usage of $24 \\cdot 10^6$. Hence, choosing K=1 will lead to a usage of $33 \\cdot 10^6$, a $37.5\\\\%$ increase. __This increase will typically get smaller as we consider models trained on larger images.__ In practice, we often observe even smaller increases, as our estimate does not include memory usage due to  backpropagation, which can be very large, e.g., in the case of residual connections.\n\nA possible mitigation strategy is to store our variable $Z$ at lower precision. Given that this variable is only accessed via multiplication with Gaussian noise, we believe that this will not change the behavior of our algorithm.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4lDprWH5HK", "reviewer": "Reviewer_WzdJ", "summary": "The paper provides a theoretical framework for BNN training (a longstanding model compression and computational speed up technique) based on Gaussian variational inference. The framework succeeds in providing theoretical grounding for practical and well established techniques for BNN training - specifically Straight Through Estimators (STEs), weight clipping and usage of sign function as special case of their formulation. Inspired by previous work by Ajanathan et al. (2019) and Meng et al. (2020) which interpret latent weights as mean field parameters, their framework allows one to model pairwise correlations between latent weights by using low rank Gaussian distributions over weights leading to a more accurate optimization problem (expressible as a SDP according to the authors). They also use sensible relaxations for tractability (Burer-Monteiro approach), leaving the door open to more sophisticated techniques (gradient flow over Bures-Wasserstein manifolds) for future work.", "questions": "- In L242-243, the authors mention taking 40 samples and average results for predictions. How much does this impact performance (aka inference time, storage etc), since one of the primary motivations of BNNs is to minimize computational storage and runtime requirements\n- It is well established in quantization literature that activations (and weights) both admit low precision representations without significant performance loss. Did you try activation precision below 32 bits since low precision architectures are quite common (and perhaps even the norm)?\n- Do you have a hypothesis for why lower K perform better for 1W1A while higher K is better for 1W32A architectures?", "limitations": "There are no major limitations", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- [Clarity, Quality] The authors contextualize their work well with respect to the past work on the topic, and the writing is clear and succinct.\n- [Originality] The paper presents a novel modelling approach to training BNNs by combining and extending several well established methods (use of gaussian variation inference as a proxy for bernoulli distributions, SDP relaxation of a combinatorial problem etc) allowing them to leverage well studied and optimized methods and numerical routines for the intractable BNN training problem (Problem (1) in the paper).\n- Their relaxation of the sample space to real numbers and usage of gaussian weights facilitates the usage of gradient descent based approaches for optimization (a challenge in earlier works where gradient based approaches use various heuristics since the objective function doesn't admit a tractable/sensible gradient)\n- The authors perform ablation to elucidate the impact of the main hyperparameters ($Z$ and $K$), as well as empirical evaluation on notable configurations of datasets and architectures\n- [Significance] The paper ties together several methods BNN optimization methods in a unifying framework, and presents an interesting relaxation which yields concrete improvement over state of the art results", "weaknesses": "- The paper can benefit from empirical evaluation over a broader class of architectures since BNNs can have applicability beyond vision as a domain. I do acknowledge that the domain is typically restricted to the class of vision based transforms, but leveraging these techniques in other major model classes of interest can have significant impact which is left unexplored.\n- The authors claim that the problem involves an SDP but the algorithm presented has very loose connections to most formalizations of semi definite programming (and variational inference). The SDP claim seems to derive only from a semi definite constraint and the problem itself fails to be convex let alone semidefinite. (See Line 163-164).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their assessment and questions. We respond to each point inline below. We encourage the reviewer to ask any follow-up question and to comment on our responses. We hope that they will consider raising their score at the end of our exchange.\n\n> The paper can benefit from empirical evaluation over a broader class of architectures since BNNs can have applicability beyond vision as a domain. I do acknowledge that the domain is typically restricted to the class of vision based transforms, but leveraging these techniques in other major model classes of interest can have significant impact which is left unexplored.\n\n__See FAQ 1 in the author rebuttal.__\n\n> The authors claim that the problem involves an SDP but the algorithm presented has very loose connections to most formalizations of semi definite programming (and variational inference). The SDP claim seems to derive only from a semi definite constraint and the problem itself fails to be convex let alone semidefinite. (See Line 163-164).\n\n__In the author rebuttal, please see FAQ 3__ for the connection to variational inference and __FAQ 4__ for the connection to semidefinite programming.  \n\n> In L242-243, the authors mention taking 40 samples and average results for predictions. How much does this impact performance (aka inference time, storage etc), since one of the primary motivations of BNNs is to minimize computational storage and runtime requirements?\n\nThis has limited impact on storage, as the algorithm can maintain the average dynamically. The impact on inference time may be as large as 40x, but may be significantly reduced by parallelization of the processing of the 40 samples. __It is an active area of focus to further reduce this impact__. In preliminary results, we find that the process may be sped by using a smaller number of correlated samples via Gaussian quadrature. In this case, $2K +1$ samples would suffice, which is as small as 3 for the $K=1$ version of our method. In conclusion, we note that __the prime runtime concern is to reduce the cost of training__, which we achieve by binarizing both weights and activations while outperforming competitors.\n\n> It is well established in quantization literature that activations (and weights) both admit low precision representations without significant performance loss. Did you try activation precision below 32 bits since low precision architectures are quite common (and perhaps even the norm)?\n\n__We conducted experiments using binary activation precision__. The results of these experiments are detailed in Tables 1, 3, and 4, which show the performance under the settings of binary activation and binary weights across different datasets and network architectures. For future work, we plan to explore other low precision configurations, including 2-bit, 3-bit, and 4-bit precision.\n\n> Do you have a hypothesis for why lower K perform better for 1W1A while higher K is better for 1W32A architectures?\n\n__See FAQ 2 in the author rebuttal.__", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ez0YIX48UL", "reviewer": "Reviewer_M9aW", "summary": "This paper aims to propose a new method training binary neural networks.\nThe method, according to the authors claims, is based on variational inference and semi-definite programming.\nDuring training, the method maintains a low-rank Gaussian distribution from which the neural network parameters are drawn.\nThe training objective is the expected loss over the Gaussian distribution.\nDuring test, the method obtains a binary neural network by rounding a random sample drawn from the Gaussian distribution.\nExperiments on CIFAR-10, CIFAR-100, Tiny-ImageNet and ImageNet demonstrate the competitiveness of the proposed method.", "questions": "NA.", "limitations": "NA.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The accuracy reported by the authors are competitive against the baselines.\n- The experiments are quite extensive including a wide range of datasets.", "weaknesses": "- Respectfully, this paper has little to do with variational inference nor semidefinite programming, contrary to what the title, the abstract, the introduction, and the related work suggest. Thus, a large fraction of the paper should be revised.\n    1. Just because you minimize the objective in Eq (2) over a Gaussian family does not imply this is variational inference.\n    If the authors disagree, please explain the posterior that the objective aims to approximate.\n    In fact, minimizing Eq (2) over a Gaussian family will collapse to a degenerate Gaussian with a zero covarianc.e\n    That is exactly why the authors have to add a moment constraint on the mean vector and the covariance matrix to prevent the collapse.\n    1. Just because you have a positive semi-definite constraint on the covariance matrix does not imply that this is a semi-definite program.\n    This formulation is not even remotely close to semi-definite programs.\n    Moreover, the algorithm proposed by the authors is not even remotely close to any prominent techniques in semi-definite programming.\n    The algorithm is basically stochastic gradient descent with Polyak's momentum and a projection step to an Euclidean sphere.\n\n- The memory consumption of this method is not reported, and I suspect that this method is way more memory-intensive than baselines.\nNote that storing a rank-$K$ covariance matrix increases the memory by a factor of $K + 1$ (storing the mean $\\mathbf{mu}$ and a low-rank matrix $\\Zv$ with $K$ columns.\nThe main results are reported using $K = 8, 4, 2$ depending on the datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer and regret to find that the connections to variational inference and semidefinite programming were not clear to them. We believe that, in the attempt to streamline the presentation of the mathematical component of our method, we may have omitted some steps that clarify such connections. __A summary of our clarifications are given in FAQ 3 and FAQ 4 of the author rebuttal__. Below, we aim to present more details of our clarifications, together with their proposed location in the  final version of the paper. Lastly, we note that the closely related works by __Ajanthan et al [1] and Meng et al [2] (see Section 3.1) also rely on the same connection to variational inference__. More recently, the paper by Shen et al (pointed out by Reviewer 6N2m) [3] describes a similar framework to ours for the (non-binarized) training of LLMs under the moniker of variational learning, a variant of variational inference. We welcome further questions and suggestions that may improve the clarity of our work.\n\n\n## CONNECTION TO VARIATIONAL INFERENCE\n\n### ENTROPY REGULARIZATION OF PROBLEM 2 (Top of Section 3)\n\nMotivated by stability and generalization concerns, we may consider an entropy regularized version of Problem 2 given by\n\n$OPT(\\theta) := \\min_{p \\in \\mathcal{P}} \\theta \\cdot \\mathbb{E}_{w \\sim p, x}[L(f(x, w), y_x)] -  H(p)$ \n\nwhere $H(p)$ is the entropy of the distribution $p.$\nThis can be seen to be the log-partition function of a probabilistic graphical model (PGM), albeit one with a complicated $n$-way sufficient statistic given by the loss function $L$, and the corresponding optimizer is the Gibbs distribution $p_\\theta.$\n\nFollowing Wainwright and Jordan [4 -Chapter 3.7 and Chapter 5.1], the task of approximating the log-partition function of a model is exactly where we may deploy the toolkit of variational inference. \n\n\n### MEAN FIELD APPROXIMATION (Line 145 in Section 3)\n\nThe mean field technique [4 - Chapter 5.2-5.3] is based on approximating $p_\\theta$ by finding its closest approximation, in KL divergence, from a family of tractable models. Meng [2] and Ajanthan [1] consider the naive mean field approximation, which corresponds to selecting the class of graphical models with no edges, i.e., product distributions. The resulting optimization problem is:\n\n$$\\min\\_{p \\in \\mathcal{P}\\_{Bernoulli}} D(p || p\\_{\\theta}) = \\min\\_{p \\in \\mathcal{P}\\_{Bernoulli}} D(p || p\\_{\\theta}) = \\min\\_{p \\in \\mathcal{P}\\_{Bernoulli}} \\theta \\cdot \\mathbb{E}\\_{w \\sim p, x}[L(f(x, w), y\\_x)] - H(p) - OPT(\\theta)$$\nwhere $\\mathcal{P}_{Bernoulli}$ is defined in the submission. \n\n\n### BNN Training via Gaussian Variational Inference (Section 3.1)\n\nGaussian variational inference ([5] and references therein) suggests to use as tractable models the family of multivariate Gaussian distributions. We then obtain: \n\n$$\\min\\_{p \\in \\mathcal{P}\\_{\\textrm{corr}}} D\\_{KL}(p || p\\_{\\theta}) = \\min\\_{p \\in \\mathcal{P}\\_{\\textrm{corr}}} \\theta \\cdot \\mathbb{E}\\_{w \\sim p, x}[L(f(x, w), y\\_x)] - H(p) - OPT(\\theta)$$\n\nAt first, this formulation may seem problematic as we are fundamentally changing the reference measure from the discrete measure over the binary hypercube $\\\\{-1,+1\\\\}^n$ to the Lesbegue measure over $\\mathbb{R}^n.$ However, this change is justified, when $\\theta$ goes to infinity, by the approximation result of Grothendieck's inequality, which formally shows that the loss minimization problem over $\\mathcal{P}\\_\\textrm{corr}$ approximates that over $\\mathcal{P}\\_{Bernoulli}$ for certain classes of objective functions.\nSee discussion in 3.1 lines 165-178. \n\nThis reasoning leads us to consider the regime in which the entropy term vanishes ($\\theta \\to \\infty$) to obtain the problem:\n$$\n\\min\\_{p \\in \\mathcal{P}\\_\\textrm{corr}}  \\mathbb{E}\\_{w \\sim p, x}[L(f(x, w), y\\_x)]\n$$\nThis is exactly the problem studied in our paper. For sake of completeness, at a preliminary stage of our investigation, we also had a version of our gradient-based SDP algorithm include the entropy term for different large values of $\\theta$. In all cases, our final algorithm ($\\theta \\to \\infty$) proved more stable, accurate and faster to run, so that we did not pursue different settings of $\\theta$ further.\n\n[1] Thalaiyasingam Ajanthan, Puneet Dokania, Richard Hartley, and Philip Torr. Proximal Mean-Field for Neural Network Quantization. ICCV, 2019.  \n[2] Xiangming Meng, Roman Bachmann, and Mohammad Emtiyaz Khan. Training Binary Neural Networks using the Bayesian Learning Rule. ICML, 2020.  \n[3] Shen et al., Variational Learning is Effective for Large Deep Networks. ICML, 2024.  \n[4]  Wainwright, M.J. and Jordan M.I., Graphical Models, Exponential Families, and Variational Inference.  \n[5] Diao, M.Z., Proximal Gradient Algorithms for Gaussian Variational Inference:Optimization in the Bures–Wasserstein Space, MIT Thesis.\n\n## CONNECTION TO SEMIDEFINITE PROGRAMMING\n\n__See FAQ 4 in the author's rebuttal.__\n\n\n## QUESTION ABOUT MEMORY USAGE\n\n> The memory consumption of this method is not reported, and I suspect that this method is way more memory-intensive than baselines. \n\nIf $n$ is the number of weights, our memory for storing relevant variables is at most $(K+1) n$ compared to $n$ for other methods. The total significance of this increase depends on the batch size and the size of the examples. For instance, in the case of ResNet18, there are roughly $9 \\cdot 10^6$ weights while, for batch size=100, the total size of a data batch is $15 \\cdot 10^6$, for a total memory usage of $24 \\cdot 10^6$. Hence, choosing K=1 will lead to a usage of $33 \\cdot 10^6$, a $37.5%$ increase. __This increase will typically get smaller as we consider models trained on larger images.__", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vl2CiVsr7j", "reviewer": "Reviewer_NfQd", "summary": "The paper proposes a Gaussian variational inference approach to training binary neural networks. Unlike traditional VI methods, the proposed VISPA algorithm is motivated by a SDP relaxation of the binary neural network objective. In experiments, VISPA gives state-of-the-art results for training binary neural networks on CIFAR-10, CIFAR-100 and TinyImageNet benchmarks.", "questions": "- Can lines 11-13 in VISPA algorithm be viewed as an orthogonal projection onto the constraint? It would be interesting to view VISPA as a projected stochastic gradient method on the problem P_corr, which would help in proving convergence and making the algorithm more theoretically sound. \n\n- Even for K=1, the method seems to perform better than other variational baselines like BayesBiNN.  Is there any intuitive explanation as of why the algorithm works so much better in practice?", "limitations": "All limitations are addressed.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and the overall problem formulation was clearly motivated. \n- The proposal to use Gaussian variational inference for directly training binary neural networks appears to be novel. \n- The experimental results are convincing, though larger-scale experiments, for example a ResNet-50 instead of the AlexNet on ImageNet or transformers would make the paper stronger.", "weaknesses": "- The non-diagonal low-rank representation of the covariance adds a significant overhead. It is unclear whether the algorithm can remain scalable. From Figure 1 (Impact of K on model accuracy) it remains unclear whether setting K > 1 is really worth the additional computational overhead. \n- The VISPA algorithm is in some sense \"heuristically\" derived, and it is unclear whether for example stationary points of the method are critical points of the original training objective.\n- I appreciate the connection to variational inference, but in the objective function there is no entropy or prior present, so it is unclear whether this algorithm actually has any connections to Bayesian inference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their assessment and questions. We respond to each point inline below. We encourage the reviewer to ask any follow-up question and to comment on our responses.\n\n> The non-diagonal low-rank representation of the covariance adds a significant overhead. It is unclear whether the algorithm can remain scalable. From Figure 1 (Impact of K on model accuracy) it remains unclear whether setting K > 1 is really worth the additional computational overhead. \n\nThe computational overhead is mostly restricted to the usage of memory. If $n$ is the number of weights, our memory for storing relevant variables is at most $(K+1) n$ compared to $n$ for other methods. The total significance of this increase depends on the batch size and the size of the examples. For instance, in the case of ResNet18, there are roughly $9 \\cdot 10^6$ weights while, for batch size=100, the total size of a data batch is $15 \\cdot 10^6$, for a total memory usage of $24 \\cdot 10^6$. Hence, choosing K=1 will lead to a usage of $33 \\cdot 10^6$, a $37.5%$ increase. __This increase will typically get smaller as we consider models trained on larger images.__ In practice, we often observe even smaller increases, as our estimate does not include memory usage due to backpropagation, which can be very large, e.g., in the case of residual connections.\n\nThe effect on the runtime is negligible because __most of the runtime is spent in forward and back-propagation__ anyways. \nIn conclusion, our method is most suitable in situations where memory is not strictly constrained or can be augmented at low cost. We plan to include this discussion in the final version of the paper.\n\n> The VISPA algorithm is in some sense \"heuristically\" derived, and it is unclear whether for example stationary points of the method are critical points of the original training objective.\n\nWe agree, but notice that the focus of this paper is to provide __a mathematically motivated algorithm that performs competitively in practice__. We believe it eminently possible to obtain the suggested result for sufficiently small step size under the assumption that the loss is smooth by adapting existing results in manifold optimization. Indeed, we can think of our algorithm in the Burer-Monteiro formulation as minimizing the loss function over the manifold given by the constraints $\\mu_i^2 + (ZZ^T)\\_{ii} = \\mu_i^2 + \\sum_j Z_{ij}^2=1$ for all $i$, which is a product of spheres, a well-studied case [1]. As suggested by the reviewer, we can view step 11 of our algorithm as __a projection step onto the manifold__, which effectively implements a retraction from the tangent space back to the manifold. To make the convergence argument complete, it only remains to analyze the effect of minibatch sampling of the loss on the gradient estimates used by the algorithm. Again, by taking the step size sufficiently small, it should be possible to make the contribution of the variance negligible  and guarantee that the algorithm obeys a descent lemma and finds a near-stationary point.\n\n[1] Boumal, N. (2023). An introduction to optimization on smooth manifolds. \n\n> I appreciate the connection to variational inference, but in the objective function there is no entropy or prior present, so it is unclear whether this algorithm actually has any connections to Bayesian inference.\n\n__Please see FAQ 3 in the author rebuttal.__ We will add the entropy term in the final version and justify why we let it vanish for our final algorithm.\n\n> Can lines 11-13 in VISPA algorithm be viewed as an orthogonal projection onto the constraint? It would be interesting to view VISPA as a projected stochastic gradient method on the problem P_corr, which would help in proving convergence and making the algorithm more theoretically sound.\n\nYes. As explained above, Step 11 of our algorithm indeed performs a projection step over the manifold of interest. We would be happy to add this explanation in the final version.\n\n> Even for K=1, the method seems to perform better than other variational baselines like BayesBiNN. Is there any intuitive explanation as of why the algorithm works so much better in practice?\n\nThe improved behavior for $K=1$ is most likely __a benefit of the randomization built into our rounding argument__, even for $K=1$. While other methods deterministically round based on the current $\\mu$-value, our algorithm for K=1 effectively adds noise to $\\mu$ via the $Zr$ term before rounding, allowing us to better explore the space of possible assignments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eEDWX1Ob0D", "reviewer": "Reviewer_6N2m", "summary": "The paper is concerned with variational inference in binary neural networks, e.g., neural networks that contain weights quantized to be either -1 or +1, with the aim of improving the overall performance of binary neural networks by performing Bayesian model averaging. For this, the authors review prior work on the topic and discuss relaxations that are typically done to optimize the optimization problem of finding a variational approximation to the posterior over the binary weights. To approach the problem, the authors show that under the assumption of Gaussian variational inference, e.g., the continuous relaxation of the discrete weights is assumed Gaussian distributed, the objective is a non-linear semi-definite program that is, hence, non-convex. Due to the non-convexity the program is solved iteratively using SGD until a local optimum is reached. The proposed approach elegantly connects the setting to hyperplane rounding and uses a low-rank approximation of the covariance of the variational family. The authors performed an extensive comparison against existing works on VGG and ResNet architectures for common benchmark tasks (CIFAR-10/100 and (Tiny)-ImageNet) based on accuracy and top-1/5 accuracy. The results look generally promising.", "questions": "1. Could the authors elaborate on connections of their methods to natural gradient-based optimization in variational inference, as done in classical and recent literature such as [Honkela] or [Shen]?\n2. The authors model correlations between all units in the network (line 182) through a low-rank approximation and need to resort to a very low rank approximation of rank smaller or equal to ten. Often a block-diagonal matrix assumption is made (assuming layers are independent). Did the authors experiment with a block-diagonal structure in which blocks are low-rank? If so, how does such an approach compare? It seem that a high-rank for each block would be feasible in this setting.\n3. After reading up about hyperplane rounding, I can see the connection to using a sign function. However, the authors only mention this without elaborating on the connection. I would recommend making those connections more explicit and elaborating on them.\n4. A critical aspect of the method seem that it relies on a low-rank approximation, however, choosing the rank might be non-trivial. Moreover, as the ablation in Fig.1 (note that this should be a table) indicates it is not clear that increasing the rank necessarily improves the results. How is the rank chosen in the experiments and how do the authors propose to chose the appropriate rank?\n\nReferences:\n[Honkela] Honkela et al. (2008), Natural Conjugate Gradient in Variational Inference.\n[Shen] Shen et al. (2024), Variational Learning is Effective for Large Deep Networks.", "limitations": "The authors highlight some open problems and future research directions. However, the limitations section could be further improved to detail the specific technical challenges associated with the approach.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "I will provide a general statement of the papers strengths below.\n\nUncertainty quantification in binary neural networks seems like a relevant and interesting direction and I appreciate the connection of non-convex optimization with the field. The technical details are well explained in most parts and the method seems interesting and promising. Moreover, I appreciate the through empirical evaluation even though limited to a single metric and the additional ablation on the rank of the deviation matrix.", "weaknesses": "I will give a general review on the papers weaknesses below and provide detailed questions in the questions section.\n\nAt parts the paper is difficult to follow as notation is sometimes not introduced or details aren't explained enough. For example, page 4 above line 151 capital L should be a curly capital L from my understanding and it was not immediately clear to me why the joint constraint in line 162 is important. Moreover, the contributions of the paper seem rather limited in the context of related work as the resulting method seem to resemble variational inference with a low-rank Gaussian approximate family using the reparameterization trick and natural gradient updates. It is possible that I misunderstood the updates in Alg.1 as I could not find derivations of them in the appendix or main text. The results look promising but are limited to a single metric, accuracy, which might not accurately the performance of each method. Additional common metrics, such as, the negative log predictive density and the calibration error, are missing at the current stage", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their assessment and questions. We respond to each point inline below. We encourage the reviewer to ask any follow-up question and to comment on our responses. We hope that they will consider raising their score at the end of our exchange.\n\n>  The results look promising but are limited to a single metric, accuracy, which might not accurately the performance of each method. Additional common metrics, such as, the negative log predictive density and the calibration error, are missing at the current stage. \n\nWe did not provide these metrics because __they are not available for our competitors__. Here we provide a selection of the results of our method using the negative log predictive density (NLPD) and the calibration error (ECE) metrics. \n\n__For the 1W1A setting on the CIFAR-10 dataset:__\n\n| Architectures | NLDP | ECE|\n|----------------|--------|-----|\n|ResNet18      |   0.60 | 0.074|\n|VGG-Small    |0.71    |0.086|\n\n\n__For the 1W32A setting on the CIFAR-10, CIFAR-100 and TinyImageNet dataset:__\n\n\n| |CIFAR-10 VGG16   | CIFAR-10 ResNET 18| CIFAR-100 VGG16  |CIFAR-100 ResNet18 | Tiny-ImageNet ResNet18|\n|-----------|------------|---------|--------------|------------------|-------------------|\n|NLPD      |  0.49        | 0.25                             | 1.59                        | 1.12                            |    2.13                              |\n|ECE         | 0.19         | 0.035                      | 0.13                            |  0.078                          |   0.13                             |\n\n\n__For the ImageNet dataset:__\n\n| |AlexNet 1W1A | AlexNet 1W32A| ResNet18 1W1A| ResNet18 1W32A  |\n|------------|----------|---------|----------|----------|\n| NLPD      |    22.8             |     23.00             |  6.91         |     6.91        |\n| ECE         |   0.88             |     1.00               |  4.50         |    4.44         |\n\n\n> Could the authors elaborate on connections of their methods to natural gradient-based optimization in variational inference, as done in classical and recent literature such as [Honkela] or [Shen]?\n\nWe thank the reviewer for pointing out these references. __Our approach is comparable to that of Shen__ in the setup of the variational learning problem and the usage of Gaussian variational distributions. However, in contrast with Shen, our method works over low-rank covariances, while Shen adopts diagonal covariance matrices. Finally, to the best of our understanding, the natural-gradient methods of Honkela and Shen are based on considering the Riemannian gradient with respect to the Fisher metric, while __our method implicitly uses the Riemannian gradient over the space of low-rank Gaussian distributions defined via the embedding $\\Sigma = Z Z^T.$__ This has the advantage of enforcing the low-rank constraint and is the basis for the Burer-Monteiro approach to semidefinite programming. We believe that __this key ingredient, together with the connection with Grothendieck’s inequality and hyperplane rounding,  is completely absent from the previous work__ cited by the reviewer. This is to be expected as that work does not have to deal with the training of binary neural networks and is not concerned with rounding from real to binary weights. In conclusion, __we cannot agree with the reviewer’s assessment regarding the limited novelty of our method__ and reassert its significant departure from previous work.\n\n> The authors model correlations between all units in the network (line 182) through a low-rank approximation and need to resort to a very low rank approximation of rank smaller or equal to ten. Often a block-diagonal matrix assumption is made (assuming layers are independent). Did the authors experiment with a block-diagonal structure in which blocks are low-rank? If so, how does such an approach compare? It seem that a high-rank for each block would be feasible in this setting.\n\n__We concur with the reviewer__ on the promise of using a block low-rank structure, a direction we have already been exploring. In our first presentation of the method we opted to model all correlations as we were interested in studying the maximum possible advantage we can obtain via a Gaussian approximation. Now that we have established the existence of this advantage, our future work will focus on how to reap it without incurring the computational costs, particularly in terms of memory, that VISPA suffers. We will include this discussion in Section 5.\n\n> After reading up about hyperplane rounding, I can see the connection to using a sign function. However, the authors only mention this without elaborating on the connection. I would recommend making those connections more explicit and elaborating on them.\n\nWhile this connection is well-known in the theoretical computer science community, we agree that it deserves more elaboration for a more general audience. Unfortunately, we were strongly constrained by space limitations in the detail we could provide in Section 3.1, as __the geometric view of hyperplane rounding requires introducing the vector embedding view of the underlying SDP__. If the reviewer deems it satisfactory, we would be happy to include a separate section detailing the geometric derivation of hyperplane rounding in the Supplementary Material.\n\n> A critical aspect of the method seem that it relies on a low-rank approximation, however, choosing the rank might be non-trivial. Moreover, as the ablation in Fig.1 (note that this should be a table) indicates it is not clear that increasing the rank necessarily improves the results. How is the rank chosen in the experiments and how do the authors propose to chose the appropriate rank?\n\n__See FAQ 2 in the author rebuttal.__ Most results presented use $K=4$, as this was experimentally showed to be a good compromise between performance and computational cost. More generally, for a robust result,  we would recommend __averaging solutions from different choices of rank.__", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their time. We are grateful that most reviewers agree on the __novelty of our framework__ based on Gaussian variational approximation, the __clarity of the submission__ and the __significance of the performance improvements__ revealed by our experiments. In this rebuttal, we address  frequently brought up questions.

# FAQS
## 1. Generalization of our method to other neural network architectures

We chose to focus our first presentation of VISPA on vision-based applications because of the __availability of well-studied binarized architectures and well-established baselines__, which isolate the performance of our method more closely.
Indeed, for transformers, there is not yet agreement on the best binarized architecture, due to the difficulty of binarizing activations in softmax layers. Only recently, researchers have made progress in bypassing this obstacle [1]. This also contributes to the scarcity of baselines and to the __absence of a standard benchmark__.  
To demonstrate the applicability of VISPA to transformers, __we include in the PDF a table of preliminary results__. See the rebuttal to Reviewer Bkmh for a more detailed discussion.

[1] He, Y., et al. "BiViT: Extremely Compressed Binary Vision Transformers." Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2023.

## 2. Setting of rank parameter K

The impact of $K$ on test accuracy is discussed in Section 4.1, Figure 1. Our expectation was that higher values of $K$ should yield better accuracy, as the algorithm can explore a larger space of distributions. As pointed out by a number of reviewers, the results do not fully support this conclusion. __Results do tend to marginally improve as K gets larger__ with the exception of VGG-Small + CIFAR-10 (1W1A).  The accuracy of ResNet18+CIFAR10 (1W32A) is essentially constant over $K$. We have two hypothesis for this lack of conclusive evidence:
- Higher values of $K$ yield a __larger feasible space and require longer time to converge__. In the case of VGG-Small + CIFAR-10 (1W1A), it is possible that the fixed number of epochs chosen for the experiments was sufficient for convergence on smaller $K$ values, but insufficient on higher values. 
- Higher values of $K$ yield a __more complex distribution__ from which it is __more expensive to sample good weights__ at inference stage. 

Based on preliminary experiments, we believe that a combination of the two may be at play for VGG-Small + CIFAR-10 (1W1A). 

## 3. Connection to Gaussian Variational Inference

Following Wainwright and Jordan [1 -Chapter 3.7], the variational inference  technique can be applied to the variational formulation of the problem of computing the __log-partition function__ of a probablistic graphical model. By adding an entropy regularization term $\frac{1}{\theta} \cdot H(p)$, the optimization problem Problem 2 (Line 134) can be put in this form for a graphical model whose sufficient statistics are described by the loss function. The fundamental idea of variational inference is then to __approximate the log-partition function__ and the corresponding Gibbs distribution $p_{\theta}$ by restricting the entropy-regularized optimization problem to a __tractable class of probability distributions__ (see [1], Chapter 5.2). The Gaussian Variational Inference approach (see [2,3] and references therein) suggests using the class of __multivariate Gaussian distributions__ as the tractable family of choice. Grothendieck's inequality, which shows that such a Gaussian distribution can be used to approximate the required function in the regime $\theta \to \infty$, leads us to __let the entropy term vanish__. Finally, this yields the same objective function as in our submission.

[1] Wainwright, M.J. and  Jordan M.I., Graphical Models, Exponential Families, and Variational Inference  
[2] Diao, M.Z., Proximal Gradient Algorithms for Gaussian Variational Inference:Optimization in the Bures–Wasserstein Space   
[3] Lambert, M. et al, Variational inference via Wasserstein gradient flows  


## 4. Connection To Semidefinite Programming

The reviewers correctly note that we use the term SDP to simply indicate the presence of a semidefinite constraint; however, this usage has become frequent in recent literature as researchers investigate the potential of semidefinite programs with non-convex objectives and constraints, in particular low-rank constraints [1]. For clarity, throughout the paper, we repeatedly reassert the non-convexity of our formulation. Despite this non-convexity, we argue that __our techniques are typical of SDP programming__. The Burer-Monteiro approach is one of the most promising practical  techniques to reduce the computational cost of SDPs [2], with a __significant amount of research__ devoted to understanding which reduction in rank preserves the global optima of the original solution [3,4,5]. Similarly, our use of __moment matching constraints__ in the definition of $\mathcal{P}_{\textrm{corr}}$ is typical of the __relaxation of combinatorial optimization problems to semidefinite programs__, which is a pillar of the theory of approximation algorithm. Indeed, our work is originally motivated by __Grothendieck’s inequality and hyperplane rounding.__

[1] Mei, S., Misiakiewicz, T., Montanari, A., & Oliveira, R. I. (2017, June). Solving SDPs for synchronization and MaxCut problems via the Grothendieck inequality.  
[2] Yurtsever, A., Tropp, J. A., Fercoq, O., Udell, M., & Cevher, V. (2021). Scalable semidefinite programming.  
[3] Boumal, N., Voroninski, V., & Bandeira, A. (2016). The non-convex Burer-Monteiro approach works on smooth semidefinite programs. \
[4] Cifuentes, D., & Moitra, A. (2022). Polynomial time guarantees for the Burer-Monteiro method.  
[5] Erdogdu, M. A., Ozdaglar, A., Parrilo, P. A., & Vanli, N. D. (2022). Convergence rate of block-coordinate maximization Burer–Monteiro method for solving large SDPs.

Meta-review: Thank you for your valuable contribution to Neurips and the ML community. Your submitted paper has undergone a rigorous review process, and I have carefully read and considered the feedback provided by the reviewers.

The paper proposes a variational approach to training binary neural networks using an SDP relaxation. The authors obtain competitive numerical results compared to state-of-the-art. Overall, the paper received mostly positive response from the reviewers. Some issues raised by the reviewers were successfully addressed in the rebuttal. These issues include (i) connections to variational inference and SDPs, (ii) memory consumption, (iii) impact of hyperparameters.

Given this positive assessment, I am willing to recommend the acceptance of your paper for publication.

I would like to remind you to carefully review the reviewer feedback and the resulting discussion. While most reviews were positive, the reviewers have offered valuable suggestions that can further strengthen the quality of the paper. Please take another careful look a the 'weaknesses' section of each reviewer comment and the issues (i)-(iii) listed above. I encourage you to use this feedback to make any necessary improvements and refinements before submitting the final version of your paper.

Once again, thank you for submitting your work to Neurips.

Best,
Area Chair
Decision: Accept (poster)