Understanding weight-magnitude hyperparameters in training binary networks
Authors: Joris Quist
Yunqiang Li
Jan van Gemert
Venue: ICLR 2023 poster (2023)
Keywords: Binary Neural Networks
Optimization
Digital Signal Processing
Inifnite Impulse Response Filter

Abstract: Binary Neural Networks (BNNs) are compact and efficient by using binary weights instead of real-valued weights. Current BNNs use latent real-valued weights during training, where several training hyper-parameters are inherited from real-valued networks. The interpretation of several of these hyperparameters is based on the magnitude of the real-valued weights. For BNNs, however, the magnitude of binary weights is not meaningful, and thus it is unclear what these hyperparameters actually do. One example is weight-decay, which aims to keep the magnitude of real-valued weights small. Other examples are latent weight initialization, the learning rate, and learning rate decay, which influence the magnitude of the real-valued weights. The magnitude is interpretable for real-valued weights, but loses its meaning for binary weights. In this paper we offer a new interpretation of these magnitude-based hyperparameters based on higher-order gradient filtering during network optimization. Our analysis makes it possible to understand how magnitude-based hyperparameters influence the training of binary networks which allows for new optimization filters specifically designed for binary neural networks that are independent of their real-valued interpretation. Moreover, our improved understanding reduces the number of hyperparameters, which in turn eases the hyperparameter tuning effort which may lead to better hyperparameter values for improved accuracy. Code is available at https://github.com/jorisquist/Understanding-WM-HP-in-BNNs

Published as a conference paper at ICLR 2023
UNDERSTANDING
HYPERPARAMETERS IN TRAINING BINARY NETWORKS
Joris Quist', Yunqiang 2 Jan van Gemert Computer Vision Lab, Delft University of technology; 2. Axelera Al
ABSTRACT
Binary Neural Networks (BNNs) are compact and efficient by using binary weights instead of real-valued weights. Current BNNs use latent real-valued weights during training, where hyper-parameters are inherited from real-valued networks. The interpretation of several of these hyperparameters the magnitude of the real-valued weights. For BNNs, however, the magnitude of binary weights is not meaningful, and thus is unclear what these hyper- parameters actually do. One example is weight-decay, which aims to keep the magnitude of real-valued weights small. Other examples are latent weight ini- tialization, the learning rate, and learning rate decay, which influence the mag- nitude of the real-valued weights. The magnitude is interpretable for real-valued weights, but loses its meaning for binary weights. In this paper we offer new interpretation of these magnitude- hyperparameters based on higher-order gradient filtering during network optimization. Our analysis makes it possible to understand how magnitude-based hyperparameters influence the training of binary networks which allows for new optimization filters specifically designed for binary neural networks that are independent of their real-valued interpreta- tion. Moreover, our improved understanding reduces the number of hyperparam- eters, which in turn eases the hyperparameter tuning effort which may lead to bet- ter hyperparameter values for improved accuracy. Code available at ht tps: / //githubs com/ jorisquist/ /onquustttuuderr iiderss rnteertanddee werrssannnnnggn WM-HP 1e---in-- BNNN
based on
INTRODUCTION
A Binary Neural Network (BNN) weight is single bit: -1 or +1, which are compact and efficient, enabling ap- plications on, for example, edge devices. Yet, training BNNs using gradient decent is difficult because the discrete binary values. Thus, BNNs are often (Kim et al., 2021b; Liu et al., 2020; Martinez et al., 2020) optimized with called 'latent', real-valued weights, which dis- cretised to -1 or +1 by, e.g., taking the positive or nega- tive sign of the real value.
5 2
Woon
0.5
The latent weight optimization depends on several essen- tial hyperparameters, such their initialization, learning rate, learning rate decay, and weight decay. These hy- perparameters are important for BNNs, as shown for ex- ample in Martinez et al. (2020), and also by Liu et al. (2021a), who both improve BNN accuracy by better tun- ing these hyperparameters.
Valgee
Figure Changes in rellev weights change their magnitude. For bi- nary weights, however, the magnitude will never change and magnitude-based hyperparameters need reinterpretation.
In this paper we investigate the latent weight hyperparameters used in a BNN, including initializa- tion, learning rate, learning rate decay, and weight decay. All these hyperparameters influence the magnitude of the latent weights. Yet, as illustrated in Fig 1, in a BNN, the binary weights are -1 signitudd-baseed hyperparameters lose
or +1, which always have constant magnitude and thus their meaning. We draw inspiration from the seminal work of Helwegen et al. (2019), who reinter- pret latent weights from inertia perspective and state that latent weights do not exist. Thus, the

---

Published as a conference paper at ICLR 2023
magnitude of latent weights also does not exist. Here, investigate what latent Bright-mannitue hyperparameters mean for a BNN, how they relate to each other, and what justification they have. We provide gradient-filtering perspective latent weight hyperparameters which main benefit is a simplified setting: fewer hyperparameters to tune, achieving similar accuracy as current, more complex methods.
2 RELATED WORK
Latent weights in BNNs. By tying each binary weight to a latent real-valued weight, continu- ous optimization approaches can be used to optimize binary weights. Some methods minimize the quantization error between latent weight and its binary variant (Rastegari et al., 2016; Bulat & Tzimiropoulos, 2019). Others focus on gradient approximation (Liu et al., 2018: Lee et al., 2021; Zhang et al., 2022), or on reviving dead weights (Xu et al., 2021; Liu et al., 2021b), or on entropy regularization (Li et al., 2022) or a loss-aware binarization (Hou et al., 2017: Kim et al., 2021a). These works directly apply traditional optimization techniques inspired by real-valued network such as weight decay, learning rate and its decay, and optimizers. The summary of De Putter & Corporaal (2022) gives good overview of these training techniques in BNNs. Recently, some papers (Liu et al., 2021a; Martinez et al., 2020; Hu et al., 2022; Tang et al., 2017) noticed that the interpreta- tion of these optimization techniques does not align with the binary weights of BNNs (Lin et al., 2017; 2020). Here, we aim to shed light on why, by explicitly analyzing latent weight-magnitude hyperparameters in a BNN.
Latent weight magnitudes. Several techniques exploit the magnitude of the latent weights during BNN optimization. Latent weights clipping is proposed in (Courbariaux et al., 2015) and followed by its extensions (Alizadeh et al., 2018; Hubara et al., 2016) to clip the latent weights within a [-1.1] 1] interval prevent the magnitude of latent weights from growing too large. Gradient clipping (Cai et al., 2017; Courbariaux et al., 2015; Qin et al., 2020) stops gradient flow if the magnitude of latent weight is too large. Work on latent weight scaling (Chen al., 2021; Qin al., 2020) standardizes the latent weights to a pre-defined magnitude. Excellent results are achieved by a two-step training strategy (Liu al., 2021a; 2020) that in the first step trains the network from scratch using only binarizing activations with weight decay, and then in the second step they fine-tune by training without weight decay. Our method reinterprets the meaning of the magnitude based weight decay hyperparameter in optimizing BNNs from a gradient filtering perspective, offering similar accuracy as two step training with simpler setting, using just a single step.
Optimization by gradient filtering. Gradient filtering is a common approach used to tackle the noisy gradient updates caused by minibatch sampling. Seminal algorithms including Momen- tum (Sutskever et al., 2013) and Adam (Kingma & Ba, 2015) which use first order infinite impulse response filter (IIR), i.e. exponential moving average (EMA) to smooth noisy gradients. Yang (2020) takes this one step further and introduces the Filter Gradient descent Framework that can use differ- ent types of filters on the noisy gradients to make a better estimation of the true gradient. In binary network optimization, Bop (Helwegen et al., 2019) and its extension (Suarez-Ramirez et al., 2021) introduce a threshold to compare with the smoothed gradient by EMA to determine whether to flip binary weight. In our paper, we build on second order gradient filtering techniques to reinterpret the hyperparameters that influence the latent weight updates.
Sound optimization approaches. Instead of using heuristics to approximate gradient descent on discrete binary values, several works take more principled approach. Peters & Welling (2018) propose a probabilistic training method for BNN, and Shekhovtsov & Yanush (2021) present a theoretical understanding of straight through estimators (STE) (Bengio et al., 2013). Meng et al. (2020) propose Bayesian perspective and Louizos et al. (2018) formulate noisy quantizer. Even though these approaches provide more theoretical justification in optimizing BNNs, they are more complex by either relying on stochastic settings or discrete relaxation training procedures. Moreover, these methods do not (yet) empirically reach a similar accuracy as current mainstream heuristic methods (Liu et al., 2018; 2020). In our paper, we build on the mainstream approaches, to get good empirical results, but add a better understanding of their properties, taking a step towards better theoretical understanding of empirical approaches.
2

---

Published as a conference paper at ICLR 2023
3 HYPERPARAMETER ANALYSIS THROUGH GRADIENT FILTERING
We start with a latent weights BNN and convert it to an equivalent weight free setting, as in Helwegen et al. (2019). To do this, we use magnitude independent setting, which means that no gradient-elipping or scaling based on the channel-wise mean of the latent-weights used.
BNN setup. We use Stochastic Gradient Descent (SGD) with weight decay and momentum starting point, as this is a commonly used setting, see Rastegari et al. (2016), Liu et al. (2018), Qin et al. (2020). Our setup is follows:
Wg init()
(1)
m, (1 7)m + V
(2)
W/ wi-1 <(m, + Aw
(3)
0, sign(w;)
(4)
if < 0;
sign(1) (x)
+1, random { 11,++1] otherwise.
if 0;
(5)
Here, Wi is latent weight iteration i which is initialized at wi- is binary weight, is the learning rate, À is the weight decay factor, m, is the momentum exponentially weighted mov- ing average with m-1
and discount factor y, Ve, is the gradient over the binary weight and ( +1] is uniformly randomly sampled -1 or +1.
We then convert to the weight free setting of Helwegen et al. (2019) where latent weights are interpreted as accumulating negative gradients. We introduce = ww; which allows working with gradients instead of with latent weights. We can then write Eq 3 as follows
9: 9i-1 c(m;
>gi-1)-
Latent weight initialization. To investigate latent weight initialization we unroll the the recursion in Eq 6 by writing out summation:
g. = (1 -A)gi + em
> r=0
(1-e))- m,
I weights are typically initialized using real-valued weight initialization techniques (Glorot & Bengio, 2010; He et al., 2015). However, since we now interpret latent weights as accumulated gradients, we argue to also initialize them gradient accumulation techniques such as Momen- tum (Sutskever et al., 2013) and simply initialize Wo = 9o = O, because at initialization there is no preference for negative or positive gradients, and their expectation is O. We do not use bias- correction as done in (Kingma& & Ba, 2015) because in practice we noticed that gradient magnitudes large in the first few iterations. Applying bias correction increases this effect, which had nega- tive effect on training. To prevent all binary weights 0 to start at the same value, we use the stochastic sign function in Eq 5 that randomly chooses sign when the input exactly 0.
Learning rate and weight decay. The learning rate appears two places in Eq 7: once outside the summation, and once inside the summation. The e outside the summation can only scale the latent weight and will not influence outcome of the sign in Eq 4 as
Σ (2) mr siign Σa- (A)- my P=0
sign
(8)
r=0
Thus, the leftmost e can removed, or set randomly without influencing the training process.
For the inside the summation of Eq 7, it appears together with the weight decay term X. Thus, there are two free hyperparameters that only control one factor, therefore one of them is redundant and can use single combined hyperparameter = eλ. Instead of setting value for the learning rate e, and setting a value for the weight decay \, we now only have to set a single value for a. Since Eq 8 shows that we can freely scale the sum with any constant factor, we scale it with a, as
9:
(1 a) m -
(9)
3

---

Published as a conference paper at ICLR 2023
which allows to re-write the with recursion, exponential moving average (EMA)
(1 00)9--1 + om.;
(10)
where 9-1 0. This shows that for BNNs under magnitude independent conditions, SGD with weight decay is just exponential moving average. This gives magnitude-free justification for using weight decay since its actual role is to act as the discount factor in an EMA. Note that is no longer possible to set to since then there number will essentially work the same. The meaning a is now clear, as in the EMA it controls how much to take the past into account.
updates anymore, but setting to small (10 20)
Learning rate decay. There no longer is a learning rate to be decayed, however, since learning rate decay scales the learning rate and a eλ, now the learning rate decay directly scales a, so from now on we apply it to alpha and will refer to it as a-decay. This also helps better explain its function: a-decay increases the window size during training, causing the filtered gradient become more stable and allowing the training to converge.
Momentum.
Now adding back the momentum term of Eq original setup yields
m; (1 -
(11)
= (1 - a 0000011111 0f77,,
(12)
0: -iign(g::)
(13)
Thus, SGD with weight decay and momentum is smoothing the gradient twice with an EMA filter.
Latent weight optimization as a second order linear infinite impulse response filter. EMAs
specific type of linear Infinite Impulse Response (IIR) Filter (Proakis, 2001). Linear filters
are filters that compute an output based on a linear combination of current and past inputs and past outputs. The general definition is given as difference equation:
y = 1 (boz, ; + b1 +... bpx; - 019i-1 - -.. -apW--o) (14)
where i is the time step, Y1 are the outputs, are the inputs, aj and are the filter coefficients and P and Q are the maximum iterations the filter looks back at the inputs and outputs to compute the current output. The maximum of P and defines the order of the filter. An EMA only looks at the previous output and the current input, so is therefore first order IIR filter. Expressing an EMA filter looks follows:
a 0
1 - a-
y/ a))ai az;
(bar; T;-1 a19i-1), dg
a
(15)
In optimizer have cascade of two EMAs applied in series to the same signal which can be represented by a filter with the order being the sum of the orders the original filters. To get the new and b vectors the original ones are convolved with each other. In our this gives:
[a7]
b
?
-
(a (-- (a
(16)
a a-
when applied to our gradient filtering setting in Eq 12 gives the difference equation:
g V7 (a + (a- 1)(7
1)9:-2
(17)
Thus, in magnitude independent setting, SGD with weight decay and momentum is equivalent to 2nd order linear IIR filter. Note that a and y have the same function: Without a decay, the values for and be swapped. This filtering perspective opens up new methods of analysis for optimizers.
Main takeaway. Our re- re-interpretations reduces the 7 hyper parameters in the latent weight view with SGD, to only hyperparameters in our filtering view, Table 1.
4

---

Published as a conference paper at ICLR 2023
Learning rale
Learning rale decay
Init
Momentum
Weight decay
Scaling
Clipping
Latent:
e-decay a-decay
Wo
Filtered:
Table 1: Hyperparameters used in the latent weight view versus filtered gradients perspective. Our filtered perspective reduces the number of hyperparameters from 7 to 3.
Filtered Gradients
gradients
2nd order (a 10
10
1st order (a= 10-1)
1st order (a-
10
filtered Gradient
0.0001
Gradenn
-0.5
-0.0001
-1.0
100
Beration
Figure 2: Gradients and filtered gradients for first and second order filter in single epoch on CIFAR-10. For better visualisation, the filter outputs are scaled up to similar range as the unfiltered gradients. It can be seen that the unfiltered gradients are noisy and that the filtered outputs are smoother. The second order filter reduces the noise even further compared to the first order filter.
4 EXPERIMENTS
We empirically validate our analysis on CIFAR-10, using the BiRealNet-200 architecture (Liu et al., 2018). Unless mentioned otherwise the networks were optimized using SGD for both the real-valued and binary parameters with as hyperparameters: learning rate=0. 1, momentum with weight decay=10-4, batch size=256 and cosine learning rate decay and cosine alpha decay. We analyze the weight flip ratio at every update, which is also known as the FF ratio (Liu et al. (2021a)).
(1 0.9),
ΣL Ewew, I
sign( wi 1) sign(w;)
Ir
FF ratio
(18)
2
Nitial
where Wi is latent weight at time i, L the number of layers, Wi the weights in layer 1, and Noxal the total number of weights.
main
scaling
1a-92
02
0.0020
1e:01
0.0020
1e+01
1es01
1c:01
FT Rate
0.0010
0.0010
0.0000
0.0000
50
100
150
200
300
350
400
100
150
200
250
300
350
400
Epoch
Esoch
(a) Varying learning rates, constant init.
(b) Vary init, constant learning rate (c=1.0).
Figure 3: In the magnitude independent setting, scaling the learning rate has the exact same effect on the flipping ratio as scaling the initial latent-weights by the inverse.

---

Published as a conference paper at ICLR 2023
80
Top-1 (%)
75
Magnitude Dependent
Magnitude Independent
Magnitude Independent init
10
10-2
10
10
10
102
10a
10a
1020
Learning Rate
Figure 4: Learning rate effect on aceuracy for three settings. (a). standard SGD, (b) Magnitude
independent in Eq 3, by removing clipping and scaling. (c) Magnitude independent with latent weight initializes of 0 in Eq 7. Setting (a) has just a single optimum. The accuracy in setting (b) is sensitive small learning rates. For setting (c), accuracy is independent of the learning rate.
0.005
0.9
1c 06
0.000
1s 05
0.7
1c: 03
0.6
0.4
0.010
0.005
0.000
0.1
50
100
150
200
300
350
400
100
150
200
250
300
350
400
Epech
alpha
Figure 5: Bit flipping (FF) ratio and accuracy for varying alphas from eq 17. BNNs are sensitive to alpha, similar to how they are sensitive weight decay. Tuning alpha is essential.
1st order vs 2nd order We visually compare filter orders by sampling real gradients from a single binary weight trained CIFAR- 10 in Figure 2. For the same 1st order filter is more noisy than a 2nd order filter. This may cause the binary weight to oscillate, even though the larger trend is that should just flip once. To reduce these oscillations with 1st order filter requires smaller alpha. This, however, causes other problems because a determines the window size of past gradients and with a smaller many more gradients are used. This means that takes much longer for a trend in the gradients to effect the binary weight. Instead, the 2nd order filter has both benefits: it can filter high frequency noise while still able to react quicker to changing trends.
Magnitude independent learning rate vs initialization In Figure 3 we show the bit flip- ping ratio for the learning rate and the initialization Jo- Multiplying c with some scal- ing factor is the same as dividing 9o by sign (1 - eX)'go + se (1
=
sign (1- (X) go + +e m,, because the magnitude has no effect on the sign. Larger 6 in Figure 3(a) and smaller 9u in Figure 3(b) are independent to scaling and have similar flipping ratios. A too small too large 9u do not reach the same flipping ratios, because their ratio insufficient to update the binary weights. For sufficiently large ratios it means that scaling both e and go has effect training, but also that scaling the scaling go with the inverse identical: as seen by comparing the two plots in Figure 3.

---

Published as a conference paper at ICLR 2023
Sensitivity to hyperparameters for weight magnitude (in)dependence We evaluate SGD in the standard magnitude dependent setting with clipping and scaling a magnitude independent setting with initializing zero. To keep
0.0025
Validation Accuracy 2
0.08
0.0020
0.0010
apha schedule
0.5
spph schedile
0.
the latent-weights the effect of weight decay constant, we scale the weight decay factor versely with the learning rate. Results in Figure 4 show that for the stan- dard magnitude dependent setting the learning rate e has to be carefully bal- anced. A too small w.r.t. to the ini- tial weights inhibits learning; while a too large c will push latent-weights to the clipping region and will stop updating. In the magnitude independent setting there is clipping. A too small c, however is still problematic because the magnitudes of the gradients are smaller when not using the scaling factor and the accuracy drops significantly. When initializing zero, as propose, this problem disappears, because there is no initial weight to hinder training and all learning rates perform equal.
cosine
costne
0.0000
0.3
200 Fpach
Figure 6: Flipping rate and accuracy for alpha decay. With decay the FF ratio goes to zero. Without decay, the flipping will continue, preventing convergence, reducing accuracy.
In Fig 5 we vary a from eq 17. Alpha strongly influences training. too large a causes too many binary weight flips per update, which hinders converging. A too small makes the network converge too quickly, which hurts the end result. Tuning a is essential.
Alpha
Alpha decay For proper conver- gence the flipping (FF) ratio should go to zero. We transform learning rate decay alpha decay. When becomes smaller, the gradients will change less, causing fewer flips, fore- ing the network to converge.
0.0025
0.08
Veletation Accumiy
0.7
Ralo
0.0015
1.6
0.0010
0.5
0.0005
Gradient Fitering
Gradient Fittering
Lacent We ghis
Latent Weights
See the plots in Figure 6 where one network has been trained with cosine alpha decay and one without alpha decay. With and without alpha decay both seem to perform well the start of training, however, the variant with- out alpha decay plateaus at the end of training while the BNN with alpha decay converges better and continues improving, leading better end result.
0.0000
i0o
Figure 7: Equivalence of latent weights (Eq 3) and our gra- dient filtering (Eq 17). They are empirically equivalent.
Equivalent interpretation Figure shows empirical validation with matching hyperparameters that the SGD setting using latent weights in Eq 3 is equivalent to our gradient filtering interpretation in Eq 17 that no longer uses latent weights,
4.1 VALIDATION OF EQUIVALENCE TO THE CURRENT STATE OF THE ART
We validate on for CIFAR-10 and Imagenet that our filtering- based optimizer is similar to the current state of the art. Several current methods use an expensive two-step optimization step. We aim to show the value of our re-interpretation by showing similar accuracy but only in single step.
CIFAR-10: We train all networks for 400 epochs. As data augmentation we use padding of pixels, followed by 32x32 crop and random horizontal flip. We use Ni-RealNet-22,, and for the real-valued parameters and latent-
when used, we use SGD with learning rate of 0.1 with cosine decay, momentum of 0.9 and on the non-BN parameters weight decay of 10 4, For filtering baasedng-based based optimizer we used an alpha of 10 with cosine decay and a gamma of 10 1 Results Table show that our re-interpretation achieves similar accuracy.

---

Published as a conference paper at ICLR 2023
Table 3: Comparison with state-of-the-art on Imagenet.
Table 2: State-of-the-ar on CIFAR-I0. The denotes that we re-ran these experi- ments ourselves.
Training Top-1 Top-5
Method
Strategy Acc(S) Acc(%)
Training
Bit-Wing Bit-width Top-1 Strategy (W7A) Acc(3)
CI-BONN (Wang et ul., 2019)
59.9
84,2
Method
Binary MobileNet (Phan et al., 20206)
60.9
82,6
MoBiNet (Phan et al., 2020a)
54.4
77.5
FP
32/32
91.7
EL (Hu et al,, 2022)
One step
56.4
DoReFa-Net (Zhou et al., 2016)
1/1
79.3
MeliusNer29 (Bethge et al., 2020)
65.8
DSQ (Gong et al., 2019))
1/1
ReActNet-A Our filtering optimizer
69.7
88.9
IR-Net (Qin et al., 2020)
One step
1/1
86.5
Ri-Real
al., 2018)
1/1
85.0
StrongBaseline (Martinez et ul., 2020)
60.9
83.0
Real + Our filtering optimizer
1/1
86.5
Real-to- Binary (Martinez al., 2020)
ReActNet-A (Liu et al., 2020)
Two stop
69.4
65.4
88.6
86.2
Bi-Real" (Liu et al., 2018)
Two stop
1/1
86.7
RcAcUNetAAAdambNN (Liu et al., 2021a)
70.5
89.1
Imagenet: We follow Liu al. (2021a): We train for 600K iterations with batch size of 510. For the real-valued parameters we use Adam with a learning rate of 0.0025 with linear learning rate decay. For the binary parameters we use our 2nd order filtering optimizer with a 10-5 which we decay linearly and 7 = 10-1 We do not use two-step training to pre-train the latent- weights. Results in Table show that ReActNet-A with our optimizer compares well to other one step training methods. It approaches the accuracy of two step training approaches, albeit without an additional expensive second step of training.
4.2 EMPIRICAL ADVANTAGE OF HAVING FEWER HYPERPARAMETER TO TUNE
Our filtering perspective significantly reduces the number of hyperparameters (Table 1). Here, we empirically verify the computational benefit having fewer hyperparameters to tune when applied in setting where the hyperparameters are unknown. To show generalization to other architectures and modality we use an audio classification task (Becker et al., 2018) with a fully connected network. Specifically, we use 4 layers with batch normalization of which the first layer is not binarized. For the latent weights we optimize 6 hyperparameters, and for our filtering perspective we optimize two hyperparameters, see Table 1. We did not tune learning rate decay this had effect on both methods. To fairly compare hyperparameter search we used Bayesian optimization (Balandat et al., 2020). The results for 25 trials of tuning, averaged over 20 independent runs are in Figure 8. We confirm that both perspectives achieve similar accuracy when their hyperparameters are well tuned. Yet, for the latent weights, it takes on average around 10 more trials when compared to the gradient filtering. This means that the latent weights would have to train many more networks, which on medium-large datasets such as Imagenet would already take several days to converge. In contrast, the gradient filtering requires much less time and energy to find good model.
5 DISCUSSION AND LIMITATIONS
One limitation of our work that we do not achieve "superior performance" in terms of accuracy. Our approach merely matches the state of the art results. Note, however, that our goal is to provide insight into how SGD and its hyperparameters behave, not improve accuracy. Our analysis ended up with an optimizer with less byperparameters, that also have a better explanation in the context of BNN optimization leading to simpler, more elegant methods. Our main empirical contribution is in the significant computational reduction in hyperparameter tuning.
Another perceived limitation that our new proposed optimizer can be projected back to a specific setting within the current SGD with leights interpretation. Thus, our analysis might not be needed. While it is true that weights can also be used, we argue that there is no disadvantage to switching to the filtering perspective, because the options are the same, but the benefit is that our hyperparameters make more sense. The option to project back to letght- also works the other way around and for those who already have a well tuned SGD optimizer could use it to make easier to switch to our filtering perspective. The benefit of our interpretations is having fewer hyperparameters to set.
8

---

Published as a conference paper at ICLR 2023
Accuracy (%)
20
Gradient Filtering
Latent Weights
3
5
11
13
15
17
19
21
23
25
Trial
Figure 8: Bayesian hyperparameter optimization for our gradient filtering vs. latent weights. We optimize the hyperparameters of Table I, except learning rate decay and alpha decay, as they had no influence. The scatter plot show the best achieved result so far at each trial for all runs. The lines show the average best result, up the current trial. Both methods perform equally for good hyperparameters. Our gradient filtering view needs much less trials.
Its also true that our method cannot use common techniques based on the magnitude such as weight clipping gradient clipping. Yet, do not really think these techniques necessary. We such methods as heuristics to reduce the bit flipping ratio over time, which helps with convergence. However, in our setting, this can also done using a good a decay schedule without reverting to such heuristics, making the optimization less complex.
We did not yet have the opportunity test the filtering-baad based optimizer on more architectures and datasets. However, since our optimizer is equivalent to specific setting of SGD, we would argue that architectures that have been trained with SGD will probably also work well with our optimizer. This is also reason why we chose to use ReActNet-A, since it was trained using Adam in both in the original paper (L.iu al., 2020) and in Liu al. (2021a). The latter specifically argues that Adam works better for optimizing BNNs, but suspect that the advantages of Adam are decreased because might not work in the same way in the magnitude invariant setting, as we see a smaller difference in accuracy. Introducing this normalizing aspect into the filtering based perspective is interesting topic for future research.
One last point to touch upon is soundness. Even though the filtering perspective provides better explanation to hyperparameters, it does not provide understanding on why optimizing BNNs with second-order low pass filters works well it does. Whereas stochastic gradient descent has extensive theoretical background, this does not exist for current mainstream BNN methods. Fully understanding BNN optimization an interesting direction for future research and our hope that this work takes step in that direction.
Ethics Statement We believe that this research does not bring up major new potential ethical con- cerns. Our work makes training BNNs easier, which might increase their use in practice.
Reproducibility Statement All our code is availabe online: https: / /ggghuu......./ com/ -MK-HP-in-BNNE Two important things for better repro- ducing our results rely on the GPUs and the dataloader. The reproduction of our ImageNet exper- iments is not trivial. First, as the teacher-student model is used in our ImageNet experiments, it will occupy a lot GPU memory. We trained on NVIDIA A40 GPUs, each A40 has 48 GB of GPU memory, with batch size of 170 per GPU for much as ten days. Second, for faster training on ImageNet, we used NVIDIA DALI dataloader to fetch the data into GPUs for the image pre- pre-processing. This dataloader could effect training as it uses slightly different image resizing algorithm than the standard Py ToTch dataloader. To keep results consistent with other methods, we do the inference with the standard PyTorch dataloader.

---

Published as a conference paper at ICLR 2023
REFERENCES
Milad Alizadeh, Javier Fernández-Marqués, Nicholas D) Lane, and Yarin Gal. An empirical study of binary neural networks' optimisation. International conference on learning representations, 2018.
Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, An- drew Gordon Wilson, and Eytan Bakshy. BoTorch: A Framework for Efficient Monte-Carib Bayesian Optimization. In Advances in Neural Information Processing Systems 33, 2020. URL htttp://aaxiii http://axxvv..rgaabb..o
www.orrgg//.... abs 11111991000666 .66003.
Sören Becker, Marcel Ackermann, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojeiech Samek. Interpreting and explaining deep neural networks for classification of audio signals. arXiv preprint 0r0.vv/807.0344 18, 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. Technical Report, 2013.
Joseph Bethge, Christian Bartz, Haojin Yang, Ying Chen, and Christoph Meinel. Meliusnet: Can binary neural networks achieve mobilenet-level accuracy? arXiv preprint arXiv:2001.05936, 2020.
Adrian Bulat and Georgios Tzimiropoulos. Xnor-net++:: Improved binary neural networks, 2019.
Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half-wave gaussian quantization. Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 5918-59182 8-5969,, 2017.
Tianlong Chen, Zhenyu Zhang, Xu Ouyang, Zechun Liu, Zhiqiang Shen, and Zhangyang Wang. ban-bn=?":: Training binary neural networks without batch normalization. In Proceedings of the IEEE/CVF conference computer vision and patlern recognition, pp. 4619-4629, 2021.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In NeurlPS, 2015.
Floran De Putter and Henk Corporaal. How to train accurate bans for embedded systems? arXiv preprint wtXXhv:2206.13222 2022.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, pp. 249-256, 2010.
Ruihao Gong, Xianglong Liu, Shenghu Jiang. Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4852-4861, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015.
Koen Helwegen, James Widdicombe, Lukas Geiger, Zechun Liu, Kwang- Ting Cheng, and Roe- land Nusselder. Latent weights do not exist: Rethinking binarized neural network optimization. Advances in neural information processing systems, 32, 2019.
Lu Hou, Quanming Yao, and James T Kwok. Loss-aware binarization of deep networks. ICLR, 2017.
Jie Hu, Ziheng Wu, Vince Tan, Zhilin Lu, Mengze Zeng, and Enhua Wu. Elastic-link for binarized neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 942-950, 2022.
Itay Hubara, Mathheu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarized neural networks. Advances in neural information processing systems, 29, 2016.
Dohyung Kim, Junghyup Lee, and Bumsub Ham. Distance- aware quantization. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pp. 5271-5280, 2021a.
10

---

Published as a conference paper at ICLR 2023
Hyungjun Kim, Jihoon Park, Changhun Lee, and Jae-Joon Kim. Improving accuracy of binary neural networks using unbalanced activation distribution. In Proceedings of the IEEE/CVF con- ference computer vision and pattern recognition, pp. 7862-7871, 2021b.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Junghyup Lee, Dohyung Kim, and Bumsub Ham. Network quantization with element gradient scaling. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 6448-6880 6457, 2021.
Yunqiang Li, Silvia-Laure Pintea, and Jan van Gemert. Equal bits: Enforcing equally distributed binary network weights. In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 36, pp.
Mingbao Lin, Rongrong Zihan Xu, Baochang Zhang, Yan Wang, Yongjian Wu, Feiyue Huang, and Chia-Wor Lin. Rotated binary neural network. ECCV, 2020.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In NeurlPS, 2017.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang. Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1 1-bit cnns with improved representational capability and advanced training algorithm. In ECCV, 2018.
Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: Towards precise binary neural network with generalized activation functions. European conference on computer vision, pp. 143-159. Springer, 2020.
Zechun Liu, Zhiqiang Shen, Shichao Li, Koen Helwegen, Dong Huang, and Kwang-Ting Cheng. How do adam and training strategies help bnns optimization. In International Conference Machine Learning, pp. 6936-6946. PMLR, 2021a.
Zechun Liu, Zhiqiang Shen, Shichao Li, Koen Helwegen, Dong Huang, and Kwang-Ting Cheng- How do adam and training strategies help bnns optimization? In International Conference on Machine Learning. PMLR, 2021b.
Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling. Re- laxed quantization for discretized neural networks. arXiv preprint arXiv:1810.01875 2018.
Brais Martinez, Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos. Training binary neural networks with real-to-binary convolutions. ICLR, 2020.
Xiangming Meng, Roman Bachmann, and Mohammad Emtiyaz Khan. Training binary neural net- works using the bayesian learning rule. In International conference on machine learning, pp- PMLR, 2020.
Jorn WT Peters and Max Welling. Probabilistic binary neural networks. arXiv preprint arXiv: 03368, 2018
Hai Phan, Yihui He, Marios Savvides, Zhiqiang Shen, et al. Mobinet: mobile binary network for image classification. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 3453-3462, 2020a.
Hai Phan, Zechun Liu, Dang Huynh, Marios Savvides, Kwang-Ting Cheng, and Zhiqiang Shen. Binarizing mobilenet via evolution based searching. In CVPR, 2020b.
John G Proakis. Digital signal processing: principles algorithms and applications. Pearson Educa- tion India, 2001.
Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. CVPR, 2020.
11

---

Published as a conference paper at ICLR 2023
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525-542- Springer, 2016.
Alexander Shekhovtsov and Viktor Yanush. Reintroducing straight-through estimators as principled methods for stochastic binary networks. In DAGM German Conference on Pattern Recognition, pp. 111-126. Springer, 2021.
Cuauhtemoc Daniel Searez-Ramirez, Miguel Gonzalez-Mendoze Leonardo Chang, Gilberto Ochoa-Ruiz, and Mario Alberto Ouran-Vega. A bop and beyond: second order optimizer for binarized neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1273-1281, 2021.
Ily Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initializa- tion and momentum in deep learning. In International conference on machine learning, 2013.
Wei Tang, Gang Hua, and Liang Wang. How to train compact binary neural network with high accuracy? In AAAI, pp. 2625-2633 2017.
Ziwei Wang, Jiwen Lu, Chenxin Tao, Jie Zhou, and Qi Tian. Learning channel-wise interactions for binary convolutional neural networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 568-577, 2019.
Zihan Xu, Mingbao Lin, Jianzhuang Liu, Jie Chen, Ling Shao, Yue Gao, Yonghong Tian, and Rongrong Ji. Recu: Reviving the dead weights in binary neural networks. ICCV, 2021.
Xingyi Yang. Stochastic gradient variance reduction by solving filtering problem. arXiv preprint arXhv:2012.124418 2020.
Yichi Zhang, Zhiru Zhang, and Lukasz Lew. Pokebnn: A binary pursuit of lightweight accuracy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12475-12488 2022.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. CoRR, 2016.
A APPENDIX
A.1 CASCADED EMAS AS 2ND ORDER FILTER
Here provide alternative solution for expressing cascaded EMAs as 2nd order filter. We first express zi time step as:
9 =(1 a)gi-1 + am,
=(1 a)::-1111111 + a (1 m) + Vo.
(19)
= a0::-)111111 + Vo + (1 7)am;;1
In time step i 1, we denote 9i-1 as:
Ji-1 =(1 - -0)ggg-2 + am;-1
(20)
where we have:
am-1 = a0)ii2
(21)
Writing the am4-1 in Eq. (21) into Eq. (19) get:
9. =ayVa, Va, (1 alg (1 (1-7) gi-1 (1 -a)9i22)
(22)
V.. - (a+ 2)g: (a 11)(7 1)9;
12


Reviews:
[{"review_id": "yBhi5G8drE", "reviewer": "Reviewer_H9io", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "For binary neural networks, some hyperparameters and optimization tricks such as weight decay lose their meaning. In this paper, the authors reinterpret these methods through a perspective of gradient filtering. Specifically, if the binary weights are taken to be the sign of latent weights the authors show that SGD with momentum and weight decay is equal to smoothing the gradient twice with an EMA filter. \n\n\n", "summary_of_the_review": "The paper visits an interesting topic and brings some relevant and thoughtful, albeit maybe simple, mathematical arguments which illuminate how hyperparameters interact when using SGD for binary networks. It is not clear how the analysis extends to adaptive optimizers, and what exactly the proposed method is is also not clear. While the authors clearly verify the theoretical results, the method does not seem to bring any performance benefits over two-stage methods. ", "strength_and_weaknesses": "Strengths:\n\n* The authors provide a clear and intuitive explanation for how hyperparameters interact when training binary networks with SGD.\n* The authors clearly verify their mathematical arguments experimentally.\n\nWeaknesses:\n\n* The mathematical arguments are rather simple and not surprising. \n* It is not clear how the analysis extends to adaptive optimizers such as Adam. Note that Adam is used for the imagenet experiments.\n* The authors refer to “our optimizer”, but it seems to me like this is just a way to parametrize well-known methods. \n* Only results matching SOTA are reached.\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "xWAlROZFFK", "reviewer": "Reviewer_YST8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides an analysis of standard optimization hyperparameters like weight initialization, learning rate, weight decay,  and momentum in the context of the latent-weight free interpretation of binary neural networks.\n\nIn a binary neural network, all weights and activations are +1 or -1. Although the weights are binary, they are typically optimized with real-valued gradients. In the latent-weight free interpretation, optimization can be viewed as a weight-flipping algorithm. When the gradients have accumulated past a certain threshold, the optimizer flips the weights.\n\nBut what should our understanding of traditional optimizer hyperparameters like weight initialization and learning rate be if we are to view optimization as weight-flipping? This paper argues that we should interpret them as impulse response filters on the gradients. This understanding reduces the number of hyperparameters that need to be tuned.", "summary_of_the_review": "This paper provides an interesting new understanding of optimization in the binary neural network setting. However, it also fails to show significant empirical benefits from such an understanding.\n\nI am on the fence about recommending this paper for acceptance, because it's not clear to me how the community can use this newfound understanding to significantly advance our theoretical understanding of binary neural networks or their empirical application. However, I also have low confidence in my assessment. I think the authors can improve the paper by adding new empirical or theoretical results, or at least sketching out how their contribution is significant to the community.\n\nEdit: Thanks for the authors' response. The quotes from relevant papers in the literature has increased my confidence in the significance of the work. Even though the empirical contributions are still weak, I see the great value this piece of work brings in filling this gap in the literature. I have updated my score to reflect that. I highly encourage the authors to re-write the Introduction section to emphasize the significance of this work by highlight the gaps in the literature and how this work fills them.", "strength_and_weaknesses": "Strengths:\n- This paper builds upon the latent-weight free interpretation of binary neural networks, and provides a new understanding of traditional optimization hyperparameters in such a setting.\n\nWeaknesses:\n- This paper fails to demonstrate a significant empirical benefit from their new interpretation of binary neural network optimization beyond a reduction in the number of hyperparameters in some cases.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "QGgvb-qlf1", "reviewer": "Reviewer_gyrn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the analysis of hyperparameters in binary neural networks (BNN). The hyperparameters could explain the BNNs with latent real-valued weights during training. However, the magnitude of binary weights is not interpretable with these hyperparameters yet. This paper provides an interpretation of these magnitude-based hyperparameters based on higher-order gradient filtering during network optimization.", "summary_of_the_review": "Overall, this paper provides some interesting insights for interpreting binary weights of BNN to magnitude-based hyperparameters. However, the major concern arises in the lack of experiments to support their statements. Also, the proposed optimizer does not offer many advantages compared with the latent-weight-based optimization.\n\n**Post after rebuttal**\n\nI appreciate the additional experimental results and detailed responses provided by the authors. After checking the latest revision in the manuscript, I decided to raise my score to encourage further exploration of hyperparameters for binary neural networks. ", "strength_and_weaknesses": "Pros:\n- This paper has good motivation for interpreting the magnitude-based hyperparameters on BNN. \n- It provides some interesting insights such as: \n  - A gradient filtering perspective on latent weight hyperparameters.\n  - A justification of which magnitude-based hyperparameters to use and offer a setting with fewer hyperparameters to tune.\n\nCons:\n- It would be better to add a summary paragraph in section 3 to state their concise conclusion of magnitude-based hyperparameters.\n- As mentioned, the performance improvement is marginal in Table 2 based on ReActNet-A. \n- The paper claimed that it provided a “simplified” setting with fewer hyperparameters to tune. However, I did not see the big advantages of the proposed optimizer compared to latent-weight optimization in terms of accuracy, and training/optimization speed.\n- The experiment section is relatively weak as the experiment is given on one architecture based on a few baseline methods, “ReActNet-A” on Imagenet and “Bi-Real” on CIFAR. It is expected to show how the proposed optimizer performs on several recent BNN optimization methods with various types of architecture and different types of datasets (modality such as visual and text, distribution such as natural, digital, fine-grained) and task (classification, detection, and segmentation.)\n- The metrics in Table 2 haven't been explained well. More details should be given.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6GxB-lqqf3", "reviewer": "Reviewer_AvuJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Binary Neural Networks use values of -1 and +1 for the network's weights instead of floating point number to save on memory. Successful training of these networks, and their ultimate performance, depends on several hyperparameters. This paper redefines or removes many of these hyperparameters. This means that there are fewer hyperparameters to tune and removes the need for latent weights. Experiments support that there is little loss in performance compared to methods with latent weights.", "summary_of_the_review": "The paper is clear and presents a nice idea. The experiments support the main claim of the paper but are limited in scope. I do not know the literature on Binary Neural Networks well enough to comment on the paper's novelty.", "strength_and_weaknesses": "**Strengths**\n- The main idea of the paper is simple.\n- Removing hyperparameters and reducing the need for tuning is important---especially in architectures that are large or expensive to train.\n- The paper is easy to follow.\n\n**Weaknesses**\n- Experiments are limited to two models and datasets, both of which are image classification.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster