Published as a conference paper at ICLR 2023

U NDERSTANDING WEIGHT- MAGNITUDE
HYPERPARAMETERS IN TRAINING BINARY NETWORKS
Joris Quist1 , Yunqiang Li1,2 , Jan van Gemert1
1. Computer Vision Lab, Delft University of technology; 2. Axelera AI

A BSTRACT
Binary Neural Networks (BNNs) are compact and efficient by using binary
weights instead of real-valued weights. Current BNNs use latent real-valued
weights during training, where hyper-parameters are inherited from real-valued
networks. The interpretation of several of these hyperparameters is based on
the magnitude of the real-valued weights. For BNNs, however, the magnitudeof binary weights is not meaningful, and thus it is unclear what these hyperparameters actually do. One example is weight-decay, which aims to keep themagnitude of real-valued weights small. Other examples are latent weight initialization, the learning rate, and learning rate decay, which influence the magnitude of the real-valued weights. The magnitude is interpretable for real-valued
weights, but loses its meaning for binary weights. In this paper we offer a newinterpretation of these magnitude-based hyperparameters based on higher-order
gradient filtering during network optimization. Our analysis makes it possible
to understand how magnitude-based hyperparameters influence the training ofbinary networks which allows for new optimization filters specifically designed
for binary neural networks that are independent of their real-valued interpretation. Moreover, our improved understanding reduces the number of hyperparameters, which in turn eases the hyperparameter tuning effort which may lead to better hyperparameter values for improved accuracy. Code is available at https://github.com/jorisquist/Understanding-WM-HP-in-BNNs

I NTRODUCTION

1.5

1

0.5
0

1.0
0.5

0.0
0.5
Weight 1

1.0

1.0

The latent weight optimization depends on several essential hyperparameters, such as their initialization, learning
rate, learning rate decay, and weight decay. These hyperparameters are important for BNNs, as shown for example in Martinez et al. (2020), and also by Liu et al.
(2021a), who both improve BNN accuracy by better tuning these hyperparameters.

-1

0.5

2

2

ht

0.5

0.5

0

ht

1.0

1

We
ig

0.0

Magnitude

1.4
1.2
1.0
0.8
0.6
0.4
0.2
0.0

Magnitude

A Binary Neural Network (BNN) weight is a single bit:
−1 or +1, which is compact and efficient, enabling applications on, for example, edge devices. Yet, training BNNs
using gradient descent is difficult because of the discrete
binary values. Thus, BNNs are often (Kim et al., 2021b;
Liu et al., 2020; Martinez et al., 2020) optimized with socalled ‘latent’, real-valued weights, which are discretisedto −1 or +1 by, e.g., taking the positive or negative signof the real value.

-0.5
-0.5

0
Weight 1

0.5

1

We
ig

1

-1

Figure 1: Changes in real-valued
weights change their magnitude. For binary weights, however, the magnitude
will never change and magnitude-based
hyperparameters need reinterpretation.

In this paper we investigate the latent weight hyperparameters used in a BNN, including initialization, learning rate, learning rate decay, and weight decay. All these hyperparameters influence themagnitude of the latent weights. Yet, as illustrated in Fig 1, in a BNN, the binary weights are −1
or +1, which always have a constant magnitude and thus magnitude-based hyperparameters lose
their meaning. We draw inspiration from the seminal work of Helwegen et al. (2019), who reinterpret latent weights from an inertia perspective and state that latent weights do not exist. Thus, the1

Published as a conference paper at ICLR 2023

magnitude of latent weights also does not exist. Here, we investigate what latent weight-magnitude
hyperparameters mean for a BNN, how they relate to each other, and what justification they have.
We provide a gradient-filtering perspective on latent weight hyperparameters which main benefit
is a simplified setting: fewer hyperparameters to tune, achieving similar accuracy as current, more
complex methods.

2

R ELATED W ORK

Latent weights in BNNs. By tying each binary weight to a latent real-valued weight, continuous optimization approaches can be used to optimize binary weights. Some methods minimize thequantization error between a latent weight and its binary variant (Rastegari et al., 2016; Bulat &Tzimiropoulos, 2019). Others focus on gradient approximation (Liu et al., 2018; Lee et al., 2021;
Zhang et al., 2022), or on reviving dead weights (Xu et al., 2021; Liu et al., 2021b), or on entropy
regularization (Li et al., 2022) or a loss-aware binarization (Hou et al., 2017; Kim et al., 2021a).
These works directly apply traditional optimization techniques inspired by real-valued network suchas weight decay, learning rate and its decay, and optimizers. The summary of De Putter & Corporaal(2022) gives a good overview of these training techniques in BNNs. Recently, some papers (Liuet al., 2021a; Martinez et al., 2020; Hu et al., 2022; Tang et al., 2017) noticed that the interpretation of these optimization techniques does not align with the binary weights of BNNs (Lin et al.,2017; 2020). Here, we aim to shed light on why, by explicitly analyzing latent weight-magnitude
hyperparameters in a BNN.
Latent weight magnitudes. Several techniques exploit the magnitude of the latent weights during
BNN optimization. Latent weights clipping is proposed in (Courbariaux et al., 2015) and followedby its extensions (Alizadeh et al., 2018; Hubara et al., 2016) to clip the latent weights within a [−1, 1]
interval to prevent the magnitude of latent weights from growing too large. Gradient clipping (Caiet al., 2017; Courbariaux et al., 2015; Qin et al., 2020) stops gradient flow if the magnitude of latent
weight is too large. Work on latent weight scaling (Chen et al., 2021; Qin et al., 2020) standardizes
the latent weights to a pre-defined magnitude. Excellent results are achieved by a two-step training
strategy (Liu et al., 2021a; 2020) that in the first step trains the network from scratch using only
binarizing activations with weight decay, and then in the second step they fine-tune by training
without weight decay. Our method reinterprets the meaning of the magnitude based weight decay
hyperparameter in optimizing BNNs from a gradient filtering perspective, offering similar accuracy
as two step training with a simpler setting, using just a single step.
Optimization by gradient filtering. Gradient filtering is a common approach used to tackle thenoisy gradient updates caused by minibatch sampling. Seminal algorithms including Momentum (Sutskever et al., 2013) and Adam (Kingma & Ba, 2015) which use a first order infinite impulse
response filter (IIR), i.e. exponential moving average (EMA) to smooth noisy gradients. Yang (2020)
takes this one step further and introduces the Filter Gradient descent Framework that can use different types of filters on the noisy gradients to make a better estimation of the true gradient. In binary
network optimization, Bop (Helwegen et al., 2019) and its extension (Suarez-Ramirez et al., 2021)
introduce a threshold to compare with the smoothed gradient by EMA to determine whether to flipa binary weight. In our paper, we build on second order gradient filtering techniques to reinterpretthe hyperparameters that influence the latent weight updates.
Sound optimization approaches. Instead of using heuristics to approximate gradient descent on
discrete binary values, several works take a more principled approach. Peters & Welling (2018)
propose a probabilistic training method for BNN, and Shekhovtsov & Yanush (2021) present atheoretical understanding of straight through estimators (STE) (Bengio et al., 2013). Meng et al.
(2020) propose a Bayesian perspective and Louizos et al. (2018) formulate a noisy quantizer. Even
though these approaches provide more theoretical justification in optimizing BNNs, they are more
complex by either relying on stochastic settings or discrete relaxation training procedures. Moreover,these methods do not (yet) empirically reach a similar accuracy as current mainstream heuristic
methods (Liu et al., 2018; 2020). In our paper, we build on the mainstream approaches, to get good
empirical results, but add a better understanding of their properties, taking a step towards better
theoretical understanding of empirical approaches.
2

Published as a conference paper at ICLR 2023

3

H YPERPARAMETER ANALYSIS THROUGH GRADIENT FILTERING

We start with a latent weights BNN and convert it to an equivalent latent-weight free setting, as
in Helwegen et al. (2019). To do this, we use a magnitude independent setting, which means that nogradient-clipping or scaling based on the channel-wise mean of the latent-weights is used.
BNN setup. We use Stochastic Gradient Descent (SGD) with weight decay and momentum as astarting point, as this is a commonly used setting, see Rastegari et al. (2016), Liu et al. (2018), Qin
et al. (2020). Our setup is as follows:
w0 = init()

mi = (1 − γ)mi−1 + γ∇θi ,(1)wi = wi−1 − (mi + λwi−1 )(3)θi = sign(wi )
if x < 0;−1,
sign(x) = +1,
if x > 0;random{−1, +1} otherwise.

(2)
(4)
(5)

Here, wi is a latent weight at iteration i which is initialized at w0 . θi is a binary weight,  isthe learning rate, λ is the weight decay factor, mi is the momentum exponentially weighted moving average with m−1 = 0 and discount factor γ, ∇θi is the gradient over the binary weight andrandom{−1, +1} is a uniformly randomly sampled -1 or +1.
We then convert to the latent-weight free setting of Helwegen et al. (2019) where latent weights areinterpreted as accumulating negative gradients. We introduce gi = −wi , which allows working withgradients instead of with latent weights. We can then write Eq 3 as follows
gi = gi−1 + (mi − λgi−1 ).
(6)
Latent weight initialization. To investigate latent weight initialization we unroll the the recursion
in Eq 6 by writing it out as a summation:gi = (1 − λ)gi−1 + mi

=



i
X

(1 − λ)i−r mr .

(7)

r=0

Latent-weights are typically initialized using real-valued weight initialization techniques (Glorot &Bengio, 2010; He et al., 2015). However, since we now interpret latent weights as accumulated
gradients, we argue to also initialize them as gradient accumulation techniques such as Momentum (Sutskever et al., 2013) and simply initialize w0 = g0 = 0, because at initialization there is
no preference for negative or positive gradients, and their expectation is 0. We do not use a biascorrection as done in (Kingma & Ba, 2015) because in practice we noticed that gradient magnitudes
are large in the first few iterations. Applying bias correction increases this effect, which had a negative effect on training. To prevent all binary weights θ to start at the same value, we use the stochastic
sign function in Eq 5 that randomly chooses a sign when the input is exactly 0.
Learning rate and weight decay. The learning rate  appears in two places in Eq 7: once outside
the summation, and once inside the summation. The  outside the summation can only scale thelatent weight and will not influence outcome of the sign in Eq 4 as
!
!
i
i
X
X
i−r
i−r
sign 
(1 − λ) mr = sign
(1 − λ) mr .
(8)
r=0

r=0

Thus, the leftmost  can be removed, or set randomly without influencing the training process.
For the  inside the summation of Eq 7, it appears together with the weight decay term λ. Thus,
there are two free hyperparameters that only control one factor, therefore one of them is redundantand can use a single combined hyperparameter α = λ. Instead of setting a value for the learning
rate , and setting a value for the weight decay λ, we now only have to set a single value for α. Since
Eq 8 shows us that we can freely scale the sum with any constant factor, we scale it with α, as
gi = αiX(1 − α)i−r mr ,
r=0

3

(9)

Published as a conference paper at ICLR 2023

which allows us to re-write the sum with a recursion, as an exponential moving average (EMA) as
gi = (1 − α)gi−1 + αmi ,(10)where g−1 = 0. This shows that for BNNs under magnitude independent conditions, SGD withweight decay is just a exponential moving average. This gives a magnitude-free justification forusing weight decay since its actual role is to act as the discount factor in an EMA. Note that it is nolonger possible to set α to 0 since then there are no updates anymore, but setting to a small (10−20 )number will essentially work the same. The meaning of α is now clear, as in the EMA it controls
how much to take the past into account.
Learning rate decay. There no longer is a learning rate to be decayed, however, since learning
rate decay scales the learning rate and α = λ, now the learning rate decay directly scales α, sofrom now on we apply it to alpha and will refer to it as α-decay. This also helps better explain its
function: α-decay increases the window size during training, causing the filtered gradient to become
more stable and allowing the training to converge.
Momentum. Now adding back the momentum term of Eq 2 in the original setup yields
mi = (1 − γ)mi−1 + γ∇θi ,gi = (1 − α)gi−1 + αmi ,θi = −sign(gi ).

(11)
(12)
(13)

Thus, SGD with weight decay and momentum is smoothing the gradient twice with an EMA filter.
Latent weight optimization as a second order linear infinite impulse response filter. EMAs
are a specific type of linear Infinite Impulse Response (IIR) Filter (Proakis, 2001). Linear filters
are filters that compute an output based on a linear combination of current and past inputs and past
outputs. The general definition is given as a difference equation:yi =1(b0 xi + b1 xi−1 + ... + bP xi−P − a1 yi−1 − a2 yi−2 − ... − aP yi−Q ),
a0

(14)

where i is the time step, yi are the outputs, xi are the inputs, aj and bj are the filter coefficients and
P and Q are the maximum of iterations the filter looks back at the inputs and outputs to computethe current output. The maximum of P and Q defines the order of the filter. An EMA only looks at
the previous output and the current input, so is therefore a first order IIR filter. Expressing an EMA
as a filter looks as follows:
 


1
α
1
,a =
.
(15)
yi = (1 − α)yi−1 + αxi = (b0 xi − b1 · xi−1 − a1 yi−1 ), b =0α−1a0In our optimizer we have a cascade of two EMAs applied in series to the same signal which can be
represented by a filter with the order being the sum of the orders of the original filters. To get the
new a and b vectors the original ones are convolved with each other. In our case this gives:    "αγ #
γ
α
b=
?
= 0 ,
0
0
0

#
 
 "
1
1
1
a=
?
= (α − 1) + (γ − 1) ,γ−1α−1(α − 1) · (γ − 1)(16)when applied to our gradient filtering setting in Eq 12 gives the difference equation:gi = αγ∇θi − (α + γ − 2)gi−1 − (α − 1)(γ − 1)gi−2(17)Thus, in a magnitude independent setting, SGD with weight decay and momentum is equivalent to a
2nd order linear IIR filter. Note that α and γ have the same function: Without α decay, the values for
α and γ can be swapped. This filtering perspective opens up new methods of analysis for optimizers.
Main takeaway. Our re-interpretations reduces the 7 hyper parameters in the latent weight view
with SGD, to only 3 hyperparameters in our filtering view, see Table 1.
4

Published as a conference paper at ICLR 2023

Learning rate

Learning rate decay

Init

Momentum

Weight decay

Scaling

Clipping–-decayα-decay

w0
–

γ
γ

λ
α

X
–

X
–

Latent:Filtered:

Filtered Gradients

Gradient

0.0002

2nd order (gradients= 10 3,= 10 1)

1e 6
1st order ( = 10 4)

1st order ( = 10 3)

1.0

Filtered Gradient

Table 1: Hyperparameters used in the latent weight view versus our filtered gradients perspective.
Our filtered perspective reduces the number of hyperparameters from 7 to 3.

0.00010.5

0.00000.0

0.0001

0.5
0

20

40

60

Iteration

80

100

1.0

Figure 2: Gradients and filtered gradients for a first and second order filter in a single epoch on
CIFAR-10. For better visualisation, the filter outputs are scaled up to a similar range as the unfilteredgradients. It can be seen that the unfiltered gradients are noisy and that the filtered outputs aresmoother. The second order filter reduces the noise even further compared to the first order filter.

4

E XPERIMENTS

We empirically validate our analysis on CIFAR-10, using the BiRealNet-20 architecture (Liu et al.,2018). Unless mentioned otherwise the networks were optimized using SGD for both the real-valued
and binary parameters with as hyperparameters: learning rate=0.1, momentum with γ = (1 − 0.9),
weight decay=10−4 , batch size=256 and cosine learning rate decay and cosine alpha decay. We
analyze the weight flip ratio at every update, which is also known as the FF ratio (Liu et al. (2021a)).
IFF =

PL P

|sign(wi+1 ) − sign(wi )|1
2

FFratio =l=1w∈Wl IFF(18)

Ntotalwhere wi is a latent weight at time i, L the number of layers, Wl the weights in layer l, and Ntotal
the total number of weights.

learning rate
1e-02
1e-01
1e+00
1e+01
1e+02
1e+20

FF Ratio

0.00200.00150.00100.00250.00050.0000Init scaling
1e+02
1e+01
1e+00
1e-01
1e-02
1e-20

0.0020FF Ratio

0.00250.00150.00100.0005

0

50

100

150

200 250
Epoch

300

350

0.0000400(a) Varying learning rates, constant init.

0

50

100

150

200 250
Epoch

300

350

400

(b) Vary init, constant learning rate (=1.0).

Figure 3: In the magnitude independent setting, scaling the learning rate has the exact same effecton the flipping ratio as scaling the initial latent-weights by the inverse.
5Published as a conference paper at ICLR 2023

85
Top-1 (%)

80
75
70

Magnitude Dependent
Magnitude Independent
Magnitude Independent - 0 init

65
10 3

10 1

10 2

101
100
102
Learning Rate

104

103

1020

Figure 4: Learning rate effect on accuracy for three settings. (a). standard SGD. (b) Magnitude
independent in Eq 3, by removing clipping and scaling. (c) Magnitude independent with latent
weight initializes of 0 in Eq 7. Setting (a) has just a single optimum. The accuracy in setting (b) is
sensitive to small learning rates. For setting (c), accuracy is independent of the learning rate.
0.035

0.025

0.8
Validation Accuracy (%)

0.030

FF Ratio

0.9

alpha
1e-06
1e-05
1e-04
1e-03

0.020
0.015
0.010
0.005

0.7
0.6
0.5
0.4
0.3
0.2
0.1

0.000
0

50

100

150

200 250
Epoch

300

350

400

0

50

100 150 200 250 300 350 400
alpha

Figure 5: Bit flipping (FF) ratio and accuracy for varying alphas from eq 17. BNNs are sensitive toalpha, similar to how they are sensitive to weight decay. Tuning alpha is essential.

1st order vs 2nd order We visually compare filter orders by sampling real gradients from a single
binary weight trained on CIFAR-10 in Figure 2. For the same α, a 1st order filter is more noisy thana 2nd order filter. This may cause the binary weight to oscillate, even though the larger trend is thatit should just flip once. To reduce these oscillations with a 1st order filter requires a smaller alpha.
This, however, causes other problems because α determines the window size of past gradients andwith a smaller α many more gradients are used. This means that it takes much longer for a trend in
the gradients to effect the binary weight. Instead, the 2nd order filter has both benefits: it can filter
out high frequency noise while still able to react quicker to changing trends.
Magnitude independent learning rate vs initialization In Figure 3 we show the bit flipping ratio for the learning rate  and the initialization
g0 . Multiplying  with some 
scal
Pi
i
i−r
ing factor s is the same as dividing g0 by s: sign (1 − λ) g0 + s r=1 (1 − λ) mr =Pisign (1 − λ)i gs0 +  r=1 (1 − λ)i−r mr , because the magnitude has no effect on the sign.
Larger  in Figure 3(a) and smaller g0 in Figure 3(b) are independent to scaling and have similar
flipping ratios. A too small  or too large g0 do not reach the same flipping ratios, because their ratio
is insufficient to update the binary weights. For sufficiently large ratios it means that scaling both and g0 has no effect on training, but also that scaling the  or scaling g0 with the inverse is identical:
as seen by comparing the two plots in Figure 3.

6

Published as a conference paper at ICLR 2023

FF Ratio

Validation Accuracy (%)

Sensitivity to hyperparameters for
0.00250.8
weight magnitude (in)dependence
0.00200.7
We evaluate SGD in the standard
0.0015
magnitude dependent setting with
0.6
0.0010clipping and scaling vs a magnitude
0.5 alpha schedulealpha schedule
independent setting with initializing
none
none
0.00050.4
cosinecosinethe latent-weights to zero. To keep
0.3
0.0000
0
100
200
300
400
0
100
200
300
400
the effect of weight decay constant,
Epoch
Epoch
we scale the weight decay factor inversely with the learning rate. Resultsin Figure 4 show that for the stan- Figure 6: Flipping rate and accuracy for alpha decay. Withdard magnitude dependent setting the decay the FF ratio goes to zero. Without decay, the flipping
learning rate  has to be carefully bal- will continue, preventing convergence, reducing accuracy.
anced. A too small  w.r.t. to the initial weights inhibits learning; while a too large  will push latent-weights to the clipping region andwill stop updating. In the magnitude independent setting there is no clipping. A too small , howeveris still problematic because the magnitudes of the gradients are smaller when not using the scaling
factor and the accuracy drops significantly. When initializing to zero, as we propose, this problem
disappears, because there is no initial weight to hinder training and all learning rates perform equal.
Alpha In Fig 5 we vary α from eq 17. Alpha strongly influences training. A too large α causes too
many binary weight flips per update, which hinders converging. A too small α makes the network
converge too quickly, which hurts the end result. Tuning α is essential.

0.00200.00150.0010
Gradient Filtering
Latent Weights
100 200 300
Epoch

0.00050.0000

Validation Accuracy (%)

0.0025FF Ratio

Alpha decay For proper convergence the flipping (FF) ratio should
go to zero. We transform learning
rate decay to alpha decay. When α
becomes smaller, the gradients willchange less, causing fewer flips, forcing the network to converge.

0.8
0.7
0.6
0.5
0.4
0.3

Gradient Filtering
Latent Weights
100
200
300
Epoch

See the plots in Figure 6 where one
0
400
0
400
network has been trained with cosinealpha decay and one without alphadecay. With and without alpha decay Figure 7: Equivalence of latent weights (Eq 3) and our graboth seem to perform well at the start dient filtering (Eq 17). They are empirically equivalent.
of training, however, the variant without alpha decay plateaus at the end of
training while the BNN with alpha decay converges better and continues improving, leading to abetter end result.
Equivalent interpretation Figure 7 shows empirical validation with matching hyperparametersthat the SGD setting using latent weights in Eq 3 is equivalent to our gradient filtering interpretationin Eq 17 that no longer uses latent weights,
4.1

VALIDATION OF EQUIVALENCE TO THE CURRENT STATE OF THE ART

We validate on for CIFAR-10 and Imagenet that our filtering-based optimizer is similar to the current
state of the art. Several current methods use an expensive two-step optimization step. We aim toshow the value of our re-interpretation by showing similar accuracy but only in a single step.
CIFAR-10: We train all networks for 400 epochs. As data augmentation we use padding of 4 pixels,
followed by a 32x32 crop and random horizontal flip. We use Bi-RealNet-20, and for the real-valued
parameters and latent-weights when used, we use SGD with a learning rate of 0.1 with cosine decay,
momentum of 0.9 and on the non-BN parameters a weight decay of 10−4 . For our filtering-based
optimizer we used an alpha of 10−3 with cosine decay and a gamma of 10−1 . Results in Table 2
show that our re-interpretation achieves similar accuracy.
7

Published as a conference paper at ICLR 2023

Table 2: State-of-the-art on CIFAR-10.
The ? denotes that we re-ran these experiments ourselves.
Training Bit-width Top-1
Strategy (W/A) Acc(%)

Method
FP

32/32

91.7

DoReFa-Net (Zhou et al., 2016)
DSQ (Gong et al., 2019)
IR-Net (Qin et al., 2020)
One step
Bi-Real? (Liu et al., 2018)
Bi-Real + Our filtering optimizer

1/1
1/1
1/1
1/1
1/1

79.3
84.1
86.5
85.0
86.5

Bi-Real? (Liu et al., 2018)

1/1

86.7

Two step

Table 3: Comparison with state-of-the-art on Imagenet.
MethodTraining Top-1 Top-5
Strategy Acc(%) Acc(%)

CI-BCNN (Wang et al., 2019)
Binary MobileNet (Phan et al., 2020b)
MoBiNet (Phan et al., 2020a)
EL (Hu et al., 2022)
MeliusNet29 (Bethge et al., 2020)
ReActNet-A + Our filtering optimizer

59.9
60.9
54.4
One step
56.4
65.8
69.7

StrongBaseline (Martinez et al., 2020)
Real-to-Binary (Martinez et al., 2020)
Two step
ReActNet-A (Liu et al., 2020)
ReActNet-A-AdamBNN (Liu et al., 2021a)

60.9
65.4
69.4
70.5

84.2
82.6
77.5
–
–
88.9
83.0
86.2
88.6
89.1

Imagenet: We follow Liu et al. (2021a): We train for 600K iterations with a batch size of 510.
For the real-valued parameters we use Adam with a learning rate of 0.0025 with linear learning
rate decay. For the binary parameters we use our 2nd order filtering optimizer with α = 10−5 ,which we decay linearly and γ = 10−1 . We do not use two-step training to pre-train the latentweights. Results in Table 3, show that ReActNet-A with our optimizer compares well to other one
step training methods. It approaches the accuracy of two step training approaches, albeit without anadditional expensive second step of training.
4.2

E MPIRICAL ADVANTAGE OF HAVING FEWER HYPERPARAMETER TO TUNE

Our filtering perspective significantly reduces the number of hyperparameters (Table 1). Here, we
empirically verify the computational benefit of having fewer hyperparameters to tune when appliedin a setting where the hyperparameters are unknown. To show generalization to other architecturesand modality we use an audio classification task (Becker et al., 2018) with a fully connected network.
Specifically, we use 4 layers with batch normalization of which the first layer is not binarized. For
the latent weights we optimize 6 hyperparameters, and for our filtering perspective we optimize two
hyperparameters, see Table 1. We did not tune learning rate decay as this had no effect on bothmethods. To fairly compare hyperparameter search we used Bayesian optimization (Balandat et al.,2020). The results for 25 trials of tuning, averaged over 20 independent runs are in Figure 8. We
confirm that both perspectives achieve similar accuracy when their hyperparameters are well tuned.
Yet, for the latent weights, it takes on average around 10 more trials when compared to the gradientfiltering. This means that the latent weights would have to train many more networks, which on
medium-large datasets such as Imagenet would already take several days to converge. In contrast,
the gradient filtering requires much less time and energy to find a good model.

5

D ISCUSSION AND LIMITATIONS

One limitation of our work is that we do not achieve “superior performance” in terms of accuracy.
Our approach merely matches the state of the art results. Note, however, that our goal is to provide
insight into how SGD and its hyperparameters behave, not to improve accuracy. Our analysis ended
up with an optimizer with less hyperparameters, that also have a better explanation in the context of
BNN optimization leading to simpler, more elegant methods. Our main empirical contribution is in
the significant computational reduction in hyperparameter tuning.
Another perceived limitation is that our new proposed optimizer can be projected back to a specificsetting within the current SGD with latent-weights interpretation. Thus, our analysis might not beneeded. While it is true that latent-weights can also be used, we argue that there is no disadvantage
to switching to the filtering perspective, because the options are the same, but the benefit is that
our hyperparameters make more sense. The option to project back to latent-weights also works theother way around and for those who already have a well tuned SGD optimizer could use it to makeit easier to switch to our filtering perspective. The benefit of our interpretations is having fewer
hyperparameters to set.
8

Published as a conference paper at ICLR 2023

90
80

Accuracy (%)

70
60
50
40
30
20

Gradient Filtering
Latent Weights

10
1

3

5

7

9

11

13
Trial

15

17

19

21

23

25

Figure 8: Bayesian hyperparameter optimization for our gradient filtering vs. latent weights. We
optimize the hyperparameters of Table 1, except learning rate decay and alpha decay, as they had
no influence. The scatter plot show the best achieved result so far at each trial for all runs. Thelines show the average best result, up to the current trial. Both methods perform equally for good
hyperparameters. Our gradient filtering view needs much less trials.
Its also true that our method cannot use common techniques based on the magnitude such as weightclipping or gradient clipping. Yet, we do not really think these techniques are necessary. We seesuch methods as heuristics to reduce the bit flipping ratio over time, which helps with convergence.
However, in our setting, this can also be done using a good α decay schedule without reverting tosuch heuristics, making the optimization less complex.
We did not yet have the opportunity to test the filtering-based optimizer on more architectures anddatasets. However, since our optimizer is equivalent to a specific setting of SGD, we would argue
that architectures that have been trained with SGD will probably also work well with our optimizer.
This is also a reason why we chose to use ReActNet-A, since it was trained using Adam in bothin the original paper (Liu et al., 2020) and in Liu et al. (2021a). The latter specifically argues thatAdam works better for optimizing BNNs, but we suspect that the advantages of Adam are decreased
because it might not work in the same way in the magnitude invariant setting, as we see a smallerdifference in accuracy. Introducing this normalizing aspect into the filtering-based perspective is aninteresting topic for future research.
One last point to touch upon is soundness. Even though the filtering perspective provides a better
explanation to hyperparameters, it does not provide understanding on why optimizing BNNs withsecond-order low pass filters works as well as it does. Whereas stochastic gradient descent has
extensive theoretical background, this does not exist for current mainstream BNN methods. Fully
understanding BNN optimization is an interesting direction for future research and our hope is that
this work takes a step in that direction.
Ethics Statement We believe that this research does not bring up major new potential ethical concerns. Our work makes training BNNs easier, which might increase their use in practice.
Reproducibility Statement All our code is availabe online: https://github.com/
jorisquist/Understanding-WM-HP-in-BNNs. Two important things for better reproducing our results rely on the GPUs and the dataloader. The reproduction of our ImageNet experiments is not trivial. First, as the teacher-student model is used in our ImageNet experiments, itwill occupy a lot of GPU memory. We trained on 3 NVIDIA A40 GPUs, each A40 has 48 GB
of GPU memory, with a batch size of 170 per GPU for as much as ten days. Second, for fastertraining on ImageNet, we used NVIDIA DALI dataloader to fetch the data into GPUs for the image
pre-processing. This dataloader could effect training as it uses a slightly different image resizing
algorithm than the standard PyTorch dataloader. To keep results consistent with other methods, wedo the inference with the standard PyTorch dataloader.
9

Published as a conference paper at ICLR 2023

R EFERENCES
Milad Alizadeh, Javier Fernández-Marqués, Nicholas D Lane, and Yarin Gal. An empirical studyof binary neural networks’ optimisation. In International conference on learning representations,
2018.
Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. BoTorch: A Framework for Efficient Monte-Carlo
Bayesian Optimization. In Advances in Neural Information Processing Systems 33, 2020. URL
http://arxiv.org/abs/1910.06403.
Sören Becker, Marcel Ackermann, Sebastian Lapuschkin, Klaus-Robert Müller, and Wojciech
Samek. Interpreting and explaining deep neural networks for classification of audio signals. arXiv
preprint arXiv:1807.03418, 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients
through stochastic neurons for conditional computation. Technical Report, 2013.
Joseph Bethge, Christian Bartz, Haojin Yang, Ying Chen, and Christoph Meinel. Meliusnet: Can
binary neural networks achieve mobilenet-level accuracy? arXiv preprint arXiv:2001.05936,
2020.
Adrian Bulat and Georgios Tzimiropoulos. Xnor-net++: Improved binary neural networks, 2019.
Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision byhalf-wave gaussian quantization. In Proceedings of the IEEE conference on computer vision andpattern recognition, pp. 5918–5926, 2017.
Tianlong Chen, Zhenyu Zhang, Xu Ouyang, Zechun Liu, Zhiqiang Shen, and Zhangyang Wang. ”
bnn-bn=?”: Training binary neural networks without batch normalization. In Proceedings of theIEEE/CVF conference on computer vision and pattern recognition, pp. 4619–4629, 2021.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neuralnetworks with binary weights during propagations. In NeurIPS, 2015.
Floran De Putter and Henk Corporaal. How to train accurate bnns for embedded systems? arXiv
preprint arXiv:2206.12322, 2022.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neuralnetworks. In AISTATS, pp. 249–256, 2010.
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, andJunjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4852–4861,
2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing
human-level performance on imagenet classification. In ICCV, 2015.
Koen Helwegen, James Widdicombe, Lukas Geiger, Zechun Liu, Kwang-Ting Cheng, and Roeland Nusselder. Latent weights do not exist: Rethinking binarized neural network optimization.
Advances in neural information processing systems, 32, 2019.
Lu Hou, Quanming Yao, and James T Kwok. Loss-aware binarization of deep networks. ICLR,
2017.
Jie Hu, Ziheng Wu, Vince Tan, Zhilin Lu, Mengze Zeng, and Enhua Wu. Elastic-link for binarizedneural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36,
pp. 942–950, 2022.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Binarizedneural networks. Advances in neural information processing systems, 29, 2016.
Dohyung Kim, Junghyup Lee, and Bumsub Ham. Distance-aware quantization. In Proceedings ofthe IEEE/CVF International Conference on Computer Vision, pp. 5271–5280, 2021a.
10

Published as a conference paper at ICLR 2023

Hyungjun Kim, Jihoon Park, Changhun Lee, and Jae-Joon Kim. Improving accuracy of binaryneural networks using unbalanced activation distribution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7862–7871, 2021b.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR, 2015.
Junghyup Lee, Dohyung Kim, and Bumsub Ham. Network quantization with element-wise gradientscaling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6448–6457, 2021.
Yunqiang Li, Silvia-Laura Pintea, and Jan C van Gemert. Equal bits: Enforcing equally distributed
binary network weights. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 1491–1499, 2022.
Mingbao Lin, Rongrong Ji, Zihan Xu, Baochang Zhang, Yan Wang, Yongjian Wu, Feiyue Huang,
and Chia-Wen Lin. Rotated binary neural network. ECCV, 2020.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In
NeurIPS, 2017.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang, Wei Liu, and Kwang-Ting Cheng. Bi-real net:
Enhancing the performance of 1-bit cnns with improved representational capability and advanced
training algorithm. In ECCV, 2018.
Zechun Liu, Zhiqiang Shen, Marios Savvides, and Kwang-Ting Cheng. Reactnet: Towards precise
binary neural network with generalized activation functions. In European conference on computer
vision, pp. 143–159. Springer, 2020.
Zechun Liu, Zhiqiang Shen, Shichao Li, Koen Helwegen, Dong Huang, and Kwang-Ting Cheng.
How do adam and training strategies help bnns optimization. In International Conference on
Machine Learning, pp. 6936–6946. PMLR, 2021a.
Zechun Liu, Zhiqiang Shen, Shichao Li, Koen Helwegen, Dong Huang, and Kwang-Ting Cheng.
How do adam and training strategies help bnns optimization? In International Conference on
Machine Learning. PMLR, 2021b.
Christos Louizos, Matthias Reisser, Tijmen Blankevoort, Efstratios Gavves, and Max Welling. Relaxed quantization for discretized neural networks. arXiv preprint arXiv:1810.01875, 2018.
Brais Martinez, Jing Yang, Adrian Bulat, and Georgios Tzimiropoulos. Training binary neuralnetworks with real-to-binary convolutions. ICLR, 2020.
Xiangming Meng, Roman Bachmann, and Mohammad Emtiyaz Khan. Training binary neural networks using the bayesian learning rule. In International conference on machine learning, pp.
6852–6861. PMLR, 2020.
Jorn WT Peters and Max Welling.
arXiv:1809.03368, 2018.

Probabilistic binary neural networks.

arXiv preprintHai Phan, Yihui He, Marios Savvides, Zhiqiang Shen, et al. Mobinet: A mobile binary network
for image classification. In Proceedings of the IEEE/CVF Winter Conference on Applications ofComputer Vision, pp. 3453–3462, 2020a.
Hai Phan, Zechun Liu, Dang Huynh, Marios Savvides, Kwang-Ting Cheng, and Zhiqiang Shen.
Binarizing mobilenet via evolution-based searching. In CVPR, 2020b.
John G Proakis. Digital signal processing: principles algorithms and applications. Pearson Education India, 2001.
Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and JingkuanSong. Forward and backward information retention for accurate binary neural networks. In CVPR,
2020.
11

Published as a conference paper at ICLR 2023

Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet
classification using binary convolutional neural networks. In European conference on computer
vision, pp. 525–542. Springer, 2016.
Alexander Shekhovtsov and Viktor Yanush. Reintroducing straight-through estimators as principled
methods for stochastic binary networks. In DAGM German Conference on Pattern Recognition,
pp. 111–126. Springer, 2021.
Cuauhtemoc Daniel Suarez-Ramirez, Miguel Gonzalez-Mendoza, Leonardo Chang, Gilberto
Ochoa-Ruiz, and Mario Alberto Duran-Vega. A bop and beyond: a second order optimizer forbinarized neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, pp. 1273–1281, 2021.
Ilya Sutskever, James Martens, George Dahl, and Geoffrey Hinton. On the importance of initialization and momentum in deep learning. In International conference on machine learning, 2013.
Wei Tang, Gang Hua, and Liang Wang. How to train a compact binary neural network with highaccuracy? In AAAI, pp. 2625–2631, 2017.
Ziwei Wang, Jiwen Lu, Chenxin Tao, Jie Zhou, and Qi Tian. Learning channel-wise interactions forbinary convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition, pp. 568–577, 2019.
Zihan Xu, Mingbao Lin, Jianzhuang Liu, Jie Chen, Ling Shao, Yue Gao, Yonghong Tian, andRongrong Ji. Recu: Reviving the dead weights in binary neural networks. ICCV, 2021.
Xingyi Yang. Stochastic gradient variance reduction by solving a filtering problem. arXiv preprint
arXiv:2012.12418, 2020.
Yichi Zhang, Zhiru Zhang, and Lukasz Lew. Pokebnn: A binary pursuit of lightweight accuracy.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
12475–12485, 2022.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training
low bitwidth convolutional neural networks with low bitwidth gradients. CoRR, 2016.

AA PPENDIX

A.1

C ASCADED EMA S AS 2 ND O RDER F ILTER

Here we provide an alternative solution for expressing cascaded EMAs as 2nd order filter. We firstexpress gi in time step i as:
gi =(1 − α)gi−1 + αmihi=(1 − α)gi−1 + α (1 − γ)mi−1 + γ∇θi(19)=(1 − α)gi−1 + αγ∇θi + (1 − γ)αmi−1
In time step i − 1, we denote gi−1 as:
gi−1 =(1 − α)gi−2 + αmi−1(20)

αmi−1 =gi−1 − (1 − α)gi−2(21)where we have:

Writing the αmi−1 in Eq. (21) into Eq. (19) we get:higi =αγ∇θi + (1 − α)gi−1 + (1 − γ) gi−1 − (1 − α)gi−2=αγ∇θi − (α + γ − 2)gi−1 − (α − 1)(γ − 1)gi−2

12(22)