Sparse Binary Neural Networks
Authors: Riccardo Schiavone
Maria A Zuluaga
Venue:  (2021)
Keywords: Binary Neural Networks
Sparsity
Deep Neural Network Compression

Abstract: Quantized neural networks are gaining popularity thanks to their ability to solve complex tasks with comparable accuracy as full-precision Deep Neural Networks (DNNs), while also reducing computational power and storage requirements and increasing the processing speed. These properties make them an attractive alternative for the development and deployment of DNN-based applications in Internet-Of-Things (IoT) devices. Among quantized networks, Binary Neural Networks (BNNs) have reported the largest speed-up. However, they suffer from a fixed and limited compression factor that may result insufficient for certain devices with very limited resources. In this work, we propose Sparse Binary Neural Networks, a novel model and training scheme that allows to introduce sparsity in BNNs by using positive 0/1 binary weights, instead of the -1/+1 weights used by state-of-the-art binary networks. As a result, our method is able to achieve a high compression factor and reduces the number of operations and parameters at inference time. We study the properties of our method through experiments on linear and convolutional networks over MNIST and CIFAR-10 datasets. Experiments confirm that SBNNs can achieve high compression rates and good generalization, while further reducing the operations of BNNs, making it a viable option for deploying DNNs in very cheap and low-cost IoT devices and sensors.

Under review as a conference paper at ICLR 2021
SPARSE BINARY NEURAL NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Quantized neural networks are gaining popularity thanks their ability to solve complex tasks with comparable accuracy as full-precision Deep Neural Networks (DNNs), while also reducing computational power and storage requirements and increasing the processing speed. These properties make them an attractive alterna- tive for the development and deployment of DNN-based applications in Internet- Of-Things (IoT) devices. Among quantized networks, Binary Neural Networks (BNNs) have reported the largest speed-up. However, they suffer from fixed and limited compression factor that may result insufficient for certain devices with very limited resources. In this work, we propose Sparse Binary Neural Networks, novel model and training scheme that allows to introduce sparsity in BNNs by using positive 0/1 binary weights, instead of the -1/+1 weights used by state-of- the-art binary networks. As result, our method is able to achieve a high compres- sion factor and reduces the number of operations and parameters at inference time. We study the properties of our method through experiments lutional networks over MNIST and CIFAR-10 datasets. Experiments confirm that SBNNs can achieve high compression rates and good generalization, while further reducing the operations of BNNs, making it viable option for deploying DNNs in very cheap and low-cost IoT devices and sensors.
linear and convo-
Keywords Binary Neural Networks; Sparsity; Deep Neural Network Compression
INTRODUCTION
The term InternetOOf-Things (IoT) became notable in the late 2000s under the idea of enabling internet access to electrical and electronic devices (Miraz et al.] 2013). thus allowing them to collect and exchange data. Since its introduction, the number of connected devices has managed to surpass the number of humans connected to the internet (Evans) 20IT). The increasing number of both mobile and embedded loT devices has led to a sensors-rich world, capable of addressing a various number of real-time applications, such security systems, healthcare monitoring, environmental meters, factory automation, autonomous vehicles and many others, where both accuracy and time matter (Al-Fuqaha al. 2015).
At the same time, Deep Neural Networks (DNNs) have reached and surpassed state-of-the-ar re- sults for multiple tasks involving images and video (Krizhevsky et al./ 2012). speech (Hinton al.] 2012) or language processing (Collobert & Weston 2008). Thanks to their ability to process large and complex multiple heterogeneous data and extract patterns needed take autonomous decisions with high reliability (LeCun et al. 2015), DNNs have the potential of enabling a myriad of new loT applications. DNNs, however, suffer from high resource consumption. in terms of required compu- tational power, memory and energy consumption (Canziani et al. 2016). Instead, most IoT devices are characterized by their limited resources. They have limited processing power, small storage ca- pabilities, they are not GPU-enabled and they are powered with batteries of limited capacity, which are expected to last over 10 years without being replaced or recharged (Global System for Mobile Communications) 2018). All these important constraints remain important bottleneck towards deploying DNN models in loT applications (Yao et al. 2018).
Achieving deployment of DNNs in IoT devices requires to compress deep neural networks to fit on loT devices, while enabling real-time time "intelligent" interactions with the environment (Yao et al.] 2018) and without degrading their accuracy. Sparsity, compression and quantization, i.e. replacing

---

Under review as a conference paper at ICLR 2021
32 bit full-precision operations and values with fixed-poin ones, reduce inference time, storage size and power consumption of DNNs. This under the constraint of keeping high accuracy.
Different studies (Denil et al. 2013 Frankle & Carbin 2019) have demonstrated that deep models contain optimal subnetworks, which can perform the same task of their related super-network with less memory and computational burden. Among the various techniques used to extract these subnet- works, pruning and quantization have shown promising results. The first one is based on removing parameters obtain a sparser network, whereas the second focuses on reducing the bit-width rep- resent the parameters. Under this principle, Binary Neural Networks (BNNs) (Courbariaux et al., 2013) and Ternrry Neural Networks (TNNs) (Hwang & Sung, 2014) are two recently proposed quantized neural networks with weights and activation functions using one and two bits, respectively. This approach avoids multiplication operations in the forward propagation, which are well-known to be computationally expensive, and replaces them with low-cost bitwise operations. This allows to speed-up the resulting networks and to compress them. For instance, BNNs with binary weights {-1,+1} can reach a compression factor w.r.t. full-precision models of up to approximately 32 times (Rastegari et al. 2016). Despite this improvement the compression factor is upper bounded to 32, which is the result of representing the network's 's weights with 1-bit instead of the full- full-precision 32-bits, and may result insufficient for certain limited size and low power embedded devices.
To address this limitation, we introduce novel quantized model denoted Sparse Binary Neural Net- work (SBNN). It shares the advantages of BNNs as it performs quantization using only one bit, while also introducing sparsity. Our SBNN uses Os and 1 weights, instead of +1s and -1s (Courbariaux iet al. 2015 Rastegari et al.) 2016). allowing to reduce the total number of required operations, and achieve higher network compression rates and lower energy consumption at inference time. To achieve this, we propose a training scheme that starts from a "nearly-empty" model, rather than from fully connected models that prune their connections, as most of-t--of-the-at works do.
The remaining parts of this work is organized as follows. Section 2 discusses previous works to achieve sparsity and quantization DNNs. The core of our contribution is described in Section In Section 4 we study the properties of the proposed method and assess its performance, in terms of accuracy and compression results, through set of experiments using MNIST and CIFAR-10 10 datasets. Finally, discussion on the results and main conclusions are drawn in Section 5
RELATED WORK
We review different approaches to address sparsity and quantization in quantized networks.
Sparsity. The concept of sparsity has been well studied beyond quantized neural networks as it reduces computational and storage requirements of the networks and prevents overfitting. Meth- ods to achieve sparsity either explicitly induce it during learning through regularization, such as Lo (Louizos et al.] 22188) or L., (Han et al., 2015) regularization; do incrementally by gradually augmenting small networks (Bello, 1992); by post hoc pruning (Srivastava et al., 2014, Sriiivas) (ct al. 2017 Gomez et al. 2019). In the context of quantized networks, Han et al. (2016) proposed magnitude-pruning of the nearly-zero parameters from the trained full-precision dense model fol- lowed by quantization step of the remaining weights. The method achieved high compression rates of x35-499 and inference speed-up losses. However it has a time-consuming train-pruning stage and a relatively limited speed-up. which is model-dependent. [Tung& & Mori (2018) tried to optimize the scheme in HHan et al. (2016) reporting an improvement in accuracy. However, their method encountered a smaller compression factor. Regarding TNNs, this type of networks naturally performs magnitude- to their quantization function which maps real-valued weights to 1,,, O, +1}. Nevertheless, some works (Faraone al., 2017 Marban et al. 2020) have achieved larger compression rates by explic- itly inducing sparsity through regularization. Current BNN implementations do not address sparsity explicitly and focus on compression improvement through quantization (Rastegari et al. (0166.. To account for sparsity, our work proposes to map real-valued weights to the positive {0. values, instead of the standard mapping of real weights to {-1, +1].
well-known DNN topologies, without incurring in accuracy
pruning thanks
Quantization.
Network quantization allows the use of fixed-point arithmetic and smaller bit- width to represent network parameters w.r.t the full-precision counterpart. As such, it has been
2

---

Under review as a conference paper at ICLR 2021
widely applied in graphs and neural networks since it reduces the power consumption and storage requirements the network, while increasing the processing speed. Representing the values using only a finite set requires quantization function that maps the original elements to the finite set. The quantization can be done either after training the model, using parameter sharing techniques (Han et al. 20T6); or by directly quantizing the weights in the forward pass, TNNS (Hwang & Sung 2 BNNs (Courbariaux et al. (2015) and other quantized networks (Hubara et al. (0017), 7). in general. Similar to previous BNNs, our model also performs quantization during the training phase,
while also introducing sparsity.
3 METHOD
We formulate the optimization problem addressed by SBNNs in section algorithm train model and the proposed quantization functions (section 3.2). The sparse initialization is introduced section 3.3 Finally, we describe implementation details and how SBNNs are more efficient in this aspect (section 3.4
3.1 then we present the
3.1 PROBLEM FORMULATION
Given a dataset of input-output pairs D = (xx y)}, the standard training full full-precision neural network can be
loss minimization problem:
arg min L(y, Dw)
(1)
w where L(-) is a loss function between true labels and the predicted values 9w. The predicted values are a function of the data inputs and the networks' weights w E RN, i.e. D.- = f(x; w).
Let us define a Sparse Binary Neural Network (SBNN) as network with positive binary weights E {0, 1) and sparsity constraint imposing maximum number M of non-zero weights in the final network such that Σ; w; < < N, with N the total number of weights .The loss mini- mization problem from eq. (T) thus becomes mixed discrete-continuous constrained optimization problem:
arg min w
M(y, Dw)
s.t.
wi E (0, Vi Vi,
Σ w S
To simplify the mixed optimization problem, we first relax the discrete constraint imposed to the binary nature the weights to deal with continuous variables. We thus allow for w E [o, 1 and we introduce non-negative function fi (w), which penalizes values far from 0 and I, while favouring the two extreme positive discrete values, i.e fi = f1(1) = O. Possible functions fi (w) shown in fig.
Furthermore, we relax the sparsity constraint, E, we S M, by introducing a non-negative function f2(w) which penalizes non-zero weights. This be achieved through L1 L2 penalization. In this way, fi (w) pushes the real-valued weights towards or 1, while f2(w) seeks to minimize the number of weights equal to 1. The effect of (wr and fo (wr) is controlled using hyperparameters X and X2 respectively. Following (Murray & Ng. 2010 Srinivas et al. (017). the relaxed optimization problem for Sparse Binary Network can expressed as:
arg min
L(y, (.)) + A Σ fi(w)) + A2 Σ f2(w;)
(3)
s.t.
E [0. 10 vi.
3.2 NETWORK TRAINING AND QUANTIZATION
The optimization problem from eq. 3 can be solved using the training algorithm proposed by Courbariaux et al. (2016) for standard BNNs with antipodal weights w E (-1,+1}. Although
3

---

Under review as a conference paper at ICLR 2021
2003 0 wf
Es eilerwee
1(4]
/102
0.7
01
6.7
0.1
Figure 1: Possible regularization functions fi()). From left to right, triangular, Leg-inppired piece-wise parabola and a 4th degree polynomial.
the adaptation to positive weight constraints, i.e w E {0,1} straightforward, the quantization function used set the final weights (0, 1} requires reformulation. In the following paragraph, first briefly describe the training procedure, followed by the formulation of the SBNN quantization function.
Binary Network Training. The binary network training algorithm (Courbariaux al. 2016) re- lies on Stochastic Gradient Descent (SGD) and it binarizes the network's weights during the forward propagation. quantize the full-precision weights and back propagate their gradients, the sign function is used in the forward pass. Due to non-differentiability, the Straight-Through Estimator (STE) (Bengio et al. 2013) is used the backward pass. During parameter update, no binarization is performed, full- full-precision values valued weights beyond the binary values range does not influence the binarization operation in the forward pass. For this reason, after every parameter update those values mapped their nearest binary value. This is done through clip operation in (Courbariaux et al. 20166). Finally, to reduce the effects of binary weight scaling, batch-normalization layer (loffe Szegedy 2015) always placed after a binary layer.
required for SGD to work. The magnitude of the real-
Quantization Functions. As the optimization problem from eq. allows for the networks weights to have continuous positive values in [0, 1]. we define quantization function to map real- valued positive weights into binary positive values {0, 1} during the training's 's forward pass and parameter update. Let us so define the weight quantization function Q.. : :0, [] - {0, 1} as:
0, i 1, otherwise
(a)
=
4))
Due to the introduction of positive weights, we modify the activation function used by each neuron, since networks with positive-valuee -valued weights in combination with the standard non-negative ReLU activation function, may lead drop in the accuracy performance. Instead, the activation function used in our networks the antipodal sign function, which can span both the negative and the positive realms, as in (Courbariaux et al. 2016). This function quantizes the outputs of each neuron in the antipodal binary domain {-1,- and is expressed by the quantization function Q, : {-1,+1} as:
1. if ea(x)= +1, otherwise
(5)
The activation function follows the batch-normalization layers. The effect of the batch- normalization with its learnable parameters can be seen as the effect of moving the threshold (eq. (5)) of the activation function to a new threshold value (Umuroglu et al., 2017). This effect needed by the network to reduce the quantization error = x Q(x), i.c. the incurred error when mapping full-precision values to the quantized binary domain. Although this results in that natch-normalization introduces more parameters and extra- corraccmmptttiions during training (e.g. mean and variance of inputs the mini-batch), when the trained model is deployed, the new threshold y can be stored using only one integer parameter (Umuroglu et al.] 2017), thus requiring only a comparison and further computations.
4

---

Under review as a conference paper at ICLR 2021
3.3 SPARSE INITIALIZATION
Differently from standard network initialization, in SBNNs every weight follows a Bernoulli distri- bution of parameter p:
P(w, = 1) B(p)-
The sparsity parameter p controls the number of initial connections in the network. A fully connected network is achieved with = 1, whereas an initial empty network is obtained with p O. Our proposed training scheme works under the hypothesis that training is faster starting from sparse initialization close to zero, and adds only the necessary connections, rather than starting from fully connected network and removing most of its connections. At the same time, too sparse initialization R 0 may induce large gradient errors, forcing the network to add great number of connections. Bernoulli distribution to remove connections is exploited also in (Ardakani et al.] 2016), with the idea of creating mask for binary and ternary weight networks that selects only percentage p of weight parameters to control the sparsity level. Their proposed approach is really effective using VLSI technology, but due to the transformation from binary weight networks with w E +11,+1} to ternary values w E {-1, 0, +1}, the sign the non-zero connections needs to be encoded in normal processors, adding overheads in the encoder and decoder, while limiting the compression factor.
3.4 IMPLEMENTATION
BNNs replace the multiplication and accumulation operations in the forward pass of full-precision networks by two bitwise operations. Multiplication is replaced by XNOR operation between binary weights at given layer and the binary inputs from previous layers, and accumulation by popcount operation.
Connections in the SBNN formulation will always have the same weight 1. As such, XNOR operations are not required allowing SNNs to further simplify the computations of the network. The only operation needed by SBNNs during the forward pass the popcount. Furthermore, is performed only among the connected input bits through the Is weights rather than the full input. The implementation gain of SBNNs w.r.t. BNNs is illustrated fig.
Bibory Neura Nemwork:
coopp
Sporse Binary Nearal Network
Example of implementation Binary Neural Networks and Sparse Binary Neural Net- works. The threshold in Binary Neural Networks is [Raaputs] with Nappute the dimension of each input sample at a given layer. In Sparse Binary Neural Networks, this accounts to [ with mis the number of connected inputs at each output neuron.
Figure 2:
EXPERIMENTS AND RESULTS
We trained and tested the proposed SBNN on classification problems from two different datasets widely used to understand the generalization
BNNs, using similar network topologies (Cour-) bariaux et al. 2015 2016 Hubara et al. 2017 Kim & Smaragdis 2015); 1) the MNIST dataset
(LeCun & Cortes 2010). consisting of 28 28 black and white images of handwritten digits; and 2) CIFAR- 10 (Krizhevsky et al.),, consisting of 32x32 32 RGB images of natural scenes. Both datasets have 10K images for testing and 10 different classes, while they have 60K and 50K images for training, respectively.
5

---

Under review as a conference paper at ICLR 2021
In the following section report experimental results obtained each dataset, where study the properties the proposed SBNN. Concretely, we use MNIST to study SBNNs using linear layers, whereas CIFAR-10 10 is used to study the behaviour of SBNNs with convolutional layers.
4.1 MNIST
This section first describes the setup used to train our SBNNs over MNIST dataset. Then we study how the sparsity parameter p. the learning rate, and the use of regularizers fi and f2 affect the SBNNs sparsity and generalization capacity. Lastly, we compare our SBNNs with state-of-the-art BNNs terms of accuracy and compression.
Setup. We the linear topology proposed by Courbariaux et al. 2016), consisting of 3 hidden layers with 1024 neurons each, followed by batch-normalization layers with learnable parameters and the sign function. We denote sparse architecture 3L-SBNN and provide further details about it and its implementation appendix A
Effects of sparsity parameter p. In this first experiment, we study the sparsity and the general- ization properties of our SBNN as function of the initial number of connections, controlled by p. Sparsity, quantified by the number of effective connections, EC W + 0//lW| (Marban et al,, 2020), and generalization, measured in terms of test accuracy, over training time (epochs) for differ- ent values of p are reported in fig. 3 The learning rate was fixed to A sparse network will have an EC close to zero, whereas fully connected network has EC 100%.
Results show that a very low number of initial connections (e.g. < 1%) seems to induce large gradient errors, leading to large increment of connections, clearly visible after the first epoch. Conversely, larger values of induce smaller variations in the number of connections over time. Re- garding training time, low values of p (e.g. p S 1%) allow for faster training, which is manifested through the fact that these achieve smaller generalization errors faster than higher values of p (fig. right).
D 0. 1% Siiina connection Nimba coneecion connection Mila%iialoooeee
Figure 3: Effective Connections (EC) as a function of the initial number connections (p) a Sparse Binary Neural Network with feed-forward hidden linear layers with 1024 neurons each, trained on MNIST dataset. Left. 0.1 (*) and p 100.0 % (x). Right. Corresponding test accuracy.
0.1 % (O), p= 1.0 % (C), p 10.0 % (A). Center. p= 50.0 %
Effects of the learning rate. Here we study the sparsity and generalization of SBNNs function of the learning rate. Using a fixed p= 1.0%, which resulted in previous experiment in a good trade- off between sparsity and generalization, fig. reports how the number of connections varies over training time along with test accuracy.
As it would be expected results show that higher learning rates lead faster convergence, mani- fested by faster increase in test accuracy. Regarding the number of network connections, however, smaller learning rates favour sparsity: the smaller the learning rate, the smaller the increase the number of connections over time. Large learning rates (Ir = 10) are catastrophic for sparsity (fig. 4 left). Overall, results indicate possible to find a compromise between learning rates that can achieve high accuracy within reasonable training times, while preserving sparsity.
Effects of the regularizers. In this last experiment, we study the role of the regularizers (eq. (3) in the SBNN's performance in terms of sparsity and accuracy. We compare EC and test accuracy

---

Under review as a conference paper at ICLR 2021
learning rate 0.01
learning
learning rate 1.0
leanning
Figure 4: Left. Effective Connections (EC) as a function the learning rate (Ir) in a Sparse Binary Neural Network with 3 feed-forward hidden linear layers with 1024 neurons each, trained on MNIST dataset. Each curve represents a different learning rate, 0.01 (O), Ir = 0.1 (O), Ir 1.0 (A), and 1v=10.0 (*). Right. Corresponding test accuracy.
Table 1: Effects of fi and fz in test accuracy (ACC) and effective connections (EC).
f2
f
1,2
1,1
ACC [%)
EC [%]]
ACC [%)
EC [%)]
triangular
97.48(110.05)
1.27(+0.00)
06.96(10.11)
1.21 1.21(10.01)
1.2-inspired
97.38(+0..16
1.17(+00.3)
97.31(H0.00)
1.01 (1.00000.))
piecewise-parabole
(6.90
97.27 (3.002)
1.17(#0.03)
4th degree polynomial
97.51(+0
1.4((10.0))
97. 97.69 (00008)
1.66(10.03)
regularizers
ACC ACC[%)]
97.78(10.07)
EC [%] 2.03(#0.03)
results with no regularization, versus the case of using different f1 regularizers: triangular, L2- inspired, piece-wiis wise parabola and 4th degree polynomial (fig. [ and an L1 and L2 penalization for f2. The models are trained for 40 epochs, with initial learning rate of 2, decreased by 10 times every 15 epochs. = X2 = 1e-5, except for the triangular function where λ 1e-6.
Results in table [ show that regularizers achieve higher sparsity levels (less final connections), al- though they tend to have slight drop in accuracy. The latter could be explained by the fact that the minimization problem does not involve only the loss function, but a sum of the loss with fi and f2- Increasing the learning rate or the number epochs should compensate for the accuracy loss.
Method Comparison. Finally, we compare the accuracy, effective connections (EC) and compres- sion performance of the proposed SBNN with state-of-the-art BNN formulations. The compression performance measured terms of the compression rate, which computed as the total number of bits to represent each network parameter at full-precision, over the number of bits of full full-precision parameters and binary parameters of the network. We use, for this purpose, three different encoders: the index encoder (IE), run-length encoder (RLE) and huffman encoder (HE); and compare them with the case with no encoder (NE). Details on the compression procedure are found in appendix (
We assess the SBNNs using two topologies: the 3L-SBNN and a hidden layers variant of it, 2L-SBNN, We compare our results with the following BNNs: 1) 3L-BNN, the 3 layer BNN from Courbariaux et al. (001)); 2) 2L-BNN, the 2 layer variant of 3L-BNN; and 3) Bitwise (Kim & Smaragdis 2015). The networks are trained for 40 epochs, starting with learning rate = 2 and decreasing it by factor of 10 every 15 epochs. For SBNNs, p 1% and X and X2 are we could not reproduce the results of Bitwise, we report those in Kim & Smaragdis (2015).
As
The results obtained for the different models are reported in table 2 As it can be seen, SBNNs achieve the best compression rates among all methods both with 3L-SBNN and 2L-SBNN topology, incurring in a loss of less than 1% w.r.t Bitwise, which the Binary Neural Network reporting the highest accuracy. In particular, SBNNs outperform other BNNs by over 4 times in compression when using a Huffman encoder. This result is of high practical interest. It suggests that SBNNs for simple tasks be implemented in very cheap and severely memory limited 1oT devices, ensuring at the same time low consumption and fast inference, therefore opening the possibility to automate various simple tasks at low cost. For instance, the process of recognizing handwritten ZIP codes

---

Under review as a conference paper at ICLR 2021
Table 2: Method comparison on MNIST dataset, Values in parentheses report standard deviation.
Model
ACC [%]]
Compression rate
EC
[%]
NE
IE
RLE
HE
2L-SBNN [our]
97.83(+00.07
32
101
111
131
2.65(
+0.03)
2L-BNN
98.30(=0.08)
32
100
Full-precision
98. 98.72(=0.07)
100
3L-SBNN [our]
97.78(=0.07)
32
128
144
173
2.03
3L-BNN
98.15(10.07)
32
100
3L- BL-Bitwise
98.67
32
100
Full-preche full-precision
98.63(+0.04)
100
reguarizers
banguuar
plece parabols
4" degree polynomial
10
709
inspired
3-
apo0ts
aposte
Figure 5: Effects fi and f2 in sparsity and test accuracy of a SBNN with convolutional topology trained on CIFAR-10. Left. Effective Connections (EC). Right. Corresponding test accuracy.
post offices or measurement reading from screens non internet-enabled meters sending them through an IoT network to a central office to process for rapid anomaly detection and localization.
In the final experiment, we make a comparison between SBNNs and 3L.-BNN with dropout (Srivas-) lava et al. 2014) as this combination could lead to final sparser BNN. Over 40 training epochs on MNIST, SBNN outperforms 3L-BNN with dropout by a large margin both in sparsity and accuracy. For this reason, the results not included but, for the sake of completeness, report them in appendix C
4.2 CIFAR-10
This section first describes the setup used to train our convolutional SBNNs over the CIFAR-10 dataset. Then we propose experiment study the relation between sparsity of convolutional SBNNs and the use of regularizers fi and fz. Lastly, we compare our SBNNs with state-of-the-art BNNs in terms of accuracy and compression.
We use convolutional topology inspired on the VGG network (imonnyn & Zisserman, 2015). We use La-inspired for fi, and the La norm for regularizers, with hyperparameters A 7e-6 6 and A = 3e-6. 6. The training set was augmented through the use of different random rotations, crop and flips every epoch and for each image. A detailed description of the network architecture and the training procedure is provided in appendix A.2
Setup.
Effects of the regularizers. We study the sparsity and the generalization properties SBNNs on convolutional networks, when regularizers are used. We assess sparsity with EC and generalization in terms of test accuracy, over training time (epochs) for the different fi (see fig. [ and f2 (L) and L2) regularizers. Results in fig. 5 are obtained over 10 training epochs with a learning rate of 0.01. As Courbariaux et al. (2015). we use an Adamax optimizer, since it shows faster generalization of BNNs. The use of regularizers in convolutional SBNNs, with linear topologies, leads to higher sparsity levels at the cost of a slower convergence. However, differently from linear networks, their presence is clearly needed to preserve sparsity.
8

---

Under review as a conference paper at ICLR 2021
Table Method comparison on CIFAR-I0 dataset. Values in parentheses report standard deviation.
Model
ACC [%]]
Compression rate
EC [%]
NE
IE RLE
HE
SBNN [our]
78.59( +0.47)
32
108
119
145
3.10(
+0.01)
BNN
08.333)
32
100
Full- coll-precision
(8..7(
100
Method Comparison. We compare the proposed SBNN with the state-of-the-art BNN formula- tions proposed in (Courbariaux et al. (0T6) in terms of accuracy, effective connections (EC) and compression performance. We use the same compression techniques and performance measure- ments used for MNIST. Both networks are trained for 300 epochs, starting with learning rate of 0.05, decreased by factor firstly after 100 epochs, and then every 50 epochs. For SBNNs, is set to 1.0%, while À1 and X2 set to 7 and 3c-6, respectively. The results for the different models reported in table 3 confirm the SBNNs superiority in terms of EC and compression rates. However, differently from the linear topology, the convolutional SBNN incurs in higher accuracy loss of ~ 10% w.r.t. BNNs.
5 CONCLUSIONS
We propose Sparse Binary Neural Network (SBNN), a method to further compress Binary Neural Networks (BNNs) by introducing sparsity, while reducing their required computations. Our ap- proach is based on quantization of weights in the 0/1 binary domain and a highly sparse initialization formulated as a mixed optimization problem and solved using modified ver-
of the network. sion of the BNN training algorithm with -1/+1 weights (Courbariaux et al.. 2016). The method has been evaluated on feed-forward linear and convolutional network on MNIST and CIFAR- 10 data sets, respectively. The achievable compression rate of SBNN is much higher than simple BNNs, making it feasible alternative for IoT devices and sensors. For instance, 2L-SBNN with a model size of 70 kB (with index encoder) and 97.83 % accuracy only in the flash memory, but also in the SRAM memory of very low-power hardware modules like the Intel Curie TM Module, which has 384kB of flash memory and only 80kB of SRAM.
MNIST, can be entirely stored, not
current weakness of SBNNs is that sparsity, speed-up and high compression rates come at the cost of reduced performance accuracy. While the generalization error in linear architectures remains competitive, this has larger drop in convolutional networks. The problem of generalization is not exclusive of SBNNs, and is common also +1/-1 BNNs. As discussed in Qin al. (2020), is still unclear what kind of network structure is suitable for binarization, so that the information passing through the network can be preserved, even after the binarization itself. As a result, multiple methods have been proposed to mitigate generalization losses by reducing the gradient (Rastegari al. 2016] Yin al. 2018 Darabi et al., 2019) and the quantization error (Kim Smaragdis] 2015 Liu et al.) 2018). Future works should continue in this direction with a specific focus on the 0/1 binary domain. Another interesting direction could be to extend the learning capabilities of SBNNs by investigating how to extract them from sparse ternary networks.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ala Al-Fuqaha, Mohsen Guizani, Mehdi Mohammadi, Mohammed Aledhari, and Moussa Ayyash. Internet of things: A survey on enabling technologies, protocols, and applications. IEEE commu- nications surveys & tutorials, 17(4):2347-2376 2015.
Arash Ardakani, Carlo Condo, and Warren J Gross. Sparsely-comnected neural networks: towards efficient vlsi implementation of deep neural networks. arXiv preprint arXiv:/611.01427, 2016.
Martin G. Bello. Enhanced training algorithms, and integrated training/architecturee selection for multilayer perceptron networks. IEEE Transactions on Neurul Networks, 3(6):864-875, 1992.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint wwrXiv:/38..445 2013.
Alfredo Canziani, Adam Paszke, and Fugenio Culurciello. An analysis of deep neural network
models for practical applications. arXiv preprint arXiv:1605.07678, 2016.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pp. 160- 2008
Matthieu Courbariaux, Yoshua Bengio, and Jean Cean-Pierre David. Binarycomnect: Training deep neural networks with binary weights during propagations. Advances in Neural Information Processing Systems, pp. 3123-3131, 2015.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to +1 or -1. arXiv preprint arXiv:1602.02830 2016.
Sajad Darabi, Mouloud Belbahri, Matthiuu Courbariaux, and Vahid Partovi Nia. Bun+: Improved binary network training. In NeurIPS19 Workshop on Energy Efficient Machine Learning and Cognitive Computing, 2019. URL arxiv: 11800
Misha Denil, Babak Shakibi, Laurent Dinh, Mare' Aurelio Ranzato, and Nando De Freitas. Pre- dicting parameters in deep learning. In Advances in Neural Information Processing Systems, pp- 2148-2156, 2013.
Dave Evans. The internet of things: How the next evolution of the internet is changing everything. CISCO white paper, 1(201 1):1-11, 2011.
Julian Faraone, Nicholas Fraser, Giulio Gambardella, Michaela Blott, and Philip HW Leong. Com- pressing low precision deep neural networks using sparsity-inducee regularization in ternrry net- works. In International Conference on Neural Information Processing, pp. 393-404, 2017.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. 2019.
Global System for Mobile Communications. 3GPP low power wide area technologies (white paper). Technical report, GSMA, 2018.
Aidan N. Gomez, Ivan Zhang, Kevin Swersky, Yarin Gal, and Geoffrey E. Hinton. Learning sparse networks using targeted dropout. ArXiv, 1tt119955...6677 2019.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in Neural Information Processing Systems, pp. 1135-1143, 2015.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In International Conference on Learning Representations (ICLR) 2016, 2016.
10

---

Under review as a conference paper at ICLR 2021
Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal processing magazine, 29(6):82-97, 2012.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Jour- nal of Machine Learning Research, (8(1):6869-889)) 2017.
David A Huffman. A method for the construction of minimum- minimum-redundanny codes. Proceedings of the IRE, 1098-110, 1952.
Kyuyeon Hwang and Wonyong Sung. Fixed-point feedforward deep neural network design using weights+ 1, O, and- In 2014 IEEE Workshop on Signal Processing Systems (SiPS), pp. 1-6, 2014.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint 1502.0330 2015.
Minje Kim and Paris Smaragdis. Bitwise neural networks. In Proceedings of the 31st International Conference on Machine Learning JMLR: W&CP, volume 2015.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced re- search). Technical report. URL http:://twwwt cs. wws.ttorntoooooo wttpt//ooodd.//i/iiiieei/ie http://www.....ooooooooo/o///aaa html
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in Neural Information Processing Systems, pp. 1097-1105, 2012.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URLhttp: lecun. com/exdb /xxd//mnist/
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436-444 2015.
Zechun Liu, Baoyuan Wu, Wenhan Luo, Xin Yang. Wei Liu, and Kwang-Ting Cheng. Bi-real net: Enhancing the performance of 1-bit cans with improved representational capability and advanced training algorithm. In Proceedings of the European Conference on Computer Vision, pp. 722-737, 2018.
Christos Louizos, Max Welling, and Diederik P Kingma. Learning sparse neural networks through in regularization. In International Conference on Learning Representations (ICLR), 2018.
Arturo Marban, Daniel Becking, Simon Wiedemann, and Wojeiech Samek. Learning sparse & ternary neural networks with cntropy-conttrained trained ternarization (ec2t). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 722-723, 2020.
Mahdi H. Miraz, Maaruf Ali, Peter S. Excell, and Rich Picking. A review on internet of things (iot), internet of everything (ioe) and internet of nano things (iont). In 2015 Internet Technologies and Applications (/TA), pp. 219-224, 2015.
Walter Murray and Kien-Minn Ng. An algorithm for nonlinear optimization problems with binary variables. Computational Optimization and Applications, 47(2):257-282 2010.
Haotong Qin, Ruihao Gong, Xianglong Liu, Xiao Bai, Jingkuan Song, and Nicu Sebe. Binary neural networks: survey, Pattern Recognition, pp. 107281, 2020.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In Proceedings of the European Con- ference Computer Vision, pp. 525-542. Springer, 2016.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations, ICLR 2015, Confer- ence Track Proceedings, 2015.
11

---

Under review as a conference paper at ICLR 2021
Suraj Srinivas, Akshayvarun Subramanya, and R Venkatesh Babu. Training sparse neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 138- 145, 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1229-195; 2014.
Frederick Tung and Greg Mori. Clip-q: Deep network compression learning by in-parallel pruning- quantization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recogni- lion, pp. 7873-7882, 2018.
Yaman Umuroglu, Nicholas Fraser, Giulio Gambardella, Michaela Blott, Philip Leong, Magnus Jahre, and Kees Vissers. Finn: A framework for fast, scalable binarized neural network inference. In Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Prev Progrdd-rrra Gate Arrays, pp. 65-74, 2017.
Shuochao Yao, Yiran Zhao, Aston Zhang, Shaohan Hu, Huajie Shao, Chao Zhang. Lu Su, and Tarek Abdelzaher. Deep learning for the internet of things. Computer, 51(5):32 01,33241,,22 2018.
Penghang Yin. Shuai Zhang, Jiancheng Lyu, Stanley Osher, Yingyong Qi. and Jack Xin. Bina- ryrelax: A relaxation approach for training deep neural networks with quantized weights. SIAM Journal on Imaging Sciences, 11(4):2205-22) -2223, 2018.
12

---

Under review as a conference paper at ICLR 2021
APPENDIX
A LINEAR AND CONVOLUTIONAL TOPOLOGIES SETUPS
This section provides details about the network configurations used in the experiments. We provide details on the linear architecture used for experiments with the MNIST dataset and the convolutional topology used with CIFAR-10.
A.1 LINEAR ARCHITECTURE
For experiments with MNIST, we use the linear topology proposed by bourbariia al. (2016). The network consists of 3 hidden layers with 1024 neurons each, followed by batch-normalizationn layers with learnable parameters and the sign function. The LogSoftMax activation function replaces the sign after the output layer, which contains 10 neurons. The loss used is the Negative Log-Likelihoo and the optimizer Stochastic Gradient Descent (SGD) with Nesterov momentum ja = 0.9. We denote this architecture 3L-SBNN (i.e. SBNN with 3 hidden linear layers). To speed up the training we used mini-batch of 200 samples.
A.2 CONVOLUTIONAL NEURAL NETWORK
For the experiments with CIFAR-10, we use convolutional topology inspired on the VGG network (Simonyan & Zisserman 2015). The 32x RGB images (3 channels) pass through 6 convolutional layers with 3x3 kernels, stride = and zero padding = I. Each layer is followed by a batch- normalization layer and the sign activation function. Every 2 layers, with a 2x2 kernel and stride = 2, before the batch-normalization layer. The first 2 convolutional layers have 128 channels, the following 2 have 256 channels, while the last ones 512 channels. The output of the last convolutional layer is flattened and followed by a linear layer with 10 neurons (one per class) followed by aatch-orrmllization layer and LogSoftMax activation function. Similarly to MNIST. the loss used is the Negative Log-Liken Likeliihdddod. We use Adamax as optimizer and scheduler which halves the learning rate after 100 epochs and then halves it every 50 epochs. The models are trained for 300 epochs in total. As regularizers, we use an Le-inspired for fi. and the Le norm for f2. The hyperparameters A and \ are set 7 and 36-6, respectively. To speed up the training, we used a mini- of 200 samples.
add max-pooling layer
Finally, to enlarge the training set, this was augmented through the use of different random rotations, crop and flips at every epoch and for each image.
B COMPRESSION TECHNIQUES
Well-known methods from source coding theory can be exploited for the compression of sparse vectors and matrices. An SBNN can be seen as a set of sparse binary matrices. To evaluate the compression potential of SBNNs, we have used three different encoders in our experiments: index encoder, run-length and Huffman.
illustration of the index encoder and the run-length encoders is depicted in fig. The first encodes only the column (or row) indexes of the is elements, the latter encodes the length of the run-length zeros. Since the distribution of the run-lengths of zeros in the matrices of SBNNs varies, the length in bits to represent each run-length, as shown in fig. 6 is chosen with full search of all possible lengths in bits from 1 to log, (maximum run-length).
The third approach, the Huffman encoder (Huffman) 1952), maps elements, in case indexes of Is elements, to bit-symbols of variable length, depending on the frequency of each element: more frequent elements encoded using less amount of bits, while less frequent elements mapped to larger in bit symbols.
The reported compression rates (table 2 and table model size over the SBNN one. The full model size is computed considering each weight stored with 32-bit plus 32-bits for each scaling factor introduced by batch-normalization layers. The SBNN
obtained as ratio of the equivalent full
13

---

Under review as a conference paper at ICLR 2021
Nows number of rows in the matrix
Nest number of columin in the matrix
Nan: best length bits for compressing the run-lengths of zeros
logg (Now) +1 bits log, (Nea) bits
index encoder:
(N
(Moulz
(N₂)2
(1 index);
(N, index);
(Ny)
(1 index);
1 row of the matrix
2 row of the matrix
16 bits
16
run-length zeros
run-length encoder:
10000100000
Nipera
run-length 10 zeros (10), 001 010
Mus =3
(Nudaa
(Niunass
(Noa)z
(5), -101
001
0
010
Noste
bits
bits
16
Nun bits
bit
if end run-length
0, otherwise
Figure 6: Schematics of the index encoder and of the run-length encoder used to compress the sparse binary weigth matrices
100
188t accuracy [%]
20
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Parp
Drapout
-3L-SBN
Figure 7: The blue curve shows the test accuracy as function of dropout probability (Parop) in the 3L-BNN with dropout, trained on MNIST dataset. The red curve represents the accuracy of Sparse Binary Neural Network with same topology, no dropout.
size is computed compressing the binary matrices with the three different encoders, while the integer thresholds y of the sign activation function stored with 16 bits each.
C DROPOUT
This section provides further details the comparison of the 3L-SBNN with 3L-BNN in combi- nation of dropout (Srivastava et al., (2014) for different values paop- the dropout probability for each neuron. Larger Pärop means less neurons. Pämp of corresponds to nearly a full-connected network, while Pdrop of to nearly empty network. For consistency, the 3L-BNN with dropout and SBNN are trained for 40 epochs with learning rate 1.0, and decreasing of 10 times every 15 epochs. No regularizers were used for the SBNN. We report the results, in terms of accuracy for different values of Parop, in fig.
Results show that SBNN outperforms the use of BNN with dropout, both in accuracy and in final sparsity large margin, suggesting that, in their current setup, BNNs are not suited for introducing sparsity.
14


Reviews:
[{"review_id": "TOnzM8A-qOU", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors are tackling the problem of sparsifying binary neural networks, i.e. to maximize the number of weights that can be set to zero, without compromising accuracy too much. This is a useful endeavor, both for deploying models on embedded devices where memory might be limited, and to reduce the computation work load at inference time, since less active weights mean less operations to be computed during the forward pass.\nThe strategy that the authors adopt is to modify binary neural networks like those in the mentioned references of Courbariaux et al. by encoding weights as 0/1, instead of -1/1. In practice, this is done by training on continuous weights between 0 and 1 and adding terms in the cost function that promote the weight values 0 and 1 over intermediate values, and a term that tends to sparsify the weights by pushing them to 0.\nThe authors test their method on MNIST and CIFAR10 training a 3 layer MLP and a modified version of VGG. In these settings the authors carefully study the effect of the regularization terms that they added and that of the sparsity of the initialization. The main result is that the resulting architectures are remarkable in their level of sparseness: the authors achieve sparseness around 2-3%, with a minimal drop of performance on MNIST but unfortunately a quite substantial one on CIFAR-10 compared to Binary NN (from 88% accuracy to 79%).\n\nWhile practically valuable as a piece as a demonstration of how tolerant Neural Networks are to reduced precision and sparsification, this paper is methodologically quite incremental, if compared to Binary Net and all the subsequent papers deriving from that one.  The demonstrations are moreover limited to MNIST and CIFAR10, meaning that there aren't any indication of how the method would fare on larger datasets like ImageNet, where the advantages of reduced precision and sparsification are arguably going to be most relevant. In addition, even for a relatively simple dataset like CIFAR10 the proposed method seems to reach good levels of sparsity at a considerable cost in terms of accuracy (more than 9%).\nLastly, previous papers on sparsifying ConvNets have pointed out that the actual advantages in terms of memory, computation and energy consumption would come from reducing the number of filters, rather than the number of active weights.", "extra_scores": {}, "extra_text": {}}, {"review_id": "CnWXsHIN6on", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of the paper:\nThis paper presents a sparse neural networks with binary weights. The binary weights are represented using 0s and 1s whereas the effective connections are encoded using the run-length encoder for efficient implementations of neural networks on IoT devices. To quantize the value of weights, a non-negative function is introduced, penalizing values far from two discrete values of 0 and 1. The performance of the obtained sparse binary neural network is measured on MNIST and CIFAR10 datasets when using fully-connected and convolutional neural networks. It was shown that sparsity degree of 98% can be achieved at the cost of an accuracy degradation.\n\nStrength:\n\n-- The paper is easy to read\n\n-- The compression rate is very high\n\n-- Comprehensive simulations for different hyper-parameters\n\nWeakness:\n\n-- Misleading notations\n\n-- Simulation results are obtained from small datasets\n\n-- Significant overlaps with previous works (limited novelty)\n\nDetailed comments:\n-- Section 2 under the label “Sparsity”: It is not clear what the authors mean by using {0, 1} for weights instead of {-1, +1}. In conventional methods such as binaryconnet, the value of weights is mapped to two discrete values of -1 and +1 where these discrete values can be represented using a single bit (i.e., 0 for representing -1 and 1 for representing +1) for hardware implementation. Do you mean that your method maps the real value of weights to positive values of 0 and 1? If this is the case, please explain how your binary neural network works with positive weights.\n\n-- Section 3.1: According to Eq. (4), 0 is an representation for -1. If this is the case, what is the intuition behind using the function f2(w) to push values towards -1? On the other hand, if 0 is in fact representing a zero-valued weight, then we have a neural networks with positive weights. Please elaborate.\n\n-- Section 3.2: In the definition of Q, we have mapping from [0, 1] to {0, 1}. However, if you use a = -1 and b = +1 in Eq. (4), then the mapping would be from [-1,+1] to {-1, +1}. Please explain this discrepancy.\n\n-- Section 3: In general, the proposed binarization process is very similar to the work of Courbariaux et al. (2016). Please list the differences.\n\n-- Section 3.3: Initializing a network with sparse connections using a Bernoulli function was introduced in [1]. Please explain the differences.\n\n[1] Sparsely-Connected Neural Networks: Towards Efficient VLSI Implementation of Deep Neural Networks (ICLR 2017)\n\n-- Section 4: I would like to see some experimental results on a bigger dataset such as ImageNet. The quantization of neural networks has been well studied during the past few years. However, they fall short in providing a good accuracy for state-of-the-art networks on ImageNet. I would like to see how the proposed method of this paper stands out against other existing works.\n\n-- Section 4: It would be interesting to see a comparison with ternary neural networks (e.g., [2]) in terms of sparsity degree and accuracy. \n\n[2] Neural Networks with Few Multiplications (ICLR 2016)\n\n-- Section 4: I am also wondering why there is significant gap between the proposed binary network and its full-precision counterpart (which is not included in this paper) given the fact that small datasets are used. Using a high compression rate could be the answer, but it would be interesting to see a curve showing accuracy for different degrees of sparsity (i.e., EC).\n\n\n--------------------------------------------------------------------------------\nPost-rebuttal update: Based on the improvements made during the rebuttal period, I have raised my rating. I believe Fig. 2 now makes the contribution of this paper more clear when compared to the conventional binary network. According to Fig. 2, the proposed method in this paper allows to multiplex and select binary inputs rather than performing XNOR operations, which makes the novelty of this paper rather limited and incremental. Therefore, I still believe this paper stands below the acceptance threshold.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "AwdH4drr0Rs", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper compresses neural networks via so called Sparse Binary Neural Network designs. The proposed idea is naïve, directly using a slightly modified sign function to quantize network weights into 0 and 1 instead of commonly defined -1 and 1. Experiments on small MNIST and CIFAR-10 datasets with two shallow and old neural networks are provided.\n\nThis paper has obvious weaknesses.\n\n--- Limited novelty\n\nThe proposed method is naïve. The authors merely replace  binary weights {-1, 1} by {0, 1}, using common quantization tricks for binary neural networks, such as straight-through estimator (STE) and a slightly modified sign function. The authors claim that such a modification can bring significantly improved compression. However, it is problematic, as it will force all quantized network weights to be non-negative, leading to serious accuracy drop. For example, in Table 3, a shallow VGG-like network on CIFAR-10 dataset shows about 10% absolute accuracy drop compared to the binary weight counterpart. Furthermore,  in the optimization, the authors add two non-negative constraints, which makes the training with STE even more challenging. I believe, experiments on large-scale image classification dataset such as ImageNet with modern CNNs will lead to more serious accuracy drops. \n\nActually, more impressive neural network compression yet with good accuracy can be achieved via combing quantization and pruning. There exist numerous works in this field,  e.g., \"Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding\" in ICLR 2016, \"Clip-q: Deep network compression learning by in-parallel pruning-quantization\" in CVPR 2018 and \"APQ: Joint Search for Network Architecture, Pruning and Quantization Policy\" in CVPR 2020. Unfortunately, they are missed by the authors.\n \n--- Poor writing\n\nThe paper is poorly written, including introduction (messy), related works (poor), proposed method (tedious) and experiments presentation (weak). \n\n--- Weak experiments\n\nThere is no comparison with state of the art CNN compression methods, combining quantization and pruning for improved compression.\n\nThe authors only conduct toy experiments, a 3-layer fully connected LeNet on MNIST dataset and a shallow VGG-like network on CIFAR-10 dataset.\n\nEven with toy experiments, results are very weak, showing serious accuracy drop even for a shallow VGG-like network on CIFAR-10 dataset, compared to the binary weight counterpart. \n\n\n  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "8e2NFFNoj1j", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "this paper propose sparse binary neural network for model compression and efficient inference on IoT devices. The paper combines pruning techniques and binary neural network techniques, to train a model that is ternary: {1, 0, -1}. the paper experimented with four regularization functions: triangular, L2-inspired, piece-wise parabola and a 4th degree polynomial. My concern is that, evaluated on MNIST dataset, the accuracy is a bit worse than the BNN, and the implementation section is thin, therefore it's not clear whether the sparsity gain can offset the implementation overhead for decoding. I'm also concerned about the novelty combining pruning techniques with binary neural network techniques. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper compresses neural networks via so called Sparse Binary Neural Network designs. All reviewers agree that the paper has limited novelty. Experiments are only performed on small datasets with simple neural networks. However, even with toy experiments, results are very weak. There is no comparison with the SOTA. Numerous related works are missed by the authors. Besides, the paper is poorly written, and there are misleading notations.
Decision: Reject