iPrune: A Magnitude Based Unstructured Pruning Method for Efficient Binary Networks in Hardware
Authors: Adithya Venkateswaran
Jean-Pierre David
Venue: ICLR 2022 Submitted (2022)

Abstract: Modern image recognition models span millions of parameters occupying several megabytes and sometimes gigabytes of space, making it difficult to run on resource constrained edge hardware. Binary Neural Networks address this problem by reducing the memory requirements (one single bit per weight and/or activation). The computation requirement and power consumption are also reduced accordingly. Nevertheless, each neuron in such networks has a large number of inputs, making it difficult to implement them efficiently in binary hardware accelerators, especially LUT-based approaches.

In this work, we present a pruning algorithm and associated results on convolutional and dense layers from aforementioned binary networks. We reduce the computation by 4-70x and the memory by 190-2200x with less than 2% loss of accuracy on MNIST and less than 3% loss of accuracy on CIFAR-10 compared to full precision, fully connected equivalents. Compared to very recent work on pruning for binary networks, we still have a gain of 1% on the precision and up to 30% reduction in memory (526KiB vs 750KiB).

Under review as a conference paper at ICLR 2022
IPRUNE: A MAGNITUDE BASED UNSTRUCTURED PRUNING METHOD FOR EFFICIENT HARDWARE IMPLE-
MENTATION OF BINARY NEURAL NETWORKS
Anonymous authors
Paper under double-blind review
ABSTRACT
Modern image recognition models span millions of parameters occupying several megabytes and sometimes gigabytes of space, making it difficult to source constrained edge hardware. Binary Neural Networks address this problem by reducing the memory requirements (one single bit per weight and/or activa- tion). The computation requirement and power consumption are also reduced accordingly. Nevertheless, in such networks, each neuron has large number of inputs, making it difficult to implement them efficiently in binary hardware accel- erators, especially LUT-based approaches.
re-
In this work, we present a pruning algorithm and associated results on convo- lutional and dense layers from aforementioned binary networks. We reduce the computation by 4-70x and the memory by 190-2200x with less than 2% loss accuracy on MNIST and less than 3% loss of accuracy on CIFAR-10 compared to full precision, fully connected equivalents. Compared to very recent work on pruning for binary networks, still have gain of 1% on the precision and up 30% reduction in memory (526KiB vs 750KiB).
INTRODUCTION
AI has become ubiquitous in our daily lives, be it the use of search engines to find information or automatically tagging photos on social media. While the predictive accuracy of these models have grown, the size and computation requirements have grown by several orders of magnitude or more, making AI a niche field where only few can participate. This practice is also proving to be environmentally unfriendly with Strubell et al. (2019) estimating the footprint of training NLP model to be 626,000 pounds of carbon dioxide equivalent.
To counter this growing computation need, it is important to develop hardware accelerators which are easily re-configurable. A Field Programmable Gate Array (FPGA) is an integrated circuit de- signed to be reconfigured by the user and acts a blank canvas for implementing custom circuits (in our case custom models). FPGAs embed millions of lookup tables (LUTTs) that "compute" (actually fetch) the output given the values of their binary inputs. The problem is that these LUTs are lim- ited in size (typically 6-12 inputs) and using multiple LUT's per neuron is less efficient. This raises the need for both binarizing our weights (and/or activation) and reducing the number of inputs per neuron.
In Sections and 3 we go through existing works on binarization and pruning of fully connected, full precision networks. In Section[4] 4 we introduce iPrune, effective way to prune the weights of binary networks. Section 5 we compare the accuracy, memory and computation requirements of these network implementations against their full precision, fully connected equivalents. In Section 6, we report some intriguing results about a- a-priori pruning versus post-training pruning. Section concludes this work.
2 BINARIZATION TECHNIQUES
Binarization is the process of converting the weights and/or activations to a 1-bit representation (#1) instead of the classic N-bit representation (typically 8-bit, 16-bit or 32-bit). This gives direct

---

Under review as a conference paper at ICLR 2022
memory saving of up to 32x. There several approaches in literature binarization. Throughout our experiments, we worked with BinaryConnect whose details are given in the next subsection.
2.1
BINARYCONNER
Courbariaux et al. (2015) introduced BinaryConnect, the first paper of its kind, with details on binarization of both convolutional and dense weights. There are sub-types based on how the binarization is performed.
The weights are binarized deterministically; specifically, using the sign function on the hird-sigmoid of the weight.
Deterministic:
Stochastic: The weights are set to -1 with probability (I-o(W:) and with probability oW;) where o is the hard-sigmoid function.
The hard-sigmoid function mentioned above is given by:
+ 1 0,1
(σ)) clip
During the back buck-propagations the full precision weights are updated based on the gradient with respect to the binary weights. The pseudocode for BinaryComnect is given in Algorithm
Algorithm Algorithm for BinaryConnect
1. Forward propagation:
w/ < binarize(w(-1)
for k = 1: do
compute ag knowing 0x-1, up and 1
end for
2. Backward propagation
for k : do Compute ac Sa
ac and knowing Bag
end for
Parameter update
Compute 8C and Then
8C and dx-1 knowing Bag
Tb aC
We + clip(w) Wt-1
77 aang
bt +- b 17 Jb.-1
80
3 PRUNING TECHNIQUES
Hoefler et al. (2021) discusses in detail different approaches pruning and used as reference for our methods. The different types of pruning techniques are given in Figure [ A few approaches described in detail in Sections 3.1 and 3.2
3.1 STRUCTURE VS UNSTRUCTURED PRUNING
Structured pruning the removal of neurons or weights structured fashion i.e. based on fixed pattern. Common methods for structured pruning neuron pruning (since this is just row/column removed from the weight matrix) and filter/trasffrmer head removal. Similarly, strided removal of weights after offset is also example of structured pruning.
On the other hand, unstructured pruning refers to the pruning method where there are no patterns in how the weights removed. While this does not help with easier matrix calculation, it tends to provide higher baselines than structured pruning and can often be converted to structured pruning through minor set of modifications.
2

---

Under review as a conference paper at ICLR 2022
Pruning Techniques
data-free
data-driven
training aware
weight similarity
loss function approximation
weight magnitude
matistical/variateee
regularization
trivial element (magnitude is -0)
correlation similarity merge
sensitivity (variance of output is -0)
Figure 1: Schemes of pruning described in Hoefler et al. (2021)
3.2 MAGNITUDE PRUNING
It is simple and effective selection scheme where the magnitude of the weights is used metric to determine which weights to drop and which weights to keep. After sparsifying the network this way, retraining done get high accuracies again.
There are three ways to perform magnitude pruning: globally, layer-wise or neuron-wise. Global magnitude pruning takes all weights from all layers and keeps the top-k weights among them. This often has the problem of vanishing gradients or even vanishing layers (because most weights have been removed from the layer). Figure 2 gives the distribution of weights before and after applying global magnitude pruning.
Liyer-wise magnitude pruming (used Guerra al. (2020) keeps the top-k weights across all neurons per layer and overcomes the issue of vanishing layers. This is still difficult to implement in LUTs, because some neurons will have large number connections while others will have very few after pruning.
Lastly, neuron-wise magnitude pruning keeps the top-k weights for each neuron. This overcomes both the issues of vanishing layers/gradients and LUT implementation, but often comes at the cost of low initial accuracy which can solved by retraining the network. In all our experiments, we neuron- wise magnitude pruning.
Distribution trained weights
Distribution masked weights
300000
500000
25,0000
Number wegghts
400000
woights
200000
300000
350000
200000
100002
Magnitudc
Magnitudk
Figure 2: Weight distribution before and after applying global magnitude pruning on custom VGG model for CIFAR-10

---

Under review as a conference paper at ICLR 2022
3.3 LOTTERY TICKET HYPOTHESIS
The Lottery Ticket Hypothesis initially introduced in Frankle & Carbin (2018), and extended to quantized networks in Diffenderfer & Kailkhura (2021) theorizes that there exists optimal sub- networks in the initialization of an overparameterized network. The authors prove theoretically and validate practically this claim their papers. They introduce a learnable parameter "score" which decide the weights to retain and the ones to drop. Diffenderter & Kailkhura (2021) claims reduce network sizes by 50% with negligible reduction or sometimes even increase accuracy compared to the fully connected counterparts.
4 IPRUNE
iPrune is magnitude based unstructured pruning technique to reduce the number of inputs to each neuron particular layer neural network.
For dense layers, the magnitude is computed and the top-k weights are chosen for each neuron of the current layers (top-k columns for each row in the weight matrix shape (out features, in features)). For convolutions, the LI norm across the kernel dimensions (kx, ky) is computed to reduce it to an (out_channell, in_channels) representation. This is then followed by pruning using top-k weights similar to the dense case.
Disconnecting neurons from previous layers can be easily accomplished most deep-learning frameworks with the use of mask. Neurons which remain connected have mask value 1 and the others have In Figure 3] for a given example mask and weight, the computation of the masked weight is shown. The figure also highlights the structure of the mask.
Algorithm provides pseudocode for iPrune. Note that the update step, performed on the full
precision weights uses the gradients with respect the binary masked weights
8C 8am
0
-2
1
-2
0
3
3
4
1
-5
1
1
0
-5
Weight fout_features, in_ features))
Mask num_input
Masked Weight
Figure 3: iPrune Masking procedure
5 EXPERIMENTAL RESULTS
We performed tests on MNIST (LeCun et al. (1988) and CIFAR-10 Krizhevsky & Hin- ton (2099) datasets. For all the results given below, the network architecture used for MNIST was (1024D 1024D - 10D) and for CIFAR-10 modified version of VGG- Small, specifically: (128C3 128C3 MP2 256C3 256C3 MP2 512C3
512C3 MP2 1024D - 1024D 10D)
For MNIST, no preprocessing was performed. We skipped the last layer while pruning. For CIFAR- 10 we applied ZCA preprocessing
suggested in Courbariaux et al. (2015) for gains of around 2-3% in accuracy. We applied weight-decay (5c-2) on the last layer (dense) of the network and used the square hinge loss the loss function. This together behaves like L2-SVM block. We the Adam optimizer with initial learning rate of 3e-3 with no learning rate scheduler. While pruning, skipped the first two convolution layers and the last dense layer.
Figure 4 shows the effect of iPrune on the weight values for the models we trained. Notice how the number of neurons with weight magnitude close reduces in both cases. At the same time the

---

Under review as a conference paper at ICLR 2022
Algorithm 2 Algorithm for iPrune
Train a fully connected model
1. Compute the mask Done only once w.mask = find_me (w)
2. Forward propagation:
wy < binarize(w)
tmm w.mask
for T.
compute ag knowing 0k-1, Wm and b end for
3. Backward propagation
for k : 2: -1 do
Compute 3og-1 knowing Dac
8C and Wm
end for
4. Parameter update
Compute ac and ac knowing ac Bay" ak-1 and w.mask
8a
w f- clip(w- 3um
BC
1I Bb
number of these weights is not zero as are applying neuron wise pruning, not global pruning. Also note that in the BinaryConnee case, the original distribution is mixture of three Gaussians centered at -1, +1 and 0 while in the FP case, there is only one Gaussian centered at O.
Distribution of original trained weights
Distribution of maskerd weights
Distribution of retrained weigles
144
1.0
100
6.6
Sumber
0.4
02-
60
02-
-3.0
-0.5
800
1.0
-1.0
-2.5
3.2
05
1.0
-18
-25
0.
055
1.0
(a) DinaryConnect model with 30% weights remaining in all layers except first two convolutional and last dense
Distribution original trained we
Distributinn masked weights
Distributio retrained weights
160000
300000
150000
125000
120000
X0000
100000
80000
79000
50000
80000
133033
20000
(b) Full Precision model with 30% weights remaining in all layers except first two convolutional and last dense
Figure 4: Weight distribution before, after pruning and after retraining a custom VGG model
T the results after applying iPrune MNIST. There is less than 2.3% difference between the fully connected, full precision network and pruned BinaryConnes deterministic model with 7/8 inputs. The memory gain is around 2200x and computation gain is around 70x.
Table gives the accuracy memory and computation requirement for different percentages of weights remaining per neuron. It is clear that weights 50%-20%) and computation by 1.9x-4.4x, compared to the fully connected Binary deterministic network, with negligible loss in accuracy (less than 1%).
reduce the memory by 1.9x-4.7x (remaining
5

---

Also note, based on Table 2 Table and our baseline accuracy for fully connected, full precision able save up 180x in Under review as a conference paper at ICLR 2022
Table 1: Accuracy on pruning models for MNIST with "Number of inputs" connections to each neuron of current layer to neurons from previous layer.
Method
Baseline (1024 inputs)
Number of inputs
Accuracy
5
94.47%
Binary Connect Deterministic
6
95.19%
98.08%
7
95.98%
8
96.01 %
5
97.51% %
6
97. 97.80%
Floating Point
98.27%
7
97.71 %
8
98.06%
Table 2: Accuracy, memory and computation requirements with different percentages of weight remaining for each neuron while pruning models for CIFAR-10.
Percentage weights remaining
Accuracy
Memory
Computation
(Ops)
1.0
72.89 %
63.787KiB
4.30361e+08
5.0 %
77.09 %
103.047KiB
3.72070e+08
10.0 %
83.41 %
187.953KiB 6.01394e+00
20.0 %
86.20 %
357.359KiB 1.05768e+09
30.0 %
87.54 %
526.047KiB 1.50689c+09
50.0 %
87.71 %
865.672KIH
2.42419c+09
80.0 %
87.86 %
1.340MiB
3.77653e+09
100.0
87.64
1.672MiB
4.69383e+09
Since we observed in our early experiments with MNIST that dense layers can be pruned further, performed gridsearch with different percentages/number
weights remaining in convolution and dense layers. The results are given in Table 3 The baseline accuracy for these models is 87 87.64%. The results mostly show a trend increasing accuracy with increasing number of dense/percentage of convolution weights per neuron apart from few anomalies. We can also observe that the com- putation and memory requirements are highly skewed towards reduction in convolution parameters than dense parameters.
model (88.2 %), that with Tess than 3% reduction accuracy, memory and up to 4.4x in computation.
Previous work this topic (Guerra al. (0200). which used layer wise unstructured pruning, was able to prune Binary models to 750 KB size with 1.4x reduction in computation while our models with 1 1% higher accuracy can be pruned to low as 526 KB with upto 3.1x reduction in computation. Compared to pruning full precision models from Li et al.) (2016) we reduce memory by upto 41x (526KB vs 21.6 6 MB) with reduction in accuracy (Note that the comparisons have been done with results of closest equivalent VGG models reported by the authors).
IPRUNE FOR RANDOMLY INITIALIZED NETWORKS
As an ablation study, we compared iPrune fully trained network for MNIST against iPrune randomly initialized network. Table shows the results of iPrune on randomly initialized network vs fully trained network.
The idea behind this is that the initialization of networks greatly affects the magnitude of the gradient and subsequently the final value after training. Existing work such "Lottery Tickets" have been discussed in Section Those methods use a learnable metric "score" to determine the mask. This score is updated while training. Conversely, in this ablation, compute the mask once, just after initialization, and assume that this the "Lottery Ticket" network. We allow the update of weights while training allow the model to fit better to the data instead of updating the mask itself.

---

Under review as a conference paper at ICLR 2022
Table Accuracy, Memory and computation requirements of different combinations of percentage convolution weights remaining per neuron and number of dense weights remaining per neuron for iPrune for CIFAR-10 BinaryConnect Deterministic.
Dense
Conv
8%
20%
30%
50 %
78.11%
80.93%
82.43%
83.65%
63.569KiB
128.819KiB
182.257KiB
291.382KiB
5.08453e+08
1.05581e+09
1,50408e+09
2.41948e+09
78.33%
81.21%
82.92%
84.30%
63.826KiB
129.076KiB
182.514KiB
291. 13993KK
5.08455e+08
1.05581e+09
1.50408e+09
2.41948e+09
10
78.40%
81.68%
83.01%
85.10%
64.344KiB
129.594KiB
183.03 KiB
292. 156K1B
5.08459c+08
1.05582++09
L.50408e+09
2.41949c+09
20
78.84%
82.07%
83.90%
85.49%
66.852KiB
102KiB
185.540KiB
294.665KiB
5.08479c+08
1.05584++09
1.50410e+00
2.41951e+09
50
78.60%
82.33%
83.49%
84.79%
74.359KiB
139. K39.609KiB
193.047KiB
302.172KiB
5.08541e+08
1,05590e+09
1.50416e+09
2.41957e+09
100
79.05%
83.29%
84.83%
85.32%
86.859KiB
152.
205.547KiB
314.672KiB 2.41967e+09
5.08643e+08
1.05600e+09
1.50427e+09
Table 4: Comparison of our results with previous works, Guerra et al. (020)) binary pruning and
Li et al. 2016) on full precision networks. The source for our results is in Table
Paper
Model
Baseline
Pruned
Memory
Memory Reduction
Computation Reduction
Li et al.
VGG-16 (Full
93.25%
93.40%
21.6MiB
28.8
1.5x
(2016)
Precision)
Guerra et al.
VGG-11 (Bi- nary Connect)
87.60%
86.53%
750KiB
1x
1.4x
(2020)
Custom VGG (30% pruned)
87.64%
87.54%
526KiB
1,4x
3.1x
Ours
Custom VGG
87.64%
86.20%
357KiB
2.0x
4.4x
(20% pruned)
Table 5: Comparison between trained, pruned and retrained network and randomly initialized, pruned and trained BinaryConnect Deterministic network. The baseline accuracy for the Train- Prune-Tie Train case is 87.64%.
Number of inputs
Train-Prune-Tras
Prune- Train
20%
86.20%
85.83%
30%
87.54%
86.13%
50%
87.71%
86.52%
80%
87.86%
86.60%
The results show that randomly initialized network can be pruned to less than 1.5% reduction in accuracy for the same training parameters. This encourages further investigation in future works on larger models to validate hypothesis. If this phenomenon always observed, it can greatly help reduce the computation cost by removing the need to train larger models initially.

---

Under review as a conference paper at ICLR 2022
7 CONCLUSION AND FUTURE WORK
We were able to demonstrate pruning for binary neural networks and reduce the memory and com- putation requirement by more than 2200x and 70x on MNIST and 190x and 4.4x on CIFAR-IO, respectively, compared to full precision, fully connected equivalents. Our CIFAR-10 results had I % increase accuracy but was only 70% of the size of the model from Guerra et al. (2020) with reduction in computation time by 3.1x vs 1 1.4x in that paper.
The ability prune randomly initialized neural networks and train reasonably high accuracies is an interesting outcome and is worth investigating. Finally, extension of these algorithms for other forms of quantization (logarithmic, k-bit, etc.) would also be an interesting avenue for research.
We are also working on porting the trained weights to an FPGA to observe the gain in energy requirements and inference time. With such low number of inputs (fan in), we expect be able to fit our neurons in lookup tables to further boost the performance.
REFERENCES
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. CoRR, abs/1511.00363, 2015. URL http:
//arxiv. org abs / 11111.11111. 11.5110000 00363
James Diffenderfer and Bhavya Kailkhura. Multi-prize lottery ticket hypothesis: Finding accu- rate binary neural networks by pruning a randomly weighted network. In International Confer- ence on Learning Representations, 2021. URL https:://////...t htttps://openrevieet netrriiewweeeeeeeeeeeeeeeeeee forum?icd
matob9iy
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Training pruned neural net- works. CoRR, abs/1803 00365, 2018. URLhttt:///axxi http://wwww.gg/b///////eaee/ 03635
Luis Guerra, Bohan Zhuang, lan D. Reid, and Tom Drummond. Automatic pruning for quantized neural networks. CoRR, abs/2002.00523, 2020. URL /RLL/tttpss///x....or///////////
00523
Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. CoRR, abs/2102.00554, 2021. URL htt.h:tpp////..m..... /tappp:///axxi..../ www.v.org/aaaa wttp/ag///aaaaaa//ee////pe 200.ab/./202554 00554
Alex Krizhevsky and Geoffrey E. Hinton. Learning multiple layers of features from tiny images. Master's 's thesis, Department of Computer Science, University of Toronto, 2009.
Yann LeCun, Leon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition". Proceedings of the IEEE, 86(1 1 11:2228-22344 November 1998.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for wbw.orrgaabbbl0g.//
efficient convnets. CoRR, abs/ 1as//600008000 2016. URL 08710
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in NLP. CoRR, abs/1906.022 022443, 2019. URL http:// http:///aaxxvv...//aaaea wrtp://.rrgabbbb/bb/bb/oo 02243
8


Reviews:
[{"review_id": "tZK1x7jmNL", "reviewer": "Reviewer_MtDD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- This paper suggested a new method of pruning binary neural networks.\n\nWeaknesses:\n- The paper writing is sloppy.\n    - Algorithms 1 and 2 use notations that are not defined. What are a_k, b, w_k, t? What are find_mask() doing? What is the range of clip function?\n    - The main part is Algorithm 2, and needs better description at the main body. The author should not let the readers to understand on their own after looking at Algorithm 2. \n    - Notation is vague. In algorithm 1, w_b is binary weight, and w_t seems to be the weight at step t. We should distinguish them instead of using the same notation of under-script.\n- I cannot get the novelty of this work.\n    - Where can we check that memory gain is around 2200x and computation gain is around 70x? The authors claim this after mentioning Table 1, but cannot find how Table 1 implies this result.\n    - In Table 4, the gain is not significant.\n- Need better comparison with other schemes\n    - Pruning binary/non-binary neural networks have been considered in various literatures. It is bit surprise that the authors only cited less than 10 papers.\n    - The authors only compared the performance with 2 papers, but I think there should be a large number of pruning techniques for binary neural networks. \n", "summary_of_the_paper": "This paper suggests iPrune, a magnitude based unstructured pruning technique which reduces the number of inputs to each neuron. This scheme reduces the memory, computation, and power consumption for training binary neural networks.\n", "summary_of_the_review": "This paper needs better illustration of their algorithm and better justification of the novelty. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "rS-m2In05F", "reviewer": "Reviewer_1172", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strengths of the paper are:\n\n1) Review and discussion of current approaches that the paper is building on.\n2) The paper presents a good amount of experimental results.\n\nHowever, they are both very limited in scope:\n\n1) The paper only presents very limited approaches on network binarization and pruning - which is a very hot topic, and a ton of research has been done in recent years.\n2) The authors focus on two very modest (easy) datasets, which don't require big models, and on which pruning/binarizations techniques are expected to work.\n\nAdditionally:\n\n3) The paper uses very limited amount of page real-estate to describe the iPrune approach, and immediately jumps to experimental results.\n4) It is not clear what the authors claim as novelty.\n5) The authors claim 190-2200x memory reduction for MNIST, which seems unsubstantiated in the paper. ", "summary_of_the_paper": "The paper introduces an algorithm for pruning binary networks. The authors combine existing binarization approaches (binary connect) and use magnitude based unstructured pruning of the binarized network.", "summary_of_the_review": "Please see above.\n\nIn order for this work to be acceptance quality, the authors should at least provide more details about the iPrune approach and compare it with more state of the art approaches. Additionally, the authors should perform experiments on more challenging datasets in order to substantiate their claims.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "REKdLjZuM9k", "reviewer": "Reviewer_TbnU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors are trying to address an important task in deep learning which is neural network compression. \n\nHowever, the authors introduced too much background of the network binarization and unstructured pruning, while the discussion about the proposed method is not enough. \n\nThe proposed methods are built heavily on the previous methods, it would be better if the authors could propose more novel ideas. \n\nMany technique details are not clear. For example, in Fig.3. the output of the masked weights are integers, which is confusing because the authors claimed that this paper is working on pruning the binary neural networks. \n", "summary_of_the_paper": "This paper introduces a neural network compression method built on magnitude-based unstructured pruning and binarization techniques. ", "summary_of_the_review": "In general, I think this paper is not ready for publish and still has a big room for improvement in terms of novelty, writing, and technique contributions. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "LG3kJNS0BKk", "reviewer": "Reviewer_r65C", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1) I think the main strength of this paper is successful exploitation of two existing method to reduce computational complexity and memory requirements of small datasets. \n2) Extensive simulation results and investigating different aspects of the proposed method is another strength of this paper.\n\nWeaknesses:\n1) The contributions of this paper are not novel. The proposed method is a combination of two existing methods and it overlaps with a previously-published work (please see https://openreview.net/pdf?id=r1fYuytex).\n2) It was tested on small datasets only. It would be great if the same level of memory and computation reduction can be achieved on larger datasets such as ImageNet.", "summary_of_the_paper": "This paper combines two existing methods (i.e., binarization and pruning) to reduce both computational complexity and memory requirements of deep neural networks (DNNs). Binarryconnect method was exploited to binarize DNNs whereas neuron-wise magnitude pruning was used to remove unnecessary input activations. It was then shown that the combination of these two methods can significantly reduce both memory and computation on CIFAR-10 and MNIST datasets. ", "summary_of_the_review": "As mentioned in the paper, this work relies on two existing works. Even though successful adaptation of existing works is challenging and valuable, I believe the contribution of this paper is still very limited since it overlaps with an existing work (i.e., https://openreview.net/pdf?id=r1fYuytex).", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper deals with a problem of significant practical relevance: memory efficient neural networks. The authors propose some pruning methods for binary networks. However, several weaknesses were identified by the reviewers (novelty, lack of extensive experiments, problems with the presentation of the paper), and several valid points of concern were raised. These points of criticism were not adequately addressed, hence the paper in its current form cannot be recommended for publication.
Decision: Reject