Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: This paper compresses neural networks via so called Sparse Binary Neural Network designs. The proposed idea is naïve, directly using a slightly modified sign function to quantize network weights into 0 and 1 instead of commonly defined -1 and 1. Experiments on small MNIST and CIFAR-10 datasets with two shallow and old neural networks are provided.

This paper has obvious weaknesses.

--- Limited novelty

The proposed method is naïve. The authors merely replace  binary weights {-1, 1} by {0, 1}, using common quantization tricks for binary neural networks, such as straight-through estimator (STE) and a slightly modified sign function. The authors claim that such a modification can bring significantly improved compression. However, it is problematic, as it will force all quantized network weights to be non-negative, leading to serious accuracy drop. For example, in Table 3, a shallow VGG-like network on CIFAR-10 dataset shows about 10% absolute accuracy drop compared to the binary weight counterpart. Furthermore,  in the optimization, the authors add two non-negative constraints, which makes the training with STE even more challenging. I believe, experiments on large-scale image classification dataset such as ImageNet with modern CNNs will lead to more serious accuracy drops. 

Actually, more impressive neural network compression yet with good accuracy can be achieved via combing quantization and pruning. There exist numerous works in this field,  e.g., "Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding" in ICLR 2016, "Clip-q: Deep network compression learning by in-parallel pruning-quantization" in CVPR 2018 and "APQ: Joint Search for Network Architecture, Pruning and Quantization Policy" in CVPR 2020. Unfortunately, they are missed by the authors.
 
--- Poor writing

The paper is poorly written, including introduction (messy), related works (poor), proposed method (tedious) and experiments presentation (weak). 

--- Weak experiments

There is no comparison with state of the art CNN compression methods, combining quantization and pruning for improved compression.

The authors only conduct toy experiments, a 3-layer fully connected LeNet on MNIST dataset and a shallow VGG-like network on CIFAR-10 dataset.

Even with toy experiments, results are very weak, showing serious accuracy drop even for a shallow VGG-like network on CIFAR-10 dataset, compared to the binary weight counterpart. 


  
review_title: A clear rejection

final_score: rating: 4: Ok but not good enough - rejection
review_content: this paper propose sparse binary neural network for model compression and efficient inference on IoT devices. The paper combines pruning techniques and binary neural network techniques, to train a model that is ternary: {1, 0, -1}. the paper experimented with four regularization functions: triangular, L2-inspired, piece-wise parabola and a 4th degree polynomial. My concern is that, evaluated on MNIST dataset, the accuracy is a bit worse than the BNN, and the implementation section is thin, therefore it's not clear whether the sparsity gain can offset the implementation overhead for decoding. I'm also concerned about the novelty combining pruning techniques with binary neural network techniques. 
review_title: novelty

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary of the paper:
This paper presents a sparse neural networks with binary weights. The binary weights are represented using 0s and 1s whereas the effective connections are encoded using the run-length encoder for efficient implementations of neural networks on IoT devices. To quantize the value of weights, a non-negative function is introduced, penalizing values far from two discrete values of 0 and 1. The performance of the obtained sparse binary neural network is measured on MNIST and CIFAR10 datasets when using fully-connected and convolutional neural networks. It was shown that sparsity degree of 98% can be achieved at the cost of an accuracy degradation.

Strength:

-- The paper is easy to read

-- The compression rate is very high

-- Comprehensive simulations for different hyper-parameters

Weakness:

-- Misleading notations

-- Simulation results are obtained from small datasets

-- Significant overlaps with previous works (limited novelty)

Detailed comments:
-- Section 2 under the label “Sparsity”: It is not clear what the authors mean by using {0, 1} for weights instead of {-1, +1}. In conventional methods such as binaryconnet, the value of weights is mapped to two discrete values of -1 and +1 where these discrete values can be represented using a single bit (i.e., 0 for representing -1 and 1 for representing +1) for hardware implementation. Do you mean that your method maps the real value of weights to positive values of 0 and 1? If this is the case, please explain how your binary neural network works with positive weights.

-- Section 3.1: According to Eq. (4), 0 is an representation for -1. If this is the case, what is the intuition behind using the function f2(w) to push values towards -1? On the other hand, if 0 is in fact representing a zero-valued weight, then we have a neural networks with positive weights. Please elaborate.

-- Section 3.2: In the definition of Q, we have mapping from [0, 1] to {0, 1}. However, if you use a = -1 and b = +1 in Eq. (4), then the mapping would be from [-1,+1] to {-1, +1}. Please explain this discrepancy.

-- Section 3: In general, the proposed binarization process is very similar to the work of Courbariaux et al. (2016). Please list the differences.

-- Section 3.3: Initializing a network with sparse connections using a Bernoulli function was introduced in [1]. Please explain the differences.

[1] Sparsely-Connected Neural Networks: Towards Efficient VLSI Implementation of Deep Neural Networks (ICLR 2017)

-- Section 4: I would like to see some experimental results on a bigger dataset such as ImageNet. The quantization of neural networks has been well studied during the past few years. However, they fall short in providing a good accuracy for state-of-the-art networks on ImageNet. I would like to see how the proposed method of this paper stands out against other existing works.

-- Section 4: It would be interesting to see a comparison with ternary neural networks (e.g., [2]) in terms of sparsity degree and accuracy. 

[2] Neural Networks with Few Multiplications (ICLR 2016)

-- Section 4: I am also wondering why there is significant gap between the proposed binary network and its full-precision counterpart (which is not included in this paper) given the fact that small datasets are used. Using a high compression rate could be the answer, but it would be interesting to see a curve showing accuracy for different degrees of sparsity (i.e., EC).


--------------------------------------------------------------------------------
Post-rebuttal update: Based on the improvements made during the rebuttal period, I have raised my rating. I believe Fig. 2 now makes the contribution of this paper more clear when compared to the conventional binary network. According to Fig. 2, the proposed method in this paper allows to multiplex and select binary inputs rather than performing XNOR operations, which makes the novelty of this paper rather limited and incremental. Therefore, I still believe this paper stands below the acceptance threshold.


review_title: Introducing sparsity in binary neural networks: high compression rate but limited novelty!

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors are tackling the problem of sparsifying binary neural networks, i.e. to maximize the number of weights that can be set to zero, without compromising accuracy too much. This is a useful endeavor, both for deploying models on embedded devices where memory might be limited, and to reduce the computation work load at inference time, since less active weights mean less operations to be computed during the forward pass.
The strategy that the authors adopt is to modify binary neural networks like those in the mentioned references of Courbariaux et al. by encoding weights as 0/1, instead of -1/1. In practice, this is done by training on continuous weights between 0 and 1 and adding terms in the cost function that promote the weight values 0 and 1 over intermediate values, and a term that tends to sparsify the weights by pushing them to 0.
The authors test their method on MNIST and CIFAR10 training a 3 layer MLP and a modified version of VGG. In these settings the authors carefully study the effect of the regularization terms that they added and that of the sparsity of the initialization. The main result is that the resulting architectures are remarkable in their level of sparseness: the authors achieve sparseness around 2-3%, with a minimal drop of performance on MNIST but unfortunately a quite substantial one on CIFAR-10 compared to Binary NN (from 88% accuracy to 79%).

While practically valuable as a piece as a demonstration of how tolerant Neural Networks are to reduced precision and sparsification, this paper is methodologically quite incremental, if compared to Binary Net and all the subsequent papers deriving from that one.  The demonstrations are moreover limited to MNIST and CIFAR10, meaning that there aren't any indication of how the method would fare on larger datasets like ImageNet, where the advantages of reduced precision and sparsification are arguably going to be most relevant. In addition, even for a relatively simple dataset like CIFAR10 the proposed method seems to reach good levels of sparsity at a considerable cost in terms of accuracy (more than 9%).
Lastly, previous papers on sparsifying ConvNets have pointed out that the actual advantages in terms of memory, computation and energy consumption would come from reducing the number of filters, rather than the number of active weights.
review_title: Conceptually sound paper but methodologically very incremental

Meta-review: This paper compresses neural networks via so called Sparse Binary Neural Network designs. All reviewers agree that the paper has limited novelty. Experiments are only performed on small datasets with simple neural networks. However, even with toy experiments, results are very weak. There is no comparison with the SOTA. Numerous related works are missed by the authors. Besides, the paper is poorly written, and there are misleading notations.
Decision: Reject