Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a method to train individual bits of weights of a neural network.  The central motivation is the question of why doesn't gradient descent "discover" inherent sparsity by setting certain weights to zero? The authors suggest that since there are so many possible states of a 32b integer number, the probability of landing on all zeros is vanishingly small. Using the bit-wise training technique, they are able to demonstrate that good performance can be achieved with few bits of weight representation. The authors also do a series of experiments where only certain bit positions are changeable to demonstrate which positions are most relevant to getting good classification performance.
main_review: The authors provide a nice background of pruning literature and sparse representations in neural networks.  I believe they were trying to provide a background for the idea that sparseness is something inherent and should/could be uncovered by gradient descent.  However, the background citations are for a slightly different, but related topic; pruning techniques are not exactly what this paper is about, but it does share some background with spares representations.

In motivation, for line" One possible explanation is that, at least for classification tasks, the usual cross–entropy loss without ad- ditional regularization techniques are not well suited for this" there needs to be citation.

The description of STE in section 3 needs more description.

The experimental results look very preliminary.  Using only LeNet and CIFAR10 might be a good way to triage a technique, but it is very different to draw any conclusions based on these.  They are too small and the results between them barely suggest any trend.  The authors essentially show a known result that training only sign bits (Ivan and Florian 2020) yields good results.  What additional science have the authors uncovered? 

The last section of encoding messages in weights seems quite unrelated to the rest of the paper.  It is interesting, but the results seem random and they don't give us any new insight into the learning process of neural networks.  
summary_of_the_review: This paper has some interesting concepts but falls short on uncovering novel insight.  They are able to recapitulate other papers' results, but even this seems a bit tenuous. The experimental support for any conclusions is too weak in this paper to draw any real conclusions.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes arithmetic decomposition and training on individual bits in order to achieve low-precision quantization.
main_review: Unfortunately, there are serious issues with this submission:

The introduction and motivation seem to be written for a paper that is on pruning not quantization, and they are out of context compared to the title, abstract, and rest of the paper. This is most likely a LaTeX error. The motivation of the paper and comparison to prior art being excluded, it is very hard to assess the quality of the work. I did however try to extrapolate what the authors intended to write, and below is my review for Section 3 and onwards.

In Section 3, what is the expression for \alpha in terms of layer dimensions and number of bits such that the He initialization standard deviation condition is satisfied? This seems like an interesting result that can be added inline in the paper, rather than just implicitly mentioning that an expression for \alpha was derived and used.

What is a He distribution? In (He, 2015), variance engineering is done and the contributions are conditions on the variance of the initialized weights. The distributions themselves are either uniform or normal. Can the author clarify what they mean by He distribution, I believe they mean He conditions on variance.

The experiments are performed on very trivial networks deployed on the MNIST and CIFAR-10 dataset. Can the authors evaluate their work on more contemporary networks such as ResNet on ImageNet and similar tasks?

Message encoding in the neural network's weight using steganography in training is interesting. However, why does it matter? And can these results be generalized on larger networks.
summary_of_the_review: Unfortunately, it seems this paper was rushed into submission. The idea sounds interesting, however, the proposed manuscript has several serious issues as raised in my main review. I therefore recommend rejection of this paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to directly train the bit values of each parameter in a neural network, instead of directly optimizing the floating point value of each weight. By varying the number of bits that are allowed to be optimized, the authors show that with less bits the network will become automatically sparser. This method has many interesting applications, including fixing some bits to be a message and only training the rest.
main_review: Strengths:
- This paper experiments with an interesting an intuitive idea.
- There are many interesting applications, e.g., more efficient networks and embedding hidden messages in network weights.
- Most figures include error bars.
- Figure 3 confirms a main hypothesis (fewer bits encourages sparser networks).

Weaknesses:
1) The paper could benefit from medium scale experiments, e.g., ImageNet. The method matches standard training on ImageNet but faces accuracy degradation on CIFAR. A concern is that this accuracy degradation would be even more substantial for problems such as ImageNet.
2) The paper would benefit substantially from a related work section. Since the paper is not 9 pages, there is definitely room for this. I am not an expert on quantization (perhaps another reviewer is) but I know that it is a very active research area. How does this papers method compare to standard methods in quantization? If the author's hypothesis is correct, networks trained with various quantization techniques should be sparse and it would be very interesting to verify this.
3) There is no discussion of how much extra compute / FLOPs is incurred by this method during training, which may be a drawback. 
summary_of_the_review: The paper is very interesting but would benefit substantially from 1) medium scale experiments (e.g., ResNet on ImageNet) and 2) a more thorough discussion and comparison with related work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a neural network training technique such that individual weight bits can be optimized separately. In detail, each weight is represented as a weighted sum of its bits weighted by powers of 2. In training, updating each bit $b$ is achieved by updating a floating-point number $x$ ($b=1, x > 0; b=0, x \le 0$).

By conducing extensive experiments, the authors find:
1) Network with shorter bit-width show more weight sparsity than that with longer bit-width. (Sec 4.)
2) With selective bit training, only a few most significant bits contribute to the final high model accuracy. The other less important bits serve as regularization. (Sec 5.)
3) The less significant bits can be used to encode other information. (Sec 6.)
main_review: Here are the strengths and weaknesses of the paper.

Strengths:
1. The paper is interesting and, in some sense, novel in that it analyze the network weights/bits and regularization in an interesting perspective. By decomposing a weight into separate bits, the function of each bit can be more easily observed and analyzed.
2. The authors conducted extensive experiments to demonstrate different phenomena from the bit-wise training idea.

Weaknesses:
1. In the second paragraph, Page 5, I guess it should be "as shown in Figure 4" instead of "as shown in Figure 5".
2. In second sentence in Sec 5.1 is quoted here: "In contrast, standard training does not reveal which weights or bits contribute most to the network’s performance". I didn't find the following paragraphs support this claim. In contrary, the paragraphs show the most significant bits contribute more to the performance than the others. In network quantization, e.g., 32-bit to 8-bit, one can also find that the least significant bits can be dropped off without hurting performance much.
3. In second paragraph of Page 4, "..., in general, neural networks can be trained well only by changing the sign of the weights and never updating their magnitudes". I don't think 4-percentage drop is small considering Cifar10 is a small dataset and ResNet-18 is relatively a large model. When using a smaller network, the performance drop might be big.
4. Throughout the paper (e.g., Sec 5.1 and Sec 6), one main claim of the paper is "a few of the most significant bits contribute to achieving a high accuracy, while the others provide regularization". This is true but not a significant observation from the perspective of network quantization. When network is big and contains redundancy, the network can be quantized to lower-bit one (e.g., 3 bit) with comparable performance (e.g., [1]). Compared with the full-precision model, the quantized model is well regularized.

Questions:
1. As the paper hypothesizes, the 32-bit model does not have much zero weights because the probability of an exactly zero-valued weight is very small (1e-31). If this is true, one should expect the trend between 2 and 14 bit-width in Figure 2 is exponential instead of flat. I didn't find an explanation in the relevant section.

[1] Zhang, Dongqing, et al. "Lq-nets: Learned quantization for highly accurate and compact deep neural networks." Proceedings of the European conference on computer vision (ECCV). 2018.
summary_of_the_review: The perspective of the paper is interesting, but some claims might need correction. Moreover, the observation is straightforward, especially from the view of network quantization.
review_title: null

Meta-review: This paper presents some interesting new ideas on training binary neural networks. However, as many reviewers point out the study is quite limited by their experimental section, and some technical issues were raised. These are criticisms that remain largely unaddressed after the author response, hence the paper is not ready for publication at its current form.
Decision: Reject