Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: The paper addresses the issue of training feed-forward neural networks with memory constraints. The idea is to start by training a very small network, binarise this network, then reuse the non-signed bits of the binarised weights to add/store new weights, and recursively repeat these steps. The cost of reducing the memory storage is the extra computation. An experiment on MNIST shows the efficacy of the proposed recursive scheme.

Quality and significance: The proposed method is a combination of the binarised neural network (BNN) architecture of Courbariaux et al. (2015; 2016) with a network growing scheme to reduce the number of bits per weight. However, the computation complexity is significantly larger. The pitch of the paper is to reduce the "high overhead of data access" when training NNs on small devices and indeed this seems to be the case as shown in the experiment. However, if the computation is that large compared to the standard BNNs, I wonder if training is viable on small devices after all. Perhaps all aspects (training cost [computation + time], accuracy and storage) should be plotted together to see what methods form the frontier. This is probably out of scope for ICLR but to really test these methods, they should be trained/stored on a real small device and trained/fine-tuned using user data to see what would work best.

The experiment is also limited to MNIST and fully connected neural networks.
review_title: Nice trick on reusing non-sign bits to recursively add more weights during training, but high computation cost and ideally need more experiments

final_score: rating: 6: Marginally above acceptance threshold
review_content: There could be an interesting idea here, but the limitations and applicability of the proposed approach are not clear yet. More analysis should be done to clarify its potential. Besides, the paper seriously needs to be reworked. The text in general, but also the notation, should be improved.

In my opinion, the authors should explain how to apply their algorithm to more general network architectures, and test it, in particular to convnets. An experiment on a modern dataset beyond MNIST would also be a welcome addition.

Some comments:
- The method is present as a fully-connected network training procedure. But the resulting network is not really fully-connected, but modular. This is clear in Fig. 1 and in the explanation in Sect. 3.1. The newly added hidden neurons at every iteration do not project to the previous pool of hidden neurons. It should be stressed that the networks end up with this non-conventional “tiled” architecture. Are there studies where the capacity of such networks is investigated, when all the weights are trained concurrently.

- It wasn’t clear to me whether the memory reallocation could be easily implemented in hardware. A few references or remarks on this issue would be welcome.

- The work “Efficient supervised learning in networks with binary synapses” by Baldassi et al. (PNAS 2007) should be cited. Although usually ignored by the deep learning community, it actually was a pioneering study on the use of low resolution weights during inference while allowing for auxiliary variables during learning.

- Coming back my main point above, I didn’t really get the discussion on Sect. 5.3. Why didn’t the authors test their algorithm on a convnet? Are there any obstacles in doing so? It seems quite important to understand this point, as the paper appeals to technical applications and convolution seems hard to sidestep currently.

- Fig. 3: xx-axis: define storage efficiency and storage requirement.

- Fig. 4: What’s an RSBL? Acronyms should be defined.

- Overall, language and notation should really be refined. I had a hard time reading Algorithm 1, as the notation is not even defined anywhere. And this problem extends throughout the paper.
For example, just looking at Sect. 4.1, “training and testing data x is normalized…”, if x is not properly defined, it’s best to omit it;  “… 2-dimentonal…”, at least major typos should be scanned and corrected.
review_title: Not ready yet; needs more work

final_score: rating: 7: Good paper, accept
review_content: The idea of this work is fairly simple. Two main problems exist in end devices for deep learning: power and memory. There have been a series of works showing how to discretisize neural networks. This work, discretisize a NN incrementally. It does so in the following way: First, we train the network with the memory we have. Once we train and achieve a network with best performance under this constraint, we take the sign of each weight (and leave them intact), and use the remaining n-1 bits of each weight in order to add some new connections to the network. Now, we do not change the sign weights, only the new n-1 bits. We continue with this process (recursively) until we don't get any improvement in performance. 

Based on experiments done by the authors, on MNIST, having this procedure gives the same performance with 3-4 times less memory or increase in performance of 1% for the same memory as regular network. 

I like the idea, and I think it is indeed a good idea for IoT and end devices. The main problem with this method that there is undiscussed payment with current hardware architectures. I think there is a problem with optimizing the memory after each stage was trained. Also, current architectures do not support a single bit manipulations, but is much more efficient on large bits registers. So, in theory this might be a good idea, but I think this idea is not out-of-the-box method for implementation.

Also, as the authors say, more experiments are needed in order to understand the regime in which this method is efficient. To summarize, I like this idea, but more experiments are needed in order to understand this method merits. 
review_title: This work suggest how to train a NN in incremental way so for the same performance less memory is needed or for the same memory higher performance can be achieved. 

Meta-review: This is an interesting paper and addresses an important problem of neural networks with memory constrains. New experiments have been added that add to the paper, but the full impact of the paper is not yet realised, needing further exploration of models of current practice, wider set of experiments and analysis, and additional clarifying discussion.
Decision: Reject