Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
This paper addresses the problem of slower training speed with low-precision training of neural nets. It presents a simple solution: first train with full precision on half of the budgeted trainining time, then train with low pecision in the remaining time. This achieves 1.2x - 1.6x speedup compared to low-precision training. 

Pro:
- The proposed idea is simple and it is nice to see that it works

Cons:
- I feel there is not enough content (in terms of ideas and experiments) to warrant a full paper. Compared to other ICLR papers, the contributions seem on the low side. See suggestions below. 
- The paper mainly compares the proposed method with low-precision training, but the results would be stronger if also compared with full-precision training followed by quantization. This is especially because all the low-precision accuracies trail behind the full precision ones in the results. 

Suggestions:
- Experiment with more quantization methods. Currently we do not know whether the proposed method is uniquely suited to the PACT method used, or is a general technique. 
- A 1.2x-1.6x speedup on a training process that takes 600 seconds (e.g. CIFG-10 result in Fig 2) seems not so impactful in the grander scheme of things. Even the 12500 second training time is just <4 hours. I understand the results should transfer, but the results would be more impressive if done on larger training runs.

Minor questions/comments:
- Please comment on the terminology. Is what you call low precision training similar to the mixed-precision training now implemented in TensorCore NVIDIA GPUs?
- Table 2: what is it/s. Is it iterations per second? 
- Table 1 shows learning rate schedule for t up to 60k in CIFAR or 450k in ImageNet, but Figure 1 stops way before those points in the x-axis. This was somewhat confusing. 
- Another clarification point about Fig 1 and 2: for the proposed method, is the full precision training part of the time included in the calculation? I believe so but just want to double-check. 

review_title: More comparisons and longer training runs will make the result more impressive

final_score: rating: 5: Marginally below acceptance threshold
review_content: #### Comments
Summary:

The authors propose a fast binarized neural network training algorithm that splits the whole training process into the full precision training stage and the binary training stage. The experimental results show some improvement about training speed in terms of iterations and wall clock time. Generally, the paper is well written.

Strength:
--The idea is reasonable and the method is presented clearly.
--The experimental results indicate that the proposed method can accelerate the convergence of the binary networks on image classification and collaborative filtering.

Weakness:
--The comparison between the proposed method and quantization-finetuning method is lacked, which seems like a closely related work.
--The analysis of the proposed method should be also enhanced. The reason why partial pretraining can improve the training of binary neural networks should be investigated more deeply.

Comments:
(1) The authors claim that the proposed method allows for faster from-scratch training of binarized neural. This seems contradictory to the partial pretraining. The authors may provide more discussion and clarification. 
(2) The improvement of the proposed method is not significant. The proposed algorithm speeds up the training marginally and cannot improve the final test accuracy. This limits the contribution.

Overall, the reviewer doesn’t recommend accepting this manuscript at its form. The author may demonstrate more differences between the proposed method and the standard quantization-finetuning method.

review_title: review

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposed one simple method called partial pre-training to speed up the training of binary neural networks (BNN). The pros and cons are as follows:

Pros:
1. The partial pre-training method is simple and easy to implement;
2. For standard binary optimizer like the straight-through-estimator (STE), the method improves the training speed to some extent;

Cons:
1. The main concern of the proposed method is kind of heuristic and there is a lack of theoretical explanation, whether rigorous or not, why this method works. 

2. As described in Section 7 by the author themselves, there are several apparent limitations of current evaluation, e.g., several dimensions of the hyper-parameters are not explored, other Binary optimizers are not considered, different learning rate schedules, etc. As a result, it is unconvinced that the partial pre-training could universally improve the speed as a general method. In addition, the improvement of speed-up are not very apparent especially for ResNet-20 and ResNet-34 as shown in Fig. 1 and Fig. 2, i.e., it took approximately the same time to reach the final precision even though the proposed method achieves higher accuracy before saturation. Given the inadequate evaluations and lack of theoretical explanations, this might be due to unfair comparison. 

3. There is a lack of explanation of the contradiction result with previous result proposed in Alizadeh et al (2019), which dismissed the approach of pre-training. Is there good explanation of such an opposite result? It would be better to show results of different split between full-precision training and low-precision training. 

4. Regarding pre-training for BNN, there are some related works from the Bayesian perspective. In Shayer et al. (2018), they used the result of full-precision training as the prior for the binary training, which improves the final result, as opposed to Alizadeh et al (2019). The Bayesian perspective provides an explanation of the effectiveness of a good prior. In Meng et al. (2020), they showed that STE could be viewed as Bayesian and obtained good result even with a uniform prior. Also, the posterior obtained after full-training (binary) could be used as prior to enable continual learning, which shows effectiveness of the prior. Given the above results, since the authors demonstrate that partial pre-training can increase the speed for STE, does this imply that partial pre-training provides a better prior than full pre-training? If so, why? 

Shayer, O., Levi, D., and Fetaya, E. Learning discrete weights using the local reparameterization trick. ICLR, 2018.
Meng, X, Bachmann. R., Khan. E. Training Binary Neural Networks using the Bayesian Learning Rule. ICML, 2020.






review_title: Providing a heuristic method to speed up low-precision training 

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper suggests a method for training binary neural networks. The proposed method is to partially train with full precision and then continue with binarized training using the straight-through estimator. The method is very simple and there is very limited technical contribution, so in order to be worthy of publication it needs to be supported with compelling experimental results. Unfortunately this is not the case.

The main claim is speeding up training by a factor of 1.2/1.6. While this can help the importance of speeding up training (unless by a much larger factor) is quiet limited. The useful speedup BNN present is at inference time, and the time to train a network is of much less importance (at least for this kind of speedup).
From Fig.1 I am not convinced that the speedup claims hold. You can see the test accuracy for binary training on 3 of the experiments reaches the Partial Pre-training level but it doesn't look like it completely flattened out yet. It looks like if you want take the best model on test, then you don't get any speedup
Results on cifar-10 seem quiet poor (both full precision and 1-bit). For example "Learning discrete weights using the local reparametrization trick" gets 93.2 1-bit acc on cifar-10 with VGG.
review_title: Little novelty and weak results

Meta-review: ## Description 

The paper asks the question whether it is possible to accelerate training a binarized neural network from scratch to a given target accuracy [by starting with training a full-precision network]. The main claimed contributions are: the idea to use *partially* pretrained networks, experimental evidence regarding the split of the training budget and measuring the speed-up.

## Review Process and Decision

All four reviewers agreed in the low rating of the paper and in the opinion that the paper is not a significant contribution. The area chair supports rejection.

## Details

It has been already observed that pre-training  in some form is needed for achieving the best accuracy: 
Rastegari (2016) XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks
Bulat (2019), "Improved training of binary networks for human pose estimation and image recognition"
Martinez (2020): "Training Binary Neural Networks with Real-to-Binary Convolutions"

Alizadeh, (2019 fig. 4) notice that pre-training can be viewed as a speed-up, but in their setup find that training from scratch gives a better accuracy. Bulat (2019) and Martinez (2020), on the contrary do use pre-training to achieve the best accuracy.  It is questionable whether the pre-training in these works is partial or not. I believe the result largely depends on the pre-training method used, which is not discussed in depth in the submission. More generally, some graduated optimization methods such as graduated smoothing or graduated non-convexity are known to help in finding better solutions / lead to faster optimization and in fact Bulat (2019) use pre-training with gradual transition from smooth activation to the sign function.  Relative to these points the technical contribution (one paragraph in the paper) is not significant. The empirical part of the contribution shows some effect, but does not indicate a breakthrough on its own. An investigation / design of pre-training schemes could make it more substantial.

The empirical analysis proposed does not rule out, and in fact supports, the methodology that for the best final accuracy, the full rather than partial pre-training is useful.

The gain of speed-up by a factor 1.3 (diminishing to close to 1 if we are interested in the best accuracy), is of little practical interest. In particular, a slight code optimization can give a similar speed-up without the complexity and hyperparameters involved in pre-training. The authors write  "we are not aware of any effort to exploit binarization during the training phase"
There are available public implementations that can optimize the forward pass of binary networks, in particular on GPU, while backward pass can stay in full precision. It could give a similar speed-up. In particular Courbariaux (2016) provides a GPU kernel and proposes a variant of BatchNorm with bit shifts rather than multiplications, specifically used at training time.
Making the emphasis on a relatively small speed-up that can be obtained to train sub-optimal models, in my view is not a good strategy to present this work. Rather the phenomenon that (partial) pre-training helps with the goal to improve the training methods more substantially I find of higher interest.

Finally, I agree with the reviewers that the lottery ticket hypothesis (Frankle, 2020) work speaks of the speed-up only hypothetically and its main (and fairly in-depth) contribution is in demonstrating and investigating an interesting phenomenon about training and initialization, which I do not see relevant to this submission.

Decision: Reject