Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes to use the mirror descent algorithm for the binary network. The key point is Theorem 3.1, which enables the mirror map. The paper is easy to read and follow, and the main contributions are clearly stated.

However, I suggest a weak rejection of this paper. The reasons are

Q1. As Review #3, it is better for authors to provide more theoretical analysis, which better includes the nonconvex objective function and the effect of annealing. 

Q2. It is not clear to me, why mirror descent is better than proximal gradient descent, i.e., proxQuant, in this application. The authors repeatedly claim "MD allows gradient descent to be performed on a more general non-Euclidean space". This cannot be told by Table 1, which is just overall performance. So, it is better to empirically show this point by an ablation study.

Q3. Since the technical contributions are not enough, I expect more experimental comparisons.
- Could the authors perform experiments on ImageNet?
- While VGG and ResNet are taken as a protocol for experimental comparison, it is better to do an extra comparison with STOA networks. VGG and ResNet are too old and easy to be compressed, compression these networks are of little practical values. EfficientNet [1], Mobilenets [2], and Shufflenet [3] can be good ones. The paper will be more convincing with these methods.

[1]. EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
[2]. Mobilenets: Efficient convolutional neural networks for mobile vision applications
[3]. Shufflenet: An extremely efficient convolutional neural network for mobile devices
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: A good paper that uses the Mirror Descent paradigm for learning quantized networks.  
Though Mirror Descent is not their original idea, but using it in the context of learning quantized network is novel and interesting.  
Empirically, they showed better results than existing method, with comparisons with reasonable baselines including using relaxed projected gradient descent.  

Overall, I don’t have much concerns, but here are some more specific comment/questions (most relates to writing)

In the intro, it would be great to mention some past success on using MD, as opposed to just saying it’s well-known. Also you mention MD can be used for more than quantization, but compression in general, it’d be better to add that discussion, or remove this sentence. 

In the beginning of Section 2.1, it'd be easier for the readers to make clear that the primal space corresponds to the quantized weights and the dual space corresponds to the unconstrained space in the rest of the paper.

At the top of page 3 you describe MD for the first time, but it’s unclear to me how y^0 is handled.

The end of section 3 and section 4 talk quite a bit about STE, maybe it'd be clear if the authors can provide a concise description.

As someone not super familiar with NN quantization, this work seems like a good contribution.  My only possible concerns would be somehow comparisons to existing methods are not comprehensive enough (if this will be pointed out by the other reviewers)


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper proposes a Mirror Descent (MD) framework for the quantization of neural networks, which, different with previous quantization methods, enables us to derive valid mirror maps and the respective MD updates. Moreover, the authors also provide a stable implementation of MD by storing an additional set of auxiliary dual variables. Experiments on CIFAR-10/100 and TinyImageNet with convolutional and residual architectures show the effective of the proposed model. 

Overall, this paper is well-written and provide sufficient material, both theoretical and experimental evidence to support the proposed method. Although the novelty of this work is somehow limited, i.e. appling MD from convex optimization to NN quantization, the authors provides sufficient effort to explore how to success to adopted it the literature. Hence, I lean to make an accept suggestion at this point. 

Concern: it would better to provide the code to validate the soundness of the model.

##post comments
The rebuttal addresses my concerns and I will not change my score. Thanks.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes a neural network (NN) quantization based on Mirror Descent (MD) framework. The core of the proposal is the construction of the mirror map from the unconstrained auxiliary variables to the quantized space. Building on that core, the authors derive some mapping functions from the corresponding projection, i.e. tanh, softmax and shifted tanh. The experimental result on benchmark datasets (CIFAR & TinyImageNet) and basic architectures (VGG & ResNet-18) showed that the proposed method is suitable for quantization. The proposed method is a natural extension of ProxQuant, which adopted the proximal gradient descent to quantize NN (a.k.a $\ell_2$ norm in MD). Different projections in NN quantization lead to different Bregman divergences in MD. 

However, the authors do not analyze the convergence of the MD with nonconvex objective function in NN quantization neither how to choose the projection for mirror mapping construction. Moreover, it is better to discuss with [Bai et al, 2019] to clarify the novelty of the proposed method. So I concern about the novelty and the theoretical contributions 

Yu Bai, Yu-Xiang Wang, Edo Liberty. 
ProxQuant: Quantized Neural Networks via Proximal Operators. ICLR 2019.
review_title: Official Blind Review #3

Meta-review: The paper proposes to use the mirror descent algorithm for the binary network. It is easy to read. However, novelty over ProxQuant is somehow limited. The theoretical analysis is weak, in that there is no analysis on the convergence and neither how to choose the projection for mirror mapping construction. Experimental results can also be made more convincing, by adding comparisons with bigger datasets, STOA networks, and ablation study to demonstrate why mirror descent is better than proximal gradient descent in this application.
Decision: Reject