Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents a refined theoretical understanding of binary connect via relating the dual-averaging method to generalized conditional gradient descent. Then, a general proximal connect algorithm is presented. Experiments on image classification datasets show marginal improvements over the comparable methods.
main_review: ## Post-rebuttal Update
I thank the authors for clarifying the questions. I would strongly encourage them to include the discussion related to MD and [1] clearly in the main paper to clarify the main contributions of the paper. Upon this condition, I increase the score to 7.

## Strengths
- The summarization of different quantization methods under one umbrella is interesting and useful.
- Even though there are many attempts to provide a wholesome theory for binary connect previously, this paper refines the understanding by including the annealing hyperparameter into the theoretical framework.
- The connection to generalized conditional gradient descent seems interesting.

## Weaknesses
- Not clear mirror descent (MD) would be a special case of proximal connect: In my understanding, dual averaging is a special case of MD [a] and generalizing dual averaging does not mean MD would be a special case of the proposed method. Furthermore, MD supports nonlinear projections (such as tanh or softmax [1]) whereas the proposed formulation supports piecewise linear projections. So I'm not sure PC would be a generalization of the methods discussed in [1]. Please clarify.

- The main idea of this paper is in Eq. 8 (and Eq. 31, if I understand correctly) where the regularizer $r$ is parametrized as opposed to ProxQuant [5]. May be this interpretation should be mentioned early so as to simplify the understanding of the final PC algorithm.


## References
- [a] Sébastien Bubeck. Convex optimization: Algorithms and complexity. Foundations and Trends R in
Machine Learning, 2015.
limitations_and_societal_impact: Adequate
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a generalized conditional gradient (GCG) framework for training weight-quantized neural networks. The framework extends existing BinaryConnect and ProxQuant. Convergence bound is established for general, non-convex problem. Compared to ProxQuant, the proposed theory supports evolving quantizers. The paper also proposes a family of smooth quantizers, parameterized by the horizontal and vertical shift. The proposed ProxConnect outperforms existing approaches on end-to-end training of ResNets.
main_review: Principled training methods for quantized neural networks is a important problem to study. This paper generalizes existing theory on training weight-quantized neural networks with a generalized conditional gradient (GCG) framework. The unified view of existing methods in Sec. 2 is interesting.

However, I think the contribution of this paper is somewhat incremental, as the convergence is already established for smooth and non-convex problems in Bai et al.. The generalization of dual averaging under the GCG framework is one of the main contribution of this paper. However, the advantage of GCG over DA is not sufficiently demonstrated. There lacks a strong showcase to explain why the general theoretical result (Theorem 5.1) is useful. If the authors want to emphasize time-dependent quantizers, I think more explanations should be included. Can we theoretically justify why the proposed quantizers can achieve better results, e.g., through a better convergence bound?

The paper is somewhat hard to read and understand. I am particularly confused on the role of Sec. 4. If my understanding is correct, in the non-convex setting, the dual problem (17) is only a lower bound of the original problem (14). In this case, why is Theorem 4.1 and Corollary 4.2 useful, as they don't directly discuss the original problem? If these theorems are not directly applicable, I would suggest to put more effort on the more relevant Sec. 5. For example, I think Theorem 5.1 needs more explanations. What does the term Delta_tau looks like if r(w) is an indicator function? Can Theorem 5.1 reduces to Theorem 5.1 in the ProxQuant paper if the quantizer is fixed?

Figure 2 and Table 1 are too messy to read. Please consider replacing the random symbols with text. 

Post Rebuttal
====

After reading the authors' response and spent several more hours in reading relevant papers I am convinced that the submission made a non-trivial contribution upon existing works (Li et al., Bai et al.), in the sense that it formally shows the convergence for BinaryConnect for non-convex problems. The natural treatment of increasingly "discrete" proximal map is also interesting. Therefore I increased my rating from 5 to 7 and the confidence from 2 to 3.
limitations_and_societal_impact: This work doesn't have any foreseeable potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper seeks to give a theoretical grounding to the study of the quantisers used in the training of neural networks. The authors use proximal maps as the foundation to construct quantisers, and demonstrate that a host of known quantisation techniques can be cast in this manner.

The authors show that BinaryConnect is a special case of the GCG algorithm, and use this as motivation to construct their own more general version - Proximal Connect - which contains BinaryConnect as a special case.

Results are shown on classification of CIFAR-10, and demonstrate that the proposed method improves performance on this task over BinaryConnect and ProxQuant (the main methods this work focuses on).
main_review: This work has two types of contribution - the theoretical understanding given about already existing quantisers, and the proposal of a new quantiser (PC) which the authors show has advantages over existing techniques.

I think in general the theoretical contributions are valuable to the community, since the quantisation techniques used in training quantised neural networks are somewhat poorly understood. Although the casting of the quantisation problem in the language of proximal maps is not novel, the relation to the various known quantisers in ML is useful.

The authors then show that BinaryConnect (the main quantiser used for binary weights) is a special case of the GCG algorithm. It was already known that BC was a special case of the regularised dual averaging problem, but the GCG does generalise it further. It is not clear to me exactly how valuable this realisation is. At the least, the authors use it to motivate the construction of their PC quantiser, and perhaps there could be more developments that are enabled by this knowledge. 

The new PC quantiser is then demonstrated to improve results on a fine-tuning task for image classification. Although this is a relatively limited evaluation (there is only one experiment, CIFAR-10, considered) there is enough theoretical meat to justify this in my eyes. The improvements are seemingly significant, in that BC and PQ each have failure cases in the given experiments, that PC does not have. Although these failure cases (for example end-to-end training of ternary weights for PQ) could be resolved by other existing techniques, it seems that the entire points of this work is to try and provide a more solid and reliable quantiser by examining overarching theory, rather than developing brittle quantisers for special cases. The PC quantiser itself seems to be relatively straightforward, after all is said and done, in that it is just a linear piecewise function of varying slopes and level intervals around quantisation points. I would say that simplicity is a good thing though, if such a technique is going to be adopted by the community.

I also find in general the work very well-written, although at points it can be dense in notation and a bit tricky to follow through the more involved sections.

Some small points:
 * the authors mention that stochastic quantisation can be cast in the proximal mapping framework. Do they have any thoughts about how the class of methods for training stochastic quantisers might relate to their framework? For example [1] and [2].
* the authors talk about the reverse BC and reverse PC algorithms, giving experimental results for the latter. Although this is clearly a scheme that we can write down, and it is motivated by the PQ algorithm, it does not seem intuitive to me. Why would we want to update the continuous parameters starting from the locations of the current quantised parameters? Perhaps this is just a gap in my optimisation intuition.

[1] LEARNING DISCRETE WEIGHTS USING THE LOCAL REPARAMETERIZATION TRICK
Oran Shayer et al. ICLR 2018

[2] Probabilistic Binary Neural Networks
Jorn W.T. Peters et al. 2018
limitations_and_societal_impact: I do not see any societal impact of the work, since it is a primarily theoretical work focused on quantisation.
review_title: null

Meta-review: After the discussion, all reviewers recommend accept (7). For example, one reviewers emphasized "the submission made a non-trivial contribution upon existing works (Li et al., Bai et al.), in the sense that it formally shows the convergence for BinaryConnect for non-convex problems.". I find this alone sufficient novel. Also, the unification and extension of previously suggested algorithms are quite interesting, and the empirical results are not bad for a theoretical paper. The only concern raising from the reviews seems to be that the paper is dense, and I hope the authors can improve improve readability, if possible.
Decision: Accept (Poster)