Title: Reducing the Computational Cost of Deep Generative Models with Binary Neural Networks
Authors: Thomas Bird
Friso Kingma
David Barber
Venue: ICLR 2021
Keywords: binary
generative
optimization
compression

Abstract: Deep generative models provide a powerful set of tools to understand real-world data. But as these models improve, they increase in size and complexity, so their computational cost in memory and execution time grows. Using binary weights in neural networks is one method which has shown promise in reducing this cost. However, whether binary neural networks can be used in generative models is an open problem. In this work we show, for the first time, that we can successfully train generative models which utilize binary neural networks. This reduces the computational cost of the models massively. We develop a new class of binary weight normalization, and provide insights for architecture designs of these binarized generative models. We demonstrate that two state-of-the-art deep generative models, the ResNet VAE and Flow++ models, can be binarized effectively using these techniques. We train binary models that achieve loss values close to those of the regular models but are 90%-94% smaller in size, and also allow significant speed-ups in execution time.


Review discussion:
The contributions of this paper are the following:
- it extends previous work on binary neural networks to the case of deep generative models which perform density estimation (VAEs and Flows)
- the results show that the proposed approach works well, with the expected trade-off but closely matching the much larger real-valued networks
- in order to do so, it introduces a novel technique for binarizing weight-normalized layers, which are used in these generative models
- the results show the advantage of this technique, and they illustrate it is particularly important for ResNet layers.
The paper is very clear (except for one element noted below). Originality is clear but not very high, as this contribution may be seen as a low-hanging fruit, albeit a useful and well-executed one.  As these kinds of architectures are becoming more and more used in various settings, the techniques used here (e.g. using ResNet layers) may be applicable more widely, increasing the significance of this work.
Minor fixes suggested:
* The semantics of equations 3 and 6 is not clear. I imagine the objective is to explain how the gradient on the binary weights and activations is converted into a gradient on the real-valued ones, but this should be explicited.
* In page 1 (bottom line) and 4 (1st line after eqn 9), the authors cite Rezende & Mohamed 2015, Dinh et al 2017 , but they should include first in this list the Dinh et al 2014 paper which actually introduced the notion found in Flow models, albeit under a different name (NICE).
* Appendix A: the authors should also show the images generated with full precision, so we can compare visually.
We thank the reviewer for their positive remarks and useful points. Find our responses below:
*The semantics of equations 3 and 6 is not clear*
We have tried to be as clear as possible here, and did give some thought to how to structure this table of equations, and we have made the forward and backwards pass for weights and activations explicit. Does the reviewer have any suggestions on how to make the equations more clear? We are happy to update them if so.
*they should include first in this list the Dinh et al 2014 paper which actually introduced the notion found in Flow models, albeit under a different name (NICE)*
This is fair - we have updated the paper with the NICE reference in these places. We did have the NICE reference in the paper, but have now simply added more reference locations.
*Appendix A: the authors should also show the images generated with full precision, so we can compare visually*
Yes we agree that this is useful - we have now updated the samples to add the samples from the models with floating-point precision weights for comparison.
The paper introduces the first way (to the best of authors knowledge) of building generative models with binary weights.
Also the case with binary activations is considered. The authors consider two SOTA generative models (flow++ and RVAE) and develop technique to binarize all weights (and possibly activcations) in residual layers. They show that residual layers can be binarized with relatively small drop in performance and further binarization of remaining blocks in computational graph leads to significant degradation. Binary modification of weight normalization is suggested although no ablation is study is performed so it is unclear how crucial is BWN for robust learning. The training process itself is pretty standard way of training binary DNNs - they use STE + truncation of real-valued weights counter-parts.
I have several questions on the proposed methodology:
1. To compute density in flow++ model you need to be able to differentiate Jacobians. How would you compute them given the inputs are binary? Have you used STE for this purpose? If so then how can we be sure that what we get is correct (normalized) density function? I think this issue should be addressed in the paper since nomalizing flows by design assume continuous variables.
2. Please define what is $n$ in you paper. Is it the total number of all weights in you network/layer or is it a number of weights in single convolution filter?
3. You provide performance of binarized generative models and show that they require a lot less memory than their real-valued analogues. It would great to see what performance can be achieved by smaller real-valued generative model that requires approximately same amount of of memory as binarized network.
Then we could understand what is better - to use large binary networks or small real-valued ones for generative modeling.
4. My major concern is the visual quality of the images obtained by binary models. To be honest after I saw them I decided to downgrade my mark from 6 to 5. It seems that with such quality the model is quite useless. Do I understand right that there is a clear visual difference between objects generated by real-valued and binarized networks of similar size?
We thank the reviewer for their review, with many interesting questions posed. Find our responses below:
*Binary modification of weight normalization is suggested although no ablation is study is performed so it is unclear how crucial is BWN for robust learning*
This point was also raised by other reviewers, so we have performed an ablation and added the results to our paper in appendix B. To summarise our findings - we find it impossible to train the generative models without any normalization, due to unstable training. This is not too surprising, since binary weights are large in magnitude.
When comparing binary weight normalization (BWN) to batch normalization (BN), we find that BN is also much more unstable during training, but can train. We find that BWN achieves better loss values than BN, probably due to BN’s unstable training. Given also that the BWN layers are faster and simpler to compute, we think these ablations validate our hypothesis that BWN is a practical normalization procedure for binary-weighted generative models.
*To compute density in flow++ model you need to be able to differentiate Jacobians. How would you compute them given the inputs are binary?*
This is an interesting question - you are correct that to train the Flow++ model we need to be able to evaluate and differentiate the log determinants of the Jacobian of each respective coupling layer. We recommend referring to section 3.2 of the original Flow++ paper to see a complete description of the method. The key point is that the log determinant of the Jacobian is the log density of the logistic mixture distribution which we parameterize at each coupling layer, so we just have to ensure that we produce a valid log density that we can differentiate through. We are guaranteed to get a normalized log-density simply by virtue of how we parameterize the logistic mixture distribution (similar to how any mean and log-variance always describe a normalized Gaussian distribution).
As such, there is no problem in using the binary weights in these flow coupling layers, since we ensure differentiability by the use of the STE.  We have added a small point to clarify this at the beginning of section 4.
*Please define what is n in you paper. Is it the total number of all weights in you network/layer or is it a number of weights in single convolution filter?*
The n you see in section 3.1 is the dimension of the binary weight vector, and we do say this just before equation 12. As per other normalization schemes, we apply this per output channel.
So for example for a binary weighted convolution layer, n = input_channels * kernel_width * kernel height. Or for a linear layer n = input_channels.
*It would great to see what performance can be achieved by smaller real-valued generative model that requires approximately same amount of of memory as binarized network*
We agree that this would be something interesting to see. We have given some results in this direction with our exploration in section 5.2 of how binary models perform with a larger architecture than a corresponding real-valued model. We show that increasing the width of the residual channels in the binary model does indeed improve performance.
In terms of shrinking the real-valued model to give it the same size as the binary model. We have a result which is close to this - our baseline method with no residual layers (table 1). This only has real-valued weights, and is similar in size to the model with binary residual layers, since the binary layers are so space efficient. It would be quite hard to make a real-valued model with a similar size as the binary model using the same architecture, since you would need to use very small filter sizes on the residual layers, or reduce the number of layers substantially. It would probably result in quite a poor model.
(we continue the response in another comment, due to reaching the character limit)
(continuing the response from part 1)
*My major concern is the visual quality of the images obtained by binary models. To be honest after I saw them I decided to downgrade my mark from 6 to 5. It seems that with such quality the model is quite useless. Do I understand right that there is a clear visual difference between objects generated by real-valued and binarized networks of similar size?*
We thank the reviewer for their honesty here, and this is an important point, since people do attribute a lot of importance to sample quality.
We have now added samples from the real-valued models for comparison.
So we urge the reviewer to look again at the samples and examine the difference. There is a lower quality in the samples from the binary models, but we think the difference is relatively marginal, and the binary samples are still of a decent quality. The difference is of course reflected in the slightly lower likelihoods achieved by the binary models, which can be seen in our quantitative results. We think the difference observable in the samples is in line with what we would expect from the difference in likelihoods (as it has to be), since in both cases the difference is noticeable but relatively small. There is going to be some degradation in the model performance when binarizing the weights, so the key problem is how to minimize it, which we attempt to do with our method.
This leads on to another important point. The class of methods we have examined are generative models with explicit densities, as opposed to implicit models such as GANs.
As such, sample quality is not the primary goal of these methods. The primary goal is density modelling performance, which has a wide range of applications in its own right (e.g. anomaly detection).
Of course, sample quality is always a good metric to examine to give a visual feel for a model, but we think that we should not place too much weight on the sample quality results, since they do not solely determine the usefulness of the model.
Thanks for answering my concerns and for providing samples from real-valued models.
Overall I think that's important topic and the quality of binarized model is not significantly worse than that of real-valued ones (although I still think it's far from being acceptably good). I raise my score to 6.
The authors propose to binarize weights and activations of generative VAE and Flow++ models. As Weight Normalization is commonly used in these models, the authors notice that Euclidean norm of binary [-1;1] vector is a square root of it’s length, such that Weight Normalization can be reduced to affine scaling. They propose to call this scaling Binary Weight Normalization, and evaluate it on CIFAR and ImageNet datasets.
Overall, motivation for binarizing generative models is unclear. To the best of my knowledge, training such generative models is an active research topic, and, unlike in image classifiers, object detectors, language and translation models, are not applied in practice in real systems where execution time and memory footprint matters. Researchers who train such generative would not apply binarization as:
- training such models is already difficult and optimization is often unstable, so adding an extra variable might make it even more difficult;
- to keep baselines simple;
- final execution time does not matter.
It is also very likely that, in case such generative models find applied use-cases where execution time and memory footprint would matter, they would have undergone several research iterations and updates such that new binarization techniques would need to be developed.
As affine scaling is no longer a weight normalization, it is probably incorrect to call it that. An experiment ablating an architecture with/without the proposed binary weight normalization is missing.
Also, as generative models are often evaluated qualitatively, the paper would benefit from including several samples from binarized models, to show that quality does not degrade.
Literature review also lacks several more recent binarization techniques such as [1] and [2] (more can be found there). It is not clear if one were to pick a simple (or more recent) binarization technique they would encounter difficulties.
Overall, due to unclear motivation, introduction of binary weight normalization which is not weight normalization, and incomplete literature review I propose rejection.
[1] Mark D. McDonnell, Training wide residual networks for deployment using a single bit for each weight, at ICLR 2018.
[2] Gu et al. Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagation, at AAAI 2019.
We thank the reviewer for their constructive feedback, with many thoughtful points. Find our responses below:
*Overall, motivation for binarizing generative models is unclear. To the best of my knowledge, training such generative models is an active research topic, and, unlike in image classifiers, object detectors, language and translation models, are not applied in practice in real systems where execution time and memory footprint matters*
We believe there is very clear motivation for binarizing generative models - there are many applications of unsupervised learning which are already in use.
For example:
* text generation models such as GPT-3, this has seen a large amount of interest recently given the breadth of the applications possible with a good text model.
* image generation and related tasks such as inpainting, super-resolution and enhancement. Given the amount of visual data being created, these are inherently useful if they can become practical.
* anomaly detection, where density models are used to detect low-probability points. This is highly relevant for applications of fraud detection in high data environments.
* representation learning and inference about the structure of data.
* compression, both lossless and lossy. Lossless compression can be performed directly by the generative models we use in this paper, along with an appropriate entropy coder. Lossy compression uses similar techniques (usually with an entropy penalisation on a bottleneck layer). This is an important application, given the explosion in the amount of data being transmitted over the internet, and the promising early results of neural compression.
All of these examples are already used by industry, and as such would stand to benefit from decreased memory usage and faster inference times. Especially for compression software, in which the execution time and size of the program is almost as important as it’s compression capabilities.
Given the power of unsupervised learning, it seems that the usages will also only increase as techniques improve.
As such, understanding how to improve the efficiency of these (often quite large) models is key to enabling their widespread deployment.
*training such models is already difficult and optimization is often unstable, so adding an extra variable might make it even more difficult*
Although the Flow++ and ResNet VAE models are stable during training, we do accept your point - binarizing the weights does add noise to the training process. We believe that this is one reason that research into increasing the efficiency of generative models has not already been done - since it is simply harder than doing the equivalent to supervised models. We see this as a reason why our research is valuable, since we have demonstrated that with a certain set of techniques, stable training can be achieved for binarized versions state-of-the-art, complex models.
*to keep baselines simple*
Baselines are not generally optimized heavily for memory and time efficiency, given that they are not designed to be deployed in the wild, but instead used for research purposes. The same point could be made for supervised learning, where there is much research into the efficiency of those models that are candidates for real-world applications.
*final execution time does not matter*
As per our above list, there are plenty of existing use cases, and also many more uses to come in the future.
Therefore, execution time does matter for all users of these models, which can range from consumers on smartphones to large corporations.
*It is also very likely that, in case such generative models find applied use-cases where execution time and memory footprint would matter, they would have undergone several research iterations and updates such that new binarization techniques would need to be developed*
We don’t believe this is necessarily true - most use cases of generative models and unsupervised learning will be off-the-shelf models. This is certainly the case in supervised learning, where popular architectures such as ResNet backbones and the YOLO architectures are used extensively. The same thing seems to be true in unsupervised learning - popular VAE, GAN and transformer architectures are used in a similar format across a variety of applications.
Also, even if the architectures are iterated upon for a particular application, which is definitely possible, then our binarization techniques can still be used as long as the resulting model has residual layers, and maybe even without (using binary weight normalization).
Therefore we think our contribution is still valuable, even if it is true that most applications of generative models required tweaking.
(We continue our response in another comment, since we have reached the character limit)
(continuing our response from part 1)
*As affine scaling is no longer a weight normalization, it is probably incorrect to call it that*
Our method is an affine transformation, in the sense that it is an element-wise product.
However, the same thing can be said for regular weight normalization, and indeed batch normalization. We believe that referring to it as an affine transformation would simply cause confusion, since it breaks the link to existing weight normalization schemes (which our method is clearly a derivative of).
*An experiment ablating an architecture with/without the proposed binary weight normalization is missing*
We agree that this is a valuable experiment, and have added an ablation in appendix B comparing binary weight normalization to batch normalization in the paper. The ablation shows that binary weight normalization achieves better results while also having faster execution and a simpler form. Note also, that the model is too unstable to train without any form of normalization. This is because using binary weights (in particular with binary activations) can result in very large pre-activations, as binary weights are large in magnitude to trained floating-point precision weights.
*the paper would benefit from including several samples from binarized models*
We do include samples from the binary models, as well as from the models with floating-point precision weights for comparison. These are in the appendix, section A. We originally split the appendix into the supplementary material, but have now included with the main text.
*Literature review also lacks several more recent binarization techniques*
We have added references to the two papers you cite, thank you for bringing them to our attention.
*It is not clear if one were to pick a simple (or more recent) binarization technique they would encounter difficulties*
There are various binarization techniques described in the literature. We have examined the most widely used technique, which is using the straight-through estimator and weight/gradient clipping, and we see this as the most useful approach to take due to the scheme’s popularity. We cannot guarantee that our approach will work for all other binarization schemes, but since (to our knowledge) these other schemes are generally much less widely used, we do not think this is much of a negative point.
This paper describes a method to binarize weights and activations of variational autoencoders and flow-based networks.  This is an important issue as these methods are valuable to solving unsupervised problems, but are rapidly growing in size, necessitating large and expensive computing systems. And, the literature of low and binary precision hasn’t considered these use-cases to date.
The choice of ResNet VAE and Flow++ are good representative candidates for methods in use widely today.
However, the largest models we see in the world today are knowledge representation and language models (transformers)). Showing that binarization will work for these models would be a valuable next step.
The authors call this a “framework” to go to binary weights and/or activations, but this is really a set of methods shown to work on a particular NN.
Further work is needed to describe a more generalized set of methods. The authors do a good job of motivating their method for the VAE and Flow++ model, but I don’t see an extraction of a fundamental principle that might be broadly applicable to most neural networks.  A major contribution of the paper is Binary Weight Normalization (BWN) and the binarizing method for residual layers.  There is something deeper there.
I like this work as it is an increasingly important consideration to data science teams in building solutions that can work within real-world constraints.
However, binarization is still seen as a method to try and reduce the computational load of something that is already being done.  This makes it difficult to build systems and computing architectures that commit to binary representations. I see this work as a steppingstone to something more general, which can be the basis of building better and cheaper computing substrates. A stronger paper would have uncovered a universal concept.
Section 5.1: It doesn’t appear there is a really principled way to decide which parameters to binarize.  There is some discussion of residual layers as being the least susceptible to performance degradation, but it would be a stronger paper if some of this was explored a bit more through experimentation. Again, extracting a principle here would be useful to the field.
Section 5.2: This vector of improvement is an entire research topic on its own. I think this might be the most interesting direction highlighted by this paper; how does precision/binarization interplay with hyperparameters to reduce the absolute computing requirements, which increasing algorithmic performance?
Section 5.3 Ablations, where is figure 3?
We thank the reviewer for their kind comments, and constructive feedback. We provide our responses below:
*There is some discussion of residual layers as being the least susceptible to performance degradation, but it would be a stronger paper if some of this was explored a bit more through experimentation. Again, extracting a principle here would be useful to the field.*
We appreciate your point regarding the possibility of extracting a general principle for the binarization procedure of generative models.
Ultimately, we believe that binarizing the residual layers is a principle that can be applied to these models. We do agree that this principle is not completely general, since not all models contain residual layers, and the extent to which they are deployed within models does vary.
However, we still think our principle is useful for practitioners, given that the majority of modern generative models do use residual layers. We also believe that we have provided sufficient evidence that the residual layers are sensible to binarize, given the ablation we perform demonstrating that if the full model is binarized then the performance is significantly worse than if only the residual layers are binarized. We do also motivate the choice to binarize the residual layers, given that are naturally designed to avoid degradation, a primary concern when binarizing weights.
*I think this might be the most interesting direction highlighted by this paper; how does precision/binarization interplay with hyperparameters to reduce the absolute computing requirements, which increasing algorithmic performance?*
We agree - indeed we believe that the best possible performance obtained for a fixed inference computation budget will be to use low-precision weights (e.g binary) for most of the network, with higher precision weights at bottlenecks.
However, the key problem (which we discuss in our paper), is that the current training methods for low-precision weights generally still utilize floating point precision weights during training.
Therefore training itself can be a memory bottleneck.
*Section 5.3 Ablations, where is figure 3?*
Figure 3 is in appendix A. We originally cut off the appendix and submitted it separately, but have now given the appendix after the main paper.

Meta-review:
role: PC
data: comment: The paper develops an approach to training generative models with binary weights. The reviewers are split. Two reviewers regard binary networks and the general theme of reducing the computational cost of training as important, and the presented work as a solid contribution. Two reviewers raise concerns about the motivation and the quality of the results. The authors' responses somewhat alleviated the quality concerns of R3, but not the concerns of R4. Overall, the reviewers lean on the positive side. There is disagreement on the importance of the problem, but there is clearly a non-trivial subset of the community that welcomes research in this direction. The AC supports acceptance.

Decision: Accept (Poster)