Reducing the Computational Cost of Deep Generative Models with Binary Neural Networks
Authors: Thomas Bird
Friso Kingma
David Barber
Venue: ICLR 2021 Poster (2021)
Keywords: binary
generative
optimization
compression

Abstract: Deep generative models provide a powerful set of tools to understand real-world data. But as these models improve, they increase in size and complexity, so their computational cost in memory and execution time grows. Using binary weights in neural networks is one method which has shown promise in reducing this cost. However, whether binary neural networks can be used in generative models is an open problem. In this work we show, for the first time, that we can successfully train generative models which utilize binary neural networks. This reduces the computational cost of the models massively. We develop a new class of binary weight normalization, and provide insights for architecture designs of these binarized generative models. We demonstrate that two state-of-the-art deep generative models, the ResNet VAE and Flow++ models, can be binarized effectively using these techniques. We train binary models that achieve loss values close to those of the regular models but are 90%-94% smaller in size, and also allow significant speed-ups in execution time.

Published as a conference paper at ICLR 2021
REDUCING THE COMPUTATIONAL COST OF DEEP GENERATIVE MODELS WITH BINARY NEURAL NETWORKS
Thomas Bird*, Friso H. Kingma" David Barber* Department of Computer Science, University College London <firstname> <aurname>@ca uul. aca.acc.uu uk fhkingma@gma il com
ABSTRACT
Deep generative models provide powerful set of tools to understand real-world data. But as these models improve, they increase in size and complexity, so their computational cost in memory and execution time grows. Using binary weights in neural networks is one method which has shown promise in reducing this cost. However, whether binary neural networks can be used in generative models is an open problem. In this work we show, for the first time, that we can successfully train generative models which utilize binary neural networks. This reduces the computational cost of the models massively. We develop a new class of binary weight normalization, and provide insights for architecture designs of these binarized generative models. We demonstrate that two state-of-the deep generative models, the ResNet VAE and Flow++ models, can be binarized effectively using these techniques. We train binary models that achieve loss values close to those of the regular models but are 90%-94% smaller in size, and also allow significant speed-ups in execution time.
INTRODUCTION
As machine learning models continue to grow in number of parameters, there is a corresponding effort to try and reduce the ever-increasing memory and computational requirements that these models incur. One method to make models more efficient is neural networks with weights and possibly activations restricted to be binary-valued (Courbariaux et al.] 2 [Cuubariiux et al.) 2016) Rastegari et al., 2016 McDonnell 2018 Gu et 2018). Binary weights and activations require significantly less memory, and also admit faster low-level implementations of key operations such as linear transformations than when using the usual floating-point precision.
Although the application of binary neural networks for classification is relatively well-studied, there has been research that be used effectively in unsupervised learning problems. Indeed, many of the deep generative models that are popular for unsupervised learning do have high parameter counts and are computationally expensive Vaswani et al. 2017 Maaløe et al. 2019; Ho et al.) 2019a). These models would stand to benefit significantly from converting the weights and activations to binary values, which we call binarization for brevity.
aware of that has examined whether binary neural networks can
In this work we focus on non non-autoregressive models with explicit densities. One such class of density model is the variational autoencoder (VAE) (Kingma & Welling 2014 Rezende et al., 2014), latent variable model which has been used to model many high-dimensional data domains accurately. The late-off-hhcart VAE models tend to have deep hierarchies of latent layers, and have demonstrated good performance relative to comparable modelling approaches (Ranganath et al.) 2016 Kingma et al. 2016; Maaløe al. 2019). Whilst this deep hierarchy makes the model powerful, the model size and compute requirements increases with the number latent layers, making very deep models resource intensive.
Another class of density model which has shown promising results are flow-based generative models (Dinh et al. 2014 Rezende & Mohamed] 2015 Dinh et al. 2017). These models perform series

---

Published as a conference paper at ICLR 2021
of invertible transformations to simple density, with the transformed density approximating the data-gent settaeeeerating distribution. Flow models which achieve state-of oftee-of-the-ar performance compose many transformations to give flexibility to the learned density (Kingma & Dhariwal 2018 Ho et al. 2019a). Again the model computational cost increases as the number of transformations increases.
To examine how to binarize hierarchical VAES and flow models successfully, we take two models which have demonstrated excellent modelling performance the ResNet VAE (Kingma et al. 2016) and the Flow++ model (Ho et al.) 2019a) and implement the majority of each model with binary neural networks. Using binary weights and actívations reduces the computational cost, but also decreases the representational capability of the model. Therefore our aim to strike balance between reducing the computational cost and maintaining good modelling performance. We show that it is possible to decrease the model size drastically, and allow for significant speed ups in run time, with only minor impact on the achieved loss value. We make the following key contributions:
We propose an efficient binary adaptation of weight normalization, reparameterizatioo technique often used in deep generative models to accelerate convergence. Binary weight normalization is the in binary neural networks.
moddlling alternative to the usual batch normalization used
We show that we can binarize the majority of weights and activations in deep hierarchical VAE and flow models, without significantly hurting performance. We demonstrate the corresponding binary architecture designs for both the ResNet VAE and the Flow++ model.
We perform experiments on different levels of binarization, clearly demonstrating the trade-off between binarization and performance.
BACKGROUND
In this section we give background on the implementation and training of binary neural networks. We also describe the generative models that we implement with binary neural networks in detail.
2.1 BINARY NEURAL NETWORKS
In order to reduce the memory and computational requirements of neural networks, there has been recent research into how to effectively utilise networks which use binary-valued weights wa and possibly also activations O'n rather than the usual real-valued weights and activations (Courbariaux
et al. 2015; Courbariaux et al. 2016) Rastegari et al. 2016 McDonnel] 2018: Gu et al. 2018). In this work, we use the convention of binary values being in 11
Motivation. The primary motivation for using binary neural networks is to decrease the memory and computational requirements of the model. Clearly binary weights require less memory to be stored: x2x less than the usual 32-bit floating-poin weights.
Binary neural networks also admit significant speed-ups. A reported 2 speed-up can be achieved by layer with binary weights and real-valued inputs (Rastegari al. 2016). This be made an additional 29 x faster if the inputs to the layer are also constrained to be binary (Rastegari et al. 2016). With both binary weights and inputs, linear operators such convolutions the inexpensive XNOR and bit-count binary operations. A simple way to ensure binary inputs to a layer have a binary activation function before the layer (Courbariaux et al. 2016 Rastegari et al.) 2016).
be implemented using
Optimization. Taking a trained model with real-valued weights and binarizing the weights has been shown to be lead to significant worsening of performance (Alizadeh et al.) 2019). So instead the binary weights are optimized. It is common to not optimize the binary weights directly, but instead optimize a set of underlying real-valued weights Wa which can then be binarized in some fashion for inference. this paper we will adopt the convention of binarizing the underlying weights using the sign function (see Equation 2). We also use the sign function as the activation function when we use binary activations (see Equation where or are the real-valued pre-activations). We define the sign
We use real-valued throughout the paper be synonymous with "implemented with floating-poin precision"..
2

---

Published as a conference paper at ICLR 2021
function as:
1-1, < 1, 2
sign(1:)
Since the derivative of the sign function zero almost everywherd the gradients of the underlying weights We and through binary activations zero almost everywhere This makes gradient-based based optimization challenging. To overcome this issue, the straight-through estimator (STE) (Bengio et al. 2013) can be used. When computing the gradient of the loss L, the STE replaces the gradient of the sign function (or other discrete output functions) with an approximate surrogate. A straightforward and widely used surrogate gradient the identity function, which we use to calculate the gradients of the real-valued weights Wp (see Equation 3). It has been shown useful to cancel the gradients when their magnitude becomes too large (Courbariaux et al. 2015: Alizadeh et al. 2019). Therefore we use a clipped identity function for the gradients of the pre-activations (see Equation 6p. This avoids satu- rating a binary activation. Lastly, the loss value only depends on the sign of the real-valued weights. Therefore, the values of the weights are generally clipped to be in [-1, 1] after each gradient update (see Equation 4.. This restricts the magnitude of the weights and thus makes it easier to flip the sign.
Weights
Activations
Forward pass:
WB sign(wR) (WR)
(2)
0g sign(aR)
(5)
Oc OL
dC =
Oc
Backward pass:
(3)
(6)
OWR
awB
daR
dag
After update:
WR + max(-1,,in1,,,u wR))
(4)
2.2 DEEP GENERATIVE MODELS
Hierarchical VAEs. The variational autoencoder (Kingma & Welling 2014 Rezende et al. 2014)
is latent variable model for observed data x conditioned on unobserved latent variables z. It consists of generative model (x z) and an inference model q (z(x). The generative model can be decomposed into the prior on the latent variables po (z) and the likelihood of our data given the latent variables pe (x(z). The inference model is a variational approximation to the true posterior, since the true posterior is usually intractable in models of interest. Training is generally performed by maximization of the evidence lower bound (ELBO), lower bound on the log- -ogliiill of the data:
log po (x) 2 (2)x) [log po (x, 2) - log (z(x)))
To give more expressive model, the latent space can be structured into hierarchy of latent variables Z1:L. In the generative model each latent layer is conditioned on deeper latents pe (z, (. A common problem with hierarchical VAEs is that the deeper latents can struggle
learn. often
"collapsing" such that the layer posterior matches the prior: 2 (z, 21+11:, x) R Po (z, [+41: One
method to help prevent posterior collapse is to use skip connections between latent layers (Kingma) al. 2016 Maaløe et al., 2019), turning the layers into residual layers (He et al. 2016).
We focus on the ResNet VAE (RVAE) model (Kingma et al. 2016). this model, both the generative and inference model structure their layers as residual layers. The ResNet VAE uses bi-directional inference structure with both a bottom-up and top-down residual channel. This is a similar structure to the BIVA model (Maalee et al., 2019). which has demonstrated state-of-the-an results for latent variable model. We give a more detailed description of the model in Appendix C
Flow models. Flow models consist of parameterized invertible transformation, fo (x), and known density P& (z) usually taken to be a unit normal distribution. Given observed data x we obtain the objective for 9 by applying ofange-of-variablle to the log- ligeliiiihoo:::
dfa
log po px()) (x)
log Pa (fo + log
det
(8)
dx
For training to be possible, it required that computation of the Jacobian determinant det(dfo) dxx is tractable. We therefore aim to specify of flow model fo which is sufficiently flexible to model the
2 AAprtt from at 0, where it is non-differentiable.
We have assumed here that the inference model factored "top-down".
3

---

Published as a conference paper at ICLR 2021
data distribution well, whilst also being invertible and having tractable Jacobian determinant. One common approach to construct fo as a composition of many simpler functions: fg = f1 o o... fL. with each invertible and with tractable Jacobian. So the objective becomes:
df.
log pe (x) log (f +
log
det
df.-1
There are many approaches to construct the f, layers (Dinh et al. 2014 Rezende & Mohamed 2015] Dinh et al. 2017 Kingma & Dhariwal 2018 Ho et al. 2019a). In this work will focus on the Flow++ model (Ho et al.) 2019a), which has state-of-the-an results for flow models, In the +low++ model, the f; are coupling layers which partition the input into X1 and x2, then transform only X2:
(x1) X1, (x2)
[MixLogCDF<xg:t(xx))
exp(a(x1)) + b(x1)
(10)
Where MixLogCDF is the CDF for mixture of logistic distributions. This is an iteration on the affine coupling layer (Dinh et al. 2014 2017). Note that keeping part of the input fixed ensures that the layer is invertible. To ensure that all dimensions are transformed in the composition, adjacent coupling layers will keep different parts the input fixed, often using an alternating checkerboard stripe pattern to choose the fixed dimensions (Dinh et al., 2017). The majority of parameters in this flow model come from the functions t, a and b in the coupling layer, and in Flow++ these parameterized as stacks of convolutional residual layers. In this work we will focus on how binarize these functions whilst maintaining good modelling performance. We give a more detailed description of the full flow model in Appendix D
BINARIZING DEEP GENERATIVE MODELS
In this section first introduce technique effectively binarize weight normalized layers, which are used extensively in deep generative model architectures. Afterwards, we elaborate on which components of the models we can binarize without significantly hurting performance.
3.1 NORMALIZATION
It is important to apply some kind of normalization after binary layer. Binary weights are often large in magnitude relative to the usual real-valued weights, and can result in large outputs which can destabilize training. Previous binary neural network implementations have largely used batch normalization, which can be executed efficiently using a shift-based implementation (Courbariaux) al. 2016).
However, common in generative modelling to use weight normalization (Salimans Kingma 2016) instead of batch normalization. For example, it is used in the Flow++ (Ho et al.) (019a) and statc-of-the-ar hicrarchical VAE models (Kingma et al.] 2016 Maalóe et al., 2019). Weight normalization factors vector of weights w into vector of the same dimension Va and magnitude both which are learned. The weight vector is then expressed as:
9
WR
VB
|RRl
(11)
Where || denotes the Euclidean norm. This implies that the norm of Wi is
Now suppose we wish binarize the parameters of a weight normalized layer. We are only able to binarize VR. since binarizing the magnitude g and bias b could result in large outputs of the layer. However, g and do not add significant compute or memory requirements, as they are applied elementwise and much smaller than the binary weight vector.
Let VB = sign(Væ) be a binarized weight vector of dimension n. Since every element of Va is one of +1, we know that |v2|| Vn
We then have:
WR = VB
(12)
*|lva| VE
4

---

Published as a conference paper at ICLR 2021
We refer to this as binary weight normalization, BWN. Importantly, this is faster to compute than the usual weight normalization (Equation IT), since we do not have to calculate the norm of Vg. The binary weight normalization requires only (1) FLOPs to calculate the scaling for VB. whereas the regular weight normalization requires (n) FLOPs to calculate the scaling for Vg- For a model of millions of parameters, this be significant speed-up. Binary weight normalization also has more straightforward backward pass, since we do not need to take gradients of the 1/llv|| term.
Furthermore, convolutions F and other linear transformations can be implemented using cheap binary operations when using binary weights, Ws, as discussed in Section 2. However, after applying binary weight normalization, the weight vector is real-valued Wg. Fortunately, since convolution is a linear transformation, we can apply the normalization factor a g/ vn either before or after applying the convolution input x.
a) = F(xx
(13)
So if we wish to utilize fast binary operations for the binary convolution layer, we need to apply binary weight normalization after the convolution. This means that the weights are binary for the convolution operation itself. This couples the convolution operation and the weight normalization, and we refer to the overall layer as binary weight normalized convolution, or BWN convolution. Note that the above process applies equally well other linear transformations. We initialize BWN layers in the same manner as regular weight normalization, but give a more thorough description of alternatives in Appendix E
3.2 BINARIZING RESIDUAL LAYERS
We aim to binarize deep generative models, in which it is common to utilize residual layers extensively. Residual layers are functions with skip connections:
Bres (x) +xx)+x
(14)
Indeed, the models we target in this work, the ResNet VAE and Flow++ models, have the majority of their parameters within residual layers. Therefore they are natural candidates for binarization, since binarizing them would result in large decrease in the computational cost of the model. To binarize them we implement Txx) in Equation 14 using binary weights and possibly activations.
The motivation for using residual layers is that they be used to add more representative capability to a model without suffering from the degradation problem (He et al.) 2016). That is, residual layers can easily learn the identity function by driving the weights to zero. So, if sufficient care taken with initialization and optimization, adding residual layers to the model should not degrade performance, helping to precondition the problem.
Degradation of performance is of particular concern when using binary layers. Binary weights and activations are both less expressive than their real-valued counterparts, and more difficult to optimize. These disadvantages of binary layers are more pronounced for generative modelling than for classification. Generative models need to be very expressive, since we wish to model complex data such as images. Optimization can also be difficult, since the likelihood of a data point is highly sensitive to the distribution statistics output by the model, and can easily diverge. This provides an additional justification for binarizing only the residual layers of generative model. By restricting binarization to the residual layers, it decreases the chance that using binary layers harms performance.
Crucially, if we were to would not be able to learn the identity function, as the binary weights cannot be set to zero. This would remove the primary motivation to binary residual layers. In contrast, using binary weight normalized layer in the residual layer, the gain g and bias can be set to zero to achieve the identity function. As such, we binarize the ResNet VAE and Flow++ models by implementing the residual layers using BWN layers.
residual binary layer without weight normalization, then the layer
"This applies when the inputs are real-valued or binary, but the speed-ups are far greater for hinary inputs
5

---

Published as a conference paper at ICLR 2021
BWN Conv2D
BWN Conv2D
BWN Gate
BWN Gate
sign
ELU
sign
BWN Conv2D
BWN ConveD
BWN Conv2D
BWN Conv2D
ELU
sign
ELU
sign
(a) RVAE
(b) RVAE
(c) Flow++ (32-bit activations)
(d) Flow++ (1-bit activations)
(32-bit activations)
(1-bit activations)
The residual blocks used in the binarized ResNet VAE and Flow++ models, using both binary and floating-point activations. The BWN Gate layer is binary weight normalized x convolution followed by gated linear unit. We display the binary valued tensors with thick red arrows.
Figure 1:
DEEP GENERATIVE MODELS WITH BINARY WEIGHTS
We now describe the binarized versions of the ResNet VAE and Flow++ model, using the techniques and considerations from Section Note that, for both the ResNet VAE and Flow++ models, we still retain valid probabilistic model after binarizing the weights. In both cases, the neural networks are simply used to output distribution parameters, which define a normalized density for any set of parameter values.
ResNet VAE. As per Section 3.2 we wish to binarize the residual layers of the ResNet VAE. The residual layers constructed as convolutional residual blocks, consisting of two 3 x 3 convolutions and non linearities, with a skip connection. This is shown in Figure [b)- To binarize the block, change the convolutions to BWN convolutions, described in Section 3.1 We either real-valued activations or binary activations. Binary activations allow the network to be executed much faster, but are less expressive. We use the ELU function the real-valued activation, and the sign function as the binary activation.
Flow++. As with the ResNet VAE, in the Flow++ model the residual layers are structured as stacks of convolutional residual blocks. To binarize the residual blocks, we change both the 3 x 3 convolution and the gated convolution in the residual block be BWN convolutions. The residual block design is shown in Figure (d). We have the option of using real-valued or binary activations.
EXPERIMENTS
We run experiments with the ResNet VAE and the Flow++ model, to demonstrate the effect of binarizing the models. We train and evaluate on the CIFAR and ImageNet (32 32) datasets. For both models we use the Adam optimizer (Kingma & Ba 2015). which has been demonstrated to be effective training binary neural networks (Alizadeh et al.] 2019).
For the ResNet VAE, we decrease the number of latent variables per latent layer and increase the width of the residual channels, as compared to the original implementation. We found that increasing the ResNet blocks in the first latent layer slightly increased modelling performance. Furthermore, we chose not to model the posterior using IAF layers Kingma et al. (2016). since we want to keep the model class general possible.
For the Flow++ model, we decrease the number of components the mixture of logistics for each coupling layer and increase the width of the residual channels, compared to the original implementation. For simplicity, we also remove the attention mechanism from the model, since the ablations the authors performed showed that this had only small effect the model performance.
Note that we do not use any techniques to try and boost the test performance of our models, such as importance sampling or using weighted averages of the model parameters. These often used in

---

Published as a conference paper at ICLR 2021
generative modelling, but since we are trying establish the relative performance of models with various degrees of binarization, we believe that these techniques are irrelevant.
5.1 DENSITY MODELLING
We display results in Table 1 We can see that the models with binary weights and real-valued activations perform only slightly worse than those with real-valued weights, for both the ResNet VAE and the Flow++ models. For the models with binary weights, we observe better performance when using real-valued activations than with the binary activations. These results are as expected given that binary values are by definition less expressive than real values. All models with binary weights perform better than a baseline model with the residual layers set to the identity, indicating that the binary layers do learn. We display samples from the binarized models in Appendix[A]
Importantly, we see that the model size significantly smaller when using binary weights 94% smaller for the ResNet VAE and 90% smaller for the Flow++ model.
These results demonstrate the fundamental trade-off that using binary layers in generative models allows. By using binary weights the size of the model can be drastically decreased, but there is slight degradation in modelling performance. The model can then be made much faster by using binary activations well as weights, but this decreases performance further.
5.2 INCREASING THE RESIDUAL CHANNELS
Binary models are less expensive in terms of memory and compute. This raises the question of whether binary models could be made larger in parameter count than the model with real-valued weights, with the aim of trying to improve performance for fixed computational budget. We examine this by increasing the number of channels in the residual layers (from 256 to 336) of the ResNet VAE. This increases the number of binary weights by approximately 40 million, but leaves the number of We can see
real-valued weights roughly constan [6] The results are shown in Table and and Figure the increase the binary parameter count does have noticeable improvement in performance. The model size increases from 13 MB to 20 MB, which is still an order of magnitude smaller than the model with real-valued weights (255 MB). It is an open question as to how much performance could be improved by increasing the size of the binary layers even further. The barrier to this approach currently is training, since we need to maintain and optimize a set of real- valued weights during training. These get prohibitively large
increase the model size significantly.
5.3 ABLATIONS
We perform ablations to verify our hypothesis from Section 3.2 that should only binarize the residual layers of the generative models. We attempt to binarize all layers in the ResNet VAE using BWN layers, using both binary and real-valued activations. The results are shown in Figure ().). As expected, the loss values attained are significantly worse than when binarizing only the residual layers. We also perform an ablation comparing the performance of BWN against batch normalization in Appendix B demonstrating the advantages of using BWN.
DISCUSSION AND FUTURE WORK
We have demonstrated that is possible drastically reduce model size and compute requirements for the ResNet VAE and Flow++ models, whilst maintaining good modelling performance. We chose these models because of their demonstrated modelling power, but the methods we used to binarize them are readily applicable to other hierarchical VAES or flow models. We believe this is useful result for any real-world application of these generative models, such as learned lossless compression (Townsend et al. 2019 Kingma et al. 2019: Townsend al. 2020 Ho et al. 2019b Hoogeboom ct al. 2019). which could be made practical with these reduced memory and compute benefits.
A key technical challenge that needs to be overcome for binary neural networks as whole is the availability of implementations of the fast binary linear operations such as binary convolutions.
'There will be slight increase, since we use real-valued weights map and from the residual channels.
7

---

Published as a conference paper at ICLR 2021
Table 1: Results for binarized ResNet VAE and Flow++ model on CIFAR and ImageNet (32 x 32) test sets. Loss values are reported in bits per dimension. We give the percentage of the model parameters that binary and the overall size of the model parameters. The weights and activations refer to those within the residual layers of the model, which are the targets for binarization.
Precision
Modelling loss
Parameters
% Binary
Memory cost
Weights
Activations
CIFAR
ImageNet (32 32)
ResNet VAE
32-bit
32-bit
3,45
4.25
56M
0%
255 MB
1-bit
32-bit
3.60
4.47
56M
97.1%
13 MB
1-bit
1-bit
3.73
4.58
56M
97.1%
13 MB
increased width
1-bit
32-bit
3,56
96M
97.7%
20 MB 20 MB
1-bit
1-bit
3.68
96M
97.7%
residual
N.A.
N.A.
3.78
1.6M
0%
6 MB
Flow++
32-bit
32-hit
3.21
4.05
34M
0%
129 MB
1-bit
32-bit
3.29
4.18
34M
90.1%
14 MB
1-bit
1-bit
3.43
4.30
34M
90.1%
14 MB
residual
N.A.
N.A.
3.54
2.2M
0%
9 MB
no rasiual layer
no residual Ityers
I-bit weights I-bit activations
3.9
1-bit weights + 32-bit activations
1-bit weights
32-bit activations
seights
weights activations
dimendinn
3.9
3.6
3.8
3.5
3,7
3.6
3.3-
3.5
3.2
3.4
100
200
300
400
500
600
opoch
(a) ResNet VAE
(b) Flow++
4.2
bit bit activations
I-bi weights
1-bit actioations mocie)
4.1
1-bit weights
1-bit activations (increased width)
Lbb weights 32-bit activations [full model)
1 bit weights bit activations (residual layers)
4.0
1 b-bit weights
32-bit activations (increased width)
weights 32-hit activations (mesidual laynes)
bits dimension
bit weights br: activotions
bits dimenoion
3.9
8.8
6-
5-
4-
3.4
200
400
6000
800
1000
1200
1400
1500
25
50
75
100
125
150
175
200
epach
(c) ResNet VAE (increased channel width)
(d) ResNet VAE (ablations)
Figure 2: Test loss values during training of the ResNet VAE and Flow++ models the CIFAR dataset. Subfigures (a) and (b): models with binary weights and either binary or real-valued activations. Compared to the model with real-valued weights and activations, and a baseline with the residual layers set to the identity. Subfigures (c) and (d): the effect of increasing the width of the residual channels, and ablations.
oooff-of-concept implementations of these binary kernels have been developed (Courbariaux et al.) 2016; Rastegari et al. 2016; Pedersoli et al.) 2018). However, there no implementation that is readily usable substitute for the existing kemels in frameworks such as PyTorc and Tensorflow.
8

---

Published as a conference paper at ICLR 2021
7 CONCLUSION
We have shown that it is possible to implement state-of-the-art deep generative models using binary neural networks. We proposed using fast binary weight normalization procedure, and shown that is necessary to binarize only the residual layers of the model to maintain modelling performance. We demonstrated this by binarizing two state-of-the-arr models, the ResNet VAE and the Flow++ model, reducing the computational cost massively. We hope this insight into the possible trade-off between modelling performance and computational cost will stimulate further research into the efficiency of deep generative models.


Reviews:
[{"review_id": "ngoSEUzib_a", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces the first way (to the best of authors knowledge) of building generative models with binary weights. Also the case with binary activations is considered. The authors consider two SOTA generative models (flow++ and RVAE) and develop technique to binarize all weights (and possibly activcations) in residual layers. They show that residual layers can be binarized with relatively small drop in performance and further binarization of remaining blocks in computational graph leads to significant degradation. Binary modification of weight normalization is suggested although no ablation is study is performed so it is unclear how crucial is BWN for robust learning. The training process itself is pretty standard way of training binary DNNs - they use STE + truncation of real-valued weights counter-parts.\n\nI have several questions on the proposed methodology:\n1. To compute density in flow++ model you need to be able to differentiate Jacobians. How would you compute them given the inputs are binary? Have you used STE for this purpose? If so then how can we be sure that what we get is correct (normalized) density function? I think this issue should be addressed in the paper since nomalizing flows by design assume continuous variables.\n2. Please define what is $n$ in you paper. Is it the total number of all weights in you network/layer or is it a number of weights in single convolution filter?\n3. You provide performance of binarized generative models and show that they require a lot less memory than their real-valued analogues. It would great to see what performance can be achieved by smaller real-valued generative model that requires approximately same amount of of memory as binarized network. Then we could understand what is better - to use large binary networks or small real-valued ones for generative modeling.\n4. My major concern is the visual quality of the images obtained by binary models. To be honest after I saw them I decided to downgrade my mark from 6 to 5. It seems that with such quality the model is quite useless. Do I understand right that there is a clear visual difference between objects generated by real-valued and binarized networks of similar size?  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "CMpHbQXDoMX", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a method to binarize weights and activations of variational autoencoders and flow-based networks.  This is an important issue as these methods are valuable to solving unsupervised problems, but are rapidly growing in size, necessitating large and expensive computing systems. And, the literature of low and binary precision hasn’t considered these use-cases to date.\nThe choice of ResNet VAE and Flow++ are good representative candidates for methods in use widely today.  However, the largest models we see in the world today are knowledge representation and language models (transformers)). Showing that binarization will work for these models would be a valuable next step.\n\nThe authors call this a “framework” to go to binary weights and/or activations, but this is really a set of methods shown to work on a particular NN. Further work is needed to describe a more generalized set of methods. The authors do a good job of motivating their method for the VAE and Flow++ model, but I don’t see an extraction of a fundamental principle that might be broadly applicable to most neural networks.  A major contribution of the paper is Binary Weight Normalization (BWN) and the binarizing method for residual layers.  There is something deeper there.\n\nI like this work as it is an increasingly important consideration to data science teams in building solutions that can work within real-world constraints.  However, binarization is still seen as a method to try and reduce the computational load of something that is already being done.  This makes it difficult to build systems and computing architectures that commit to binary representations. I see this work as a steppingstone to something more general, which can be the basis of building better and cheaper computing substrates. A stronger paper would have uncovered a universal concept.\n\nSection 5.1: It doesn’t appear there is a really principled way to decide which parameters to binarize.  There is some discussion of residual layers as being the least susceptible to performance degradation, but it would be a stronger paper if some of this was explored a bit more through experimentation. Again, extracting a principle here would be useful to the field.\n\nSection 5.2: This vector of improvement is an entire research topic on its own. I think this might be the most interesting direction highlighted by this paper; how does precision/binarization interplay with hyperparameters to reduce the absolute computing requirements, which increasing algorithmic performance?\n\nSection 5.3 Ablations, where is figure 3?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "4wjkGi5AfCj", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The contributions of this paper are the following:\n - it extends previous work on binary neural networks to the case of deep generative models which perform density estimation (VAEs and Flows)\n - the results show that the proposed approach works well, with the expected trade-off but closely matching the much larger real-valued networks\n - in order to do so, it introduces a novel technique for binarizing weight-normalized layers, which are used in these generative models\n - the results show the advantage of this technique, and they illustrate it is particularly important for ResNet layers.\n\nThe paper is very clear (except for one element noted below). Originality is clear but not very high, as this contribution may be seen as a low-hanging fruit, albeit a useful and well-executed one.  As these kinds of architectures are becoming more and more used in various settings, the techniques used here (e.g. using ResNet layers) may be applicable more widely, increasing the significance of this work.\n\nMinor fixes suggested:\n\n* The semantics of equations 3 and 6 is not clear. I imagine the objective is to explain how the gradient on the binary weights and activations is converted into a gradient on the real-valued ones, but this should be explicited.\n\n* In page 1 (bottom line) and 4 (1st line after eqn 9), the authors cite Rezende & Mohamed 2015, Dinh et al 2017 , but they should include first in this list the Dinh et al 2014 paper which actually introduced the notion found in Flow models, albeit under a different name (NICE).\n\n* Appendix A: the authors should also show the images generated with full precision, so we can compare visually.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2lBrBGxZ187", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to binarize weights and activations of generative VAE and Flow++ models. As Weight Normalization is commonly used in these models, the authors notice that Euclidean norm of binary [-1;1] vector is a square root of it’s length, such that Weight Normalization can be reduced to affine scaling. They propose to call this scaling Binary Weight Normalization, and evaluate it on CIFAR and ImageNet datasets.\n\nOverall, motivation for binarizing generative models is unclear. To the best of my knowledge, training such generative models is an active research topic, and, unlike in image classifiers, object detectors, language and translation models, are not applied in practice in real systems where execution time and memory footprint matters. Researchers who train such generative would not apply binarization as:\n- training such models is already difficult and optimization is often unstable, so adding an extra variable might make it even more difficult;\n- to keep baselines simple;\n- final execution time does not matter.\nIt is also very likely that, in case such generative models find applied use-cases where execution time and memory footprint would matter, they would have undergone several research iterations and updates such that new binarization techniques would need to be developed.\n\nAs affine scaling is no longer a weight normalization, it is probably incorrect to call it that. An experiment ablating an architecture with/without the proposed binary weight normalization is missing. Also, as generative models are often evaluated qualitatively, the paper would benefit from including several samples from binarized models, to show that quality does not degrade.\n\nLiterature review also lacks several more recent binarization techniques such as [1] and [2] (more can be found there). It is not clear if one were to pick a simple (or more recent) binarization technique they would encounter difficulties.\n\nOverall, due to unclear motivation, introduction of binary weight normalization which is not weight normalization, and incomplete literature review I propose rejection.\n\n[1] Mark D. McDonnell, Training wide residual networks for deployment using a single bit for each weight, at ICLR 2018. \n[2] Gu et al. Projection Convolutional Neural Networks for 1-bit CNNs via Discrete Back Propagation, at AAAI 2019.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper develops an approach to training generative models with binary weights. The reviewers are split. Two reviewers regard binary networks and the general theme of reducing the computational cost of training as important, and the presented work as a solid contribution. Two reviewers raise concerns about the motivation and the quality of the results. The authors' responses somewhat alleviated the quality concerns of R3, but not the concerns of R4. Overall, the reviewers lean on the positive side. There is disagreement on the importance of the problem, but there is clearly a non-trivial subset of the community that welcomes research in this direction. The AC supports acceptance.
Decision: Accept (Poster)