QGAN: Quantize Generative Adversarial Networks to Extreme low-bits
Authors: Peiqi Wang
Yu Ji
Xinfeng Xie
Yongqiang Lyu
Dongsheng Wang
Yuan Xie
Venue:  (2020)
Keywords: generative adversarial networks
quantization
extreme low bits

Abstract: The intensive computation and memory requirements of generative adversarial neural networks (GANs) hinder its real-world deployment on edge devices such as smartphones. Despite the success in model reduction of convolutional neural networks (CNNs), neural network quantization methods have not yet been studied on GANs, which are mainly faced with the issues of both the effectiveness of quantization algorithms and the instability of training GAN models. In this paper, we start with an extensive study on applying existing successful CNN quantization methods to quantize GAN models to extreme low bits. Our observation reveals that none of them generates samples with reasonable quality because of the underrepresentation of quantized weights in models, and the generator and discriminator networks show different sensitivities upon the quantization precision. Motivated by these observations, we develop a novel quantization method for GANs based on EM algorithms, named as QGAN. We also propose a multi-precision algorithm to help find an appropriate quantization precision of GANs given image qualities requirements. Experiments on CIFAR-10 and CelebA show that QGAN can quantize weights in GANs to even 1-bit or 2-bit representations with results of quality comparable to original models.

Under review as a conference paper at ICLR 2020
QGAN: QUANTIZE GENERATIVE ADVERSARIAL NET- WORKS TO EXTREME LOW-BITS
Anonymous authors Paper under double-blind review
ABSTRACT
The intensive computation and memory requirements of generative adversarial neural networks (GANs) hinder its real-world deployment on edge devices such as smartphones. Despite the success in model reduction of convolutional neural networks (CNNs), neural network quantization methods have not yet been studied GANs, which are mainly faced with the issucs of both the effectiveness of quantization algorithms and the instability of training GAN models. In this paper, start with extensive study on applying existing successful CNN quantization methods to quantize GAN models to extreme low bits. Our observation reveals that none of them generates samples with reasonable quality because of the underrep- resentation of quantized weights in models, and the generator and discriminator networks show different sensitivities upon the quantization precision. Motivated by these observations, we develop novel quantization method for GANs based on EM algorithms, named as QGAN, We also propose a multi- cenlti-preciionn algorithm to help find an appropriate quantization precision of GANs given image qualities requirements, Experiments on CIFAR- 10 and CelebA show that QGAN can quan- tize weights in GANs to even 1-bit or 2-bit representations with results of quality comparable to original models.
INTRODUCTION
Generative adversarial networks (GANs) have obtained impressive success in wide range of applications, such as super-resolution image generation, image-to-image translation, and so on (Bulat al. 2018: Ao al. 2018). Despite their success generating high-quality samples, these models are hard to be deployed into real-world applications demands for computation and storage capacity. For example, the BigGAN model (Brock et al. 2018), developed by Google, contains up to 0.2 TOPs in the inference phase and its model size over 1.3 GB. becomes more urgent as the growth of privacy and security concerns about running the inference cloud platforms.
edge devices because of their huge
This challenge exists in the deployment of various neural network models besides GANs. State-of-the art techniques compress model scales include pruning, quantization, low-rank approximate (Han) et al. 2015) Courbariaux et al. [016 [ainath et al. 2013). Among these techniques, quantization is the most easy-to-use and scalable method, which uses fewer bits for data representations than the 32-bit precision. Quantization has the following three advantages over other techniques. First, the compression rate significant. For example, the model size of a 2-bit quantized model is reduced by 16x. Second, the quantization technique does not change the neural network architectures. Thus it is orthogonal to the study of algorithms for neural network architecture exploration. Finally, is easy to be deployed into off-the-shelf devices with little hardware co-design to obtain significant performance and energy benefits. The use of quantization methods requires little knowledge from algorithm researchers to hardware.
Although quantizing neural network models has achieved impressive success on convolutional neural networks (CNNs) (Zhou et al., 2016 Rastegari et al. 2016 Zhu al., 2016). there is still no successful attempt to quantize GAN models, especially in extreme low bits domain. In this paper, we first study the effectiveness of typical quantization methods on GAN models. Despite the success of these methods on CNNs, we observe that they are not directly applicable to quantize GAN models to extreme low bits because of their underrepresentation of original values. Besides, we observe that

---

Under review as a conference paper at ICLR 2020
different components from GAN model have different sensitivities to the quantization precision. Based on our observations, we develop QGAN, novel quantization method based on Expectation Maximization (EM) algorithm, and novel pretti-peecison quantization algorithm. Finally, our experiments show that the proposed QGAN can quantize weights in GAN models into 1-bit or 2-bit representations while generating samples of comparable quality, and our multi-precision method helps to search a better quantization configuration according to a given demand.
In summary, work has following contributions:
We conduct a comprehensive study on existing quantization methods to understand their effec- tiveness to GAN models and the sensitivity of GAN models to the quantization precision. The observations of these empirical studies further guide the development of our QGAN method.
We obtain several insightful observations through the sensitivity study. First, the discriminator is more sensitive than the generator to the number of quantized bits. Second, a converged quantized discriminator is conductive to the convergence of the entire quantized GAN model. Third, quantizing both the discriminator and generator is more stable than only quantizing generator networks.
Based on the observation of the underrepresentation problem in existing quantization methods, we propose QGAN, a novel quantization method for GAN models based on EM algorithm to overcome this problem in the extreme low-bit situation. Our experiments demonstrate that GAN models with 2-bit or even 1-bit weights quantized by QGAN can generate samples of comparable quality.
Based on observations from the sensitivity study, we develop a multi-precision quantization algorithm. This algorithm provides the quantization precision low possible to satisfy the quality requirement of generated samples.
BACKGROUND
2.1
GENERATIVE ADVERSARIAL NETWORKS
Generative Adversarial Network (GAN) is composed of two components, the generator and the discriminator. The generator network, usually denoted as G, is trained to generate samples similar distribution of real data while the discriminator network, usually denoted as D, is trained to discriminate whether the input is generated by G or from real data. The generator takes a sampled noise where (O, 1) or 1),, the input each time generate data sample. Both samples from real data and generated from are taken as inputs, denoted as I, to the discriminator, and the discriminator estimates the probability, D(x), that the input is from real data. The training process of a GAN model can be formulated as a min-max game between the generator and the discriminator. The objective function of this min-max game be formulated as:
min max V(D, G) (x-ppate [log (xx) + Ex [log(1 (log( G(z)] D The generator aims to minimize this objective function while the discriminator aims to maximize Both of them converge Nash equilibrium point where neither of them has any better action further improve objects.
To improve the quality of generated samples, prior studies focus better neural network architec- tures (Radford et al. 20155 Mirza & Osindero 2014 Karras et al. 2017). Some studies propose new objective functions for better convergence the training process, such adding new constraints (Ar) jovsky et al., 2017; Gulrajani et al., 2017) and using smoother non-vanishing or non-exploding gradients (Mao et al. 2017 Zhao al.] 2016). Our work focuses on using fewer bits for data repre- sentations to compress models for more efficient deployment on edge devices, further expanding the application scope of large advanced models.
2.2 QUANTIZATION
Quantization is promising technique to reduce neural network model size and simplify arithmetic operations by reducing the number of bits in the data representation. A quantization method is a mapping method for a variable z from continuous space C discrete space D, which can be formulated as
((x) 5 (round
2

---

Under review as a conference paper at ICLR 2020
where denotes a full-precision value from C, Q(x) is the quantized discrete value, and f(-) is a scaling function. Different quantization methods use different f(-) functions. Among various quantization methods, Minmax (Minmax-Q) (Jacob et al. 2018). Logarithmic minmax (Log- Q) (Miyashita et at.) (016), and Tanh (Tanh-Q) (Hubara et al. 2016) are three representative quantization methods and their scaling functions are shown in Equation [ where X is the tensor that variable z comes from, A is the number of bits used by quantized values, and is an extreme small constant to avoid the appearance of -00.
I min(X)
fm(x)= mar(X) (X) min(X) x(2 1)
(3)
fi(a)) = Jm (1og((1)) + ())
(4)
(x) tanh(x) + x (2* 1)
(5)
The accuracy loss in the process of quantization is the main challenge when quantizing NN models. To train a quantized model, the floating- -ainnttpoontttto parameters X are quantized to low-bit representation by Q(-) at first, and then used in the forward process of the model. In the backward propagation, the non-differentiable components round(.) (Bengio et al., (003)), which can be considered as an operator with the arbitrary forward operation and unit derivative. Therefore, the derivative of Q respect to X is simplified to 1, After training, parameters are quantized to low bits and the quantized model is saved for further inference.
Qxx²) adopt the straight-througs estimator method
EFFECTIVENESS AND SENSITIVITY STUDY
In this section, we provide comprehensive studies on the effectiveness of typical CNN quantization methods on GANs and the sensitivity of different components in GANs to the quantization precision. Observations from these studies motivate us for better quantization method on GAN models.
3.1 EFFECTIVENESS STUDY
We take the deep convolutional generative adversarial network (DCGAN) (Radford et al. 2015) as an example GAN model to investigate the effectiveness of the aforementioned typical CNN quantization methods (Section (2.2) GANs. All evaluations in this section adopt the DCGAN model on CIFAR-10 dataset (Krizhevsky & Hinton 2009). To fit the 32x32 images in the dataset, we reduce the final convolutional layer in the original DCGAN generator and the first convolutional layer in the discriminator, keeping all other hyperparameters consistent with the prototype implemented based on pytorch The quality of generated samples is measured in Inception Score (IS) (the higher the better) in Salimans et al. (2016) and Frehet Inception Distance (FID) (the lower the better) in Heusel et al. 2017). We apply the pretrained Inception-v3 network for the computation of IS and FID, and scores are calculated using 5000 generated images.
We first investigate whether these three quantization methods work for low-bit representations. We apply them to DCGAN by quantizing all weights in both the discriminator and generator to 2-bit data representation. The results are demonstrated in Table [ and the baseline is the original model with full-precision (32-bit). The significant quality gap between the full-precision DCGAN and quantized DCGAN indicates that these methods can not directly applied to quantizing GAN models.
In order to understand the reason for such failure, we visualize the distributions of the weights from both discriminator and generator in Figure We take the weights in the first convolutional layer of the discriminator and the last convolutional layer of the generator as an example. The distributions of original weights in full-precision are shown in Figure T(a) and Figure [(e). and the rest of sub-figures show the distributions of quantized weights in 2-bit using different quantization methods. We observe from Figure | that the underrepresentation of original values in quantized states leads to the quality gap of these methods in quantizing DCGAN. Specifically, Figure T shows
The baseline we used here the pytorch version t treee/master/dogan
The pretrained inception model comes from https: htttss//ddonl1e http///wwwoddddooco/m////m http::///dwoooddytttrohhhh werttorch.orgmmdee.. org/modell: ioppptionvvv___eee 1a9a5a 14. The split parameter in IS computation equals to 10.
3

---

Under review as a conference paper at ICLR 2020
3.
28
Frequency
5.
2o 50 2-
03 64
6. 6:60
02 63
61 21
6166 61 03.03. 03
0. 03-42-22-61 02-0000000000000 61
We gits
Weights
Weights
Weights
(a) weights in D
(b) 2-bit minmax- Qn
(c) 2-bit log-Qlq--QQ in
(d) 2-bit tanh- in D
Frequancy
1.0
255
30
25
220
15
-15 -65 05 1. 15
-5 -I0 -65
90
L0
255
-15 -50 -65 05
16 155
0.0
-15 -50 -65
05 05 16 15
weights
weights
weights
weights
(e) weights G
(1) 2-bit minmax- in G
(g) 2-bit log- Qag-Quie in G
(h) 2-bit tanh- in G
Figure 1: The distribution of weights the first convolutional layer in discriminator (D) and the last convolutional layer in generator (G). (a) and (e) show original weight distributions in full precision, (b) and (f) use the Minmax-Q quantization, (c) and (g) use the Log-Q quantization, (d) and (h) use the Tanh-Q Q quantization. The model here DCGAN trained on CIFAR-10 dataset, and all quantization schemes quantize full precision data to 2 bits.
Table 1: The Inception Score (IS, higher is better) and Frehet Inception Distance (FID, lower better) of 2-bit DCGAN CIFAR-I0 dataset using different quantization methods.
Methods
Baseline
Minmax-Q
Log-Q
Tanh-Q Q
IS FID
5.30 28.4 2.65 132.4 1.17 421.2
1.28 437.8
that none of the aforementioned quantization methods use the full capacity of 2-bit precision. All distributions quantized values manifest two peaks using only two discrete states. However, a 2-bit precision number has the capacity to represent 4 different discrete states. In minmax-Q method, two states represent the extremums, where few data distribute around. log-Q method, two states are used for the +h. In tan-Q method, two states are projected to too0 by arclanh()). which cannot be shown in figures. Such an under-utilization of 4 states 2-bit representation makes the distribution of quantized values fail approach the distribution of original values.
3.2 SENSITIVITY STUDY
Despite the failure of three typical quantization methods on quantizing GAN models into low-bit representations, we investigate the sensitivity of generator and discriminator the number of bits used data representations to understand the minimum number of bits prior methods can achieve. We take the log-Q method
study.
Figure aaa) shows the training curve of only quantizing weights in the discriminator (D) from scratch, while the generator (G) in full- full-precision. Figure 2(b) shows the training curve of quantizing weights in both and G. Figure 2 shows the training curve of only quantizing weights in G. From training curves, we can observe three different training states: convergent, unstable, and failed. The difference between states, unstable and failed, is that the Inception Score (IS) of an unstable state oscillates when the number of epochs increases while the IS of failed state does not change from the very beginning. The Frchet Inception Distance (FID) metric shows the same trend. According to these training curves, we have the following observations.
Inception Score w a in
Inception Score o
Inception Score
26R
36R
w
7b2.
8bR
100
20
40
60
80
100
100
Epoch
Epoch
Epoch
(a) Quantized D only
(b) Quantized both D and G
(c) Quantized G only
Figure 2: The training curves DCGAN using logarithmic minmax quantization different bits.
4

---

Under review as a conference paper at ICLR 2020
First, D is more sensitive than G to the namber of bits used in data representations. As shown in Figure 2(a) quantizing only D to different numbers of bits will result in either a convergent or failed state. Besides, quantizing only G' in Figure 2 will result in either convergent or unstable state. Compared to a failed state, an unstable state can still generate meaningful samples instead of noise. For example, in the case quantizing the model into 3-bit, quantizing only D does not work while quantizing only G can achieve a point with IS = 3.42 during thrashing.
Second, converged D is conducive the whole quantized GAN model to converge. As shown in Figure 2(a) and Figure 2 both of them have only two states in cases with different numbers of bits. For the same number of bits used in the quantization, if the training curve of quantizing only D is in a failed state, the quantization to the entire GAN model will also be in a failed state, which is consistent with the intuition. Additionally, comparing Figure 2(b) and 2(c) the failure of convergence in only quantizing G does not necessarily lead to the quantization fai failure of the whole quantized GAN model.
Third, quantizing both D and G is more stable than only quantizing G. Take the case of 4-bit quantization as example, which is shown in the green lines of Figu Figure [ig)re[[cc] only quantizing G could lead to an unstable state while quantizing both D and G makes a convergent state. Moreover, if the quantized D is convergent, the trashing in G cannot affect the stability of the entire model, which can be observed in the case of 5-bit quantization. One of the empirical GAN training guidelines is balancing the capability of discriminators and generators, which is consistent with our observation.
In summary, these observations indicate the different sensitivities of D and G in the quantization process for GAN models, which further motivates us to develop a mulli-ppeecisiee quantization method find the number of bits used in the quantization as low as possible meet the quality requirement.
4 QGAN
In order to address the data underrepressentation problem identified by our case study, we introduce our novel quantization method based on the Dopectation-Maximizaae weights in GAN models to even 1-bit or 2-bit with little quality loss. Besides, to leverage observations from our case study, we propose multi- multi-precision quantization strategy to provide the precision configuration as low as possible to satisfy the quality requirement for generated samples.
algorithm, which can quantize
4.1 QUANTIZATION BASED ON EM ALGORITHM
To overcome data underrepresentation problem, it is important to fully utilize the quantized states and narrow the gap between the distribution of quantized values and original values. Therefore, formulate the quantization method as an optimization problem with the L2-norm loss function as the objective function to measure the difference between original weights and quantized weights. To simplify the problem, we select the linear function fem (x) = (x - B) a as our scaling function in the quantization method. The proper choice of scaling parameters a and / crucial to the final quality of quantized models. We propose an EM-based algorithm to find the optimal scaling parameters.
Given full-precision input weights W- [w;]. < < N, the quantization method quantizes them to bibit intermediate values 2; E [0. 2n - 1] at first, and then rescales them back to get the quantized weights Wa_ = {f "(zz; (,,;,,3), 3) Then, the optimization problem can be formulated as
N 2(m: (=;0,
1 N
0,3 3 = arg min a
(6)
Here we turn the optimization problem into a maximum likelihood problem. Considering a generative
model p(w, 2//a, () which generates the parameter candidates, can obtain p(w, 2, 2//a, () oc exp (w0 (2;; ox, }) when 2; = arg min=(w - -=; at 8) and p(w,, 24 a0, (3) equals to 0 otherwise. The likelihood of this model is T.(a:, 83 W, Z) p(W, Za, B). Therefore, solving the optimization problem shown in Equation (6) is equivalent to maximizing this likelihood. Finding the optimal a and 8 to maximize the likelihood can be solved by the EM algorithm, which iteratively applies two steps, Expectation and Maximization.
Expectation step: Define E(a, 3 (1)) ,())) as the expected value of the log likelihood function of and 8, with respect to the current conditional distribution of Z given W and the current estimates
5

---

Under review as a conference paper at ICLR 2020
of the parameters (4) and g(t) at the time step t. This expected value can be derived
) a an po W. = -w -w
0)) (7)
where C' is a constant value. In the current time step I, the parameter (1 and 300 are in fixed value, thus we can obtain the current best intermediate discrete values 2; given wi by
8(0
arg min (w, (z; ",, ,300) round( (1)
(8)
Maximization step: The maximization step is going find the parameters that maximize the expected value E for the next time step / + 1.
(1+1),, glet1) arg max E(a, () ,())) arg min o,
Σ
(w a2; - =)2
(9)
a,
Therefore, the optimal parameters of time step + 1 are
a (++1) [[wz)= - (()) g(++1) = B(w) - (++1)(2))
(10)
[==²] (B(=))2
After applying the Expectation and Maximization steps iteratively, parameters (t)) and z will converge values which are optimal values found by the EM algorithm. According to converged values, a and 8", our quantization method uses the scaling function fem(z) to quantize weights in original GAN models from full precision any number of bits.
4.2 MULTI-PRECISION QUANTIZATION
Our sensitivity study in Section 3.2 motivates us to develop multi-precision method to figure out a quantization configuration as low-bit as possible for GANs to satisfy given requirement for generated image quality. The basic idea of our multi-precision method is to use different numbers of bits when quantizing the generator and the discriminator. Our observations in Section 3.2 indiccae that the discriminator is more sensitive than the generator to the number of bits. Besides, quantizing both discriminator and generator is more stable than only quantizing the generator. Therefore, our multi-precision method greedily first quantizes the discriminator, and then quantizes the generator. Overall, our multi-prest precision method has two steps. In the first step, our method finds the lowest number of bits needed by the weights in discriminator to meet the given quality requirement while the weights of the generator keep in fuel-precision. In the second step, our method uses the quantized discriminator obtained from the first step to figure out the lowest number of bits needed by the generator to meet the given requirement. The effectiveness of our multi qulti-precision quantization method will be demonstrated in Section 5.2 where we apply this method to various GAN models.
EXPERIMENTS
In this section, evaluate the effectiveness of our quantization method, QGAN, three typical GAN models: DCGAN (Radford et al.] 2015), WGAN-GP (Gulrajani et al. 2017). and LSGAN (Mao) let al. 2017). We use the datasets of CIFAR-10 10 (Krizhevsky & Hinton 2009) with 32x32 colorful images, CelebA (Liu et al. 2015) with 64 x 64 and 128x 128 face images here. The metries are Inception Score (IS) (Salimans al. 2016) and Frchet Inception Distance (FID) (Heusel et 2017). which are the same as the case study of Section 3.1 Generally, higher IS or lower FID indicates a better quality of generated images. We implement -uee-precision baseline models in pytorch, and the configuration of hyper-pane papper-prramterss such as the learning rate, are the same as the configuration shown in the original papers of evaluated GAN models. Our evaluation consists of two parts. First, we demonstrate that our EM-based quantization method used in QGAN is superior to prior quantization methods in Section Then, we demonstrate the effectiveness of our multi-precision quantization process in Section 5.2
5.1 QUANTIZATION BASED ON EM ALGORITHM
To demonstrate that our EM-based quantization method in QGAN is superior to other prior quantiza- tion methods, we evaluate all of these methods for the DCGAN on CIFAR- 10 dataset. Specifically,

---

Under review as a conference paper at ICLR 2020
Frequency 0x10
2.5
3.5
2.0
1.5
2.0
Frequet
1.0
0.5
0.5
0.0
-..4-0.3-..2-0100
0.0
-1.5
1.5 -0.5
0.5
weights
weights
(a) 2-bit QGAN in D
(b) 2-bit QGAN in G
Figure 3: The distribution of weights in quantized DCGAN using 2-bit QGAN.
Table The IS (higher better) and FID (lower better) of DCGAN using different quantization methods on CIFAR-10 (with baseline in IS 5.30 FID 28.41)
1-bit
2-bit
3-bit
4-bit
IS / FID
IS / FID
IS / FID
IS FID
Minmax-Q
1.16 / 407.9
2.65 / 132.4
4.35 / 65.1
4.74/ 40.3
Log-Q
N/A
1.17 4219
1.16 440.3
4.15 / 60.6
Tanh-Q
N/A
1.28 437.88
1.20 466.7
1.13 460.2
QGAN
3.32 / 96.7
4.15 54.3
4.46/514
4.60 / 39.6
we compared QGAN with prior methods, Minmax- Log-Q. Tanh-Q. We use all of these methods to quantize pretrained full-precision DCGAN models and fintune CIFAR-10 1- from 1-bit to 4-bit. set the inside iterations EM algorithm to 32 in 4-bit case while to 16 in other cases. The quantization process needs much smaller number epoches compared to training from scratch, and the running time not a bottleneck. To simplify comparisons, we quantize weights in both discriminator and generator into the cases of log-Q Q and tanh-Q are because they degenerate to the and +o00 respectively and cannot work all.
number of bits. Results shown in Table The lost points in 1-bit
Results in Table 2 show that QGAN gets the best or comparable results in all cases. In the 4-bit case, QGAN performs as good as Minmax-Q with little bit lower IS but better FID. Measuring the image quality is well-known difficult task, and these popular metrics are positively related instead of strictly monotonous to image quality. We inspect the distribution of quantized states in QGAN, quantization based on the EM algorithm can overcome the problem of data underrepresentation, thus resulting better fit of quantized states to the distribution of original weights. Besides, these results also show that QGAN can still work in the case using extreme low-bit data representations, specifically 1-bit where GAN models become binary neural networks. Although there is still quality gap between the 1-bit model quantized by QGAN and the baseline full full-precision model, all other quantization methods either fail or generate noise in this extreme case.
which is shown in Figure 3 Compared to Figure [
5.2 MULTI-PRECISION QUANTIZATION
To demonstrate the effectiveness of our multi-precision method in extreme low bit cases, we apply all mentioned quantization methods DCGAN on CIFAR- 10, and the results are shown in Figure
The trends of the IS are good reflection of our three observations in Section 3.2 In Figure ((a) the scores all methods except QGAN exist a jumping-point, which means quantized discriminator changing from failed state to convergence. Because QGAN works well in extreme low-bit quantized situations, even in the 1-bit case, there is no such phenomenon. The inception scores shown in Figure 4(c) vary within small range due to the less sensitivity the number of bits in the generator.
Inception Scatee
Incesttion Score
Log-0
Incepation
Minmax-Q
OGAN
Bits
Bits
Bits
(a) Quantized only
(b) Quantized both and
(c) Quantized G only
Figure 4: The IS of quantized DCGANs in different bits using different methods on CIFAR-10.

---

Under review as a conference paper at ICLR 2020
Table 3: The IS and FID of different GAN models using multi-precision QGAN
Model
Dataset
D-bit
G-bit
IS
FID
IS-32bits
FID-32bits
DCGAN
CIFAR-10 CIFAR-10
2
4.33
56.1
5.30
28.41
WGAN-GP
3
3.41
74.1
4.20
46.97
LSGAN
CIFAR-10
3
3
3.55
85.4
4.91
38.27
DCGAN
CelebA-64x64
3
2.68
56.6
2.67
12.85
DCGAN
CelebA-128> 128
3
2.05
52.2
2.19
23.5
(a) DCGAN baseline
(b) LSGAN baseline
(c) WGAN-GP baseline
(d) DCGAN baseline
(g) ID3G WGAN-GP Figure 5: The generated samples various GAN models on CIFAR-10 10 dataset and DCGAN on CelebA dataset using QGAN. The kDjG denotes h-bit D and j-bit G.
(e) ID2G DCGAN
(I) 3D3G LSGAN
(h) ID3G DCGAN
Furthermore, the tendency in Figure and and 4(b) in Section 3.2 The FID metric shows the same trend this situation.
is very similar, which reflects the second observation
Finally, demonstrate the overall effectiveness of QGAN, we apply it to three GAN models on two datasets and results are shown in Table Due to the difficulty of quantitative image quality evaluation, we manually inspect the generated images when IS FID is significantly worse than baseline models. We present images generated by both quantized models and their baseline models for the purpose comparison in Figure [5] As shown in Figure 5 the marginal difference between images generated by quantized models and baseline models can hardly be distinguished by human eyes. As demonstrated Table 3 and Figure our QGAN method can quantize GAN models to extreme low precision (less than 4-bit) successfully, and the quality of images generated by quantized GANs on a par with the image quality of full-precision models.
CONCLUSION
In this paper, we study the problem of quantizing generative adversarial networks (GANs). We first conduct extensive study on the effectiveness typical quantization methods which are widely used in CNNs. Our observation reveals that the underrepresentation of original values in quantized states leads to the failure of these methods in quantizing GAN. The observation motivates us to propose QGAN, which operates with a linear scaling function based on EM algorithm and achieves high utliiization quantized states. Besides, we observe from the sensitivity study that the discriminator more sensitive than the generator to the number of quantized bits. To leverage this observation, introduce a multi- precision quantization approach to find the lowest number of bits for quantizing GAN models to satisfy the quality requirement for generated samples. Our experiments various GANs and different datasets show that QGAN can generate samples in a comparable quality in cases using even only 1-bit or 2-bit.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Dongyang Ao, Corneliu Dumitru, Gottfried Schwarz, and Mihai Datcu. Dialectical gan for sar image translation: from sentinel-1 to terrasar-x. Remote Sensing, 10(10))1597, 2018.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. arXhv:1701.07875. 2017.
Wasserstein gan. arXiv preprint
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint 3r8.v:/30..3433 2013.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: /096, 2018.
Adrian Bulat, Jing Yang, and Georgios Tyimiropoulos. To learn image super-resolution, use GAN to learn how to do image degradation first. CoRR, abs/1807.11458. 2018.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1. arXiv preprint 1602. 02830, 2016.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems, pp. 5767-5777, 2017.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.000 2015.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-66337 2017.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran EI- Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations, 2016.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient ariger--rithmeeic-onnyy inference. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2704 2704-2713, 2018
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), 2015.
Xudong Mao, Qing Li. Haoran Xie, Raymond YK Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative ad adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, pp. 2813-2821. IEEE, 2017.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 2014.
Daisuke Miyashita, Edward H. Lee, and Boris Murmann. Convolutional neural networks using logarithmic data representation. CoRR, abs//660.00102 2016.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06433 2015.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary comvolutional neural networks. In European Conference on Computer Vision, pp. 525-542. Springer, 2016.

---

Under review as a conference paper at ICLR 2020
Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low- rank matrix factorization for deep neural network training with high-dimensional output targets. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pp. 6655-6659 IEEE, 2013.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems, pp. 2234-2242, 2016.
Junbo Zhao, Michael Mathieu, and Yann LeCun. Energy-based generative adversarial network. arXiv preprint arXiv: 03126, 2016.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06160, 2016.
Chenzhuo Zhu Song Han, Huizi Mao, and William Dally. Trained terarry quantization. arXiv preprint
2016.
10


Reviews:
[{"review_id": "S1gsfgPVtS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors address the quantization of Generative Adversarial Networks (GANs). The paper first performs a sensitivity study for both the generator and the discriminator to quantization methods. Building upon the conclusion of this study, the authors propose a scalar quantization method (QGAN) and compress models to 1 bit of 2 bits weights and show generated images and metrics by the compressed models. \n\nStrengths of the paper:\n- As well stated in the introduction, the compression of GANs (in particular the generator, which is used at inference time) is of practical interest and, to the best of my knowledge, novel. This novelty can be explained by (1) the fact that it takes tome for quantization methods to percolate the entire deep learning field and/or (2) the fact that quantizing GANs has its specificities and own challenges that have not been yet addressed (this is the claim of the authors).\n- The sensitivity study is of interest for the community that can build upon this work. The conclusions (discriminator more sensitive than generator to quantization, quantizing both generator and discriminator helps) are sensible and interesting. \n\nWeaknesses of the paper:\n- The related work section could be greatly improved, thereby showing the limited novelty of the proposed method (QGAN). Indeed, the authors propose to learn the optimal scaling parameters alpha and beta. Many works perform this already and are currently missing in this section, see for instance the two recent surveys: \"A Survey on Methods and Theories of Quantized Neural Networks\", Guo, \"A Survey of Model Compression and Acceleration for Deep Neural Networks\", Cheng et al. \n- Results. The results are not sufficient to justify the performance of the method for two reasons. (1) First, the scale is crucial in assessing the performance of a quantization method. As an example, it is easier to quantize small ResNets on CIFAR-10 than large ResNets on ImageNet. Thus, scales enables to better discriminate between various approaches. I acknowledge that this requires large computational resources but this would greatly strengthen the paper (2) Second, GAN metrics have known shortcomings (see for instance \"How good is my GAN?\", Shmelkov et al.), so the strength of Table 2 is limited. This is in part alleviated by the authors by showing generated images (which is a good practice), but again echoing point (1), larger images would have helped assess better the quality of the quantization.\n\nJustification of rating: \nThe authors propose a sensitivity study that is interesting for the community. However, the proposed method lacks novelty and the results are not convincing enough. I encourage the authors to pursue in this interesting direction which has important practical implications.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BkxGc0P1qS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper propose to study the quantization of GANs parameters. They show that standard methods to quantize the weights of neural networks fails when doing extreme quantization (1 or 2-bit quantization). They show that when using low-bit representation, some of the bits are used to model extremal values of the weights which are irrelevant and lead to numerical instability. To fix this issue they propose a new method based on Expectation-Maximization to quantify the weights of the neural networks. They then show experimentally that this enables them to quantize the weights of neural networks to low bit representation without a complete drop of performance and remaining stable.\n\nI'm overall in favour of accepting this work. The paper is well motivated, the authors clearly show the benefits of the proposed approach compared to other approach when using extreme quantization.\n\nMain argument:\n+ Great overview of previous methods and why they fail when applying extreme quantization\n+ Great study of the influence of the sensitivity to the number of bits used for quantization\n- It would have been nice if the author had provided standard deviation for the results by running each method several times. In particular figure 2.c seem to show that they might be a lot of variance in the results when using low bit quantization.\n- I feel some details are missing or at least lack some precision. For example are the networks pre-trained with full precision in all experiments ? if so can you precise it in section 3.1 also ? \n- The proposed approach seem very similar in spirit to vector quantization, can the author contrast their method to vector quantization ?\n- In equation (7) doesn't the constant C also depend on alpha and beta ?\n- In section 5.1 do you also use the two phase training described in section 4.2 ?\n- Figure 4.c seems to indicate that quantize the generator only is no more a problem ? Can you explain why this figure is very different from figure 2.c \n- In table 3 how is the number of bits chosen, did you try several different values and report the best performance ?\n\nMinor:\n- Some of the notations are a bit confusing. You call X the tensor of x, I think it would be more clear to say that X is the domain of x.\n- I'm surprised by the results in section 3.1, wouldn't the issue described in this section when training standard neural networks ? wasn't this known before ?\n- There is some typos in the text", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxy0i2a5B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a fairly simple yet seemly effective method for quantizing GAN. Existing quantization methods (namely minmax, log, and tanh quantization) for CNN/RNN fail brutally under GAN setting. From empirical observation of the distribution of quantized weights, the authors conjecture the reason being under-utilization of the low-bit representation, called under-representation in the paper. Based on such observation, linear scaling with EM is proposed and experimental results seem to be effective. \n\n[Advantage]\nThe paper is clearly written and easy to follow. The proposed method is well-motivated from the empirical observation presented in Sec 3, and seems to mitigate the difficulties from the discussion in Sec 5.\n\n[Disadvantage & Improvement]\nWhile I am not a direct expert in this area, I do have some concerns regarding the novelty of the method and comparison to previous works. Linear quantization seems to be a common/intuitive method and there are various improvement techniques built upon it (e.g. cliping, ocs,... etc [1,2]). These are related works, yet neither included nor discussed in this paper. How is the presented linear+EM method comparing with these variants, in term of effectiveness on training GAN and the reported test-time metrics? In short, the comparison to previous works seems insufficient in my point of view.\n\nAlso, can you comment on Defensive Quantization (DQ) [3]? The quantization method is specifically designed for adversarial attack/perturbation setting and seems applicable under GAN setting. \n\nLast, there is a typo at the end of Sec 4.2: should it be f_{em}(x) instead of f_{e}m(x)?\n\n[1] Low-bit Quantization of Neural Networks for Efficient Inference\n[2] Improving Neural Network Quantization using Outlier Channel Splitting\n[3] Defensive Quantization: When Efficiency Meets Robustness\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: main summary:  method for quantizing GAN

discussion:
reviewer 1: well-written paper, but reviewer questions novelty
reviewer 2: well-written, but some details are missing in the paper as well as comparisons to related work
reviewer 3: well-written and interesting topic, related work section and clarity of results could be improved
recommendation: all reviewers agree paper could be improved by better comparison to related work and better clarity of presentation. Marking paper as reject.
Decision: Reject