not-so-big-GAN: Generating High-Fidelity Images on Small Compute with Wavelet-based Super-Resolution
Authors: Seungwook Han
Akash Srivastava
Cole Lincoln Hurwitz
Prasanna Sattigeri
David Daniel Cox
Venue:  (2021)
Keywords: deep generative modeling
GAN
super-resolution
wavelet transformation
energy efficient

Abstract: State-of-the-art models for high-resolution image generation, such as BigGAN and VQVAE-2, require an incredible amount of compute resources and/or time (512 TPU-v3 cores) to train, putting them out of reach for the larger research community. On the other hand, GAN-based image super-resolution models, such as ESRGAN, can not only upscale images to high dimensions, but also are efficient to train. In this paper, we present not-so-big-GAN (nsb-GAN), a simple yet cost-effective two-step training framework for deep generative models (DGMs) of high-dimensional natural images. First, we generate images in low-frequency bands by training a sampler in the wavelet domain. Then, we super-resolve these images from the wavelet domain back to the pixel-space with our novel wavelet super-resolution decoder network. Wavelet-based down-sampling method preserves more structural information than pixel-based methods, leading to significantly better generative quality of the low-resolution sampler (e.g., 64×64). Since the sampler and decoder can be trained in parallel and operate on much lower dimensional spaces than end-to-end models, the training cost is substantially reduced. On ImageNet 512×512, our model achieves a Fréchet Inception Distance (FID) of 10.59 – beating the baseline BigGAN model – at half the compute (256 TPU-v3 cores).

Under review as a conference paper at ICLR 2021
NOT-SO-BIG-GAN: GENERATING HIGH-FIDELITY IM- AGES ON SMALL COMPUTE WITH WAVELET-BASED SUPER-RESOLUTION
Anonymous authors Paper under couble-blind review
ABSTRACT
State-of-the-art models for high-resolution image generation, such as BigGAN and VQVAE-2, require an incredible amount of compute resources and/or time (PU-V cores) to train, putting them out of reach for the larger research com- munity, On the other hand, GAN-based image super-resolution models, such ESRGAN, can not only upscale images to high dimensions, but also are efficient train. In this paper, we present NOT-SO-BIG-GAN cost-effective two-step training framework for deep generative models (DGMs) of high-dimensional natural images. First, we generate images in low-frequency bands by training a sampler in the wavelet domain. Then, we super-resolve these images from the wavelet domain back to the pixel-space with our novel wavelet super-resolution decoder network. Wavelet-based down-sampling method pre- serves more structural information than pixel-based methods, leading to signifi- cantly better generative quality of the low-resolution sampler (e.g., 64) Since the sampler and decoder can be trained in parallel and operate on much lower dimensional spaces than end-to-end models, the training cost is substantially re- duced. On ImageNet 512x512, our model achieves Fréchet Inception Distance (FID) of 10.59 beating the baseline BigGAN model at half the compute (256 TPU-v3 cores).
INSB-GAN), simple yet
INTRODUCTION
Generative modeling of natural images has achieved great success in recent years (Kingma &
Welling 2013 Goodfellow et al., 2014 Arjovsky el. [017 Menick & Kalchbrenner 2019 Zhang
et al.) 201
models (Miyato Advancements in scalable computing and theoretical understanding of generative
al. 2018 Zhang et al. 2018a| Guirajani al. 2017, Mescheder et al. 2018
2017 Roth et al. 2017, Nowozin et al.) 2016 Srivastava et al.] [017] 2020, Karras et al. 2020).
have, for the first time, enabled the state-of of the-artt techniques to generate photo-realistic images in
higher dimensions than ever before (Brock et al. 2018 Razavi et al. 2019 Karras et al. 2020
Yet, generating high-dimensional complex data, such as ImageNet, still remains challenging and ex- tremely resource intensive. At the forefront of high-resolution image generation is BigGAN (Brock) jt al. 2018), a generative adversarial network (GAN) (Goodfellow et al. (014)) that tackles the curse of dimensionality (CoD) head-on, using the latest in scalable GPU-computing. This allows for train- ing BigGAN with large mini-batch sizes (e.g., 2048), which greatly helps to model highly diverse, large-scale datasets like ImageNet. But, BigGAN's s ability to scale high- nigh-dimensional data comes at the cost of a hefty compute budget. A standard BigGAN model at 256 256 resolution can require up to a month or more of training time on as many as eight Tesla V100 graphics processing units (GPUs). This compute requirement raises the barrier to entry for using and improving upon these technologies as the wider research community may not have access to any specialized hardware (e.g., Tensor processing units (TPUs) (Jouppi et al.) 2017). The environmental impact of training large-scale models can also substantial as training BigGAN on 512x512 images with 512 TPU cores for two days reportedly used as much electricity the average American household does in about six months (Schwab) 2018).
Motivated by these problems, we present NOT-SO-BIL S-O--IIG-GA (NSB-GAN) a small compute training alternative BigGAN, for class-conditional modeling of high-resolution images. In end-to-end

---

Under review as a conference paper at ICLR 2021
generative models of high-dimensional data, such VQVAE-2 Razavi et al. 2019) and Karras et al. (2017), the lower layers transform noise into low resolution images, which are subsequently upscaled I.c. super-resolved to higher dimensions in the higher layers. Based this insight, in NSB-GAN we propose to split the end-to-end generative model into two separate neural networks, sampler and up-scaling decoder that can be trained in parallel on much smaller dimensional spaces. In turn, we drastically reduce the compute budget of training. This split allows the sampler be trained in up 16-times lower dimensional space, not only making it compute efficient, but also alleviating the training instability of and-to-end approaches. To this end, we propose wavelet- space training of GANs, As compared to pixel-based interpolation methods for down- hown-ssmpling images, wavelet-transform (WT) (Haar) 1909 Daubechies] 1992. Antonini al., 1992) based down- sampling preserves much more structural information, leading to much better samplers in fairly low resolutions (Sekar et al. 2014). When applied to 2D image, wavelet transform slices the image into four equally-sized image-like patches along different frequency bands. This process can be recursively applied multiple times in order to slice large image into multiple smaller images, each representing the entire image in different bandwidths. This is diagrammatically shown Figure [ Here, the top-left patch (TL) lies in the lowest frequency band and contains most of the structure of the original image and therefore the only patch preserved during downsampling. The highly sparse top-right (TR), bottom-left (BL) and bottom-right (BL) patches lie in higher bands of frequency and are therefore dropped. But wavelet-space sampling prohibits the use of existing pixel-space super- resolution models, such as Ledig al. (017)); Wang et al. (2018). to upscale the samples. Thus, we introduce two wavelet-space space image encoding, while matching the performance of equivalent pixel-space methods. Training our decoders is extremely compute efficient (e.g., 3 days on the full ImageNet dataset), and, once trained on diverse dataset like ImageNet, can generalize beyond the original training resolution and dataset.
super-resolution decoder networks that can work directly with wavelet-
Down-Sample
Top-left (T.)
LPF
Down-Sample
Down-Sample
Top-Right (TR)
LPF
Down-Sample
Bottom (BL)
HPF
Down-Sample
Down-Sample
Bottom Bottom-Aight (BR)
Figure 1: Wavelet transformation consists of low-pass and high-pas, sigh-pass&ee filter, followed by down- sampling step that splits the image into two equal-sized patches. Each these two patches undergo the same operation again resulting in four equal-sized patches, TL., TR, BL and BR.
Our main contributions are the following:
We introduce simple training framework NSB- GAN for large generative models and show that super-resolving res-uusotttion samplers reduces the cost of training by orders of mag- nitude without sacrificing image quality and even outperforms the baseline BigGAN model at higher resolution (e.g., 512 x 512).
In addition simple pixel-based version of NSB-GAN, introduce wavelet-space train- ing of DGMs and demonstrate that it leads to better image quality compared to pixel-space NSB-GAN model average.
While conceptually simple (especially in the pixel-space), training large models to work in tandem requires careful engineering. Therefore, make entire ASB-GAN code-base publicly available to the wider research community.
BACKGROUND AND RELATED WORK
Given set X {2g/Vi N). E R2} of samples from the true data distribution p(x), the task of deep generative modeling is to approximate this distribution using deep neural networks. All generative models of natural images assume that p(x)) is supported on a K-dimensionnl, smaller
2

---

Under review as a conference paper at ICLR 2021
manifold of R.. As such, they model the data using conditional distribution pz(2]2), where 0 represents the model parameters and Z E IRK is the latent variable with the prior p(z). Z is marginalized to obtain the log- rgglikelihood (LLH) log (z log f p(z) (z)z)dz of X under the model. Generative model are trained by either maximizing this L.H. or its lower-bound. There are two types of deep generative models, explicit models like variational autoencoders (VAE) (Kingmal & Welling, 2013) that have an explicit form for the LLH and implicit models such as GANs (Good- fellow al., 2014) that do not have such a form. VAES use decoder network Dg to parameterize the model distribution po (z|22 and an encoder network Eg to parameterize a variational distribution (=(2)) as an approximation to the true posterior under the model. They can then be trained using the evidence lower-bound (ELBO), po 2 - [=][((([])]]
+ (=(2)))) log po (z/lz)dz. KL here refers to the Kullback-Leible divergence between ( .r) and p(z). Unlike VAžs, GANs do not make any assumptions about the functional form of the conditional distribution (=(x]z) and, therefore, do not have a tractable likelihood function. Instead, they directly attempt to minimize either variational lowerbound to chosen f-divergence (Nowozin et al. 2016
Srivastava et al.
2017 or integral probability metric (IPM) (Arjovsky et al. [017]] Li et al.
2017 Srivastava et al,,
2020 between the model distribution po(2) and p(z) using binary classifier based density ratio estimators,
Though the idea of multiscale modeling in GANs has been considered throughly in Denton et al. 2013); K et al. (2017) 2020); Liu et al. (2019b), our method is more closely related VQVAE-
2 (Razavi et al. (019) and the Subscale Pixel Network (SPN) (Menick & Kalchbrenner 2019) mod- els. 2QVAE-2 is a hierarchical version of the VQVAE model (van den Oord et al.. 2017) that unlike VAEs, does not impose KL-based regularization on the latent space. Instead, VQVAE/VQVAR2 models use vector quantization as a strong functional prior. As such, sampling from the latent space is not straightforward and requires a large compute budget to train auto-regressive PixeISNAIL. (Chen et al. (017) models to estimate the density on the latent space to generate samples. [Menick] & Kalchbrennerr 2019) shows that it is easier to generate high-resolution images by splitting the process into two steps. In step one, they slice the image into small patches and then learn to gen- erate these patches using patch-level autoregressive model. This step allows for capturing the general structure of the image but misses the finer details. Therefore, in step two, they train another network that leams to fill in the missing details. The SPN model has a significant shortcoming, however, as its patch-level autoregressive approach fails to capture long term dependencies (e.g. eyes on a face looking in different directions). Similar to SPN's multi-scale approach, NSB-GAN decouples structure learning from adding details. Unlike SPN, however, it does not suffer from the long-term dependency problem. This is because NSB- -SB-GAN uses wavelet transformation to slice the image, which naturally decouples the structure (TL) from the details (TR, BL, BR) in the image. We demonstrate this difference diagrammatically the Appendix F More importantly, wavelet trans- form alleviates the need for autoregressive modeling of the patches each patch represents the entire image in different frequency bands.
Image Super Resolution Since the pioneering work of SRCNN (Dong et al. 2014) to tackle the problem of single image super- resouttrso, recovering a high-resolution image from a low-resolution image, with a deep neural network, deep convolutional neural network approaches have demon- strated great success reconstructing realistic high-resolution images. ESRGAN (Wang et al.) 2018) is the state-of-the-art GAN model that majorly builds on SRGAN, RDN, and EDSR (Ledig al. 2017 /Zhang et al., 20185; Lim et al., 2017). Improving upon its predecessor SRGAN, modifies the SRGAN's architecture from SRResNet batch normalization, finetunes the perceptual loss implemented with the VGG-19 model (Simonyan & Zisserman 2015). and utilizes a relativistic GAN-like Molicoeur-Martinee
Residual-in-Residuu Dense Block (RRDB) without
2019) adversarial loss to predict relative realness. First, only the generator is trained as Peak Signal-to-Noise Ratio (PSNR)-oriented model with LI loss. Then, the GAN is trained as a whole, initialized from this pre-training, with carefully balanced LI, perceptual, and adversarial losses. With such, ESRGAN is able to reconstruct realistic high-frequency texture and details missing in the low-resolution images.
METHOD
LAB-GAN training introduces a fairly simple change to the original BigGAN model. Instead of training the generator directly on the full dimensionality of the data, D = 256 (say, for ImageNet dataset), train the generator in K 64. Then, in order to upscale the 64 x 64 image back
3

---

Under review as a conference paper at ICLR 2021
to 256, train up-scaling decoder. We now describe two specific instances of the NSB-GAN framework, NSB-GAN- W and NSB-GAN-P in detail.
3.1 NSB-GAN-W
NSB- ASB-GAN-T W is best described an autoencoder that operates in the frequency domain instead of the usual pixel space. It is comprised of a deterministic encoder, a learned decoder and learned prior network, i.e. sampler. These three components form a full generative model that can produce high-quality, high-resolution samples with week of training commodity hardware.
3.1.1 DETERMINISTIC ENCODER.
The USB-GANA encoder is deterministic function that recursively applies wavelet transform to the input image, retaining only the TL patches at each level. Each TL patch is quarter of the size of the previous TL patch, resulting in pyramid-like stack of 2D patches after multiple transformations. The last TL contains lowest frequencies and, therefore, the most structural information about the image. We use it as a compressed, lossy representation of the input image in our encoding step.
For N dimenssiol image X, let us denote the matrix obtained after wavelet transform WXX), which is a N x block matrix with the following structure,
(X) ().(()) (X) W22(X) ]
W(X)
Here, with slight abuse of notation, Wi. represents the ( 3) dimeesiooall TL patch of image X. NSB-GAN's encoder & can then be defined recursively as follows,
Eo(X) X { = ()-(((--(())) I
(1)
where / is the number of wavelet transforms applied to the input image. As can be seen, the size of the retained TL patch decreases exponentially with l.
3.1.2 DECODER
After wavelet transform, the original image can be deterministically recovered from the TL, TR, BL, and BR patches using IWT. NSB-GAN-W. however, discards the high-frequency patches dur- ing its encoding step, rendering a fully deterministic decoding impossible. Thus, the NSB-GAN decoder first learns to recover the missing high-frequency patches (using neural network) and then deterministically combines them using IWT to reconstruct the original input. We now present two NSB-GAN-W decoder designs based ResNet (He al. 2016: Ledig al. 2017) and UNet Ronneherger et al. 2015) architectures respectively.
ESRGAN-W Since NSB-GAN-V W sampler generates samples in the wavelet domain, we modify the basic ESRGAN architecture (Wang al. 2018) to allow it accept images in the wavelet- domain, up-scale them, and then project them back to the pixel-space. Specifically, we replace the generator in ESRGAN with 16-block-deep SRResNet without batch normalization (Ledig al. 2017]. denoted by fa, and replaced the bilinear interpolation based up-scaling with IWT. We can then define the decoder as,
W
0 0 0
)
= IWT =IW
+ foW).))
This design not only allows us to directly input wavelet-encoded images the decoder, but also lets us use the carefully balanced GAN-based training of ESRGAN since our decoder still operates in the pixel space to recover the missing higher frequencies. Similarly to ESRGAN, we pre-train SRResNet with only L1 Loss, initialize the generator at this state, and then start the full adversarial training with perceptual loss and adversarial loss. We refer to these two models as ResNet-W and ESRGAN-W. respectively. We defer further training details of the decoder to the Appendix
4

---

Under review as a conference paper at ICLR 2021
UNet-Decoder While the ESRGAN-W decoder works well with wavelet-encoded inputs, it does not take full advantage of the compression that wavelet space modeling brings about. Therefore, we also introduce UNet-based decoder that takes full advantage of the wavelet decomposition of 2D data to completely bypass the original dimensionality of the target image. Due to space limitations, defer the details of this design to the Appendix
3.1.3 SAMPLER: LEARNED PRIOR NETWORK
The functional prior imposed by our deterministic encoder leads to highly structured represen- tation space made up of low frequency TL patches of images. In order to generate from NSB- GAN-W, one must be able to draw samples from this space. We posit that, compared to sampling from equivalemll--sized representation spaces for AE and VAFs, it is easier to sample from a low- dimensional, image-like like latent space using generative methods such as GANs, as they repeatedly have been shown to excel in learning to samples from image distributions. Therefore, we train a BigGAN sampler on this representation space. As the dimensionality is considerably lower than the original image, it only takes a single week and two Tesla V100 GPUs to to train the sampler. Since the values of the TL patch may not lie in the [o, 1], range, care needs to be taken in order to normalize the encoded samplers properly as failing to do so prevents the BigGAN sampler to learn the correct distribution.
Pre-trained Sampler. It is possible to use a pre-trained BigGAN model at a lower resolution within NSB-GAN-W to generate higher resolution images. Consider pre-trained BigGAN model that can generate 128 x 128 samples in the pixel-space. By simply projecting the samples into the frequency-space using our deterministic encoder (down to 64 x 64) and then up-scaling it through our decoder network, one can generate samples at 256x256 without any significant loss in quality. In fact, these samplers can outperform the end-to-end baseline BigGAN model not only in compute requirement, but also in terms of the FID at 512x512 resolution.
3.2 NSB-GAN-P
NSB- - the pixel counterpart of NSB-GAN-W Instead of encoding, generating, and decod- ing in the wavelet space, it operates in the pixel space. The deterministic encoder sub-samples the original image in the pixel space to generate a low-resolution latent representation. Specifically, we realize the sub-sampling as smoothing with a Gaussian filter and sub-sampling with respect to the upscaling factor r (Jo et al. 2018). For the NSB-GAN-P decoder, we use the ESRGAN model with the generator replaced with SRResNet without batch normalization, as in NSB-GAN- refer to as ESRGAN-P and the BigGAN model on low-dimensional space for the prior network. The training procedure is same in ESRGAN-W W. where first we pre-train SRResNet with only L1 loss (namely, ResNet-P) and then start adversarial training. Similar to the NSB-GAN-W W model, we also have a UNet-version of the decoder but it reconstructs directly in the original data dimensionality like the ESRGAN based decoder. Further details are in Appendix C.2
which we
3.3 TRAINING
As before, let X be the dataset of nigh-dimensional natural images. The NSB-GAN encoder can be defined as a deterministic function E : R + R that uses the WT to produce Z = {2,1aj e NJ. 2j E R Using this paired dataset {X, 2}, we treat NSB-GAN as a fully observable model and train the decoder function Ds IR + R8 D to reconstruct X from 2 by minimizing the negative log-liielioood (NLL)- Ep(,,,) [log po [x|=)] of the conditional probability distribution
that it parameterizes. Learning of the generator function Go : RK + RKK which is referred to as learning the prior in previous literature (Razavi al. 2019 De Fauw et al., 2019). is essentially fitting a generative model to the marginal distribution of Z. LC. p(z). While we could use a sim-
ilar approach as above and fit G& that parameterizes the model distribution p=(2) by minimizing -Epte) [log Ps (z)], we chose to instead use a BigGAN model to directly minimize the (variational lower-bound to the) f f-divergence [p(p()) (=) (Nooozin et al. 2016 between the true marginal distribution p(z) and the model distribution it provides better fit for natural-image like distribu- tions compared to other approaches. All together, NSB-GAN simply specifies a generative model over X and Z jointly and is, therefore, trained by minimising the NLI.,
(p)(,)) (log po ( (I. =)] Ep((,s) [log po (x]=)) Ep(a)) Dog [o)))()))]
5

---

Under review as a conference paper at ICLR 2021
Sampler
Decoder
Resolution
min FID/IS
Compute Budget
Learned-P-64
ESRGAN-P
256
32.66 / 89.81
8 days on 4 x V100 GPUs
Learned-W-64
ESRGAN-W
256
21.82/ / 119.8
7 days on 4 x V100 GPUs
Learned Baseline
None
256
> 200 / < 10 7 days on 8 x V100 GPUs
The learned samplers above are realized with BigGANriigGAN-derr architecture. On small compute budget, it is not only possible to train the Learned W-64 sampler well, but also the NSB-GAN model (with Learned-W-or sampler) beats the state-of-the-arr VQVAE-2 model (before truncation/rejection sampling), which has extremely high cost of training similar that of the BigGAN model. We defer a more detailed discussion about truncation and rejection sampling in Appendix B.] and showcase more detailed set of results in Table in the Appendix B.1
Table 1:
4 EXPERIMENTS
In this section, we quantitatively and qualitatively evaluate the training efficiency of the NSB-GAN approach against the end-to-end training approach of the BigGAN model by benchmarking the com- pute budget required vs. the quality of the generated samples. While quantifying image quality re- mains challenging problem (Borji 2019), we report the Frechet Inception Distance (FID) (Heusel iet al., 2 and Inception Score (IS) (Salimans et al., 2016) as proxy measures to allow for direct comparison with BigGAN. All results are reported on the ImageNet dataset (Deng al. 2009) at two different resolutions, 256 and 512x512.
Compute Budget. For our main experiments on training efficiency, depending on the sampler used, we define two different compute budgets. For the learned samplers, we train the NSB-GAN- W and NSB-GAN-P models for a total of 168 hours (7 full days). Our training is performed on a single machine with four Telsa V100 GPUs with 16GB of VRAM each. We found that training the baseline model (BigGAN) at 256x with only 4 GPUs in 7 days is not possible on the default setting, allow total of 8 GPUs for the baseline training. Since the training of these BigGAN-basee samplers are highly unstable (Zhao et al.) 2020), we train 5 instances in total (3 BigGANNdeep and BigGAN samplers) and report results using the instance with the best FID score. For the pre-traines sampelrs, the compute budget is computed in terms of the TPUs used in their original training and does not include the decoder training. This is because the compute budget required to train the decoder is negligible compared to that of the sampler.
Hyperparameters and Setup. For the deterministic encoder using wavelet transformation, we instantiate it with biorthogonal 2.2 wavelet. For the NSB-GAN sampler, we use batch size of 512 and learning rates of 10 and 4 x 10a for the generator and discriminator, respectively. For the pre- training of ESRGAN-W and ESRGAN-P with L1 loss, rate is initialized 1 x 10-4 and decayed by a factor of 2 every 2 x 10a mini-batch updates. During adversarial training of ESRGAN-W and ESRGAN-P, the learning rate is set to 10-4 and halved at [50k; 100k] iterations. In the case of the UNet-decoders, for the first-level decoder, we use a batch size of 128 and learning rate of 10 For the second-level decoder, we use the same learning rate as the first-level decoder, but a smaller batch size of 64. We allocate two GPUs to the NSB-GAN sampler, and the other two for the decoders. For the UNet-decoder, this implies one GPU per level. Each component is trained in parallel and independently from each other. To train the BigGAN baseline model at the native resolution of 256 x 256, we use the same learning rates as for the NSB- GAN sampler and recommended batch size of 2048 (Brock et al. 2018). The baseline model trains with eight GPUs, instead of four like with NSB-GAN to meet its large memory requirement, given the batch size and resolution of the image. We provide training and evaluation code for NSB-GAN sampler and the other models, respectively here: https: http::ttttssssso/ttttoooo
batch size of 32, and the learning
cca 3 f2e- 5c27- bdd -ele-aaaeee-edddbeeeee 38c1/

---

Under review as a conference paper at ICLR 2021
4.1 RESULTS
4.1.1 LEARNED SAMPLERS
To establish the training efficiency and competitive image quality of our NSB-GAN approach, compare the FID and IS that the NSB-GAN- W. NSB-GAN- and baseline BigGAN models reach in the compute budget of 7 days with four (8 for the baseline) Tesla V100 GPUs in Table Both NSB- GAN-P and NSB-GAN-W W models reach very competitive FIDs of 32.66 and 21.82, respectively. To put these results in perspective, even SAGAN (Zhang et al. 2018a). which requires twice the compute and operates on half the resolution (128 X 128), reaches an FID of 19. Furthermore, in the same amount of time but twice the compute, the baseline BigGAN model fails to generate any meaningful samples across all five runs. This illustrates that, compared to end-to-end models, the NSB- BAN models are significantly more compute efficient and can reach competitive image quality.
In the aforementioned models, we use the public PyTorch implementation of BigGAN (kindly pro- vided by the authors) to provide fair comparisons between the NSB-GAN models and the baselines. important to note, however, that the Pytorch implementation not optimal as it does not use (SBN) which is essential for large mini-batch training. This implementation has been reported to not reach the FID and IS reported in the original paper (ajbrock] 2018).
When the sampler is trained in the low-dimensional space of 64 x 64, NSB-GAN- W clearly out- performs NSB-GAN-P. We posit that this sharp difference in FID stems from the difference in the loss of structural information using the two down-sampling methods: pixel-based interpolation and wavelet-space encoding (Sekar al. 2014). As illustrated in Figure 2 in Appendix A WT encod- ing seems to preserve more structural information than pixel-based interpolation which misses key features of the image due to arbitrary sub-sampling. This finding suggests that the down-sampled distribution at 64 x 64 becomes sufficiently different from the original distribution at 256 x 256 such that when trained on the down-samppl approximate the structure in the original data distribution. If this were true, the NSB-GAN-P per- formance should improve as the level of down-sampling is decreased. With the following set of experiments with pre-trained samplers, we test this hypothesis and demonstrate that NSB-GAN framework can also be used with pre-trained samplers to make image generation at high resolution (256 x 256, 512 x 512) efficient and even beat the BigGAN baseline model in some cases.
distribution, ASB-GAN-P sampler fails to sufficiently
4.1.2 PRE-TRAINED SAMPLERS
We consider two pairs of pre-trained BigGAN samplers combined with our NSB-GAN models to ul- timately generate at two different resolutions of 256x255 and 512x512, To clarify, our pre-trained samplers are simply serial combination of pre-trained BigGAN models ample, Pretrained- 128-64 is created by generating samples from pre-trained sampler at 128 x 128 and then applying down-sampling operation (pixel or wavelet) to obtain 64 64 samples. Then, our decoders up-scale the low-resolution image four times to 256 x 256. We report the FID/IS scores for all the samplers and compare to the baseline BigGAN models pre-trained at the respective reso- lutions in Table 2 First, notice that NSB-GAN-P performance increases drastically, supporting hypothesis. Next, both NSB-GAN-V W and NSB-GAN-P models require significantly less compute (up to 4 times less TPU-v3 cores, as shown in the last row of Table 2 to train. Most importantly, note how approach outperforms the original BigGAN model in terms of FID at 512x512 resolution, despite using exactly half the original compute budget. We hypothesize that this is partly due to the way in which ImageNet 512x512 dataset is generated, but defer this discussion to Appendix H
[adells[annd and our encoders. For ex-
Furthermore, note that we did not have to re-train the decoders when applying to different resolutions and samplers. All results in this experiment are done with the decoders from the previous experiment that were only trained to up-scale images from 64 x 64 to 256 x 256. They evidently generalize well across different resolutions. This amortization of training across resolutions further reduces the training cost when more than one generator is trained.
UNet-Decoders As shown in Table 4 of the Appendix B.1 our ESRGAN-based decoders clearly outperform our UNet-decoders on FID and IS. It is important to note, however, that this difference primarily because the ESRGAN decoders employ adversarial training which has been shown to drastically improve image quality as measured by FID. As shown in Table 3 not considering
We use the pre-trained models from huggingface's Py re-implementation of BigGAN model.

---

Under review as a conference paper at ICLR 2021
Sampler
Decoder
Resolution
min FID /TD//S
Compute
BigGAN-128
None
128
10.58 43.72
128 TPUs
BigGAN-226
None
256
10.68 52.16
256 TPUs
Pretrained- cetrained-128-44 128-64
ESRGAN-P
256
12.28
46.06
128 TPUs
Pretrained- 128-64
ESRGAN-W
256
12.66 45.54
128 TPUs
BigGAN-552
None
512
11.32/ 49.37
512 TPUs
Pretrained-256-4
ESRGAN-P
512
10.30 / 213.4
256 TPUs
seerained-226-111
ESRGAN-W
512
10.59
52.14
256 TPUs
Pretrained- --8
ESRGAN-P
512
13.55 47.70
128 TPUs
Table All pretrained samplers above are realized with BigGAN. In higher dimensions, compared to the baseline BigGAN models at the respective resolutions, NSB-GAN models reduce the training compute budget by up to four times (last row) while suffering only minor increase in FID. All pre-trained samplers above are trained for approximately two days on the compute described, Note that the decoder only trained once, but generalizes across all the resolutions. This amortization further reduces the training cost drastically, Pretrained- 12--44, for example, indicates that the model generates at 128 x 128 resolution and we dlon--sampe to 64 x 64 resolution with our encoders for MSB-GAN models. As empirically tested and confirmed by Razavi et al. (2019), IS is highly sensitive to slight blurs and perturbations. Therefore, we include an expanded set of guantitative results with various truncation and rejection sampling levels in Table 5 in the Appendix B.2
UNet-P
UNet-W
VQVAE-2
Train MSE
0.0061
0.0045
0.0047
Valid MSE
0.0074
0.0049
0.0050
Table 3: MSE on training and validation set for UNet-P, UNet-Waand VQVAE-2 models. Small difference between the training and validation error suggests that the models generalize well.
adversarial decoder, our UNet-W W decoder leads to the best reconstruction error compared to the pixel-space decoder, including the state-of-the-ar VQVAE-2 model.
Evaluation on LSUN Church with StyleGAN-2-W The NSB-GAN approach can be imple- mented with models other than BigGAN. To demonstrate this, replace the BigGAN sampler chitecture in our model with the StyleGAN-2 architecture and re-run the experiments on the LSUN- Church dataset. The results in Table 10 of the Appendix demonstrate that our NSB-GAN models can generalize to different samplers and datasets.
Qualitative Results. Due to space limitations, we have deferred all qualitative results to the Ap- pendix. Please see the Appendix A. A.2 and A.4 for a detailed qualitative comparison of all the models.
CONCLUSION
In this work, present genre of compute- tfffutte-ffficeent generative models, NOT-SO-BIG-GAN that achieve comparable image quality to the current SoTA DGM (BigGAN) with dramatically lower compute budget. Surprisingly, NOT-SO-RIG-GAN is even able to outperform BigGAN in image quality at 512 512 resolution. Overall, we hope that our work inspires others to develop low- compute generative models that be utilized and iterated by the wider research community.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
ajbrock.
Training results(is and fid) are not good as yours with same training process
issue 23 pbbroch 2018. URL https://githut: wtt.:atth..o...aa.eeeoeeeeeeae //jook//
Big CaAAAN AewA--CyNnncciiiee Pww.oooooooochsss/2222222
Mare Antonini, Michel Barlaud, Pierre Mathieu, and Ingrid Daubechies. Image coding using wavelet transform. IEEE Transactions on image processing, 1(2):005-220, 1992.
Martin Arjovsky, Soumith Chintala, and Léon Bottou.
Wasserstein gan. arXiv preprint
MrXfv:1701.0785. 2017.
Ali Borji. Pros and cons of gan evaluation measures. Computer Vision and Image Understanding, 179:41-653 66, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2018.
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved autore- gressive generative model. arXiv preprint arXiv: 2017.
Ingrid Daubechics. Ten lectures on wavelets, volume 61. Siam, 1992.
Jeffrey De Fauw, Sander Dieleman, and Karen Simonyan. Hierarchical autoregressive image models with auxiliary decoders. arXiv preprint arXiv: /93xv3/903.0493,, 2019.
Deng. W. Dong, R. Socher, L. Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pallern Recognition, pp. 248-255 2009.
Emily Denton, Soumith Chintala, Rob Fergus, et al. Deep generative image models using a laplacian pyramid of adversarial networks. In Advances neural information processing systems, pp. 1486-1494, 2015.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning deep convolutional net- work for image super-resolution. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuyte- laars (eds.), Computer Vision ECCV 2014, pp. 184-199, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10593-2.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In Neural Information Processing Systems, 2014.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of wasserstein gans. Advances in neural information processing systems, pp. 5767-5777, 2017.
Alfred Haar. the theory of orthogonal functional systems. Georg August University, Gottingen., 1909.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANS trained by time-scale update rule converge local nash equilibrium. In Neural Information Processing Systems, 2017.
Xiaodan Hu, Mohamed A Naiel, Alexander Wong, Mark Lamm, and Paul Fieguth. Runet: A robust unet architecture for image super-resolution. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
Y. Jo, S. W. Oh, J. Kang, and S. J. Kim. Deep video super-resolution network using dynamic upsampling filters without explicit motion compensation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3224-3232, 2018.

---

Under review as a conference paper at ICLR 2021
Alexia SalcoeurrMaatineaat The relativistic discriminator: key element missing from stan- dard GAN. In International Conference on Learning Representations, 2019. URL https:
Norman P Jouppi, Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analy- sis tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture, pp. 1-12, 2017.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint 17X0v 10196, 2017.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8110-8119, 2020.
Diederik P Kingma and Max Welling. arXiv:1312.6114, 2013.
Auto-encoding variational bayes.
arXiv preprint
Christian Ledig. Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo-realistic sin- gle image super-resolution using a generative adversarial network. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4681 1898-, 2017.
Chun-Liang Li, Wei-Cheng Chang, Yu Cheng, Yiming Yang, and Barnabás Póezos. Mmd gan: Towards deeper understanding of moment matching network. In Advances in Neural Information Processing Systems, pp. 2200-2201111 2017.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep resid- ual networks for single image super-resolution, 2017.
Jinlin Liu, Yuan Yao, and Jianqiang Ren. An acceleration framework for high resolution image synthesis, 2019a.
Jinlin Liu, Yuan Yao, and Jianqiang Ren. An acceleration framework for high resolution image synthesis. arXiv preprint arXiv: (tt9::999000000 0, 2019b.
Jacob Menick and Nal Kalchbrenner. GENERATING HIGH FIDELITY IMAGES WITH SUB- SCALE PIXEL NETWORKS AND MULTIDIMENSIONAL UPSCALING. In International Conference Learning Representations, 2019. URU/tttsss//opeerviviee net/ /formm?
Hdd
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. The numerics of gans. In Advances in Neurul Information Processing Systems, pp. 1825-1835, 2017.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? arXiv preprint arXiv: 1801.0000, 2018
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. International Conference on Learning Representations, 2018. URLAAttps://ppneeeee wwwp:/rwwwwwwwwweweeeeeeeeeeeeee net wtt.:ttteoorrmma/dd-oo02
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances neural information processing systems, pp. 271-279, 2016.
Ali Razavi, Aaron den Oord, and Oriol Vinyals. Generating diverse high- images with vq-vae-2. In Advances in Neural Information Processing Systems, pp. 14837-14847, 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241, Springer, 2015.
10

---

Under review as a conference paper at ICLR 2021
Kevin Roth, Aurelien Lucchi, Sebastian Nowozin, and Thomas Hofmann. Stabilizing training of generative adversarial networks through regularization. In Advances in neural information pro- cessing systems, pp. 2018-2022 2017.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234 2242, 2016.
Katharine Schwab. A google intern built the ai behind these shockingly good fake images, October 2018. URL
wtwp::/wwww.t.tttaa....ooooaaaa com/ 90244767/
http::/ehoooccooo-eaaaaaaaee-eee line; posted 10-02-18].
[On-
K. Sekar, V. Duraisamy, and A. M. Remimol. An approach of image scaling using dwt and bicu- bic interpolation. In 2014 International Conference Green Computing Communication and Electrical Engineering (ICGCCEE), pp. 1-5, 2014.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, 2015.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. In Advances in Neural Infor- mation Processing Systems, pp. 3308-3318, 2017.
Akash Srivastava, Kai Xu, Michael U. Gutmann, and Charles Sutton. Generative ratio matching networks. In International Conference on Learning Representations, 2020. URL https:// openreview net wwwet//formmiddddS3 /
Aaron den Oord, Oriol Vinyals, al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 6306-6315, 2017.
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong. Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 0-0, 2018.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. arXiv preprint arXiv:1805.088 08838 2018a.
Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution, 2018b.
Yang Zhao, Chunyuan Li, Ping Yu, Jianfeng Gao, and Changyou Chen. Feature quantization im- proves gan training. arXiv preprint arXiv:2004.02088, 2020.
A APPENDIX
A.1 NSB-GAN SAMPLES WITH LEARNED SAMPLER
Refer to Figure
for samples from NSB-GAN models.
A.2 NSB-GAN SAMPLES WITH PRE-TRAINED SAMPLER
Refer to Figures 4b and 4a for class-conditional samples from NSB-GAN-P and NSB-GAN-W models at 256 256, respectively. Refer to Figures 7a
8a 5a
and ba for class-conditional samples 512. Samples from BigGAN models at
from NSB-G. -EB-GANPP and NSB- BSB-GAN-W models at 256 x 256 and 512 x 512 are also included in 10 and T1 for comparison. We intentionally include random samples from the same classes for all the models to allow for direct and easy comparison.
A.3 FULL RESOLUTION SAMPLES FROM NSB-GAN-W WITH PRE-TRAINED SAMPLER
Refer Figures
12
13
14
and
1.5 for full resolution samples from NSB-GAN-V W 256 256.
Refer Figures
16
17
18 and
19 for 512 512. These samples fitted to the page.

---

Under review as a conference paper at ICLR 2021
Tu mMm
Figure 2: The above images are the original, down-sampled version using wavelet encoding, and down-sampled version using pixel-based interpolation (bilinear) in the column-wiss order. Please note that the pixel-based interpolation loses key structural information the images face of the person, structure of the boat, and accordion keyboard whereas wavelet encoding does not.
Sampler
Decoder
Resolution
min FID / IS
FID / max IS
Leamed-P-64l
ESRGAN-P
256
32.66 / 89.81
33.61 / 16.122
Learned --6m
ESRGAN-W
256
21.82 / 119.75
27.52 / 219.4
Learned-P-6h
UNet-W
256
35.60
84.68
38.58 / 129.4
Table 4:
Minimum FID / IS (column 4) and FID / Minimum IS (column 5) attained with different levels of truncation [1.0, 0.8, 0.6, 0.4, 0.2, 0.1] and rejection sampling [0.70, 0.80, 0.90, 0.95).
A.4 FULL RESOLUTION SAMPLES FROM NSB-GAN-P WITH PRE-TRAINED SAMPLER
Refer to Figures 20
21
22
and
23
for full resolution samples from NSB-GAN-V W 256 x 256.
Refer to Figures
24
25
26
and
27
for 512 512. These samples fitted to the page.
B RESULTS WITH TRUNCATION AND REJECTION SAMPLING
B. LEARNED SAMPLERS
For a more thorough analysis of our NSB-GAN models with learned samplers, apply trun- cation and rejection sampling to study their effects on FID and IS. We apply truncation at [1.0, 0.8, 0.6, 0.4, 0.2, 0.1]. and, given the truncation level at which the model outputs the best FID, we apply rejection sampling with a pre-trained Inception V3 model on ImageNet. We test rejection sampling thresholds 0.70, 0.80, 0.90, and 0.95. Refer to Table 4 for quantitative results FID and IS. We also include FID and IS results for UNet-W W decoder that was only trained with a MSE loss.
12

---

Under review as a conference paper at ICLR 2021
(a) NSB-GAN-P with learned sampler
(b) NSB-GAN-W with learned sampler
Figure 3: Class-conditional random samples 256 256 from NSB-GAN models with learned samplers. Classes from the top row: 12 house finch, 129 spoonbill, 200 Tibetan terrier, 323 monarch butterfly, 493 chiffonier, 727 planetarium, 780 schooner, 853 thatch, 970 alp. and 992 agaric.
Sampler
Decoder
Resolution
min FID IS
FID / max IS
Pretrained- -1atnned--1---4
ESRGAN-P
256
12.28 46.06
13.85 / 229.6
Pretrained- 128-64
ESRGAN-W
256
15.55
20.44 / 285.5
Pretrained-256-128
ESRGAN-P
512
10.30/213.35
21.85 / 338.4
Pretrained-256-122
ESRGAN-W
512
10.59 / 52.14
22.26 332.7
Table Minimum FID IS (column 4) and FID Minimum IS (column 5) attained with different levels of truncation [1.0, 0.8, 0.6, 0.4, 0.2, 0.1] and rejection sampling [0.70, 0.80, 0.90, 0.95).
B.2 PRE-TRAINED SAMPLERS
We conduct the same detailed analysis for our ASB-GAN models with pre-trained samplers as the above. Refer to Table for quantitative results FID and IS.
13

---

Under review as a conference paper at ICLR 2021
(a) NSB-GAN-W with pre-trained sampler
(b) NSB-GAN-P with pre-trained sampler
Figure 4: Class-conditional random samples 256 x 256 from NSB- -BN models with pre- pre-trained trained samplers. Classes from the top row: 12 house finch, 129 spoonbill, 200 Tibetan terrier, 323 monarch butterfly, 493 chiffonier, 727 planetarium, 780 schooner, 853 thatch, 970 alp. and 992 agaric.
C ARCHITECTURE, HYPERPARAMETERS, AND TRAINING DETAILS
C.1 SAMPLER
Refer for details about the architecture and hyperparameters of the learned samplers (Learned- P-64 and Learned-W-6e We attempted to train the sampler with both BigGAN and BigGAN-deep architectures and report on the models that achieved the best FID for each. Empirically, we found the models trained on the pixel-space much more unstable than on the wavelet-space, with four out of five models diverging.
C.2 NSB-GAN DECODERS
ESRGAN-W and ESRGAN-P A similar training procedure in ESRGAN is conducted for both of our decoders, ESRGAN-W and ESRGAN-P. First. SRResNet, with batch normalization removed, is trained with Jn loss: Ta = Er, [((x;) ylh, where is the down-sampled image and is the target image. After training with this I, loss for 150k iterations, the GAN is trained with perceptual and adversarial losses added. Therefore, the total loss for the generator becomes:
L Lepercep XLG nTI
14

---

Under review as a conference paper at ICLR 2021
(a) NSB-GAN- W with pre- me-trrined sampler
Figure 5: Class-conditional random samples at 512 x 512 from NSB-GAN-W model with pre- trained sampler. Classes from the top row: 12 house finch, 129 spoonbill, 200 Tibetan terrier, 323 monarch butterfly, and 493 chiffonier.
Perceptual loss is implemented with pre-trained VGG-19 model ImageNet and we use the features of the fourth layer before the fifth max-pooling layer. This GAN model then trained for another 150k iterations.
As in the training for the learned sampler in wavelet domain, because the wavelet-encoded input image does not lie in the standard [0, 1] range for ESRGAN-W, a normalization step is applied transform the range into [0. By using the minimum and maximum values of the wavelet- encoded input images, pre-calculated in the pre-processing step, we use the following normalization
[mminm where is the wavelet-encoded input and min, and maz, technique: Ztporm [maz,
the pre-calculated minimum and maximum values of the wavelet-encoded input images.
Both ESRGAN-W and ESRGAN-P share the architecture design, except for how the input is up-scaled and added to the leamed features in order to induce the residual learning paradigm. Refer Table for specific details of the architecture and hyperparameters.
C.3 UNET
UNet in Wavelet Domain After wavelet transform, the original image be deterministically recovered from the TL, TR, BL, and BR patches using IWT. NSB-GAN-W however, discards the high-frequency patches (TR, BI, and BR) during its encoding step, rendering a fully deterministic
15

---

Under review as a conference paper at ICLR 2021
(a) NSB-GAN W with pre-trained sampler
Figure 6: Additional class-conditional random samples 512 x 512 from NSB-GAN-W model with pre-trained sampler. Classes from the top row: 727 planetarium. 780 schooner, 853 thatch, 970 alp, and 992 agaric.
decoding impossible. To resolve this, the NSB-GAN decoder first learns to recover the missing high-frequency patches (using neural network) and then deterministically combines them using IWT to reconstruct the original imput.
Since NSB-GAN's encoding operation recursively applies wavelet transforms and discards the high- frequency components each of the L encoding levels, train L decoder networks to reconstruct the missing frequencies at each corresponding level of decoding. To parallelize the training of these decoder networks, perform multiple wavelet transforms to the original high-resolution dataset, generating L training sets of TL, TR, BL and BR patches. This allows us to independently train each of the decoder networks (in supervised manner) reconstruct the missing high-frequency patches conditioned on the corresponding TL. This parallelization boosts the convergence rate of NSB-GAN, allowing for fully trained decoder in under 48 hours.
Again with slight abuse of notation, let W1,1 be the TL patch level and IWT WT We can write the decoder for level as
W
f/. (W),)
])
Di(Wf, =IWT[ fa.. (Wi,)
IWT
fer. (W)
for 1515L. L.
16

---

Under review as a conference paper at ICLR 2021
(a) NSB-GAN-W W with pre-trained sampler
Figure 7: Class-conditional random samples at 512 x 512 from NSB-GAN-P model with pre-trained sampler. Classes from the top row: 12 house finch, 129 spoonbill, 200 Tibetan terrier, 323 monarch butterfly, and 493 chiffonier.
Here, each fl is deep neural network that is trained to reconstruct one of the TR, BL, or BR patches at level conditioned on the TL patch (W1,).
The main challenge in high-resolution image generation lies in overcoming the curse of dimension- ality. But by leveraging IWT, the original dimensionality of the image completely bypassed in the NSB-GAN decoder. Therefore, in practice, at each level, we further divide the TR, BL and BR patches by applying WT until we reach the patch dimensionality of 32x32. This be done irre- spective of the original dimensionality of the image. We then reconstruct these patches in parallel and recover the patches and the original image by recursively applying IWT. This ability of NSB- GAN to bypass the original dimensionality of the input separates it from all other SR methods that operate in the pixel space.
Refer to Figure 28a and Table for architecture details and hyperparameters of the UNet decoder for the wavelet-space.
Refer to Figure 30 for the training schematic with the UNet decoders in the wavelet-space.
Architecture We realize each of the L decoder neural networks fe, with a slightly modified ver- sion of the UNet architecture (Ronneberger et al. 2015) rather than the commonly used transposed- convolution based architecture. UNet is typically used for image segmentation and object detection tasks. As shown in Figure 28. UNet is an autoencoder architecture that has skip-connections from
17

---

Under review as a conference paper at ICLR 2021
(a) NSB-GAN-W W with pre-trained sampler
Figure 8: Additional class-conditional random samples at 512 512 from NSB-GAN-P model with pre-trained sampler. Classes from the top row: 12 house finch, 129 spoonbill, 200 Tibetan terrier, 323 monarch butterfly, and 493 chiffonier.
each encoding layer to its corresponding decoding layer. These skip connections copy and paste the encoding layer's output into the decoding layer, allowing the decoder to exclusively focus on recon- structing only the missing, residual information. This architectural design makes it compelling fit for decoding in NSB-GAN. We modify the UNet architecture by appending three shallow networks to its output with each one reconstructing one of the three high-frequency patches. This setup allows us to capture the dependencies between the capacity to capture to their individual differences.
patches while also allowing sufficient
UNet in Pixel Domain The UNet-P decoder is also partly-learned function that uses mod- ified UNet-based architecture. First, the encoded image is deterministically upsampled using an interpolation-based method. This leads to a low-quality, blurry image at the same size as the origi- nal image. We then train UNet to fill in the missing details in similar approach image super- resolution methods (Hu et al. 2019). Unlike the NSB-GAN decoder that circumvents the CoD by avoiding reconstructions in the original data dimensionality, the NSB-GAN-P decoder still has to operate on the full image size, resulting in a larger decoder.
Refer to Figure 28b and Table for architecture details and hyperparameters of the UNet decoder for the pixel-space.
18

---

Under review as a conference paper at ICLR 2021
Figure 9: Class-conditional random samples at 256 x 256 from BigGAN, Classes from the top row: 12 house finch, 129 spoonbill, 200 Tibetan terrier, 323 monarch butterfly, 493 chiffonier, 727 planetarium, 780 schooner, 853 thatch, 970 alp, and 992 agaric.
D NSB-GAN vs NSB-GAN-P INFORMATION CONTENT
As we show Figure 2 the information content the latent embedding NSB-GAN and NSB- GAN-P are drastically different. The TL patches from NSB-GAN preserve more structural infor- mation in the image than the down-sampled image from ASB-GAN-P. that pixel-based samm-sam method misses key structures and features, such the face of the person, structure of the boat, and structure of the accordion keyboard, whereas wavelet encoding does not.
evident the figure
19

---

Under review as a conference paper at ICLR 2021
Figure 10: Class-conditional random samples at 512 x 512. Classes from the top row: 12 house finch, 129 spoonbill, 200 Tibetan terrier, 323 monarch butterfly, and 493 chiffonier.
Learned-W-64
Learned-P-64
Best model type
BigGAN-deep
BigGAN
Batch size
512
512
Learning rate of generator
le-4
le-4
Learning rate of discriminator
4c-4
4e-4
Attention resolution
128
120
Dimension of random noise (z dim)
32
32
Number of resblocks per stage in
2
generatorldisrriiiaatr
Adam optimizer B1
0
Adam optimizer B3
0.999
0.999
Adam optimizer
1e-8
1e-8
Training iterations
250000
250000
Table 6: Hyperparameters of learned samplers (wavelet and pixel)
20

---

Under review as a conference paper at ICLR 2021
Figure 11: Additional class-conditional random samples at 512 x 512. Classes from the top row: 727 planetarium, 780 schooner, 853 thatch, 970 alp. and 992 agaric.
ESRGAN-W / ESRGAN-P
Input size
64
64
Batch size
32
Learning rate
le-4
Number of residual blocks
16
Cony filter size
Adam optimizer 31
0.9
Adam optimizer 32
0.99
L1 loss weight (1))
le-2
Lo loss weight (A)
5e-3
Training iterations
150000
Table 7: Hyperparameters of ESRGAN decoders
21

---

Under review as a conference paper at ICLR 2021
Figure 12: Full 256 x 256 resolution sample from NSB-GAN-W with pre-trained sampler.
Figure 13: Full 256 x 256 resolution sample from NSB-GAN-W with pre-trained sampler.
E EVALUATION ON LSUN CHURCH WITH STYLEGAN-2
To demonstrate that the NSB-GAN paradigm can be different samplers and that our decoder trained on ImageNet generalizes fairly well to an unseen dataset, we conduct a set of experiments with the BigGAN sampler replaced with StyleGAN-2 sampler. We test the StyleGAN-2 sampler with
22

---

Under review as a conference paper at ICLR 2021
Figure 14: Full 256 x 256 resolution sample from NSB-GAN-W with pre-trained sampler.
Figure 15: Full 256 x 256 resolution sample from NSB-GAN-W with pre-trained sampler.
two decoders: one trained on ImageNet and another trained on LSUN Church. Illustrated in 10 as expected, the decoder that was only trained the LSUN Church dataset outperforms decoder trained on ImageNet. However, even with our decoder, we reach a very competitive FID of 13.53, beating other recent two-step approaches, such (Liu al. 2019a).
23

---

Under review as a conference paper at ICLR 2021
Figure 16: Full 512 x 512 resolution sample from NSB-GAN-W with pre-trained sampler.
Level Decoder
Level Decoder
Input size
32 x 32
32 x 32
Batch size
128
64
Learning rate
le-4
le-4
Layers
16
16
Cony filter size
Number of added shallow networks
12
48
Adam optimizer 31
0.9
0.9
Adam optimizer 32
0.999
0.999
Adam optimizer e
1e-8
1c-8
Training iterations
288000
232000
Table Hyperparameters of UNet-W W decoders (level and level 2)
24

---

Under review as a conference paper at ICLR 2021
Figure 17: Full 512 x 512 resolution sample from NSB-GAN-W with pre-trained sampler.
Level Decoder
Level Decoder
Input size
64 64
128 128
Batch size
64
64
Learning rate
le-4
le-4
Layers
Cony filter size
Adam optimizer 31
0.9
0.9
Adam optimizer /2
0.999
0.999
Adam optimizer
1e-8
1e-8
Training iterations
615000
178000
Table 9: Hyperparameters of UNet-P decoders (level 1 and level 2)
25

---

Under review as a conference paper at ICLR 2021
Figure 18: Full 512 x 512 resolution sample from NSB-GAN-W with pre-trained sampler.
Sampler
Decoder
Resolution
FID
2retrained-256-64
ESRGAN-W
256
13.53
3.182
Pretrained-256-64
ESRGAN-W (Church)
256
7.886
2.784
Table 10:
The above pre-trained samplers are StyleGAN-2 model trained on LSUN Church dataset. Even our decoder trained only on ImageNet reaches competitive FII) of 13.53, but evidently can be improved by training a new decoder on LSSUN Church dataset. These results show that our NSB- GAN models can work with different samplers than BigGAN and that our decoder generalizes fairly well to unseen dataset. Therefore, we do not go about proving the compute efficiency this case.
SLICING
shows the difference between the slicing operations SPN and NSB-GAN. SPN slices the images across the spatial dimensions thus introducing long-tem dependencies. In contrast, NSB- GAN decomposes the image along different frequency bands that preserves the global structure of the image in each of the patches.
Figure 32b
26

---

Under review as a conference paper at ICLR 2021
Figure 19: Full 512 x 512 resolution sample from NSB-GAN-W with pre-trained sampler.
G WAVELET TRANSFORM.
Wavelet transformation of an image (illustrated in Figure I is two-step recursive process that splits the original image into four equal-sized patches, each representation of the entire image in different frequency bands. In the first step, low-pass filter (LPF) and high pass-filter (HPF) are applied to the original image. This produces two patches of the same size as the original image. Since the application of LPF and HPF leads to redundancy, we can apply Shannon-Nyquist theo- rem to downsample these patches by half without losing any information. In step two, the same process is repeated on the output of step one, splitting the original image into four equally- patches (TL, TR, BL and BR). TR, BL and BR contain increasingly higher frequencies of the input image, preserving horizontal, vertical and diagonal edge information, respectively (contributing to the sharpness of the image).
H NSB-GAN AT 512x512
At 512 x 512 resolution, our model outperforms the BigGAN model. We believe this to be the case mainly because of how the 512x512 training data is generated. The ImageNet dataset is na- tively at x56x256 (approximately). When training the BigGAN model to generate 512x512, an interpolation-basee method is used upsample ImageNet images 512x512, resulting in noise
27

---

Under review as a conference paper at ICLR 2021
Figure 20: Full 256 256 resolution sample from NSB-GAN-P with pre-trained sampler.
Figure 21: Full 256 x 256 resolution sample from NSB-GAN-P with pre-trained sampler.
and blurriness in the upsampled images. In comparison, method takes samples at 128x 128 and upsamples them using a learned SISR model. This leads to substantially sharper images and there- our model over end-to-end
fore better FID scores. In fact, based on study, it is better DGM when learning to generate samples beyond the native resolution of the dataset. To clearly demonstrate this difference, show samples from the bilinearly interpolated 512x512 ImageNet.
28

---

Under review as a conference paper at ICLR 2021
Figure 22: Full 256 256 resolution sample from NSB-GAN-P with pre-trained sampler.
Figure 23: Full 256 256 resolution sample from NSB-GAN-P with pre-trained sampler.
data and super-resolved version of the same samples with decoder in Figure 31 Clearly, inter- polated images are blurrier than super-resolved images. Since BigGAN is trained to generate this blurry data, compared to our NSB-GAN approach, it performs sub-optimally.
29

---

Under review as a conference paper at ICLR 2021
Figure 24: Full 512 x 512 resolution sample from NSB-GAN-P with pre-trained sampler.
NSB-GAN beats the BigGAN baseline model on FID at 512x512 resolution. We hypothesize that this difference in performance is due the fact that ImageNet dataset 512 x 512 is generated by using pixel-basee based interpolation up-sampling of the original data.
I OFFICIALLY PRE-TRAINED ESRGAN
Applying an officially pre-trained ESRGAN does not perform as well the ESRGAN-P and W models. Specifically at 512x512 resolution, it suffers slight increase in FID (10.73) and substantial decrease (>100) IS (52.25).
J SAMPLING TIME
Sampling from ASB-GAN takes 0.039 s image, whereas sampling from the BigGAN model takes 0.029 image, at 512x512 resolution. Therefore, the overhead of upsampling (ESRGAN) is relatively small (0.010 s image).
30

---

Under review as a conference paper at ICLR 2021
Figure 25: Full 512 x 512 resolution sample from NSB-GAN-P with pre-trained sampler.
K FINETUNING NSB-GAN- W USING PRE-TRAINED BIGGAN
Since the decoders are trained with real samples from the ImageNet dataset and tested on generated samples from our BigGAN models (learned and pre-trained), our up-sampled outputs could have visual artifacts due to the distribution mismatch. Therefore, we tried finetuning NSB-GAN-W end- to-end with pre-trained BigGAN samples at 256x resolution. We found end-to-end finetuning to increase high-frequency artifacts the up- sampled images and showcase few examples in Figure 32 Finetuning also results in worse FID and IS scores of 13.73 and 44.02, respectively. Note that finetuning our model end-to-end requires ground truth samples from an already trained BigGAN the target resolution. This somewhat defeats the purpose of approach, which tries to avoid training a BigGAN at the target resolution to begin with.
31

---

Under review as a conference paper at ICLR 2021
Figure 26: Full 512 x 512 resolution sample from NSB-GAN-P with pre-trained sampler.
32

---

Under review as a conference paper at ICLR 2021
Figure 27: Full 512 x 512 resolution sample from NSB-GAN-P with pre-trained sampler.
(a) NSB-GAN
(b) NSB-GAN-P
Figure 28: UNetd decoding architecture for NSB-GAN and NSB-GAN-P models.
33

---

Under review as a conference paper at ICLR 2021
WT
Slice
(a) SPN Slicing Operation
(b) NSB-GAN Slicing Operation
Figure 29: Unlike SPN, NSB-GAN slices the images in the frequency domain. As result each patch contains the entire global structure of the input image. This helps alleviate any long-term dependency issues.
leamed
Lostl
Levl
Ht
INT
Disarded
Inel
Iee 2
(a) Training
(b) Reconstruction
(c) Sampling
Figure 30: NSB-GAN schematic for (a) training, (b) reconstruction and sampling.
34

---

Under review as a conference paper at ICLR 2021
Figure 31: The left column consists of images from the ImageNet dataset at 512 x 512 and the right column consists of super-resolved version of the same images using our decoder (from 128 128 to 512 x 512. Clearly, the ImageNet dataset at 512 x 512 is blurrier than super-resolved images.
35

---

Under review as a conference paper at ICLR 2021
(a) Sample
(b) Sample
Figure 32: Generated samples at 256 x 256 after end-to-end finetuning of NSB-GAN- W with pre- trained BigGAN-256. After finetuning, high frequency noise (e.g., checkerboard-like patterns) can be observed in the images.
36


Reviews:
[{"review_id": "rVG0ALVHmb1", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary**\n\nThis paper proposes a cost-effective two-step training GAN framework (NSB-GAN). NSB-GAN contains a learned sampler in the wavelet domain, and a decoder to super-resolve images from the wavelet domain to the RGB space. Compared with the baseline BigGAN model their method reduces the cost of training and generates higher quality images at 512x512 resolution.\n\n**Strengths**\n- Contributions clearly stated and validated and the paper is clearly written and easy to understand.\n- Comprehensive experiments to show the effectiveness of their method. \n- Cost-effective than BigGAN and performance is good.\n\n**Weaknesses**\n- The idea of this work is not so novel. It seems low-resolution sampler + super-resolution decoder which is a straightforward idea and lacks novelty. And the two parts are trained independently. Why the authors didn't finetune the sampler and decoder networks together in the end so that to get a better sampler from a more correct sampling space.\n- For (x4) super-resolution networks, e.g., EDSRGAN, they often fail to generate a high-quality HR image with rich details when the input LR image is low quality. Therefore, what if the LR images from the learned sampler contain do not contains any desired patterns/textures?\n- For the NSB-GAN sampler, the authors use a batch size of 512, which is much smaller than that of BigGAN (2048). The authors should give an ablation study to analyze the effect of batch size on the proposed framework.\n- From Figure 2 in Appendix A, it seems that wavelet-based downsampling loses more structural information than pixel-based downsampling, which is not consistent with the description in the paper and violates the motivation of wavelet-based super-resolution as well. Please check it.\n- I can not understand the claim that \"NSB-GAN models reduce the training compute budget by up to four times\" in the caption of Table 2. Because Table 2 shows the compute of NSB-GAN is half of BigGAN, not 1/4.\n- I notice that NSB-GAN outperforms BigGAN at 512 resolution in terms of min FID, but it shows worse performance at 256 resolution. Which leads to these results? The authors are suggested to give some analysis on this problem.\n\n**Post rebuttal**\nI appreciate that the authors answer my questions. After reading through the rebuttal and other reviews,  I partly agree with R1's comments and I would like to downgrade my score by 1. My main concern is the novelty of this method. I disagree that the decoupling of generation from upsampling is interesting, it seems more like an engineering problem. Besides, I found that the authors changed the images in Figure 2 in Appendix A, not only changed the order, which is not consistent with their explanation in rebuttal. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "WYdzxYS1_Tt", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a two-stage generative image model: first, a GAN is trained to output low-resolution images, and another model to then perform single-image superresolution on the results of the first model. The claim is that the resulting model is slightly better than BigGAN-512 using half the compute requirements, in terms of FID. Two variants are described: one that generates in the wavelet decomposition domain (-W), and another that operates in pixel space (-P).\n\nThe idea of applying SISR to a GAN output seems potentially novel and useful, but as it stands, I find the paper convoluted and lacking a clear message. I cannot recommend acceptance at this point.\n\nThe main issues I see are the following:\n\n1. I suspect much of the image processing is performed incorrectly, casting doubt on the validity of the -P model and rendering the comparison to the wavelet case moot.\n2. Results are only reported in numeric form as FID and IS.\n3. The argumentation about the properties of the wavelet decomposition seem vague and without clear technical counterparts in the (well-developed) literature on wavelets and image processing.\n4. Simultaneous lack of details and overall verbosity; I find it difficult to find the big picture from this paper even after hours of trying.\n\n\n1. I believe the bilinear downsampling, the basis of the -P variant, is implemented incorrectly. This is visible as clear aliasing in the supplemental Figure 2, rightmost “pixel-space” column. To verify, I extracted the dog and sailboat images from the PDF and applied bilinear downsampling in Matlab – which uses proper pre-filtering before downsampling to remove aliases, unlike for instance the bilinear grid_sample operation in PyTorch – and get a significantly different result; one that does not have the signature aliasing artifacts that remain in the images shown. If, on the other hand, I explicitly turn off antialiasing, the result quite closely matches the right-hand column. To be clear, this a rudimentary mistake in image processing (which is surprisingly prevalent in the ML and vision literature).\n\nThis makes me suspect all results of the -P variants are not to be trusted: teaching a GAN to generate aliased images, and then another model to up-res those aliased images, seems like a task that is fundamentally harder than if the aliasing wasn’t there. Hence the worse results are not unexpected.\n\nIn particular: I find the conclusions drawn from the results in Table 1 all potentially invalid.  On the other hand, using pretrained samplers, the pixel space versions appear to actually do a little *better* (in terms of FID) than the wavelet ones in Table 2. Comparing the results in the appendix does not appear to reveal large differences.\n\n\n2. It is well known that metrics like FID and IS do not capture the notion of the quality of a GAN well. They are useful in drawing a picture of how the model performs. While some metrics that better correlate with result quality are known, a satisfactory one hardly exists, so visual inspection and analysis of the results cannot be skipped. I do not approve of pushing them to the appendix.\n\n3. Example: \"The functional prior imposed by our deterministic encoder leads to a highly structured representation space made up of low frequency TL patches of images.” What does this mean, precisely? The repeated application of the wavelet approximation coefficient filter followed by decimation by 2 is equivalent to a particular linear downsampling operation applied to the original image; a poor one at that, because the kinds of critically sampled wavelets employed here are known for their aliasing issues (which has long ago led to a preference of using overcomplete bases). Similar language about the “structuredness” of the wavelet representation can be found near Figure 2, where the pixel-space comparison is, I believe, incorrect, as I detail above.\n\n4. What precisely is going on with Equation 2? IWT(…) would appear to be a reconstruction operation that combines the approximation and detail coefficients into an image of 2x the size, in pixel space; then addition of f(W^l_1,1) seems to add hallucinated detail on top. Does f do anything random or is it deterministic? And more pressingly, how is this actually different from the pixel-space version..?\n\nI do not understand the paragraph 3.1.2 “U-Net decoder”. Why “does [it] not take full advantage of the compression that wavelet space modeling brings about”? \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "JeStZAqlEsY", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new framework NSB-GAN for high-resolution natural image generation with a low computation budget. They first utilize the BigGAN (or StyleGANv2) to generate a low-resolution image, and the wavelet- or pixel-based SR network decodes the intermediate image to the desire resolution. With this simple two-step approach, they successfully train the model on the limited resources where vanilla BigGAN cannot be trained. In addition, NSB-GAN outperforms the BigGAN on high-resolution (>512x512) maybe due to the instability of the BigGAN at extremely high resolution.\n\nStrengths:\n+ The motivation of compute-efficient deep generative models for large images is a very important issue, but not many methods have taken this into account. The proposed method tackles this issue in a simple but effective manner.\n+ Since the proposed approach does not modify the network architecture, it is flexible, allowing any model compression techniques can be adapted accordingly.\n\nWeaknesses:\n- Lack of some critical comparisons: 1) Does the proposed method also outperform the approach that first generates using a pre-trained BigGAN-256 then upsamples using an officially pre-trained ESRGAN? 2) What about the inference time (or complexity) of NSB-GAN compared to BigGAN?\n- If the decoders are trained with real samples only (drawn from the imagenet dataset), the upsampled outputs may have visual artifacts due to the mismatched distribution between the train (real) and test (fake gen from sampler) images. For example, the images of the cabinet (Fig3, 4, 5th row) have overly sharpened artifacts that BigGAN does not suffer.\n\nOverall, the suggested work effectively decreases the training time of the BigGAN using a simple idea. However, there are missing comparisons and analyses such as 1) comparison with pre-trained BigGAN -> ESRGAN 2) How the capacity of the SR model affects the FID. And lastly, since the proposed method is pipelining, there are some unexpected artifacts.\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers brought up significant concerns that were not resolved by the authors' responses. The concerns are too significant for the paper to be accepted at this time.
Decision: Reject