Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 2: Strong rejection
review_content: The paper describes a two-stage generative image model: first, a GAN is trained to output low-resolution images, and another model to then perform single-image superresolution on the results of the first model. The claim is that the resulting model is slightly better than BigGAN-512 using half the compute requirements, in terms of FID. Two variants are described: one that generates in the wavelet decomposition domain (-W), and another that operates in pixel space (-P).

The idea of applying SISR to a GAN output seems potentially novel and useful, but as it stands, I find the paper convoluted and lacking a clear message. I cannot recommend acceptance at this point.

The main issues I see are the following:

1. I suspect much of the image processing is performed incorrectly, casting doubt on the validity of the -P model and rendering the comparison to the wavelet case moot.
2. Results are only reported in numeric form as FID and IS.
3. The argumentation about the properties of the wavelet decomposition seem vague and without clear technical counterparts in the (well-developed) literature on wavelets and image processing.
4. Simultaneous lack of details and overall verbosity; I find it difficult to find the big picture from this paper even after hours of trying.


1. I believe the bilinear downsampling, the basis of the -P variant, is implemented incorrectly. This is visible as clear aliasing in the supplemental Figure 2, rightmost “pixel-space” column. To verify, I extracted the dog and sailboat images from the PDF and applied bilinear downsampling in Matlab – which uses proper pre-filtering before downsampling to remove aliases, unlike for instance the bilinear grid_sample operation in PyTorch – and get a significantly different result; one that does not have the signature aliasing artifacts that remain in the images shown. If, on the other hand, I explicitly turn off antialiasing, the result quite closely matches the right-hand column. To be clear, this a rudimentary mistake in image processing (which is surprisingly prevalent in the ML and vision literature).

This makes me suspect all results of the -P variants are not to be trusted: teaching a GAN to generate aliased images, and then another model to up-res those aliased images, seems like a task that is fundamentally harder than if the aliasing wasn’t there. Hence the worse results are not unexpected.

In particular: I find the conclusions drawn from the results in Table 1 all potentially invalid.  On the other hand, using pretrained samplers, the pixel space versions appear to actually do a little *better* (in terms of FID) than the wavelet ones in Table 2. Comparing the results in the appendix does not appear to reveal large differences.


2. It is well known that metrics like FID and IS do not capture the notion of the quality of a GAN well. They are useful in drawing a picture of how the model performs. While some metrics that better correlate with result quality are known, a satisfactory one hardly exists, so visual inspection and analysis of the results cannot be skipped. I do not approve of pushing them to the appendix.

3. Example: "The functional prior imposed by our deterministic encoder leads to a highly structured representation space made up of low frequency TL patches of images.” What does this mean, precisely? The repeated application of the wavelet approximation coefficient filter followed by decimation by 2 is equivalent to a particular linear downsampling operation applied to the original image; a poor one at that, because the kinds of critically sampled wavelets employed here are known for their aliasing issues (which has long ago led to a preference of using overcomplete bases). Similar language about the “structuredness” of the wavelet representation can be found near Figure 2, where the pixel-space comparison is, I believe, incorrect, as I detail above.

4. What precisely is going on with Equation 2? IWT(…) would appear to be a reconstruction operation that combines the approximation and detail coefficients into an image of 2x the size, in pixel space; then addition of f(W^l_1,1) seems to add hallucinated detail on top. Does f do anything random or is it deterministic? And more pressingly, how is this actually different from the pixel-space version..?

I do not understand the paragraph 3.1.2 “U-Net decoder”. Why “does [it] not take full advantage of the compression that wavelet space modeling brings about”? 


review_title: Neat idea, but technical flaws cast doubt on conclusions

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a new framework NSB-GAN for high-resolution natural image generation with a low computation budget. They first utilize the BigGAN (or StyleGANv2) to generate a low-resolution image, and the wavelet- or pixel-based SR network decodes the intermediate image to the desire resolution. With this simple two-step approach, they successfully train the model on the limited resources where vanilla BigGAN cannot be trained. In addition, NSB-GAN outperforms the BigGAN on high-resolution (>512x512) maybe due to the instability of the BigGAN at extremely high resolution.

Strengths:
+ The motivation of compute-efficient deep generative models for large images is a very important issue, but not many methods have taken this into account. The proposed method tackles this issue in a simple but effective manner.
+ Since the proposed approach does not modify the network architecture, it is flexible, allowing any model compression techniques can be adapted accordingly.

Weaknesses:
- Lack of some critical comparisons: 1) Does the proposed method also outperform the approach that first generates using a pre-trained BigGAN-256 then upsamples using an officially pre-trained ESRGAN? 2) What about the inference time (or complexity) of NSB-GAN compared to BigGAN?
- If the decoders are trained with real samples only (drawn from the imagenet dataset), the upsampled outputs may have visual artifacts due to the mismatched distribution between the train (real) and test (fake gen from sampler) images. For example, the images of the cabinet (Fig3, 4, 5th row) have overly sharpened artifacts that BigGAN does not suffer.

Overall, the suggested work effectively decreases the training time of the BigGAN using a simple idea. However, there are missing comparisons and analyses such as 1) comparison with pre-trained BigGAN -> ESRGAN 2) How the capacity of the SR model affects the FID. And lastly, since the proposed method is pipelining, there are some unexpected artifacts.



review_title: compute-efficient deep generative models for large images

final_score: rating: 5: Marginally below acceptance threshold
review_content: **Summary**

This paper proposes a cost-effective two-step training GAN framework (NSB-GAN). NSB-GAN contains a learned sampler in the wavelet domain, and a decoder to super-resolve images from the wavelet domain to the RGB space. Compared with the baseline BigGAN model their method reduces the cost of training and generates higher quality images at 512x512 resolution.

**Strengths**
- Contributions clearly stated and validated and the paper is clearly written and easy to understand.
- Comprehensive experiments to show the effectiveness of their method. 
- Cost-effective than BigGAN and performance is good.

**Weaknesses**
- The idea of this work is not so novel. It seems low-resolution sampler + super-resolution decoder which is a straightforward idea and lacks novelty. And the two parts are trained independently. Why the authors didn't finetune the sampler and decoder networks together in the end so that to get a better sampler from a more correct sampling space.
- For (x4) super-resolution networks, e.g., EDSRGAN, they often fail to generate a high-quality HR image with rich details when the input LR image is low quality. Therefore, what if the LR images from the learned sampler contain do not contains any desired patterns/textures?
- For the NSB-GAN sampler, the authors use a batch size of 512, which is much smaller than that of BigGAN (2048). The authors should give an ablation study to analyze the effect of batch size on the proposed framework.
- From Figure 2 in Appendix A, it seems that wavelet-based downsampling loses more structural information than pixel-based downsampling, which is not consistent with the description in the paper and violates the motivation of wavelet-based super-resolution as well. Please check it.
- I can not understand the claim that "NSB-GAN models reduce the training compute budget by up to four times" in the caption of Table 2. Because Table 2 shows the compute of NSB-GAN is half of BigGAN, not 1/4.
- I notice that NSB-GAN outperforms BigGAN at 512 resolution in terms of min FID, but it shows worse performance at 256 resolution. Which leads to these results? The authors are suggested to give some analysis on this problem.

**Post rebuttal**
I appreciate that the authors answer my questions. After reading through the rebuttal and other reviews,  I partly agree with R1's comments and I would like to downgrade my score by 1. My main concern is the novelty of this method. I disagree that the decoupling of generation from upsampling is interesting, it seems more like an engineering problem. Besides, I found that the authors changed the images in Figure 2 in Appendix A, not only changed the order, which is not consistent with their explanation in rebuttal. 
review_title: A cost-effective two-step GAN framework

Meta-review: The reviewers brought up significant concerns that were not resolved by the authors' responses. The concerns are too significant for the paper to be accepted at this time.
Decision: Reject