Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a novel single image generation framework that is able to generalize to multi-sample case, which is different from previous work where each model only works on a single sample, leveraging hyper network. The author has provided solid theoretical analysis to support their design. And the effectiveness is also validated by their produced results.
main_review: Pros:
1. The paper is well written and easy to follow.
2. The author has provided solid theoretical analysis of the proposed method.
3. The idea of leverage hyper-net to generate the model weights for generator and discriminator is quite novel and intuitive.
4. The paper has made a meaningful progress to generalize the single-sample GAN models to multi-sample case.
5. A collection of variants (shared gen/dis hyper) are also studied to validate the effectiveness of the proposed method.
6. The author has also provided code for reference.

Cons:
1. The author has pointed out the model size might be one of the limitations. I would like to know if there is any failure case existing w.r.t the the generation quality under some circumstances? For example, does the proposed method also works for human face images, etc?
limitations_and_societal_impact: This technique might have a negative societal impact. Some people might be able to leverage this method to produce some fake images based on a given photo. The produced fake images might be harmful.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper aims at improving single-image GAN models by training them on larger datasets. The authors use a meta-learning approach based on the hypernetworks scheme. The model consists of a hyper-generator and a hyper-discriminator. The Hyper-generator network is decomposed into an embedding network that is shared among scales and a linear projection per scale. The hyper-discriminator network is also decomposed into an embedding network and a set of projections. The model is trained with adversarial and reconstruction loss functions. Authors show that the proposed model reduce the training time per image without loss in performance. They also introduce novel
capabilities, such as interpolation and feedforward modeling of novel images. 
main_review: **Strengths**:


- The paper is well-written, and the proposed approach is simple and intuitive. The authors motivate the problem well. 


- Experiments are thorough and demonstrate efficacy of the proposed approach qualitatively and quantitatively. The authors also define new tasks such as interpolation and feedforward modeling which enables new capacities in single-image GAN models. 


**Weaknesses**:


- The proposed approach employs hyper-networks in SinGAN models, which is an instance of apply X to Y. Considering that hyper-networks are applied to other generative models and various discriminative models, this makes novelty of the proposed approach moderate. There are existing works such as [A] that use hyper-networks in generative models. There are also recent works such as [B, C, D] which also use hyper-networks for generation (but are not published by the submission deadline for this conference). 
However, the good results and improvements of the proposed approach can make up for this.    


- The authors note the limitation of their work in terms of model size, convergence, runtime and GPU memory usage in section 6, but do not provide further details. It would be good to have comparisons with SinGAN on this.  


[A] HCNAF: Hyper-Conditioned Neural Autoregressive Flow and its Application for Probabilistic Occupancy Map Forecasting; Oh et al.; CVPR 2020


[B] Adversarial Generation of Continuous Images; Skorokhodov et al.; CVPR 2021


[C] pi-GAN: Periodic Implicit Generative Adversarial Networks for 3D-Aware Image Synthesis; Chan et al.; CVPR 2021


[D] HyperVAE: A minimum description length variational hyper-encoding network; Nguyen et al.; 2020

------------------------------------------------------------------------------------------------------------------------------------------------

Update after the rebuttal: I read the rebuttal and other reviews. Overall, I think the paper meets the acceptance criteria for the conference. I keep my score, and I hope the authors clarify concerns of the reviewers in the final version of the paper.  

------------------------------------------------------------------------------------------------------------------------------------------------
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a method to learn hypernetworks, trained on a larger dataset (500-5000 images), for single-image GANs. These hypernetworks are then used to initialize the GAN for a specific single image to speed-up training time and improve image quality/diversity. The single-image GAN follows the same architecture as SinGAN and the individual generators and discriminators for a specific image are initialized by the trained hypernetwork and then fine-tuned for the specific image. The resulting single-image GANs outperform related work (SinGAN, ConSinGAN, ...) based on SIFID, diversity, and training time.
main_review: The idea of using a larger dataset to speed up the training of single-image GANs on a single image is promising. Using hypernetworks and meta-learning may be one suitable approach for this. The method in this paper seems to deliver promising results while only needing 5 minutes of fine-tuning for a given image. 

It would be more interesting to see how well this approach performs on more complex images and how much it benefits from large-scale training in this case. SinGAN-like approaches are not good at modeling complex structures/objects and it seems to me that using hypernetworks would be most helpful in these cases. Landscapes do not contain many complex structures so it would be better to test this approach on other datasets, e.g. LSUN churches or something similar.

Also, is it necessary to train an image encoder as you do given that there are many pre-trained image encoders already available? Have you tried using a pretrained (frozen) image encoder as conditioning for your hypernetworks? 

While I get that your approach supports applications that are not supported by traditional single-image GANs (e.g. image interpolation and mixing) I think it would still be beneficial to compare against at least some of the original applications directly in the main paper (unconditional synthesis, harmonization, editing, retargeting, ...). At the moment this is all hidden in the supplementary which makes it difficult to directly qualitatively compare your approach to the baselines.

I believe some (most) of section 2 (2.1 can be summarized in one paragraph like 2.2) can be moved to the supplemental to free up more space to run more experiments and evaluations. Similarlay, section 3 can be shortened (e.g. equations 9-11 are a repetition of equations 6-8).

Overall, I believe this paper would benefit from a more thorough analysis of the approach on different datasets (that also contain more complex structures than landscapes), more direct comparisons to the baselines on some applications (generation, harmonization, editing, ...), and possibly running a user study to get direct comparisons between the different models besides SIFID/diversity (both of which have been shown to have weaknesses).

***
Update: Raised score to 6, see comment.
***
limitations_and_societal_impact: The authors address limitations and societal impacts.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces an advanced version of SinGAN by incorporating the hypernetwork into the framework so that the proposed model can perform all the tasks that the original SinGAN can do, but with much faster test inference. 
main_review: The idea of employing hypernetwork to train both generators and discriminators is very interesting, and the motivation of fast inference is convincing enough for me. 


My main concern is on the quality of the results (Figure 3 and 4). I partly agree (but with some conditions) that if there is no shortage of unlabeled images, single-image GAN is unjustifiable. This must go with conditions that the test image performance is better than or at least comparable to the original SinGAN. 


Zooming at Figure 4, however, the quality of the results does not seem to match with SinGAN (even on train data with the model trained on the larger dataset!). For example, the V5000 results does not look crisp nor realistic when looking at the boundaries of the buildings. These qualitative results do not match with the quantitative results shown in Table 1 and Table 3. This needs more explanations.


In Table 3, using a smaller dataset gives better results on training data, and this trend is opposite in the test data. What would be the explanation for the former case? (+ why doesn’t this go along with the qualitative results in Figure 4?)


Speaking of the results, I don’t see the benefit of the availability to interpolate between single image GANs (Figure 3). I understand that the framework allows this to perform and indeed this is new, but cannot see what kinds of advantage it would bring by doing so. This needs more discussion, or reduce the weight it takes in the paper (Currently, it is mentioned with the other major contributions).


In addition, one of the contributions that the authors argue is that they are the first (as far as they know) to perform adversarial training with hypernetworks. In fact, there exists a very similar work that did use hypernetwork for generator and train it using the adversarial loss: “Generative Models as Distributions of Functions”. Although it does not use the hypernetwork scheme for both generator and discriminator, this is noteworthy. However, I do not think this compromises the novelty much since the work is quite recent and has not been published anywhere yet. 


The organization of the paper can be also improved. For example, in my personal opinion, it looks better to spare more space in Backgound section for introducing the intuition behind the SinGAN and its characteristics than providing the technical details (like eq. (2), (3), and (4)), which are not important to understand the main idea and context. 


Overall, the idea is interesting and the advantages of using the framework are clearly seen. However, since the results do not go along with the first two benefits (Line 24 – Line 27), which are very critical, it is hard to give a strong score to the paper. 

---

Update: Raised score to 6, please see the comment.

---

limitations_and_societal_impact: No comment
review_title: null

Meta-review: Meta-review of "Meta Internal Learning"

This paper proposes a framework for single image generation from the perspective of meta-learning on larger datasets. The method uses hypernetworks as the generator and discriminator so that the weights of these networks can be conditionally adapted from the single image from a set of projections, and can be trained with adversarial and reconstruction losses. They discuss capabilities of the method such as interpolation and modeling of novel images.

Most reviewers agree that this paper is well written, easy to follow, includes solid analysis of the proposed methods, and also generally agree with the novelty. The fast inference can find important uses in real-world applications. The weaknesses are highlighted by the authors and discussed in detail with reviewers, and the authors have included several comparisons to previous work. As suggested by reviewer FnNF (who had increased their score), the authors are strongly encouraged to better reorganize the manuscript in the camera ready version, in which the authors agreed to do so.

For all these reasons stated, I recommend acceptance of this work.

Decision: Accept (Poster)