Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a novel framework, InfinityGAN, which could generate arbitrary-sized images with infinite pixels.

By disentangling global appearance, local structures and textures, InfinityGAN generates local and global consistent large images with a seamless patch-by-patch paradigm. 

Experiments validate its superiority compared to some other baselines, and several applications are raised based on their approaches.
main_review: Strength:
1) The idea to generate infinity-pixel images in a seamless patch-by-patch manner is quite novel.
2) This approach considers both global and local consistency in generating large images. Though straightforward, the designed framework based on it is effective and elegant.
3) The experiments and the appendix are comprehensive to reproduce this work.

Weakness:
No obvious weakness but there are some issues.
1) It is better to add the symbols in Sec 3.2 & 3.3 to Figure 2, e.g., $G_s$, $Z_s$, $G_T$, $p_c$.
2) Although the concurrent work ALIS (SKOROKHODOV ET AL., 2021) has been discussed in appendix A, it is suggested to make a fair comparison in Table 1 with the same metrics in the future version, since ALIS address nearly the same task to generate the infinity-pixel images.

summary_of_the_review: Based on its strength and weakness, I think it is a good and quite novel paper considering both its framework and applications. I recommend accepting this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed a new model called InfinityGAN, for generating images of arbitrary sizes. These images are generated holistically: the authors proposed using a structure synthesizer first to generate a latent structural representation. Then a modified StyleGAN based texture synthesizer generates the image patches at the corresponding locations. The positional encoding and PFG enable generating patches in an arbitrary large image and aggregating these patches seamlessly. The experiments on the Flickr-Landscape dataset show promising large image synthesis performance. On the Place365 and Flickr-Scenery datasets, the experiment results show superior image outpainting performance. The authors also demonstrated other applications including spatial style fusion and image inbetweening.
main_review: I think this is a very solid paper. The writing is good and the proposed model seems novel.  The task is very interesting and the authors addressed several challenges in very smart ways. For example, I think generating two patches that can be combined seamlessly is hard. The authors proposed a modification to StyleGAN to eliminate zero padding. Position information is further added to generate plausible results.
The experiments are sufficient and comprehensive. A lot of details are provided in the supplementary material. Applications including spatial style fusion, multimodal outpainting, and image inbetweening are also impressive.
 

The major concern I have is about the problem formulation. The whole image generation pipeline involves 3 random variables as input: $z_g$, $z_l$, and $z_n$. $z_n$ is used as noise input in StyleGAN. The structural latent variable $z_S$ is generated from $G_S(z_g, z_l, c)$. However, $z_S$ is only used in $G_T$, while there is another input $z_g$ in G_T. I was wondering if the $z_g$ input in $G_S$ is necessary? As shown by Fig 6, $z_l$ determines the layout while $z_g$ in $G_T$ for style, I did not see the role of $z_g$ in $G_S$. In other words, what if we use different $z_g$ in $G_S$ and $G_T$? Which $z_g$ will determine the style?
 
Here are some minor issues that will not affect my rating on this paper. But I would appreciate it if the authors could share more insight:

1 In the previous work section, I wondered if the authors also want to include autoregressive models like PixelCNN. These models can generate images of arbitrary sizes or shapes by nature. The authors did cite the VQVAE2 paper and mentioned it could only generate images of fixed sizes.
 
2 I found the visual result on the Flickr-Landscape data is better than that on the LSUN bridge or tower datasets. For example, Fig 27 top left shows repetitive tower patterns. Is this limited by the proposed pipeline, or the datasets, or StyleGAN?
summary_of_the_review: I did not find big issues with this submission. It is a very good paper, and I think the proposed techniques can benefit other large image generation research. I am only concerned about the $z_g$ in both $G_S$ and $G_T$, which I think might be redundant. Therefore I recommend accepting this submission.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a framework to generate high resolution images of outdoor scenes with any desired style. The approach extends image synthesis techniques and proposes a system that uses both global and local information. This ensures that the model can be trained using low resolution patches (e.g. 101 x 101) and, at inference time, it can be used to generate high resolution renderings: images up to 4096 x 4096 are showed.

The combination of the global and local appearance, coupled with interesting modules such as the padding-free generator, ensure high quality results that are consistently outperforming other approaches.
main_review: + The paper advances the state of art in image synthesis and there are multiple contributions that could be re-used in other contexts. For example, I really liked the intuition behind the padding-free generator and how the authors showed its effectiveness in both quantitative and qualitative results.
+ The paper is well written and justified. All the technical details are well elaborated and additional supplementary material provides all the details needed to ensure reproducibility. The contributions seem to be well placed with respect to the state of art
+ The experimental section is impressive, plenty of examples showing the effectiveness of the approach. The supplementary materials show multiple comparisons and visuals as well as implementation details for the spatial style fusion.

There are a couple, mostly minor weak points, that I hope could help the authors to make the paper even stronger:
- Evaluation on temporal consistency: it would be great to assess how the method performs on temporal sequences and assess how stable/consistent it is. I understand that the method was not designed for this purpose, but could still provide a baseline for future research.
- Ablation study: there are lots of interesting contributions here, from architecture specific modules to loss functions. I would suggest to elaborate more the ablation study section, which mostly focuses on the padding-free generator at the moment.
summary_of_the_review: This paper is a solid submission, that really pushes the state of art in image synthesis for outdoor scenes. The problem is well justified and placed with respect to related work and the contributions are well described. The results prove that the method is outperforming baselines with comparisons on both quantitative and qualitative results. Given the complexity of the framework, I would have appreciated a more in-depth analysis (i.e. ablation study) of the various components to help other researchers to reproduce the results and focus on the right modules, however I acknowledge that authors did their best with experiments and evaluations: the supplementary material is actually impressive. 

I would suggest authors to also show some results on temporal sequences, to assess temporal consistency across frames and/or long sequences. This would also help to establish a baseline for future work in this area.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a novel framework, InfinityGAN, for infinitely-wide landscape image generation, combining recent advances on style-based generative adversarial networks and implicit neural representations. Leveraging parallelizable inference, patch-based generative modelling and hand-crafted landscape image priors, the proposed framework can synthesize landscape images of seemingly infinite width and a large height. The paper further proposes a new evaluation metric, ScaleInv FID, and a set of applications for their framework, including image outpainting and inbetweening, and spatial style fusion. The results of InfinityGAN are validated using user studies and some image-space metrics. 
main_review: This paper proposes a novel framework, InfinityGAN, for infinitely-wide landscape image generation, combining recent advances on style-based generative adversarial networks and implicit neural representations. Leveraging parallelizable inference, patch-based generative modelling and hand-crafted landscape image priors, the proposed framework can synthesize landscape images of seemingly infinite width and a large height. The paper further proposes a new evaluation metric, ScaleInv FID, and a set of applications for their framework, including image outpainting and inbetweening, and spatial style fusion. The results of InfinityGAN are validated using user studies and some image-space metrics. 

The paper proposes very interesting ideas on image generation, including a padding-free StyleGAN architecture, feature unfolding and neural implicit representations for adding variation to the generated image contents, as well as hand-crafted priors on landscape image generation, including positional encodings and disentanglements between global and local styles. The paper is well-written and presented and is extended with a very valuable supplementary material. It includes sufficient details on implementation for allowing reproducibility. The proposed applications could be impactful in the image generation literature. 

Overall, this paper shows potential for somewhat significant algorithmic contributions for image generative models, but it falls short on important aspects that need to be adressed before it can be recommended for acceptance. First, several of the claims of the paper should be toned down. The paper argues it proposes a generic framework for infinite image synthesis, but the hand-crafted priors embedded in the method and the datasets used for validation are very specific for landscape photography, making it more limited in its scope than the paper claims. Second, many decisions taken throughout the design of the framework are not well justified or validated empirically, making it hard to understand what the actual contributions of the paper really are.  Finally, this paper misses important references and comparisons to prior work, the comparisons that are present in the paper are somewhat arbitrary and the proposed evaluation metric lacks implementation details and justification. 

*Detailed review* 

- This paper should tone down its claims of generality. The paper claims to present "a novel framework for arbitrary-sized image generation". However, the design of the method and the datasets used for evaluating it are limited to landscape photography, which shows particularities that InfinityGAN exploits in clever ways but limits its generality. In particular, the hand-crafted prior of self-similarity on the horizontal axis but rapid saturation on the vertical axis only holds for this particular image domain. Therefore, it is unlikely to work in other domains where arbitrary-sized image generation is important, including texture synthesis or satellite photography. This limitation is implicitly acknowledged on the conclusions section, but is not properly discussed throughout the paper.

- Several important references are missing on the related work. In the context of neural implicit representations, relevant recent advances should be included and discussed (eg Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains, Tancik et al, NeurIPS 2020; Implicit Neural Representations with Periodic Activation Functions, Sitzmann et al, NeurIPS 2020). 

- Important comparisons with previous work on image generation are missing: TileGAN: Synthesis of Large-Scale Non-Homogeneous 
Textures, Frühstück et al, SIGGRAPH 2019; Learning Texture Manifolds with the Periodic Spatial GAN,  Bergmann et al, ICML 2017;  Improved Techniques for Training Single-Image GANs, Hinz et al, WACV 2021. Concurrent work that could be mentioned includes Taming Transformers for High-Resolution Image Synthesis, Esser et al, CVPR 2021. Finally, the paper proposes an inbetweening application, which should be better contextualized on the image inpainting literature. 

- Some parts of the design of the framework need to be better empirically demonstrated. 
1) What will happen if the self-similarity prior is introduced on both axes? 
2) What is the impact of the period T of the positional encoding? 
3) What is the impact of the mode-seeking diversity loss introduced to the generator training? 
4) Why is the auxiliary task of the generator needed? What happens if it is not done? 
5) What will happen when a non-landscape dataset is used? 

- With regards to Section 4:
1) The proposed ScaleInv FID seems rather arbitrary and its implementation is not clear. Could the authors elaborate further on this metric? From the current level of detail about this metric, it is quite hard to draw conclusions from it.
2) The comparison with SinGAN is rather unfair to their method. The paper could include more implementation details on how they implement this comparison. It also remains unclear if the improved results of InfinityGAN are due to its inductive biases (hand crafted priors, neural implicit representations, etc), or simply because InfinityGAN is trained on tens of thousands of images, whilst SinGAN is trained on a single image.
3) The user study needs further explanation. What are the demographics of this user set? Which images were they presented with (eg what dataset). What were the questions they were asked?
4) Why is the inception score (IS) used on Table 2 but not on Table 1? Could the authors extend these analyses with other metrics, including SSIM or LPIPS?

In terms of results, on the Supplementary Material, it is clear that InfinityGAN struggles to add variation to the uppermost parts of the images (eg Figures 20 and 21). This is likely due to the tanh() operation done to the vertical coordinates, which limits the variation on this axis compared to the horizontal, positonal-encoded, axis. Could the authors elaborate on this and maybe show more examples on vertical expansion?

I have other minor comments:
- Was any type of data augmentation considered, rather than random cropping? This could alleviate the need for such massive datasets. (Training Generative Adversarial Networks with Limited Data, Karras et al 2020)
- It would be really interesting to see spatial fusion and outpainting applications in the LSUN dataset and other architectural datasets. Did the authors consider this?
- Could this method be extended for 360º photography for VR applications? For example, by making the positional encoding periodic for both axes? It would be great if authors could ellaborate on this.


** Update after rebuttal period** 


The authors have successfully adressed many of my concerns during the rebuttal period, including new extensive comparisons with other methods, ablation studies, discussion of limitations and clarifications on the implementation of the Scale-Invariant FID metric. I am overall more positive about this submission and thereby recommend it for acceptance, as this paper proposes novel and well-evaluated ideas, despite some of the claims being somewhat oversold. The responses to concerns by other reviewers were also convincing. 


summary_of_the_review: This paper proposes very interesting ideas on image generation, including a padding-free StyleGAN architecture, feature unfolding and neural implicit representations for adding variation to the generated image contents, as well as hand-crafted priors on landscape image generation, including positional encodings and disentanglements between global and local styles. It includes sufficient details on implementation for allowing reproducibility. The proposed applications could be impactful in the image generation literature. 

Overall, even though this paper shows potential for somewhat significant algorithmic contributions for image generative models, it falls short on important aspects that need to be adressed before it can be recommended for acceptance. First, several of the claims of the paper should be toned down. The paper argues it proposes a generic framework for infinite image synthesis, but the hand-crafted priors embedded in the method and the datasets used for validation are very specific for landscape photography, making it much more limited in its scope than the paper claims to be. Second, many decisions taken throughout the design of the framework are not well justified or validated empirically, making it hard to understand what the actual contributions of the paper really are.  Finally, this paper misses important references and comparisons to prior work, the comparisons that are present in the paper are somewhat arbitrary and the proposed evaluation metric lacks implementation details and justification. 

** Update after rebuttal period** 
The authors have adressed many of my concerns during the rebuttal period, including new extensive comparisons with other methods, ablation studies and discussion of limitations. I am overall more positive about this submission and thereby recommend it for acceptance.
review_title: null

Meta-review: 3 reviewers recommend accept, 1 rates the paper marginally above acceptance. The authors provided satisfactory answers to criticism -- all in all this is a paper worth accepting at ICLR. Please make sure that criticism in the reviews is adequately addressed in the final version, e.g. include various experimental results in the rebuttal, add the symbols in sec 3.2 & 3.3 to fig. 2, add a related discussion on ablations when the model is fully trained, etc.
Decision: Accept (Poster)