Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper focuses on visual synthesis, especially for high-resolution cases. To model global dependencies between patches, 
this work proposes a patch-level render-and-optimize strategy, i.e., once a patch is predicted, its hidden states are saved as contexts for the next round of prediction. In addition, an arbitrary direction relative position embedding method is proposed, which shows promising performance on free-direction visual extension. The experimental results show the good performance of the proposed method on visual synthesis.  
strengths_and_weaknesses: **Strengths**

+Clear Motivation
The motivation of this paper is clear. The story flow is smooth. 

+Interesting idea
Although the render-and-optimize strategy is intuitive, the idea is interesting and reasonable. 

+Goog performance
From the shown results, the performance of the proposed method is promising.

+Sufficient experiments
This paper shows sufficient experimental results in the main paper and supplementary material. It would be good if more baselines could be included for comparison.


**Weakness**

-Difference from previous works
Although the paper claimed the differences between the proposed method and several previous methods, the specific differences are not clear.
For example, this work also uses the VQGAN as the Taming. Are there any differences between VQGANs? This work is also like MAE. Is it possible for MAE to generate such infinite visual synthesis? More explanations about the uniqueness are expected. 

-Technique questions
The proposed render-and-optimize strategy first generates the first patch, then eventually generates other patches taking previous patches as conditions, like a Markov chain. Thus, the first patch is significant for the final result. It seems like a basis. What the results would be like if the first patch prediction is not much good? Such analysis is necessary and also reflects the robustness. 


questions: Besides the questions in the weaknesses, here are some other questions. 
1. As claimed, the proposed method brings two advantages (i) The autoregressive
rendering process with information transfer between contexts provides an implicit
global probabilistic distribution modeling. Why could provide an implicit global probabilistic distribution modeling? Any evidence could be provided? 
2. The visual synthesis could only generate similar cases with the training data or could generate some out-of-distribution cases? What is the generalization capability? For example, does it generate any natural scenes? 

limitations: Yes, the authors adequately the limitations of their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work presents an infinite image and video synthesis method based on a render and optimize strategy. The proposed autoregressive model iteratively predicts next patch according to contexts in order to take the dependency between patches into account. The generated results look less repetitive and more coherent than those by existing methods. And the property of infinite generation step is a good progress towards generating images in ultra-resolution compared to fixed size generation through a big generation network like GANs.
strengths_and_weaknesses: + Modeling the global distribution of a large visual data so that each generated patch has taken the context into consideration.

+ Impressive results in ultra-resolution which is rarely seen in previous work

+ The proposed method is demonstrated both in spatial dimension of images and temporal dimension of videos.

Overall, it is a nice work that brings the extrapolation or outpainting task to a new level. I have following concerns:

(1) It looks the proposed pipeline share similar idea with some early autoregressive image generation models like PixelRNN or PixelCNN which are generating each pixel iteratively based on the previously generated pixels, though this work is operating on the patch level. What are the main differences compared with these related works?

(2) Is the proposed work mainly for unconditional generation, based on an optional text prompt? Could it be used for conditional generation, like image extrapolation to extend its original field of view?

(3) To what extent could the long-range dependency between patches be learned? Or how far is the relation of two patches modeled? I understand it might be not across all image regions but it is better to show and analysis on this so that it could help control the generation steps or image resolution for the best quality.
questions: See comments above.
limitations: No. I'm wondering if the proposed method mainly works well on generating scene or landscape. Generating complex object shape should be still challenging.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a render-and-optimize framework that supports autoregressive models to synthesize in arbitrary directions. Some memory components are being proposed to keep track of longer-term dependencies. 
strengths_and_weaknesses: Strengths:
1. The concept of synthesize in arbitrary direction is the strongest point of the paper. But it is not very well-discussed in the paper. 
2. The visual results are with decent qualities. But it is not clear what are the proposed components that contribute/improve the visual quality, and what are the challenges that prior methods (InfinityGAN[17] and ALIS[30]) fail to address.

Weaknesses:
1. Some statements are pretty strong, but lacking results backing up. For instances,
(a) In the abstract, "However, since they fail to model global dependencies between patches, the quality and consistency of the generation can be limited." Some methods, such as InfinityGAN, model the global dependencies among all pixels with a global latent variable. But there is no comparison or any results supporting the statement is correct. The paper is cited in the first paragraph of the introduction but omitted in the second paragraph.
(b) In conclusion, "This strategy allows us to generate images and videos of infinite size." This is not the task that the authors first propose or address, which causes a mistaken over-claim. The authors should instead consider discussing what has been improved with the proposed framework.

2. The paper emphasizes the method takes care of the global dependency, but the algorithm design, such as the nearby context pool and the rendering model, only attends to the local region within a limited number of steps. Could the authors clarify which part of the pipeline models the global dependencies? It should be highlighted and carefully ablated if it is a critical intuition and contribution.

3. The comparisons to important baselines are missing, InfinityGAN and ALIS[30]. Especially, both of the baselines compared in the submission (VQGAN[8] and MaskGIT[2]) were not designed for infinite visual synthesis. The authors should compare with more relevant baselines.

4. The paper proposes modules that keep track of memory states, but there are no experiments showing the memory mechanisms are working properly. Especially, consider methods such as ALIS[30] that works completely fine without memory. The authors should demonstrate the importance of memory, and perhaps show that memory states indeed affects the results in distant patches. For instance, compare the differences between with and without a distant patch, is it just some random perturbations, or is it indeed provide some long-range dependency information?

5. The paper features infinite visual synthesis, but all the provided samples are only a few steps of extension. Could the author provide some visual results after 1k steps? The intermediate results can be omitted to save space. It is important to show that the algorithm can still produce plausible results after a large number of steps.

6. The additional applications in image-to-video and image-outpainting are all presented without careful comparisons. These are not new applications that lack baselines. For video synthesis, some recent video synthesis frameworks (e.g., StyleGAN-V) can already synthesize infinite-length videos. The authors should compare with them in the video synthesis problem. And the image outpainting is also relevant to InfinityGAN. 

7. Many citations are broken.

8. (Minor citation error) To the best of my knowledge, the first divide-and-conquer generation paper is COCO-GAN, but only its follow-up works are cited.
questions: See above.
limitations: I think the main problem and comparisons are not well discussed and compared. It is impossible to discuss the limitations without resolving the previous issues.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes an infinite visual synthesis approach using patch-level rendering and optimization. The main contribution includes (1) a "render-and-optimize" strategy; (2) arbitrary direction extension; (2) achieving good performance compared with baseline results.
strengths_and_weaknesses: Strengths:

+ The proposed "render-and-optimize" strategy is novel and effective.

+ The experimental results are significantly better than baseline results.

+ The ablation study is extensive and solid.

Weaknesses:

- Although 5 shows video frames individually, the video itself is not included in the supplementary material. This makes it hard to see the temporal smoothness, which is usually critical for video quality evaluation.

- The algorithm is claimed to be fast. However, there's no evaluation about speed, and the factors affecting the speed.
questions: How is Fig. 1 generated? What is the input? It would be better if details can be included in the caption.
limitations: The limitations are mentioned in the supplementary materials. But it would be good if the authors mention them briefly in the main paper.
review_title: null

Meta-review: metareview: This paper proposes a patch-level autoregressive model for infinite visual synthesis based on two extensions: (1) transfer information across patches via context vectors and (2) timely optimization for each patch. Note that hierarchical autoregressive models have been explored in previous works such as VQ-VAE2 (https://arxiv.org/pdf/1906.00446.pdf). But the idea of context pool and arbitrary direction modeling seems interesting. The paper has received consistently positive reviews. Reviewers found the idea intuitive and the results visually compelling. The rebuttal further addressed the concerns such as the missing comparisons (w/ InfinityGAN and ALIS), running speed, and clarifications w.r.t existing works. The AC agreed with the reviewers’ consensus and recommended accepting the paper. 
confidence: Certain
Decision: Accept