NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesis
Authors: Jian Liang
Chenfei Wu
Xiaowei Hu
Zhe Gan
Jianfeng Wang
Lijuan Wang
Zicheng Liu
Yuejian Fang
Nan Duan
Venue: NeurIPS 2022 Accept (2022)
Keywords: Image synthesis
video synthesis

Abstract:  Infinite visual synthesis aims to generate high-resolution images, long-duration videos, and even visual generation of infinite size. Some recent work tried to solve this task by first dividing data into processable patches and then training the models on them without considering the dependencies between patches. However, since they fail to model global dependencies between patches, the quality and consistency of the generation can be limited. To address this issue, we propose NUWA-Infinity, a patch-level \emph{``render-and-optimize''} strategy for infinite visual synthesis. Given a large image or a long video, NUWA-Infinity first splits it into non-overlapping patches and uses the ordered patch chain as a complete training instance, a rendering model autoregressively predicts each patch based on its contexts. Once a patch is predicted, it is optimized immediately and its hidden states are saved as contexts for the next \emph{``render-and-optimize''} process. This brings two advantages: ($i$) The autoregressive rendering process with information transfer between contexts provides an implicit global probabilistic distribution modeling; ($ii$) The timely optimization process alleviates the optimization stress of the model and helps convergence.  Based on the above designs, NUWA-Infinity shows a strong synthesis ability on high-resolution images and long-duration videos. The homepage link is \url{https://nuwa-infinity.microsoft.com}.

NUWA-Infinity: Autoregressive over Autoregressive
Generation for Infinite Visual Synthesis
Jian Liang'*
Chenfei 2u Xiaowei Hu Zhe Gan
Jianfeng Wang Yuejian Fang't t Nan Duan
Lijuan Wang? Zicheng Liu
Peking University Microsoft Research Asia MMicrosoft Azure Al
{j. S..iinggstu,,fan fangy jangyjess)..m pku. edu. en
(chewu, xiaowei. hu, zhe. gan,jianti 1i jijuan,, zliu, manduan))miccooooft..o
Abstract
Infinite visual synthesis aims to generate high-resolution images, long-duration videos, and even visual generation of infinite size. Some recent work tried solve this task by first dividing data into processable patches and then training the models on them without considering the dependencies between patches. However, since they fail to model global dependencies between patches, the quality and consistency of the generation can be limited. To address this issue, we propose NUWA-Infinity, patch-level "render-and-opp" synthesis. Given a large image or a long video, NUWA-Infinit first splits it into non-overlapping patches and uses the ordered patch chain complete training instance, a rendering model autoregressively predicts each patch based on its contexts. Once patch is predicted, it is optimized immediately and its hidden states are saved as contexts for the next "render-and and-optimize" process. This brings advantages: (i) The autoregressive rendering process with information transfer between contexts provides an implicit global probabilistic distribution modeling; (ii) The timely optimization process alleviates the optimization stress of the model and helps convergence. Based on the above designs, NUWA-Infinity shows a strong synthesis ability on high-resolution images and long-duration videos. The homepage link is wihhtts://// //nwwa-
strategy for infinite visual
Introduction
The field of deep visual synthesis has witnessed great advances recent years, with a notable
trend of generating images from low to high resolutions. Specifically, many works have shifted from
to 512 x 512 [ and 1024 x 1024 (24), The same trend is witnessed in the video field as well, as the number of supported frames is increased from 25 (35) and 48 [4] to even
generating images with resolution of 256 x 256 [25 36 [0 37
even infinite ones of 256 x DC 29 and 512
17
2
infinity [8]] By increasing the spatial resolution and the number of temporal frames infinity, these models have shown great visual quality with both high fidelity and creativity.
To support "infiiity"", most works follow the divide and conquer strategy to first divide a large image into several processable patches and then train them in a separate way. GAN-based models [32] (2) at- tempt to divide large images into several patches and optimize each of them from global or coordinate latents separately. Since different regions have no explicit dependency, these models struggle to merge different patches during inference and can easily lead to inconsistent results. To address this issue, auto-regressive models g incorporate sliding window to make connections between different
Both authors contributed equally this research. i Conreppondigg author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Figure 1:
The RiverSide of Qingming Festival painted by our proposed NUWA-Infinin model in resolution of 2048 x 38912. Trained from scratch on the original painting.
patches during inference. However, since different patches are still trained separately, the sliding window only relieves the inconsistency between patches, failing to model the global probability distribution a large image or video.
We propose NUWA-Infinity, a patch-leve large image, rendering model auto-repressively predicts each patch based
"render-and-oplimiie"" strategy for infinite visual synthe-
sis. When training on its contexts. Once a patch is predicted, the loss is calculated and the parameters are optimized immediately and its hidden states are saved contexts for the next "render-and and-optimize" process. This brings two benefits: (i) The autoregressive rendering process with information transmission between contexts provides an implicit global probabilistic distribution modeling of the whole image; (ii) The timely optimization process alleviates the optimization stress of the model and helps conver- gence. Further, an arbitrary direction modeling is proposed help NUWA-Infinity auto-repressively learn in arbitrary directions.
Based the above designs, NUWA-Infinity shows a strong synthesis ability on even extreme high-resolution images (see Fig. D.
To sum up, the major contributions of this paper
We propose NUWA-Infinity a patch- cunder-and-optinne strategy that models the global probabilistic distribution of large image or video with fast convergence.
We propose an arbitrary direction relative position embedding, which enables the free- direction extension of KUWA-Infinity
NUWA-Infinity shows surprisingly good performance on both image and video synthesis. We also conduct detailed ablations show the trade-off's between different rendering model.
2 Related Work
High-Resolution Visual Synthesis High-resolution visual synthesis is a hot topic recently. Most models, such Cogview [7] and DALLE-2 (24) firstly generate small image, and then gradually scale it up. However, these two separate steps will result in a transmission gap between the small and the high-resolution images. Since small image only contains few limited objects, the super- resolution model can only scale it up, but cannot generate new ones. This paper focuses on Infinite Visual Synthesis, which aims to generate images or videos of arbitrary size without super resolution.
Infinite Visual Synthesis Existing GAN-based models mainly embed global latent vectors 32 9 coordinates conditions I7 30 into the latent space model digh-dimensional visual features,
28
Since this paper focus infinite visual synthesis, patch represents large region, for example 256x 256.

---

Patch Contexts
11
16
21
Rendering Model
Context:
22
Optimize
13
NAR
Prompt:
The green mountain
5
10
15
20
25
reflects on the green lake.
Figure 2: The overview NUWA-Infinit model. When training visual data, a rendering model autoregressively predicts each patch based on its contexts. Once a patch is predicted, it is optimized immediately and its hidden states are saved as contexts for the next rrender-and-a and-optimize" process.
and then high- resolutonn images can be fast sampled from target distribution, but usually faces the problem of complicated optimization, unstable training, and low diversity. Mask-Predie [3] 38 2
as progressive non-autoregressive generation models designing complex sampling strategies on bidirectional masked language models, which generate several visual tokens in one step to speed up inference time, yet needs to feed entire sequence into the bidirectional model which makes impossible to scale up to a larger resolution. Auto-regressive models [8 2 incorporate sliding window to make connection between different patches during inference. However, since different patches are still trained separately, the sliding window only relieves the inconsistency between patches, yet still failing to model the global probability distribution data. This paper is also based on auto-regressive architectures, but aims to directly train large image video in efficient way.
Long-Range Sequence Modeling To efficiently model long Dang-range sequences, on the one hand, the sparse computing is designed, like Axial Attention [14] and 3D Nearby Attention [36), they can reduce a lot of computation while maintaining visual quality; on the other hand, linking long sequences through memory information, memory-based models 6 23 5 11] process the sequence in several segments and keep the sequence integrity with auxiliary historical memory. This paper proposes a "render-and-optmizz"" strategy and context pool to accelerate convergence and keep long-term memory for sequence modeling.
Arbitrary Direction Extension Image out-painting and video prediction are considered as spatial and temporal extension respectively, therefore auto-regressive model can follow the orientation of different axes to extend images videos naturally. However, since the direction of training AR is fixed, it can only be limited to extension in a certain direction. To support the extension in arbitrary directions, coordinate-bac based generative models [30 [7 27 21 are proposed, They embed coordinate codes into latent space to generate complete image, but this easily leads to repetition. Mask-Predict models 3 benefit from bidirectional language model, They iteratively mask and predict new regions to expand in any direction, while the context it relies on is only within the training window, resulting in lack the global consistency, Our approach is to change the training strategy of the AR model, the autoregressive direction during training covers the directions that can be extended, and this way allows us to use the AR model for arbitrary direction extension.
Method
Given optional text prompt y. the infinite visual synthesis task aims to generate an arbitrary-size visual target æ (e.g., a large image or a long video), i.e., (xxly) Since æ could be extremely large, current models [8] mainly try to split into several patches {p) P...... and assume they are i.i.d during training, i.e.. -I P(plly ly), where N the number of patches, but this is strong assumption. To consider the dependencies between patches, a simple idea is to model different patches in an ruto-regressive manner, i.e., - IP (pdleei y). but this requires the model to fully consider all previous information. Hence challenges turn to how to properly model pez with affordable computing resources.

---

To address the above issues, we propose NUWA NUWA-Infinity, Infinity, a patch-level "render-and pnnder-andooptiizze strategy that offers efficient training and inference whiling modeling Pei the same time. We achieve this by incorporating context as latent variable in Eq.
P (xlyy)
II Pppppappee
II P(lcc)
where e, should encode the information of Pei. Eq. consists of two components:
Rendering Model. We define plug-and-play rendering model that can be selected according to the balance of performance and speed. A rendering model P (p: (c, y) that receives visual context C; and an optional text prompt y. and then generates the current patch p;.
Rendering Strategy. We define patch-by pay-patch rendering strategy that repeatedly invoke the rendering model. The patches rendered in the past P<i are used to generate a context c;, which can be then used for the next rendering step.
Our approach uses a powerful visual rendering model (oxlee,y) though can only generate fixed-size patch each time, it considers the context of surrounding patches to ensure visual continuity and integrity. Through our designed patch-by-patel rendering approach, this rendering model will be repeatedly called to generate each patch; therefore, the modeling of I' (p,, c/Ppci, y) allows us to synthesize images or videos of any desired size.
3.1 Rendering Strategy
3.1.1 Arbitrary Direction Modeling
We encode raw visual pixels into discrete tokens by VQGAN [ which significantly reduces the sequence length. Let & E / be the visual tokens with height h. width w and frames. Images are considered special videos with one frame. Firstly, we split z several non-overlapping patches p E 181""="" Then can get (H W F)) patches, where /
W -u fr There are multiple ways to arrange these patches as an auto-regressive sequence. To enable the spatial extension in arbitrary direction, we define the generation direction r, including Down-Rite Right-Foowarrdd Down-Left-Forward, Right-Up-Forward and gight-Down-Forweee
shown Fig (b), we take Diwn-Right- Finn-Riaht-Forward direction an example. This direction denotes the patches are flattened from up down as first sort, from left to right as a second sort and from back to forward as the third sort. We consider that the coordinates of all patches are fixed, and different directions only change the time order of generation. Therefore, we define T to convert coordinate (i, j, k) into time order n with certain direction and 7 can deduce coordinates from time order and direction r.
Pixel-level
Patch-eeel
2eah
3rd
2el +1
2e
(a) Token-level 3D representation
(b) Patch-level 3D representation
[c) Patch-level 3D relative position
Figure 3: 3D representation for visual data. visual tokens obtained by using VQGAN. (b) Each patch consists of several tokens, and the numbers on the patches indicate the time order of generation. (c) Relative position is based on the patches.
To be able to extend infinitely, how to efficiently encode position infinite size is crucial. Absolute position embedding 34 is unable to include all coordinates, while all coordinates can be represented by using a fixed number of relative position codes [15] 20 Assuming that each patch only focuses on the information around it, can define receptive field expansion size as (e" e e!) in Fig

---

(c), denoting the expansion size in the height, width and temporal axis respectively. Therefore we only need create a learnable relative bias B (12===))2e-))(2222 +1) which can represent all position relationships between patches to represent infinite positional embeddings.
3.1.2 Nearby Context Pool
To make connections across patches, we propose a nearby context pool q. It can store the hidden states mc; of the previous patch p<i- Then these hidden states will form the context G; as the rendering conditions of the patch y): S.c mentioned meettoonee that each patch will pay attention to the surrounding information within (e", e" e²), so the pool just needs to save the hidden states around each patch that have not yet been rendered. Although the pool does not store all hidden states, each hidden states generated based on the previous hidden states, so this builds an information transfer chain to enable patches to obtain long-range contexts. We define three operations on this pool:
Select the contexts of patch at position (i.j,k from the pool with direction r.
(H..)) (2mm( "' e - = , K & T( TIC,,, T(......))) (2)
Add the hidden states patch at position ji, k)) into the pool g:
q = q (mbaam)
(3)
Remove the expired hidden states from the pool q position (i, with direction
q :==0 9 of { m (( T(min((' e", H..HH,mmn min(j c", W), min(k' + +el, ((..kkr))
3.1.3 Training and Inference Strategy
We propose an infinite patch-by-pate rendering strategy, which is based the nearby context pool, the training and inference method are defined below.
Algorithm 1:
Training Strategy
Algorithm 2: Inference Strategy
Data: images or videos x, optional text y
Input: optional text y. target size Output: generated images or videos initial context pool q 0 initial generation < 0;
Result: optimized rendering model
initial context pool q +00
p [I...N] +- (x);
r < sample training direction;
initial + 1;
for all from / N do
while not reached target size 8 do
i.j, + T (n,
((.j,k) Select(n):
d(..)) Seleect((,
Pn; m, Render (Cay y);
2(.Jil),m
(1 (plddr( c(3..)) ((k.))))))))))))......))))
q - q (mdd q q. Remoove();
g (t.
9 + Addd((p));
q. Remeovve);; optimize ((33M); end
n n + 1;
end
Return merge(g):
As summarized in Algorithm I Given a visual data and optional text Firstly, we split into several patches, and then we get (H x W F) patches, for each training batch we randomly sample a direction r. Finally, sliding rendering window to use "render-and and-optimize the rendering model gets conditions and outputs loss of current patch, each loss of patch will be optimized separately, and then the updated parameters are used immediately for the next rendering.
TTii equation only denotes Down-Right Right-Forward Forward direction example.

---

3.2 Rendering Model
The Rendering model I (pslc), y) is a conditional generative model, receives a context e; selected from the nearby context pool and optional text prompt y, and then concentrate on generating the current patch Pt- We first provide the overall equation below:
s = SA(LN(Q + 2 k+ [2' d' 1 - d (l)))))) + +z--1,
2 crass = CC + Ladts f k + y', + g't) + 2505s
(5)
2' "MMLPLLN (A + +2
where SA and CA are Self-Attention and Cross-Attention respectively, LN is LaryerNorm and M LP is linear layer. In SA, - and d² denote the output and nearby context in (1 - 1) layer, b' b--1 denotes the learnable relative position bias. we propose a pre- pre-perceived relative position embedding (Pre-RPE), different from post- pesceeivdd relative position embedding (Post-RPE) [9], we add b into the key. When producing attention-map, the location relationship between patches and contexts can be perceived in advance with Pre-RPF, instead of adjusting the already generated attention-map through Post-RPE. As a result, the SA outputs visual features with context. The CA produces cross-modal features zoross by interaction between visual outputs y and text features y'. The each layer output will be used as hidden states into the nearby context pool for next patch rendering.
We explore three rendering models:
Autoregressive (AR): AR a serial generation P(s] 2/ Ct. C,,y), where each discrete token s/ will be predicted autoregressively. The input of first layer in the AR model is a sequence, which includes a BOS token and discrete visual tokens by VQGAN, we use the learnable axial absolute position encoding for this visual sequences. Tokens are predicted sequentially based on previous tokens:
Spttch -EeELLAPee (o) [log Po(s:)s<e:c,,)).
(6)
Non-Autorege Antoortrrrrresvvee (NAR): The tokens in NAR are independent of each other, and they only depend on the input context G. and text prompt y. Therefore, P(p(ec,)) y) where patch p; can be generated directly in parallel. There are no serial dependencies between tokens in NAR, the sequence will be generated in parallel, and this greatly accelerates sampling speed. The first layer input of NAR is only an initialization sequence with position encoding, and do not require the BOS. In addition, since the visual input during training is position embeddings without the target tokens, we need to run NAR twice, the first time is used to optimize the model by pparo 7 the second running get the each layer output, which will be saved cache into nearby context pool.
Sparch -Eagit, [log Po(sel,c.
To improve the generation performance of NAR. In P-NAR P(p, Pr. C. y) the previously generated sequence pi will be iteratively optimized. P-NAR can be considered a Mask-Predict [38] 2 , it predicts all target tokens when given a fully-masked sequence at the first iteration, and then iteratively re-mask and re-predict subset of tokens with low probability scores for constant number of iterations. Follwing MaskGIT (2), We use the cosine mask scheduling function y Cosine 1)), then masked tokens 8m sampling from D ? (h" w f*). Eventually, through the mon-masked tokens sm. predicts the masked tokens:
Progressive Non-Autoregreesiive (P-NAR):
Cpateh -EmeD (log (9( C, y)]
(8)
Experiments
4.1 Experiment Setup
Datasets. For image synthesis, we trained unconditional generation model on the LHQ [30, which consists of 90k high-resolution (> 1024²) nature landsacapes. In addition to support text prompt, we added a caption for each image of LHQ to create a new dataset called LHQC, where 85k as training data and 5k as test data. For video synthesis, we downloaded 120k high-resolution videos from pexels website and ran a pretrained Mask R-CNN [2] to remove the videos that likely contain objects on them. As result we keeped 40k high-quality videos called LHQ-V.
Metrics. For image synthesis, we use Inception Score (IS) 26 and CLIP Similarity Score (CLIP- SIM) 2 to evaluate the sample diversity and semantic consistency between images and text. In

---

field of yollow flowors mountain the background
lake with mountain the background
Taming
MaskGIT
NUWA WWWA-InfinityyOurs
Figure 4: High Resolution text-to-image, the left 1024x 1024, the right 1024x4096.
addition, we propose a Block-FID, which splits the large image into blocks to calculate Fréchet Inception Distance [13], can avoid the downsampling caused by resizing images and work for infinite-size synthesis. Since rendering windows and blocks (256x256 in our experiments) maybe overlap, when measuring our model will generate larger images, and move half of the block size. For video metrics, because the resize operation has little effect on the motion of the video, we use Fréchet Video Distance (FVD) directly to evaluate the quality of the generated videos.
Implementation Details. During training, images are cropped into 1024x1022 and videos are cut into 1024x1024x5 with 5fps, then, they will be encoded into discrete tokens using the VQGAN model with compression rate of 16 and a codebook of 16384. Sec. 3.2 the rendering size of the three models is 256x256. In Sec. 3.1 based on the nearby sparsity, images and (e" e ef) (1, 1, for videos. We train the model using Adam optimizer [16] with learning rate of le-4, a batch size of 256, and warm-up 5% of total 50 epochs.
set (e" e" ee) 2, for
4.2 Evaluation on Visual Synthesis
High Resolution Text-to-Image We evaluate NUWA-Infinity on high resolution text-to-image task with LHQC dataset in Tab. [ Under the 1024x1024 resolution same the size of the training dataset, our model exceeds other methods with Block-FID of 9.71, as well as better text-image semantic consistency of 0.2807 and more diversity of 4.98. When generating larger images such as 1024x4096 (x4) resolution, the performance of MaskGIT (2) decreases rapidly due to limited Window, but our model can still maintain higher visual quality with Block-FID of 15.65.
Method
Block-FIDJ
Block-FID(x4)L
ISt
CLIP-SIM
Taming
38.89
46.37
4.58
0.2662
MaskGIT 24.33
45.76
4.61
0.2754
Our (AR)
9.71
15.65
4.98
0.2807
Table 1: High resolution text-to-image synthesis on LHQC, the size of sample is 1024x1024 by default, x4 means 1024x4096.

---

NAR
1-10
1:20
1:40 6000
1+50
P-NAR
AR
1+30 14:0
Figure 5: Long-duration video prediction, generating 60 frames from frame.
High Resolution Image-to- Image-to-Video Since our framework generates patches autoregressively. and easily expand from image synthesis to video synthesis. We compare the performance of three different rendering models for video prediction in Tab. and Fig. 5
Method
FVD.
StyleGAN-V
143.76
Our (NAR)
368.16
Our (P-NAR)
165.39
Our (AR)
62.57
Table 2: Video prediction on LHQ-V, V, the size of sample is 1024x1024x8.
Block-FIDJ
Method
Right Extend +
Left Extend +
Down Extend I
Up Extend f
Taming
8
22.53
N/A
26.38
N/A
MaskGIT
14.68
14.81
25.57
25.38
InfinityGAN 17 w/o text
17.93
19.76
24.62
23.59
Our (AR) w/o text
6.43
6.71
11.47
8.03
Our (AR) w/ text
6.45
6.72
9.84
7.43
Table Arbitrary image extension on LHQC, calculating Block-FID between the extended area and the same area in the test set, not using full images. Taming and MaskGIT use text prompt by default.
Arbitrary Image Extension Tab. 3 and Fig. demonsttaaes that proposed approach achieves state-of-the-art performance on arbitrary image extension task. Traditional AR models like Taming [8] only generate samples along certain order and can not achieve image outpainting in arbitrary direction. MaskGIT benefits from a bidirectional masked language model, which can add new mask area in any direction and predict it to extend images. However, since MaskGIT is inconsistent between the mask area of training images and extending images, and this damage its performance. In addition, found that the prompt does help when extending the image left right, while the text prompt brings performance improvements when extending up or down. We think it is because upper lower half of the image contains less information, while the left or right half has more visual semantic hints.

---

a fenced area with doudy sky above
road that going down hill
large rock formation the desert with cloudy sky
tree reflected in the water of river
Input
Taming
MaskG
NUWA-infinity((uur
Figure 6: Arbitrary direction image extension, the input is 1024x512.
4.3 Ablation Studies
Rendering model Tab. 4a shows that the AR rendering model has the best performance but sacrificed the generation speed. P-NAR optimizes the results of NAR multiple times and achieve good performance at not bad speed. Scaling up the model in Tab. 4b using Pre-RPE in Tab. 4c can improve the model performance. visual quality and smaller rendering size will lead to worse performance, especially for 1024x4096 (x4) resolution generation. is because of too small patches as low-level visual features, and is not conducive to model learning the global consistency. When rendering size is equal to 256, this phenomenon will disappear, but size continues to expand, it easily leads to out of memory.
noticed Fig. 7a that the rendering size has a great impact
Training way Tab. 4e shows that our synthesis anwpiis--nd--opmimiii training strategy can accelerate convergence from 70 epochs to 50 epochs. If the model accumulates the gradient of the full images to optimize, not conducive to sharing the parameters between each patch, so that the rendering model is too concerned about the global layout without focusing connection between each patch. In addition, each patch optimized by itself and this can provide more data distribution. The success of warthesis-and-optine context transfer. Tab, 4d proves that context transfer will bring huge improvements, where no transfer means the cache generated by each patch only contains its own information.
strategy also benefits from the long-term connections between patches by
Nearby extension Although implicit context transfer can make the cross-patce connection distance farther, explicitly increasing expansion size in nearby context pool will bring more stable connections. Fig. 7b shows the impact of spatial expansion size on image generation, as eh and e increase, performance will quickly improve until size is equal to 2, but FLOPs will continue to grow. Fig. presents the impact of temporal expansion size on video prediction. Without considering context transfer, when el is equal to the model can obtain the previous one statie frame. When el equal to 2, the model can obtain the action between the two static frames, but when el reaches 3, the model

---

Render model Block-FIDJ
Block-FID(X 4)
CLIP-SIMÍ
Inference Speedt
NAR
92.34
98.67
0.2451
95x
P-NAR
19.86
38.59
0.2726
15x
AR
10.05
17.78
0.2753
1x
(a) Render model.
Parameters
Depth
Dim
Block-FIDJ
RPE
Block-FIDI
Block-FIIDx4)) 4
202M (Base)
16
768
10.05
Pre
10.05
17.78
809M (Large)
1280
9.71
Post
10.47
18.89
(b) Model size.
(c) Relative position embedding.
Context
Block-FIDD
Block-FID(x4) |
Loss
Block-FIDJ
Convergence epoch.
w/o transfer
15.8
38.32
Patch
10.05
50
w/ transfer
10.21
23.62
Full
11.62
70
(d) Context transfer.
(c) Loss region.
Table Ablation experiments with text-to-image on LHQC, the experiments use base model with layers of 16 and dim of 768 except (b). Default setting are marked in gray.
can obtain the movement between two dynamic visual features. So starts from ef =3 the growth of the performance is not obvious.
Hondtiak RO
12/4
Tooss
30C
346
17.8
10/35
1813
22:1
6.31
18
Spattal
neadby
Tamporal estero on stm
neadly contect
(a) Rendering size.
(b) Spatial extension.
(c) Temporal extension.
Figure 7: Ablation results on rendering size and nearby expansion size, (a) (b) are based on text-to-image synthesis, (c) is based on video prediction with 8 frames. Note that they all use the base model with layers of 16 and dim of 768.
Conclusion
In this work, we propose NUWA-Infinity, a patch-level "render-and-optimizz strategy for infinite visual synthesis. training stage, visual data are spitied into non-overlapping patches with different time order, then rendering model autorepressively predicts each patch based its contexts. the patch is optimized at each time step, and the optimized parameters are immediately used in the next rendering. This strategy allows to 1) generate images and videos of infinite size 2) visual extension in any direction 3) information transfer brings long-term memory 4) accelerate model convergence.
Acknowledgements
The paper is supported by the National Key Research and Development Project (Grant No. No.2020AAA01066000,
10

---

References
[1] Eric R Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 5799-5809, 2021.
[2] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T. Freeman. MaskGIT: Masked Generative Image Transformer. MaskGIT.
[3] Jaemin Cho, Jiasen Lu, Dustin Schwenk, Hannaneh Hajishirzi, and Aniruddha Kembhavi. X-Ixmert: Paint, caption and answer questions with multi-moda transformers. arXiv preprint arXiv:2009. /278, 2020.
[4] Aidan Clark, Jeff Donahue, and Karen Simonyan. Adversarial Video Generation on Complex Datasets. 48.
[5] Marcella Corna,, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed-memory transformer for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2020.
[6] Zihang Dai, Zhilin Yang. Yiming Yang, William W. Cohen, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. Transformer-XL..
[7] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, and Jie Tang. Cog View: Mastering Text-to-lmage Genera- tion via Transformers. CogView.
[8] Patrick Esser, Robin Rombach, and Björn Ommer. Taming Transformers for High- Image Synthesis. Comment: Changelog can be found in the supplementary.
[9] Anna Frühstück, Ibraheem Alhashim, and Peter Wonka. Tilegan: synthesis of large-scale non-homogeneous textures. ACM Transactions on Graphics (ToG), 38(4):1-11, 2019.
[10] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. arXiv preprint arXiv:2111.14822. 2021.
[11] Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao. Xiangyang Xue, and Zheng Zhang. Star- tturtrrasffrmer. arXiv preprint 1902::190..091133 2019,
[12] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn.
[13] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[14] Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimensional transformers. arXiv preprint 19Xiv:1912.12180. 2019.
[15] Zhiheng Huang, Davis Liang, Peng Xu, and Bing Xiang. Improve transformer models with better relative position embeddings. arXiv preprint arXiv:2009. 13658, 2020.
[16] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:/412.69880 22.000, 2014.
[17] Chich Hubert Lin, Hsin- Hsin-Ying Lee, Yen-Chi Cheng, Sergey Tulyakov, and Ming-Hsuan Yang. Infinitygan: Towards infinite-resolution image synthesis. arXiv e-prints, pages arXiv-2104, 2021.
[18] Andrew Liu, Richard Tucker, Varun Jampani, Ameesh Makadia, Noah Snavely, and Angjoo Kanazawa. Infinite nature: Perpetual view generation of natural scenes from single image. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 14458- 14467. Infinite Nature.

---

[19] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[20] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu, Video swin transformer. arXiv preprint arXiv:2106. 1 13230, 2021.
[21] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional genera- tive neural feature fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11453-1 11464, 2021.
[22] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763. PMLR, 2021.
[23] Jack W Rac, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv: 191.:5911.00507, 2019.
[24] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical Cext-Conditional Image Generation with CLIP Latents.
[25] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Iyy Sutskever. Zero-Shot Text-to-Image Generation. DALL-E DALLE.
[26] Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. Advances in neural information processing systems, 29, 2016.
[27] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. Advances in Neural Information Processing Systems, 0122014-220666 2020.
[28] Tamar Rott Shaham, Tali Dekel, and Tomer Michaeli. Singan: Learning a generative model from single natural image. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4570-4580, 2019.
[29] Ivan Skorokhodov, Grigorii Sotnikov, and Mohamed Elhoseiny. Aligning latent and image spaces to connect the unconnectable. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14144-14153. ARIS.
[30] Ivan Skorokhodov, Grigorii Sotnikov, and Mohamed Elhoseiny. Aligning latent and image spaces to connect the unconnectable. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 14144-14153, 2021.
[31] Ivan Skorokhodov, Sergey Tulyakov, and Mohamed Elhoseiny. Stylegan-v: A continuous video generator with the price, image quality and perks of stylegan2. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6626-3636 2022.
[32] \ Lukasz Struski, Szymon Knop, Jacek Tabor, Wiktor Daniec, and Przemyst law Spurek. LocoGAN-Locally Convolutional GAN. LocoGAN.
[33] Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint 18/2.0111 2018
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008. fixed Transformer.
[35] Ruben Villegas, Arkanath Pathak, Harini Kannan, Dumitru Erhan, Quoe V. Le, and Honglak Lee. High fidelity video prediction with large stochastic recurrent neural networks. 32. 25.
12

---

[36] Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. NY'UWA: Visual Synthesis Pre-training for Neural visUal World creAtion.
[37] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2 10..4477, 2021.
[38] Zhu Zhang, Jianxin Ma, Chang Zhou, Rui Men, Zhikang Li, Ming Ding, Jie Tang, Jingren Zhou, and Hongxia Yang. M6-ufc: Unifying multi-modal controls for conditional image synthesis. arXiv preprini Axiv:21055.1221,, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Supplementary material
(C) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
(b) Did you include complete proofs of all theoretical results? [Yes] See Section
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [No]
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Supplementary material
4. If you using existing assets (e.g., code, data, models) or curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "R24yX96tUeK", "reviewer": "Reviewer_j1NF", "summary": "The paper proposes a render-and-optimize framework that supports autoregressive models to synthesize in arbitrary directions. Some memory components are being proposed to keep track of longer-term dependencies. ", "questions": "See above.", "limitations": "I think the main problem and comparisons are not well discussed and compared. It is impossible to discuss the limitations without resolving the previous issues.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The concept of synthesize in arbitrary direction is the strongest point of the paper. But it is not very well-discussed in the paper. \n2. The visual results are with decent qualities. But it is not clear what are the proposed components that contribute/improve the visual quality, and what are the challenges that prior methods (InfinityGAN[17] and ALIS[30]) fail to address.\n\nWeaknesses:\n1. Some statements are pretty strong, but lacking results backing up. For instances,\n(a) In the abstract, \"However, since they fail to model global dependencies between patches, the quality and consistency of the generation can be limited.\" Some methods, such as InfinityGAN, model the global dependencies among all pixels with a global latent variable. But there is no comparison or any results supporting the statement is correct. The paper is cited in the first paragraph of the introduction but omitted in the second paragraph.\n(b) In conclusion, \"This strategy allows us to generate images and videos of infinite size.\" This is not the task that the authors first propose or address, which causes a mistaken over-claim. The authors should instead consider discussing what has been improved with the proposed framework.\n\n2. The paper emphasizes the method takes care of the global dependency, but the algorithm design, such as the nearby context pool and the rendering model, only attends to the local region within a limited number of steps. Could the authors clarify which part of the pipeline models the global dependencies? It should be highlighted and carefully ablated if it is a critical intuition and contribution.\n\n3. The comparisons to important baselines are missing, InfinityGAN and ALIS[30]. Especially, both of the baselines compared in the submission (VQGAN[8] and MaskGIT[2]) were not designed for infinite visual synthesis. The authors should compare with more relevant baselines.\n\n4. The paper proposes modules that keep track of memory states, but there are no experiments showing the memory mechanisms are working properly. Especially, consider methods such as ALIS[30] that works completely fine without memory. The authors should demonstrate the importance of memory, and perhaps show that memory states indeed affects the results in distant patches. For instance, compare the differences between with and without a distant patch, is it just some random perturbations, or is it indeed provide some long-range dependency information?\n\n5. The paper features infinite visual synthesis, but all the provided samples are only a few steps of extension. Could the author provide some visual results after 1k steps? The intermediate results can be omitted to save space. It is important to show that the algorithm can still produce plausible results after a large number of steps.\n\n6. The additional applications in image-to-video and image-outpainting are all presented without careful comparisons. These are not new applications that lack baselines. For video synthesis, some recent video synthesis frameworks (e.g., StyleGAN-V) can already synthesize infinite-length videos. The authors should compare with them in the video synthesis problem. And the image outpainting is also relevant to InfinityGAN. \n\n7. Many citations are broken.\n\n8. (Minor citation error) To the best of my knowledge, the first divide-and-conquer generation paper is COCO-GAN, but only its follow-up works are cited.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QaOb3DWQSi4", "reviewer": "Reviewer_msGb", "summary": "The paper proposes an infinite visual synthesis approach using patch-level rendering and optimization. The main contribution includes (1) a \"render-and-optimize\" strategy; (2) arbitrary direction extension; (2) achieving good performance compared with baseline results.", "questions": "How is Fig. 1 generated? What is the input? It would be better if details can be included in the caption.", "limitations": "The limitations are mentioned in the supplementary materials. But it would be good if the authors mention them briefly in the main paper.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n+ The proposed \"render-and-optimize\" strategy is novel and effective.\n\n+ The experimental results are significantly better than baseline results.\n\n+ The ablation study is extensive and solid.\n\nWeaknesses:\n\n- Although 5 shows video frames individually, the video itself is not included in the supplementary material. This makes it hard to see the temporal smoothness, which is usually critical for video quality evaluation.\n\n- The algorithm is claimed to be fast. However, there's no evaluation about speed, and the factors affecting the speed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Bt9igq4CpkI", "reviewer": "Reviewer_o5hC", "summary": "This paper focuses on visual synthesis, especially for high-resolution cases. To model global dependencies between patches, \nthis work proposes a patch-level render-and-optimize strategy, i.e., once a patch is predicted, its hidden states are saved as contexts for the next round of prediction. In addition, an arbitrary direction relative position embedding method is proposed, which shows promising performance on free-direction visual extension. The experimental results show the good performance of the proposed method on visual synthesis.  ", "questions": "Besides the questions in the weaknesses, here are some other questions. \n1. As claimed, the proposed method brings two advantages (i) The autoregressive\nrendering process with information transfer between contexts provides an implicit\nglobal probabilistic distribution modeling. Why could provide an implicit global probabilistic distribution modeling? Any evidence could be provided? \n2. The visual synthesis could only generate similar cases with the training data or could generate some out-of-distribution cases? What is the generalization capability? For example, does it generate any natural scenes? \n", "limitations": "Yes, the authors adequately the limitations of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n+Clear Motivation\nThe motivation of this paper is clear. The story flow is smooth. \n\n+Interesting idea\nAlthough the render-and-optimize strategy is intuitive, the idea is interesting and reasonable. \n\n+Goog performance\nFrom the shown results, the performance of the proposed method is promising.\n\n+Sufficient experiments\nThis paper shows sufficient experimental results in the main paper and supplementary material. It would be good if more baselines could be included for comparison.\n\n\n**Weakness**\n\n-Difference from previous works\nAlthough the paper claimed the differences between the proposed method and several previous methods, the specific differences are not clear.\nFor example, this work also uses the VQGAN as the Taming. Are there any differences between VQGANs? This work is also like MAE. Is it possible for MAE to generate such infinite visual synthesis? More explanations about the uniqueness are expected. \n\n-Technique questions\nThe proposed render-and-optimize strategy first generates the first patch, then eventually generates other patches taking previous patches as conditions, like a Markov chain. Thus, the first patch is significant for the final result. It seems like a basis. What the results would be like if the first patch prediction is not much good? Such analysis is necessary and also reflects the robustness. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "B9X4vEVtQtR", "reviewer": "Reviewer_q8JV", "summary": "This work presents an infinite image and video synthesis method based on a render and optimize strategy. The proposed autoregressive model iteratively predicts next patch according to contexts in order to take the dependency between patches into account. The generated results look less repetitive and more coherent than those by existing methods. And the property of infinite generation step is a good progress towards generating images in ultra-resolution compared to fixed size generation through a big generation network like GANs.", "questions": "See comments above.", "limitations": "No. I'm wondering if the proposed method mainly works well on generating scene or landscape. Generating complex object shape should be still challenging.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ Modeling the global distribution of a large visual data so that each generated patch has taken the context into consideration.\n\n+ Impressive results in ultra-resolution which is rarely seen in previous work\n\n+ The proposed method is demonstrated both in spatial dimension of images and temporal dimension of videos.\n\nOverall, it is a nice work that brings the extrapolation or outpainting task to a new level. I have following concerns:\n\n(1) It looks the proposed pipeline share similar idea with some early autoregressive image generation models like PixelRNN or PixelCNN which are generating each pixel iteratively based on the previously generated pixels, though this work is operating on the patch level. What are the main differences compared with these related works?\n\n(2) Is the proposed work mainly for unconditional generation, based on an optional text prompt? Could it be used for conditional generation, like image extrapolation to extend its original field of view?\n\n(3) To what extent could the long-range dependency between patches be learned? Or how far is the relation of two patches modeled? I understand it might be not across all image regions but it is better to show and analysis on this so that it could help control the generation steps or image resolution for the best quality.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept