InfinityStar: Uniﬁed Spacetime AutoRegressive Modeling for Visual Generation
Authors: Jinlai Liu
Jian Han
Bin Yan
Wuhui
Fengda Zhu
Xing Wang
Yi Jiang
BINGYUE PENG
Zehuan Yuan
Venue: NeurIPS 2025 oral ()
Keywords: Visual Synthesis
Visual AutoRegressive Modeling
Text to Video Generation
Generative models
Primary area: deep_learning
TL;DR: InfinityStar is the first discrete autoregressive video generator capable of producing industrial-level 720p videos.

Abstract: We introduce InfinityStar, a unified spacetime autoregressive framework for high-resolution image and dynamic video synthesis. Building on the recent success of autoregressive modeling in both vision and language, our purely discrete approach jointly captures spatial and temporal dependencies within a single architecture. This unified design naturally supports a variety of generation tasks such as text-to-image, text-to-video, image-to-video, and long-duration video synthesis via straightforward temporal autoregression. Through extensive experiments, InfinityStar scores 83.74 on VBench, outperforming all autoregressive models by large margins, even surpassing diffusion competitors like HunyuanVideo. Without extra optimizations, our model generates a 5s, 720p video approximately 10$\times$ faster than leading diffusion-based methods. To our knowledge, InfinityStar is the first discrete autoregressive video generator capable of producing industrial-level 720p videos. We release all code and models to foster further research in efficient, high-quality video generation.

InfinityStar: Unified Spacetime AutoRegressive Modeling for Visual Generation
Jinlai Liu", Jian Han",
Bin Yan
Hui Wu,
Fengda Zhu,
Xing Wang
Yi Jiang, Bingyue Peng,
Zehuan Yuan
ByteDance
QLiujinlai.1ie licio,cciiii hanjian. thu123, bin. yan, wuhui. wwhui.321, (en.ul...,,,,,,eeeaaiaaa teae.aeaeegggguuuuuuuttaaaaa... com,
{xing. wraing.wang.o wang wanggwann..jaaagn. piaagni..nnn.. bingyue peng, peng, senggayaanzhhhanayyyyeaaeee...
Codes and models: https: htops://githin http://tt....///ttaaaa/////////t
Abstract
We introduce InfinityStar,, a unified spacetime autoregressive framework for high- resolution image and dynamic video synthesis. Building the recent success of autoregressive modeling in both vision and language, our purely discrete approach jointly captures spatial and temporal dependencies within single architecture. This unified design naturally supports variety of generation tasks such as text- to-image, text-to-video, image-to-video, and long-duration video synthesis via straightforward temporal autoregression. Through extensive experiments, Infini- tyStar scores 83.74 on VBench, outperforming all autoregressive models by large margins, even surpassing diffusion competitors like Hunyuan Without extra optimizations, our model generates 5s, 720p video approximately 10x faster than leading diffusion-baas methods. To our knowledge, InfinitySSar is the first discrete autoregressive video generator capable of producing industrial-level 720p videos. We release all code and models to foster further research in efficient, high-quality video generation.
1 Introduction
Visual synthesis has witnessed remarkable progress in recent years, largely propelled by the scaling of Transformer architectures. In particular, video generation has attracted growing interest from both academia and industry, owing its wide-ranging applications in content creation, world simulation, etc. At present, diffusion models[3, 17, 16, 26, 3,] lead the field by iteratively denoising latent representations to produce high-fidelity clips. Concurrently, autoregressive models[15, 28, 8] have been explored for their potential to unify image and video generation and generalize over longer time horizons.
Despite their successes, each paradigm exhibits critical shortcomings. Video diffusion models excel synthesizing fixed-length frame sequences by exploiting bidirectional attention, yet they incur substantial computational cost due to tens or even hundreds of sequential denoising steps, and they struggle to extend seamlessly to video extrapolation. Autoregressive methods based on next-token prediction, while inherently capable of streaming generation, often fall short in visual fidelity and suffer from prohibitive latency due tens of thousands of inference steps.
These observations motivate the need for a generation framework that simultaneously possess high visual quality, efficiency and temporal generalization. Recently, Visual AutoRegressive modeling (VAR)[23] redefined image generation as a coarse-to- fine next-scale prediction. Its follow-up work, Infinity [12] further introduces bitwise modeling and scales up the vocabulary size, achieving
Equual contribution. *Corresponding author: Saanzzhuunn@byeddnncee.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

comparable performance to diffusion models while offering significant advantages in inference speed. Inspired by the success of VAR [23] and Infinity [12]. we present InfinityStar, a Spacetime Pyramid Modeling for unified text-to-image, text-to-video, zero-shot image-to-videns and zero-shot video extrapolation. This framework models video as an image pyramid and multiple clip pyramids, not only naturally inheriting the text-to-image capabilities but also decoupling static appearance from dynamic motions videos. Furthermore, we introduce several key improvements. First, we improve discrete reconstruction quality by leveraging knowledge inheritance from continuous video tokenizer. Second, we introduce Stochastic Quantizer Depth during training of the tokenizer to alleviate the imbalanced information distribution across scales. Third, propose Semantic Scales Repetition, which refines the predictions of earlier semantic scales in a video, significantly enhancing fine-grained details and complex motions of the generated videos.
We train Inai on large-scale video corpora to support up to 720p resolution and variable durations. On the VBench benchmark[36], InfinityStar establishes a new state-of-the-art among autoregressive video models, even surpassing industry-leading Hunyuan Video[[6] (83.74 v.s 83.24). Besides, Infinity Star shows great advantage in terms of speed. Using visual tokenizers of the same compression rate, InfinityStar achieves a 10x reduction in inference latency relative to leading diffusion models.
In summary, the main contributions of our work are as follows:
1. We propose InfinityStar, a novel spacetime pyramid modeling framework that unifies diverse visual generation tasks, demonstrating superior flexibility and versatility.
2. InfinityStar is the first discrete autoregressive model capable of generating high-quality videos, outperforming existing autoregressive text-to-video models and matching the perfor- mance of leading diffusion models.
3. Compared to the inefficiency of existing autoregressive models and diffusion models, Infini- tyStar significantly accelerates high-quality video generation.
2 Related Work
2.1 Video Diffusion Models
Diffusion models excel at generating high-fidelity data by gradually denoising random noise and can naturally extend to video generation. Early attempts [2, 4, 34] are built on U-Net architectures, demonstrating the feasibility of this approach but falling short in producing sharp, temporally coherent frames due to limited model capacity. The advent of Diffusion Transformers (DiT [19]) marked turning point. SORA [3] harnessed DiT's scaling ability to process spatio-temporal patches at scale, dramatically improving both video consistency and generation quality. Inspired by SORA's success, industry efforts [32, 16, 26] have further advanced the field, pushing video generation to new heights. Although video diffusion models deliver outstanding quality, their slow generation speed hinders the production of high-resolution, mun-duration videos.
2.2 Video AutoRegressive Models
Another class of methods [28, 8, 15] employs autoregressive models for video generation. Inspired by the success of LLMs, these works predict video tokens in specific orders using autoregressive Transformer. For example, Emu3 [28] performs next-token prediction along both spatial and temporal axes, while NOVA [8] first predicts spatial tokens set-by-set and subsequently proceeds frame-by- frame in the temporal dimension. Although achieving preliminary progress, they require hundreds to thousands of inference steps, resulting in prohibitively low generation efficiency. In contrast, recent advances in next-scale prediction [23, 12] have demonstrated state-of-the-art performance in image synthesis, offering both improved quality and markedly faster inference. In this work, we extend the next-scale prediction paradigm to the unified tasks of image and video generation.
2.3 Discrete Video Tokenizers
For long time, discrete [31, 33] and continuous [16, 32, 26] video tokenizers have been developed independently. Although some works [1, 27] provide both discrete and continuous tokenizers, the

---

Image Pyramid
Clip Pyramid T H W
2 Clip Pyramid
7 x.M.
Pred
Spacetime Autoregressive Transformer
midi
text
text to image
text video
image to video
video extrapolation
Figure I: Spacetime pyramid modeling of InfinityStar. Built with an unified autoregressive pipeline, InfinityStar capable of performing text-to-image, text-to-video, image-to-video, video extrapolation tasks all in one model.
network configurations are usually not aligned. For example, Cosmos [1] chooses 6 and 16 as latent dimensions in its discrete and continuous variants respectively. This misalignment hinders the knowledge reuse between two types of tokenizers. As result, most mainstream discrete video tokenizers are either trained from scratch [1] or starting from a pretrained discrete image tokenizer [33, 27]. However, these training strategies have the following drawbacks. First, training from scratch is inefficient and converges slowly. Second, weights pretrained on static images are not optimal for video reconstruction. To alleviate these deficiencies, which inherits the architecture and knowledge of a trained continuous video tokenizer. Experiments show that this strategy significantly boosts the convergence of discrete video tokenizers.
propose new training strategy,
3 InfinityStar Architecture
3.1 Preliminaries
Infinity for Image Generation. Infinity [12] decomposes an image into sequence of hierarchical token blocks using a visual tokenizer and models the relationship between tokens by visual autoregressive Transformer (VAR Transformer). To cover images of various sizes, Infinity pre-defines list of token block sizes {(h,, w1), (/ux, wx)). called scale schedule. The size (hi, w:) in scale schedule grows as increases, forming a pyramid-like structure, which we refer as image pyramid in later discussion. Next we introduce the training and inference procedure of Infinity.
In the first training stage, a visual tokenizer learns reconstruct the raw image and compress into a sequence of discrete tokens, which can be modeled by the VAR Transformer in the next stage. Specifically, the tokenizer first encodes the raw images into compact latents, then transforms latents into K discrete residual token blocks (r1, 72, ..., using a bitwise multi-scale residual quantizer [12]. Each token block r; consists of h; X w; discrete tokens of d-dim with vocabulary size of 2d Then in the second stage, VAR Transformer is trained to predict next residual token block F사 conditioned on text embedding (1) and former tokens blocks T'ck- Formally, in each step, VAR Transformer predicts conditional probability p(rk|rek, ()))). During the inference, Infinity generates an image by running the VAR Transformer K times autoregressively, merging the predicted tokens and running the tokenizer decoder once.
3

---

3.2 Spacetime Pyramid Modeling for Unified Generation
Extending the spatial-only next-scale prediction paradigm of Infinity [12] to video generation presents primary challenge: how to incorporate the temporal dimension. The straightforward strategies are either letting time grows uniformly, i.e., from (1, 1, 1) to (T, H, W). or keeping time constant, Le., from (T. 1) to (T, H. W). We empirically found that letting time grow uniformly produces flickering videos. As for the constant time pyramid, we refer to it as the pseudo-spacetime pyramid. Despite its conceptual simplicity, it suffers from two fundamental limitations. First, the treatment of videos differs markedly from that of images, preventing text-to-video (T2V) model from effectively leveraging the knowledge learned by text-to-image (T2I) model and complicating its extension to tasks such as image-to-video (12V). Second, because appearance and motion in videos are coupled in this design, the model faces significant difficulty in accurately fitting both aspects.
To overcome these challenges, we propose a novel spacetime pyramid modeling framework as shown in Fig. I. Each video is decomposed into sequential clips {c1. c2. ,,N, ). We regard the first frame as Ci (i.e., T = 1) to encode video main static appearance cues specifically and other clips share an equal duration T > 1. Each clip is modeled as a 3D volume pyramid similar as Infinity [12]. In particular, for each clip, there are K scales with each represented as residual token block F of (T, hg. w&) dimension. It is worth noting that all scales in the pyramid are extended only in spatial dimension instead of time. Mathematically, the clip tokens are generated auto-regressively across scales as:
K
p(r rk)
II p(rk 1r (.)) km1
(I)
For inter-clip predictions, clips are generated sequentially conditioned prior clip predictions and the text input in an autoregressive manner. In this way, we could generate infinitely long videos theoretically. Formally, the autoregressive likelihood of the whole video can be expressed as:
K
p(rl
x = IIII p(r ri........ (()).
c=1k=1
3.3 Visual Tokenizer
Training video tokenizers faces greater challenges than training image tokenizers. First, training tokenizers on videos of tens of frames is much computationally heavier than training static images. Therefore, training a video tokenizer from scratch is extremely time-consuming and suffers from slow convergence. Second, the scale schedule in videos leads to more imbalanced information distribution, where most information is concentrated in the last few scales. This brings great difficulties to the optimization of VAR Transformer. To solve these challenges, we introduce two techniques, knowledge inheritance from continuous video tokenizer and stochastic quantizer depth.
Knowledge Inheritance from Continuous Video Tokenizer. Instead of designing and training a discrete video tokenizer from scratch, we inherit the architecture and weights of trained continuous video tokenizer, i.e. video VAE. Specifically, we first insert parameter-free quantizer between the pre-trained VAE encoder and the decoder. The quantizer based on binary spherical quantization [38], being similar to that of Infinity [1 [12] but with new spacetime pyramid scale schedule. This does not introduce any new parameter like codebook in VQ [24] and well retains knowledge of the original VAE. As shown in Fig.5, the discrete video tokenizer reconstructs videos decently, even without any fine-tuning. To further improve the reconstruction quality, we fine-tune the new tokenizer jointly on images and videos like previous works [27, I]. During the fine-tuning, the KL loss of the original VAE is replaced with the commitment loss plus the entropy penalty [38]. As shown in Fig.5, with the help of knowledge of continuous video VAE, the convergence accelerates dramatically.
Stochastic Quantizer Depth. When tokenizing videos using the spacetime pyramid schedule, the information distribution on different scales gets extremely imbalanced. Specifically, there are only a few tokens in the early scales, while there are tens of thousands of tokens in the last scales. Thus the tokenizer tends to reconstruct videos solely relying on tokens from the last few scales and not to learn useful representation in early scales as shown in Fig.6 (left). However, this imbalanced distribution difficult to model using VAR Transformer because the dependence between the latter token blocks and the former ones is weak. To alleviate this problem, we propose regularization called stochastic quantizer depth. During training, each one of the last N scales has probability p of being discarded.

---

In this way, there are 2d possible scale schedules during training. This requires the tokenizer to reduce the reliance on last scales and store more information in tokens of early scales. As in Fig.6 (right), with the help of this regularization, the reconstruction results of early scales become much clearer. This balanced information distribution makes the training of VAR Transformer easier.
3.4 Spacetime Autoregressive Transformer
To accommodate the newly introduced temporal dimension, enhance the quality of generated videos, and alleviate the substantial computational overhead associated with large number of tokens, we propose the following modifications to the VAR Transformer: Spacetime RoPE, Semantic Scale Repetition, and Spacetime Sparse Attention. We put Spacetime RoPE in the appendix.
Semantic Scale Repetition. With carefully crafted positional encodings, InfinityStar can already generate videos of acceptable quality. However, we observe that the structural coherence and motion dynamics in these outputs remain suboptimal. As shown in Figure 6, the overall layout and the placement of foreground objects are determined by the early scales of the clip pyramid-whi what term the "semantic scales." This observation motivates semantic scales. To this end, we introduce simple yet effective technique called semantic scale repetition. Concretely. if a clip pyramid comprises K scale tuples, we repeat the first K, tuples N times, thereby reinforcing the semantic-level representations. undergoes multiple rounds of refinement, improving the generation quality of semantics and the performance in complex scenarios with large motion. Given that the tokens at these early scales account for only a small fraction of the total token count, the additional computational overhead incurred by repeating them is negligible.
to enhance generation fidelity at these
this way, every earlier residual rk
Cb2
VAR-ikke Attention mask
Switti-lkke Attention mask
Spacetime Attention maek
Figure 2: Illustration of three causal attention variants. We plot three pyramids on the scale size (1,2,3) for visualization simplicity. From left to right, VAR block-wise causal mask with full history. Switti block-wise non-causal mask with full history, and spacetime sparse attention.
Spacetime Sparse Attention. Autoregressive video generation faces significant challenges due to the high computational costs of long context. As on the left of Fig.2, Infinity [12] employs a block- blsek-wise causal mask for single pyramid modeling. Switti [25] verifies that conditioning next-scale predictions solely on imputs from preceding scales is sufficient, resulting in a sparser attention mask as on the middle of 22g.2. For long video generation, it's necessary to attend history tokens to achieve temporal consistency. However, attending full history leads to an explosively long sequence. Considering each clip corresponds to 5s, which is sufficient maintain temporal consistency, here only attend to the last scale of the preceding clip. Finally, we obtain a highly sparse attention as show in Fig.2 (right). Our spacetime sparse attention drastically reduces attention computational overhead during both training and inference, all while delivering better performance.
Experiment
4.1 Implementation
Datasets. The training data of InfinityStaa includes text-to-image data and text-to-video data. We curated 130M pretraining and 70M high-quality text-to-image data. To balance the data distribution and improve overall aesthetics, we also involve 5M high-quality synthetic data. In terms of text-to- video data, we curated around 16M video data. All videos are longer than seconds. Among them 13M videos under x 192 resolution used for pre-training. They mainly from Panda-70M[6],
5

---

OOOOOO
Jonath brown Happpars sctting indieeines obyants slightly her speaking.
dark-cotored necilace. blreed,
Adiibs shiri and bluc bolding sbutchleok with both hands. He standing outdoors front building with bluc facedo and gevenory. The flipping though pages dutcchook points difTennt drawings
gendy
pliding
expensivv feeling tanpilit
the smontilly
lanéscape abrupt perepective peesenting
Figure 3: Text to image and text to vidco examples.
Mi]a[14], and other internal video-text pairs. Apart from those 192p videos, we also curated 3M 480p and 50K 720p high-quality videos for fine-tuning.
Model and Training. After inserting the patchify and unpatchify layers between Wan 2.1 VAE's encoder and decoder, we obtain a video tokenizer with compression rate of x 16 x 16 and a latent dimension 64. Multi-scale BSQ quantization adopted to obtain discrete tokens. contrast to using vocabulary size of 264 for all scales, scales and 264 for the latter large scales. We empirically find that it boosts convergence and has negligible impact on the reconstruction quality. Starting with the pretrained weights of Wan 2.1 VAE, the discrete tokenizer is fine-tuned jointly on images 256 256, 512 512, 768 x 768 and videos of 256 256 81 for 30K iterations. The learning rate is 1e-4.
vocabulary size of 26 for the former small
The autoregressive Transformer of InfinityStari is trained progressively four stages, including T2I pre-training and three T2V fine-tuning on 192p, 480p, 720p respectively. Each time we increase the training resolution, we preserve scale schedule of lower resolutions and append several larger scales, which enables better inheritance. The global batch size for 192p is 2048 and that of 480p and 720p is 1024. The learning rate for 192p is 2e Then we decay it to le -4 for 480p and 720p. We train the model on videos of 192p, 480p. 720p for 50K, 8K, 3K iterations, respectively. Specifically, each clip pyramid is composed 80 frames 16 fps, and the first K, 12 semantic scales are repeated by N times.
4.2 Text-to-Image Generation
The upper part of Fig.3 shows generated images from our InfinityStar- model, showcasing InfinityStar's strength in generating high-fidelity and photo-realistic images across various categories and image styles. We also carry out the quantitative evaluation on the GenEval[1]] [1]] and DPG[13] benchmarks. As Tab. 1, InfinityStar achieves the best overall score of 0.79 on the GenEval bench with prompt rewriter. It's worth noting that InfinityStar exceeds Infinity by 6% on overall score. We attribute the significant improvement the larger model size and the architectural innovations. On the DPG bench, Itai reaches overall score of 86.55, surpassing Infinity by 3.09%. These

---

Table 1: Evaluation the GenEval [11] and DPG [13]
benchmark. f result is with prompt rewriting.
GenEvvaf
DPGT
Methods
Parums
Two Ohj.
Position
Color Attri.
Overall
Global
Relation
Overall
Diffusion Models
SDXL [20]
2.6B
0.74
0.15
0.23
0.55
83.27
86.76 86.39
74.7
PixArt-Sigma [5]
0.6B
0.62
0.14
0.27
0.55
86.89
80.5
S133 (d=38) [9]
0.34
0.71
SANA-1.0 [29]
1.6B
0.66
84.8
FLUX-dev [18]
12B
0.67
84.0
H..UX-schmell [18]
0.71
84.8
AutoRegressive Models
LammaGen [21]
0.88
0.07
65.2
Chameleon [22]
7B
0.39
Show-o [30]
1.3B
0.80
0.31
0.50
0.68
67.5
Emnu3 [28]
8B
0.817
0.49"
0.451
0.661
81.6
Infinity (12)
0.857
0.49
0,571
0.731
93.11
90.76
83.46
InfinityStar-TT
0.90
0.62
0.79
91.68
91.87
86.55
quantitative results demonstrate InfinityStar's strong capabilities of image generation following users' prompts.
4.3 In the lower part of Fig.3, we present the generated videos of InfinityStar regarding user prompts. The generated videos successfully capture the semantic information in user prompts while maintaining high aesthetics and visual quality. Especially for the second example in Fig.3, the generated video accurately restores the delicate movements the characters flipping through sketchbooks, talking while pointing to different parts of the drawings. In Tab.2, we compare InfinityStar with leading diffusion and autoregressive approaches on VBench-a comprehensive video benchmark spanning 16 evaluation dimensions. Our model achieves overall score of 83.74. outperforming all open- source autoregressive baselines by substantial margin. Moreover, InfinityStar surpasses diffusion- based competitors such OpenSora[39], Cog Vi2coX[32], and Hunyuan Video These results demonstrate that, through its novel spacetime autoregressive design, Infinity capabilities of discrete autoregressive video models but also attains performance par with and in some cases superior to state-of ofatt--oftthear the-art diffusion methods.
Generation
Star Star not only pushes the
Intrapoined Video
10s
realisii enarpphoee cassaily
follow
backgrund, clithing wall patioms
slowly sct againa beight indoer buckgrond
year
appbing moosunazng lipstck,
adocning
Figure Zero-shot video extrapolation examples.
Zero-shot Generation. Although trained exclusively T2V data, InfinityStar can generate videos conditioned on an image or video as historical without any fine-tuning. Fig.4 shows video ex- trapolation results. The synthesized videos exhibit strong temporal coherence with the reference while faithfully capturing the semantic nuances of texts. Zero-shot I2V samples are presented the appendix.
4.4 Ablation Study
Visual Tokenizer. As shown in Fig.5 and Tab.3, loading weights of continuous video tokenizer significantly speeds up the convergence and achieves the best reconstruction results. As shown in Fig.6, stochastic quantizer depth largely improves the reconstruction quality of early scales. In terms of generation, using VAE with SQD leads to notable improvement in VBench scores (81.28 v.s. 81.07 as shown in Tab.4). Moreover, we observe that SQD contributes to faster convergence during the vidco generation training.

---

Table 2: Evaluation on the VBench benchmark, i result with prompt rewriting.
Human Action
Multiple Objects
Appear. Style
Quality Score
Semantic Score
Models
Params
Scene
Overall
Diffusion Models
AnimateDiff-Ve V2
1.5B
92.60
50.19
22.42
69.75
80.27
VideoCrafier-220(44
1.5B
95.00
55.29
25.13
73.42
OpenSora V1.2[39]
1.IB
85.80
42.47
58.41
23.89
80.71
73.30
79.23
Show-1[35]
95.60
45.47
23.06
72.98
78.93
Gen-3 [10]
96.40
54.57
53.64
24.31
84.11
75.17
82.32
Cog VideoX-5B|32]
99.40
53.20
24.91
82.75
77.04
81.61
Hunyuan [
13B
94.40
53.88
68.55
19.80
85.09
75.82
83.24
Wan 2.1
14B
98.80
53.67
81.44
21.13
85.64
80.95
84.70
AutoRegressive Models
[8]
0.6B
95.20
54.06
77.52
20.92
80.39
79.05
80.12
Emu3[28]
8B
77.71
37.11
44.64
20.92
84.09
68.43
80.96
InfinityStart
8B
96.43
52.08
78.66
21.81
84.73
79.78
83.74
Oriiial
PNN
301
2002
Wahattonaadddee veigle
Trald with www.erpeettteee......... wigho
Figure 5: Influence of pretrained weights on reconstruction and convergence. The left sub-figure shows the reconstructed frames using different pretrained weights wi without finetuning. Loading weights of continuous video tokenizer achieves the best results. The right sub-figure shows that training with pretrained video tokenizer converges significantly faster than the other two strategies.
Table 3: Reconstruction metrics internal high-motion video benchmark (480p 81 frames).
Pretrained Weights
PSNR(1)
SSIM(1)
LPIPS(J)
Continuous Video VAE
33.37
0.94
0.065
Image VAE
29.10
0.90
0.123
None
30.04
0.90
0.124
2t
3r
Global Semantic Info
S4
S4
S6
S6
5g
Sg
Visual
S11
$11
Details
Without Stochastic Quantizer Depth
With Stochastic Quantizer Depth
Figure The influence of stochastic quantizer depth. Sub-figure (s,, nt) represents the reconstructed frame nt using all tokens from the image pyramid plus tokens of first i scales in the clip pyramid. SQD significantly improves the reconstruction quality of early scales. Besides, the earlier scales correspond to global semantics, while the later ones responsible for local visual details.
Pseudo-Spacetime Pyramid v.s. Spacetime Pyramid. As illustrated in Fig.7, videos generated by pseudo-spacetime pyramid lack visual details and deliver simpler motion. contrast, spacetime pyramid generates videos with richer details and higher motion. Besides, spacetime pyramid improves VBench's overall score from 80.30 81.23 as illustrated in Tab.4. These experiments support hypothesis that spacetime pyramid could decouple appearance and temporal information. The image

---

Single Pyramid
Image Clips
man stands well-lia kiccen with white cohinets, large window, and various kitchen items the counter and shelves. He gestures speaking, then turss towards pot
pot.
Single Pyramid
Image Clips
system citysape with huildings and river, pertially abscural by fog. car moves from right lowseds left.
foreground, mountiinous with greemery. bookground.
Figure 7: Comparison between Pseudo-Spacetio Spacetime Pyramid and Spacetime Pyramid. Spacetime Pyramid could generate videos with richer details and higher motion.
pyramid corresponds to appearance information and clip pyramids focus on subsequent motions. This decoupling makes it easier to learn video motions. In addition to advances in performance, spacetime pyramid could unify T21, T2V, 12V tasks into one framework.
Semantic Scale Repetition. In Fig.6, we can observe that the earlier scales correspond to semantic information, while the later ones are responsible for high-frequency details. Here we compare the generation results with and without semantic scale repetition. As shown in Fig.8, semantic scale repetition highly effective in improving the structure stability and motion quality. The quantitative results further confirm the significant gains. As shown in Tab.4, semantic scale repetition improves VBench's overall score from 75.72 to 81.28.
Spacetime Sparse Attention. In Tab.4 and Tab.5, we compare different attention mechanisms. Spacetime sparse attention shows superior performance full attention in the Vbench total score (81.28 v.s. 80.77), while showing significant advantage in saving computation and GPU VRAM. SSA reaches 1.5x speedup when generating 192p 161 frames. The efficiency advantage becomes larger as the resolution and duration grow. For 480p 161 frames, full attention fails due to OOM while SSA completes it within 44.7s using 63GB VRAM. We hypothesize that SSA produces better results than full attention because it reduces exposure bias. Full attention is more susceptible to accumulated errors. The reason we do not condition on smaller scales of the preceding clip is that misses the former clips' visual details and brings visual inconsistency between clips. Although it reaches 1.1x 1.5x speedup for 192p and 480p 161 frames, we observe a significant performance drop in Vbench from 81.28 to 80.75 as shown in Tab.4. Therefore, the proposed spacetime sparse attention strikes a better balance between computational efficiency and visual quality.
4.5 Inferency Latency
As shown in Tab.6, we report the end-to-end inference latency measured on single GPU, including both the text encoder and VAE decoder. [26- and Nova[8] were evaluated using their default GitHub configurations. Even without employing stronger compression, InfinityStar achieves a 32x speedup over Wan-2.1. Furthermore, despite its larger model size, InfinityStar delivers 6x speedup compared to Nova. These results highlight our model's significant efficiency advantage over both diffusion and autoregressive approaches.
5 Conclusion and Limitation
We introduce Star, unified spacetime autoregressive framework capable of synthesizing high-resolution images and dynamic, high-motion videos. By seamlessly integrating spatial and temporal prediction within a purely discrete architecture, InfinityStar supports diverse generation tasks while maintaining both state-of th-tt--artttttt quality and exceptional efficiency. Our extensive evaluation

---

W/ SSR
SSR
with SSR
Figure 8: Semantic Scale Repetition (SSR) greatly improves structure stability and motion quality.
Table 4: Comprehensive ablation studies. Experiment with 1M 192p training data, batch_sttt size = 40, and 30K iterations. We evaluate the results on the Vbench benchmark.
total score
quality Score
semantic
Vbench
score
InfinityStar (Our Model)
81.28
81.56
80.16
Attend to former clip's largest scale
Ablation by removing/replacing core components
w/o Semantic Scale Repetition(SSR,
75.72
76.73
71.68
w/o Spacetime Pyramid (using Pseudo )pruud--paceemme)
80.30
80.81
78.28
w/o Stochastic Quantizer Depth(SQD)
81.07
81.21
80.54
Comparison of different Attention Mechanism variants
Full Attention
80.77
81.15
79.23
Attend to former clip's 3rd largest scale
80.86
81.26
79.26
Attend to former clip's 6th largest scale
80.75
80.98
79.80
Table 5: Computational efficiency comparison of attention mechanisms on single GPU.
(192p 65 frames)
(192p 161 frames)
(480p 161 frames)
Full Attention
8.6s 40.8GB
24.3s / 57GB
0OM
Attend former clip's largest scale
7.7s 38.5GB
16.7s / 40GB
44.7s 63 GB
Attend former clip's 3rd largest scale
7.4s 38.2GB
15.8s 39GB
34.5s 58 GB
Attend to former clip's 6th largest scale
7.3s 37.9GB
15.2s / 38GB
30.5s / 55GB
Table 6: Computational efficiency comparison.
Method
Model
# Parameters
Durations(s)
Frames
Resolution
Time(s)
Speedup
Diffusion
Wan 2. [[66]
14B
5
81
720p
1864
AR
Nova[8]
0.6B
81
480p
354
AR
InfinityStar
8B
5
81
720p
58
32
demonstrates that InfinityStar outperforms prior autoregressive video models and rivals leading diffusion-based approaches, producing a 720p video of 5s in one-tenth the inference time. As the first discrete autoregressive model to deliver industrial-gradd 720p video synthesis, we anticipate that InfinityStar will catalyze future research rapid, long video generation.
While InfinityStar sets new record in discrete video generation, several limitations remain. Specifi- cally, there is trade-off between image quality and motion fidelity in high-motion scenes, where sometimes grained visual details can be compromised. Additionally, due to limited computa- tional resources, we have not scaled our model training or parameter size to match those of leading diffusion models, which constrains the upper bound of the performance. Furthermore, our inference pipeline has not yet been fully optimized, indicating room for future improvement.
10

---

6 Acknowledgments
The authors appreciate the valuable support provided by colleagues from ByteDance, including Yuqi Zhang, Yifu Zhang, Hao Yang, Yifei Hu, Chuang Lin, Xiaofeng Mei, Ruibiao Lu, and Jiawei Duan. Their contributions to data processing and related technical aspects are essential for the advancement of this research.
References
[1] N. Agaraal, A. Ali, M. Bala, Y. Balaji, E. Barker, T. Cai, P. Chattopadhyay, Y. Chen, Y. Cui, Y. Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint 2rX1v
2025. 2, 3, [2] A. Blattmann. T. Dockhom, S. Kulal, D. Mendelevitch, M. Kilian, D. Lorenz, Y. Levi, Z. English, V. Voleti,
A. Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datascts. arXiv
preprint arXix:2311.
2023. 2
[3] T. Brooks, Peebles, C. Holmes, W. DePue, Guo, L.. Jing, D. Schnurr, J. Taylor, T. Luhman, F. Luhman,
C. Ng. R. Wang. and A. Ramesh. Video generation models as world simulators. OpenAI, 2024. 2 [4] H. Chen, M. Xia, Y. He, Y. Zhang, X. Cun, S. Yang, J. Xing, Y. Liu, Q Chen, X. Wang, et al. Vidcocrafterl:
Open diffusion models for high-quality video generation, arviv preprint arXiv:2310. 195112 2023, 2, 8 [5] Chen, C. Ge, E. Xie, Y. Wu, L. Yao, X. Ren, Z. Wang. Luo. H. Lu, and Li. Pixart-sigma: Weak-to- strong training of diffusion transformer for 4k text-to-image generation. arXiv preprint arXiv:2403.0446 2024. 7
[6] T.-S. Chen, A. Siarohin, W. Mcnapace, F. Deyneka, H.-w. Chao, B. F. Jcon, Y. Fang, H.-Y. Lee, J. Ren, M.-
H. Yang. et al. Panda-70m: Captioning 70m videos with multiple cross- pross-modality teachers. In Proceedings
of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13320-13331, 2024.
[7] G. DeepMind, Vco 3. hutps://deppiind howpmmiinnggog//tteeeeeeeeeeeeee wtw.e//cccooooiiev//veoooooo
2025.00 05. [8] H. Deng. T. Pan, H. Diao, Z. Luo, Y. Cui, H. Lu, S. Shan, Y. Qi, and X. Wang. Autoregressive video generation without vector quantization. arXiv preprint arXiv:2412. 141.1919, 2024. 2, 9, 10 [9] P. Esser, S. Kulal, A. Blattmann, R. Entezari. J. Müller, H. Saini, Y. Levi, D. Lorenz, A. Sauer. F. Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
[10] A. Germanidis. Introducing gen-3 alpha: A new frontier for video generation. Runway, 2024,
[11] D. Ghosh, H. Hajishirzi, and Schmidt. Geneval: An focuse- framework for evaluating text-to-
image alignment. Advances in Neural Information Processing Systems, 36. 2024. 6,
[12] J. Han, J. Liu, Y. Jiang, B. Yan, Y. Zhang, Z., Yuan, B. Peng, and X. Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. arXiv preprint arXiv:2412.04431, 2024. 1, 2.
5.
[13] X. Hu, R. Wang, Y. Fang, B. Fu, P. Cheng, and G. Yu. Ella: Equip diffusion models with Ulm for enhanced semantic alignment. arxiv preprint arXiv:2403.00 05100. 2024. 6, 6.7
[14] X. Ju, Y. Gao. Z. Zhang. Z. Yuan, X. Wang. A. Zeng. Y. Xiong. Q Xu. and Y. Shan. Miradata: A large-scale
video dataset with long durations and structured captions. 2024. 6
[15] D. Kondratyuk, L. Yu, X. Gu, J. Lezama, J. Huang, G. Schindler, R. Hornung, V. Birodkar, J. Yan, M.-C. Chiu, K. Somandepalli, H. Akbari, Y. Alon, Cheng, J. Dillon, A. Gupta, M. Hahn, A. Hauth, D. Hendon, A. Martinez, Minnen, M. Sirotenko, K. Sohn, X. Yang, H. Adam, M.-HI Yang. Essa, H. Wang, D. A. Ross, B. Seybold, and L. Jiang. Videopoet: A large language model for zero-shot video generation, 2024.
1.2
[16] W. Kong, Q. Tian, Z. Zhang, R. Min, Z. Dai, J. Zhou, J. Xiong, X. Li, B. Wu, J. Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXiv:24/2.03300 2024. 1,2,
7, 8
[17] Kuaishou. Kling ai. https:/kkingaii kwwishou....iiccccoouo com// 2024.06.
[18] B. F. Labs. Flux. https: http://baacoffeestscoee
a1/ //annnnnnninggg
2024. 7
[19] W. Peebles and S. Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF
International Conference
Computer Vision, pages 4195-4205, 2023.
[20] D. Podell, 7. English, K. I Lacey, A, Blattmann, T. Dockhorn, J. Müller, J. Penna, and R. Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arX1v:2307.00952.
2023. 7
[21] P. Sun, Y. Jiang, Chen, S. Zhang, B. Peng. P. Luo, and Z. Yuan. Autoregressive model beats diffusion:
Llama for scalable image generation. arXiv preprint arXiv:2406.06522 2024. 7
[22] C. Team. Chameleon: Mixed-modal early- foundation models. arXiv preprint arXiv:2405.099
2024. 7
[23] K. Tian, Y. Jiang, Z. Yuan, B. Peng, and L. Wang. Visual autoregressive modeling: Scalable image
generation via next-scale prediction. arXiv preprint arXiv:2404.02900 2024, 1.2
[24] A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[25] A. Voronov, D. Kuznedelev, M. Khoroshikh, V. Khrulkov, and D. Baranchuk. Switti: Designing scale- wise
transformers for text-to-inne eex-to--image synthesis. arXiv preprint arXiv:2412.01819, 2024. 5
11

---

[26] A. Wang. B. Ai, B. Wen, C. Mao, C.-W. Xie. D. Chen, F. Yu, H. Zhao, J. Yang, Zeng, et al. Wan: Open and advanced large-scale video generative models. arXiv preprini arXiv:2503.203: 2025. 2, 8, 10 [27] J. Wang. Y. Jiang, Z. Yuan, B. Peng, Z. Wu, and Y.GG. Jiang. Omnitokenizer: joint image-vidco tokenizer for visual generation. Advancey in Neural Information Processing Systems, 37:28281-28295, 2024. 3,
[28] X. Wang. X. Zhang. Z. Luo, Q. Sun, Y. Cui, J. Wang. F. Zhang. Y. Wang. Z. Li. Q. Yu, al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409. 18869. 2024, 7.
[29] E. Xic, J. Chen, J. Chen, H. Cai, H. Tang, Y. Lin, Z. Zhang, M. Li, Zhu, Y. Lu, et al. Sana: Efficient high-resolution image synthesis with linear diffusion transformers. arXiv preprint arXiv:2410. 10629, 2024.
[30] Xie, W. Mao, Z. Bai, D. J. Zhang, W. Wang, K. Q. Lin, Y. Gu, Z. Chen, Z. Yang, and M. Z. Shou. Show-o: One single transformer to unity multimodal understanding and generation. arXiv preprint
arXiv:2408. 12528, 2024.7 7
[31] W. Yan. Y. Zhang. P. Abbeel, and A. Srinivas. Videogpt: Video generation using vg-vae and transformers.
arXiv preprint arXiv:2104. 101555 2021. 2021., 2
[32] Z. Yang. J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y. Yang. W. Hong, X. Zhang, G. Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408. 06072,
2024. 2, 7.
[33] L. Yu, Lezama. N. B. Gundavarapu, L. Versari, K. Sohn, D. Minnen, Y. Cheng, A. Gupta. X. Gu, A. Hauptmann, et al. Language model beats
is key to visual generation. arXiv preprint
Sti::2310.057377 2023. 2.3
[34] Y. Zeng, G. Wei, J. Zheng, J. Zou, Y. Wei, Y. Zhang, and H. Li. Make pixels dance: High-dynamic vidco generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition,
pages 8850-8860, 2024. 2
[35] D. J. Zhang. J. Z. J. J.-W. Liu, R. Zhao, L. Ran, Y. Gu, D. Gao, and M. Z. Shou. Show-1: 1: Marrying pixel and latent diffusion models for text-to-video generation. International Journal of Computer Vision,
pages 1-15, 2024.
[36] F. Zhang, S. Tian, Huang. Y. Qiao, and Liu. Evaluation agent: Efficient and promptable evaluation
framework for visual generative models. arXiv preprint arXiv:2412.094: 09645, 2024. 2
[37] Y. Zhang, H. Yang. Y. Zhang. Y. Hu, Zhu, C. Lin, X. Mei, Y. Jiang. B. Peng, and Yuan. Waver: Wave 2025.
your way to lifelike video generation. arXiv preprint 2rxiv
[38] Y. Zhao, Y. Xiong, and P. Krähenbühl. Image and video tokenization with binary spherical quantization.
arXiv preprint arXiv:2406.07555 07548, 2024. 4
[39] Z. Zheng, X. Peng. T. Yang, C. Shen, S. Li, H. Liu, Y. Zhou, T. Li, and Y. You. Open-sora: Democratizing 2024. 7,
efficient video production for all. arXiv preprint
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
This paper claims the contributions that we are the first discrete autoregressive model to achieve high-quality 720p video generation with the fastest generation speed in the abstract and introduction
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss our limitations Section 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiattinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer: [Yes]
Justification: Our proof, such as the autoregressive likelihood formula, has been validated and verified through experimental comparisons, demonstrating the effectiveness of the proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
We will open-source our model and code, and use publicly available data to guide people on how to reproduce our results, as training large-scale video generation models is extremely resource-intensive.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
14

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [No]
Justification: Currently, the code needs to be cleaned, and the data needs to be maintained as an open-source subset to ensure the community can follow along. We will soon open-source the code along with documentation for some of the data sources.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips.ccl wttppcccppppcccccccccccccccccccc hwwpuuululuuulssspeeepppbbaaaaal gww./s//eee//ee/// CooeeSummii issisni ssssssoliccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have outlined the multi-stages of our training, including the training steps and relevant training steps, and learning rate for each stage.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For both our benchmark and ablation study evaluations, we run multiple trials and report the average results to ensure the statistical significance the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
15

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We use internal clusters to train our models. In terms of estimation, the four-stage video generation process consumes 5,000, 40,000, 30,000, and 30,000 GPU hours respectively. The video VAE requires 2,000 GPU hours. The ablation study has total cost of 10,000 GPU hours, and the evaluation consumes 1,000 GPU hours.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: ptttps:/neeurpgee cc cc/public/ eshuid
Answer: [Yes]
Justification: We have clear filters in place for privacy, inappropriate and harmful content for both image and video data.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
16

---

Visual generation has broad impact on society. On one hand, video generation models significantly improve production efficiency. On the other hand, we closely monitor the potential societal risks related to the misuse of information by models trained on the data we use. We have filtered the harmful and copyrighted data in our training dataset
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [Yes]
Justification: First, our training data will undergo strict filtering, and we will ensure rigorous verification before open-sourcing to check for any obvious issues like harmful visual content. After open-sourcing, we will provide a model that includes a default safeguards model to filter any unsuitable generated content.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
17

---

In the paper, we specify the sources of the open-source data used and provide proper citations. Regarding the code, we will acknowledge the relevant contributors on our GitHub repository when we make it publicly available.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: In the appendix, we will provide more visual samples with details. All these materials will be anonymized. The code is our proprietary property, and the data collection is compliant.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
18

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: We use LLM for writing and formatting
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy (https: http://rpp.cccoooeeeeeeeeeeoeeee for what should or should not be described.
19


Reviews:
[{"review_id": "hWbGgLEOmu", "reviewer": "Reviewer_7YkR", "summary": "This paper proposes Infinity*, a unified spacetime autoregressive framework for image and video synthesis under a purely discrete, autoregressive pipeline. Building on the prior Infinity and VAR frameworks, it extends next-scale prediction to videos by introducing a spacetime pyramid that incorporates the temporal axis. To enable this extension effectively, the paper introduces: (1) knowledge inheritance from continuous video VAEs for efficient discrete tokenizer training, (2) stochastic quantizer depth to address information imbalance across scales in the tokenizer, (3) semantic scale repetition to improve the generation quality of semantic tokens in early scales, and (4) spacetime sparse attention to reduce computational cost. The framework supports text-to-image, text-to-video, image-to-video, and video extrapolation within a single model, achieving state-of-the-art or near state-of-the-art results. It outperforms prior autoregressive methods and even some diffusion methods such as HunyuanVideo while being 10× faster than leading diffusion models when generating 5s 720p videos.", "questions": "Regarding semantic scale repetition, I would like clarification on whether the early scales are generated multiple times (each round conditioned on all preceding rounds) with supervision applied only to the final round. Additionally, I am interested in understanding how much quantitative improvement this repetition design brings to generation quality.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1. The paper proposes a highly effective framework that extends next-scale prediction to video generation, enabling support for text-to-image, text-to-video, image-to-video, and video extrapolation tasks within a single unified model.\n\n2. The proposed framework achieves very strong performance, attaining state-of-the-art or near state-of-the-art results on benchmarks such as VBench and GenEval while significantly outperforming prior autoregressive methods and even surpassing some diffusion models like HunyuanVideo.\n\n3. I enjoyed reading this paper; it is well written, with clear and well-motivated design choices for both the tokenizer and transformer components, supported by ablation studies that validate these decisions.\n\nWeaknesses:\n\nThis is not strictly a weakness, but I am curious about the proposed Spacetime Sparse Attention design, which attends only to the last scale of the preceding clip. Since I assume the last scale contains the most tokens, I would like to understand how much computation this sparse attention actually saves and how much, if any, performance is lost compared to using full attention. I wonder why the current clip’s scales do not condition on the smaller scales of the preceding clip, which seems more natural and would likely save more computation. Has this alternative been tried, and how does it compare in terms of efficiency and quality?", "was_revised": true, "final_justification": "This work is a well-designed extension of VAR framework to video generation and achieves competitive performance. The authors conducted extensive experiments to address my concerns. I believe this work can benefit the community of AR video generation.", "rebuttal": "## Response to Reviewer ``7YkR`` (Rating: Accept)\n\nWe thank reviewers for your valuable comments on Infinity$^*$, acknowledging strong performance with fast speed (``7YkR``, \n``qR4k``, ``vUvY``, ``tNTA``, ``KwMu``), well-motivated design (``7YkR``, ``vUvY``, ``KwMu``), valuable contribution to the generative model community (``qR4k``, ``tNTA``, ``KwMu``), with solid experimental results (``qR4k``, ``KwMu``). Your advice will help us enhance the paper.\n\n#### **Q1:  About the proposed spacetime sparse attention, how much computation does it save and how much performance loss does it incur compared to full attention?**\nIn Table 1, we compare our spacetime sparse attention with full attention. Spacetime sparse attention shows superior performance to full attention in the Vbench total score (81.28 v.s. 80.77), while showing a significant advantage in saving computation and GPU VRAM. It's worth noting that spacetime sparse attention achieves 1.5$\\times$ speedup when generating 192p 161 frames. The efficiency advantage becomes larger as the resolution and duration grow. For 480p 161 frames, full attention fails due to OOM while spacetime sparse attention completes it within 44.7s using 63GB VRAM. We conclude that spacetime sparse attention significantly improves efficiency without performance loss. We hypothesize that only attending to the previous clip's largest scale yields better results than full attention because it reduces exposure bias. In full attention, since the context is all ground truth during training, it is more susceptible to accumulated errors during inference.\n\n\n\n| Method                                    | total score | quality score | semantic score |\n|---|:---:|:---:|:---:|\n| Full Attention                                 | 80.77       | 81.15         | 79.23          |\n| Attend to former clip's largest scale     | **81.28**       | **81.56**         | **80.16**          |\n| Attend to former clip's 3rd largest scale | 80.86       | 81.26         | 79.26          |\n| Attend to former clip's 6th largest scale | 80.75       | 80.98         | 79.80          |\n\nTable 1. Performance comparison of various attention mechanisms\n\n\n#### **Q2:  Why the current clip do not condition on smaller scales of the preceding clip, which seems more natural and would likely save more computation. Has this alternative been tried, and how does it compare in terms of efficiency and quality?**\n\nThe reason we do not condition on smaller scales of the preceding clip is that it misses the former clips' visual details and brings severe visual inconsistency between clips. As illustrated in Table 2, although conditioning on smaller scales  reaches 1.1$\\times$, 1.5$\\times$ speedup for 192p and 480p 161 frames, we observe a significant performance drop in Vbench from 81.28 to 80.75 as shown in Table 1. Our qualitative comparison also shows that attending to smaller scales bring obvious visual inconsistency between clips. Therefore, attending to the largest scale of the preceding clip strikes a better balance between computational efficiency and visual quality.\n\n|      Method                           | (192p 65 frames, 4s 16fps) | (192p 161 frames, 10s 16fps) | (480p 161 frames, 10s 16fps) |\n|---|:---:|:---:|:---:|\n| Full Attn                               | 8.6s / 40.8GB              | 24.3s / 57GB                 | OOM                          |\n| Attend to former clip's largest scale   | 7.7s / 38.5GB              | 16.7s / 40GB                 | 44.7s / 63GB                 |\n| Attend to former clip's 3rd largest scale | 7.4s / 38.2GB              | 15.8s / 39GB                 | 34.5s / 58GB                 |\n| Attend to former clip's 6th largest scale | 7.3s / 37.9GB              | 15.2s / 38GB                 | 30.5s / 55GB                 |\n\nTable 2. Computational efficiency comparison of attention mechanisms measured on a single NVIDIA H100 GPU with 8B parameters.\n\n\n#### **Q3:  Whether the early scales are generated multiple times with supervision applied only to the final round?**\nThat's not the case. As a matter of fact, repeated early scales are treated as independent scales in VAR's residual quantification procedure. Therefore, both the original scale and the $n$ repeated scales receive their own corresponding ground truth and supervision signals.\n\n#### **Q4:  How much quantitative improvement does this repetition design bring to generation quality?**\nAs shown in Table 3 , semantic scale repetition improves VBench's overall score from 75.72 to 80.97. More repetition literally yileds better results from 80.97 to 81.28. Here we set the repetition times $n=6$ to balance the performance and efficiency.\n\n| Method |  total score  |  quality score  |   semantic score   |\n|---|:---:|:---:|:---:|\n| w/o SSR | 75.72 | 76.73 | 71.68 |\n| With SSR (repetition=4) | 80.97 | 81.38 | 79.32 |\n| With SSR (repetition=6) | **81.28** | **81.56** | **80.16** |\n\nTable 3. Comparison of different repetition times in Semantic Scale Repetition (SSR)\n\n\n#### **Experiments Setup**\nAll our ablation experiments were conducted using a 1M training set with a batch size of 40, trained at 192p resolution for 30,000 iterations. We evaluated the results on the Vbench benchmark."}, {"review_id": "t6OcZsQBSS", "reviewer": "Reviewer_qR4k", "summary": "The authors present Infinity-star, a discrete autoregressive framework that unifies text-to-image, text-to-video, image-to-video and zero-shot video extrapolation. It extends the “next-scale” paradigm of Infinity to the temporal domain by decomposing a video into an image pyramid plus successive clip pyramids that grow only in space, allowing static appearance and motion to be modeled separately. The introduced model achieve competitive performance on benchmarks of several tasks. More importantly, the proposed approach significantly reduces inference speed compared to existing diffusion models.", "questions": "- It would be helpful if the author could list the sequence length of each sample (summing all scales) during training.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths: \n- The proposed model achieve fairly competitive results with much faster inference speed compared to diffusion/flow matching. \n- The paper is well-written, organized and technically sound. Several design details are included for reproducibility.\n- The author validate their motivation for extending spatial infinity to temporal dimension with simple baselines. They also report ablation study to demonstrate the effectiveness of proposed approach.\n- The open-source weight would be beneficial and have large impact for community \n\n## Weakness: \n- The author experiment with several design choices and chose the best one but only report qualitative results. It would be helpful if they can also report quantitative results for a clearer picture for reader i.e., to understand how significant of each component that the paper introduce.", "was_revised": true, "final_justification": "After reading other review and the author rebuttal, i believe this is a good paper and I don't have any major concerns. Therefore, I keep my initial score.", "rebuttal": "## Response to Reviewer ``qR4k`` (Rating: Accept)\n\nWe thank reviewers for your valuable comments on Infinity$^*$, acknowledging strong performance with fast speed (``7YkR``, \n``qR4k``, ``vUvY``, ``tNTA``, ``KwMu``), well-motivated design (``7YkR``, ``vUvY``, ``KwMu``), valuable contribution to the generative model community (``qR4k``, ``tNTA``, ``KwMu``), with solid experimental results (``qR4k``, ``KwMu``). Your advice will help us enhance the paper.\n\n\n#### **Q1:  The author experiment with several design choices and chose the best one but only report qualitative results. It would be helpful if they can also report quantitative results for a clearer picture for reader i.e., to understand how significant of each component that the paper introduce.**\nWe appreciate your suggestion to provide more quantitative results. In response, we have included additional ablation studies focusing on the proposed spacetime sparse attention, stochastic quantizer depth, semantic scales repetition, and Spacetime Pyramid  in Tables 1~4, respectively. The results further validate the contribution of each component to the overall performance of our method.\n\n---\n\n| Method                                    | total score | quality score | semantic score |\n|---|:---:|:---:|:---:|\n| Full Attention                                 | 80.77       | 81.15         | 79.23          |\n| Attend to former clip's largest scale     | **81.28**       | **81.56**         | **80.16**          |\n| Attend to former clip's 3rd largest scale | 80.86       | 81.26         | 79.26          |\n| Attend to former clip's 6th largest scale | 80.75       | 80.98         | 79.80          |\n\nTable 1. Performance comparison of various attention mechanisms\n\n---\n\n| Method |  total score  |  quality score  |   semantic score   |\n|---|:---:|:---:|:---:|\n| w/o SSR | 75.72 | 76.73 | 71.68 |\n| With SSR (repetition=4) | 80.97 | 81.38 | 79.32 |\n| With SSR (repetition=6) | **81.28** | **81.56** | **80.16** |\n\nTable 2. Comparison of different repetition times in Semantic Scale Repetition (SSR)\n\n---\n\n| Method   | total score | quality score | semantic score |\n|---|:---:|:---:|:---:|\n| w/o SQD  | 81.07       | 81.21         | **80.54**          |\n| With SQD | **81.28**       | **81.56**         | 80.16          |\n\nTable 3. Ablation study of the proposed stochastic quantizer depth\n\n---\n\n| Method                 | total score | quality score | semantic score |\n|---|:---:|:---:|:---:|\n| Pseudo-Spacetime Pyramid | 80.30       | 80.81         | 78.28          |\n| Spacetime Pyramid      | **81.28**       | **81.56**         | **80.16**          |\n\nTable 4. Ablation study of the proposed Spacetime Pyramid\n\n#### **Q2:  It would be helpful if the author could list the sequence length of each sample (summing all scales) during training.**\n\nAs shown in Table 5, we list the total sequence length for each sample by summing across all scales at different resolutions during training.\n\n|  Resolution | seq_len (81 frames) | seq_len (161 frames) |\n|:---:|:---:|:---:|\n| 192p      | 14,721              | 28,741               |\n| 384p      | 48,489              | 94,669               |\n| 480p      | 81,879              | 159,859              |\n| 720p      | 205,863             | 401,923              |\n\nTable 5. sequence length of training sample\n\n#### **Experiments Setup**\nAll our ablation experiments were conducted using a 1M training set with a batch size of 40, trained at 192p resolution for 30,000 iterations. We evaluated the results on the Vbench benchmark."}, {"review_id": "Zq8FEqMHuD", "reviewer": "Reviewer_vUvY", "summary": "This paper proposes a unified spacetime autoregressive framework for high-resolution image and dynamic video synthesis. Building on the success of VAR models, it models videos as an image pyramid and multiple clip pyramids, naturally decoupling static appearance from dynamic motion. Through key improvements including knowledge inheritance from continuous video tokenizers, stochastic quantizer depth, and semantic scale repetition, the model achieves  a good result on VBench, outperforming all VAR model and be comparible with Diffusion models.", "questions": "see weakness.", "limitations": "see weakness.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "## Strengths \n1. It is so impressive to see such a great dynamic degree on VBench. And achieve compariable performance as SOTA on VBench but get a much faster speed.\n2. Decomposes videos into image pyramids and clip pyramids, cleverly decoupling appearance and motion, enabling unified handling of T2I, T2V, I2V tasks.\n3. By inheriting weights from pretrained continuous video VAE, significantly accelerates discrete tokenizer convergence and improves reconstruction quality.\n4. Stochastic Quantizer Depth to address imbalanced information distribution, Semantic Scale Repetition to improve structural stability and motion quality, and Spacetime Sparse Attention to reduce computational overhead. The design of these module is novel and useful.\n\n## Weaknesses\n1. Authors mentioned that due to GPU and training data constraints, they didn't match the training scale of leading diffusion models like Wan 2.1, potentially limiting current performance.\n2. Experiments mainly focus on short videos, lacking validation of longer temporal generation capabilities.\n3. Performance still lags behind some diffusion models in certain evaluation dimensions (it is a weakness, but it is already very good for the VAR model, with considering a much faster generation speed.)", "was_revised": true, "final_justification": "Authors solved my concerns, I think this paper is well contributed and have a great novelty and impact, I suggest to accept.", "rebuttal": "## Response to Reviewer ``vUvY`` (Rating: Accept)\n\nWe thank reviewers for your valuable comments on Infinity$^*$, acknowledging strong performance with fast speed (``7YkR``, \n``qR4k``, ``vUvY``, ``tNTA``, ``KwMu``), well-motivated design (``7YkR``, ``vUvY``, ``KwMu``), valuable contribution to the generative model community (``qR4k``, ``tNTA``, ``KwMu``), with solid experimental results (``qR4k``, ``KwMu``). Your advice will help us enhance the paper.\n\n\n#### **Q1:   Authors mentioned that due to GPU and training data constraints, they didn't match the training scale of leading diffusion models like Wan 2.1, potentially limiting current performance.**\nYes, it is one of the limitations of our work. We have not yet matched the training scale of leading diffusion models like Wan 2.1 in terms of GPU resources and data. By scaling up both, our results shall keep improving.\n\n#### **Q2:  Experiments mainly focus on short videos, lacking validation of longer temporal generation capabilities.**\nThank you for the insightful comment. We acknowledge that the experiments in the submitted manuscript primarily focus on short video clips (less than 10 seconds). This is largely due to the limited availability of datasets containing long videos paired with rich, multi-segment captions. Nevertheless, our autoregressive framework is inherently more suitable for long video generation than diffusion-based models, which typically operate with fixed-length sequences. In future work, we plan to curate more appropriate data and extend our approach to thoroughly evaluate its capabilities in long video generation.\n\n#### **Q3: Performance still lags behind some diffusion models in certain evaluation dimensions(it is a weakness, but it is already very good for the VAR model, with considering a much faster generation speed.)**\nThank you for acknowledging our results. We agree that our method currently lags behind some leading diffusion models in certain aspects. Our work explores discrete video generation based on VAR and Infinity, opening up new possibilities in video generation field. While diffusion models have been extensively studied, we will continue to improve our approach to bridge the gap while maintaining our speed advantage."}, {"review_id": "ZtvhvbF8tM", "reviewer": "Reviewer_tNTA", "summary": "In this work, the authors propose Infinity*, a pyramid modeling autoregressive (AR) model for image and video generation. Infinity* builds a discrete vision tokenizer that tokenizes videos into discrete tokens in a pseudo-spacetime pyramid. In particular, each video is encoded into a a sequential tokens that models video clips of different scales. An AR model is trained on the pseudo-spacetime pyramid tokenizer with spacetime attention mask for efficient generation and time constant generation. Infinity* demonstrates competitive performance on text-to-image and text-to-video generation benchmarks. Also, it experimentally shows effectiveness on design choices like stochastic quantizer depth and semantic scales repetition.", "questions": "Major:\n1. The work mentioned pre-trained video VAE with continuous latents is used. How is the VAE pretrained? Is it from previous open-source work or trained from scratch? More details about the VAE are appreciated.\n2. For comparison of different models in Table 1/2, is it possible to include the scale of training data besides # params so that people can better compare Infinity* with baselines. \n3. Also in Table 2, it would be beneficial if # params of different models are listed as in Table 1. \n4. Are there quantitative ablation studies about the proposed stochastic quantizer depth and semantic scales repetition as well as how the hyperparameters are determined? For example, number of repetitions for semantic scales.  \n\nMinor:\n1. In page 5 line 197, it should be \"Patchify\" not \"Patchfy\".", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The paper is well written and easy to follow.\n2. The work shows how AR model can achieve competitive performance in text-to-video generation, which is a valuable contribution to generative model community. \n3. The work shows competitive performance on vision generation benchmarks.\n\nWeaknesses:\n1. Overall, the paper is largely built on previous VAR and Infinity and shows design choices that makes scale-wise AR model work for video generation. Though design choices like stochastic quantizer depth and semantic scales repetition help improve the generation quality, they may be adhered to certain AR model instead of benefiting vision generative models in general.", "was_revised": true, "final_justification": "The work has shown strong performance empirically with comprehensive experiments supporting the design choices after rebuttal. I believe this work can benefit building efficient AR-based video generation.", "rebuttal": "## Response to Reviewer ``tNTA`` (Rating: Borderline Accept)\n\nWe thank reviewers for your valuable comments on Infinity$^*$, acknowledging strong performance with fast speed (``7YkR``, \n``qR4k``, ``vUvY``, ``tNTA``, ``KwMu``), well-motivated design (``7YkR``, ``vUvY``, ``KwMu``), valuable contribution to the generative model community (``qR4k``, ``tNTA``, ``KwMu``), with solid experimental results (``qR4k``, ``KwMu``). Your advice will help us enhance the paper.\n\n#### **Q1:   The paper is largely built on previous VAR and Infinity and shows design choices that makes scale-wise AR model work for video generation. Though design choices like stochastic quantizer depth (SQD) and semantic scales repetition (SSR) help improve the generation quality, they may be adhered to certain AR model instead of benefiting vision generative models in general.**\n\nThanks for your suggestions. We agree that Infinity* is built on VAR and Infinity. However, both VAR and Infinity focus on image generation. Simply extending them to video generation yields inferior results, and there is currently no relevant work in academia. In this paper, we successfully solved the difficulty of handling the additional temporal dimension while preserving high efficiency. Compared to vanilla AR and diffusion models, Infinity$^*$ reaches 10$\\sim$30$\\times$ speedup and better generation quality. Therefore, we believe our work is non-trivial in fostering better and faster video generation.\n\nAs for the universality of SQD and SSR, we agree that they are designed for VAR video generation and may not be suitable for vanilla AR models. This lies in the hierarchical structure of VAR, which is different to vanilla AR. But these innovations are applicable to other hierarchical generation models, like FlowAR and FractalAR. We will explore applying SQD and SSR to other models.\n\n#### **Q2:  The work mentioned pre-trained video VAE with continuous latents is used. How is the VAE pretrained? Is it from previous open-source work or trained from scratch? More details about the VAE are appreciated.**\nThank you for pointing this out. We chose the powerful open-sourced WAN 2.1 VAE as the continuous VAE. It is a 3D causal VAE with a latent dimension of 16, compressing videos by 4x8x8 times. As mentioned in the Wan2.1 paper, Wan-VAE is progressively trained in three stages: (1) image pretraining (2) low-resolution, short video pretraining (3) high-quality video fine-tuning. In Infinity$^*$, we construct our discrete tokenizer by inserting our quantizer between the encoder and the decoder of Wan-VAE and initialize it using the official weights of Wan-VAE. Then we fine-tune it on images and videos jointly. With the help of pretrained weights of continuous VAE, the convergence speed of the discrete tokenizer has been significantly improved as shown in Figure 5 of the submitted manuscript. We will clarify this in the revised paper.\n\n#### **Q3:  For comparison of different models in Table 1/2, is it possible to include the scale of training data besides params so that people can better compare Infinity\\* with baselines.**\nThanks for your advice. As shown in Table 1, we list the training data size for different T2I and T2V models if their original papers have declared. Wanx 2.1 and HunyuanVideo have not provided the information. We will add the training data size to Table 1/2 in the revised paper.\n\n---\n|   T2I Model |   #Param  |   Data Size   |   T2V Model   |    #Param   |   Data Size   |\n|---|:---:|:---:|---|:---:|:---:|\n| PixArt-Sigma | 0.6B | 33M | VideoCrafter2 | 2B | 10M |\n| LlamaGen | 2.6B | 60M | OpenSora v1.2 | 1B | 30M |\n| SD3 | 8B | 1000M | NOVA | 0.6B | 20M |\n| Chameleon | 7B | 1400M | HunyuanVideo | 2B | undeclared |\n| Infinity | 2B | 130M | Wan 2.1 | 2B | undeclared |\n| Infinity*| 8B | 70M | Infinity* | 2B | 13M |\n\nTable 1. The number of parameters and training data size for different T2I/T2V models\n\n#### **Q4:  Also in Table 2, it would be beneficial if params of different models are listed as in Table 1.**\nWe agree with your suggestion. As illustrated in Table 1, we present the number of parameters for open-source T2I/T2V models. Specifically, our model has 8B parameters, WAN 2.1 has 14B, HunyuanVideo has 13B, NOVA has 0.6B, and Emu3 has 8B parameters. We will add this information to the revised paper.\n\n #### **Q5: Are there quantitative ablation studies about the proposed stochastic quantizer depth and semantic scales repetition as well as how the hyperparameters are determined?**\n\nThank you for your valuable feedback. We provide quantitative ablation studies on both the stochastic quantizer depth (SQD) and semantic scales repetition (SSR) in Table 2 and Table 3, respectively. As shown in Table 2, incorporating SQD leads to a notable improvement in VBench scores across several metrics. Moreover, we observe that SQD contributes to faster convergence during video generation training: at 5k iterations, the variant with SQD achieves a gFVD of 257.03, compared to 349.46 without SQD. Regarding SSR, Table 3 demonstrates that introducing this mechanism improves the VBench score from 75.72 to 80.97. Increasing the number of repetition steps further boosts the score to 81.28, although with diminishing gains. Based on this trade-off between performance and computational efficiency, we set the repetition count to $n=6$.\n\n---\n| Method   | total score | quality score | semantic score |\n|---|:---:|:---:|:---:|\n| w/o SQD  | 81.07       | 81.21         | **80.54**          |\n| With SQD | **81.28**       | **81.56**         | 80.16          |\n\nTable 2. Ablation study of the proposed stochastic quantizer depth\n\n\n\n\n---\n| Method |  total score  |  quality score  |   semantic score   |\n|---|:---:|:---:|:---:|\n| w/o SSR | 75.72 | 76.73 | 71.68 |\n| With SSR (repetition=4) | 80.97 | 81.38 | 79.32 |\n| With SSR (repetition=6) | **81.28** | **81.56** | **80.16** |\n\nTable 3. Comparison of different repetition times in Semantic Scale Repetition (SSR)\n\n\n #### **Q6: On page 5 line 197, it should be ''Patchify'' not ''Patchfy''.**\n\nThank you for pointing out the typo on page 5, line 197. We appreciate your careful review. We will correct ''Patchfy'' to ''Patchify\" in the revised paper.\n\n\n#### **Experiments Setup**\nAll our ablation experiments were conducted using a 1M training set with a batch size of 40, trained at 192p resolution for 30,000 iterations. We evaluated the results on the Vbench benchmark."}, {"review_id": "GOG4OZTcbK", "reviewer": "Reviewer_KwMu", "summary": "This paper proposes Infinity∗, a spacetime pyramid modeling framework that unifies diverse visual generation tasks, including t2i, t2v, and i2v. It systematically investigates the various challenges encountered when applying the Infinity architecture to video generation and proposes solutions for each issue. The method elevates the performance of autoregressive video generation approaches to a level comparable to cutting-edge diffusion models.", "questions": "1. How to determine the video generation length, frame count, or when to stop generation.\n2. Does Spacetime Sparse Attention affect performance?\n3. In Table 4, it is better to list the frame numbers instead of durations to ensire comparative fairness.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Infinity∗ is a vision generation model based on the Infinity architecture, covering a variety of visual generation tasks. Notably, in video generation, it significantly outperforms other autoregressive text-to-video models of the same category, and the experimental results are solid.\n2. The paper is well-written and clear. Starting from Infinity, it explains the challenges and solutions of the VAR architecture in visual generation.\n3. Overall, although it is an extension of Infinity, it also holds considerable value in unifying diverse visual generation tasks, providing inspiration for future methods in autoregressive visual generation.\n\nWeaknesses:\n1. Some content in Section 3.2 is confusing. For example, the \"Pyramid\" in Figure 1 still appears to be a spatial pyramid at each timestep, making the term \"spacetime pyramid\" misleading.\n2. Some experimental details are insufficient. Are there any experiments demonstrating that Spacetime Sparse Attention has minimal impact on performance?", "was_revised": true, "final_justification": "Thanks for the author's response. Most of my concerns are resolved. I keep the original rating unchanged.", "rebuttal": "## Response to Reviewer ``KwMu`` (Rating: Accept)\n\nWe thank reviewers for your valuable comments on Infinity$^*$, acknowledging strong performance with fast speed (``7YkR``, \n``qR4k``, ``vUvY``, ``tNTA``, ``KwMu``), well-motivated design (``7YkR``, ``vUvY``, ``KwMu``), valuable contribution to the generative model community (``qR4k``, ``tNTA``, ``KwMu``), with solid experimental results (``qR4k``, ``KwMu``). Your advice will help us enhance the paper.\n\n#### **Q1:  Some content in Section 3.2 is confusing. For example, the ''Pyramid\" in Figure 1 still appears to be a spatial pyramid at each timestep, making the term ''spacetime pyramid\" misleading.**\n\nThank you for your insightful comment. We named our scale schedule the \"spacetime pyramid\" for two primary reasons: (1) To distinguish it from the conventional space pyramid schedule used in image generation. Specifically, images are two-dimensional, involving only height and width. Consequently, the scale schedule in images consists of a series of coordinate pairs $(h_i, w_i)$. Videos, however, are a spatiotemporal medium with three dimensions (time, height, and width). Accordingly, the new schedule we propose is composed of triplets $(t_i, h_i, w_i)$. The term \"spacetime\" is intentionally used to emphasize that our method is designed to process videos with their inherent spatiotemporal structure.\n    (2) To reflect the hierarchical nature of our schedule, which is the core characteristic of a pyramid. Although the temporal dimension t remains constant across the different scales in our schedule, we observe a clear hierarchical pattern, as illustrated in Figure 6 of the submitted manuscript. The earlier scales encode global semantic information, while the later scales focus on local, fine-grained details. As one progresses from the initial to the final scales, semantics gradually decreases while details become progressively stronger. Since this hierarchical structure is the most fundamental attribute of a pyramid, we believe the term \"pyramid\" accurately describes our approach. \n\n#### **Q2:  any experiments demonstrating that Spacetime Sparse Attention has minimal impact on performance?**\nIn Tables 1~2, we compare our spacetime sparse attention with full attention. Spacetime sparse attention shows better performance than full attention in the Vbench's score (81.28 vs 80.77), while showing a significant advantage in saving computation and GPU VRAM.  We conclude that spacetime sparse attention significantly improves efficiency without performance loss. We hypothesize that spacetime sparse attention yields better results than full attention because it reduces exposure bias. In full attention, since the context is all ground truth during training, it is more susceptible to accumulated errors during inference.\n\n| Method                                    | total score | quality score | semantic score |\n|---|:---:|:---:|:---:|\n| Full Attention                                 | 80.77       | 81.15         | 79.23          |\n| Attend to former clip's largest scale     | **81.28**       |  **81.56**         | **80.16**          |\n| Attend to former clip's 3rd largest scale | 80.86       | 81.26         | 79.26          |\n| Attend to former clip's 6th largest scale | 80.75       | 80.98         | 79.80          |\n\nTable 1. Performance comparison of various attention mechanisms\n\n---\n\n|           Method                   | (192p 65 frames, 4s 16fps) | (192p 161 frames, 10s 16fps) | (480p 161 frames, 10s 16fps) |\n|---|:---:|:---:|:---:|\n| Full Attn                               | 8.6s / 40.8GB              | 24.3s / 57GB                 | OOM                          |\n| Attend to former clip's largest scale   | 7.7s / 38.5GB              | 16.7s / 40GB                 | 44.7s / 63GB                 |\n| Attend to former clip's 3rd largest scale | 7.4s / 38.2GB              | 15.8s / 39GB                 | 34.5s / 58GB                 |\n| Attend to former clip's 6th largest scale | 7.3s / 37.9GB              | 15.2s / 38GB                 | 30.5s / 55GB                 |\n\nTable 2. Computational efficiency comparison of attention mechanisms measured on a single NVIDIA H100 GPU with 8B parameters.\n\n#### **Q3:  How to determine the video generation length, frame count, or when to stop generation?**\n\nDuring the training phase, variable-length videos are decomposed into clips, where each clip contains 80 frames (5s 16FPS). The caption regarding each video is appended with ''duration=xs\". During the inference phase, Infinity* stops generating once the user-specified duration is reached.\n\n#### **Q4:  In Table 4, it is better to list the frame numbers instead of durations to ensire comparative fairness.**\n\nThanks for your valuable suggestion. In Table 4 in the paper, all compared models generate 16 FPS videos with 81 frames. We will replace ''duration\" with ''frames numbers\" in the revised paper.\n\n#### **Experiments Setup**\nAll our ablation experiments were conducted using a 1M training set with a batch size of 40, trained at 192p resolution for 30,000 iterations. We evaluated the results on the Vbench benchmark."}]

Author rebuttal:

Meta-review: This paper extends the VAR framework to video generation by adding the temporal dimension into its pyramid design. The authors introduce a new sparse attention mechanism that conditions only on the final scale of the previous clip, striking a balance between latency and temporal consistency. They also propose additional techniques to improve generation quality, such as repeatedly generating the first few scales and training the tokenizer with stochastic quantizer depth.

The reviewers gave consistently positive feedback, highlighting the method’s versatility for video generation tasks, computational efficiency, strong empirical results, and thorough ablation studies. Some concerns were raised about design choices, latency analysis, and the need for more empirical evidence and implementation details to support the claims. The authors addressed these issues well, and it is strongly recommended that they include the corresponding clarifications and results in the revision.

Overall, this paper introduces a scalable visual autoregressive generation framework that can handle challenging long-duration video generation tasks, while also unifying text-to-image, text-to-video, and image-to-video generation in a single model. This is an important step forward in the visual generation field, complementing existing autoregressive and diffusion approaches.
Empirically, the paper demonstrates both strong performance and efficiency. It surpasses prior AR models and even some diffusion baselines, while generating videos about 10 times faster. This makes the work highly impactful for real-world use. In addition, the newly designed modules are insightful and could inspire further research.
Besides, the reviewers were consistently positive about its versatility, efficiency, and thoroughness. 
Considering these points, the AC recommended this paper for an oral presentation.
Decision: Accept (oral)