Compositional 3D-aware Video Generation with LLM Director
Authors: Hanxin Zhu
Tianyu He
Anni Tang
Junliang Guo
Zhibo Chen
Jiang Bian
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D-aware Video Generation; LLM; Compositional
Primary area: generative_models

Abstract: Significant progress has been made in text-to-video generation through the use of powerful generative models and large-scale internet data. However, substantial challenges remain in precisely controlling individual elements within the generated video, such as the movement and appearance of specific characters and the manipulation of viewpoints. In this work, we propose a novel paradigm that generates each element in 3D representation separately and then composites them with priors from Large Language Models (LLMs) and 2D diffusion models. Specifically, given an input textual query, our scheme consists of four stages: 1) we leverage the LLMs as the director to first decompose the complex query into several sub-queries, where each sub-query describes each element of the generated video; 2) to generate each element, pre-trained models are invoked by the LLMs to obtain the corresponding 3D representation; 3) to composite the generated 3D representations, we prompt multi-modal LLMs to produce coarse guidance on the scale, location, and trajectory of different objects; 4) to make the results adhere to natural distribution, we further leverage 2D diffusion priors and use score distillation sampling to refine the composition. Extensive experiments demonstrate that our method can generate high-fidelity videos from text with flexible control over each element.

Compositional 3D-aware Video Generation with LLM Director
Hanxin ; He?, Anni Tang', Junliang Guo", Zhibo Chen', Jiang Bian? UUniversity of Science and Technology of China
Microsoft Research Asia
Shanghai Jiao Tong University
hanrinzhu@mail, www.uut.c...... eattc.ed.... en, tianyuhe@microooftt com, memory97dsjtu, edu cn, junl
com, chenzhibo@ustc, edu. cn, jiang. bwwnbmiccosoot..commmmo com
Abstract
Significant progress has been made text-to-video generation through the use powerful generative models and large-scale internet data. However, substantial challenges remain in precisely controlling individual concepts within the generated video, such as the motion and appearance of specific characters and the movement viewpoints. In this work, we propose a novel paradigm that generates each concept in 3D representation separately and then composes them with priors from Large Language Models (LLM) and 2D diffusion models. Specifically, given an input textual prompt. our scheme consists of three stages: 1) We leverage LLM as the director to first decompose the complex query into several sub-prompts that indicate individual concepts within the video (c.g., scene, objects, motions), then we let LLM to invoke pre- pre-trained expert models to obtain corresponding 3D representations of concepts. 2) To compose these representations, we prompt multi-modal LLM to produce coarse guidance on the scales and coordinates trajectories for the objects. To make the generated frames adhere natural image distribution, we further leverage 2D diffusion priors and use Score Distillation Sampling refine the composition. Extensive experiments demonstrate that our method can generate high-fidelity videos from text with diverse motion and flexible control over each concept. Project page: https: //aka. ms/c3v
Introduction
Benefitting from large-scale data and the advancement of the generative models 212), have witnessed plenty of astonishing results across a wide array of tasks. For example, Large Language Models (L.M)) pre-trained web-scale datasets are revolutionizing machine learning with strong capability of zero-shot learning [3] and planning [45), while diffusion models [6] empower text-to-
image generation with a rapid surge in both quality and diversity 77...
To harness the power of text-to-image models in text-to-video generation, modern solutions directly view video as multiple images. In this way, tremendous efforts have been dedicated extending text-to-image models with temporal interaction to ensure consistency between frames [7] However, generating visual content conditioned the textual prompt alone struggles express multiple concepts with precise spatial layout control [ [18-20] To tackle this issue, LVD (1 and VideoDirectorGiP (22 propose first generate spatiotemporal bounding boxes of each object based on the textual prompt with I.M,, and then condition the video generation on the obtained layouts. Although rough layout control can be realized, they still have inherent limitations for detailed concept control, e.g. the motion and appearance of specific characters, and the movement of viewpoints.
"This work is accomplished Microsoft, April 2024.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

In nature, our understanding of the world is compositional [23 24 20], and the interaction with the world takes place in a 3D. Motivated by this, in contrast to the prior endeavors that implicitly learn different concepts in 21) space, we are interested in exploring an alternative solution that explicitly composes concepts in 3D space for video generation. To this end, we in particular identify two key technical challenges: 1) Since textual prompt contains multiple concepts, how to coordinate the generation of various concepts? 2) Given the generated concepts, how to compose them to follow common sense in the real world?
In this work, we introduce text-guided compositional 3D-aware video generation (C3V), novel paradigm that regards LLM as director and 3D as structural representation for video generation. C3V consists of three main stages: 1) Given a textual prompt, to coordinate the generation of various concepts. we leverage LLM to disassemble the input prompt into sub-prompts, where each sub- prompt describes an individual concept, e-g., the scene, objects, and motion. For each concept, a pre-trained expert model is assigned by L.M to generate its corresponding 3D representation (e.g., 3D Gaussians 25 SMPL parameters (26] according to the textual description. 2) To provide coarse instruction for composition (i.e., the scale and trajectory of each object in the scene), we further resort to the priors in multi-modal LLM by querying it with the rendered scene image and the textual goals. However, directly instructing multi-modal LLM to return the scale and trajectory of each object leads to unexpected results, as it is challenging for LLM to estimate visual dynamics. Therefore, we follow step-by-step reasoning philosophy [27] by representing the object with the bounding boxes and dividing the trajectory estimation into sub-tasks, L.e., estimating the starting points, ending points, and trajectories step-by-step 3) After obtaining the coarse trajectories from the language space, also propose to refine the scales, rotations, and exact locations with priors from large-scale visual data. Specifically, taking inspiration from DreamFusion (28). which proposes to distill generative priors from pre- pre-trained image diffusion models into 3D objects, we employ Score Distillation Sampling (SDS) 28 optimize the transformation matrix of each object in 3D space.
Our system has three main advantages: 1) Because each concept is represented by individual 3D representations, it naturally supports flexible control and interaction of each concept. 2) It inherently excels at synthesizing complex and long videos such as drama, etc. 3) The viewpoint is controllable.
Extensive experiments demonstrate that our proposed method can generate 3D-aware videos with diverse motion and high visual quality, even from complex queries that contain multiple concepts and relationships. We also illustrate the flexibility of C3V by editing various concepts of the generated videos. The generated videos are presented on our project page, To the best of our knowledge, we make the first attempt towards text-guided compositional 3D-aware video generation. We hope it can inspire further explorations on the interplay between video and 3D generation.
2 Related Works
2.1 Video Generation with LLM
Recently, there have been substantial efforts in training text-to-video models on large-scale datasets with autoregressive Transformer [29 30 [7] or diffusion models [3] A prominent approach for text-to-video generation is to extend a pre-trained text-to-image model by inserting temporal layers into its architecture, and fine-tuning models on video data. However, although effective, it remains challenging to generate objects with specific attributes or positions. To address this challenge, series of studics proposed to exploit knowledge from LLM 32] to achieve controllable generation [2] 19
22
33, zero-shot generation [36-39], or long video generation [40]. For
example, Free--looo
36 and DirecT2V (38 used LLM to transform the input textual prompt
into a sequence of sub- hoo-prompts that describe each frame. LVD (1] and VideoDirectorGPT [22 employed LLM to generate spatiotemporal bounding boxes to control the level dynamics in video generation.
In light of the above success of exploiting LLM to direct video generation in 2D space, we view LLM as director in 3D, which differs from previous methods not only terms of technical route but also in benefits: providing free interaction with individual concepts and flexible viewpoint control.

---

2.2 Compositional 3D Generation
Generating 3D assets from textual prompt has garnrred significant attention owing to its promising applications in various fields such as AR [1] VR 42 and autonomous driving [43]. However, due to the lack of large-scale 3D data, it is challenging to apply 2D generative models to 3D directly.
Therefore, building upon Dream Fields 44 DreamFusion introduced the Score Distillation Sampling
(SDS) (28] a technique enhancing 3D generation by distilling 2D diffusion priors from pre-trained text-to-image generative models. Motivated by the success of DreamFusion [28) dedicated effort have been made to improve SDS [45 Though achieving remarkable results, these methods struggle to generate scenes with multiple distinct elements. To mitigate this issue, several techniques was proposed to guide 3D generation with additional conditions like layout priors, which we refer to compositional 3D generation [48 50). However, these works still focus static compositional 3D generation and lack visual dynamic modeling.
Recently, two concurrent works Comp4D [ST] and TC4D 52 also achieved compositional 4D generation (i.e., dynamic 3D generation). However, they only considered composition between objects, and the trajectory of these methods is either formulated by kinematics- equations or pre-defined by users. Differently, we explore 3D-aware video generation with integrated 3D scenes and compose various concepts with priors from both LLM and 2D diffusion models.
3 Preliminaries
3.1 3D Gaussian Splatting
3D Gaussian Splatting (3DGS) (25 has been attracting lot of interest for novel view synthesis, due to its photorealistic visual quality and real-time rendering. 3DGS utilizes a set of anisotropic ellipsoids (i.e., 3D Gaussians) to encode 3D properties, in which each Gaussian is parameterized by position μ E IR3, covariance E E 33x3 (obtained from scale s E R and rotation E R²), opacity R, and color c E R
To render novel view, 3DGS adopts a tile-based rasterization, where 3D Gaussians are projected onto the image plane 2D Gaussians. The final color c(p) of pixel p is denoted as:
c(p) Σ čØ à)
where è and â represent the individual color and opacity values of a series of 2D Gaussians con- tributing to this pixel. 3DGS are then optimized using LI loss and SSIM (53) loss in a per-view optimization manner. Thanks to the nature of modeling 3D scenes explicitly, optimized 3D Gaussians can be easily controlled and edited.
3.2 Score Distillation Sampling
Different from text-to-image generation which benefits from a large number of text-image pairs available, text-to-3D generation suffers from severe lack of data. To mitigate this issue, Score Distillation Sampling (SDS) (28] was proposed to distill generative priors from pretrained diffusion- based rext-to-image models Φ. Specifically, for 3D representation parameterized by 0, SDS is served as way to measure the similarity between the rendered images x = g(f) and the given textual prompts y, where g represents the rendering operation. As a result, the gradients used to update 0 computed as follows:
VeLsDs(ő = g(0) =
- e)]
(2)
where is the noise level, e is the ground-trute noise, w(t) is a weighting function, &o is the estimated noise given noised images 24 with text embeddings y. Please refer to DreamFusion (8 for details.
4 Method
Overview. To achieve text-guided compositional 3D-aware video generation (C3V), we regard L.M as director and 3D as structural representation. To this end, our method consists of three stages. To begin with, we utilize LLMs to decompose the input textual prompts into three sub-prompts, each
3

---

Magician's megical cabiin alone serene forest, an elien welking on the floor, starting from the cabin's door to the mow nearthe bottom right image.
Multimodal Large Language Model
Task Decorpposer
Trajectory Generator
Complex query into sub prompts
Scale trajectory estimation
Composition with 2D Diffusion Priors
Magician's... forest
Please trajectory represents that allon walking on the floor, starting from the cabin's 's door the the bottom right corner this image.
3D Scene
Generation
"In Magician's of this image.
A:
Diffusion Priors
an alien
Step-by-Step
Scale:
Estimation
(380,450)
Scale
Object Generation
End:
Flstllation Sumpuin
Location
(204, 8821
Rafiremem
Trajectory:
Rotation
(380, 4550
walking on the floor
(408,461)
Sptutinn
Motion Generation
(624, (80)
(668,66
(704,662)
Figure Illustration of our method. consists of three stages: 1) The input textual prompt is decomposed into individual concepts by the LLM. Then generate each concept in the form of 3D with the corresponding pre-trained expert model (left & Sec. 4.1). 2) We leverage knowledge in multi-modal LLM to estimate the 2D trajectory of objects step-by-step (middle & Sec. 4.2p. 3) After lifting the estimated 2D trajectory into 3D as initialization, we refine the scales, locations, and rotations of objects within the 3D scene using 2D diffusion priors (right & Sec. 4.3)
of which provides description for generating a corresponding concept (i.e,, scene, object, motion, etc.) respectively (STc. Subsequently, we leverage multi-modal L.MM obtain coarse-grained scales and trajectories for each animatable object (Sec. 4.2). Finally, we emplay 2D diffusion priors to refine the objects' location, scale, and rotation for a fine-grained composition (Sec. 4.3)
4.1 Task Decomposition with LLM
Task Instructions. Given a textual prompt, we invoke LLM (e.g., GPT-4V 32) to decompose it into several sub-prompts, Each sub-prompt describes an individual concept such as the scene, object, and motion. Specifically, for an input prompt y, we query LLM with the instruction like: "Please decompose this prompt into several sub-prompts, each describing the scene, objects the scene, and the objects' motion." from which we obtain the corresponding sub-prompts.
3D Representation. After obtaining the sub nub-prompt for each concept, aim generate its corresponding 3D representations using the pre-trained expert models. In this work, we build structural representation 3DGS (25), which is explicit form and therefore flexible enough compose or animate. Concerning concepts like motion, our framework can generalize to arbitrary animatable 3D Gaussian-barts objects. For simplicity, because it is general for various scenarios. In order to obtain diverse human motions, we take autreval-augmented approach [54] acquire motion in the form of SMPL-X parameters [55] from large motion libraries [56] according to the motion-relate sub- prompt.
take human motion as an instantiation
Instantiation. To illustrate the scheme formally, consider the following example. We have sub- prompts 3/, 3/2 and ya that describe scene, object, and motion respectively. Additionally, we have corresponding pre- pre-trained text-guided expert models Φ1. 62, and Øy that are selected by the LLM. The concept generation can be formulated as follows:
G (1()). G2 Φ2(U2, M), M ds((3).

---

trajentory ropresents
starting the cabin
th image.
Direct Estimation
Step-by-Steg Estimation
Please
Use boundinn cabints the coorcinese
GZ Use a bouncing bax reproset the mav
QU
Trajectory
Irpectory represents that allnn maiking on the Foor, star ing from the cabins door to the mow
Taak: Giwen the imaga, imagine a harman woster
(880450))
near the bottom ngnnt comer of the imge. what the pien ocordinato the conter of the bounding box?
Pleaso gonerae oto one ressonaee abbls path,
4011
the center of the bounding box
rop esented by 10 bouncing boves teach
bounding box with recolution 220 1210
(409)417))
near the botoo right comve thes imaga
the image, from bod to boeill ceecriinng that
the human avaca walking on
star
(066.004)
(360 453) (704, 892) at difforent timestopa
Ac Trajectory
Al: (260 650)
AZ (704
should resecraids should
the
The path should smooth line
Output:
Give me the plecl ccordinate of each bord
box's cenner, both the x x-coordinate and
conrdieete should ln between (1- 287
Figure 2: Illustration of coarse gaatse-rained trajectory generation with LLM. Instead of querying multi-
modal LLM to estimate dynamic trajectory directly, we generate trajectory in step-by-step manner: estimating the locations of starting and ending points first, then reasoning the path between them.
where 11 and G2 represent the 3D Gaussians, and M means the motions used to drive G2. In the following sections, we will provide details on the composition of the generated concepts.
4.2 Coarse-grained Trajectory Generation with LLM
Given the generated concepts, we aim to compose them into dynamic 3D representation to render videos that align with the input textual prompt. Achieving this requires scales and trajectories of the objects to indicate their sizes and locations within the scene. To this end, we propose to leverage knowledge encoded in multi-modal LLMs (i.e., GPT-4V (3) to provide priors.
For the scale of the object, we find that directly querying GPT-4V with the input prompt and rendered scene image can yield reasonable estimation of its resolution (H2p and W2D). However, this not the case for trajectory estimation. As demonstrated in Fig. 2 directly querying GPT-4V for trajectory will lead result that deviates conspicuously from common sense. Based on this observation, we conclude two issues: I) it is too difficult for GPT-4V to generate the trajectory within a single query, as it lacks priors on visual dynamics; 2) since GPT-4V is trained to generate text, has limitations on imagining visual content.
To mitigate this, introduce two simple yet effective techniques. 1) Although GPT-4V lacks visual knowledge of the object, we can alleviate this by representing the object as a bounding box with the estimated resolution. 2) We follow a step-by-step reasoning philosophy (27 and propose to let GPT-4V estimate the locations of starting and ending points first, then reason the path between them.
Overall, formulate the above process follows:
(1 p(yp: I, L.. Le);
S - (), D(ys: I), L. &
where & represents the multi-modal LLM (i.e., GPT-4V), I denotes the rendered scene image, S represents the estimated scale of the object given textual prompt and I, L,, and L, represent the locations of starting and ending points respectively, (L, IN represent N locations indicating the path between L.g and T.g. Notably, all locations (i.e., L.x, Les (T..) ) are represented by 2D pixel coordinates on 7.
4.3 Fine-grained Composition with 2D Diffusion Priors
Lift Trajectory from 2D to 3D. In Sec. 4.2 we obtain the 2D pixel coordinates L = (pé. pb)) of the estimated trajectory. However, 2D trajectory is not enough for composition in 3D space. Therefore, we convert it into corresponding 3D world coordinate T.p = (xf,8) ,,"" ,²). Specifically, we first predict the depth map D of the rendered scene image with a monocular depth estimator (57).
5

---

Then, we use the depth value of the center point of the lower boundary of the bounding box as the trajectory's 's depth. As result, we can transform 2D trajectory into 3D:
H2D
2 pyy 1)" D(p' + H2D
HAD _000,00,, (5)
p
where R and K represent camera extrinsic and intrinsic respectively, Hnn and Wen represent the resolution of the 2D bounding box. HnD represent the actual height of the 3D bounding box of this object within the scene.
Composition Refinement with 2D Diffusion Priors. With the lifted 3D trajectory, we then integrate the object into the scene. However, the trajectory estimated by LLM is still rough and may not obey natural image distribution. To address this, we propose to further refine the object's ss scale, location, and rotation by distilling generative priors from pre-trained image diffusion models ( into 3D space. Specifically, we treat the parameters for these attributes as optimizable variables and use SDS (Eq. (2) to improve the fidelity of rendered images. As result, scale refinement can be formulated as follows:
V SDS EE,glu(((((essss)(Ca) (S o - 2) G2);8, e)]
(6)
where S represents the optimizable variable, S represents the scale estimated by GPT--(V), means the Sigmoid function, т₁ is a threshold, G2 represents the 3D gaussians the object, and ø; is the noised 2D image given L&D and scaled G2-
After obtaining more precise scale, we then refine the locations of the estimated 3D trajectory similarly, where the location refinement is denoted as:
Location = Ba..l - TI La SDS
2 (S Ta (2)))..,))((])) (
(7)
where L represents the optimizable variable, TL is threshold.
For the rotation of the object different timesteps, can directly compute the corresponding
rotation matrix, based on the assumption that the object at the current time step should face the
location of the object at the next time step. As result, the rotation matrix Ri time step can be computed using the following equation:
Lx2 + try + 2s
Lry 28 Laz + ys ty² +
tyz 2S tyz + to +
(8)
txz ys
= 1 - c, = cos((), (0n (2, 2)
where 0 and u represent the rotation angle and axis obtained through the cross product of (L (IL+1). Tipp (T.t)
TI.) and (0, 0, 1)T.
Inference. After obtaining individual concepts in the form of 3D and the optimized parameters that indicate how to compose various concepts, we can render the 3D representation into 2D video with flexible camera control in real time 255
Experiments
In this section, we instantiate C3V with three concepts: scene, humanoid object, and human motion, to generate 3D-aware video from text. We compare our proposed method with state-of-the-art text-to- 4D models (4D-FY (58), ) compositional 4D generation models (Comp4D (5T) and text-to-video
models (VideoCrafter2 (59) Videos are available on our anonymous project page
Implementation Details. generate 3D scenes, humanoid objects and motions respectively. To realize SDS, we utilize Stable Diffusion 77] as the image diffusion model. All the videos of our proposed method are rendered at resolution of 512 x 512 in real time. Please refer to the appendix for more details.
use LucidDreamer
60
HumanGaussian 61 and Motion-X X (56 to

---

(D-FY
CaupD
VidkoCrafter2
Ours
(a) Text prompt: "In a Magician's magical cabin alone in serene forest, on alien walking the floor, starting from the cabin's door the mow near the bottom right corner of this image".
40-YY
CampID
Ours VidkoCrafter2
(b) Text prompt: "Four characters stood on the stage. In front of the stage, man and woman are performing Kung Fu and dancing respectively. On the right side of the stage, a skeleton man is dancing, and behind them, clown is performing".
Figure 3: Qualitative comparisons with baselines. When prompting complex queries, the baseline methods fail to follow the queries in terms of the number of objects and the corresponding motion. In contrast, our method excels in yiclding both diverse motion and high visual quality.
Metrics.
Following Comp4D (51), we choose Q-Align 62 as the referee to measure the quality
and aesthetics the video. The Q-Align score is number ranging from (worst) to (best) where higher score indicates better performance. We also report the CLIP score [3 to measure the alignment between the generated videos and the input texts.
5.1 Comparison with Competitors
In Fig. we conduct a comparative analysis of our method against 4D-FY (58) Comp4D 51. and VideoCrafter2 [59 with the same textual prompt. It can be observed that all three baselines fail to provide diverse motion from the textual prompt, while our method excels in yielding large motion and high visual quality. For example, our scheme successfully obeys the complex query in terms of

---

Table Quantative comparisons with competitors. Our method consistently outperforms all baseline methods in terms of both the video quality and the alignment with textual prompts.
Metric
4D-FY (58)
Comp4D [5T]
VideoCrafter2 [59]
Ours
QAlign-in img-quality
1.681
1.687
3.839
4.030
QAlign-ing laee-in---tstttee" [62]
1,475
1.258
3.199
3,471
QAlign-vid vidu-aiiyy
62
2.154
2.142
3.868
4.112
QAlisn-vid-eestheeicc]
02
1,580
1.425
3.159
3.723
CLIP Scoret
63
30.47
27.50
35.20
38,36
Direct traeetoor estimation.
(II) Trajectory estimation with bounding box indicating objects.
Trajectory estimation step by-step mamner.
(IV) Ours.
(a) blation studies trajectory estimation with multi-moda LLM.
Without SDS refinement
(II) With scale refinement.
(III) With trajectory refinement.
(IV) With rotation refinement.
(b) A Ablation studies on composition with 2D diffusion models. Figure Ablation studies on framework design. Each ablation is prompted with the same text.
the number of objects and the corresponding motion. In addition, since 4D-FY and Comp4D focus on object-centris generation, they fail to generate videos with natural backgrounds. In Tab. 3 we perform quantitative comparisons by utilizing Q-Align Score [62] and CLIP Score [63] to assess the quality of generated videos. Our method consistently outperforms the baseline models in terms of both the video quality and the alignment with textual prompts. More results are available in the appendix.
5.2 Ablation Studies
Ablations on Trajectory Estimation with Multi-modal LLM. As shown in Fig. (T), direct prompt of GPT-4V will lead to obvious unsatisfactory trajectory estimation. When only depending on bounding boxes to indicate the location of objects within the scene (Fig. (II), though roughly better trajectory can be achieved, it still leads to unreasonable results, such as several floating bounding boxes. Similarly, using only the se-eby-step estimation strategy described Sec. 4.2 typically results in trajectory that is merely simple straight line connecting the starting and ending points (Fig. 4 (II()))). With both of the two techniques, we can achieve the best performance, with a more reasonable and smooth trajectory (Fig. (a))))).
Ablations on Composition with 2D Diffusion Models. To figure out whether it is necessary to conduct fine- ane-grained composition with 2D generative priors, we gradually refine the scales, locations, and rotations with SDS and visualize the results in Fig. 4b). All results are generated with the same textual prompt: "An alien walking on the floor in front of the cabin's 's door.". shows that when we optimize the attributes with SDS, we can obtain consistently improved performance with reasonable scale (Fig. g (II,, accurate locations that are aligned with the input prompt (Fig. [b)( and orientation that accords with common sense (Fig. (I))))).

---

Source object
Edited object
(a) Object Editing
Source motion
Edited motion
(b) Motion Editing
Source scene
Edited scene
(c) Scene Editing
Figure 5: Our method offers flexible control individual concepts. We demonstrate this by editing different concepts: the appearance and motion of the actors, and the scenes.
Table 2: Quantative comparisons of ablation studies on trajectory estimation with multi-modal LLM.
Estimation using bounding box.
Methods
Estimation.
Step-by-step estimation.
QAlign-img-qualinn
62
2.056
2.894
3.752
4.030
DAlign-img-aesthell aes-teettteetiiceeeee.e
62
1.568
2,156
3.047
3.471
Aligg-vid-quultty
62
2.178
3.043
3,904
4.112
QAlign- aesthetic
62
1.680
2.346
3.342
3.723
CLIP Score
63
25.68
29.84
36.73
38.36
5.3
Applications on Controllable Generation
Due underlying 3D structural representation, scheme has the natural merits of editing individual concepts. We illustrate this character in Fig. 5 by editing three different concepts: the appearance and motion of the actors, and the scenes. For the appearance and motion the actor, we can seamlessly replace them in a zero-shot manner according to the textual prompt (Fig5 ab)), while this is still challenging for implicit models [64 65). For scene editing, ensure a smooth composition of objects within the target scene, we re-estimate the trajectory of the objects given the target scene. Kindly refer appendix for more results.
Conclusion
In this paper, we present a novel paradigm for 3D-aware video generation by conceptualizing videos as compositions of independent concepts represented 3D space. To this end, we leverage L.M as director to decompose the input textual prompts into individual concepts and then invoke pre-trained expert models to generate them separately. To compose various concepts, we first prompt multi-modal LLM in ste-by-step manner provide coarse guidance the scale and trajectory of objects,

---

Table Quantative comparisons of ablation studies on composition with 2D diffusion models.
Methods
Without SDS.
With scale refinement.
With trajectory refinement.
Ours
Qllign-img leenn-ingqualitt t
662
3.045
3.674
3.826
4.030
QAlign-img-aesthenn
62
2.752
3.046
3.341
3,471
Qiligitvid-yuality 1 [6
3.129
3,794
3.983
4.112
Qilign-vvilaeeshettie
62
2.704
3.468
3.603
3.723
CLIP Score f 66
31.35
35.27
37.04
38.36
then refine the composition with 2D generative priors. We verify our scheme in different scenarios, demonstrating its superiority over the baseline methods.
Limitations and Future Works. Although demonstrate promising results in 3D-aware video generation, there still are limitations to be improved in the future. First, our framework is instantiated with limited concepts in this work, Le., scene, humanoid object, and human motion. It is exciting to generalize the framework to more concepts like animals, vehicles, etc. Second, the composition between concepts is conducted with priors from LLM and 2D diffusion priors in our method. However, is still interesting to introduce physically grounded dynamics into 3D representation [66]. Third, though our method is naturally suitable for maintaining the consistency of actors across different scenes, it still needs further exploration on long video generation with multiple scenes, e.g. a full-length film.
Ethics Statement. C3V is exclusively a research initiative with no current plans for product integration or public access. We are committed to adhering Microsoft AI principles during the ongoing development of our models. The model is trained on Al Al-generated content, which has been thoroughly reviewed ensure that they do not include personally identifiable information or offensive content. Nonetheless, as these generated data are sourced from the Internet, there may still be inherent biases. To address this, we have implemented a rigorous filtering process on the data to minimize the potential for the model to generate inappropriate content.
Acknowledgement.
This work was supported in part by NSFC under Grant 62371434, 62021001.
References
[1] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 188):9, 2019.
[2] Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS), 33:6840-68511 6851, 2020.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 33:1877- 1901, 2020.
[4] Timo Schick, Jane Dwivedi-Y Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024.
[5] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. Conference on Robot Learning, pages 540-562. PMLR, 2023.
[6] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems (NeurIPS), 34:8780-8794, 2021.
[7] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 10684-10695 2022.
10

---

[8] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[9] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems (NeurIPS), 35:36479-36494, 2022.
[10] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. FxXv:2204.033588 2022.
[11] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.023003 2022.
[12] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22563-22575, 2023.
[13] David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint arXiv:2309. 15818, 2023.
[14] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang. Xiaodong Cun, Shaoshu Yang. Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19511 2023.
[15] Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Emu video: Factorizing text-to-video generation by explicit image conditioning. arXiv preprint arXiv:2311.10709 2023.
[16] Agrim Gupta, Lijun Yu, Kihyuk Sohn, Xiuye Gu, Meera Hahn, Li Fei-Fei, Irfan Essa, Lu Jiang, and José Lezama. Photorealistic video generation with diffusion models. arXiv preprini arXiv:2312.06660 2023.
[17] Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero shot video generation. arXiv preprint arXiv:2312.1415 14125, 2023.
[18] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836-3847, 2023.
[19] Weixi Feng. Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuchai Hc, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. Advances in Neural Information Processing Systems, 36, 2024.
[20] Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In European Conference on Computer Vision, pages 423 4399. Springer, 2022.
[21] Long Lian, Baifeng Shi, Adam Yala, Trevor Darrell, and Boyi Li. LIm-grounded video diffusion models. In International Conference on Learning Representations, 2024.
[22] Han Lin, Abhay Zala, Jaemin Cho, and Mohit Bansal. Videodirectorgpt: Consistent multi-scene video generation via Ilm- -inigddedd planning. arXiv preprint arXiv:2309. 2023.
[23] Noam Chomsky. Aspects of the Theory of Syntax. Number 11. MIT press, 2014.
[24] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 33022266) 2015.
11

---

[25] Bernhard Kerbl, Georgios Kopanas. Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):1-14, 2023.
[26] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: skinned multi-person linear model. ACM Transactions on Graphics (TOG), 34(6):1-16. 2015.
[27] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2024.
[28] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations, 2023.
[29] Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kindermans, Herman Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Learning Representations, 2023.
[30] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. In International Conference on Learning Representations, 2023.
[31] OpenAI. Chatgpt. https://openallcom/chhggg 2022.
[32] OpenAL. Gpt-4v(ision) system card. http:://penni..omm/mddd//g--vvv
2023.
[33] Sitong Su, Litao Guo, Lianli Gao, Hengtao Shen, and Jingkuan Song. Motionzero: Exploiting motion priors for zero-shot text-to-video generation. arXiv preprint arXiv:2311.16635, 2023.
[34] Yash Jain, Anshul Nasery, Vibhav Vineet, and Harkirat Behl. Peekaboo: Interactive video generation via masked-diffusion. arXiv preprint arXiv:2312.07509 2023.
[35] Sixiao Zheng, Jingyang Huo, Yu Wang, and Yanwei Fu. Intelligent director: An automatic framework for dynamic visual composition using chatgpt. arXiv preprint 47X1v:2402.15746
2024.
[36] Hanzhuo Huang, Yufan Feng, Cheng Shi, Lan Xu, Jingyi Yu, and Sibei Yang. Free-bloom: Zero-shot text-to-video generator with Ilm director and Idm animator. Advances in Neural Information Processing Systems, 36, 2024.
[37] Yu Lu, Linchao Zhu, Hehe Fan, and Yi Yang. Flowzero: Zero-shot text-to-video synthesis with llm-driven dynamic scene syntax. arXiv preprint arXiv:2311.15811 2023.
[38] Susung Hong, Junyoung Seo, Heeseong Shin, Sunghwan Hong, and Seungryong Kim. Direct2v: Large language models frame-level directors for zero-shot text-to-video generation. arXiv preprint 230iv: 14330, 2023.
[39] Gyeongrok Oh, Jaehwan Jeong, Sieun Kim, Wonmin Byeon, Jinkyu Kim, Sungwoong Kim, Hyeokmin Kwon, and Sangpil Kim. Mtvg: Multi-text video generation with text-to-video models, arXiv preprint XrXiv:2312.04086 2023.
[40] Shaobin Zhuang, Kunchang Li, Xinyuan Chen, Yaohui Wang, Ziwei Liu, Yu Qiao, and Yali Wang. Vlogger: Make your dream vlog. arXiv preprint 4rXiv:2401.09411 09440444444 2024.
[41] Chenliang Chang, Kiseung Bang, Gordon Wetzstein, Byoungho Lee, and Liang Gao. Toward the next-generation vr/ar optics: a review of holographic near-eye displays from a human-centric perspective. Optica, 7(11): 15:3 1578, 2020.
[42] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mydream: Multi- view diffusion for 3d generation. arXiv preprint arXiv:2308.1653 2023.
12

---

[43] Xiangyu Yue, Bichen Wu, Sanjit A Seshia, Kurt Keutzer, and Alberto L Sangiovanni- Vinciinliitttllli lidar point cloud generator: from a virtual world to autonomous driving. In Proceedings of the 2018 ACM on International Conference on Multimedia Retrieval, pages
2018.
[44] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 867-876, 2022.
[45] Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024.
[46] Yukun Huang, Jianan Wang, Yukai Shi, Xianbiao Qi. Zheng-Jun Zha, and Lei Zhang. Dream- time: An improved optimization strategy for text-to-3d content creation. arXiv preprint arXiv:2306. 12422, 2023.
[47] Yixun Liang, Xin Yang, Jiantao Lin, Haodong Li, Xiaogang Xu, and Yingcong Chen. Lucid- dreamer: Towards high-fidelity text-to-3d generation via interval score matching. arXiv preprint arXiv:2311. 12284, 2023.
[48] Ryan Po and Gordon Wetzstein. Compositional 3d scene generation using locally conditioned diffusion. arXiv preprint arXiv:2303. 2023.
[49] Gege Gao, Weiyang Liu, Anpei Chen, Andreas Geiger, and Bernhard Schölkopf. Graphdreamer: Compositional 3d scene synthesis from scene graphs. arXiv preprint arXiv:2312.0009,, 2023.
[50] Xiaoyu Zhou, Xingjian Ran, Yajiao Xiong, Jinlin He, Zhiwei Lin, Yongtao Wang. Deqing Sun, and Ming-Hsuay Yang. Gala3d: Towards text-to-3d complex scene generation via layout-guided generative gaussian splatting. arXiv preprint arXiv:2402.072077 2024.
[51] Dejia Xu, Hanwen Liang, Neel P Bhatt, Hezhen Hu, Hanxue Liang, Konstantinos N Plataniotis, and Zhangyang Wang. Comp4d: LIm-guided compositional 4d scene generation. arXiv preprint 1 2024.
[52] Sherwin Bahmani, Xian Liu, Yifan Wang. Ivan Skorokhodov. Victor Rong, Ziwei Liu, Xihui Liu, Jeong Joon Park. Sergey Tulyakov, Gordon Wetzstein, et al. Tc4d: trajectory-conditionn text-to-4d generation. arXiv preprint arXiv:2403. 17920, 2024.
[53] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Ecro P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, (44):600 612, 2004.
[54] Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. General- ization through memorization: Nearest neighbor language models. In International Conference on Learning Representations, 2020.
[55] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael Black. Expressive body capture: 3d hands, face, and body from single image. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10975-10985, 2019.
[56] Jing Lin, Ailing Zeng, Shunlin Lu, Yuanhao Cai, Ruimao Zhang, Haoqian Wang, and Lei Zhang. Motion-x: large- scale 3d expressive whole-body human motion dataset. Advances Neural Information Processing Systems, 36, 2024.
[57] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the IEEF/CVF international conference on computer vision, pages 12179- 12188, 2021.
[58] Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B Lindell. 4d-fy: Text-to-4d generation using hybrid score distillation sampling. arXiv preprint arXiv:2311.17995 2023.
13

---

[59] Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprini arXiv:2401.09047, 2024.
[60] Jaeyoung Chung. Suyoung Lee, Hyeongjin Nam, Jaerin Lee, and Kyoung Mu Lee. Lu- ciddreamer: Domain-fres free generation of 3d gaussian splatting scenes. arXiv preprint arXiv:2311.13384, 2023.
[61] Xian Liu, Xiaohang Zhan, Jiaxiang Tang, Ying Shan, Gang Zeng. Dahua Lin, Xihui Liu, and Ziwei Liu. Humangaussian: Text-driven 3d human generation with gaussian splatting, arXiv preprint arXiv:2311.17061 2023.
[62] Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, et al. Q-align: Teaching Imms for visual scoring via discrete text-defined levels. arXiv preprint arXiv:2312.17090, 2023.
[63] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 3748-8763 PMLR, 2021.
[64] Wilson Yan, Andrew Brown, Pieter Abbeel, Rohit Girdhar, and Samaneh Azadi. Motion- conditioned image animation for video editing. arXiv preprint arXiv:231 1.88277, 2023.
[65] Jianhong Bai, Tianyu He, Yuchi Wang, Junliang Guo, Haoji Hu, Zuozhu Liu, and Jiang Bian. Uniedit: A unified tuning-free framework for video motion and appearance editing. arXiv preprint 13115 2024.
[66] Tianyi Xie, Zeshun Zong, Yuxin Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. Physgaussian: Physics-integratee 3d gaussians for generative dynamics. arXiv preprint 1rX11:2311.12198, 2023.
14

---

A Implementation Details
A.1 Experimental Settings
During the process of multi-modal LLMs-based trajectory estimation, we use 20 locations by default indicate the trajectory between the starting point and the ending point, (i.e. N 20 in Eq. where the path between adjacent locations is assumed as a straight line. For scale refinement (Eq. 6 refine all the twenty locations. The training iterations for each location is 1000 and TL is set to 0.1. All experiments are conducted using single NVIDIA A100 GPU.
Ts is set to 0.1. For location refinement (Eq. [7). we apply
A.2 Pre-trained Expert Models
LucidDreamer [60). As powerful 3D scene generation method, LucidDreamer adopts an iterative view generation strategy, where series of views are dreamed and aligned via -hrrppng based inpainting networks. After obtaining these multiview-consiseee images, 3D gaussians are optimized construct a high-qualle 3D scene, by means of typical training pipeline of 3DGS.
We choose HumanGaussian as the method for human generation due to its capability of producing drivable avatars on the basis of 3DGS. Specifically, HumanGaussian starts with SMPL-X prior to densely sample Gaussians on the human mesh surface as initial center positions, followed by texture-structure joint model and an annealed negative prompt guidance strategy to obtain high-fidelity outputs.
HumanGaussian [61].
Motion-X [56]. Motion-X a large-scale 3D expressive whole-body human motion dataset which comprises 15.6M precise 3D whole-body pose annotations (i.e., SMPI.-X) covering 81.1K K motion sequences from massive scenes with sequence-level semantic labels. By calculating the similarity of text embeddings between the motion-related sub-prompt and sequence labels, we find the most matching sequence and acquire motion data in the form of SMPL-X parameters [55].
A.3 Metries
Given the lack of ground truth videos for specific text queries, we utilize pre-trained quality- assessment models evaluate the generated videos and their individual frames. In line with Comp4D) [51] we employ Q-Align (62) the benchmarking tool to assess the quality and aes- theties of the videos. The Q-Align rating, which ranges from (worst) to 5 (best), is considered state-of-the-arr closely aligning with human judgments across established quality assessment bench- marks. Additionally, we include the CLIP score [63] to measure the alignment between the generated videos and the input texts, where higher scores signify better alignment.
B Results of Different Stages
As shown in Fig. 6 Fig. and Fig. 8 we provide a detailed visualization of results obtained during different stages, including results of based task decomposition, results of coarse-grained trajectory generation with GPT-4V, and results the final rendered videos.
C More Results of Controllable Generation
In this section, we present additional results on controllable generation. As demonstrated in Fig. g Fig. aO]and Fig. [I] we can achieve fine- ime-grained control of the target without affecting other concepts in the 3D space.
15

---

Prompt: In Magician's magical cabin alone serene forest, alien walking on the floor, starting from the cabin's door to mow near the bottom right corner of image.
Task Decomposition with LLM
Magician's magical cabin alone
forest
alien
walking on the floor
Trajectory Generation with LLM
Use a bounding box to represent cabin's door, pixel coordinate of the center of the bounding box?
bounding rupresent the mow near the bottom right comer of this image, what the pixel coordinate of the center the bounding bux?
Estimated trajectory
Rendered Video
Figure 6: Results of different stages given the textual prompt: "In Magician's 's magical cabin alone in serene forest, an alien walking on the floor, starting from the cabin't 's door to the the bottom right corner of this image.".
16

---

Prompt: Inside cozy livingroom Christmas, astrologer performing ballet on the floor, starting from wooden floor behind red ammchair near the bottom left of this image to the in the bottom right comer of this image.
Task Decomposition with LLM
a cozy livingroom Christmas
astrologer
performing ballet on the floor
Trajectory Generation with LLM
( sse a bounding box to represent wooden floor behind bottom left of this image, what is the pixel coordinate of the center the bounding box?
Use bounding box to represent soća the bottom right comer of this image, what is the pixel coordinate bounding box?
armchaic near the
Estimated trajectory
Rendered Video
Figure 7: Results of different stages given the textual prompt: "Inside cozy livingroom in Christmas, astrologer performing ballet the floor, starting from wooden floor behind the red armchair the bottom left of this image to the sofa in the bottom right corner of this image.".
17

---

Prompt: On simple stage, man with fedora and denim jacket and woman wearing ski clothes are performing Kungfu and dancing respectively, on the left side and right side of this stage.
Task Decomposition with LLM
simple
stage
mun with black felom deain jacket
wortac weeding sid clothes
performing Kungfu
dancing
Trajectory Generation with LLM
Use bounding hox represent left side stagu in this image, what is the pixel coordinete the center bounding box?
bounding box represent right side of the stage this image, pixel coordinate of the center of the bounding
Estimated trajectory
Rendered Video
Figure 8: Results of different stages given the textual prompt: "On a simple stage, a man with a black fedora and denim jacket and woman wearing ski clothes are performing Kungfu and dancing respectively, on the left side and right side of this stage.".
18

---

(a) Prompt: A black wearing green t-shirt playing Kungfu the stage.
(b) Prompt: Turn the character fairy.
(c) Prompt: Turn the character into warlock.
Figure Results of actor's appearance editing.
(a) Prompt: A demon slayer performing a han and tang dance on the stage.
Prompt: Turn the motion of the character into ballet.
(c) Prompt: motion of character kin kong exotic dance.
Figure 10: Results of actor's motion editing
19

---

(a) Prompt: A revenant dancing Daily Lovedive in a ultra-modern mega villa by the sea with swimming pool.
(b) Prompt: Turn the scene into long anime-style road with anime-blocks and little anime-grass.
(c) Prompt: Turn the character into a postapocalyptic city in desert.
Figure 11: Results scene editing.
20

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We clarify our contributions in Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We clarify our limitations Sec. 6.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide implementation details in Sec. 5 and Sec. A.I.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer:
[No]
Justification:
We do not provide open access to the data and code in supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.com wttpcccpppbccccceee guwdes/dde/eeee Cooeeeoodeesssseeeeeee http://dd///uuuiddd/d/oe//eea/s/
ssinislicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide implementation details Sec. 5 and Sec. A.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because they would be too computationally expen- sive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide computer resources used in Sec. A.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have reviewed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We report possible impacts of our method in Sec. I and Sec. 6.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite all models and datasets mentioned in the paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
The paper proposes novel paradigm for text-to-video generation, named C3V. We plan to release the model's code along with comprehensive documentation to facilitate its use and replication.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The human avatars showed in this paper are all virtual humans that have relationships with real human beings.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The human avatars showed in this paper are all virtual humans that have relationships with real human beings.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
26

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "jMSnKHNvFi", "reviewer": "Reviewer_6K4s", "summary": "The paper presents a pipeline for 3D-aware video generation by composing scenes, objects, and motions. One key idea is to use existing LLM to provide coarse guidance on the scale and trajectory of objects, and then refine the coarse rendering with SDS. The method is compared with multiple methods.", "questions": "I think the paper tackles an interesting problem but the effectiveness of the solution remains to be justified.", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea is explainable as it composes the scene, objects, and motions in an explicit manner. So potentially it is easy to improve the pipeline by building better modules for each component.\n\nThe paper shows both quantitative results and qualitative examples. The paper is overall easy to read.", "weaknesses": "$\\textbf{Method}$\n\n1. It would be great to clarify what are considered to be the main contributions of the paper. The pipeline uses multiple existing modules for 3D generation, and then using LLM to generate a series of bounding boxes for the guidance of the rendering. The idea of using LLM to generate a trajectory does not look novel enough given the existing volume of literature.\n\n2. For the refinement stage, is the scale and 3D location refined for each image individually? If so, how does the scale and location maintain temporal consistency?\n\n3. How does the motion be compatible to the scene? Suppose it's a rough terrain or mountain area, the motion has to be adjusted?\n\n4. To generate the bounding box trajectory, is it correct that a single image is used as input, which means no 3D information is used for LLM? If the 3D scene is existing, do you think adding 3D information and generating 3D locations could be a better option rather than 2D bounding boxes?\n\n\n$\\textbf{Experiments}$\n\n1. One concern is that the overall quality of the human actors is still limited. Since the resolution is not high enough, it is hard to tell whether the method is harmonizing the image well given the examples presented. \n\n2. It is also hard to see the 3D viewpoints changes given the examples, it seems like for most examples, the viewpoints are not changing much for the sequence of renderings. I imagine identity-preserving character inpainting methods can show similar examples? Maybe show more examples demonstrating the uniqueness of the proposed method.  \n\n3. The baselines do not include the strong motion prior as the proposed method so I am not sure if this is fair. How does the method compare to methods with explicit 3D priors [1]?\n\n[1] Image Sculpting: Precise Object Editing with 3D Geometry Control. CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 6K4s, we are grateful for your careful review and the valuable feedback that you provided for our paper. We appreciate that you found our paper easy to read and our ideas explainable. We hope the following comments address your concerns.\n\n**Q: Main contributions of this paper.**\n\nA: As recognized by Reviewer 7svJ and Reviewer BLtC, the main contributions of this paper can be summarized as follows:\n\n1. We approach video generation as a compositional task by leveraging existing strong domain-specific models. To the best of our knowledge, this is the first work that can realize video/4D generation from the perspective of compositional scene, object, and motion generation.\n\n2. To compose different models into a harmonic whole automatically, we provide a novel method that utilizes LLMs as a director in a step-by-step manner, which is able to generate reasonable trajectories based solely on the background scene.\n\n3. To further refine details of composed dynamic scenes, we propose leveraging 2D diffusion priors (i.e., SDS) to ensure the rendered images align more closely with natural image distributions.\n\n4. We have conducted extensive experiments to demonstrate the effectiveness of our method, showing that high-fidelity videos with diverse motion and flexible control over each concept can be achieved from text.\n\n**Q: Is the scale and 3D location refined for each image individually?**\n\nA: No. Since the scale describes the relative size of the object within the 3D scene, it should remain consistent across different time steps. Therefore, we refine the scale for the first frame and apply it to all subsequent frames. For the 3D location, we optimize it for each image individually to achieve better composition.\n\n**Q: How to accommodate both scenes and motions?**\n\nA: Since the motion is generated using expert models, complex object motions will be produced for more intricate scenes. We then use SDS to enhance the compatibility of these motions with the scenes.\n\n**Q: Whether 3D information is used for LLM?**\n\nA: No. In this paper, we use only a single image as input without 3D information available. However, we believe that injecting 3D information into LLM would be beneficial for better composition, which we will explore in the future. Thank you for pointing this out!\n\n**Q: The resolution is limited.**\n\nA: In this paper, all generated videos have a resolution of 512 × 512, which is higher than that of many previous methods (e.g., Comp4D, 4D-fy, etc.). We also provide more examples in the attached PDF file to demonstrate the effectiveness of our method.\n\n**Q: Hard to see 3D viewpoints changing.**\n\nA: In this paper, we focused on highlighting the composition result by cutting out the region of the moving object, which may have made viewpoint changes less noticeable. We have included additional examples with more varied viewpoints in the attached PDF. Furthermore, we believe that applying identity-preserving character inpainting methods directly may not be effective, as 3D consistency is difficult to ensure.\n\n**Q: Comparisons with methods using different motion priors.**\n\nA: Actually, methods like Comp4D and 4D-fy also use strong implicit motion priors derived from pre-trained video diffusion models, whereas we use explicit motion priors to achieve better composition and controllability. In Image Sculpting, 2D objects are projected into 3D for image editing, we share a similar concept on 3D structural representation, but focuses on composing various 3D concepts into a 4D scene for video generation. Since Image Sculpting is not designed for video generation, a direct comparison is challenging. We plan to explore integrating methods from Image Sculpting in the future to incorporate more robust motion priors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1lp6B21JRK", "reviewer": "Reviewer_N95p", "summary": "In this work, the authors propose a framework for 3D-aware video generation using guidance from LLM. Specifically, this work follows previous studies on LLM for video generation that takes language model as a director to do below sub-tasks:\n1) expand and decompose the prompt into different aspects, and then use off-the-shelf expert models to generate objects/motions/scenes.\n2) plan the trajectory and other scene configurations.\n3) put all of them together and update with SDS loss.\nExperiments are conducted to verify the effectiveness of this work.", "questions": "How do the authors handle the situation where scene configurations are not properly", "limitations": "The limitations are discussed.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1) The focused setting is novel, which enables LLM-guided text-to-video generation with 3D awareness.\n2) The proposed pipeline generally makes sense to me.", "weaknesses": "1) The novelty of this work is limited. It seems like to be the combination of 3D scene generation + 3D avatar generation + motion generation + LLM + SDS. It would be subjected to the performance upper bound set by each expert model, and combining them altogether would make the quality even worse.\n2) In TC4D, the trajectory can also be generated by LLMs, which should not be a drawback of that work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer N95p, thank you for taking the time to review our work and for providing thoughtful feedback. We are pleased that you found our setting novel and the pipeline coherent. We address your concerns as follows.\n\n**Q: Novelty of this work.**\n\nA: As noted by Reviewers 7svJ and BLtC, our work transcends a mere combination of expert models. It is rooted in the concept that our understanding of the world is inherently compositional, and we achieve this by leveraging priors from expert models. Specifically, to seamlessly combine different models into a cohesive whole, we introduce a novel method that employs LLMs as a step-by-step director, capable of generating plausible trajectories with only the background scene. Subsequently, to enhance the details of the composed dynamic scenes, we propose using 2D diffusion priors (i.e., SDS) to ensure that the rendered images more closely match natural image distributions. Additionally, as acknowledged by Reviewer 6K4s, our pipeline can be easily enhanced by developing better modules for each component, highlighting the potential of our method.\n\n**Q: Trajectory in TC4D.**\n\nA: Thank you for bringing this to our attention, and we apologize for any confusion caused. As an outstanding approach for text-to-4D generation, TC4D enables trajectory-conditioned 4D creation, accommodating trajectories that are either user-defined or estimated directly by LLMs, and facilitating applications such as generating 4D scenes from arbitrary trajectories and synthesizing compositional 4D scenes. Our method differs from TC4D in several key aspects: **1)** we use 3D Gaussians instead of a deformable NeRF for representing 4D scenes, as 3D Gaussians offer real-time rendering and easier editing; **2)** for trajectory estimation, we query the LLM in an step-by-step manner, which can estimate reasonable trajectories using only the background scene; **3)** in addition to object-level composition, our method achieves composition at the scene level, including interactions between complex scenes and objects, marking a novel advancement in scene-level composition. We will make the necessary corrections in a future version to address this issue and ensure that our work meets the highest standards of accuracy and clarity.\n\n**Q: Solutions when scene configurations are not properly.**\n\nA: Based on the estimated scene configurations provided by LLMs, we will use SDS for refinement to ensure the composed 4D scene renders more realistic images that align with human intuition. Specifically, in this paper, we treat properties such as scale, location, and rotation as optimizable variables, refined by SDS. In the future, we plan to consider additional factors, such as illumination and spherical harmonic coefficients, to achieve more realistic video generation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dqSodWEHap", "reviewer": "Reviewer_7svJ", "summary": "This paper proposes a method to synthesize dynamic 3D videos, with moving objects and camera. It treats the tasks compositionally, generating the (static) background and foreground figures separately. The generation process is orchestrated by an LLM, which provides prompts to separate models that specialize in different scene components (and which are pretrained); these components each use gaussian spats allowing assembly into a single dynamic 4D scene. This compositionality also enables controllable generation, where certain scene elements are replaced depending on user input. The method is demonstrated on several text prompts, and shown to out-perform three baselines.", "questions": "- How many prompts were in the evaluation set? How were these selected?\n- What are the quantitative results from the ablation study?", "limitations": "There is adequate discussion of limitations. There is an exceedingly brief discussion of broader impacts, borderline adequate for this task.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The idea of treating 3D/4D video generation as a compositional task is elegant, and it is sensible to leverage existing strong domain-specific models; it is also a nice idea to use an LLM here to automatically determine a suitable sequence of domain models to apply, and how to combine them.\n- The proposed pipeline is novel, and fairly natural for the task. The choice of stages/components is clearly motivated.\n- The method successfully generates videos from at least two text prompts, showing somewhat plausible motion. Qualitative results from two prompts show significantly better visual quality than the selected baselines (VideoCrafter, Comp4D, 4Dfy)\n- Quantitative results based on CLIP score (adherence of frames to prompt) and Q-Align again exceed the baselines\n- As well as text-conditioned generation, the method also supports certain other kinds of controllability. Since the scene representation is compositional, foreground humanoids can be replaced by others, motion can be modified, and the background can be replaced. This affords a degree of precise control that is missing from 'monolithic' video generation models\n- Ablation experiments were conducted, removing three components, aiming to establish their importance in the overall pipeline\n- The writing is generally clear (modulo a few grammar issues); the paper is well organized.", "weaknesses": "- Very few qualitative examples are given (and presumably these were cherry-picked rather than random). In particular, only two text-conditioned generations are shown (same in paper and supplementary). This makes it difficult for the reader to judge the visual quality of the model outputs, which is vital for such a task\n- Even in the given two examples, prompt adherence is poor, with the \"skeleton man\" missing, incorrect positioning (\"in front\" of the stage vs at the front; \"in\" a cabin vs outside), and unrealistic lighting (no shadowing). This is problematic given that object compositionality being guided by the LLM is claimed as a key contribution of the work\n- It is unclear how many videos were used for the quantitative evaluation, nor where the set of prompts was drawn from. This makes it hard to judge the significance of these results.\n- The ablation experiments are on a prompt and qualitative only. This means they are statistically meaningless, and the benefits of the different components need to be demonstrated more rigorously.\n- The exact prompting strategy for the LLM is unclear, in particular the initial stage of creating the sub-tasks, and the creation of the scale/trajectory estimation prompt.\n- The method is limited by the choice of 'experts' that synthesize parts of the scene (currently static background from LucidDreamer, humanoids from HumanGaussian, and humanoid motion from Motion-X). While 'delegating' generation subtasks is a neat idea, it seems that significant engineering work is required to incorporate each 'expert', and there is not a clear path to adding e.g. other dynamic object types such as quadruped animals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 7svJ, thank you for your thoughtful feedback and for looking into every detail of our work. We are pleased that you found our idea elegant, novel, clearly motivated, and well-organized. We hope the following comments address your concerns.\n\n**Q: More qualitative examples should be provided.**\n\nA: Thanks for pointing this out. In fact, the two examples in this paper were randomly selected rather than cherry-picked. In the attached PDF file, we provide additional five examples with different 3D viewpoints and corresponding depth maps, demonstrating similar performance. We will release more results in the future version of our paper.\n\n**Q: Prompt adherence is poor.**\n\nA: Compared to competitors like 4D-fy, Comp4D, and VideoCrafter2, our method significantly outperforms them in generating results that more accurately adhere to the text prompts. However, despite this improvement, some inconsistencies in detail are currently unavoidable due to the limitations of the expert models adopted. For example, the generated \"skeleton man\" from HumanGaussian is indeed the actor on the right side of the stage (Fig. 3(b) in the paper), and it will be difficult for the LLM to estimate a trajectory \"inside the cabin\" when the 2D image it relies on represents scenes \"outside the cabin.\" Additionally, this paper does not consider factors such as illumination, as this is the first work towards compositional 3D-aware video generation, and we only consider several basic properties. As mentioned by Reviewer 6K4s, our pipeline can be easily improved by incorporating better modules for each component, and we plan to address this in future works.\n\n**Q: Ways to obtain prompts and number of videos used for quantitative evaluation.**\n\nA: Since no public benchmarks are available for compositional 3D-aware video generation, we first obtain sub-prompts for each expert model following their schemes. These sub-prompts are then used as key terms and composed into a complete input prompt using LLMs. For quantitative evaluation, we **randomly generated 400 videos** with varied scenes, objects, or motions, avoiding cherry-picking, and reported the average value of CLIP Score and Q-Align Score.\n\n**Q: Quantitative ablation studies should be provided.**\n\nA: We provide the ablation experiments quantitatively as follows:\n\n**Table. 1**: Quantitative comparisons of ablation studies on trajectory estimation with multi-modal LLM\n| Methods                                 | Direct Estimation. | Estimation using bounding box. | Step-by-step estimation. | Ours.  |\n|-----------------------------------------|--------------------|-------------------------------|--------------------------|-------|\n| **QAlign-img-quality** $\\uparrow$       | 2.056              | 2.894                         | 3.752                    | **4.030** |\n| **QAlign-img-aesthetic** $\\uparrow$     | 1.568              | 2.156                         | 3.047                    | **3.471** |\n| **QAlign-vid-quality** $\\uparrow$       | 2.178              | 3.043                         | 3.904                    | **4.112** |\n| **QAlign-vid-aesthetic** $\\uparrow$     | 1.680              | 2.346                         | 3.342                    | **3.723** |\n| **CLIP Score** $\\uparrow$               | 25.68              | 29.84                         | 36.73                    | **38.36** |\n\n\n**Table. 2**: Quantitative comparisons of ablation studies on composition with 2D diffusion models\n| Methods                           | Without SDS. | With scale refinement. | With trajectory refinement. | Ours  |\n|-----------------------------------|--------------|------------------------|-----------------------------|-------|\n| **QAlign-img-quality** $\\uparrow$ | 3.045        | 3.674                  | 3.826                       | **4.030** |\n| **QAlign-img-aesthetic** $\\uparrow$ | 2.752        | 3.046                  | 3.341                       | **3.471** |\n| **QAlign-vid-quality** $\\uparrow$ | 3.129        | 3.794                  | 3.983                       | **4.112** |\n| **QAlign-vid-aesthetic** $\\uparrow$ | 2.704        | 3.468                  | 3.603                       | **3.723** |\n| **CLIP Score** $\\uparrow$         | 31.35        | 35.27                  | 37.04                       | **38.36** |\n\nAs shown in the tables above, using a direct prompt with a multi-modal LLM for trajectory estimation results in clearly unsatisfactory outcomes. Relying solely on bounding boxes to indicate object locations within the scene yields improved but still limited performance. While the step-by-step estimation strategy offers noticeable improvements, the best results are achieved by combining both approaches. Similarly, for SDS-based refinement, applying SDS incrementally to adjust scale, location, and rotation results in substantial performance improvements.\n\n**Q: The exact prompting strategy for the LLM.**\n\nA: As demonstrated in Line 136, for an input prompt, we query the LLM with the instruction: *\"Please decompose this prompt into several sub-prompts, each describing the scene, objects in the scene, and the objects' motion.\"* From this, we obtain the corresponding sub-prompts. The creation of the scale/trajectory estimation prompt is shown in Fig. 2 in the paper.\n\n**Q: Method is limited by expert models.**\n\nA: As recognized by Reviewer 6K4s, we can easily improve the performance of our method by leveraging more powerful expert models, without the need for significant engineering work. To add other dynamic objects such as animals, we can use methods such as [1], which is a 3D Gaussian-based method that can drive animals. We will explore this in future works.\n\n[1] GART: Gaussian Articulated Template Models, CVPR 2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OY41cbUNrU", "reviewer": "Reviewer_BLtC", "summary": "This paper presents an LLM-involved three-stage pipeline for text-guided compositional 3D-aware video generation. In the first stage, an LLM is employed as director to decompose input textual prompts into sub-prompts including scene, object and motion. Subsequently, it leverages multi-modal LLM to make an initial estimation about scales and trajectories for each object. Moreover, 2D diffusion priors are further leveraged to refine the 3D generation results with SDS loss. Extensive experiments demonstrate that the effectiveness of the proposed method in 3D-aware high-fidelity video generation.", "questions": "1.Rendering 2D video from 3D representation is real-time as said in the paper, what about the time spent on 3D representation generation and trajectory generation ?  \n2.How is the influence of selection\t of pretrained expert models and open-source LLM/MLLM as director?", "limitations": "Yes, limitations have been discussed in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The idea of decompose the generation task into scene, object and its motion in a more organized way with LLM is interesting.  \n2.The method of generating trajectories following a step-by-step manner is a reasonable solution and is validated in the experiments.  \n3.Using SDS loss to distill generative priors from pretrained LDM/SD models helps refine the quality of generation and is validated in the ablation study.  \n4.With LLM as director and 3D as structural representation, the proposed method is able to generate 3D-aware videos with diverse motion and high visual quality.", "weaknesses": "1.How to choose pretrained expert models? How would those models influence generation performance?  \n2.Since the proposed method adopts the divide-and-conquer strategy, I wonder how it perform in case of complex scene with multiple objects, especially with interaction among objects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer BLtC, thank you for taking the time to review our work and for your positive and insightful feedback. We are pleased to hear that you found our idea interesting and that it demonstrated improved performance. We hope the following comments address your concerns.\n\n**Q: Criteria for choosing pretrained expert models.**\n\nA: In this paper, for scene and object generation, we choose models based on 3D Gaussians considering their explicit structure, which is beneficial for composition and editing. For motion generation, we use models capable of leveraging the generated motions to drive the object's 3D Gaussians.\n\n**Q: Performance with multiple objects.**\n\nA: As shown in Fig. 3(b) in the paper and the results in the attached PDF file, we can still achieve satisfactory results when generating videos with multiple objects and complex scenes. By decomposing the video generation task into several sub-tasks, we can pre-generate multiple objects' motions and then compose them into the same scene to produce a coherent video. In the future, we plan to incorporate more physics-based priors to generate more complex interactions.\n\n**Q: Time spent on 3D generation and trajectory generation.**\n\nA: Since we use pre-trained expert models to generate corresponding 3D representations, the time needed depends on the model itself. Specifically, for trajectory estimation, it takes about one minute for GPT-4V to make an inference. For scene generation using LucidDreamer, it takes about 15 minutes to generate a scene. For object generation with HumanGaussian, it takes approximately one and a half hours to generate an object. However, as noted by Reviewer 6K4s, the performance can be improved by leveraging more powerful expert models, such as [1] for scene generation and [2] for object generation, which will significantly accelerate the generation process. We will explore this in future works.\n\n[1] InstantSplat: Unbounded Sparse-view Pose-free Gaussian Splatting in 40 Seconds \n\n[2] Animatable 3D Gaussian: Fast and High-Quality Reconstruction of Multiple Human Avatars\n\n**Q: Influence on choosing different expert models.**\n\nA: According to our criteria for selecting pretrained expert models, when using models based on 3D Gaussians, no significant performance gap is observed. For trajectory estimation, since we follow a step-by-step approach, we believe that reasonable trajectories can also be generated by open-source MLLMs. We will validate this in future works.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We extend our sincere thanks to all the reviewers for their time and effort. We appreciate your positive feedback, noting that our work was described as "novel and elegant" (N95p, 7svJ, BLtC), "effective and achieving better results" (N95p, 7svJ, BLtC), and "easy to read and well-organized" (6K4s, 7svJ). In response to your comments, we have addressed each reviewer's concerns individually and uploaded a PDF file with additional qualitative examples showcasing the superiority of our proposed method. We would be grateful if you would consider increasing your score based on our responses.

Meta-review: This paper proposes a method for compositional 3D-aware video generation from text prompt. The main contribution is a meticulous design and leverage of existing LLM and diffusion models for object, motion, scene synthesis. Reviewers are positive about the rationality of the overall framework and competitive performance compared to baselines. During rebuttal, reviewers raised concerns about the novelty, insufficient evaluation and questions about other technical details. Most of the concerns have been addressed by the rebuttal and reviewers lean to the positive side of the paper, thus this paper is recommended to be accepted. We encourage the authors to include all additional results in the revision. Moreover, reviewers have raised reasonable concerns about the limitation (e.g., interaction between scene and character) of the current method and these shall also be discussed in the paper.
Decision: Accept (poster)