Animate3D: Animating Any 3D Model with Multi-view Video Diffusion
Authors: Yanqin Jiang
Chaohui Yu
Chenjie Cao
Fan Wang
Weiming Hu
Jin Gao
Venue: NeurIPS 2024 poster (2024)
Keywords: 4D generation
Multi-view video diffusion model
Primary area: generative_models

Abstract: Recent advances in 4D generation mainly focus on generating 4D content by distilling pre-trained text or single-view image conditioned models. It is inconvenient for them to take advantage of various off-the-shelf 3D assets with multi-view attributes, and their results suffer from spatiotemporal inconsistency owing to the inherent ambiguity in the supervision signals. In this work, we present Animate3D, a novel framework for animating any static 3D model. The core idea is two-fold: 1) We propose a novel multi-view video diffusion model (MV-VDM) conditioned on multi-view renderings of the static 3D object, which is trained on our presented large-scale multi-view video dataset (MV-Video). 2) Based on MV-VDM, we introduce a framework combining reconstruction and 4D Score Distillation Sampling (4D-SDS) to leverage the multi-view video diffusion priors for animating 3D objects. Specifically, for MV-VDM, we design a new spatiotemporal attention module to enhance spatial and temporal consistency by integrating 3D and video diffusion models. Additionally, we leverage the static 3D model’s multi-view renderings as conditions to preserve its identity. For animating 3D models, an effective two-stage pipeline is proposed: we first reconstruct coarse motions directly from generated multi-view videos, followed by the introduced 4D-SDS to model fine-level motions. Benefiting from accurate motion learning, we could achieve straightforward mesh animation. Qualitative and quantitative experiments demonstrate that Animate3D significantly outperforms previous approaches. Data, code, and models are open-released.

Animate3D: Animating Any 3D Model with Multi-view Video Diffusion
Yanqin Jiang 222 Chaohui 2u Chenjie Cao Fan Wang
Weiming Hu 1:2.5 Jin Gao
State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA
School of Artificial Intelligence, University of Chinese Academy of Sciences DAMO Academy, Alibaba Group "Hupan Lab
School Information Science and Technology, ShanghaiTech University
j jiangyangin2021011 aww.eaaaiaa....................e cn
(huakun ych, caochenjie cej.ii...ccjjj..............ee fan ftap:..aaaaaaaaaaaaaaaaaaa wwt.:u/aaaaabaieeeecccc. com
{jin. wmhh))nnlpr.ii ia.ac......
sithhu.... io/
Time
Front
Back
Front
Back
View
3D object
Video Diffusion (T2V)
Video+3D Diffusion
MV-VDM
(SVD + Zerol23)
(Ours)
Figure I: Different supervision for 4D generation. MV-VDM shows superior spatiotemporal consis-
tency than previous models. Based MV-VDM, propose Animate3D animate any 3D model.
Abstract
Recent advances in 4D generation mainly focus on generating 4D content by distill- ing pre-trained text single-view image-conditioned models. It is inconvenient for them to take advantage of various off-the-shelf 3D assets with multi-view attributes, and their results suffer from spatiotemporal inconsistency owing to the inherent ambiguity in the supervision signals. In this work, we present Animate3D, novel framework for animating any static 3D model. The core idea is two-fold: 1) We propose novel multi-view video diffusion model (MV-VDM) conditioned on multi-view renderings of the static 3D object, which is trained on our presented large-scale multi-view video dataset (MV-Video). 2) Based on MV-VDM, we introduce a framework combining reconstruction and 4D Score Distillation Sam- pling (4D-SDS) to leverage the multi-view video diffusion priors for animating
Equual contribution. Work done during Yanqin's 's internship at DAMO Academy, Alibaba Group Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

3D objects. Specifically, for MV-VDM, we design a new spatiotemporal attention module to enhance spatial and temporal consistency by integrating 3D and video diffusion models. Additionally, we leverage the static 3D model's multi-view renderings as conditions to preserve its identity. For animating 3D models, an effective two-stage pipeline is proposed: we first reconstruct motions directly from generated multi-view videos, followed by the introduced 4D-SDS to refine both appearance and motion. Benefiting from accurate motion learning, we could achieve straightforward mesh animation. Qualitative and quantitative experiments demonstrate that Animate3D significantly outperforms previous approaches. Data, code, and models are open-released.
1 Introduction
3D content creation has garnered significant attention due to its wide applicability in AR/VR, gaming, and movie industry. With the development of diffusion models [45, 44, 39, 20, 51, 6, 19] and large-scale 3D object datasets [13, 12, 37, 65, 70, 14], recent 3D foundational generations have seen extensive exploration through fine-tuned text-to-image (T2I) diffusion models [31, 40, 41, 52, 32, 49], as well as training large reconstruction models from scratch [22, 61, 47, 67, 54], leading the 3D assets creation to a new era. Despite the significant progress in static 3D representation, this momentum has not been paralleled in the realm of dynamic 3D content generation, also known as 4D generation.
4D generation is more challenging due the difficulty of simultaneously maintaining spatiotemporal consistency in visual appearance and dynamic motion. In this paper, we mainly focus on two challenges: 1) No foundational 4D generation models to unify both spatial and temporal consistency. Though recent 4D generation works [43, 24, 5, 69, 29, 38, 63, 59, 4, 57, 66, 16] separately distill knowledge from pre-trained T2I and 3D diffusion models [39, 41, 31] and video diffusion models [51, 2, 6] to model multi-view spatial appearance and temporal motions respectively, we clarify that such detached learning way suffers from inevitable accumulation of appearance degradation as the motion changed, as shown in Fig. 1 (SVD+Zerol 23). 2) Failing to animale existing 3D assets through multi- view conditions. With the development of 3D generations, animating existing high-quality 3D content becomes common demand. However, previous works about 4D modeling from video [24, 57] or based on generated 3D assets [S, 29, 38, 66] are all based on text [39, 41] or single-view [31, 40] conditioned models, struggling to faithfully preserve multi-view attributes during the 4D generation, such as the back of butterfly in Fig. is ignored by Zero 123 [31].
To address these issues, we advocate for an approach better suited for 4D generation, that is, ani- mating any off-the-shelf 3D models with unified spatiotemporal consistent supervision. this way, it would be convenient to directly take advantage of various fast-developing 3D generation and reconstruction approaches based on a single foundational model, eliminating the accumulation of errors in modeling appearance and motion.
To this end, we propose novel 4D generation framework called Animate3D in this paper, which can be divided into a foundational 4D generation model and a joint 4D Gaussian Splatting (4DGS) optimization. Formally, the foundational 4D model is a Multi- Multi-View Video Diffusion Model (MV- VDM) built upon the 3D generation model, MVDream [41], which can synchronously synthesize multi-view images with various temporal motions. Specifically, to better inherit the prior in previous 3D and video diffusion models trained on large-scale data, we propose learnable plug-and-play spatiotemporal attention module, building upon the motion module video diffusion [18, 6, 20] to expand the attention learning from the temporal domain to the spatial and temporal domain. Moreover, MV-VDM also includes the ability to refer to multi-view images, sufficienlly preserving the identity and details of off-the- 3D assets. Specifically, given multi-view images rendered from existing 3D assets or collected from real-world objects, we expand adaptive image-to-vide work, 12V- 1da-Ada [17], to multi-view version, called MV2V-Adapter, incorporating multi-view conditions to 4D learning through additionally spatial features and text embeddings. Enhanced by the milti-view appearance injection, we can disentangle the appearance learning from the motion learning, ensuring MV-VDM focuses on learning natural and coherent dynamic motions.
To further enable impressive animations from 3D objects that can be observed at any viewpoint and time, we jointly optimize the 4DGS [55] through both reconstruction and 4D Score Distillation Sampling (4D-SDS) losses based on our unified MV-VDM. Benefiting from the spatiotemporal

---

consistent multi-view video generations, 4DGS can be roughly converged to proper results with only reconstruction loss, while 4D-SDS further improves the details and fine-graines motions. The Gaussian trajectory learned by our framework is surprisingly accurate and could be used to directly animate the mesh.
The main dilemma in building a foundational 4D generation model lies in the rarity of large-scale 4D datasets, which is non-trivial to collect but the key factor drive our MV-VDM. In this work, we make the first attempt to build a large-scale multi-view video (4D) dataset, dubbed MV-Video. Specifically, VVdVideo comprises about 84K animations that are available under public license, consisting of about 38K animated 3D objects at all, which are rendered into over 1.3M multi-view videos with minigpt4-video [3] generated prompts, serve as the training dataset for our 4D foundation model.
We highlight the contribution of this paper as follows: 1) Animate3D is the first 4D generation framework animate any 3D objects with detailed multi-view conditions. The framework is further extended to achieve mesh animation without skeleton rigging. 2) We propose the foundational 4D generation model, MV-VDM, to jointly model spatiotemporal consistency. 3) We present the largest 4D datasets MV-Video collected with about 84K animations and over 1.3M multi-view videos. Extensive experiments demonstrate that our data-driven approach can generate spatiotemporal consistent 4D objects, significantly outperforming previous counterparts.
2 Related Work
3D Generation. Early 3D generation works optimized single 3D object with CLIP loss [36] or Score Distillation Sampling (SDS) [33] from 2D text-to-image (T2I) diffusion models. Since the models providing supervision lacked 3D prior, those works usually suffered from spatial inconsistency, ie., multi-face Janus problem [50, 28, 533. To tackle this problem, on the one hand, some works [31, 40, 41,32, 35] lifted the T2I diffusion to multi-view image diffusion by injecting new spatial attention layers and fine-tuning on large-scale 3D synthetic datasets [13, 12]. Although 3D consistency was improved, these optimization- based methods still required relatively long time to optimize a 3D object. On the other hand, some feed-forward 3D generation foundation models [22, 61, 47, 67, 54, 60, 30], also trained on large-scale 3D datasets, were able to produce a good-quality 3D object in several seconds in an inference-only way. Inspired by the success of the data-driven approaches in 3D generation, we aim to construct a large-scale 4D generation dataset and take the pioneering step towards developing foundation models for 4D generation.
Video Generation. Video generation works started with text-to-video (T2V) generation [20, 42, 2, 18, 6, 19, 7, 10], subsequently followed by image-to-video (12V) approaches [64, 17, 58, 6]. Previous T2V works usually built upon T21 diffusion models [20, 42, 19, 18, 21]. leveraging their pre-trained weights by leaving the spatial blocks unchanged and inserting new temporal blocks to model the temporal camera or object motions. The 12V works [58, 17, 64], building upon the aforementioned T2V methods, typically incorporate image semantics into video models. This is achieved through cross-attention mechanisms between noisy frames and the conditional image, while retaining the motion module design from the T2V models unaltered. We draw inspiration from the development paradigm of video generation to design our 4D generation foundation model, which is a multi-view image conditioned multi-view video diffusion model building upon pre-traines multi-view 3D and video diffusion models.
4D Generation. The pioneering work of 4D generation is MAV3D [43], which is text and image- conditioned 4D generation framework. MAV3D first proposed a multi-stage pipeline to optimize the static 3D object generation through the T2I model and subsequently learn motions from the T2V model [42]. Following works [68, 5, 29, 69, 4, 59] adopted similar pipeline, and they further found that employing T2I [39] and 3D-SDS [41] are crucial for both object generation and motion learning stages. Without them, the quality of the generated object's appearance suffered a remarkable decline, and the leaiong process was prone to failure. Very recently, Consistent4D [24] proposed video-to-4D generation task, which used single-view video reconstruction and SDS from Zero 1 23 [31] for motion and appearance learning. This paradigm was adopted by following works [38, 63, 66, 57, 16, 11] and extended to text/image-oo-viee then video-to-4D generation. All aforementioned works heavily depend on the foundational model for SDS to preserve objects' appearance and attributes. However, existing 3D diffusion models struggle to refer to multi-view
3

---

MV-VDDM
"4 dod is (o00)
Camera
X glbresfj
Text Ensodee
Speiil Fnceding
Temporal Encoding
Multvvew 3D Atientin
3D Alleation
Adaptes
Aoss-Attn
3D Attemiion
Malti-iio
(Pre-tuinnzi
Temporal Motion Module
W
Project In
SelfAAm. ]x2
Multi- SSead Alin.
Spatislemporal Attention
Alpha Blander
Spatetesspoaal Block
Mulli-vew Reiderings
Spaiioonparal Attertion
MV2V-Acapeer
Motion Module
Animatec3
Neose
Course Motion
Fine Mosion
MV-VDM
4D-SDS
Gen. Recon. 3DGS
4DGS
Generated Multi-vie Videos
Random Mulli-view Video Renderings
Pronpe: "A bind is Jlying
Figure 2: Illustration of proposed multi-view video diffusion model MV-VDM (upper part) and our Animate3D framework (lower part). MV-VDM, trained on our presented large-scale 4D dataset MV-Video, generate spatiotemporal consistent multi-view videos. Animate3D, based MV-VDM, combines reconstruction and 4D-SDS optimization to animate any static 3D models.
conditions, restricting their broader applications to animate various off-the-shelf 3D assets without losing their multi-view attributes.
Furthermore, is worth noting that existing generation methods suffer from another issue, i.e. spatial and temporal inconsistency [43, 5, 29, 24, 38]. Because the diffusion models used for SDS were never trained with multi-view video (4D) datasets, missing the critical capacity to formulate spatial and temporal consistency simultaneously. Thus previous methods failed properly trade off a good balance between appearance and motion learning. Please refer to Sec. in appendix for detailed discussion and comparison.
In this work, we resort to disentangle 3D object generation/reconstructioon and motion learning through a foundational 4D generation model, and propose novel framework to animate any static 3D object with consistent multi-view attributes.
3 Method
Given static 3D model, goal animate it with text prompt and use its multi-view renderings as image condition. This 4D generation task is particularly challenging as it requires ensuring spatial and temporal consistency of the appearance and motion, compatibility with the prompt, and preserving the identity of the static object. To address these challenges more fundamentally. we propose novel framework, Animate3D. to animate any static 3D object. As depicted in Fig. we divide the task into two parts: learning multi-view video diffusion model (MV-VDM), and animating 3D object with multi-view videos generated by MV-VDM.
3.1 Multi-view Video Diffusion Model (MV-VDM)
We propose novel multi-view image conditioned multi-vies video diffusion model, named MV- VDM. To inherit prior knowledge acquired by spatially consistent 3D models and temporally consis- tent video models trained on large-scale datasets, we advocate a baseline architecture by integrating them to utilize their pre-trained weights. In this work, we take MVDreem [41] and AnimateDiff [18] for the 3D and the video diffusion model, respectively. To enhance the spatiotemporal consistency and ensure compatibility with the prompts and the object's multi-view images, we propose an efficient plug-and-play spatiotemporal attention module combined with an image-conditioning approach. Our

---

MV-VDD is trained on our presented large-scale multi-view video dataset, MV.Video, which is introduced in Sec. 4.
Spatiotemporal Attention Module. As illustrated in Fig. 2, we insert a novel spatiotemporal attention module after cross-attention layers. The proposed spatiotemporal attention module comprises two parallel branches: the left branch is for spatial attention, and the right branch is for temporal attention. For spatial attention, we adopt the same architecture as the multi-view 3D attention in MVDream [41]. Specifically, the original 2D self-attention layer is converted into 3D by connecting n different views. In addition, we incorporate 2D spatial encoding, specifically sinusoidal encoding, into the latent features to enhance spatial consistency. As for temporal attention, we keep all designs of the temporal motion module from the video diffusion model [18] unchanged in order to reuse their pre-traie pre-trained weights. Based on the features of these two branches, we employ an alpha blender layer with a learnable weight to achieve features with enhanced spatiotemporal consistency. It worth noting that we do not apply spatiotemporal attention across all views and frames due to the prohibitive GPU memory requirements that render training infeasible. Instead, our parallel-brand efficient and practical alternative. Specifically, we first reshape the input feature of spatiotemporal attention module X E
design offers an
into two forms, X; E R (bx f)x (axhxw)xx x for spatial branch and X, E (bbxxxxx)xxfx for temporal branch. The spatiotemporal attention is then computed as: Xout = 14 Attentionspatiial (XiW X W,W + (1 (1- 11) - Altentiontemporal (X, Q+ X,W X,W?)
where y denotes the learnable weight, Wast, w wy/ Wos represent the corresponding projec- tion matrices. b, n, f h, w, e are the batch size, views, frames, height, width, and channels of the image features, respectively.
Multi-view Images Conditioning. Inspired by 12V-Adapter [17], add new attention layer, termed MV2V-Adapter, parallel to the existing frozen multi-view 3D self-attention layer within the proposed spatiotemporal block, shown in Fig. 2. Concretely, noisy frames are first concatenated along the spatial dimension. These are then used to query the rich contextual information from the multi-view conditional frames, which are extracted using the frozen 3D diffusion model. Next, we add the output of the MV2V-Adapter layer to that of the original multi-view 3D attention layer of MVDream. Thus, for each frame conditional frames' features as X X1:n,3, X x1:m,3 out, and X1m1,1, we have:
(I,..., S}. denoting the multi-view input, output, and
Xland out
Attention (X W.. xl.m.KK. Imm Wy Woo + (tttitioon!!m.WW......
(2)
Attention(X I... X xlm..... W,, X I:n. (m.llw))'''s
where Wa. WK. Wv and Wo are projection matrices in original self-attention layer, while Wq' and Wo' are those in the newly added layer. We find this simple cross-attention operator can effectively improve the object's 's appearance consistency in the generated video. After that, as shown in the spatiotemporal block in Fig. 2, we employ two cross-attention layers to align the text prompt and preserve the object's identity, respectively. The left one is inherited from MVDream while the right one is pre-trained in Ad-Atere [62].
Training Objectives. The training process of our multi-view video diffusion model is similar to Latent Diffusion Model [39]. Specifically, the sampled multi-view video data to 1:n,l:f are first encoded into latent feature z I.m.l:f via encoder E frame by frame and view by view. Then we add noise using the forward diffusion scheduler: z, 11m.2:f Varza 1:m.2:f + /I āşe, where ae is a weighted parameter and e is Gaussian noise. Note that, following V-V-Adapeer, we keep the first frame, i.e., the condition multi-view frames clean, and only add noise to the rest of the frames. During training, the 1mm.2:f text prompt
proposed MV-VDM takes as input the clean latent code 2 Ilm,1 noisy latent code =e embedding y, and the camera parameters E1. and outputs the noise strength, supervised by Le loss. The training objective of our MV-VDM is calculated as:
LMV Eg (0. |||c co(=) 1:m.2:f ., Σ 1, 111)
where 9 denotes the diffusion model. It is important that we keep the entire multi view 3D attention module frozen and only train the MV2V-Adapter layer and the spatiotemporal attention module to conserve GPU memory and accelerate training. Moreover, as the multi-view images of the first frame, 11:,1 serves as the condition images, we calculate the loss only for the latter f - frames, i.e.,
20 lm.22:f
2g
5

---

3.2 Reconstruction and Distillation of 4DGS
Based on our 4D generation foundation model MV,, we propose to animate any off-the-sher 3D object. For efficiency, we take 3D Gaussian Splatting (3DGS) [25] as the static 3D object representation, and animate by learning motion fields represented by Hex- Hex-planes, as in [55].
4D Motion Fields. As in 4D Gaussian Splatting (4DGS) [55], we represent the motion fields by Hex-planes [15, 8]. Denoting the static 3DGS as represent the position, color, opacity, rotation, and scale, respectively. The motion module 'D predicts changes in position, rotation, and scale for each Gaussian point in frame by interpolating the Hex-planes R. The motion fields computation can be formulated as:
(X' C, a, r, where X, C, a, r, and
F UII mnterp((1S, (X,
(4)
AX -r
(5)
where equals to the scales in Hex-plane, and interp() denotes interpolating the Gaus- sian points on the specific plane Ç to obtain corresponding motion features. We have Ç E {(x, v). (z, 2). (y, 2), (1, t), (y. t), (z, t)}. Therefore, Gaussian G' at time updated as follows: g' = fx' + C.K,C,a, Ar, s + As}.
(6)
To better preserve the appearance of static 3D objects, we keep certain attributes, specifically opacity a and color C, unchanged.
Coarse Motion Reconstruction. Based on the spatiotemporal consistent multi-view videos generated by VDM, we first leverage a 4DGS reconstruction stage to directly reconstruct the coarse motions. Specifically, we use simple but effective L2 loss as our Live, which is calculated as:
Lree ΣΣ l|c - 11c- ==
where C and é denote the multi-view and multi-frame renderings and the corresponding ground truth. As verified in Fig. 3, this reconstruction stage can already learn high-quality coarse motions by leveraging the generated multi-view videos of MV-VDM.
4D-SDS Optimization. To better model the fine-level motions, introduce 4D-SDS optimization stage to distill the knowledge of our multi-view video diffusion model. The 4D-SDS loss LAD-SDS is variant of z zg-ooccontrrttio SDS loss and can be formulated as:
24 GgCo
LDD- (G, D, = E(g((())) = E.xlelll== - 201131 20
(8)
where z and 2g are latent feature of the rendered image and the estimation of clean latent feature from current noise prediction ca, respectively, 9 represents the rendering function. ag and or are the signal and noise scale controlled by the noise scheduler, respectively.
Training Objectives. In addition Lree and LAD-SDS, introduce variant of As-Rigid-As- Possible (ARAP) loss [46] to facilitate the rigid movement learning as well as the maintenance of the high-quality appearance of the static object. The ARAP loss Carap in our work is defined as:
Carap (p;)
Σ
Σ
W3 ph) R2((F) - p
2 KEN,
where RG is estimated from a rigid transformation using Singular Value Decomposition (SVD) according to [46]:
R =
Σ AEA.
wy - -Pt R,((F) - -
(10)
NN. denotes the set of points within fixed radius of pj. and Wj,k exp(- d,z where djx is the distance between center of p, and pk, measuring the impact of pk on py. This loss encourages the generated dynamic object to be locally rigid, and it enhances the learning with rigid movement.
d
In summary, the training objectives for animating off-the-shelf 3DGS object is:
L= Cree + A2LAD-SDS + Agharap:
(11)
where \1, dg. and Ag are weighted parameters.

---

3.3 Extension to Mesh Animation
To directly utilize high-quality mesh generated from commercial 3D generation tools and crafted by human experts, we extent our framework to mesh animation, producing animated mesh compatible with standard 3D rendering pipelines.
We initialize the 3DGS representation of the given object by vertices and triangles of the static mesh. Specifically, the color is determined by vertex color and average the connected edges for the scale. Opacity and rotation are set to fully visible and zero rotation quaternion, respectively. The coarse 3DGS is animated following the motion reconstruction steps as described in the above sections. We utilize the per-vertex Gaussian trajectory to deform the static mesh in straightforward way without skeleton rigging, control point selection complicated deformation algorithms. As shown in Fig. and our project page, the results are surprisingly good despite the simplicity of the solution.
Experiment
4.1 Setup
Training Dataset. To train our MV-VDM, we build a large-scale multi-view video dataset, MV-Video.
Concretely, render multi-view videos of 37,857 animated 3D models collected from Sketchfab [1]. Each model has 2.2 animations on average, resulting in 83,716 animations in total. Each animation is 2 seconds long at 24 fps. Note that animated models that are not allowed to be used to generate Al programs are filtered. The statistical information of our MV-Video dataset is reported in Table 1. For more details about the rendering settings and data examples, please refer to our Appendix ( (). We will release this dataset to further advance the field of 4D generative research.
Table 1: Statistical information for our multi-view video (MV- Videed dataset.
Model ID
Animations Avg. Animations per ID Max Animations per ID
Multi-view Videos
37,857
83,716
2.2
6.0
1,339,456
Implementation Details. We sample 8 frames evenly for each animation to train our MV-VDM. We use the Adawm optimizer with a learning rate of de 4 and weight decay 0.01, and train the model for 20 epochs with a batch size of 2048. When inference, we set the sampling step to 25 and adopt freeinit [56] to get stable results when animating 3D objects. As for 4D generation, the resolution and feature dimension of the Hex- fle--llanes are set to [100, 100, 8] and 16, respectively. We perform coarse motion reconstruction for the first 1000 iterations with batch size of 32 (4 views, 8 frames), and then add 4D-SDS optimization for another 400 iterations. Learning rate is 0.0015 initially and decreases linearly to 0.0005 the end of reconstruction stage. λ1. A2 and Ag in Eq. 11 set to 10.0, 0.01 and 0.5., respectively. It costs 3 days to train our MV-VDM on 32 80G A800 GPUs, and the optimization for 4D generation takes around 30 minutes a single A800 GPU per object.
Evaluation Dataset. For the evaluation of MV-VDM, we render multi-view images from 128 static 3D objects and then generate multi-view videos conditioned on them. Following the evaluation setting of VBench [23], we use four different random seeds for each object and report the average results. For the evaluation of 4D generation, we generate 25 objects across various categories using the large 3DGS reconstruction model GRM [61]. Input images for GRM and animation prompts used in this work are provided in our Appendix 1..
Evlaution Metrics. We adopt the evaluation protocol proposed in VBench [23], which a popular video generation benchmark consisting of both T2V and 12V evaluation tools. The 12V evaluation protocol contains evaluation metrics, and we choose for our evaluation, i.e., I2V Subject, Motion Smoothness, Dynamic Degree, and Aesthetic Quality, measuring the consistency with the given image, the motion smoothness, the motion degree, and the appearance quality, respectively. We abbreviate them as 12V. M. Sm., Dy. Deg, and Aest. Q. in the experiment section. Values of all metrics are the higher, the better, except for Dynamic Degree, since we observe that sometimes completely failed results present extremely high dynamic degree. For more details about the introduction and calculation of the evaluation metrics, please refer to our Appendix (E.2).
Comparison Methods. We compare our work with 4Dfy [5] and DreamGaussian4D (DG4D) [38] on the task of animating any given 3D object using their official implementations. They represent the state-of-the. thefthhe-rtttt in 4D generation methods, starting by generating static 3D object using 3D-SDS in
7

---

Input 1=0
4Dfy(Gau.)
DG4D
Reconstruction (Ours)
Ours
t=t
t=b b
t=4
t=
t=1 4
t=
t=
t=
"Bear wearing cartoon swimming goggles, ready plunges water, goggles on.
"A cartoon frog with a friendly, wide mouthed smile is walking."
"A cartoon penguin walks awkwardly:: Figure 3: Qualitative comparison with state-of-the-art methods.
the initial stage, and subsequently animating it via video SDS and single-view video reconstruction in later stages. For 4D representation, 4Dfy and DG4D adopt dynamic NeRF [34, 26] and 4DGS [55], respectively. For fair comparison, our work and DG4D, and also apply ARAP loss for motion regularization. For DG4D, we keep the 4DGS representation and motion regularizations in their work unchanged.
replace the dynamic NeRF in 4Dfy with 4DGS used in both
4.2 Comparison with State-of-the-Art
In this section, we perform comprehensive comparisons with previous works, including quantitative and qualitative comparisons and user studies.
Table 2: Quantitative comparisons with state-of-the of-the-rttrtttt methods.
Comparison video gencration metrics.
Comparison study.
12V t M. Sm.
Dy. Deg.
Aest. Q.
Align. Text Align, 3D.
Mot.
Appr.
4Dfy (Gau.) [S]
0.783
0.996
0.0
0.497
4Day(Gau.) [S]
2,028
1.608
1.534
1.84
DG4D [38]
0.898
0.986
0.477
0.529
DG4D [38]
2,824
3.52
2.284
3.108
Ours
0.982
0.991
0.597
0.581
Ours
4.386
4.734
4,288
4.528
Quantitative Comparison. As shown in Tab. 2a, our method significantly outperforms 4Dfy and DG4D in terms of 12V, Dy. Deg., and Aest. Q. This indicates our generation results have good alignment with the given static 3D object (I2V Subject), dynamic motion (Dynamic Degree), and superior appearance (Aesthetic Quality). For Motion Smoothness, we slightly lag behind 4Dfy, since 4Dfy always generates nearly static results, as illustrated by the 0.0 Dynamic Degree in the first row of Tab 2a. Generally, motion, at almost no cost of sacrificing their high-quality appearance, facilitating customized and high-quality dynamic 3D object creation.
method is able animate 3D object with smooth and dynamic
Qualitative Comparison. shown in Fig. 3, is obvious that 4Dfy's 's results are blurred and deviate much from the given 3D object, owing to the use of text-conditioned diffusion models to optimize the motion and appearance. Additionally, the beginning of the training process, the noisy rendered image sequence, i.e., the input to the T2V model, has no temporal changes, which misleads the video diffusion model to generate almost static supervisions, as illustrated in Fig 1. For DG4D, its results align well with the given 3D object in front
generated objects are almost static. This is because at

---

Input t=0
Ours w/o S.T. Attn [- lg
Ours w/o Pre-train
Ours
1-
t=
1
t= 4
t= 12
t= L
"A basketball player is playing basketball." "
Figure 4: Ablation for multi-view video diffusion.
Input (-0 0
Ours w/o SDS loss
Ours w/o ARAP loss
Ours
1
1 lp
[- 4
f
"Cow in cartoon style lifts its front hooves,"
Figure 5: Ablation for 3D object animation.
view, L.e., the view used for generating guided video. However, doesn't it align with the object in the novel view, as indicated by the tail in Bear and Penguin, distorted goggles in bear, and blurred back and side views in Fig 3. This is because adopts Zero123 to optimize the novel views. Zero only conditions the front view, leading to NVS optimization favoring pre-trained data distributions, which could lead to potential appearance degradation. More importantly, DG4D fails when the object in the guided video is assigned with movements toward the camera. For example, the frog is moving forth and back in the guided video, however, DG4D interprets it as enlargement and reduction of the object. The same thing comes the penguin which nods towards the camera and leans forward. This misinterpretation usually results in blurry effect and strange appearance.
In contrast, our method, leveraging the spatiotemporal consistent muti-view prior, manages to deal with motion towards the camera, as demonstrated by the bear's 'ss' raised front paw (our model takes the front view and its orthogonal views as the condition view, not depicted in the image). Besides, successfully maintain the high quality appearance of the given 3D object when generating natural motion. Please refer the videos in our supplementary material for more intuitive comparison.
User Study. We conduct a user study among 20 people on the 25 dynamic objects and report the averaged results in Tab. 2b. The participants are asked to score the generated dynamic objects from to 5, according to the alignment with the given text (Align. Text) and static object (Align. 3D), motion quality (Mot.), and appearance quality (Appr.). The user study proves the superiority of our method. Please refer to the Appendix (E.3) for more details.
4.3 Ablation
Table 3: Ablation Studies
(a) Ablation Multi- View Diffusion
(b) Ablation
4D Generation
12V 1
Sm. 1
Dy. Deg.
Aest. Q.
12V
M. Sm.
Dy. Deg.
Aest. Q. f
w/o S.T. Att.
0.915
0,980
0.958
0.531
w/o SDS loss
0.978
0.990
0.657
0.572
w/o Pre train
0.910
0.981
0.944
0.531
w/o ARAP loss
0.970
0.990
0.573
0.557
Ours
0.935
0.988
0.710
0.532
Ours
0.983
0.997
0.597
0.581
Multi-view Video Diffusion. In Tab 3a, we validate the effectiveness of the proposed SpatioTemporal Attention (short as S.T. Att.) and the pre-trained weight from video diffusion model (short as Pre- train). We replace the proposed spatiotemporal block with temporal block from AnimateDiff, and

---

"A wooden dragon head shakes from right left.
"A cute dog is running and jumping." Figure 6: Visualizations of mesh animation. We present RGB and textureless renderings of two mesh animation results. Best viewed by zooming in.
this leads to performance drop in I2V Subject, Motion Smoothness and Aesthetic Quality. Dynamic Degree seems to be enhanced, but that caused by the increase of unstable failure cases. The same tendency could be observed in experiments w/o pre-trained video diffusion weight. Therefore, we think both designs are necessary for generating multi-view videos consistent with the given multi-view images and with high-quality appearance and motion. Qualitative ablations in Fig further demonstrate this point.
4D Object Optimization.
The ablations for 4D object optimization are shown in Tab. 3b and Fig. 5. The quantitative results Tab. indicate both SDS and ARAP losses improve the alignment with the 3D object (I2V Subject), Motion Smoothness, and Aesthetic Quality. However, the Dynamic Degree decreases. We suppose the decrease in dynamic degree is mainly caused by the removal of floaters and blurry effects, which are also taken into account of dynamic degree, as shown Fig. 5, The two losses might slightly decrease the motion amplitude, but generally, we think the overall performance is improved when applying them.
4.4 Mesh Animation
We provide mesh animation results in Fig. 6. Static meshes are generated by commercial 3D generation tools. For more results, please visit our project page.
Conclusion
In this work, we present Animate3D, novel framework for animating any off-the-self 3D object. Animate3D disentangles the 4D object generation into a foundational 4D generation model, MV- VDM, and a joint 4DGS optimization pipeline based MV-VDM. MV-VDM is the first 4D foundation model, which can generate spatiotemporal consistent multi-view videos conditioned on multi-view renderings of static 3D object. To train MV-VDM, we present the largest multi-view video (4D) dataset, Videoo,o, containing about 84K animations with over 1.3M multi-view videos. Based on MV-VDM we propose an effective pipeline animate any static 3D object by jointly optimizing the 4DGS via both reconstruction and 4D-SDS. Animate3D is a highly practical solution for downstream 4D applications since can animate any generated or reconstructed 3D objects. Data, codes, and pre-trained weights will be released to facilitate the research in 4D generation.
10

---

6
Acknowledgements
The authors would like to thank the anonymous reviewers for their valuable comments and suggestions. This work was supported in part by the Natural Science Foundation of China (Grant No. 62192782, U22B2056, 62422317), the Beijing Natural Science Foundation (Grant No. L223003, JQ22014), the Natural Science Foundation of China (62036011, 1, U2033210, 62102417, 62222206, 62172413), the Project of Beijing Science and technology Committee (Project No. Z231 100005923046) Jin Gao was also supported in part by the Youth Innovation Promotion Association, CAS. This work was also supported by Damo Academy through Damo Academy Research Intern Program.
References
[i] Sketchfab. https: https://sketchfall com/. Accessed: 2024-5-21.
[2] Zeroscope text-to-video model. httpe://huggingfaan co/ co/cerspense/eerccopee.W _2__ee__________ Ac- cossed: 2023-10-31.
[3] Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny, Minigpt4-videos Advancing multimodal Ilms for video understanding with interleaved visual-textual tokens. arXiv preprint arXiv:2404.034/ 2024.
[4] Sherwin Bahmani, Xian Liu, Yifan Wang, Ivan Skorokhodov, Victor Rong, Ziwei Liu, Xihui Liu, Jeong Joon Park, Sergey Tulyakov, Gordon Wetzstein, et al. Te4d: Trajdtoorycoondiiineedd text-to-4d generation. arXiv preprint arXiv:2403. 17920, 2024.
[5] Sherwin Bahmani, Ivan Skorokhodov, Victor Rong. Gordon Wetzstein, Leonidas Guibas, Peter Wonka. Sergey Tulyakov, Jeong Jonn Park, Andrea Tagliasacchi, and David I Lindell, 4d-fy: Text-to-4d generation using hybrid score distillation sampling. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[6] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, al. Stable video diffusion: Scaling latent video diffusion models to large datasets, arXiv preprint arX1v:211..15127. 2023.
[7] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn. Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: Higg-resotution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22563- 22575, 2023.
[8] Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 130-141. 2023.
[9] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-super vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 9650-9660, 2021.
[10] Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang. Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Vidcocrafterl: Open diffusion models for high-quality video generation. arXiv preprint 2r3v 19512, 2023.
[11] Wen-Hsuan Chu, Lei Ke, and Katerina Fragkiadaki. Dreamscene4d: Dynamic multi-object scene generation from monocular videos. arXiv preprint arXiv:2405. 02280, 2024.
[12] Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems, 36, 2024.
[13] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi. and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142- 13153, 2023.
[14] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google seanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA), pages 2553- 2560. IEEE, 2022.

---

[15] Sara Sridovich-Keil. Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12479-12488. 2023.
[16] Quankai Gao, Qiangeng Xu, Zhe Cao, Ben Mildenhall, Wenchao Ma, Le Chen, Danhang Tang, and UIrich Neumann. Gaussianflow: Splatting gaussian dynamics for 4d content creation. arXiv preprint 10X1v:2403.13365, 2024.
[17] Xun Guo, Mingwu Zheng, Liang Hou, Yuan Gao, Yufan Deng, Chongyang Ma, Weiming Hu, Zhengjun Zha, Haibin Huang, Pengfei Wan, et al. 12v-adppeer A general image-to-video adapter for vidco diffusion models. arXiv preprint arXiv:2312. 16693, 2023.
[18] Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao. Dahua Lin, and Bo Dai. Animated- iff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.0477 044755 2023.
[19] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole. Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.0230: 2022.
[20] Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Floct. Video diffusion models, Advances in Neural Information Processing Systems, 35:8633-8646, 2022.
[21] Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205. 1.55.15860 15868, 2022.
[22] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lim: Large reconstruction model for single image to 3d. arXiv preprint vrXiv:2311.04400 2023.
[23] Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tranxing Wu, Qingyang Jin, Nattapol Champaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
[24] Yanqin Jiang. Li Zhang, Jin Gao, Weiming Hu, and Yao Yao. Consistent4d: Consistent 360" dynamic object generation from monocular video. In The Twelfth International Conference on Learning Representations. 2024.
[25] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):1-14, 2023.
[26] Sixu Li, Chaojian Li, Wenbo Zhu, Boyang Yu, Yang Zhao, Cheng Wan, Haoran You, Huihong Shi, and Yingyan Lin. Instant-3d: Instant neural radiance field training towards on-device ar/vr 3d reconstruction. In Proceedings of the 50th Annual International Symposium on Computer Architecture, pages 1-13, 2023.
[27] Zhen Li, Zuo-Liang Zhu, Ling-Hao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng. Amt: All-pairs multi-field transforms for efficient frame interpolation. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[28] Chen- Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d 3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300-309, 2023.
[29] Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, and Karsten Kreis. Align your gaussians: Text-to-id with dynamic 3d gaussians and composed diffusion models. arXiv preprint ar3 2023.
[30] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2- 3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2024.
[31] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1-to-3: Sero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9298 9309, 2023.
12

---

[32] Yuan Liu, Cheng Lin, Zijiao Zeng. Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview--cosiiteett images from single-view image. arXiv preprint urXiv:2309.0345. 2023.
[33] Ben Poole. Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 149888880. 2022.
[34] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10318-103222 10327, 2021.
[35] Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi Zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d. arXiv preprint arXiv:2311.16918, 2023.
[36] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763, PMLR, 2021.
[37] Jeremy Reizenstein. Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In International Conference on Computer Vision, 2021.
[38] Jiawei Ren, Liang Pan. Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu. Dreamgaussian4d: Generative 4d gaussian splatting. arXiv preprint arXiv:2312. 17142, 2023.
[39] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[40] Ruoxi Shi, Hansheng Chen. Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zerol23++: a single image to consistent multi-view diffusion base model, 2023.
[41] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mydream: Multi-view diffusion for 3d generation. arXiv preprint Xiv::3338 16512, 2023.
[42] Uriel Singer, Adam Polyak. Thomas Hayes. Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et Make-a-video: Text-lo- generation without text-video data. arXiv preprint or Xiv:2209. 14792, 2022.
[43] Uriel Singer, Shelly Sheynin, Adam Polyak. Oron Ashual, lurii Makarov. Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et al. Text-to-4d dynamic scene generation. arXiv preprint or Xiv:2301. /280, 2023.
[44] Jascha Sohl-Dicker Dickkstinn, Eric Weiss, Niru Maheswaranathan. and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[45] Jiaming Song. Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint 2xiv:2010.02502. 2020.
[46] Olga Sorkine and Marc Alexa. -s-rri surface modeling. In Symposium on Geometry processing, volume 4, pages 109-116. Citeseer, 2007.
[47] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng. and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054. 2024.
[48] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer EisionEECCy 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 402-419. Springer, 2020.
[49] Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts. David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. arXiv preprint a4Xiv:2403. /2008, 2024.
[50] Haochen Wang. Xiaodan Du Jiahao Li. Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffiusion models for 3d generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12619 12629, 2023.
13

---

[51] Jiuniu Wang. Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang. and Shiwei Zhang, Modelscope text-to-video technical report. arXiv preprint arXiv:2308.0657, 2023.
[52] Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:2312.0220,, 2023.
[53] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems, 36, 2024.
[54] Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su. and Zexiang Xu. Meshlrm: Large reconstruction model for high-quality mesh. arXiv preprint WrXiv2240.112855 2024.
[55] Guanjun Wu, Taoran Yi. Jiemin Fang. Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528, 2023.
[56] Tianxing Wu, Chenyang Si, Yuming Jiang, Ziqi Huang, and Ziwei Liu. Freeinit: Bridging initialization gap in video diffusion models. arXiv preprint ar a331:2331..77537, 2023,
[57] Zijie Wu, Chaohui Yu, Yanqin Jiang, Chenjie Cao. Fan Wang, and Xiang Bai. Sc4d: Sparse-controlled video-to-4d generation and motion transfer. arXiv preprint 2rXiv:2404.03736, 2024.
[58] Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open domaid images with video diffusion priors. arXiv preprint arXiv:2310. 12190, 2023.
[59] Dejia Xu, Hanwen Liang. Neel P Bhatt, Hezhen Hu, Hanxue Liang, Konstantinos N Plataniotis. and Zhangyang Wang. Comp4d: LIm-guided compositional 4d scene generation. arXiv preprint vxXiv:2403.1699/ 2024.
[60] Jiale Xu, Weihao Cheng. Yiming Gao, Xintao Wang. Shenghua Gao, and Ying Shan. Instantmesh: Efficient mesh generation from single image with sparse- view large reconstruction models. arXiv preprint arXiv:2404.0719/ 2024.
[61] Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arXiv preprint 24035 14621, 2024.
[62] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to- image diffusion models. arXiv preprint arXiv:2308.0677: 2023.
[63] Yuyang Yin, Dejia Xu, Zhangyang Wang, Yao Zhao, and Yunchao Wei. 4dgen: Grounded 4d content generation with -patial-temporal consistency. arXiv preprint 172255, 2023.
[64] Jiwen Yu, Xiaodong Cun, Chenyang Qi. Yong Zhang, Xintao Wang. Ying Shan, and Jian Zhang. An- imatezero: Video diffusion models are zero-shot image animators. arXiv preprint arXiv:2312.03793, 2023.
[65] Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Tianyou Liang, Guanying Chen, and Xiaoguang Han. Mvimgnet: A large-scale datasct of multi-vien images. In CVPR, 2023.
[66] Yifei Zeng. Yanqin Jiang, Siyu Zhu, Yuanxun Lu, Youtian Lin, Hao Zhu, Weiming Hu, Xun Cao, and Yao Yao. Stag4d: Spatial-temporal anchored generative 4d gaussians. arXiv preprint arXiv:2403. 14939, 2024.
[67] Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. arXiv preprint arXiv:2404. 10002240441972. 2024.
[68] Yuyang Zhao, Zhiwen Yan, Enze Xie, Lanqing Hong. Zhenguo Li, and Gim Hee Lee. Animate124: Animating one image to 4d dynamic scene. arXiv preprint arXiv:2311.14600 2023.
[69] Yufeng Zheng, Xueting Li. Koki Nagano. Sifei Liu, Otmar Hilliges, and Shalini De Mello. A unified approach for text-and image-guides
scene generation. arXiv preprint arX1v:2311.1654. 2023.
[70] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint 1811: 2018.
14

---

Input t= 0
4Duy(Gau.)
DreanGaussian4D
Ours
t=t
t=
t=
t= t2
t=
t=
"A spiderman is walking.
"A monstrous dog beast attempts to attuck." "
S
"A cool superman is dancing."
Figure Qualitative comparison with state-of-ther ofrthe--rttttttt methods on reconstructed 3D objects.
A More Visualizations
We provide more qualitative results as below. Please refer to the videos attached in our supplementary materials for more intuitive visualizations.
Comparison. In the main paper, we only present the animation results for objects generated by large reconstruction models. But our method is able to animate reconstructed objects as well as generate 3D models from text/image/video, and shows a great advantage over other methods as shown in Fig. 7. In Fig. 7, 4Dfy produces results inconsistent with the given 3D object, and they are almost static. Distortions and blunr effects could be observed in results generated by DreamGaussian4D. and it sometimes deviates from the given object in novel views. In contrast, approach can generate results not only spatially and temporally coherent but also consistent with the input object.
Ablation. We provide more qualitative ablations of MV-VDM and 4D generation in Fig. 8.
B Discussion with respect to Previous 4D Generation Methods
Previous two-stage 4D generation works attempted to disentangle motion learning from appearance learning by adopting different types of supervision signal, i.c., video diffusion/mmonocular video for
15

---

Input
t- 0
Ours w/o S.T. Attm
t=t
t=
Ours w/o Pre-train t= 1
1=4
1=4
Ours
1=t
"A pink dinosaur is waving its little hands"
"A blue treasure chest is being opened."
(a) More ablation studies MV-VDM.
Input t= 0
Reconstmuctinn (Ours)
Ours w/o ARAP loss
Ours
1=b
1=4
t= l2
"Magical panda in 3D cartoon style, with wizard hat and cloak waving its hands cast spell." "
(b) More ablation studies of 4D generation.
Figure 8: Ablations of MV-VDM and 4D optimization.
16

---

motion and image/3D diffusion for appearance. However, the motion and appearance supervisions adopted their work are not orthogonal, and sometimes have negative effect on each other.
For example, is commonly agreed that video-diffusion-SDD usually brings unappealing visual effect the appearance of the object [68, 69, 29]. Meanwhile, the appearance supervision signal prevents 4D object from updating along the direction that follows the exact score function of the video diffusion model, leading less natural motion. Small motion amplitude in [5, 69] and shaky appearance motion learning, previous work [38, 63, 66] rely on 3D diffusion model (Zero123 [31]) supervise both motion and appearance in novel view. Since Zero123-SD 2D-SSDS is applied per-frame, temporal consistency in novel view cannot be guaranteed. Moreover, monocular video doesn't 't provides information about depth/distance, so moving closer to or farther away from the camera can be perceived as the magnification or reduction of the object, resulting appearance distortion.
in [29] could partly support this point. As for monocular-video-guiie
In contrast, our method takes the unified supervision signal from MV-VDM for motion learning and appearance preservation. Our motion and appearance supervision signal inherently don't conflict with each other, since MV-VDM is conditioned on multi-view attributes of the 3D object to generate multi-view videos. Besides, multi-view motion supervision in our work enables more natural motion generation when compared with single-view motion supervision in other works. Thus, we achieve superior performance in terms of both motion generation and appearance preservation in the task of animating any off-the-shelf 3D object.
To provide comprehensive understanding of the difference between our method and previous work, we conduct comparisons with more open-sourced previous works using their official implementations, and summarize the results in Tab. 4.
Table 4: Comparison on 4D Generation. Note that TC4D [4] takes pre-defined object trajectory as the input.
12V
M. Sm.
Dy. Deg.
Aest. Q. 1
CLIP-I f
4Dfy [5] (4DGS)
0.783
0.996
0.0
0),497
0.786
4Dfy [5] (NeRF)
0,817
0.990
0.010
0.549
0.834
Animate 24 [68]
0.845
0.986
0.313
0.563
0.845
4DGen [63]
0.833
0.994
0.187
0,453
0.776
TC4D [4]
0.856
0.992
0.830
0.565
0.859
Dream-in-4D [69]
0.938
0.994
0.0
0.551
0.895
DG4D [38]
0.898
0.986
0.477
0.529
0.860
Ours (8- fr-armme)
0.982
0.991
0.597
0.581
0.946
Ours (16-frame)
0.983
0.991
0.750
0.572
0.937
C Limitations
Despite the promising performance in generating spatiotemporal consistent 4D objects, our method still has few limitations. First, takes relatively long time (about 30 minutes) animate an existing 3D object. Second, there is a trade-off between temporal coherence and motion amplitude in multi-view videos generated from the proposed MV-VDM. Specifically, the larger the motion amplitude, the higher the risk of temporal incoherence. Third, our model sometimes fails to animate realistic scenes due to the domain gap between synthetic training data and real-world test data. At last, current evaluation metrics in 4D generation are not sufficient, as they mainly rely on vidco generation metrics and user studies. Designing more suitable metrics for 4D generation will be an important future work.
D
More Details of Videideo Dataset
D.1 Rendering Details.
For the rendering settings, we first centralize the model according to the bounding box of the object in the first frame. Then, we adjust the camera distance make sure the object stays the scope of view during the animation process. Sixteen views are evenly sampled in terms of azimuth, starting from
17

---

values randomly selected between - and 11.25". The elevation angle is randomly sampled within the range of O0 30". To stabilize training, we manually filter out objects that we identify as challenging to learn due to factors such as large movements, complex environmental interactions, high speed, or sudden changes in appearance.
D.2 Data Examples.
As shown in Fig. 12 and Fig. 13, showcase more examples of our multi-view video dataset (MV-Video).
Furthermore, as shown in Fig. 9, extracted all nouns from the text captions of our MV-Video dataset, which are generated by minigpt4-video [3], and plotted a word cloud of the Top-1000 nouns. We that our MV- Video dataset contains diverse categories of animated 3D objects, including humans, characters, animals, plants, mechanical structures, and ect.,
activities
YARAIN aoca tinn
food
US aisemusic expression equi pmont
car
vehicle,
e
geal
@
Clothing
head
dance
a
M
type
platform
naa
envvronnen
Fonnents
IO
50
S
pose KT
SWOTES
Tocus
inoestonn
machine
denonstratet
Foy
captur
vatte
Bir
clip
FiSh
wa
series
Rnteraal
FAIK
body
nenurs
W
tree: eye
orange
carmore
dragan
a
Ion
circle
ang
le
saages
aa
version
car
mode
effect
Big
DOgE...
sccenar
rock
k
change
Compu
Feot
taetaal
ChiLE
thing
part
wing
TACK
D Faco
fore
people
Dall
LEOP
Stage
V
Lten
oroonnd
shot
time
AT
Lop
Jog
Fist
creature
amim
device
mait
Lifer sinstee
t
Lon
gun
We
eapon
ccootee
Figure 9: Illustration of the word cloud of the top 1000 nouns extracted from the text captions of our
MV-Video dataset.
E 4D Generation Evaluation
E.1
Evaluation Dataset
In Fig. 10, we provide the input images for image-to-3D generation and corresponding text prompts for 4D animation used Sec. 4.2.
E.2 Evaluation Metrics
VBench [23] provides six evaluation metries for 12V evaluation", i.e., I2V Subject, I2V Background, Camera Motion, Subject Consistency, Background Consistency, Motion Smoothness, Dynamic Degree, Aesthetic Quality and Imaging Quality. Since our gener- ated results have no background, and the evaluation cameras are fixed, metrics related to background and camera motion are used. The metric Imaging Quality, which is affected by ambient light, is also not used here. Subject Consistency is also omitted since its calculation process is similar to I2V Subject, except for the choice of reference frame. takes the first generated frame, instead of the input image used in I2V Subject, as the reference frame for similarity score calculation, which is not suitable for our task of animating 3D objects. We briefly introduce the evaluation metrics used in our work as below:
18

---

curtoon bear with seft, cuddly for along, fur swaying gently.
cartoun dolphin with playfed smile and sparkling eyes leaps joyfully from the waves.
A cartoon eugle with grand, ouistretched wings circles flying the sky.
A cartoon elephant with colorful, patchwork shi pattern lumbers stendily, patterns shiftiin.
A curtoon frog with friendlly, wide mouthed smile walking.
curtoon parrot with bright, multi colored feathers stretching neck to look around.
cartoon penguin walks awkwardly.
A cartoon rubbit with long, flopyy cars hops along, bobbing merrily,
A curtoon sheep with fluffi cloud like wool wool rippling gently.
A cutte fox, 3D animation rests in place, gendly nodding its head.
Bear tutue, looking bushful shuffles shyly, swaying,
Bear wearing swimming gugeles, reudy plunges water, goggles on.
curtoon elephant with colorful, patchwork skin pattern lumbers stendity, patterns shifting.
Bison with 3D cartoon winter coot shakes off flakes, coat ruffling.
Car cartoon colorfully pointed driving along road.
Cut with cartoor pirate eye patch trying reach for something.
Chichen cartoon, with superhern struts valiantly, cape floving.
Chicken cartnon style, donning tiny crown confidently around the coop.
in style lifts lts front hooves.
Dolphin with sunglasses, stylish leaps gracefully from the
Shadowy ninja fax poised for action, in dynamic animation unsheathes glinting katana.
Monkey with cartoon sunglasses, looks around.
Magical panda in 3D cartoon style, with wizard hat and cloak waving its spell.
Sheep in cartoon style, with reuding glasses reads book.
Space shuttle cartoon style, bousters ready, flying.
Figure 10: Illustration the input images for image-to-3D generation and corresponding prompts for 4D animation.
I2V Subject assess whether the appearance of the object in the generated video remains consistent with that in the input image. To this end, DINO [9] feature similarity across frames calculated.
Motion Smoothness evaluates whether the motion in the generated video is smooth, and follows the physical law of the real world. The motion prior in the video frame interpolation model [27] is utilized for evaluation.
Dynamic Degree employs RAFT [48] to estimate the degree of dynamics in synthesized videos.
Aesthetic Quality is calculated by the LAION aesthetic predictor, which reflects the artistic and beauty value perceived by humans towards each frame.
E.3 User Study Template
As illustrated Fig. 11, picture of the user study page is depicted. The survey contains 25 dynamic objects, which are shown in Fig. 10. The participants are asked to score the generated 4D objects from to 5, according to the alignment with the given static object and text prompt, appearance quality, and motion quality.
19

---

Every three constitute group, representing the animation generations for the same object sing three different methods (the order has been shuffled). Please compare the generated res ults within the same group and then provide scores.
1.
Please rate the animated results for the static 3D model in conjunction with the video. (The first row fea tures reference images of the 3D model, while the second row displays the generated animation results. The first and second columns represent two different viewpoints, respectively.)
The desired text description for the animation is: "Sheep in cartoon style, with reading glasses, reads a book."
Score:
2
5
Consistency with the static 3D model:
Consistency with text prompt:
Aesthetic quality of the results:
Movement quality of results:
Gif
2.
Figure 11: The layout of our user study,
F Border Impacts
This paper exploited 4D generation based on our proposed multi-view video diffusson model, which can generate spatiotemporal consistent multi-view videos. Because of the advanced generative capacity, our models may output misinformation or fake videos. Thus, we sincerely remind the users pay attention to it. Note that our method only focuses on the technical aspect. All the code, dataset, and trained models will be released.
20

---

Figure 12: More examples of our MV-Video dataset.
21

---

Figure 13: More examples of our VV-Video dataset.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
our abstract clearly clarifies our contributions: 1) We propose Animate3D, the first 4D generation pipeline to animate any 3D objects. 2) We propose the first 4D generation foundation model, MV-VDM. 3) We present the largest 4D datasets, MV-Video.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: we have included our limitation in Sec. C.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiattinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
23

---

Answer:
[NA]
Justification:
This paper did not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: experiments are reproducible, and all codes, dataset, and pre-trained models will be open-relcased.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
24

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
All data, codes, and pre-trained weights used in experiments will be open-released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips...
oeeiiooolloooo icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wctpublic//aeccccccccccccccc /gwpi/eg///////eeeeee /Cae//ooeubbiiic soteeuubbisiiono. ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: we include detailed implementations in See. 4.1. The full details can be seen in our codes, which will be open-released.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
As a visual generative model, we provide sufficient visualization results
instead.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
25

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: we provide the information on the computer resources in Sec. 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //tpprr///////////////e/////ssss ttt.ppppccpppppii//////ccpcpcccc csGuideliness
Answer: [Yes]
Justification: our work conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
we provide the it in Sec. F.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
26

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: we have cited related works and codes.
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwode.hco....c. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
27

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release new assets for now. Codes, datasets, and trained models will be open-released soon.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
28


Reviews:
[{"review_id": "h19x1822FI", "reviewer": "Reviewer_Tnsn", "summary": "This paper focuses on animating 3D objects with multi-view diffusion models. To improve spatial and temporal consistency, this work builds a large-scale multi-view video dataset, MV-Video and designs an attention module to enhance spatial and temporal consistency by integrating 3D and video diffusion models. To enhance animations from 3D objects, this work jointly optimizes the 4DGS through reconstruction and 4D-SDS. Experiments show the proposed method can more consistent 4D generation.", "questions": "* It is better to validate the effectiveness of the proposed dataset, can it also improve single-view diffusion models (e.g., T2V) or multi-view image-to-3D generation models (e.g., MVDream). Then, can these models further improve other methods, like DreamGaussian4D?\n* Although the work verify the effectiveness of the S.T attention. The effect of MV2V-adapter is not validated.\n* If the approach leverages better 4DGS reconstruction methods, such as [1] for the coarse motion reconstruction, can the 4D-SDS still improve the coarse results?  \n[1] Yang, Zeyu, et al. \"Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting.\" arXiv preprint arXiv:2310.10642 (2023).  \n* As this work designs a multi-view video diffusion model, can the authors compare it with existing single-view video diffusion models?\n* For Figure 4, both the ablation models show that the player touches the basketball. However, the full model does not show this interaction. I wonder if the full model can do better for this interaction.", "limitations": "The paper has discussed the limitations.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* This work builds a large-scale multi-view video dataset to facilitate the training of multi-view video diffusion models.\n* The method designs an attention module to encourage spatial and temporal consistency for multi-view diffusion models. \n* The approach introduces 4D-SDS to leverage multi-view video diffusion models for animating 3D objects.\n* The paper is easy to read and understand.", "weaknesses": "* This work focuses on 3D objects, however, the paper title claims \"Animating Any 3D Models\". It is a bit inappropriate.\n* The effectiveness of the proposed dataset is not fully validated. Can this dataset improve single-view diffusion models or multi-view image-to-3D generation models, like MVDream?\n* The diversity of the proposed method is unclear. Can one object perform different actions? For example, in Figure 3, can the frog perform swimming?\n* The picture quality is a bit low, it is hard for readers to distinguish the advantages of the generated objects from the proposed method. \n* For animating from 3D object, this work first leverages 4DGS to reconstruct coarse motions and then uses 4D-SDS for refinement. If the method uses better 4DGS algorithms, maybe the improvement from 4D-SDS will become little.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Inappropriate title: 3D objects instead of 3D models**\n\nThanks for the advice, we will consider revising it in the revised version.\n\n**W2: Effectiveness of the proposed dataset in video or 3D diffusion models**\n\nGiven limited time, we only finetune SVD on a subset (20\\%) of our dataset, and glad to see some improvements. For evaluation, we evaluate the finetuned model on the validation dataset used for the ablation of MV-VDM. \n\n| Method         | **I2V $\\uparrow$**   | **M. Sm. $\\uparrow$** | **Dy. Deg.** | **Aest. Q. $\\uparrow$**   |\n|----------------|-----------|------------|---------------|----------------|\n| SVD            | 0.927     | 0.985      | 0.849         | 0.538          |\n| SVD-finetuned  | **0.950** | **0.990**   | 0.708         | **0.557**      |\n\n\nWe also use the generated video from finetuned model as the input for DG4D in 4D generation and report the results as below.\n\n|    |   **I2V $\\uparrow$**   |   **M. Sm. $\\uparrow$**   |  **Dy. Deg.**  |  **Aest. Q. $\\uparrow$**   |\n|---|--------|-----------|------------|------------|\n| **DG4D w/o finetuned SVD** | 0.898 | 0.986 | 0.477 | 0.529 |\n| **DG4D w/ finetuned SVD** | **0.907** | **0.989** | 0.407 | **0.533** |\n\nPlease note that the decrease in **Dy. Deg.** is because SVD w/o fintuning leads to artifacts and noise in the generated input video, hence leading to the failure of 4D generation. We believe that further improvements could be achieved by fintuning it on the full training dataset.\n\n**W3: Motion Diversity: object performing different actions.**\n\nThanks for your insightful question. Please refer to our global response (3. Motion Diversity and Dataset).\n\n**W4: About the picture quality.**\n\nActually, the picture quality is mainly affected by the quality of static 3D generated/reconstructed object, since our method is good at preserving the appearance of the given object. We should clarify that the reconstruction quality is not the primary contribution of this paper, which is potentially influenced by the hyper-parameter adjustment of GRM (or other reconsturction method). Besides, The compression of images in PDF can lead to a certain degree of loss in image quality.\n\nWe provide high-quality images in Figure 6-8 in PDF attached. Although they do not have the same quality as the original renderings at 1024$\\times$1024 resolution, they are better than those provided before.\n\n**W5: Better 4DGS algorithms will make 4D-SDS unnecessary**\n\nThough MV-VDM generates spatiotemporally coherent multi-view videos as the ground truth for reconstruction, **the ground truth has only 4 views**. Existing 4DGS algorithms **cannot straightforwardly address this sparse multi-view video reconstruction.**\n\nFollowing your suggestion, we have experimented with an improved 4DGS reconstruction algorithm [2]. The results are reported in the table below. Better 4DGS[2] doesn't improve the reconstruction results primarily because the task here involves sparse multi-view video reconstruction using only 4 views. Better 4DGS[2] is not designed to tackle such scenarios.\nIn our work, we apply arap loss to 4DGS[1] to effectively handle sparse views. We fail to apply arap loss to Better 4DGS[2] given that arap loss is designed to constrain the motion of the 3D Gaussians, however, Better 4DGS[2] doesn't learn the motion (It regards the time dimension as a property of Gaussian points, similar to scale/rotation/opacity). We find the proposed 4D-SDS improves the results of Better 4DGS. We will add this discussion in our revised paper.\n\n| Reconstruction | 4D-SDS | **I2V $\\uparrow$**  | **M. Sm. $\\uparrow$** |  **Dy. Deg.**  | **Aest. Q. $\\uparrow$**|\n|---|---|---|---|---|---|\n| 4DGS [1] |   | 0.978 | 0.990 | **0.657** | 0.572 |\n| Better 4DGS [2] |   | 0.972 | 0.990 | 0.621 | 0.561 |\n| 4DGS [1] | $\\checkmark$ | 0.983 | **0.997** | 0.597 | **0.581** |\n| Better 4DGS [2] | $\\checkmark$ | **0.984** | **0.997** | 0.610 | 0.573 |\n\n[1] Wu, Guanjun, et al. \"4d gaussian splatting for real-time dynamic scene rendering.\" (CVPR2024)\n\n[2] Yang, Zeyu, et al. \"Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting.\" (ICLR2024)\n\n**Q1: Validate the effectiveness of the proposed dataset.**\n\nPlease refer to the response of W2.\n\n**Q2: Ablation of MV2V-adapter**\n\nPlease refer to our global response (2. Ablation of MV2V-Adapter)\n\n**Q3: The effectiveness of better 4DGS algorithms.**\n\nPlease refer to the response of W5.\n\n**Q4: Comparison with existing single-view video diffusion models**\n\nWe conducted some empirical comparisons between our MV-VDM and SVD, finding that our model outperforms SVD when animating 3D object renderings, as we specifically trained our model in this domain. However, we struggled to animate realistic images with complicated backgrounds, which is an area where SVD excels. It’s important to note that the comparison is made between the multi-view video results of our model and the single-view video results of SVD, which may be an unfair assessment for our model. For quantitative evaluation, we tested SVD on the dataset used for the ablation study of MV-VDM, and the results are reported in the table below.\n\n| Method |**I2V $\\uparrow$** |**M. Sm. $\\uparrow$** | **Dy. Deg.** | **Aest. Q. $\\uparrow$** |\n| --- | ---| ---| ---| ---|\n| SVD | 0.927 | 0.985 | 0.849 | **0.538** |\n| Ours | **0.935** | **0.988** | 0.710 | 0.532 |\n\n**Q5: Issue about the basketball interaction in Fig. 4**\n\nYes, it has the interaction. Please refer to the Figure 3 in our attached PDF.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sDZCD3Ru0U", "reviewer": "Reviewer_9MgC", "summary": "This paper proposes an animation method that animates a 3D model in a 4D one. A  Multi-View image conditioned multi-view Video Diffusion Model  (MV-VDM)  is presented to generate multi-view videos from multi-view renderings of a static 3D object. The MV-VDM is leveraged to train the 4D Gaussian Splatting (4DGS), where As-Rigid-As-Possible (ARAP) loss, reconstruction loss and SDS are used as objectives. In addition, a multi-view video dataset is constructed to train MV-VDM. The paper conducts experiments to show the effectiveness of the proposed method.", "questions": "Please refer to my question above.", "limitations": "The authors have provided  limitations and  societal impact of their work", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea is well-motivated and straightforward.\n2. A large-scale multi-view video dataset is presented.\n3. A multi-view video diffusion model is presented, where spatiotemporal attention is introduced to animate multi-view images with motions.", "weaknesses": "1. The comparison is not very fair. The paper focuses on a new task, i.e., 3D animation, while 4Dfy focuses on  text-to-4D, which is a different task. Furthermore, text-to-4D is more challenging than 3D animation since multi-view images are unavailable.  The better performance gain of the proposed method may partly attributed to additional multi-view images. In other words, the worse performance of 4Dfy and DG4D may not be due to the methods themselves. In addition, the paper replaces dynamic NeRF in 4Dfy with 4DGS. However, some hyperparameters of 4Dfy are set according to dynamic NeRF, rather than 4DGS. Instead, the paper can compare the proposed method with AYG, which directly trains 4DGS. In addition, the paper can compare Animate234, which is an image-to-4D method.\n\n2. The motivation for the proposed spatiotemporal attention block is not clear. Besides the temporal motion module and temporal encoding, the block introduces a multi-view 3D attention module. What is the multi-view 3D attention module used for? Why are temporal motion modules and temporal encoding not enough to generate motions for multi-view videos? \n\n3. What is the influence of Alpha Blender on the performance of the proposed method?\n \n\n4. Does the proposed method train all modules in the spatiotemporal attention block? It's not clear whether the \"Multi-view 3D Attention\" and \"Temporal Motion Module (Pre-trained VDM)\" are trained or frozen.\n\n5. The implementation details of Alpha Blender are not clear. Is it implemented as a layer of MLP? \n\n6. The temporal encoding is unclear. According to Figure 2, there is a temporal encoding module in the spatiotemporal attention block, but there is no description of what kind of temporal encoding is used.  \n\n7. How many views does each animated 3D object contain in the dataset? Only four orthogonal views or more?\n\n8. Although the dataset contains 38K animated 3D objects, it is still much smaller compared to 2D video datasets. Can the proposed method trained on this dataset of this magnitude animate any 3D model?\n\n9. Training a multi-view video diffusion model requires camera parameters. Are the camera parameters in the multi-view video dataset processed to ensure that they are consistent with the prior knowledge learned in the MVDream", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: (1) Issues about unfair comparison: 4Dfy and DG4D do not leverage multi-view images; (2) 4Dfy is based on nerf instead of 4DGS; (3) Add comparison with AYG and Animate124.**\n\n**(1)**: We proposed a new task of animating any off-the-shelf 3D object, and there is **no previous work specially designed for this task to compare with**, so we could only chose two representative 4D generation works to serve as the comparison baselines. Due to the lack of unified spatiotemporal consistent supervision, existing works (e.g., 4Dfy, DG4D) can only focus on using text or single-view image conditioned diffusion models for motion modeling. Experiments have shown that this often leads to spatiotemporally inconsistent 4D generation results. Our Animate3D is the first 4D generation framework to animate any 3D objects with detailed multi-view conditions. We believe this workflow is more suitable for generating spatiotemporal consistent 4D objects by leveraging advanced static 3D object reconstruction and generation literature.\n\n**(2)** and **(3)** Thanks for the suggestion. Please refer to our global response (1. Comparison methods)\n\n**W2: Motivation and effectiveness of the proposed spatiotemporal attention block.**\n\nTo enhance the spatial and temporal consistency of our MV-VDM, we design a new spatiotemporal attention module, which mainly contains a temporal attention branch and a spatial attention branch. This block is build upon the motion module of video diffusion model to inherit the temporal prior. In the early experiments, we found that simply adopting the temporal attention branch is not enough. This is because the features are likely to lose multi-view consistency after being processed by the temporal branch. Therefore, we add a parallel spatial attention branch to address this issue. The effectiveness is validated in **Tab. 3(a) and Fig. 4 in our original paper**. Note that **w/o S.T. Attn** means we replace the proposed spatiotemporal block with motion modules in video diffusion models. We will add this details in our revision.\n\n**W3: The influence of the Alpha Blender.**\n\nIn our spatiotemporal attention block, we employ an alpha blender layer with a learnable weight to fuse the features from both temporal and spatial branches. In the following table, we show the influence of the alpha blender layer and the results validate that it can enhance the spatiotemporal consistency of the 4D generation results.\n\n| Method | **I2V $\\uparrow$**  | **M. Sm. $\\uparrow$** |  **Dy. Deg.**  | **Aest. Q. $\\uparrow$**|\n| --- | ---| ---| ---| ---|\n| w/o Alpha Blender | 0.911 | 0.982 | 0.958 | 0.528 |\n| Ours | **0.935** | **0.988** | 0.710 | **0.532** |\n\n**W4: Implementation details: trainable modules in spatiotemporal attention block.**\n\nYes, we train all modules in our proposed spatiotemporal attention block, as illustrated in Figure 2 in our paper. Thanks for your reminder, we will add these details in implementation section in our revised version.\n\n**W5: Implementation details: Alpha blender**\n\nAs illustrated in Eq. (1) in our paper, we perform alpha blender with a learnable weight $\\mu$, which is implemented via nn.Parameter. We will add this detail in the revised version.\n\n**W6: Implementation details: Temporal encoding**\n\nThanks for your reminder, we adopt sinusoid positional encoding in AnimateDiff as temporal encoding. We will add these details in our revised version.\n\n**W7: Implementation details: number of views of each aniamted 3D object in our dataset.**\n\nPlease refer to the Appendix (C.1 Rendering Details), 16 views are evenly sampled in terms of azimuth, starting from values randomly selected between $-11.25^{\\circ}$ and $11.25^{\\circ}$. The elevation angle is randomly sampled within the range of $0^{\\circ}$ to $30^{\\circ}$.\n\n\n**W8: Training dataset size is limited, is it capable of animating any 3D object?**\n\nAdimittedly, our dataset is much smaller than 2D video datasets, but since we inherit the prior knowledge learned by 3D and video diffusion models, our model can animate most dynamic 3D objects commonly seen in daily life. We have tested various categories of objects, including humans, mammals, reptiles, birds, insects, vehicles, weapons, etc, and obtained good results. We provide more examples in the **PDF** attached. Please also refer to our global response (3. Motion Diversity and Dataset) to see our discussion about the dataset.\n\n**W9: Implementation details: camera parameters consistent with MVDream?**\n\nYes, the camera parameters in our multi-view video dataset are processed to ensure that they are consistent with the prior knowledge learned in the MVDream. Please refer to the Appendix (C.1 Rendering Details) in our paper, 16 views are \\textbf{evenly} sampled in terms of azimuth. Specifically, during training, we randomly sample four orthogonal views to form a multi-view video for each animation, which is consistent with MVDream's camera setting.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ypZ8wl1w0X", "reviewer": "Reviewer_C8Es", "summary": "This work presents Animate3D, a framework for animating static 3D models. The core idea involves two main components:\n1. A multi-view video diffusion model (MV-VDM) conditioned on multi-view renderings of the static 3D object, trained on a large-scale multi-view video dataset (MV-Video).\n2. A framework that combines reconstruction and 4D Score Distillation Sampling (4D-SDS) to utilize multi-view video diffusion priors for 3D object animation.\nThe animation process involves a two-stage pipeline: coarse motion reconstruction from generated multi-view videos, followed by 4D-SDS to model fine-level motions. Quantitative and qualitative evaluations show enhancements to previous methods.", "questions": "1. The results of 4D-fy appear strange. When I check the 4D-fy results in TC4D and 4DGen, they look more reasonable.\n\n2. I think the authors should also conduct ablation studies on MV2V-Adapter.", "limitations": "The authors have discussed the limitations and potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. Performance: The proposed method achieves state-of-the-art results. The experiments well validate the effectiveness of the proposed methods.\n\n2. Clarity: The paper is well-written and easy to follow.\n\n3. Technical Novelty: The main contributions of this paper are threefold: 1) The first 4D generation framework to animate any 3D objects with detailed multi-view conditions, which are incorporated by the proposed MV2V-Adapter.  2) The authors propose to animate any off-the-shelf 3D models with unified spatiotemporal consistent supervision, which can get better results. 3) The collected 4D dataset, MV-Video.", "weaknesses": "1. Missing Reference [a]: It is understandable that you did not compare Animate3D with Animate124 in this paper, as 4D-fy and DreamGaussian4D both demonstrate better performance compared to Animate124. However, it is unusual that you did not discuss Animate124 at all, given its relevance in previous comparisons by 4D-fy and DreamGaussian4D.\n\n2. Limited Comparison Scope: The authors only compare Animate3D with 4D-fy and DreamGaussian4D, which seems insufficient. It would be more comprehensive to include comparisons with 4DGen and TC4D, as both claim superior performance over 4D-fy and have released their code. Additionally, while the modification of 4D-fy to a Gaussian representation for fair comparison is understandable, the original 4D-fy results should also be included for a thorough ablation study.\n\n[a] Animate124: Animating One Image to 4D Dynamic Scene, \\\n[b] 4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency, \\\n[c] TC4D: Trajectory-Conditioned Text-to-4D Generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Missing Reference: Animate124**\n\nSorry for missing Animate124, which is a great pioneering work. Comparison is in the global response (1. Comparison methods), required by other reviewers. Reference will be added in our revised paper.\n\n\n**W2: More comparisons with 4DGen, TC4D and original 4Dfy.**\n\nThanks for your suggestion. Please refer to the global response(1. Comparison methods)\n\n**Q1: 4Dfy results is strange**\n\nThanks for this point. Actually, 4Dfy is not designed for animating off-the-shelf 3D objects, instead, it is designed for text-to-4D (the task in TC4D/4Gen). So it doesn't fit our task and presents not so good results. Technically, we follow the default training setting in 4Dfy's official implementation for animating off-the-shelf static Gaussian/NeRF. Notably, that implementation reduces the learning rate of static Gaussian/NeRF to a relatively low value in the dynamic stage in hope that high-quality appearance would be preserved. Admittedly, when the static object is generated by image/3D SDS loss and **the exact image/3D SDS loss** is continually used in the dynamic stage, the high-quality appearance could be preserved, as shown in previous works. However, the image/3D SDS loss doesn't match the off-the-shelf 3D object in our task, so using it in the dynamic stage as the original paper results in appearance changes. As the learning rate for static NeRF is very low, the appearance change doesn't finish at the end of the training, so the results are strange.  We tried to use a higher learning rate, but found that resulted in very low **I2V** values, as the object appearance was changed completely.\n\nBesides, we found 3DGS sometimes fail to generate good results when supervised by MVDream SDS loss. This problem was discussed by other researchers in issue section of threestudio-3dgs repo.\n\n**Q2: Ablation of MV2V-Adapter**\n\nPlease refer to our global response (2. Ablation of MV2V-Adapter)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V6rUADsFjE", "reviewer": "Reviewer_G4mL", "summary": "This paper proposes Animate3D, a 4D generation framework that consists of a multi-view video diffusion model (MV-VDM) followed by 4D Gaussian Splatting optimization, as well as a dataset of 38K animated 3D objects (MV-Video) that is used to provide spatiotemporal supervision to train the model. Different from text-to-4D approaches, Animate3D specifically tackles the problem of animating 3D assets which requires spatiotemporal consistency as well as identity preservation. The proposed architecture is based on a multi-view diffusion model (MVDream) and a video diffusion model (AnimateDiff) with additional spatio-temporal attention blocks. To animate a given 3D asset, the trained model is used for both reconstruction and distillation. First, a multi-view video is generated conditioned on multiple views of the asset. The video is used to optimize a 4D Gaussian Splatting resulting in a model with coarse motion. The same MV-VDM model is used for 4D Score Distillation Sampling to refine the motion. The method compares favorably to state-of-the-art 4D generation methods.", "questions": "**Q1. [Based on W1]** Please elaborate on the difference between _3D animation via multi-view video diffusion_ and _two-stage text-to-4D approaches_. In the latter case, the second stage could be viewed as 3D animation. I understand that a multi-view video foundation model would offer spatiotemporal consistency that other approaches can likely not reach, but I would appreciate a more in-depth discussion about the difference to and the challenges of text-to-3D-and-3D-to-4D approaches than what is currently provided in the paper. \n\n---\n\n**Q2. [Based on W1]** Ideally, the authors should provide also empirical comparisons to additional methods (e.g., Dream-in-4D), if possible.   \n\n---\n\n**Q3. [Based on W4]** The authors state that\n>We find this simple cross-attention operator [MV2V-Adapter] can effectively improve the object’s appearance consistency in the generated video.\n\nThe effect of MV2V-Adapter should be better demonstrated with an ablation. \n\n---\n\n**Q4. [Based on W4]** Are the cross-attention layers described in L170-172 necessary for identity preservation and alignment with the text prompt? Their effect should be also demonstrated in the ablation studies. \n\n---\n\n**Q5.** The proposed 3D animation approach uses both reconstruction and distillation. It first uses the generated multi-view video to optimize a 4D Gaussian Splatting with coarse motion. Then SDS is used to refine the motion. It seems counter-intuitive that SDS would be good at modeling the finer details of the animation, since SDS is typically known for its over-smoothing behavior. The authors should further discuss the motivation behind using SDS during 4D optimization and provide additional examples to prove its effectiveness. \n\n---\n\n**Q6. [Based on W6]** Could the authors elaborate on the apparent lack of diversity in the generated motions? It may be helpful to provide some statistics from motion descriptions included in the dataset. Or is this a limitation of the model instead?\n\n---", "limitations": "Limitations are adequately discussed in Appendix B.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**[S1]** The problem setting in this paper is interesting and timely, in an area that has recently attracted significant attention. The paper takes a slightly different angle compared to text-to-3D, text-to-4D or video-to-4D methods, focusing on animating 3D assets based on a textual description instead of generating them from scratch. \n\n**[S2]** This problem setting also opens up a new challenge, which is to preserve multi-view attributes during the animation. It is nice to see that the identity preservation and the animation are significantly improved compared to state-of-the-art methods.\n\n**[S3]** The authors tackle this problem by training a multi-view video diffusion model. Given the success of image (2D), video (2D+time), and multi-view (3D) diffusion models, MV-VDM (4D) is a reasonable next step to ensure spatial and temporal consistency simultaneously.  Both the model and the dataset used to train it are likely to influence future work and could thus have significant impact. \n\n**[S4]** The proposed architecture aims to reuse existing multi-view and video diffusion models, which are already pre-trained on a larger scale of data, thus taking advantage of the priors built into these models.\n\n---", "weaknesses": "**[W1]** The data used in this paper are animated 3D models collected from Sketchfab (cited). The authors promise to distribute the collected dataset. However, there is not sufficient information provided about copyrights and licensing. The only piece of information provided is that models with the following clause are excluded: \n\n>NoAI: This model may not be used in datasets for, in the development of, or as inputs to generative AI programs.\n\nBased on the paper checklist, the paper is flagged for ethics review.  \n\n---\n\n**[W2]** Lack of clarity in the method section and method figure\n\n**(a)** The method section is very hard to parse. The notation is overloaded but not very well explained. The section would benefit from thorough revision to improve the overall clarity and quality. \n\n**(b)** It is almost easier to understand the method through Fig. 2, but even that is dense and inconsistent, and often the notation does not align with what is written in Sec. 3.1. For example, in L175 latent features $z$ seem to be the output of the image encoder. In Fig. 2 (middle), $z$ appears to be the input to the spatiotemporal attention module. In Eq. 1, $X_l$ and $X_r$ are instead the inputs to the same module. At the same time, in Fig. 2 (left) the input to the spatiotemporal module seems to be the sum of the output of the cross-attention layers, not directly $z$. \n\nThis makes reading and understanding the method extremely difficult to the point where the reviewer cannot fully judge the technical contribution and similarities or differences to related models. \n\n---\n\n**[W3]** Discussion with respect to 4D generation methods could be expanded\n\nExisting work in text-to-4D generation often splits the problem into two stages: a static 3D asset generation and a deformation field to model motion. This disentanglement makes it possible to learn the 3D motion from video diffusion models, without the need for multi-view video. The paper briefly touches upon this and compares to two 4D generation methods, 4Dfy and DreamGaussian4D. However, the paper could further elaborate on the similarities and differences of the proposed approach compared to existing methods and the advantages of a multi-view video diffusion model. \n\n---\n\n**[W4]** Certain architectural components are not ablated and their contribution to the overall performance is unclear. See Questions. \n\n---\n\n**[W5]** The temporal extent of MV-VDM is only 8 frames, which seems rather limited.\n\n---\n\n**[W6]** In some of the qualitative examples, the faithfulness to the text prompt is questionable. Some of the animations provided in the supplementary video are not accompanied by a text prompt, while quite a few animations appear very similar. \n\n---\n\n**[Minor]** Typos and suggestions:\n\n- L103: pioneer → pioneering\n- L113: All these manners aforementioned → All aforementioned work\n- L131-132: this sentence can be improved for clarity since the two parts sound a bit repetitive\n- L135: spatial → spatially; temporal → temporally \n- L156: It should be better explained how what $X_l$ and $X_r$ represent and how these features are obtained.  \n- L292: levering → leveraging; manage → manages\n\n---", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Data copyrights and licensing**\n\nWe confirm that all models downloaded from Sketchfab have a distributable Creative Commons license and were obtained using Sketchfab’s public API. Besides, models marked as ``NoAI'' and restricted due to objectionable or adult thematic content were excluded from the dataset. We provide a detailed elaboration of the licensing information as below.\n\n| License               |   Number       |\n|-----------------------|---------------|\n| CC Attribution        |   37,081      |\n| CC Attribution-NonCommercial    | 460    |\n| CC Attribution-NonCommercial-NoDerivs   | 112   |\n| CC Attribution-ShareAlike       |    73 |\n| CC Attribution-NonCommercial-ShareAlike     |    66  |\n| Free Standard         |    38       |\n| CC Attribution-NoDerivs     |    18  |\n| CC0 Public Domain      |    9      |\n| **Total**                   |  37,857 |\n\n**W2: Clarity of the details of our method.**\n\nThanks very much for careful reading. Sorry for the confusing presentation. We will revise the figures and the text in the updated version.\n\n**W3: Discussion with respect to 4D generation methods**\n\nBenefiting from the unified spatiotemporal consistent supervision of our 4D foundation model (MV-VDM), our approach can leverage the multi-view attributes of 3D objects to achieve more consistent 4D generation. In contrast, existing methods based on separated text or single-view diffusion models and video diffusion models struggle to preserve spatiotemporal consistency.\n\nMore importantly, they also fail to faithfully maintain the multi-view attributes of an existed 3D object. \n\n**Due to limited space，we would like to discuss this further in the discussion phase**.\n\n**W4: More components ablations.**\n\nWe provide the ablation for MV2V-Adapter, alpha blender and cross-attn layer (image) as follows. Note that we don't ablate the cross-attn layer (text) since it is a necessary component of MVDream and we freeze MVDream in our pipeline. The results indicate that all components are effective.\n\n| Method | **I2V $\\uparrow$** | **M. Sm. $\\uparrow$** | **Dy. Deg.** | **Aest. Q. $\\uparrow$** |\n| --- | --- | --- | --- | --- |\n| w/o cross-attn layer (image) | 0.887 | 0.978 | 0.966 | 0.504 |\n| w/o alpha-blender | 0.911 | 0.982 | 0.958 | 0.528 |\n| w/o MV2V-adapter | 0.927 | 0.986 | 0.961 | 0.526 |\n| **Ours** | **0.935** | **0.988** | 0.710 | **0.532** |\n\n\n**W5: The outputs are only 8 frames**\n\nThanks for this good point. To validate the scalability of our MV-VDM, we have trained a 16-frame version and reported the quantitative and qualitative results in global response (1. Comparison methods) and Figure 8 in the attached **PDF**, respectively. We find that 16-frame model generates larger amplitude motion while maintaining similar identity preservation ability as the 8-frame version.\n\n**W6: Faithfulness to the text prompt, similar animations**\n\nFor the concerning about faithfulness of prompts, we clarify that our MV-VDM is capable of producing generations well-aligned with the prompts. \nFirst, our MV-Video dataset is designed to encompass a wide variety of motion prompts, as illustrated in the word cloud presented in the rebuttal **PDF** (Figure 4). \nSecond, for the same object, our model can synthesize various motions according to different prompts as verified in the rebuttal **PDF** (Figure 7).\n\nHowever, we acknowledge that achieving extremely precise motion control through detailed prompts presents certain challenges. \nThis limitation is primarily due to the constraints of the CLIP model, which serves as the text encoder for MV-VDM. It would be interesting future work to further develop more powerful 4D models for superior text alignment.\n\nPlease also refer to our global response (3. Motion Diversity and Dataset) for more discussions about motion diversity.\n\nActually, we have omitted some text prompts to make the video more clear. Most objects in the supplementary video are from our test set and the prompts are listed in Appendix (Figure 9). Text prompts for reconstructed objects  presented in the video are listed in Appendix (Figure 6). Below we list the rest text prompts:\n\n* A glowing blue butterfly is flying.\n* A cute cartoon dog is dancing.\n* A monster dog is walking.\n* A cartoon bear wearing a swimming goggles is getting ready to dive.\n* A cartoon frog is jumpping up and down.\n* A panda with very adorable features is dancing.\n* A cartoon sea lion, adorned with an expressive and charmingly animated face, is singing.\n* A eagle in cartoon style is flapping its wings.\n* A gaint panda in cartoon is walking.\n* A cool spiderman is dancing.\n* A cute lemur is dancing.\n\n**W7: Issues about typos.**\n\nThanks, we appreciate your careful review. We will correct it in the revised paper and check if there remains any typos.\n\n**Q1: In-depth discussion between 3D animation via multi-view video diffusion and two-stage text-to-4D approaches**\n\nPlease refer to W3.\n\n**Q2: empirical comparisons to additional methods (e.g., Dream-in-4D)**\n\nPlease refer to global response (1. Comparison methods)\n\n**Q3: Ablation of MV2V-Adapter**\n\nPlease refer to W4\n\n**Q4: Ablation: Cross-attention layers for image-condition and text prompt**\n\nPlease refer to W4.\n\n**Q5: The use of 4D-SDS for fine-level animation is counter-intuitive**\n\nThe role of 4D-SDS is to alleviate small floaters, similar to smoothing but without the blurry effect. Our coarse reconstruction is \\textbf{sparse-view} reconstruction, i.e., we only have 4 views, and the reconstruction results are inevitably not perfect in novel views, especially when the number of Gaussian points is large. Similar SDS techniques are adopted in sparse-view 3D reconstruction works, such as ReconFusion[CVPR2023] and Sparse3D[CVPR2023]. We also provide detailed qualitative comparisons in the **PDF** attached (Figure 2).\n\n**Q6: Issues about the diversity in the generated motions.**\n\nPlease refer to the global response (3. Motion Diversity and Dataset).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "S3dYCM7oKI", "reviewer": "Reviewer_2Yu3", "summary": "This paper introduces a framework for 4D generation to animate static 3D models, consisting of two components: MV-VDM, a multi-view video generation model, and a framework combining reconstruction and 4D score distillation sampling (4D-SDS). A spatiotemporal attention module enhances consistency, using multi-view renderings to preserve the model's identity. The proposed two-stage pipeline for animating 3D models first reconstructs coarse motions and then refines them with 4D-SDS. Experiments show that Animate3D outperforms previous methods.", "questions": "1. Regarding time, the paper reports an optimization time of 30 minutes per object on a single A800 GPU. How much time does it take to reconstruct coarse motions and refine motions using the 4D-SDS model, respectively?", "limitations": "The authors have discussed the limitations and broader impacts of the study in the paper.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a large-scale dataset for multi-view video generation, consisting of 84,000 animations and over 1.3 million multi-view videos, which addresses the challenge of limited 4D datasets.\n2. Qualitative experiments demonstrate the ability to generate high-quality 4D objects. The generated shapes appear to have better quality and texture compared to other methods.\n3. The paper is clearly written, with a clear motivation for using a multi-view diffusion model to enhance spatiotemporal consistency.", "weaknesses": "1. The method appears to be a straightforward engineering pipeline, combining several existing techniques (e.g., MVDream and IP-Adapter in the MV-VDM stage, reconstruction + SDS in the animating stage) without significant innovation.\n2. Motion diversity is a concern. The method shifts the core problem to the quality of the multi-view video. The dataset achieves this effect, but the maximum number of animations per ID is only six, which is insufficient for capturing the diverse motions of a single 3D object. This approach lacks generality.\n3. Motion controllability is limited. The results shown in the paper exhibit simple, repetitive motions with small amplitudes. Although text prompts are used as conditions, they are relatively simple, and the generated motions are not complex. The method lacks a clear approach to achieving more controllable motions.\n4. Experimental results are lacking. DreamGaussian4D provides CLIP-I as a quantitative comparison to assess the consistency between prompts and generated results, which is missing in the paper. Additionally, the qualitative results provided in terms of views and moments are too few.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**W1: Pipeline without significant innovation.**\n\nThanks. Our work enjoys good novelty in both task formulation and solution pipeline.\nFirstly, we redefine the concept of 4D generation by introducing a novel task: animating any off-the-shelf 3D objects. This innovative task holds significant relevance in fields with limited exploration (Lines 30-41).\nGiven the rapid advancements in 3D generation, our proposed task is both practical and meaningful to directly drive these high-quality 3D assets.\nAlthough some SDS-based methods have been investigated in previous 4D works, the community is still in need of a fast, feed-forward model specifically designed to animate 3D objects.\nFurthermore, such a model has the potential to facilitate mesh generation with PBR materials, which could enhance commercial applications in software platforms like Rodin-Gen1 and Meshy. This positions our work as a crucial step forward in the evolution of 4D generation technology.\n\nAs for the novelty of pipeline, the main goal of the proposed pipeline is to **learn accurate motions of static Gaussians**. Note that unlike previous text-video-4D works, e.g., DG4D, 4DGen, STAG4D, we don't update any gaussian points and don't include gaussian densification/pruning. This is because **only in this way, we could learn the accurate trajectory of each points in static gaussian and when initializing static gaussian by the vertices of the mesh, we can achieve mesh animation, as shown in PDF attached (Figure 6).** Once we could animate mesh, 4D generation could benefit from high-quality 3D assets generated by commercial tools.\n\n**W2: Concern about the motion diversity.**\n\nPlease refer to global response (3. Motion Diversity and Dataset).\n\n**W3: Concern about motion controllability.**\n\nAdmittedly, text prompt could not achieve precise control. We'll include more controls such as monocular video in the future.\n\n**W4: Lack of experimental result CLIP-I; provide more views and moments for qualitative comparison.**\n\nPlease refer to global response (1. Comparison methods) for CLIP-I metric. Please refer to the attached **PDF** for more visualization results.\n\n**Q1: Time cost of reconstruction and 4D-SDS.**\n\nFor 8-frame version model, the Reconstruction and 4D-SDS both costs around 15 minutes, totaling 30 minutes (Line 242). For 16-frame, they both takes around 20 minutes.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank for the reviewers' appreciation of our work, as they give positive comments of "problem setting is interesting, well-motivated and straightforward" (R2, R4), "achieve state-of-the-art performance of 4D generation" (R3, R5), "the large-scale 4D dataset could have significant influence on this area" (R1, R2, R3, R4, R5), "paper is well-written" (R1, R3, R5). Below we respond to some common concerns raised by the reviewers. 

**Many reviewers ask for more visualizations of our methods beyond the demo video we provided. Due to the anonymity policy, we cannot share the link to our project page with more than 100 $1024\times1024$ resolution animation videos, so we provide some high-quality images in the attached PDF**. Additionally, we validate **the Gaussian trajectory learned by our model is quite accurate** and even could be used to **directly animate the MESH**, obtaining **animated mesh** that could be used in standard 3D rendering pipelines. (See Figure 6 in the PDF attached. Text prompts there are "A dragon head is roaring", "A cute dog is walking" and "A cute rabbit is dancing" from left to right.)

**1. Comparison methods**: Since we propose **a new task** of animating any off-the-shelf 3D object and there are **no previous methods** specially designed for this task, we compared our work with two state-of-the art 4D generation methods of different categories (4Dfy (4DGS) and DG4D). Although we think our experiments in the paper are solid enough to support our claims, we further provide the additional comparisons with all the methods requested by the reviewers except for AYG, which is not open-released and hard to be reproduced during the limited rebuttal period. We use official implementations of all comparison methods, and load our pre-trained static NeRF/3D Gaussians. Note that NeRF-based methods usually require training time from hours to several dozens of hours. Additionally, we add **CLIP-I** as the evaluation metric and provide results of **16-frame version of our model** as suggested.

**Quantitative results are as below and qualitative results are depicted in Figure 8 in PDF attached.** (We indicate the best and second best results in bold and italics.)

|       | **I2V $\uparrow$** | **M. Sm. $\uparrow$** | **Dy. Deg.** | **Aest. Q. $\uparrow$** | **CLIP-I $\uparrow$** |
|-------|-------------------|----------------------|--------------|------------------------|------------|
| 4Dfy (4DGS) | 0.783             | **0.996**       | 0.0          | 0.497                   | 0.786      |
| 4Dfy (NeRF)  | 0.817             | 0.990                 | 0.010        | 0.549                   | 0.834      |
| Animate124   | 0.845             | 0.986                 | 0.313        | 0.563                   | 0.845      |
| 4DGen        | 0.833             | *0.994*    | 0.187        | 0.453                   | 0.776      |
| TC4D         | 0.856             | 0.992                 | **0.830** | 0.565                   | 0.859      |
| Dream-in-4D  | 0.938             | *0.994*    | 0.0          | 0.551                   | 0.895      |
| DG4D         | 0.898             | 0.986                 | 0.477        | 0.529                   | 0.860      |
| Ours (8-frame)| *0.982*| 0.991                 | 0.597        | **0.581**         | **0.946**|
| Ours (16-frame)| **0.983**  | 0.991                 | *0.750*| *0.572*      | *0.937*|

As the comparison methods are **not specifically designed for this task**, i.e., they do not take multi-view attributes of the given 3D object into consideration, they perform not well in persevering the identity of the object (indicated by **I2V** and **CLIP-I**). Besides, they also struggle with learning motion with relatively large amplitude since they have to balance between appealing appearance and large motion (indicated by **Dy. Deg.**). **TC4D is a special case as it takes pre-defined object trajectory as the global motion**. 

Our methods are superior to comparison ones in terms of both appearance and motion. **Specially, we find our 16-frame version model can generate motion with larger amplitude while having similar appearance preservation ability as our 8-frame version model. Please refer to PDF attached for a better understanding.**

**2. Ablation of MV2V-Adapter**: We ablate the MV2V-Adapter as follows. MV2V-Adapter improves almost all metrics. The decrease in **Dy. Deg.** is because w/o MV2V-Adapter the generated results are noisy and the motion is incoherent.  Qualitative comparison is in Figure 5 in PDF attached (Text prompt: "A flame rock monster is launching an attack".)


| Method | **I2V $\uparrow$** | **M. Sm. $\uparrow$** | **Dy. Deg.** | **Aest. Q. $\uparrow$** |
|---|---|---|---|---|
| w/o MV2V-adapter | 0.927 | 0.986 | 0.961 | 0.526 |
| Ours | **0.935** | **0.988** | 0.710 | **0.532** |


**3.  Motion Diversity and Dataset**: We present more generated results in **Figure 7 in PDF attached**, demonstrating that the model trained on our dataset can perform various diverse animations on the same 3D objects. This proves the generalization capability of our dataset. Admittedly, our MV-Video dataset contains only 38K animated objects with 84K animations, which is still smaller and limited compared to 2D video datasets. 

However, it is worth noting that our multi-view video (4D) dataset is more rare and challenging to obtain compared to web-scale 2D video datasets, especially with regard to the multi-view camera parameters. Our dataset is much larger than previous 4D datasets. Large-scale 4D dataset is crucial for learning multi-view spatiotemporal consistent 4D foundation model, which cannot be achieved with 2D video datasets.
 
We've created a 4D dataset and confirmed its effectiveness. We are expanding the dataset and plan to release it, and believe it will grow  with community support, driving advancements in the 4D domain. We hope the reviewers will consider this.

Meta-review: Pre-rebuttal, reviewers praised the overall relevance and novelty of the tackled problem, and the intuitive solution proposed to solve it. Recurring concerns included insufficient evaluation against too few baselines, missing ablations, concerns about the diversity of the generated motions, and incomprehensibility of the method section.

Authors provided a strong rebuttal. All reviewers unanimously agreed to accept the paper conditioned on:
1. Authors include additional comparisons and ablations in their final version.
2. Significantly improve the writing quality in the method section.

The AC thus accepts the paper conditioned on the fact that authors fix both 1. and 2. above in the final version of their paper.
Decision: Accept (poster)