Title: Animate3D: Animating Any 3D Model with Multi-view Video Diffusion
Venue: NeurIPS 2024
Accepted: yes
Keywords: 4D generation

Multi-view video diffusion model
Abstract: Recent advances in 4D generation mainly focus on generating 4D content by distilling pre-trained text or single-view image conditioned models. It is inconvenient for them to take advantage of various off-the-shelf 3D assets with multi-view attributes, and their results suffer from spatiotemporal inconsistency owing to the inherent ambiguity in the supervision signals. In this work, we present Animate3D, a novel framework for animating any static 3D model. The core idea is two-fold: 1) We propose a novel multi-view video diffusion model (MV-VDM) conditioned on multi-view renderings of the static 3D object, which is trained on our presented large-scale multi-view video dataset (MV-Video). 2) Based on MV-VDM, we introduce a framework combining reconstruction and 4D Score Distillation Sampling (4D-SDS) to leverage the multi-view video diffusion priors for animating 3D objects. Specifically, for MV-VDM, we design a new spatiotemporal attention module to enhance spatial and temporal consistency by integrating 3D and video diffusion models. Additionally, we leverage the static 3D model’s multi-view renderings as conditions to preserve its identity. For animating 3D models, an effective two-stage pipeline is proposed: we first reconstruct coarse motions directly from generated multi-view videos, followed by the introduced 4D-SDS to model fine-level motions. Benefiting from accurate motion learning, we could achieve straightforward mesh animation. Qualitative and quantitative experiments demonstrate that Animate3D significantly outperforms previous approaches. Data, code, and models are open-released.

Animate3D: Animating Any 3D Model with Multi-view Video Diffusion

Abstract

Recent advances in 4D generation mainly focus on generating 4D content by distilling pre-trained text or single-view image-conditioned models. It is inconvenient for them to take advantage of various off-the-shelf 3D assets with multi-view attributes, and their results suffer from spatiotemporal inconsistency owing to the inherent ambiguity in the supervision signals. In this work, we present Animate3D, a novel framework for animating any static 3D model. The core idea is two-fold: 1) We propose a novel multi-view video diffusion model (MV-VDM) conditioned on multi-view renderings of the static 3D object, which is trained on our presented large-scale multi-view video dataset (MV-Video). 2) Based on MV-VDM, we introduce a framework combining reconstruction and 4D Score Distillation Sampling (4D-SDS) to leverage the multi-view video diffusion priors for animating

3D objects. Specifically, for MV-VDM, we design a new spatiotemporal attention module to enhance spatial and temporal consistency by integrating 3D and video diffusion models. Additionally, we leverage the static 3D model's multi-view renderings as conditions to preserve its identity. For animating 3D models, an effective two-stage pipeline is proposed: we first reconstruct motions directly from generated multi-view videos, followed by the introduced 4D-SDS to refine both appearance and motion. Benefiting from accurate motion learning, we could achieve straightforward mesh animation. Qualitative and quantitative experiments demonstrate that Animate3D significantly outperforms previous approaches. Data, code, and models are open-released.

1 Introduction

3D content creation has garnered significant attention due to its wide applicability in AR/VR, gaming, and movie industry. With the development of diffusion models [45, 44, 39, 20, 51, 6, 19] and large-scale 3D object datasets [13, 12, 37, 65, 70, 14], recent 3D foundational generations have seen extensive exploration through fine-tuned text-to-image (T2I) diffusion models [31, 40, 41, 52, 32, 49], as well as training large reconstruction models from scratch [22, 61, 47, 67, 54], leading the 3D assets creation to a new era. Despite the significant progress in static 3D representation, this momentum has not been paralleled in the realm of dynamic 3D content generation, also known as 4D generation.

4D generation is more challenging due to the difficulty of simultaneously maintaining spatiotemporal consistency in visual appearance and dynamic motion. In this paper, we mainly focus on two challenges: 1) No foundational 4D generation models to unify both spatial and temporal consistency. Though recent 4D generation works [43, 24, 5, 69, 29, 38, 63, 59, 4, 57, 66, 16] separately distill knowledge from pre-trained T2I and 3D diffusion models [39, 41, 31] and video diffusion models [51, 2, 6] to model multi-view spatial appearance and temporal motions respectively, we clarify that such a detached learning way suffers from inevitable accumulation of appearance degradation as the motion changed, as shown in Fig. 1 (SVD+Zero123). 2) Failing to animate existing 3D assets through multiview conditions. With the development of 3D generations, animating existing high-quality 3D content becomes a common demand. However, previous works about 4D modeling from video [24, 57] or based on generated 3D assets [5, 29, 38, 66] are all based on text [39, 41] or single-view [31, 40] conditioned models, struggling to faithfully preserve multi-view attributes during the 4D generation, such as the back of butterfly in Fig. 1 is ignored by Zero123 [31].

To address these issues, we advocate for an approach better suited for 4D generation, that is, animating any off-the-shelf 3D models with unified spatiotemporal consistent supervision . In this way, it would be convenient to directly take advantage of various fast-developing 3D generation and reconstruction approaches based on a single foundational model, eliminating the accumulation of errors in modeling appearance and motion.

To this end, we propose a novel 4D generation framework called Animate3D in this paper, which can be divided into a foundational 4D generation model and a joint 4D Gaussian Splatting (4DGS) optimization. Formally, the foundational 4D model is a Multi-View Video Diffusion Model (MVVDM) built upon the 3D generation model, MVDream [41], which can synchronously synthesize multi-view images with various temporal motions. Specifically, to better inherit the prior in previous 3D and video diffusion models trained on large-scale data, we propose a learnable plug-and-play spatiotemporal attention module, building upon the motion module in video diffusion [18, 6, 20] to expand the attention learning from the temporal domain to the spatial and temporal domain. Moreover, MV-VDM also includes the ability to refer to multi-view images, sufficiently preserving the identity and details of off-the-shelf 3D assets. Specifically, given multi-view images rendered from existing 3D assets or collected from real-world objects, we expand adaptive image-to-video work, I2V-Adapter [17], to multi-view version, called MV2V-Adapter, incorporating multi-view conditions to 4D learning through additionally spatial features and text embeddings. Enhanced by the multi-view appearance injection, we can disentangle the appearance learning from the motion learning, ensuring MV-VDM focuses on learning natural and coherent dynamic motions.

To further enable impressive animations from 3D objects that can be observed at any viewpoint and time, we jointly optimize the 4DGS [55] through both reconstruction and 4D Score Distillation Sampling (4D-SDS) losses based on our unified MV-VDM. Benefiting from the spatiotemporal

consistent multi-view video generations, 4DGS can be roughly converged to proper results with only reconstruction loss, while 4D-SDS further improves the details and fine-grained motions. The Gaussian trajectory learned by our framework is surprisingly accurate and could be used to directly animate the mesh.

The main dilemma in building a foundational 4D generation model lies in the rarity of large-scale 4D datasets, which is non-trivial to collect but the key factor to drive our MV-VDM. In this work, we make the first attempt to build a large-scale multi-view video (4D) dataset, dubbed MV-Video. Specifically, MV-Video comprises about 84K animations that are available under a public license, consisting of about 38K animated 3D objects at all, which are rendered into over 1.3M multi-view videos with minigpt4-video [3] generated prompts, to serve as the training dataset for our 4D foundation model.

We highlight the contribution of this paper as follows: 1) Animate3D is the first 4D generation framework to animate any 3D objects with detailed multi-view conditions. The framework is further extended to achieve mesh animation without skeleton rigging. 2) We propose the foundational 4D generation model, MV-VDM, to jointly model spatiotemporal consistency. 3) We present the largest 4D datasets MV-Video collected with about 84K animations and over 1.3M multi-view videos. Extensive experiments demonstrate that our data-driven approach can generate spatiotemporal consistent 4D objects, significantly outperforming previous counterparts.

2 Related Work

3D Generation. Early 3D generation works optimized single 3D object with CLIP loss [36] or Score Distillation Sampling (SDS) [33] from 2D text-to-image (T2I) diffusion models. Since the models providing supervision lacked 3D prior, those works usually suffered from spatial inconsistency, i.e. , multi-face Janus problem [50, 28, 53]. To tackle this problem, on the one hand, some works [31, 40, 41, 32, 35] lifted the T2I diffusion to multi-view image diffusion by injecting new spatial attention layers and fine-tuning on large-scale 3D synthetic datasets [13, 12]. Although 3D consistency was improved, these optimization-based methods still required a relatively long time to optimize a 3D object. On the other hand, some feed-forward 3D generation foundation models [22, 61, 47, 67, 54, 60, 30], also trained on large-scale 3D datasets, were able to produce a good-quality 3D object in several seconds in an inference-only way. Inspired by the success of the data-driven approaches in 3D generation, we aim to construct a large-scale 4D generation dataset and take the pioneering step towards developing foundation models for 4D generation.

Video Generation. Video generation works started with text-to-video (T2V) generation [20, 42, 2, 18, 6, 19, 7, 10], subsequently followed by image-to-video (I2V) approaches [64, 17, 58, 6]. Previous T2V works usually built upon T2I diffusion models [20, 42, 19, 18, 21], leveraging their pre-trained weights by leaving the spatial blocks unchanged and inserting new temporal blocks to model the temporal camera or object motions. The I2V works [58, 17, 64], building upon the aforementioned T2V methods, typically incorporate image semantics into video models. This is achieved through cross-attention mechanisms between noisy frames and the conditional image, while retaining the motion module design from the T2V models unaltered. We draw inspiration from the development paradigm of video generation to design our 4D generation foundation model, which is a multi-view image conditioned multi-view video diffusion model building upon pre-trained multi-view 3D and video diffusion models.

4D Generation. The pioneering work of 4D generation is MAV3D [43], which is a text and imageconditioned 4D generation framework. MAV3D first proposed a multi-stage pipeline to optimize the static 3D object generation through the T2I model and subsequently learn motions from the T2V model [42]. Following works [68, 5, 29, 69, 4, 59] adopted a similar pipeline, and they further found that employing T2I [39] and 3D-SDS [41] are crucial for both object generation and motion learning stages. Without them, the quality of the generated object's appearance suffered a remarkable decline, and the motion-learning process was prone to failure. Very recently, Consistent4D [24] proposed a video-to-4D generation task, which used single-view video reconstruction and SDS from Zero123 [31] for motion and appearance learning. This paradigm was adopted by following works [38, 63, 66, 57, 16, 11] and extended to text/image-to-video then video-to-4D generation. All aforementioned works heavily depend on the foundational model for SDS to preserve objects' appearance and attributes. However, existing 3D diffusion models struggle to refer to multi-view

<latexi sh

1_b

64="9

OM5I

2

/df

R

AVc8+u0

>

C

H

j

LS

N

FD

r

p

Z

gk

w

K

B

JEmn

W

o

vX

Q

7

P

y

U

Y

3

T

z

G

q

<latexi sh

1_b

64="9

OM5I

2

/df

R

AVc8+u0

>

C

H

j

LS

N

FD

r

p

Z

gk

w

K

B

JEmn

W

o

vX

Q

7

P

y

U

Y

3

T

z

G

q

<latexi sh

1_b

64="Krj7ORZE

I

z

HS

2

C

k

>A

c

V

L

N

FD

pfU

dug

0

y

m

Y

W

nG

Q

3

P+B

v

8

/

w

5M

J

9

q

T

X

o

<latexi sh

1_b

64="pIn2U

+

QX

P

B

fk

Y

V8

>A

CzH

cj

LS

N

FD

r

Zdug

w

R

7

q

E

M

K

9

J

/

mo

3

v

W

y

T

O

5

G

0

<latexi sh

1_b

64="07

M

L2GEzQA

mXY

>

C

H

cj

S

N

FD

pf

R

ZdO

ygn

W

T

v

J

B

/

+

5

U

8

o

I

<latexi sh

1_b

64="fSD7

T8LZy

F50

vz

C

X

I

>A

H

cjVG

N

E

w

g

K

RY

U

Q

B

k

Wq

O

P

/

J

M

2

3u

r

n

+

d

m

p

o

9

<latexi sh

1_b

64="E

S

+

m

ICWg8

kAR

V

d

PTYF

>

zH

cj

L

N

D2

r

pf

Z

u

w

U

X7

q

n

G

K

0

/

o

3

J

v

9

y

5

B

Q

O

M

<latexi sh

1_b

64="VjSAHz

r

NO3M

F5+IT

Zp

C

>

c

L

D2

f

R

u

m

d

W

0

B

7E

/g

9

P

o

w

U

Xn

8

y

Y

v

G

J

Q

<latexi sh

1_b

64="Q38L

q

mzoGf

+

j5

>A

C

n

VH

SgM

FD

O7

r

RXB

uJ

W

T

U

E

w

K

Z

k

9

I

0d

P

N

y

Y

/

Spatiotemporal Block

Text Encoder

Image Encoder

Multi-view

3D Attention

…

⊕

…

Spatiotemporal Attention

MV2V-Adapter

Spatial

Encoding

Multi-view

3D Attention

Temporal

Encoding

Temporal Motion

Module

(

Pre-trained VDM

)

Cross-Attn.

T=1

T=2

T=

…

f

Motion Module

'A bird is flying'

W

′

Q

W

V

Multi-Head Attn.

W

′

O

W

K

Camera

Cross-Attn.

Timestep

⊕

⊕

⊕

Multi-view 3D Attention

Project In

Project Out

Self-Attn.

×

2

Multi-view Renderings

Gen./Recon.

3DGS

MV-VDM

Coarse Motion

Reconstruction

Fine Motion

4D-SDS

…

…

…

…

f

Prompt:

'A bird is flying'

4DGS

Noise

…

…

…

…

…

…

Time

X

1:

n,

1

X

1:

n,i

Adapter

MV-VDM

Spatiotemporal Attention

X ∈ R(bxnxf)xcxhxw

(nxhxw)xeX ∈R(6

xhxw)

→

+

↑

m- 0

conditions, restricting their broader applications to animate various off-the-shelf 3D assets without losing their multi-view attributes.

Furthermore, it is worth noting that existing 4D generation methods suffer from another issue, i.e. , spatial and temporal inconsistency [43, 5, 29, 24, 38]. Because the diffusion models used for SDS were never trained with multi-view video (4D) datasets, missing the critical capacity to formulate spatial and temporal consistency simultaneously. Thus previous methods failed to properly trade off a good balance between appearance and motion learning. Please refer to Sec. B in appendix for detailed discussion and comparison.

In this work, we resort to disentangle 3D object generation/reconstruction and motion learning through a foundational 4D generation model, and propose a novel framework to animate any static 3D object with consistent multi-view attributes.

3 Method

Given a static 3D model, our goal is to animate it with a text prompt and use its multi-view renderings as image condition. This 4D generation task is particularly challenging as it requires ensuring spatial and temporal consistency of the appearance and motion, compatibility with the prompt, and preserving the identity of the static object. To address these challenges more fundamentally, we propose a novel framework, Animate3D, to animate any static 3D object. As depicted in Fig. 2, we divide the task into two parts: learning a multi-view video diffusion model (MV-VDM), and animating 3D object with multi-view videos generated by MV-VDM.

3.1 Multi-view Video Diffusion Model (MV-VDM)

We propose a novel multi-view image conditioned multi-view video diffusion model, named MVVDM. To inherit prior knowledge acquired by spatially consistent 3D models and temporally consistent video models trained on large-scale datasets, we advocate a baseline architecture by integrating them to utilize their pre-trained weights. In this work, we take MVDream [41] and AnimateDiff [18] for the 3D and the video diffusion model, respectively. To enhance the spatiotemporal consistency and ensure compatibility with the prompts and the object's multi-view images, we propose an efficient plug-and-play spatiotemporal attention module combined with an image-conditioning approach. Our

MV-VDM is trained on our presented large-scale multi-view video dataset, MV-Video, which is introduced in Sec. 4.

Spatiotemporal Attention Module. As illustrated in Fig. 2, we insert a novel spatiotemporal attention module after cross-attention layers. The proposed spatiotemporal attention module comprises two parallel branches: the left branch is for spatial attention, and the right branch is for temporal attention. For spatial attention, we adopt the same architecture as the multi-view 3D attention in MVDream [41]. Specifically, the original 2D self-attention layer is converted into 3D by connecting n different views. In addition, we incorporate 2D spatial encoding, specifically sinusoidal encoding, into the latent features to enhance spatial consistency. As for temporal attention, we keep all designs of the temporal motion module from the video diffusion model [18] unchanged in order to reuse their pre-trained weights. Based on the features of these two branches, we employ an alpha blender layer with a learnable weight to achieve features with enhanced spatiotemporal consistency. It is worth noting that we do not apply spatiotemporal attention across all views and frames due to the prohibitive GPU memory requirements that render training infeasible. Instead, our parallel-branch design offers an efficient and practical alternative. Specifically, we first reshape the input feature of spatiotemporal attention module X ∈ R ( b × n × f ) × c × h × w into two forms, X l ∈ R ( b × f ) × ( n × h × w ) × c for spatial branch and X r ∈ R ( b × n × h × w ) × f × c for temporal branch. The spatiotemporal attention is then computed as:

where µ denotes the learnable weight, W s/t Q , W s/t K , W s/t V , W s/t O represent the corresponding projection matrices. b, n, f, h, w, c are the batch size, views, frames, height, width, and channels of the image features, respectively.

Multi-view Images Conditioning. Inspired by I2V-Adapter [17], we add a new attention layer, termed MV2V-Adapter, parallel to the existing frozen multi-view 3D self-attention layer within the proposed spatiotemporal block, as shown in Fig. 2. Concretely, noisy frames are first concatenated along the spatial dimension. These are then used to query the rich contextual information from the multi-view conditional frames, which are extracted using the frozen 3D diffusion model. Next, we add the output of the MV2V-Adapter layer to that of the original multi-view 3D attention layer of MVDream. Thus, for each frame i ∈ { 1 , ..., f } , denoting the multi-view input, output, and conditional frames' features as X 1: n,i , X 1: n,i out , and X 1: n, 1 , we have:

where W Q , W K , W V and W O are projection matrices in original self-attention layer, while W Q ′ and W O ′ are those in the newly added layer. We find this simple cross-attention operator can effectively improve the object's appearance consistency in the generated video. After that, as shown in the spatiotemporal block in Fig. 2, we employ two cross-attention layers to align the text prompt and preserve the object's identity, respectively. The left one is inherited from MVDream, while the right one is pre-trained in IP-Adapter [62].

Training Objectives. The training process of our multi-view video diffusion model is similar to Latent Diffusion Model [39]. Specifically, the sampled multi-view video data q 1: n, 1: f 0 are first encoded into latent feature z 1: n, 1: f 0 via encoder E frame by frame and view by view. Then we add noise using the forward diffusion scheduler: z 1: n, 2: f t = √ ¯ α t z 1: n, 2: f 0 + √ 1 -¯ α t ϵ , where α t is a weighted parameter and ϵ is Gaussian noise. Note that, following I2V-Adapter, we keep the first frame, i.e. , the condition multi-view frames clean, and only add noise to the rest of the frames. During training, the proposed MV-VDM takes as input the clean latent code z 1: n, 1 0 , noisy latent code z 1: n, 2: f t , text prompt embedding y , and the camera parameters Σ 1: n , and outputs the noise strength, supervised by L 2 loss. The training objective of our MV-VDM is calculated as:

where θ denotes the diffusion model. It is important that we keep the entire multi-view 3D attention module frozen and only train the MV2V-Adapter layer and the spatiotemporal attention module to conserve GPU memory and accelerate training. Moreover, as the multi-view images of the first frame, z 1: n, 1 0 , serves as the condition images, we calculate the loss only for the latter f -1 frames, i.e. , z 1: n, 2: f 0 .

3.2 Reconstruction and Distillation of 4DGS

Based on our 4D generation foundation model MV-VDM, we propose to animate any off-the-shelf 3D object. For efficiency, we take 3D Gaussian Splatting (3DGS) [25] as the static 3D object representation, and animate it by learning motion fields represented by Hex-planes, as in [55].

4D Motion Fields. As in 4D Gaussian Splatting (4DGS) [55], we represent the motion fields by Hex-planes [15, 8]. Denoting the static 3DGS as G = {X , C , α, r, s } , where X , C , α , r , and s represent the position, color, opacity, rotation, and scale, respectively. The motion module D predicts changes in position, rotation, and scale for each Gaussian point in frame i by interpolating the Hex-planes R . The motion fields computation can be formulated as:

where l equals to the scales in Hex-plane, and interp() denotes interpolating the Gaussian points on the specific plane ζ to obtain corresponding motion features. We have ζ ∈ { ( x, y ) , ( x, z ) , ( y, z ) , ( x, t ) , ( y, t ) , ( z, t ) } . Therefore, Gaussian G ′ at time t is updated as follows:

To better preserve the appearance of static 3D objects, we keep certain attributes, specifically opacity α and color C , unchanged.

Coarse Motion Reconstruction. Based on the spatiotemporal consistent multi-view videos generated by MV-VDM, we first leverage a 4DGS reconstruction stage to directly reconstruct the coarse motions. Specifically, we use a simple but effective L 2 loss as our L rec , which is calculated as:

where C and ̂ C denote the multi-view and multi-frame renderings and the corresponding ground truth. As verified in Fig. 3, this reconstruction stage can already learn high-quality coarse motions by leveraging the generated multi-view videos of MV-VDM.

4D-SDS Optimization. To better model the fine-level motions, we introduce a 4D-SDS optimization stage to distill the knowledge of our multi-view video diffusion model. The 4D-SDS loss L 4D -SDS is a variant of z 0 -reconstruction SDS loss and can be formulated as:

where z and z 0 are latent feature of the rendered image and the estimation of clean latent feature from current noise prediction ϵ θ , respectively, g represents the rendering function. α t and σ t are the signal and noise scale controlled by the noise scheduler, respectively.

Training Objectives. In addition to L rec and L 4D -SDS , we introduce a variant of As-Rigid-AsPossible (ARAP) loss [46] to facilitate the rigid movement learning as well as the maintenance of the high-quality appearance of the static object. The ARAP loss L arap in our work is defined as:

where ˆ R j is estimated from a rigid transformation using Singular Value Decomposition (SVD) according to [46]:

N c j denotes the set of points within a fixed radius of p j , and w j,k = exp( -d jk ¯ d ) where d jk is the distance between center of p j and p k , measuring the impact of p k on p j . This loss encourages the generated dynamic object to be locally rigid, and it enhances the learning with rigid movement.

In summary, the training objectives for animating off-the-shelf 3DGS object is:

where λ 1 , λ 2 , and λ 3 are weighted parameters.

3.3 Extension to Mesh Animation

To directly utilize high-quality mesh generated from commercial 3D generation tools and crafted by human experts, we extent our framework to mesh animation, producing animated mesh compatible with standard 3D rendering pipelines.

We initialize the 3DGS representation of the given object by vertices and triangles of the static mesh. Specifically, the color is determined by vertex color and we average the connected edges for the scale. Opacity and rotation are set to fully visible and zero rotation quaternion, respectively. The coarse 3DGS is animated following the motion reconstruction steps as described in the above sections. We utilize the per-vertex Gaussian trajectory to deform the static mesh in a straightforward way without skeleton rigging, control point selection or complicated deformation algorithms. As shown in Fig. 6 and our project page, the results are surprisingly good despite the simplicity of the solution.

4 Experiment

4.1 Setup

Training Dataset. To train our MV-VDM, we build a large-scale multi-view video dataset, MV-Video. Concretely, we render multi-view videos of 37,857 animated 3D models collected from Sketchfab [1]. Each model has 2.2 animations on average, resulting in 83,716 animations in total. Each animation is 2 seconds long at 24 fps. Note that animated models that are not allowed to be used to generate AI programs are filtered . The statistical information of our MV-Video dataset is reported in Table 1. For more details about the rendering settings and data examples, please refer to our Appendix ( D). We will release this dataset to further advance the field of 4D generative research.

Implementation Details. We sample 8 frames evenly for each animation to train our MV-VDM. We use the Adawm optimizer with a learning rate of 4 e -4 and a weight decay 0 . 01 , and train the model for 20 epochs with a batch size of 2048. When inference, we set the sampling step to 25 and adopt freeinit [56] to get stable results when animating 3D objects. As for 4D generation, the resolution and feature dimension of the Hex-planes are set to [100 , 100 , 8] and 16, respectively. We perform coarse motion reconstruction for the first 1000 iterations with a batch size of 32 (4 views, 8 frames), and then add 4D-SDS optimization for another 400 iterations. Learning rate is 0.0015 initially and decreases linearly to 0.0005 at the end of reconstruction stage. λ 1 , λ 2 and λ 3 in Eq. 11 are set to 10 . 0 , 0 . 01 and 0 . 5 , respectively. It costs 3 days to train our MV-VDM on 32 80G A800 GPUs, and the optimization for 4D generation takes around 30 minutes on a single A800 GPU per object.

Evaluation Dataset. For the evaluation of MV-VDM, we render multi-view images from 128 static 3D objects and then generate multi-view videos conditioned on them. Following the evaluation setting of VBench [23], we use four different random seeds for each object and report the average results. For the evaluation of 4D generation, we generate 25 objects across various categories using the large 3DGS reconstruction model GRM [61]. Input images for GRM and animation prompts used in this work are provided in our Appendix ( E.1).

Evlaution Metrics. We adopt the evaluation protocol proposed in VBench [23], which is a popular video generation benchmark consisting of both T2V and I2V evaluation tools. The I2V evaluation protocol contains 9 evaluation metrics, and we choose 4 for our evaluation, i.e. , I2V Subject , Motion Smoothness , Dynamic Degree , and Aesthetic Quality , measuring the consistency with the given image, the motion smoothness, the motion degree, and the appearance quality, respectively. We abbreviate them as I2V , M. Sm. , Dy. Deg. and Aest. Q. in the experiment section. Values of all metrics are the higher, the better, except for Dynamic Degree , since we observe that sometimes completely failed results present extremely high dynamic degree. For more details about the introduction and calculation of the evaluation metrics, please refer to our Appendix ( E.2).

Comparison Methods. We compare our work with 4Dfy [5] and DreamGaussian4D (DG4D) [38] on the task of animating any given 3D object using their official implementations. They represent the state-of-the-art in 4D generation methods, starting by generating a static 3D object using 3D-SDS in

DG4D

4Dfy(Gau.)

Ours

Input

t = t

1

t = t

2

t = t

1

t = t

2

t = t

1

t = t

2

t = 0

Reconstruction (Ours)

t = t

1

t = t

2

'A cartoon penguin walks awkwardly.'

the initial stage, and subsequently animating it via video SDS and single-view video reconstruction in later stages. For 4D representation, 4Dfy and DG4D adopt dynamic NeRF [34, 26] and 4DGS [55], respectively. For a fair comparison, we replace the dynamic NeRF in 4Dfy with 4DGS used in both our work and DG4D, and also apply ARAP loss for motion regularization. For DG4D, we keep the 4DGS representation and motion regularizations in their work unchanged.

4.2 Comparison with State-of-the-Art

In this section, we perform comprehensive comparisons with previous works, including quantitative and qualitative comparisons and user studies.

(a) Comparison on video generation metrics.

(b) Comparison via user study.

Quantitative Comparison. As shown in Tab. 2a, our method significantly outperforms 4Dfy and DG4D in terms of I2V , Dy. Deg. , and Aest. Q. . This indicates our generation results have good alignment with the given static 3D object ( I2V Subject ), dynamic motion ( Dynamic Degree ), and superior appearance ( Aesthetic Quality ). For Motion Smoothness , we slightly lag behind 4Dfy, since 4Dfy always generates nearly static results, as illustrated by the 0.0 Dynamic Degree in the first row of Tab 2a. Generally, our method is able to animate 3D object with smooth and dynamic motion, at almost no cost of sacrificing their high-quality appearance, facilitating customized and high-quality dynamic 3D object creation.

Qualitative Comparison. As shown in Fig. 3, it is obvious that 4Dfy's results are blurred and deviate much from the given 3D object, owing to the use of text-conditioned diffusion models to optimize the motion and appearance. Additionally, its generated objects are almost static. This is because at the beginning of the training process, the noisy rendered image sequence, i.e. , the input to the T2V model, has no temporal changes, which misleads the video diffusion model to generate almost static supervisions, as illustrated in Fig 1. For DG4D, its results align well with the given 3D object in front

Ours

Ours w/o S.T. Attn

Ours w/o Pre-train

Input

t = t

1

t = t

2

t = 0

t = t

1

t = t

2

t = t

1

t = t

2

'

A basketball player is playing basketball.'

Ours

Input

t = t

1

t = t

2

t = t

1

t = t

2

t = t

1

t = t

2

t = 0

Ours w/o SDS loss

Ours w/o ARAP loss

'

Cow in cartoon style lifts its front hooves.

'

Figure 5: Ablation for 3D object animation.

view, i.e. , the view used for generating guided video. However, it doesn't align with the object in the novel view, as indicated by the tail in Bear and Penguin , distorted goggles in bear , and blurred back and side views in Fig 3. This is because it adopts Zero123 to optimize the novel views. Zero123 only conditions on the front view, leading to NVS optimization favoring pre-trained data distributions, which could lead to potential appearance degradation. More importantly, DG4D fails when the object in the guided video is assigned with movements toward the camera. For example, the frog is moving forth and back in the guided video, however, DG4D interprets it as enlargement and reduction of the object. The same thing comes to the penguin which nods towards the camera and leans forward. This misinterpretation usually results in blurry effect and strange appearance.

In contrast, our method, leveraging the spatiotemporal consistent muti-view prior, manages to deal with motion towards the camera, as demonstrated by the bear's raised front paw (our model takes the front view and its orthogonal views as the condition view, not depicted in the image). Besides, we successfully maintain the high quality appearance of the given 3D object when generating natural motion. Please refer to the videos in our supplementary material for a more intuitive comparison.

User Study. We conduct a user study among 20 people on the 25 dynamic objects and report the averaged results in Tab. 2b. The participants are asked to score the generated dynamic objects from 1 to 5, according to the alignment with the given text ( Align. Text ) and static object ( Align. 3D ), motion quality ( Mot. ), and appearance quality ( Appr. ). The user study proves the superiority of our method. Please refer to the Appendix ( E.3) for more details.

4.3 Ablation

Multi-view Video Diffusion. In Tab 3a, we validate the effectiveness of the proposed SpatioTemporal Attention (short as S.T. Att.) and the pre-trained weight from video diffusion model (short as Pretrain). We replace the proposed spatiotemporal block with temporal block from AnimateDiff, and

'

A cute dog is running and jumping.

'

this leads to performance drop in I2V Subject , Motion Smoothness and Aesthetic Quality . Dynamic Degree seems to be enhanced, but that is caused by the increase of unstable failure cases. The same tendency could be observed in experiments w/o pre-trained video diffusion weight. Therefore, we think both designs are necessary for generating multi-view videos consistent with the given multi-view images and with high-quality appearance and motion. Qualitative ablations in Fig 4 further demonstrate this point.

4D Object Optimization. The ablations for 4D object optimization are shown in Tab. 3b and Fig. 5. The quantitative results in Tab. 3b indicate both SDS and ARAP losses improve the alignment with the 3D object ( I2V Subject ), Motion Smoothness , and Aesthetic Quality . However, the Dynamic Degree decreases. We suppose the decrease in dynamic degree is mainly caused by the removal of floaters and blurry effects, which are also taken into account of dynamic degree, as shown in Fig. 5. The two losses might slightly decrease the motion amplitude, but generally, we think the overall performance is improved when applying them.

4.4 Mesh Animation

We provide mesh animation results in Fig. 6. Static meshes are generated by commercial 3D generation tools. For more results, please visit our project page.

5 Conclusion

In this work, we present Animate3D, a novel framework for animating any off-the-self 3D object. Animate3D disentangles the 4D object generation into a foundational 4D generation model, MVVDM, and a joint 4DGS optimization pipeline based on MV-VDM. MV-VDM is the first 4D foundation model, which can generate spatiotemporal consistent multi-view videos conditioned on multi-view renderings of a static 3D object. To train MV-VDM, we present the largest multi-view video (4D) dataset, MV-Video, containing about 84K animations with over 1.3M multi-view videos. Based on MV-VDM, we propose an effective pipeline to animate any static 3D object by jointly optimizing the 4DGS via both reconstruction and 4D-SDS. Animate3D is a highly practical solution for downstream 4D applications since it can animate any generated or reconstructed 3D objects. Data, codes, and pre-trained weights will be released to facilitate the research in 4D generation.

References

Sketchfab. https://sketchfab.com/ . Accessed: 2024-5-21.

Zeroscope text-to-video model. https://huggingface.co/cerspense/zeroscope_v2_576w . Accessed: 2023-10-31.

Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny. Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens. arXiv preprint arXiv:2404.03413 , 2024.

Sherwin Bahmani, Xian Liu, Yifan Wang, Ivan Skorokhodov, Victor Rong, Ziwei Liu, Xihui Liu, Jeong Joon Park, Sergey Tulyakov, Gordon Wetzstein, et al. Tc4d: Trajectory-conditioned text-to-4d generation. arXiv preprint arXiv:2403.17920 , 2024.

Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B. Lindell. 4d-fy: Text-to-4d generation using hybrid score distillation sampling. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2024.

Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 , 2023.

Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2256322575, 2023.

Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 130-141, 2023.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafter1: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19512 , 2023.

Wen-Hsuan Chu, Lei Ke, and Katerina Fragkiadaki. Dreamscene4d: Dynamic multi-object scene generation from monocular videos. arXiv preprint arXiv:2405.02280 , 2024.

Matt Deitke, Ruoshi Liu, Matthew Wallingford, Huong Ngo, Oscar Michel, Aditya Kusupati, Alan Fan, Christian Laforte, Vikram Voleti, Samir Yitzhak Gadre, et al. Objaverse-xl: A universe of 10m+ 3d objects. Advances in Neural Information Processing Systems , 36, 2024.

Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1314213153, 2023.

Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA) , pages 2553-2560. IEEE, 2022.

Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12479-12488, 2023.

Quankai Gao, Qiangeng Xu, Zhe Cao, Ben Mildenhall, Wenchao Ma, Le Chen, Danhang Tang, and Ulrich Neumann. Gaussianflow: Splatting gaussian dynamics for 4d content creation. arXiv preprint arXiv:2403.12365 , 2024.

Xun Guo, Mingwu Zheng, Liang Hou, Yuan Gao, Yufan Deng, Chongyang Ma, Weiming Hu, Zhengjun Zha, Haibin Huang, Pengfei Wan, et al. I2v-adapter: A general image-to-video adapter for video diffusion models. arXiv preprint arXiv:2312.16693 , 2023.

Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725 , 2023.

Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 , 2022.

Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems , 35:8633-8646, 2022.

Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868 , 2022.

Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint arXiv:2311.04400 , 2023.

Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, Yaohui Wang, Xinyuan Chen, Limin Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. VBench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024.

Yanqin Jiang, Li Zhang, Jin Gao, Weiming Hu, and Yao Yao. Consistent4d: Consistent 360° dynamic object generation from monocular video. In The Twelfth International Conference on Learning Representations , 2024.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics , 42(4):1-14, 2023.

Sixu Li, Chaojian Li, Wenbo Zhu, Boyang Yu, Yang Zhao, Cheng Wan, Haoran You, Huihong Shi, and Yingyan Lin. Instant-3d: Instant neural radiance field training towards on-device ar/vr 3d reconstruction. In Proceedings of the 50th Annual International Symposium on Computer Architecture , pages 1-13, 2023.

Zhen Li, Zuo-Liang Zhu, Ling-Hao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng. Amt: All-pairs multi-field transforms for efficient frame interpolation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 300-309, 2023.

Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, and Karsten Kreis. Align your gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models. arXiv preprint arXiv:2312.13763 , 2023.

Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-23-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems , 36, 2024.

Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9298-9309, 2023.

Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consistent images from a single-view image. arXiv preprint arXiv:2309.03453 , 2023.

Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988 , 2022.

Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10318-10327, 2021.

Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi Zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d. arXiv preprint arXiv:2311.16918 , 2023.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

Jeremy Reizenstein, Roman Shapovalov, Philipp Henzler, Luca Sbordone, Patrick Labatut, and David Novotny. Common objects in 3d: Large-scale learning and evaluation of real-life 3d category reconstruction. In International Conference on Computer Vision , 2021.

Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu. Dreamgaussian4d: Generative 4d gaussian splatting. arXiv preprint arXiv:2312.17142 , 2023.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero123++: a single image to consistent multi-view diffusion base model, 2023.

Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308.16512 , 2023.

Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 , 2022.

Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, Iurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, et al. Text-to-4d dynamic scene generation. arXiv preprint arXiv:2301.11280 , 2023.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning , pages 2256-2265. PMLR, 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Olga Sorkine and Marc Alexa. As-rigid-as-possible surface modeling. In Symposium on Geometry processing , volume 4, pages 109-116. Citeseer, 2007.

Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054 , 2024.

Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16 , pages 402-419. Springer, 2020.

Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from a single image using latent video diffusion. arXiv preprint arXiv:2403.12008 , 2024.

Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12619-12629, 2023.

Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571 , 2023.

Peng Wang and Yichun Shi. Imagedream: Image-prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:2312.02201 , 2023.

Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems , 36, 2024.

Xinyue Wei, Kai Zhang, Sai Bi, Hao Tan, Fujun Luan, Valentin Deschaintre, Kalyan Sunkavalli, Hao Su, and Zexiang Xu. Meshlrm: Large reconstruction model for high-quality mesh. arXiv preprint arXiv:2404.12385 , 2024.

Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528 , 2023.

Tianxing Wu, Chenyang Si, Yuming Jiang, Ziqi Huang, and Ziwei Liu. Freeinit: Bridging initialization gap in video diffusion models. arXiv preprint arXiv:2312.07537 , 2023.

Zijie Wu, Chaohui Yu, Yanqin Jiang, Chenjie Cao, Fan Wang, and Xiang Bai. Sc4d: Sparse-controlled video-to-4d generation and motion transfer. arXiv preprint arXiv:2404.03736 , 2024.

Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Wangbo Yu, Hanyuan Liu, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv preprint arXiv:2310.12190 , 2023.

Dejia Xu, Hanwen Liang, Neel P Bhatt, Hezhen Hu, Hanxue Liang, Konstantinos N Plataniotis, and Zhangyang Wang. Comp4d: Llm-guided compositional 4d scene generation. arXiv preprint arXiv:2403.16993 , 2024.

Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. Instantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruction models. arXiv preprint arXiv:2404.07191 , 2024.

Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang, Sida Peng, Yujun Shen, and Gordon Wetzstein. Grm: Large gaussian reconstruction model for efficient 3d reconstruction and generation. arXiv preprint arXiv:2403.14621 , 2024.

Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721 , 2023.

Yuyang Yin, Dejia Xu, Zhangyang Wang, Yao Zhao, and Yunchao Wei. 4dgen: Grounded 4d content generation with spatial-temporal consistency. arXiv preprint arXiv:2312.17225 , 2023.

Jiwen Yu, Xiaodong Cun, Chenyang Qi, Yong Zhang, Xintao Wang, Ying Shan, and Jian Zhang. Animatezero: Video diffusion models are zero-shot image animators. arXiv preprint arXiv:2312.03793 , 2023.

Xianggang Yu, Mutian Xu, Yidan Zhang, Haolin Liu, Chongjie Ye, Yushuang Wu, Zizheng Yan, Tianyou Liang, Guanying Chen, and Xiaoguang Han. Mvimgnet: A large-scale dataset of multi-view images. In CVPR , 2023.

Yifei Zeng, Yanqin Jiang, Siyu Zhu, Yuanxun Lu, Youtian Lin, Hao Zhu, Weiming Hu, Xun Cao, and Yao Yao. Stag4d: Spatial-temporal anchored generative 4d gaussians. arXiv preprint arXiv:2403.14939 , 2024.

Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-lrm: Large reconstruction model for 3d gaussian splatting. arXiv preprint arXiv:2404.19702 , 2024.

Yuyang Zhao, Zhiwen Yan, Enze Xie, Lanqing Hong, Zhenguo Li, and Gim Hee Lee. Animate124: Animating one image to 4d dynamic scene. arXiv preprint arXiv:2311.14603 , 2023.

Yufeng Zheng, Xueting Li, Koki Nagano, Sifei Liu, Otmar Hilliges, and Shalini De Mello. A unified approach for text-and image-guided 4d scene generation. arXiv preprint arXiv:2311.16854 , 2023.

Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. arXiv preprint arXiv:1805.09817 , 2018.

DreamGaussian4D

4Dfy(Gau.)

Ours

Input

t = t

1

t = t

2

t = t

1

t = t

2

t = t

1

t = t

2

t = 0

'

A spiderman is walking.

'

'A monstrous dog beast attempts to attack.'

'A cool superman is dancing.'

A More Visualizations

We provide more qualitative results as below. Please refer to the videos attached in our supplementary materials for more intuitive visualizations.

Comparison. In the main paper, we only present the animation results for 3D objects generated by large reconstruction models. But our method is able to animate reconstructed objects as well as generate 3D models from text/image/video, and shows a great advantage over other methods as shown in Fig. 7. In Fig. 7, 4Dfy produces results inconsistent with the given 3D object, and they are almost static. Distortions and blurry effects could be observed in results generated by DreamGaussian4D, and it sometimes deviates from the given object in novel views. In contrast, our approach can generate results not only spatially and temporally coherent but also consistent with the input object.

Ablation. We provide more qualitative ablations of MV-VDM and 4D generation in Fig. 8.

B Discussion with respect to Previous 4D Generation Methods

Previous two-stage 4D generation works attempted to disentangle motion learning from appearance learning by adopting different types of supervision signal, i.e., video diffusion/monocular video for

Ours

Ours w/o S.T. Attn

Ours w/o Pre-train

Input

t = t

1

t = t

2

t = 0

t = t

1

t = t

2

t = t

1

t = t

2

A pink dinosaur is waving its little hands'

'

A blue treasure chest is being opened.'

(a) More ablation studies of MV-VDM.

Ours

Input

t = t

1

t = t

2

t = t

1

t = t

2

t = t

1

t = t

2

t = 0

Reconstruction (Ours)

Ours w/o ARAP loss

(b) More ablation studies of 4D generation.

'

motion and image/3D diffusion for appearance. However, the motion and appearance supervisions adopted in their work are not orthogonal, and sometimes have negative effect on each other.

For example, it is commonly agreed that video-diffusion-SDS usually brings unappealing visual effect to the appearance of the object [68, 69, 29]. Meanwhile, the appearance supervision signal prevents 4D object from updating along the direction that follows the exact score function of the video diffusion model, leading to less natural motion. Small motion amplitude in [5, 69] and shaky appearance in [29] could partly support this point. As for monocular-video-guided motion learning, previous work [38, 63, 66] rely on 3D diffusion model (Zero123 [31]) to supervise both motion and appearance in novel view. Since Zero123-SDS is applied per-frame, temporal consistency in novel view cannot be guaranteed. Moreover, monocular video doesn't provides information about depth/distance, so moving closer to or farther away from the camera can be perceived as the magnification or reduction of the object, resulting in appearance distortion.

In contrast, our method takes the unified supervision signal from MV-VDM for motion learning and appearance preservation. Our motion and appearance supervision signal inherently don't conflict with each other, since MV-VDM is conditioned on multi-view attributes of the 3D object to generate multi-view videos. Besides, multi-view motion supervision in our work enables more natural motion generation when compared with single-view motion supervision in other works. Thus, we achieve superior performance in terms of both motion generation and appearance preservation in the task of animating any off-the-shelf 3D object.

To provide a comprehensive understanding of the difference between our method and previous work, we conduct comparisons with more open-sourced previous works using their official implementations, and summarize the results in Tab. 4.

C Limitations

Despite the promising performance in generating spatiotemporal consistent 4D objects, our method still has a few limitations. First, it takes a relatively long time (about 30 minutes) to animate an existing 3D object. Second, there is a trade-off between temporal coherence and motion amplitude in the multi-view videos generated from the proposed MV-VDM. Specifically, the larger the motion amplitude, the higher the risk of temporal incoherence. Third, our model sometimes fails to animate realistic scenes due to the domain gap between synthetic training data and real-world test data. At last, current evaluation metrics in 4D generation are not sufficient, as they mainly rely on video generation metrics and user studies. Designing more suitable metrics for 4D generation will be an important future work.

D More Details of MV-Video Dataset

D.1 Rendering Details.

For the rendering settings, we first centralize the model according to the bounding box of the object in the first frame. Then, we adjust the camera distance to make sure the object stays in the scope of view during the animation process. Sixteen views are evenly sampled in terms of azimuth, starting from

values randomly selected between -11 . 25 ◦ and 11 . 25 ◦ . The elevation angle is randomly sampled within the range of 0 ◦ to 30 ◦ . To stabilize training, we manually filter out objects that we identify as challenging to learn due to factors such as large movements, complex environmental interactions, high speed, or sudden changes in appearance.

D.2 Data Examples.

As shown in Fig. 12 and Fig. 13, we showcase more examples of our multi-view video dataset (MV-Video).

Furthermore, as shown in Fig. 9, we extracted all nouns from the text captions of our MV-Video dataset, which are generated by minigpt4-video [3], and plotted a word cloud of the Top-1000 nouns. We can see that our MV-Video dataset contains diverse categories of animated 3D objects, including humans, characters, animals, plants, mechanical structures, and ect. .

perform

cameras

cocationg

bea chren

hotatue

foodUsemusic

expressionEequipmentplane

car

vehicle

angle

suny!

mstick

scenario

ti

attention

uo

credit

skya

body

品

u

e

S

t

t

qfire

appearance

ougroc

peopl

machine

capabilitiesareawolt

terrain

picture

clip

"behavior

attack

portray

gesture

eature animation

grass

nature,

can

text

history

t0o1

dark

K

e

images

horse

platform

force

type

claw

emotion

size

table

group

top

DO ttom

land

demonstrates

n8

egg

shoe

insectea

blant

base

material

rabbit

hand

art

S

h

at

norh

environments

tchange

title

emphasis

ettank

posing

shoot

shark

jump

lift

weay

set

eye.

appear

trees

control

shoulder

rope

point

toy

fish

talk

七

iduc

movie

piece

100king

boat

second

swimming

(beginu)

S

e

O

e

ups

dot

task

forra

stage

feet

for

V1.

meta.

t

uo

L

e

frame

combat

ew

technique

clothing

close

see

child

ange!

capture

event

C

0

e

zoom

wave

Aarsue

rod

1thing

th

品

puty

army

eg

path

we11

spider

O

windco

dragon

level

box

fanger

wa

give

rstep

target

stop

teeth

scenery

battle

component

it

night

line

1

wa11

n

t

S

repr

walki

ing

flying

0

esentation

eas

work

shirt

s

exercise

ground

visua

structure

series

hair

V1deo:

warr

floor

surface

part

e

sides

aspect

S

S

attire

fighter

ior

bat

Bop

quence

shape

Urole

chest

0

oop

gunin

action

ride

tire

verslonplay

fun

rifle

wing

foot

shot

cat

rotate

ca

situation

weapon'

graphic

way

back

play

rtoon

oad

gear

name

-time s'

sitting

mode

e

circle

alr

aaos

FLS

mouth

gamer

field

hat

girl

1ady

day

cup

speed

element

iewer

Lonfocus

peurr

distance

other

S

effect

dance

form

des,ign

earth

b0

5

stance

stone

space

00m

light

'system

pant

ship

irong

spot

running

E 4D Generation Evaluation

E.1 Evaluation Dataset

In Fig. 10, we provide the input images for image-to-3D generation and corresponding text prompts for 4D animation used in Sec. 4.2.

E.2 Evaluation Metrics

VBench [23] provides six evaluation metrics for I2V evaluation 3 , i.e. , I2V Subject , I2V Background , Camera Motion , Subject Consistency , Background Consistency , Motion Smoothness , Dynamic Degree , Aesthetic Quality and Imaging Quality . Since our generated results have no background, and the evaluation cameras are fixed, metrics related to background and camera motion are not used. The metric Imaging Quality , which is affected by ambient light, is also not used here. Subject Consistency is also omitted since its calculation process is similar to I2V Subject , except for the choice of reference frame. It takes the first generated frame, instead of the input image used in I2V Subject , as the reference frame for similarity score calculation, which is not suitable for our task of animating 3D objects. We briefly introduce the evaluation metrics used in our work as below:

A cartoon bear with

soft, cuddly fur

ambles along, fur

swaying gently.

A cartoon dolphin with

a playful smile and

sparkling eyes leaps

joyfully from the waves.

A cartoon eagle with

grand, outstretched

wings circles is flying in

the sky.

A cartoon elephant with a

colorful, patchwork ski

pattern lumbers steadily,

patterns shifting.

A cartoon frog with a

friendly, wide mouthed

smile is walking.

A cartoon parrot with

bright, multi colored

feathers is stretching its

neck to look around.

A cartoon penguin

walks awkwardly.

A cartoon rabbit with

long, floppy ears hops

along, ears bobbing

merrily.

A cartoon sheep with

fluffy, cloud like wool

hops, wool rippling

gently.

A cute fox, 3D

animation rests in place,

gently nodding its head.

Bear in a cartoon

tutu, looking bashful

swaying.

Bear wearing cartoon

swimming goggles,

ready plunges into the

water, goggles on.

A cartoon elephant with a

colorful, patchwork skin

pattern lumbers steadily,

patterns shifting.

Bison with a 3D cartoon

winter coat shakes off

flakes, coat ruffling.

Car in cartoon form,

colorfully painted

driving along the road.

Cat with a 3D

cartoon pirate eye

patch trying to reach

for something.

Chicken in 3D cartoon,

with superhero cape

struts valiantly, cape

flowing.

Chicken in cartoon style,

donning a tiny crown

struts confidently

around the coop.

Cow in cartoon style

lifts its front hooves.

Dolphin with cartoon

sunglasses, stylish leaps

gracefully from the

waves.

Shadowy ninja fox poised

for action, in dynamic 3D

animation unsheathes a

glinting katana.

Monkey with cartoon

sunglasses, looks

around.

Magical panda in 3D

cartoon style, with a wizard

hat and cloak waving its

hands to cast a spell.

Sheep in cartoon style,

with reading glasses

reads a book.

Space shuttle in cartoon

style, boosters ready, is

flying.

I2V Subject assess whether the appearance of the object in the generated video remains consistent with that in the input image. To this end, DINO [9] feature similarity across frames is calculated.

Motion Smoothness evaluates whether the motion in the generated video is smooth, and follows the physical law of the real world. The motion prior in the video frame interpolation model [27] is utilized for evaluation.

Dynamic Degree employs RAFT [48] to estimate the degree of dynamics in synthesized videos.

Aesthetic Quality is calculated by the LAION aesthetic predictor, which reflects the artistic and beauty value perceived by humans towards each frame.

E.3 User Study Template

As illustrated in Fig. 11, a picture of the user study page is depicted. The survey contains 25 dynamic objects, which are shown in Fig. 10. The participants are asked to score the generated 4D objects from 1 to 5, according to the alignment with the given static object and text prompt, appearance quality, and motion quality.

Please rate the animated results for the static 3D model in conjunction with the video. (The first row fea tures reference images of the 3D model, while the second row displays the generated animation results. The first and second columns represent two different viewpoints, respectively.)

The desired text description for the animation is: " Sheep in cartoon style, with reading glasses, reads a book. '

2.

…

Gif

Score:

1

2

3

4

5

Consistency with the static 3D model:

Consistency with the text prompt:

Aesthetic quality of the results:

Movement quality of the results:

Every three constitute a group, representing the animation generations for the same object u sing three different methods (the order has been shuffled). Please compare the generated res ults within the same group and then provide scores.

0O0OO

000O0

0000O

F Border Impacts

This paper exploited 4D generation based on our proposed multi-view video diffusion model, which can generate spatiotemporal consistent multi-view videos. Because of the advanced generative capacity, our models may output misinformation or fake videos. Thus, we sincerely remind the users to pay attention to it. Note that our method only focuses on the technical aspect. All the code, dataset, and trained models will be released.

##

Sn

n

SA

木

V

不

不

天

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: our abstract clearly clarifies our contributions: 1) We propose Animate3D, the first 4D generation pipeline to animate any 3D objects. 2) We propose the first 4D generation foundation model, MV-VDM. 3) We present the largest 4D datasets, MV-Video.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: we have included our limitation in Sec. C.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: This paper did not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: our experiments are reproducible, and all codes, dataset, and pre-trained models will be open-released.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All data, codes, and pre-trained weights used in our experiments will be open-released.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: we include detailed implementations in Sec. 4.1. The full details can be seen in our codes, which will be open-released.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: As a visual generative model, we provide sufficient visualization results instead.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: we provide the information on the computer resources in Sec. 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: our work conforms to the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: we provide the it in Sec. F.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: our paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: we have cited related works and codes.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: the paper does not release new assets for now. Codes, datasets, and trained models will be open-released soon.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: the paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a framework for 4D generation to animate static 3D models, consisting of two components: MV-VDM, a multi-view video generation model, and a framework combining reconstruction and 4D score distillation sampling (4D-SDS). A spatiotemporal attention module enhances consistency, using multi-view renderings to preserve the model's identity. The proposed two-stage pipeline for animating 3D models first reconstructs coarse motions and then refines them with 4D-SDS. Experiments show that Animate3D outperforms previous methods.\n\nSTRENGTHS:\n1. The paper introduces a large-scale dataset for multi-view video generation, consisting of 84,000 animations and over 1.3 million multi-view videos, which addresses the challenge of limited 4D datasets.\n2. Qualitative experiments demonstrate the ability to generate high-quality 4D objects. The generated shapes appear to have better quality and texture compared to other methods.\n3. The paper is clearly written, with a clear motivation for using a multi-view diffusion model to enhance spatiotemporal consistency.\n\nWEAKNESSES:\n1. The method appears to be a straightforward engineering pipeline, combining several existing techniques (e.g., MVDream and IP-Adapter in the MV-VDM stage, reconstruction + SDS in the animating stage) without significant innovation.\n2. Motion diversity is a concern. The method shifts the core problem to the quality of the multi-view video. The dataset achieves this effect, but the maximum number of animations per ID is only six, which is insufficient for capturing the diverse motions of a single 3D object. This approach lacks generality.\n3. Motion controllability is limited. The results shown in the paper exhibit simple, repetitive motions with small amplitudes. Although text prompts are used as conditions, they are relatively simple, and the generated motions are not complex. The method lacks a clear approach to achieving more controllable motions.\n4. Experimental results are lacking. DreamGaussian4D provides CLIP-I as a quantitative comparison to assess the consistency between prompts and generated results, which is missing in the paper. Additionally, the qualitative results provided in terms of views and moments are too few.\n\nQUESTIONS:\n1. Regarding time, the paper reports an optimization time of 30 minutes per object on a single A800 GPU. How much time does it take to reconstruct coarse motions and refine motions using the 4D-SDS model, respectively?\n\nLIMITATIONS:\nThe authors have discussed the limitations and broader impacts of the study in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Animate3D, a 4D generation framework that consists of a multi-view video diffusion model (MV-VDM) followed by 4D Gaussian Splatting optimization, as well as a dataset of 38K animated 3D objects (MV-Video) that is used to provide spatiotemporal supervision to train the model. Different from text-to-4D approaches, Animate3D specifically tackles the problem of animating 3D assets which requires spatiotemporal consistency as well as identity preservation. The proposed architecture is based on a multi-view diffusion model (MVDream) and a video diffusion model (AnimateDiff) with additional spatio-temporal attention blocks. To animate a given 3D asset, the trained model is used for both reconstruction and distillation. First, a multi-view video is generated conditioned on multiple views of the asset. The video is used to optimize a 4D Gaussian Splatting resulting in a model with coarse motion. The same MV-VDM model is used for 4D Score Distillation Sampling to refine the motion. The method compares favorably to state-of-the-art 4D generation methods.\n\nSTRENGTHS:\n**[S1]** The problem setting in this paper is interesting and timely, in an area that has recently attracted significant attention. The paper takes a slightly different angle compared to text-to-3D, text-to-4D or video-to-4D methods, focusing on animating 3D assets based on a textual description instead of generating them from scratch. \n\n**[S2]** This problem setting also opens up a new challenge, which is to preserve multi-view attributes during the animation. It is nice to see that the identity preservation and the animation are significantly improved compared to state-of-the-art methods.\n\n**[S3]** The authors tackle this problem by training a multi-view video diffusion model. Given the success of image (2D), video (2D+time), and multi-view (3D) diffusion models, MV-VDM (4D) is a reasonable next step to ensure spatial and temporal consistency simultaneously.  Both the model and the dataset used to train it are likely to influence future work and could thus have significant impact. \n\n**[S4]** The proposed architecture aims to reuse existing multi-view and video diffusion models, which are already pre-trained on a larger scale of data, thus taking advantage of the priors built into these models.\n\n---\n\nWEAKNESSES:\n**[W1]** The data used in this paper are animated 3D models collected from Sketchfab (cited). The authors promise to distribute the collected dataset. However, there is not sufficient information provided about copyrights and licensing. The only piece of information provided is that models with the following clause are excluded: \n\n>NoAI: This model may not be used in datasets for, in the development of, or as inputs to generative AI programs.\n\nBased on the paper checklist, the paper is flagged for ethics review.  \n\n---\n\n**[W2]** Lack of clarity in the method section and method figure\n\n**(a)** The method section is very hard to parse. The notation is overloaded but not very well explained. The section would benefit from thorough revision to improve the overall clarity and quality. \n\n**(b)** It is almost easier to understand the method through Fig. 2, but even that is dense and inconsistent, and often the notation does not align with what is written in Sec. 3.1. For example, in L175 latent features $z$ seem to be the output of the image encoder. In Fig. 2 (middle), $z$ appears to be the input to the spatiotemporal attention module. In Eq. 1, $X_l$ and $X_r$ are instead the inputs to the same module. At the same time, in Fig. 2 (left) the input to the spatiotemporal module seems to be the sum of the output of the cross-attention layers, not directly $z$. \n\nThis makes reading and understanding the method extremely difficult to the point where the reviewer cannot fully judge the technical contribution and similarities or differences to related models. \n\n---\n\n**[W3]** Discussion with respect to 4D generation methods could be expanded\n\nExisting work in text-to-4D generation often splits the problem into two stages: a static 3D asset generation and a deformation field to model motion. This disentanglement makes it possible to learn the 3D motion from video diffusion models, without the need for multi-view video. The paper briefly touches upon this and compares to two 4D generation methods, 4Dfy and DreamGaussian4D. However, the paper could further elaborate on the similarities and differences of the proposed approach compared to existing methods and the advantages of a multi-view video diffusion model. \n\n---\n\n**[W4]** Certain architectural components are not ablated and their contribution to the overall performance is unclear. See Questions. \n\n---\n\n**[W5]** The temporal extent of MV-VDM is only 8 frames, which seems rather limited.\n\n---\n\n**[W6]** In some of the qualitative examples, the faithfulness to the text prompt is questionable. Some of the animations provided in the supplementary video are not accompanied by a text prompt, while quite a few animations appear very similar. \n\n---\n\n**[Minor]** Typos and suggestions:\n\n- L103: pioneer → pioneering\n- L113: All these manners aforementioned → All aforementioned work\n- L131-132: this sentence can be improved for clarity since the two parts sound a bit repetitive\n- L135: spatial → spatially; temporal → temporally \n- L156: It should be better explained how what $X_l$ and $X_r$ represent and how these features are obtained.  \n- L292: levering → leveraging; manage → manages\n\n---\n\nQUESTIONS:\n**Q1. [Based on W1]** Please elaborate on the difference between _3D animation via multi-view video diffusion_ and _two-stage text-to-4D approaches_. In the latter case, the second stage could be viewed as 3D animation. I understand that a multi-view video foundation model would offer spatiotemporal consistency that other approaches can likely not reach, but I would appreciate a more in-depth discussion about the difference to and the challenges of text-to-3D-and-3D-to-4D approaches than what is currently provided in the paper. \n\n---\n\n**Q2. [Based on W1]** Ideally, the authors should provide also empirical comparisons to additional methods (e.g., Dream-in-4D), if possible.   \n\n---\n\n**Q3. [Based on W4]** The authors state that\n>We find this simple cross-attention operator [MV2V-Adapter] can effectively improve the object’s appearance consistency in the generated video.\n\nThe effect of MV2V-Adapter should be better demonstrated with an ablation. \n\n---\n\n**Q4. [Based on W4]** Are the cross-attention layers described in L170-172 necessary for identity preservation and alignment with the text prompt? Their effect should be also demonstrated in the ablation studies. \n\n---\n\n**Q5.** The proposed 3D animation approach uses both reconstruction and distillation. It first uses the generated multi-view video to optimize a 4D Gaussian Splatting with coarse motion. Then SDS is used to refine the motion. It seems counter-intuitive that SDS would be good at modeling the finer details of the animation, since SDS is typically known for its over-smoothing behavior. The authors should further discuss the motivation behind using SDS during 4D optimization and provide additional examples to prove its effectiveness. \n\n---\n\n**Q6. [Based on W6]** Could the authors elaborate on the apparent lack of diversity in the generated motions? It may be helpful to provide some statistics from motion descriptions included in the dataset. Or is this a limitation of the model instead?\n\n---\n\nLIMITATIONS:\nLimitations are adequately discussed in Appendix B.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents Animate3D, a framework for animating static 3D models. The core idea involves two main components:\n1. A multi-view video diffusion model (MV-VDM) conditioned on multi-view renderings of the static 3D object, trained on a large-scale multi-view video dataset (MV-Video).\n2. A framework that combines reconstruction and 4D Score Distillation Sampling (4D-SDS) to utilize multi-view video diffusion priors for 3D object animation.\nThe animation process involves a two-stage pipeline: coarse motion reconstruction from generated multi-view videos, followed by 4D-SDS to model fine-level motions. Quantitative and qualitative evaluations show enhancements to previous methods.\n\nSTRENGTHS:\n1. Performance: The proposed method achieves state-of-the-art results. The experiments well validate the effectiveness of the proposed methods.\n\n2. Clarity: The paper is well-written and easy to follow.\n\n3. Technical Novelty: The main contributions of this paper are threefold: 1) The first 4D generation framework to animate any 3D objects with detailed multi-view conditions, which are incorporated by the proposed MV2V-Adapter.  2) The authors propose to animate any off-the-shelf 3D models with unified spatiotemporal consistent supervision, which can get better results. 3) The collected 4D dataset, MV-Video.\n\nWEAKNESSES:\n1. Missing Reference [a]: It is understandable that you did not compare Animate3D with Animate124 in this paper, as 4D-fy and DreamGaussian4D both demonstrate better performance compared to Animate124. However, it is unusual that you did not discuss Animate124 at all, given its relevance in previous comparisons by 4D-fy and DreamGaussian4D.\n\n2. Limited Comparison Scope: The authors only compare Animate3D with 4D-fy and DreamGaussian4D, which seems insufficient. It would be more comprehensive to include comparisons with 4DGen and TC4D, as both claim superior performance over 4D-fy and have released their code. Additionally, while the modification of 4D-fy to a Gaussian representation for fair comparison is understandable, the original 4D-fy results should also be included for a thorough ablation study.\n\n[a] Animate124: Animating One Image to 4D Dynamic Scene, \\\n[b] 4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency, \\\n[c] TC4D: Trajectory-Conditioned Text-to-4D Generation.\n\nQUESTIONS:\n1. The results of 4D-fy appear strange. When I check the 4D-fy results in TC4D and 4DGen, they look more reasonable.\n\n2. I think the authors should also conduct ablation studies on MV2V-Adapter.\n\nLIMITATIONS:\nThe authors have discussed the limitations and potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an animation method that animates a 3D model in a 4D one. A  Multi-View image conditioned multi-view Video Diffusion Model  (MV-VDM)  is presented to generate multi-view videos from multi-view renderings of a static 3D object. The MV-VDM is leveraged to train the 4D Gaussian Splatting (4DGS), where As-Rigid-As-Possible (ARAP) loss, reconstruction loss and SDS are used as objectives. In addition, a multi-view video dataset is constructed to train MV-VDM. The paper conducts experiments to show the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The idea is well-motivated and straightforward.\n2. A large-scale multi-view video dataset is presented.\n3. A multi-view video diffusion model is presented, where spatiotemporal attention is introduced to animate multi-view images with motions.\n\nWEAKNESSES:\n1. The comparison is not very fair. The paper focuses on a new task, i.e., 3D animation, while 4Dfy focuses on  text-to-4D, which is a different task. Furthermore, text-to-4D is more challenging than 3D animation since multi-view images are unavailable.  The better performance gain of the proposed method may partly attributed to additional multi-view images. In other words, the worse performance of 4Dfy and DG4D may not be due to the methods themselves. In addition, the paper replaces dynamic NeRF in 4Dfy with 4DGS. However, some hyperparameters of 4Dfy are set according to dynamic NeRF, rather than 4DGS. Instead, the paper can compare the proposed method with AYG, which directly trains 4DGS. In addition, the paper can compare Animate234, which is an image-to-4D method.\n\n2. The motivation for the proposed spatiotemporal attention block is not clear. Besides the temporal motion module and temporal encoding, the block introduces a multi-view 3D attention module. What is the multi-view 3D attention module used for? Why are temporal motion modules and temporal encoding not enough to generate motions for multi-view videos? \n\n3. What is the influence of Alpha Blender on the performance of the proposed method?\n \n\n4. Does the proposed method train all modules in the spatiotemporal attention block? It's not clear whether the \"Multi-view 3D Attention\" and \"Temporal Motion Module (Pre-trained VDM)\" are trained or frozen.\n\n5. The implementation details of Alpha Blender are not clear. Is it implemented as a layer of MLP? \n\n6. The temporal encoding is unclear. According to Figure 2, there is a temporal encoding module in the spatiotemporal attention block, but there is no description of what kind of temporal encoding is used.  \n\n7. How many views does each animated 3D object contain in the dataset? Only four orthogonal views or more?\n\n8. Although the dataset contains 38K animated 3D objects, it is still much smaller compared to 2D video datasets. Can the proposed method trained on this dataset of this magnitude animate any 3D model?\n\n9. Training a multi-view video diffusion model requires camera parameters. Are the camera parameters in the multi-view video dataset processed to ensure that they are consistent with the prior knowledge learned in the MVDream\n\nQUESTIONS:\nPlease refer to my question above.\n\nLIMITATIONS:\nThe authors have provided  limitations and  societal impact of their work", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on animating 3D objects with multi-view diffusion models. To improve spatial and temporal consistency, this work builds a large-scale multi-view video dataset, MV-Video and designs an attention module to enhance spatial and temporal consistency by integrating 3D and video diffusion models. To enhance animations from 3D objects, this work jointly optimizes the 4DGS through reconstruction and 4D-SDS. Experiments show the proposed method can more consistent 4D generation.\n\nSTRENGTHS:\n* This work builds a large-scale multi-view video dataset to facilitate the training of multi-view video diffusion models.\n* The method designs an attention module to encourage spatial and temporal consistency for multi-view diffusion models. \n* The approach introduces 4D-SDS to leverage multi-view video diffusion models for animating 3D objects.\n* The paper is easy to read and understand.\n\nWEAKNESSES:\n* This work focuses on 3D objects, however, the paper title claims \"Animating Any 3D Models\". It is a bit inappropriate.\n* The effectiveness of the proposed dataset is not fully validated. Can this dataset improve single-view diffusion models or multi-view image-to-3D generation models, like MVDream?\n* The diversity of the proposed method is unclear. Can one object perform different actions? For example, in Figure 3, can the frog perform swimming?\n* The picture quality is a bit low, it is hard for readers to distinguish the advantages of the generated objects from the proposed method. \n* For animating from 3D object, this work first leverages 4DGS to reconstruct coarse motions and then uses 4D-SDS for refinement. If the method uses better 4DGS algorithms, maybe the improvement from 4D-SDS will become little.\n\nQUESTIONS:\n* It is better to validate the effectiveness of the proposed dataset, can it also improve single-view diffusion models (e.g., T2V) or multi-view image-to-3D generation models (e.g., MVDream). Then, can these models further improve other methods, like DreamGaussian4D?\n* Although the work verify the effectiveness of the S.T attention. The effect of MV2V-adapter is not validated.\n* If the approach leverages better 4DGS reconstruction methods, such as [1] for the coarse motion reconstruction, can the 4D-SDS still improve the coarse results?  \n[1] Yang, Zeyu, et al. \"Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting.\" arXiv preprint arXiv:2310.10642 (2023).  \n* As this work designs a multi-view video diffusion model, can the authors compare it with existing single-view video diffusion models?\n* For Figure 4, both the ablation models show that the player touches the basketball. However, the full model does not show this interaction. I wonder if the full model can do better for this interaction.\n\nLIMITATIONS:\nThe paper has discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-22"}