Animate3D: Animating Any 3D Model with Multi-view Video Diffusion

Authors: yanqin  jiang
chaohui  yu
chenjie  cao
fan  wang
weiming  hu
jin  gao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 15:08:22

Abstract: Recent advances in 4D generation mainly focus on generating 4D content by distilling pre-trained text or single-view image conditioned models. It is inconvenient for them to take advantage of various off-the-shelf 3D assets with multi-view attributes, and their results suffer from spatiotemporal inconsistency owing to the inherent ambiguity in the supervision signals. In this work, we present Animate3D, a novel framework for animating any static 3D model. The core idea is two-fold: 1) We propose a novel multi-view video diffusion model (MV-VDM) conditioned on multi-view renderings of the static 3D object, which is trained on our presented large-scale multi-view video dataset (MV-Video). 2) Based on MV-VDM, we introduce a framework combining reconstruction and 4D Score Distillation Sampling (4D-SDS) to leverage the multi-view video diffusion priors for animating 3D objects. Specifically, for MV-VDM, we design a new spatiotemporal attention module to enhance spatial and temporal consistency by integrating 3D and video diffusion models. Additionally, we leverage the static 3D model’s multi-view renderings as conditions to preserve its identity. For animating 3D models, an effective two-stage pipeline is proposed: we first reconstruct coarse motions directly from generated multi-view videos, followed by the introduced 4D-SDS to model fine-level motions. Benefiting from accurate motion learning, we could achieve straightforward mesh animation. Qualitative and quantitative experiments demonstrate that Animate3D significantly outperforms previous approaches. Data, code, and models are open-released.

Main paper: 
 introduction: 
 3D content creation has garnered significant attention due to its wide applicability in AR/VR, gaming, and movie industry. With the development of diffusion models [45, 44, 39, 20, 51, 6, 19] and large-scale 3D object datasets [13, 12, 37, 65, 70, 14] , recent 3D foundational generations have seen extensive exploration through fine-tuned text-to-image (T2I) diffusion models [31, 40, 41, 52, 32, 49] , as well as training large reconstruction models from scratch [22, 61, 47, 67, 54] , leading the 3D assets creation to a new era. Despite the significant progress in static 3D representation, this momentum has not been paralleled in the realm of dynamic 3D content generation, also known as 4D generation.4D generation is more challenging due to the difficulty of simultaneously maintaining spatiotemporal consistency in visual appearance and dynamic motion. In this paper, we mainly focus on two challenges: 1) No foundational 4D generation models to unify both spatial and temporal consistency. Though recent 4D generation works [43, 24, 5, 69, 29, 38, 63, 59, 4, 57, 66, 16] separately distill knowledge from pre-trained T2I and 3D diffusion models [39, 41, 31] and video diffusion models [51, 2, 6] to model multi-view spatial appearance and temporal motions respectively, we clarify that such a detached learning way suffers from inevitable accumulation of appearance degradation as the motion changed, as shown in Fig. 1 (SVD+Zero123). 2) Failing to animate existing 3D assets through multiview conditions. With the development of 3D generations, animating existing high-quality 3D content becomes a common demand. However, previous works about 4D modeling from video [24, 57] or based on generated 3D assets [5, 29, 38, 66] are all based on text [39, 41] or single-view [31, 40] conditioned models, struggling to faithfully preserve multi-view attributes during the 4D generation, such as the back of butterfly in Fig. 1 is ignored by Zero123 [31] .To address these issues, we advocate for an approach better suited for 4D generation, that is, animating any off-the-shelf 3D models with unified spatiotemporal consistent supervision. In this way, it would be convenient to directly take advantage of various fast-developing 3D generation and reconstruction approaches based on a single foundational model, eliminating the accumulation of errors in modeling appearance and motion.To this end, we propose a novel 4D generation framework called Animate3D in this paper, which can be divided into a foundational 4D generation model and a joint 4D Gaussian Splatting (4DGS) optimization. Formally, the foundational 4D model is a Multi-View Video Diffusion Model (MV-VDM) built upon the 3D generation model, MVDream [41] , which can synchronously synthesize multi-view images with various temporal motions. Specifically, to better inherit the prior in previous 3D and video diffusion models trained on large-scale data, we propose a learnable plug-and-play spatiotemporal attention module, building upon the motion module in video diffusion [18, 6, 20] to expand the attention learning from the temporal domain to the spatial and temporal domain. Moreover, MV-VDM also includes the ability to refer to multi-view images, sufficiently preserving the identity and details of off-the-shelf 3D assets. Specifically, given multi-view images rendered from existing 3D assets or collected from real-world objects, we expand adaptive image-to-video work, I2V-Adapter [17] , to multi-view version, called MV2V-Adapter, incorporating multi-view conditions to 4D learning through additionally spatial features and text embeddings. Enhanced by the multi-view appearance injection, we can disentangle the appearance learning from the motion learning, ensuring MV-VDM focuses on learning natural and coherent dynamic motions.To further enable impressive animations from 3D objects that can be observed at any viewpoint and time, we jointly optimize the 4DGS [55] through both reconstruction and 4D Score Distillation Sampling (4D-SDS) losses based on our unified MV-VDM. Benefiting from the spatiotemporal consistent multi-view video generations, 4DGS can be roughly converged to proper results with only reconstruction loss, while 4D-SDS further improves the details and fine-grained motions. The Gaussian trajectory learned by our framework is surprisingly accurate and could be used to directly animate the mesh.The main dilemma in building a foundational 4D generation model lies in the rarity of large-scale 4D datasets, which is non-trivial to collect but the key factor to drive our MV-VDM. In this work, we make the first attempt to build a large-scale multi-view video (4D) dataset, dubbed MV-Video. Specifically, MV-Video comprises about 84K animations that are available under a public license, consisting of about 38K animated 3D objects at all, which are rendered into over 1.3M multi-view videos with minigpt4-video [3] generated prompts, to serve as the training dataset for our 4D foundation model.We highlight the contribution of this paper as follows: 1) Animate3D is the first 4D generation framework to animate any 3D objects with detailed multi-view conditions. The framework is further extended to achieve mesh animation without skeleton rigging. 2) We propose the foundational 4D generation model, MV-VDM, to jointly model spatiotemporal consistency. 3) We present the largest 4D datasets MV-Video collected with about 84K animations and over 1.3M multi-view videos. Extensive experiments demonstrate that our data-driven approach can generate spatiotemporal consistent 4D objects, significantly outperforming previous counterparts.2 Related Work 3D Generation. Early 3D generation works optimized single 3D object with CLIP loss [36] or Score Distillation Sampling (SDS) [33] from 2D text-to-image (T2I) diffusion models. Since the models providing supervision lacked 3D prior, those works usually suffered from spatial inconsistency, i.e., multi-face Janus problem [50, 28, 53] . To tackle this problem, on the one hand, some works [31, 40, 41, 32, 35] lifted the T2I diffusion to multi-view image diffusion by injecting new spatial attention layers and fine-tuning on large-scale 3D synthetic datasets [13, 12] . Although 3D consistency was improved, these optimization-based methods still required a relatively long time to optimize a 3D object. On the other hand, some feed-forward 3D generation foundation models [22, 61, 47, 67, 54, 60, 30] , also trained on large-scale 3D datasets, were able to produce a good-quality 3D object in several seconds in an inference-only way. Inspired by the success of the data-driven approaches in 3D generation, we aim to construct a large-scale 4D generation dataset and take the pioneering step towards developing foundation models for 4D generation.Video Generation. Video generation works started with text-to-video (T2V) generation [20, 42, 2, 18, 6, 19, 7, 10] , subsequently followed by image-to-video (I2V) approaches [64, 17, 58, 6] . Previous T2V works usually built upon T2I diffusion models [20, 42, 19, 18, 21] , leveraging their pre-trained weights by leaving the spatial blocks unchanged and inserting new temporal blocks to model the temporal camera or object motions. The I2V works [58, 17, 64] , building upon the aforementioned T2V methods, typically incorporate image semantics into video models. This is achieved through cross-attention mechanisms between noisy frames and the conditional image, while retaining the motion module design from the T2V models unaltered. We draw inspiration from the development paradigm of video generation to design our 4D generation foundation model, which is a multi-view image conditioned multi-view video diffusion model building upon pre-trained multi-view 3D and video diffusion models.4D Generation. The pioneering work of 4D generation is MAV3D [43] , which is a text and imageconditioned 4D generation framework. MAV3D first proposed a multi-stage pipeline to optimize the static 3D object generation through the T2I model and subsequently learn motions from the T2V model [42] . Following works [68, 5, 29, 69, 4, 59 ] adopted a similar pipeline, and they further found that employing T2I [39] and 3D-SDS [41] are crucial for both object generation and motion learning stages. Without them, the quality of the generated object's appearance suffered a remarkable decline, and the motion-learning process was prone to failure. Very recently, Consistent4D [24] proposed a video-to-4D generation task, which used single-view video reconstruction and SDS from Zero123 [31] for motion and appearance learning. This paradigm was adopted by following works [38, 63, 66, 57, 16, 11] and extended to text/image-to-video then video-to-4D generation. All aforementioned works heavily depend on the foundational model for SDS to preserve objects' appearance and attributes. However, existing 3D diffusion models struggle to refer to multi-view … < l a t e x i t s h a 1 _ b a s e 6 4 = " 9 9x O M 5 I s O 2 O / d f a t d d R / A V c 8 + u 0 = " > A A A C x H i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R L x t S w K 4 r I F + 4 B a J E m n d W h e Z C Z C K f o D b v X b x D / Q v / D O O A W 1 i E 5 I c u b c e 8 7 M v d d P Q y 6 k 4 7 w W r L n 5 h c W l 4 n J p Z X V t f a O 8 u d U S S Z 4 F r B k k Y Z J 1 f E + w k M e s K b k M W S f N m B f 5 I W v 7 o 3 M V b 9 + x T P A k v p L j l P U i b x j z A Q 8 8 S V R j c F O u O F V H L 3 s W u A Z U Y F Y 9 K b / g G n 0 k C J A j A k M M S T i E B 0 F P F y 4 c p M T 1 M C E u I 8 R 1 n O E e J d L m l M U o w y N 2 R N 8 h 7 b q G j W m v P I V W B 3 R K S G 9 G S h t 7 p E k o L y O s T r N 1 P N f O i v3 N e 6 I 9 1 d 3 G 9 P e N V 0 S s x C 2 x f + m m m f / V q V o k B j j V N X C q K d W M q i 4 w L r n u i r q 5 / a U q S Q 4 p c Q r 3 K Z 4 R D r R y 2 m d b a 4 S u X f X W 0 / E 3 n a l Y t Q 9 M b o 5 3 d U s a s P t z n L O g d V B 1 j 6 t H j c N K 7 c y M u o g d 7 G K f 5 n m C G i 5 R R 1 N 7 P + I J z 9 a F F V r C y j 9 T r Y L R b O P b s h 4 + A E O c j 3 U = < / l a t e x i t > G E z Q A 1 m X Y = " > A A A C 3 H i c j V H L S s N A F D 2 N r 1 p f V R c u 3 A S L 4 K q k 4 m t Z d O O y g n 1 A W 0 u S T t v Q v J h M x B K 6 c y d u / Q G 3 + j 3 i H + h f e G d M Q S 2 i E 5 K c O f e e M / f O t U L X i Y R h v G a 0 m d m 5 + Y X s Y m 5 p e W V 1 L b + + U Y u C m N u s a g d u w B u W G T H X 8 V l V O M J l j Z A z 0 7 N c V r e G Z z J e v 2 Y 8 c g L / U o x C 1 v b M v u / 0 H N s U R= " > A A A C x H i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w V R L x t S w K 4 r I F + 4 B a J E m n d W h e Z C Z C K f o D b v X b x D / Q v / D O O A W 1 i E 5 I c u b c e 8 7 M v d d P Q y 6 k 4 7 w W r L n 5 h c W l 4 n J p Z X V t f a O 8 u d U S S Z 4 F r B k k Y Z J 1 f E + w k M e s K b k M W S f N m B f 5 I W v 7 o 3 M V b 9 + x T P A k v p L j l P U i b x j z A Q 8 8 S V R j c F O u O F V H L 3 s W u A Z U Y F Y 9 K b / g G n 0 k C J A j A k M M S T i E B 0 F P F y 4 c p M T 1 M C E u I 8 R 1 n O E e J d L m l M U o w y N 2 R N 8 h 7 b q G j W m v P I V W B 3 R K S G 9 G S h t 7 p E k o L y O s T r N 1 P N f O i v 3 N e 6 I 9 1 d 3 G 9 P e N V 0 S s x C 2 x f + m m m f / V q V o k B j j V N X C q K d W M q i 4 w L r n u i r q 5 / a U q S Q 4 p c Q r 3 K Z 4 R D r R y 2 m d b a 4 S u X f X W 0 / E 3 n a l Y t Q 9 M b o 5 3 d U s a s P t z n L O g d V B 1 j 6 t H j c N K 7 c y M u o g d 7 G K f 5 n m C G i 5 RY F 8 = " > A A A C z H i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w I S U R X 7 g q u n E l F e x D a p U k n d a h e T G Z C K V 0 6 w + 4 1 e 8 S / 0 D / w j t j C m o R n Z D k z L n 3 n J l 7 r x v 7 P J G W 9 Z o z p q Z n Z u f y 8 4 W F x a X l l e L q W j 2 J U u G x m h f 5 k W i 6 T s J 8 H r K a 5 N J n z V g w J 3 B 9 1 n D 7 p y r e u G c i 4 V F 4 K Q c x a w d O L + R d 7 j m S q K v m z d A + D n f s 0 W 2 x Z J U t v c x J Y G e g h G x V o + I L r t F B B A 8 p A j C E k I R 9 O E j o a c G G h Z i 4 N o b E C U J c x x l G K J A 2 p S x G G Q 6 x f f r 2 a N f K 2 J D 2 y j P R a o 9 O 8 e k V p D S x R Z q I 8 g R h d Z q p 4 6 l 2 V u x v 3 k P t q e 4 2 o L + b e Q X E S t w R + 5 d u n P l f n a p F o o s j X Q O n m m L N q O q 8 z C X V X V E 3 N 7 9 U J c k h J k 7 h D s U F Y U 8 r x 3 0 2 t S b R t a v e O j r + p j M V q / Z e l p v i X d 2 S B m z / H O c k q O + W 7 Y P y / s V e q X K S j T q P D W x i m + Z 5 i A r O U E W N v A M 8 4 g n P x r k h j a E x + k w 1 c p l m H d + W 8 f A B a W G S Q w = = < / l a t e x i t > X 1:n,1 < l a t e x i t s h a 1 _ b a s e 6 4 = " p I n 2 U 4 + 1 Q X l P 4 1 B Q P t i f k p f Y + V 8 = " > A A A C z H i c j V H L S s N A F D 2 N r 1 p f V Z d u g k V w I S U R X 7 g q u n E l F e x D a p U k n d a h e Z F M h F K 6 9 Q f c 6 n e J f 6 B / 4 Z 1 x C m o R n Z D k z L n 3 n J l 7 r x v 7 P B W W 9 Z o z p q Z n Z u f y 8 4 W F x a X l l e L q W j 2 N s s R j N S / y o 6 T p O i n z e c h q g g u f N e O E O Y H r s 4 b b P 5 X x x j 1 L U h 6 F l 2 I Q s 3 b g 9 E L e 5 Z 4 j i L p q 3 g z t 4 3 C H j 2 6 L J a t s q W V O A l u D E v S q R s U X X K O D C B 4 y B G A I I Q j 7 c J D S 0 4 I N C z F x b Q y J S w h x F W c Y o U D a j L I Y Z T j E 9 u n b o 1 1 L s y H t p W e q 1 B 6 d 4 t O b k N L E F m k i y k s I y 9 N M F c + U s 2 R / 8 x 4 q T 3 m 3 A f 1 d 7 R U Q K 3 B H 7 F + 6 c e Z / d b I W g S 6 O V A 2 c a o o V I 6 v z t E u m u i J v b n 6 p S p B D T J z E H Y o n h D 2 l H P f Z V J p U 1 S 5 7 6 6 j 4 m 8 q U r N x 7 O j f D u 7 w l D d j + O c 5 J U N 8 t 2 w f l / Y u 9 U u V E j z q P D W x i m + Z 5 i A r O U E W N v A M8 4 g n P x r k h j K E x + k w 1 c l q z j m / L e P g A 7 p m S e w = = < / l a t e x i t > X 1:n,iSpatiotemporal Attention Spatiotemporal Attention conditions, restricting their broader applications to animate various off-the-shelf 3D assets without losing their multi-view attributes. Furthermore, it is worth noting that existing 4D generation methods suffer from another issue, i.e., spatial and temporal inconsistency [43, 5, 29, 24, 38] . Because the diffusion models used for SDS were never trained with multi-view video (4D) datasets, missing the critical capacity to formulate spatial and temporal consistency simultaneously. Thus previous methods failed to properly trade off a good balance between appearance and motion learning. Please refer to Sec. B in appendix for detailed discussion and comparison.In this work, we resort to disentangle 3D object generation/reconstruction and motion learning through a foundational 4D generation model, and propose a novel framework to animate any static 3D object with consistent multi-view attributes. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: we have included our limitation in Sec. C.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 Given a static 3D model, our goal is to animate it with a text prompt and use its multi-view renderings as image condition. This 4D generation task is particularly challenging as it requires ensuring spatial and temporal consistency of the appearance and motion, compatibility with the prompt, and preserving the identity of the static object. To address these challenges more fundamentally, we propose a novel framework, Animate3D, to animate any static 3D object. As depicted in Fig. 2 , we divide the task into two parts: learning a multi-view video diffusion model (MV-VDM), and animating 3D object with multi-view videos generated by MV-VDM.We propose a novel multi-view image conditioned multi-view video diffusion model, named MV-VDM. To inherit prior knowledge acquired by spatially consistent 3D models and temporally consistent video models trained on large-scale datasets, we advocate a baseline architecture by integrating them to utilize their pre-trained weights. In this work, we take MVDream [41] and AnimateDiff [18] for the 3D and the video diffusion model, respectively. To enhance the spatiotemporal consistency and ensure compatibility with the prompts and the object's multi-view images, we propose an efficient plug-and-play spatiotemporal attention module combined with an image-conditioning approach. Our MV-VDM is trained on our presented large-scale multi-view video dataset, MV-Video, which is introduced in Sec. 4.As illustrated in Fig. 2 , we insert a novel spatiotemporal attention module after cross-attention layers. The proposed spatiotemporal attention module comprises two parallel branches: the left branch is for spatial attention, and the right branch is for temporal attention. For spatial attention, we adopt the same architecture as the multi-view 3D attention in MVDream [41] . Specifically, the original 2D self-attention layer is converted into 3D by connecting n different views. In addition, we incorporate 2D spatial encoding, specifically sinusoidal encoding, into the latent features to enhance spatial consistency. As for temporal attention, we keep all designs of the temporal motion module from the video diffusion model [18] unchanged in order to reuse their pre-trained weights. Based on the features of these two branches, we employ an alpha blender layer with a learnable weight to achieve features with enhanced spatiotemporal consistency. It is worth noting that we do not apply spatiotemporal attention across all views and frames due to the prohibitive GPU memory requirements that render training infeasible. Instead, our parallel-branch design offers an efficient and practical alternative. Specifically, we first reshape the input feature of spatiotemporal attention module X ∈ R (b×n×f )×c×h×w into two forms, X l ∈ R (b×f )×(n×h×w)×c for spatial branch and X r ∈ R (b×n×h×w)×f ×c for temporal branch. The spatiotemporal attention is then computed as:EQUATIONwhere µ denotes the learnable weight, Ws/t Q , W s/t K , W s/t V , W s/tO represent the corresponding projection matrices. b, n, f, h, w, c are the batch size, views, frames, height, width, and channels of the image features, respectively.Multi-view Images Conditioning. Inspired by I2V-Adapter [17] , we add a new attention layer, termed MV2V-Adapter, parallel to the existing frozen multi-view 3D self-attention layer within the proposed spatiotemporal block, as shown in Fig. 2 . Concretely, noisy frames are first concatenated along the spatial dimension. These are then used to query the rich contextual information from the multi-view conditional frames, which are extracted using the frozen 3D diffusion model. Next, we add the output of the MV2V-Adapter layer to that of the original multi-view 3D attention layer of MVDream. Thus, for each frame i ∈ {1, ..., f }, denoting the multi-view input, output, and conditional frames' features as X 1:n,i , X 1:n,i out , and X 1:n,1 , we have:EQUATIONwhere W Q , W K , W V and W O are projection matrices in original self-attention layer, while W Q ′ and W O ′ are those in the newly added layer. We find this simple cross-attention operator can effectively improve the object's appearance consistency in the generated video. After that, as shown in the spatiotemporal block in Fig. 2 , we employ two cross-attention layers to align the text prompt and preserve the object's identity, respectively. The left one is inherited from MVDream, while the right one is pre-trained in IP-Adapter [62] .Training Objectives. The training process of our multi-view video diffusion model is similar to Latent Diffusion Model [39] . Specifically, the sampled multi-view video data q 1:n,1:f 0 are first encoded into latent feature z 1:n,1:f 0 via encoder E frame by frame and view by view. Then we add noise using the forward diffusion scheduler:z 1:n,2:f t = √ ᾱt z 1:n,2:f 0 + √ 1 -ᾱt ϵ, where α t is a weighted parameter and ϵ is Gaussian noise. Note that, following I2V-Adapter, we keep the first frame, i.e., the condition multi-view frames clean, and only add noise to the rest of the frames. During training, the proposed MV-VDM takes as input the clean latent code z 1:n,1 0 , noisy latent code z 1:n,2:f 0 , text prompt embedding y, and the camera parameters Σ 1:n , and outputs the noise strength, supervised by L 2 loss. The training objective of our MV-VDM is calculated as:EQUATIONwhere θ denotes the diffusion model. It is important that we keep the entire multi-view 3D attention module frozen and only train the MV2V-Adapter layer and the spatiotemporal attention module to conserve GPU memory and accelerate training. Moreover, as the multi-view images of the first frame, z 1:n,1 0 , serves as the condition images, we calculate the loss only for the latter f -1 frames, i.e., z 1:n,2:f 0 .Based on our 4D generation foundation model MV-VDM, we propose to animate any off-the-shelf 3D object. For efficiency, we take 3D Gaussian Splatting (3DGS) [25] as the static 3D object representation, and animate it by learning motion fields represented by Hex-planes, as in [55] .4D Motion Fields. As in 4D Gaussian Splatting (4DGS) [55] , we represent the motion fields by Hex-planes [15, 8] . Denoting the static 3DGS as G = {X , C, α, r, s}, where X , C, α, r, and s represent the position, color, opacity, rotation, and scale, respectively. The motion module D predicts changes in position, rotation, and scale for each Gaussian point in frame i by interpolating the Hex-planes R. The motion fields computation can be formulated as:EQUATIONEQUATIONwhere l equals to the scales in Hex-plane, and interp() denotes interpolating the Gaussian points on the specific plane ζ to obtain corresponding motion features. We have ζ ∈ {(x, y), (x, z), (y, z), (x, t), (y, t), (z, t)}. Therefore, Gaussian G ′ at time t is updated as follows:EQUATIONTo better preserve the appearance of static 3D objects, we keep certain attributes, specifically opacity α and color C, unchanged.Coarse Motion Reconstruction. Based on the spatiotemporal consistent multi-view videos generated by MV-VDM, we first leverage a 4DGS reconstruction stage to directly reconstruct the coarse motions. Specifically, we use a simple but effective L 2 loss as our L rec , which is calculated as:EQUATIONwhere C and C denote the multi-view and multi-frame renderings and the corresponding ground truth. As verified in Fig. 3 , this reconstruction stage can already learn high-quality coarse motions by leveraging the generated multi-view videos of MV-VDM.4D-SDS Optimization. To better model the fine-level motions, we introduce a 4D-SDS optimization stage to distill the knowledge of our multi-view video diffusion model. The 4D-SDS loss L 4D-SDS is a variant of z 0 -reconstruction SDS loss and can be formulated as:EQUATIONwhere z and z 0 are latent feature of the rendered image and the estimation of clean latent feature from current noise prediction ϵ θ , respectively, g represents the rendering function. α t and σ t are the signal and noise scale controlled by the noise scheduler, respectively.Training Objectives. In addition to L rec and L 4D-SDS , we introduce a variant of As-Rigid-As-Possible (ARAP) loss [46] to facilitate the rigid movement learning as well as the maintenance of the high-quality appearance of the static object. The ARAP loss L arap in our work is defined as:EQUATION)where Rj is estimated from a rigid transformation using Singular Value Decomposition (SVD) according to [46] :EQUATION)N cj denotes the set of points within a fixed radius of p j , and w j,k = exp(-d jk d) where d jk is the distance between center of p j and p k , measuring the impact of p k on p j . This loss encourages the generated dynamic object to be locally rigid, and it enhances the learning with rigid movement.In summary, the training objectives for animating off-the-shelf 3DGS object is:EQUATIONwhere λ 1 , λ 2 , and λ 3 are weighted parameters.To directly utilize high-quality mesh generated from commercial 3D generation tools and crafted by human experts, we extent our framework to mesh animation, producing animated mesh compatible with standard 3D rendering pipelines.We initialize the 3DGS representation of the given object by vertices and triangles of the static mesh. Specifically, the color is determined by vertex color and we average the connected edges for the scale. Opacity and rotation are set to fully visible and zero rotation quaternion, respectively. The coarse 3DGS is animated following the motion reconstruction steps as described in the above sections. We utilize the per-vertex Gaussian trajectory to deform the static mesh in a straightforward way without skeleton rigging, control point selection or complicated deformation algorithms. As shown in Fig. 6 and our project page, the results are surprisingly good despite the simplicity of the solution.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
setup: 
 Training Dataset. To train our MV-VDM, we build a large-scale multi-view video dataset, MV-Video. Concretely, we render multi-view videos of 37,857 animated 3D models collected from Sketchfab [1] . Each model has 2.2 animations on average, resulting in 83,716 animations in total. Each animation is 2 seconds long at 24 fps. Note that animated models that are not allowed to be used to generate AI programs are filtered. The statistical information of our MV-Video dataset is reported in Table 1 . For more details about the rendering settings and data examples, please refer to our Appendix ( D). We will release this dataset to further advance the field of 4D generative research. Implementation Details. We sample 8 frames evenly for each animation to train our MV-VDM. We use the Adawm optimizer with a learning rate of 4e -4 and a weight decay 0.01, and train the model for 20 epochs with a batch size of 2048. When inference, we set the sampling step to 25 and adopt freeinit [56] to get stable results when animating 3D objects. As for 4D generation, the resolution and feature dimension of the Hex-planes are set to [100, 100, 8] and 16, respectively. We perform coarse motion reconstruction for the first 1000 iterations with a batch size of 32 (4 views, 8 frames), and then add 4D-SDS optimization for another 400 iterations. Learning rate is 0.0015 initially and decreases linearly to 0.0005 at the end of reconstruction stage. λ 1 , λ 2 and λ 3 in Eq. 11 are set to 10.0, 0.01 and 0.5 , respectively. It costs 3 days to train our MV-VDM on 32 80G A800 GPUs, and the optimization for 4D generation takes around 30 minutes on a single A800 GPU per object.Evaluation Dataset. For the evaluation of MV-VDM, we render multi-view images from 128 static 3D objects and then generate multi-view videos conditioned on them. Following the evaluation setting of VBench [23] , we use four different random seeds for each object and report the average results. For the evaluation of 4D generation, we generate 25 objects across various categories using the large 3DGS reconstruction model GRM [61] . Input images for GRM and animation prompts used in this work are provided in our Appendix ( E.1).Evlaution Metrics. We adopt the evaluation protocol proposed in VBench [23] , which is a popular video generation benchmark consisting of both T2V and I2V evaluation tools. The I2V evaluation protocol contains 9 evaluation metrics, and we choose 4 for our evaluation, i.e., I2V Subject, Motion Smoothness, Dynamic Degree, and Aesthetic Quality, measuring the consistency with the given image, the motion smoothness, the motion degree, and the appearance quality, respectively. We abbreviate them as I2V, M. Sm., Dy. Deg. and Aest. Q. in the experiment section.Values of all metrics are the higher, the better, except for Dynamic Degree, since we observe that sometimes completely failed results present extremely high dynamic degree. For more details about the introduction and calculation of the evaluation metrics, please refer to our Appendix ( E.2).Comparison Methods. We compare our work with 4Dfy [5] and DreamGaussian4D (DG4D) [38] on the task of animating any given 3D object using their official implementations. They represent the state-of-the-art in 4D generation methods, starting by generating a static 3D object using 3D-SDS in Reconstruction (Ours) t = t 1 t = t 2 the initial stage, and subsequently animating it via video SDS and single-view video reconstruction in later stages. For 4D representation, 4Dfy and DG4D adopt dynamic NeRF [34, 26] and 4DGS [55] , respectively. For a fair comparison, we replace the dynamic NeRF in 4Dfy with 4DGS used in both our work and DG4D, and also apply ARAP loss for motion regularization. For DG4D, we keep the 4DGS representation and motion regularizations in their work unchanged.In this section, we perform comprehensive comparisons with previous works, including quantitative and qualitative comparisons and user studies. which could lead to potential appearance degradation. More importantly, DG4D fails when the object in the guided video is assigned with movements toward the camera. For example, the frog is moving forth and back in the guided video, however, DG4D interprets it as enlargement and reduction of the object. The same thing comes to the penguin which nods towards the camera and leans forward. This misinterpretation usually results in blurry effect and strange appearance.In contrast, our method, leveraging the spatiotemporal consistent muti-view prior, manages to deal with motion towards the camera, as demonstrated by the bear's raised front paw (our model takes the front view and its orthogonal views as the condition view, not depicted in the image). Besides, we successfully maintain the high quality appearance of the given 3D object when generating natural motion. Please refer to the videos in our supplementary material for a more intuitive comparison.User Study. We conduct a user study among 20 people on the 25 dynamic objects and report the averaged results in Tab. 2b. The participants are asked to score the generated dynamic objects from 1 to 5, according to the alignment with the given text (Align. Text) and static object (Align. 3D), motion quality (Mot.), and appearance quality (Appr.). The user study proves the superiority of our method. Please refer to the Appendix ( E.3) for more details. 4D Object Optimization. The ablations for 4D object optimization are shown in Tab. 3b and Fig. 5 . The quantitative results in Tab. 3b indicate both SDS and ARAP losses improve the alignment with the 3D object (I2V Subject), Motion Smoothness, and Aesthetic Quality. However, the Dynamic Degree decreases. We suppose the decrease in dynamic degree is mainly caused by the removal of floaters and blurry effects, which are also taken into account of dynamic degree, as shown in Fig. 5 . The two losses might slightly decrease the motion amplitude, but generally, we think the overall performance is improved when applying them.We provide mesh animation results in Fig. 6 . Static meshes are generated by commercial 3D generation tools. For more results, please visit our project page. 
conclusion: 
 In this work, we present Animate3D, a novel framework for animating any off-the-self 3D object. Animate3D disentangles the 4D object generation into a foundational 4D generation model, MV-VDM, and a joint 4DGS optimization pipeline based on MV-VDM. MV-VDM is the first 4D foundation model, which can generate spatiotemporal consistent multi-view videos conditioned on multi-view renderings of a static 3D object. To train MV-VDM, we present the largest multi-view video (4D) dataset, MV-Video, containing about 84K animations with over 1.3M multi-view videos. Based on MV-VDM, we propose an effective pipeline to animate any static 3D object by jointly optimizing the 4DGS via both reconstruction and 4D-SDS. Animate3D is a highly practical solution for downstream 4D applications since it can animate any generated or reconstructed 3D objects. Data, codes, and pre-trained weights will be released to facilitate the research in 4D generation.  
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: we include detailed implementations in Sec. 4.1. The full details can be seen in our codes, which will be open-released.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: As a visual generative model, we provide sufficient visualization results instead.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: we provide the it in Sec. F. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: our paper poses no such risks. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: we have cited related works and codes. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: We provide more qualitative results as below. Please refer to the videos attached in our supplementary materials for more intuitive visualizations.Comparison. In the main paper, we only present the animation results for 3D objects generated by large reconstruction models. But our method is able to animate reconstructed objects as well as generate 3D models from text/image/video, and shows a great advantage over other methods as shown in Fig. 7 . In Fig. 7 , 4Dfy produces results inconsistent with the given 3D object, and they are almost static. Distortions and blurry effects could be observed in results generated by DreamGaussian4D, and it sometimes deviates from the given object in novel views. In contrast, our approach can generate results not only spatially and temporally coherent but also consistent with the input object.Ablation. We provide more qualitative ablations of MV-VDM and 4D generation in Fig. 8 . motion and image/3D diffusion for appearance. However, the motion and appearance supervisions adopted in their work are not orthogonal, and sometimes have negative effect on each other.Ours Input t = t 1 t = t 2 t = t 1 t = t 2 t = t 1 t = t 2 t =For example, it is commonly agreed that video-diffusion-SDS usually brings unappealing visual effect to the appearance of the object [68, 69, 29] . Meanwhile, the appearance supervision signal prevents 4D object from updating along the direction that follows the exact score function of the video diffusion model, leading to less natural motion. Small motion amplitude in [5, 69] and shaky appearance in [29] could partly support this point. As for monocular-video-guided motion learning, previous work [38, 63, 66] rely on 3D diffusion model (Zero123 [31] ) to supervise both motion and appearance in novel view. Since Zero123-SDS is applied per-frame, temporal consistency in novel view cannot be guaranteed. Moreover, monocular video doesn't provides information about depth/distance, so moving closer to or farther away from the camera can be perceived as the magnification or reduction of the object, resulting in appearance distortion.In contrast, our method takes the unified supervision signal from MV-VDM for motion learning and appearance preservation. Our motion and appearance supervision signal inherently don't conflict with each other, since MV-VDM is conditioned on multi-view attributes of the 3D object to generate multi-view videos. Besides, multi-view motion supervision in our work enables more natural motion generation when compared with single-view motion supervision in other works. Thus, we achieve superior performance in terms of both motion generation and appearance preservation in the task of animating any off-the-shelf 3D object.To provide a comprehensive understanding of the difference between our method and previous work, we conduct comparisons with more open-sourced previous works using their official implementations, and summarize the results in Tab. 4. Despite the promising performance in generating spatiotemporal consistent 4D objects, our method still has a few limitations. First, it takes a relatively long time (about 30 minutes) to animate an existing 3D object. Second, there is a trade-off between temporal coherence and motion amplitude in the multi-view videos generated from the proposed MV-VDM. Specifically, the larger the motion amplitude, the higher the risk of temporal incoherence. Third, our model sometimes fails to animate realistic scenes due to the domain gap between synthetic training data and real-world test data. At last, current evaluation metrics in 4D generation are not sufficient, as they mainly rely on video generation metrics and user studies. Designing more suitable metrics for 4D generation will be an important future work.D.1 Rendering Details.For the rendering settings, we first centralize the model according to the bounding box of the object in the first frame. Then, we adjust the camera distance to make sure the object stays in the scope of view during the animation process. Sixteen views are evenly sampled in terms of azimuth, starting from values randomly selected between -11.25 • and 11.25 • . The elevation angle is randomly sampled within the range of 0 • to 30 • . To stabilize training, we manually filter out objects that we identify as challenging to learn due to factors such as large movements, complex environmental interactions, high speed, or sudden changes in appearance.As shown in Fig. 12 and Fig. 13 , we showcase more examples of our multi-view video dataset (MV-Video).Furthermore, as shown in Fig. 9 , we extracted all nouns from the text captions of our MV-Video dataset, which are generated by minigpt4-video [3] , and plotted a word cloud of the Top-1000 nouns.We can see that our MV-Video dataset contains diverse categories of animated 3D objects, including humans, characters, animals, plants, mechanical structures, and ect.. In Fig. 10 , we provide the input images for image-to-3D generation and corresponding text prompts for 4D animation used in Sec. 4.2.VBench [23] provides six evaluation metrics for I2V evaluation3 , i.e., I2V Subject, I2V Background, Camera Motion, Subject Consistency, Background Consistency, Motion Smoothness, Dynamic Degree, Aesthetic Quality and Imaging Quality. Since our generated results have no background, and the evaluation cameras are fixed, metrics related to background and camera motion are not used. The metric Imaging Quality, which is affected by ambient light, is also not used here. Subject Consistency is also omitted since its calculation process is similar to I2V Subject, except for the choice of reference frame. It takes the first generated frame, instead of the input image used in I2V Subject, as the reference frame for similarity score calculation, which is not suitable for our task of animating 3D objects. We briefly introduce the evaluation metrics used in our work as below:A cartoon dolphin with a playful smile and sparkling eyes leaps joyfully from the waves.A cartoon frog with a friendly, wide mouthed smile is walking.A cartoon penguin walks awkwardly.A I2V Subject assess whether the appearance of the object in the generated video remains consistent with that in the input image. To this end, DINO [9] feature similarity across frames is calculated.Motion Smoothness evaluates whether the motion in the generated video is smooth, and follows the physical law of the real world. The motion prior in the video frame interpolation model [27] is utilized for evaluation.Dynamic Degree employs RAFT [48] to estimate the degree of dynamics in synthesized videos.Aesthetic Quality is calculated by the LAION aesthetic predictor, which reflects the artistic and beauty value perceived by humans towards each frame.As illustrated in Fig. 11 , a picture of the user study page is depicted. The survey contains 25 dynamic objects, which are shown in Fig. 10 . The participants are asked to score the generated 4D objects from 1 to 5, according to the alignment with the given static object and text prompt, appearance quality, and motion quality. This paper exploited 4D generation based on our proposed multi-view video diffusion model, which generate spatiotemporal consistent multi-view videos. Because of the advanced generative capacity, our models may output misinformation or fake videos. Thus, we sincerely remind the users to pay attention to it. Note that our method only focuses on the technical aspect. All the code, dataset, and trained models will be released. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.https://github.com/Vchitect/VBench/tree/master/vbench2_beta_i2v 


Decision: Pre-rebuttal, reviewers praised the overall relevance and novelty of the tackled problem, and the intuitive solution proposed to solve it. Recurring concerns included insufficient evaluation against too few baselines, missing ablations, concerns about the diversity of the generated motions, and incomprehensibility of the method section.

Authors provided a strong rebuttal. All reviewers unanimously agreed to accept the paper conditioned on:
1. Authors include additional comparisons and ablations in their final version.
2. Significantly improve the writing quality in the method section.

The AC thus accepts the paper conditioned on the fact that authors fix both 1. and 2. above in the final version of their paper.

Reviews: review: paper_summary: paper_summary: This paper focuses on animating 3D objects with multi-view diffusion models. To improve spatial and temporal consistency, this work builds a large-scale multi-view video dataset, MV-Video and designs an attention module to enhance spatial and temporal consistency by integrating 3D and video diffusion models. To enhance animations from 3D objects, this work jointly optimizes the 4DGS through reconstruction and 4D-SDS. Experiments show the proposed method can more consistent 4D generation.
strength_weakness: strength_weakness: * This work focuses on 3D objects, however, the paper title claims "Animating Any 3D Models". It is a bit inappropriate.
* The effectiveness of the proposed dataset is not fully validated. Can this dataset improve single-view diffusion models or multi-view image-to-3D generation models, like MVDream?
* The diversity of the proposed method is unclear. Can one object perform different actions? For example, in Figure 3, can the frog perform swimming?
* The picture quality is a bit low, it is hard for readers to distinguish the advantages of the generated objects from the proposed method. 
* For animating from 3D object, this work first leverages 4DGS to reconstruct coarse motions and then uses 4D-SDS for refinement. If the method uses better 4DGS algorithms, maybe the improvement from 4D-SDS will become little.
questions: questions: * It is better to validate the effectiveness of the proposed dataset, can it also improve single-view diffusion models (e.g., T2V) or multi-view image-to-3D generation models (e.g., MVDream). Then, can these models further improve other methods, like DreamGaussian4D?
* Although the work verify the effectiveness of the S.T attention. The effect of MV2V-adapter is not validated.
* If the approach leverages better 4DGS reconstruction methods, such as [1] for the coarse motion reconstruction, can the 4D-SDS still improve the coarse results?  
[1] Yang, Zeyu, et al. "Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting." arXiv preprint arXiv:2310.10642 (2023).  
* As this work designs a multi-view video diffusion model, can the authors compare it with existing single-view video diffusion models?
* For Figure 4, both the ablation models show that the player touches the basketball. However, the full model does not show this interaction. I wonder if the full model can do better for this interaction.
limitations: limitations: The paper has discussed the limitations.
paper_summary: paper_summary: This paper proposes an animation method that animates a 3D model in a 4D one. A  Multi-View image conditioned multi-view Video Diffusion Model  (MV-VDM)  is presented to generate multi-view videos from multi-view renderings of a static 3D object. The MV-VDM is leveraged to train the 4D Gaussian Splatting (4DGS), where As-Rigid-As-Possible (ARAP) loss, reconstruction loss and SDS are used as objectives. In addition, a multi-view video dataset is constructed to train MV-VDM. The paper conducts experiments to show the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1. The comparison is not very fair. The paper focuses on a new task, i.e., 3D animation, while 4Dfy focuses on  text-to-4D, which is a different task. Furthermore, text-to-4D is more challenging than 3D animation since multi-view images are unavailable.  The better performance gain of the proposed method may partly attributed to additional multi-view images. In other words, the worse performance of 4Dfy and DG4D may not be due to the methods themselves. In addition, the paper replaces dynamic NeRF in 4Dfy with 4DGS. However, some hyperparameters of 4Dfy are set according to dynamic NeRF, rather than 4DGS. Instead, the paper can compare the proposed method with AYG, which directly trains 4DGS. In addition, the paper can compare Animate234, which is an image-to-4D method.

2. The motivation for the proposed spatiotemporal attention block is not clear. Besides the temporal motion module and temporal encoding, the block introduces a multi-view 3D attention module. What is the multi-view 3D attention module used for? Why are temporal motion modules and temporal encoding not enough to generate motions for multi-view videos? 

3. What is the influence of Alpha Blender on the performance of the proposed method?
 

4. Does the proposed method train all modules in the spatiotemporal attention block? It's not clear whether the "Multi-view 3D Attention" and "Temporal Motion Module (Pre-trained VDM)" are trained or frozen.

5. The implementation details of Alpha Blender are not clear. Is it implemented as a layer of MLP? 

6. The temporal encoding is unclear. According to Figure 2, there is a temporal encoding module in the spatiotemporal attention block, but there is no description of what kind of temporal encoding is used.  

7. How many views does each animated 3D object contain in the dataset? Only four orthogonal views or more?

8. Although the dataset contains 38K animated 3D objects, it is still much smaller compared to 2D video datasets. Can the proposed method trained on this dataset of this magnitude animate any 3D model?

9. Training a multi-view video diffusion model requires camera parameters. Are the camera parameters in the multi-view video dataset processed to ensure that they are consistent with the prior knowledge learned in the MVDream
questions: questions: Please refer to my question above.
limitations: limitations: The authors have provided  limitations and  societal impact of their work
paper_summary: paper_summary: This work presents Animate3D, a framework for animating static 3D models. The core idea involves two main components:
1. A multi-view video diffusion model (MV-VDM) conditioned on multi-view renderings of the static 3D object, trained on a large-scale multi-view video dataset (MV-Video).
2. A framework that combines reconstruction and 4D Score Distillation Sampling (4D-SDS) to utilize multi-view video diffusion priors for 3D object animation.
The animation process involves a two-stage pipeline: coarse motion reconstruction from generated multi-view videos, followed by 4D-SDS to model fine-level motions. Quantitative and qualitative evaluations show enhancements to previous methods.
strength_weakness: strength_weakness: 1. Missing Reference [a]: It is understandable that you did not compare Animate3D with Animate124 in this paper, as 4D-fy and DreamGaussian4D both demonstrate better performance compared to Animate124. However, it is unusual that you did not discuss Animate124 at all, given its relevance in previous comparisons by 4D-fy and DreamGaussian4D.

2. Limited Comparison Scope: The authors only compare Animate3D with 4D-fy and DreamGaussian4D, which seems insufficient. It would be more comprehensive to include comparisons with 4DGen and TC4D, as both claim superior performance over 4D-fy and have released their code. Additionally, while the modification of 4D-fy to a Gaussian representation for fair comparison is understandable, the original 4D-fy results should also be included for a thorough ablation study.

[a] Animate124: Animating One Image to 4D Dynamic Scene, \
[b] 4DGen: Grounded 4D Content Generation with Spatial-temporal Consistency, \
[c] TC4D: Trajectory-Conditioned Text-to-4D Generation.
questions: questions: 1. The results of 4D-fy appear strange. When I check the 4D-fy results in TC4D and 4DGen, they look more reasonable.

2. I think the authors should also conduct ablation studies on MV2V-Adapter.
limitations: limitations: The authors have discussed the limitations and potential negative societal impact.
paper_summary: paper_summary: This paper proposes Animate3D, a 4D generation framework that consists of a multi-view video diffusion model (MV-VDM) followed by 4D Gaussian Splatting optimization, as well as a dataset of 38K animated 3D objects (MV-Video) that is used to provide spatiotemporal supervision to train the model. Different from text-to-4D approaches, Animate3D specifically tackles the problem of animating 3D assets which requires spatiotemporal consistency as well as identity preservation. The proposed architecture is based on a multi-view diffusion model (MVDream) and a video diffusion model (AnimateDiff) with additional spatio-temporal attention blocks. To animate a given 3D asset, the trained model is used for both reconstruction and distillation. First, a multi-view video is generated conditioned on multiple views of the asset. The video is used to optimize a 4D Gaussian Splatting resulting in a model with coarse motion. The same MV-VDM model is used for 4D Score Distillation Sampling to refine the motion. The method compares favorably to state-of-the-art 4D generation methods.
strength_weakness: strength_weakness: **[W1]** The data used in this paper are animated 3D models collected from Sketchfab (cited). The authors promise to distribute the collected dataset. However, there is not sufficient information provided about copyrights and licensing. The only piece of information provided is that models with the following clause are excluded: 

>NoAI: This model may not be used in datasets for, in the development of, or as inputs to generative AI programs.

Based on the paper checklist, the paper is flagged for ethics review.  

---

**[W2]** Lack of clarity in the method section and method figure

**(a)** The method section is very hard to parse. The notation is overloaded but not very well explained. The section would benefit from thorough revision to improve the overall clarity and quality. 

**(b)** It is almost easier to understand the method through Fig. 2, but even that is dense and inconsistent, and often the notation does not align with what is written in Sec. 3.1. For example, in L175 latent features $z$ seem to be the output of the image encoder. In Fig. 2 (middle), $z$ appears to be the input to the spatiotemporal attention module. In Eq. 1, $X_l$ and $X_r$ are instead the inputs to the same module. At the same time, in Fig. 2 (left) the input to the spatiotemporal module seems to be the sum of the output of the cross-attention layers, not directly $z$. 

This makes reading and understanding the method extremely difficult to the point where the reviewer cannot fully judge the technical contribution and similarities or differences to related models. 

---

**[W3]** Discussion with respect to 4D generation methods could be expanded

Existing work in text-to-4D generation often splits the problem into two stages: a static 3D asset generation and a deformation field to model motion. This disentanglement makes it possible to learn the 3D motion from video diffusion models, without the need for multi-view video. The paper briefly touches upon this and compares to two 4D generation methods, 4Dfy and DreamGaussian4D. However, the paper could further elaborate on the similarities and differences of the proposed approach compared to existing methods and the advantages of a multi-view video diffusion model. 

---

**[W4]** Certain architectural components are not ablated and their contribution to the overall performance is unclear. See Questions. 

---

**[W5]** The temporal extent of MV-VDM is only 8 frames, which seems rather limited.

---

**[W6]** In some of the qualitative examples, the faithfulness to the text prompt is questionable. Some of the animations provided in the supplementary video are not accompanied by a text prompt, while quite a few animations appear very similar. 

---

**[Minor]** Typos and suggestions:

- L103: pioneer → pioneering
- L113: All these manners aforementioned → All aforementioned work
- L131-132: this sentence can be improved for clarity since the two parts sound a bit repetitive
- L135: spatial → spatially; temporal → temporally 
- L156: It should be better explained how what $X_l$ and $X_r$ represent and how these features are obtained.  
- L292: levering → leveraging; manage → manages

---
questions: questions: **Q1. [Based on W1]** Please elaborate on the difference between _3D animation via multi-view video diffusion_ and _two-stage text-to-4D approaches_. In the latter case, the second stage could be viewed as 3D animation. I understand that a multi-view video foundation model would offer spatiotemporal consistency that other approaches can likely not reach, but I would appreciate a more in-depth discussion about the difference to and the challenges of text-to-3D-and-3D-to-4D approaches than what is currently provided in the paper. 

---

**Q2. [Based on W1]** Ideally, the authors should provide also empirical comparisons to additional methods (e.g., Dream-in-4D), if possible.   

---

**Q3. [Based on W4]** The authors state that
>We find this simple cross-attention operator [MV2V-Adapter] can effectively improve the object’s appearance consistency in the generated video.

The effect of MV2V-Adapter should be better demonstrated with an ablation. 

---

**Q4. [Based on W4]** Are the cross-attention layers described in L170-172 necessary for identity preservation and alignment with the text prompt? Their effect should be also demonstrated in the ablation studies. 

---

**Q5.** The proposed 3D animation approach uses both reconstruction and distillation. It first uses the generated multi-view video to optimize a 4D Gaussian Splatting with coarse motion. Then SDS is used to refine the motion. It seems counter-intuitive that SDS would be good at modeling the finer details of the animation, since SDS is typically known for its over-smoothing behavior. The authors should further discuss the motivation behind using SDS during 4D optimization and provide additional examples to prove its effectiveness. 

---

**Q6. [Based on W6]** Could the authors elaborate on the apparent lack of diversity in the generated motions? It may be helpful to provide some statistics from motion descriptions included in the dataset. Or is this a limitation of the model instead?

---
limitations: limitations: Limitations are adequately discussed in Appendix B.
paper_summary: paper_summary: This paper introduces a framework for 4D generation to animate static 3D models, consisting of two components: MV-VDM, a multi-view video generation model, and a framework combining reconstruction and 4D score distillation sampling (4D-SDS). A spatiotemporal attention module enhances consistency, using multi-view renderings to preserve the model's identity. The proposed two-stage pipeline for animating 3D models first reconstructs coarse motions and then refines them with 4D-SDS. Experiments show that Animate3D outperforms previous methods.
strength_weakness: strength_weakness: 1. The method appears to be a straightforward engineering pipeline, combining several existing techniques (e.g., MVDream and IP-Adapter in the MV-VDM stage, reconstruction + SDS in the animating stage) without significant innovation.
2. Motion diversity is a concern. The method shifts the core problem to the quality of the multi-view video. The dataset achieves this effect, but the maximum number of animations per ID is only six, which is insufficient for capturing the diverse motions of a single 3D object. This approach lacks generality.
3. Motion controllability is limited. The results shown in the paper exhibit simple, repetitive motions with small amplitudes. Although text prompts are used as conditions, they are relatively simple, and the generated motions are not complex. The method lacks a clear approach to achieving more controllable motions.
4. Experimental results are lacking. DreamGaussian4D provides CLIP-I as a quantitative comparison to assess the consistency between prompts and generated results, which is missing in the paper. Additionally, the qualitative results provided in terms of views and moments are too few.
questions: questions: 1. Regarding time, the paper reports an optimization time of 30 minutes per object on a single A800 GPU. How much time does it take to reconstruct coarse motions and refine motions using the 4D-SDS model, respectively?
limitations: limitations: The authors have discussed the limitations and broader impacts of the study in the paper.

Comments: title: Response to rebuttal
Post Rebuttal
In-depth Discussion with respect to Previous 4D generation Methods
comment: Thanks for your response. It has addressed my concerns. I will keep my score.
Thanks for the response. We are carefully revising our manuscript, including figures, tables, text and demo video. The revision will feature significant updates.
Thank you for the detailed rebuttal. Most of my concerns have been addressed and I intend to keep a positive rating. I appreciate the extended discussion regarding existing methods and I would suggest integrating it into the main paper or the appendix. 

I also appreciate the authors' efforts in providing additional experiments and comparisons during the rebuttal phase. 

One concern that remains is still about the writing and presentation, which I hope the authors will improve substantially in their revision.
Thank the authors for addressing my questions. I am still advocating for accepting the paper.
Due to limited space, our initial response did not permit a comprehensive discussion of previous 4D generation works, now we detail the in-depth discussion as suggested: 

Previous two-stage 4D generation works attempted to **disentangle motion learning** from appearance learning by adopting **different types of supervision signal**, i.e., video diffusion/monocular video for motion and image/3D diffusion for appearance. However, the motion and appearance supervisions adopted in their work are **not orthogonal**, and sometimes have **negative effect** on each other. 

For example, it is commonly agreed that video-diffusion-SDS usually brings unappealing visual effect to the appearance of the object [Animate124, Dream-in-4D, AYG]. Meanwhile, the appearance supervision signal prevents 4D object from updating along the direction that follows the exact score function of the video diffusion model, leading to less natural motion. Small motion amplitude in [4Dfy, Dream-in-4D] and shaky appearance in [AYG] could partly support this point. As for monocular-video-guided motion learning, previous work [DG4D, 4DGen, STAG4D] relies on 3D diffusion model (Zero123) to supervise both motion and appearance in novel view. Since Zero123-SDS is applied per-frame, temporal consistency in novel view cannot be guaranteed. Moreover, monocular video doesn't provides information about depth/distance, so moving closer to or farther away from the camera can be perceived as the magnification or reduction of the object, resulting in appearance distortion.

In contrast, our method takes the **unified** supervision signal from MV-VDM for motion learning and appearance preservation. Our motion and appearance supervision signal inherently don't conflict with each other, since MV-VDM is conditioned on multi-view attributes of the 3D object to generate multi-view videos.
Besides, **multi-view motion supervision** in our work enables more natural motion generation when compared with single-view motion supervision in other works. Thus, we achieve superior performance in terms of both motion generation and appearance preservation in the task of animating any off-the-shelf 3D object.

Thanks for your insightful suggestion of in-depth discussion, and we'll add this in revision. If you have any further questions, discussions are welcomed.
Thank the authors for addressing my questions. I plan to keep my positive score.
Thanks for the efforts of the authors. They have conducted additional experiments to support their claims, and my concerns have been resolved. I will maintain my scores.