AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning
Authors: Yuwei Guo
Ceyuan Yang
Anyi Rao
Zhengyang Liang
Yaohui Wang
Yu Qiao
Maneesh Agrawala
Dahua Lin
Bo Dai
Venue: ICLR 2024 spotlight (2024)
Keywords: Deep Learning
Diffusion Model
Video Generation
Primary area: generative models
TL;DR: In this paper, we present AnimateDiff, a practical framework for animating personalized text-to-image diffusion models without requiring model-specific tuning.

Abstract: With the advance of text-to-image (T2I) diffusion models (e.g., Stable Diffusion) and corresponding personalization techniques such as DreamBooth and LoRA, everyone can manifest their imagination into high-quality images at an affordable cost. However, adding motion dynamics to existing high-quality personalized T2Is and enabling them to generate animations remains an open challenge. In this paper, we present AnimateDiff, a practical framework for animating personalized T2I models without requiring model-specific tuning. At the core of our framework is a plug-and-play motion module that can be trained once and seamlessly integrated into any personalized T2Is originating from the same base T2I. Through our proposed training strategy, the motion module effectively learns transferable motion priors from real-world videos. Once trained, the motion module can be inserted into a personalized T2I model to form a personalized animation generator. We further propose MotionLoRA, a lightweight fine-tuning technique for AnimateDiff that enables a pre-trained motion module to adapt to new motion patterns, such as different shot types, at a low training and data collection cost. We evaluate AnimateDiff and MotionLoRA on several public representative personalized T2I models collected from the community. The results demonstrate that our approaches help these models generate temporally smooth animation clips while preserving the visual quality and motion diversity. Codes and pre-trained weights are available at https://github.com/guoyww/AnimateDiff.

Published as a conference paper at ICLR 2024
ANIMATEDIFF: ANIMATE YOUR PERSONALIZED TEXT-TO-IMAGE DIFFUSION MODELS WITHOUT
SPECIFIC TUNING
Yuwei Guo Ceyuan Yangat Anyi Rao Zhengyang Liang" Yaohui Wang"
Yu Qiao² Maneesh Agrawala²
Dahua Lin Bo Dai
The Chinese University of Hong Kong SShanghai Artificial Intelligence Laboratory
Stanford University
(cartoon) Iboy, dark skin, play- ing guitar. concert,
(oil painting) black pearl pirate ship, night time, ...
(realistic) Lamborghini on road, fireworks, high detail.
zoom-in
rolling
z00m-out + rolling
right + up
Figure 1: AnimateDiff directly turns existing personalized text-to-image (T2I) models to the corre- sponding animation generators with pre-trained motion module. First row: results by combining AnimateDiff with three personalized T2Is in different domains; Second row: results of further com- bining AnimateDiff with MotionLoRA (s) to achieve shot type controls. Best viewed with Acrobat Reader. Click the images to play the animation clips.
ABSTRACT
With the advance of text-to-image (T2I) diffusion models (e.g., Stable Diffusson) and corresponding personalization techniques such DreamBooth and LoRA, everyone can manifest their imagination into high-quality images at an afford- able cost. However, adding motion dynamics existing high-quality personal- ized T2Is and enabling them to generate animations remains an open challenge. In this paper, we present AnimateDiff, a practical framework for animating per- sonalized T2I models without requiring model-specific tuning. At the core of our framework is plug-and-play motion module that can be trained once and seamlessly integrated into any personalized T2Is originating from the same base T2I. Through our proposed training strategy, the motion module effectively learns transferable motion priors from real-world videos. Once trained, the motion mod- ule can be inserted into personalized T2I model to form personalized animation
Corresponding Author.

---

Published as a conference paper at ICLR 2024
generator. We further propose MotionLoRA, a lightweight fine-tuning technique for AnimateDiff that enables pre-trained motion module to adapt to new motion patterns, such as different shot types, at low training and data collection cost. We evaluate AnimateDiff and MotionL.oRA on several public representative personal- ized T21 models collected from the community. The results demonstrate that our approaches help these models generate temporally smooth animation clips while preserving the visual quality and motion diversity. Codes and pre-trained weights are available at https: / / thub. com/ guo / An i ma t teDiff.
INTRODUCTION
text-to-image (T2I) diffusion models (Nichol et al., 2021; Ramesh et al., 2022; Saharia et al., 2022; Rombach et al., 2022) have greatly empowered artists and amateurs to create visual content using text prompts. To further stimulate the creativity of existing T2I models, lightweight personalization methods, such as DreamBooth (Ruiz et al., 2023) and LoRA (Hu et al., 2021) have been proposed. These methods enable customized fine-tuning on small datasets using consumer-grade hardware such as a laptop with an RTX3080, thereby allowing users to adapt a base T2I model to new do- mains and improve visual quality relatively low cost. Consequently, a large community of Al artists and amateurs has contributed numerous personalized models on model-sharing platforms such as Civitai (2022) and Hugging Face (2022). While these personalized T2I models can generate remarkable visual quality, their outputs are limited to static images. On the other hand, the ability to generate animations more desirable real-world production, such as the movie and cartoon industries. In this work, we aim to directly transform existing high-qualit personalized T2I models into animation generators without requiring model-specific fine-tuning, which is often impractical in terms of computation and data collection costs for amateur users.
We present AnimateDiff, an effective pipeline for addressing the problem of animating personalized T2Is while preserving their visual quality and domain knowledge. The core of AnimateDiff is an approach for training ald-paay motion module that learns reasonable motion priors from video datasets, such WebVid-I0M (Bain al., 2021). At inference time, the trained motion module can be directly integrated into personalized T2Is and produce smooth and visually appealing animations without requiring specific tuning. The training of the motion module in AnimateDiff consists of three stages. Firstly, we fine-tune a domain adapter on the base T2I to align with the visual distribution of the target video dataset. This preliminary step guarantees the motion module concentrates on learning the motion priors rather than pixel-level details from the training videos. Secondly, we inflate the base T2I together with the domain adapter and introduce newly initialized motion module for motion modeling. We then optimize this module on videos while keeping the domain adapter and base T2I weights fixed. By doing so, the motion module learns generalized motion priors and can, via module insertion, enable other personalized T2Is to generate smooth and appealing animations aligned with their personalized domains. The third stage of AnimateDiff, also dubbed Loti aims adapt the pre-trained motion module to specific motion patterns with a small number of reference videos and training iterations. We achieve this by fine-tuning the motion module with the aid of Low-Rank Adaptation (LoRA) (Hu et al., 2021). Remarkably, adapting to new motion pattern can be achieved with as few as 50 reference videos. Moreover, a MotionLoRA model requires only approximately 30M of additional storage space, further enhancing the efficiency of model sharing. This efficiency is particularly valuable for users who are unable to bear the expensive costs of pre-training but desire to fine-tune the motion module for specific effects.
We evaluate the performance of AnimateDiff and MotionLoRA on diverse set of personalized T2I models collected from model-sharing platforms (Civitai, 2022; Hugging Face, 2022). These models encompass a wide spectrum of domains, ranging from 2D cartoons to realistic photographs, thereby forming a comprehensive benchmark for our evaluation. The results of our experiments demon- strate promising outcomes. In practice, we also found that Transformer (Vaswani et al., 2017) architecture along the temporal axis is adequate for capturing appropriate motion priors. We also demonstrate that our motion module can be seamlessly integrated with existing content-controlling approuches (Zhang et al., 2023; Mou et al., 2023) such as ControlNet without requiring additional training, enabling AnimateDiff for controllable animation generation.
2

---

Published as a conference paper at ICLR 2024
In summary, (1) we present AnimateDiff, practical pipeline that enables the animation generation ability of any personalized T2Is without specific fine-tuning; (2) we verify that a Transformer archi- tecture is adequate for modeling motion priors, which provides valuable insights for video genera- tion; (3) we propose MotionI LoRA, a lightweight fine-tuning technique to adapt pre-trained motion modules new motion patterns; (4) we comprehensively evaluate our approach with representative community models and compare it with both academic baselines and commercial tools such as Gen- 2 (2023) and Pika Labs (2023). Furthermore, we showcase its compatibility with existing works for controllable generation.
2 RELATED WORK
Text-fo-image diffusion models. Diffusion models (Ho et al., 2020; Dhariwal & Nichol, 2021; Song et al., 2020) for text-to-image (T2I) generation (Gu et al., 2022; Mokady et al., 2023; Podell et al., 2023; Ding et al., 2021; Zhou et al., 2022b; Ramesh et al., 2021; Li et al., 2022) have gained significant attention in both academic and non-academic communities recently. GLIDE (Nichol et al., 2021) introduced text conditions and demonstrated that incorporating classifier guidance leads to more pleasing results. DALL-E2 (Ramesh et al., 2022) improves text-image alignment by lever- aging the CLIP (Radford et al., 2021) joint feature space. Imagen (Saharia et al., 2022) incorporates a large language model (Raffel et al., 2020) and a cascade architecture to achieve photorealistic re- sults. Latent Diffusion Model (Rombach et al., 2022), also known Stable Diffusion, moves the diffusion process to the latent space of an auto-encoder to enhance efficiency. eDiff-I (Balaji et al., 2022) employs an ensemble of diffusion models specialized for different generation stages.
Personalizing T2I models. To facilitate the creation with pre-trained T21s, many works focus on efficient model personalization (Shi et al., 2023; Lu et al., 2023; Dong et al., 2022; Kumari al., 2023), i.e., introducing concepts or styles to the base T2I using reference images. The most straight- forward approach to achieve this is complete fine-tuning of the model. Despite its potential to significantly enhance overall quality, this practice can lead to catastrophic forgetting (Kirkpatrick al., 2017; French, 1999) when the reference image set small. Instead, DreamBooth (Ruiz al., 2023) fine-tunes the entire network with preservation loss and uses only few images. Textual In- version (Gal et al., 2022) optimize a token embedding for each new concept. Low-Rank Adaptation (LoRA) (Hu et al., 2021) facilitates the above fine-tuning process by introducing additional LoRA layers to the base T2I and optimizing only the weight residuals. There are also encoder- ap- proaches that address the personalization problem (Gal et al., 2023; Jia et al., 2023). In our work, we focus on tuning-based methods, including overall fine-tuning, DreamBooth (Ruiz et al., 2023), and LoRA (Hu et al., 2021), as they preserve the original feature space of the base T2I.
Animating personalized T21s. There are not many existing works regarding animating personal- ized T21s. Text2Cinemagraph (Mahapatra al., 2023) proposed to generate cinematography via flow prediction. In the field of video generation, it is common to extend a pre-trained T2I with temporal structures. Existing works (Esser et al., 2023; Zhou et al., 2022a: Singer et al., 2022: Ho et al., 2022b,a; Ruan et al., 2023; Luo et al., 2023; Yin et al., 2023b,a; Wang et al., 2023b; Hong et al., 2022; Luo et al., 2023) mostly update all parameters and modify the feature space of the original T2I and is not compatible with personalized ones. Align- Your-Latents (Blattmann et al., 2023) shows that the frozen image layers in a general video generator can be personalized. Re- cently, some video generation approaches have shown promising results in animating personalized T21 model. Tune-a- (Wu et al., 2023) fine-tune small number of parameters on a single video. Video- (Khachatryan al., 2023) introduces training-free method to animate pre-trained T2I via latent wrapping based on a pre-defined affine matrix.
3 PRELIMINARY
We introduce the preliminary of Stable Diffusson (Rombach et al., 2022), the base T2I model used in work, and Low-Rand Adaptation (LoRA) (Hu et al., 2021), which helps understand the domain adapter (Sec. 4.1) and MotionLoRA (Sec. 4.3) in AnimateDiff.
Stable Diffusion. We chose Stable Diffusion (SD) the base T2I model in this paper since it is open-sourced and has a well-developed community with many high-quality personalized T2I models for evaluation. SD performs the diffusion process within the latent space of pre-trained autoen-
3

---

Published as a conference paper at ICLR 2024
coder E() and D(-). In training, encoded image 2u (xr) is perturbed to 2 by the forword diffusion:
Va + vI- e 1),
(1)
for t 1,...,T, T, where pre-defined de determines the noise strength at step t. The denoising network co(-) learns to reverse this process by predicting the added noise, encouraged by an MSE loss:
C = (. [lle [16--c0(=,44
where y is the text prompt corresponding to Fo: To(-) is a text encoder mapping the prompt to a vector sequence. In SD, co(-) is implemented as UNet (Ronneberger et al., 2015) consisting of pairs of down/up sample blocks at four resolution levels, as well as a middle block. Each network block consists of ResNet (He al., 2016), spatial self-attention layers, and cross-attention layers that introduce text conditions.
Low-rank adaptation (LoRA).. LoRA (Hu et al., 2021) is approach that accelerates the fine- tuning of large models and is first proposed for language model adaptation. Instead of retraining all model parameters, LoRA adds pairs of rank-decomposition matrices and optimizes only these newly introduced weights. By limiting the trainable parameters and keeping the original weights frozen, LoRA less likely to cause catastrophic forgetting (Kirkpatrick al., 2017). Concretely, the aankdecomposition matrices serve as the residual of the pre-trained model weights W e Rmmx.. The new model weight with LoRA is
W = W AW = W + AB".
where A E Ruxr, B E RR are pair of cankdecompooition matrices, is hyper-parameter, which is referred to as the rank LoRA layers. In practice, LoRA only applied to attention layers, further reducing the cost and storage for model fine-tuning.
ANIMATEDIFF
The core of our method is learning transferable mo- tion priors from video data, which be applied to personalized T2Is without specific tuning. As shown Fig. 2, inference time, our motion module (blue) and the optional MotionLoRA (green) can be directly inserted into a personalized T2I to constitute the an- imation generator, which subsequently generates ani- mations via an iterative denoising process.
Motion Modules of AnimateDiff
Training Pipeline
Module Insert
M Denoise
We achieve this by training three components of Ani- mifeDiff,, namely domain adapter, motion module, and MotionLoRA. The domain adapter in Sec. 4.1 is only used in the training to alleviate the negative effects caused by the visual distribution gap between the base T2I pre-training data and our video training data; the motion module in Sec. 4.2 is for learning the motion priors; and the MotionL LoRA in Sec. 4.3, which is optional in the case of general animation, is for adapting pre-trained motion modules new motion patterns. Sec.4.4 elaborates the training (Fig. 3) and inference of AnimateDiff.
21
2
Image Layers of Personalized
Figure 2: Inference pipeline.
4.1 ALLEVIATE NEGATIVE EFFECTS FROM TRAINING DATA WITH DOMAIN ADAPTER
Due to the difficulty in collection, the visual quality of publicly available video training datasets much lower than their image counterparts. For example, the contents of the video dataset We- bVid (Bain et al., 2021) are mostly real-world recordings, whereas the image dataset LAION- Aesthetic (Schuhmann et al., 2022) contains higher-quality contents, including artistic paintings and professional photography. Moreover, when treated individually as images, each video frame can contain motion blur, compression artifacts, and watermarks. Therefore, there is a non-negligible quality domain gap between the high-quality image dataset used train the base T21 and the target
4

---

Published as a conference paper at ICLR 2024
1. Alleviate Negative Effects
2. Learn Motion Priors
3. (optional) Adapt to New Patterns
spromptss
sprompts
cprompts
V
Sampled Frames
Dataset
Video
20-50 Ref. Videos
Precrained
image Layers
Cioss-Atteziion
Motion Module (Temporal Transformer)
MSE Loss
Pretroined image Layers (frozen)
RoiNet Block
>
W Arlapten(s)
Proj. In
Self- Attention
Proj. Out (zero initializze)
Domain Adapter (trainable
Motion Module
(erainable (trainable stage
W Adapter(s)
Position Ere.
xN
MationLoRA
Figure 3: Training pipeline of AnimateDifT. AnimateDiff consists of three training stages for the corresponding component modules. Firstly, a domain adapter (Sec. 4.1) is trained to alleviate the negative effects caused by training videos. Secondly, motion module (Sec. 4.2) is inserted and trained on videos to learn general motion priors. Lastly, MotionLoRA (Sec. 4.3) is trained on a few reference videos adapt the pre-trained motion module to new motion patterns.
video dataset we use for learning the motion priors. We argue that such a gap can limit the quality of the animation generation pipeline when trained directly on the raw video data.
To avoid learning this quality discrepancy as part of motion module and preserve the knowledge of the base T2I, we propose to fit the domain information to separate network, dubbed as domain adapter. We drop the domain adapter at inference time and show that this practice helps reduce the negative effects caused by the domain gap mentioned above. We implement the domain adapter layers with LoRA (Hu et al., 2021) and insert them into the self-/crros--ttennion layers in the base T2I, as shown in Fig. 3. Take query (Q) projection as an example. The internal feature after projection becomes
Q 2º 2 AdapterLayer(2) W AB
(4)
where a = is a scalar and can be adjusted to other values at inference time (set to 0 to remove the effects of domain adapter totally). We then optimize only the parameters of the domain adapter on static frames randomly sampled from video datasets with the same objective in Eq. (2).
4.2 LEARN MOTION PRIORS WITH MOTION MODULE
To model motion dynamics along the temporal dimension top of pre-trained T2I. must 1) inflate the 2-dimensional diffusion model to deal with 3-dimensional video data and 2) design a sub-module to enable efficient information exchange along the temporal axis.
Network Inflation. The pre-trained image layers in the base T2I model capture high-quality con- tent priors. To utilize the knowledge, preferable way for network inflation is to let these image layers independently deal with video frames. To achieve this, we adopt a practice similar to recent works (Ho al., 2022b; Wu et al., 2023; Blattmann et al., 2023), and modify the model that it takes 5D video tensors I E ppxexxxxxxx as input, where and f represent batch axis and frame- time axis respectively. When the internal feature maps go through image layers, the temporal axis is ignored by being reshaped into the b axis, allowing the network to process each frame inde- pendently. We then reshape the feature map to the 5D tensor after the image layer. On the other hand, our newly inserted motion module ignores the spatial axis by reshaping h, w into b and then reshaping back after the module.
Module Design. Recent works on video generation have explored many designs for temporal mod- eling. In DifmateDiff, we adopt the Transformer (Vaswani et al., 2017) architecture as our motion module design, and make minor modifications adapt it to operate along the temporal axis, which we refer to as "temporal Transformer" in the following sections. We experimentally found this design is adequate for modeling motion priors. As illustrated in Fig. 3, the temporal Transformer consists of several self-attention blocks along the temporal axis, with sinusoidal position encoding encode the location of each frame in the animation. As mentioned above, the input of the motion module is the reshaped feature map whose spatial dimensions are merged into the batch axis. When we divide the reshaped feature map along the temporal axis, it can be regarded as vector sequences with length of f, Le., {=1,-., Zy; 2i E T (bxhxxx) The vectors will then be projected and go
5

---

Published as a conference paper at ICLR 2024
through several self-attentio blocks, i.e.
(5) where 2, K W and V = WV are three separated projections. The attention mech- anism enables the generation of the current frame to incorporate information from other frames. As result, instead of generating each frame individually, the T2I model inflated with our motion module learns to capture the changes of visual content over time, which constitute the motion dy- namics in animation clip. Note that sinusoidal position encoding added before the self-attention is essential; otherwise, the module is not aware of the frame order in the animation. To avoid any harmful effects that the additional module might introduce, we zero initialize (Zhang et al., 2023) the output projection layers of the temporal Transformer and add a residual connection so that the motion module is an identity mapping the beginning of training.
Zout Attention (Q, K.KV V) Softmax(KK/VVO)
4.3 ADAPT TO NEW MOTION PATTERNS WITH MOTIONLORA
While the pre-trained motion module captures general motion priors, a question arises when we need to effectively adapt it to new motion patterns such as camera zooming, panning and rolling, etc., with a small number of reference videos and training iterations. Such efficiency is essential for users who cannot afford expensive pre-training costs but would like to fine-tune the motion module for specific effects. Here comes the last stage of AnimateDiff, also dubbed as MotionLoRA (Fig. 3), an efficient fine-tuning approach for motion personalization. Considering the architecture of the motion module and the limited number of reference videos, add LoRA layers to the self-attention layers of the motion module in the inflated model described in Sec. 4.2, then train these LoRA layers on the reference videos of new motion patterns.
We experiment with several shot types and get the reference videos via rule-based data augmenta- tion. For instance, to get videos with zooming effects, we augment the videos by gradually reducing (zoom-in) or enlarging (zoom-out) the cropping area of video frames along the temporal axis. We demonstrate that our MotionLoRA can achieve promising results even with few 20 50 ref- erence vidcos, 2,000 training iterations (around 1 ~ 2 hours) as well as about 30M storage space, enabling efficient model tuning and sharing among users. Benefited by the low-rank property, Mo- tionLoRA also has the composition capability. Namely, individually trained MotionLoRA models can be combined to achieve composed motion effects at inference time.
4.4 ANIMATEDIFF IN PRACTICE
Training. As illustrated in Fig. 3, AnimateDiff consists of three trainable component modules to learn transferable motion priors. Their training objectives are slightly different. The domain adapter is trained with the original objective as in Eq. (2). The motion module and MotionLoRA, as part of an animation generator, use a similar objective with minor modifications to accommodate higher dimension video data. Concretely, a video data batch 1:f E ppxxxx/xxxx is first encoded into the latent codes =0" 1:ff frame-wisely via the pre-trained auto-encoder of SD. The latent codes are then noised using the defined forward diffusion schedule as Eq. (1)
1:f Va
1:f + -I- -.
(6)
The inflated model inputs the noised latent codes and corresponding text prompts and predicts the added noises. The final training objective of our motion modeling module is:
L E Eg(o 1..
[ll--c(=]** co(=) 1:f ,t,t,,,, t, (o))))
It's worth noting that when training the domain adapter, the motion module, and the MotionL LoRA, parameters outside the trainable part remain frozen.
Inference. At inference time (Fig. 2). the personalized T21 model will first be inflated in the same way discussed in Section 4.2, then injected with the motion module for general animation genera- tion, and the optional MotionLoRA A for generating animation with personalized motion. As for the domain adapter, instead of simply dropping it during the inference time, in practice, we can also in- ject it into the personalized T2I model and adjust its contribution by changing the scaler in Eq. (4). An ablation study on the value of a is conducted in experiments. Finally, the animation frames can be obtained by performing the reverse diffusion process and decoding the latent codes.

---

Published as a conference paper at ICLR 2024
RCNZ Cartoon 3d
TUSUN
epiC Realism
Toon You
golden Labrador, nat- ural lighting,
cute Pallas's Cat walk- ing in the snow,
photo of 24 Noo woman, night street,
coastline, lighthouse, waves, sunlight. Oil painting
MeinaMix
Realistic Vision
MoXin
Igirl, white hair, purple eyes, dress, petals. ...
a cyberpunk city street, night time,
a bird sits on a branch, ink painting. ...
sunset, orange sky, fish- ing boats, waves,
Figure 4: Qualitative Result. Each sample corresponds distinct personalized T2I. Best viewed with Acrobat Reader. Click the images to play the animation clips.
EXPERIMENTS
We implement AnimateDiff upon Stable Diffusion V1.5 and train motion module using the Web Vid- 10M (Bain et al., 2021) dataset. Detailed configurations can be found in supplementary materials.
5.1
QUALITATIVE RESULTS
Evaluate on community models. We evaluated the AnimateDiff with diverse set representative personalized T2Is collected from Civitai (2022). These personalized T2Is encompass wide range of domains, thus serving as a comprehensive benchmark. Since personalized domains in these T2Is only respond to certain "trigger words", abstain from using common text prompts but refer the model homepage to construct the evaluation prompts. In Fig. 4, we show eight qualitative results of AnimateDiff. Each sample corresponds to distinct personalized T2I. the second row of Figure 1, we present the outcomes obtained by integrating AnimateDiff with MotionLoRA to achieve shot type controls. The last two samples exhibit the composition capability of MotionLoRA, achieved by linearly combining the individually trained weights.
Compare with baselines. In the absence of existing methods specifically designed for animating personalized T2Is, we compare our method with two recent works in video generation that can be adapted for this task: 1) Text2Video-Zero (Khachatryan et al., 2023). We also compare AnimateDiff with two commercial tools: 3) Gen-2 (2023) for text- to-video generation, and 4) Pika Labs (2023) for image animation. The results are shown in Fig. 5.
al., 2023) and 2) Tune-a-Vides Video (Wu
5.2 QUANTITATIVE COMPARISON
We conduct the guantitative comparison through user study and CLIP metrics. The comparison focuses on three key aspects: text alignment, domain similarity, and motion smoothness. The results shown in Table 1. Detailed implementations can be found in supplementary materials.
User study, In the user study, we generate animations using all three methods based on the same personalized T21 models. Participants are then asked to individually rank the results based the

---

Published as a conference paper at ICLR 2024
Tune-A A-Video
AnimateDiff
T2V-Zero
Animate Diff
raccoon is playing guitar, soft lighting.... ...
horse galloping the street, best quality. ...
Figure 5: Qualitative Comparison. Best viewed with Acrobat Reader. Click the images to play the
animation clips.
Table 1: Quantitative comparison. A higher score indicates superior performance.
User Study (f)
CLIP Metric (f)
Method
Text.
Domain.
Smooth.
Text.
Domain.
Smooth.
Video-zdoo
1.620
2.620
1.560
32.04
84.84
96.57
Tune-a- Video
2.180
1.100
1.615
35.98
80.68
97.42
Ours
2.210
2.280
2.825
31.39
87.29
98.00
above three aspects. We use the Average User Ranking (AUR) as preference metric where a higher score indicates superior performance. Note that the corresponding prompts and images are provided for reference for text alignment and domain similarity evaluation.
CLIP metric. We also employed the CLIP (Radford et al., 2021) metric, following the approach taken by previous studies (Wu et al., 2023; Khachatryan et al., 2023). When evaluating domain similarity, it is important to note that the CLIP score was computed between the animation frames and the reference images generated using the personalized T21s.
5.3 ABLATIVE STUDY
Domain adapter. To investigate the impact of the domain adapter in AnimateDiff, we conducted study by adjusting the scaler in the adapter layers during inference, ranging from 1 (full impact) to O (complete removal). As illustrated in Figure 6, as the scaler of the adapter decreases, there is an improvement in overall visual quality, accompanied by reduction in the visual content distri- bution learned from the video dataset (the watermark in the case of Web Vid (Bain et al., 2021))- These results indicate the successful role of the domain adapter in enhancing the visual quality of DnimateDiff by alleviating the motion module from learning the visual distribution gap.
Motion module design. We compare our motion module design of the temporal Transformer with its full convolution counterpart, which is motivated by the fact that both designs are widely em- ployed in recent works on video generation. We replace the temporal attention with ID temporal convolution and ensured that the two model parameters were closely aligned. As depicted in sup- plementary materials, the convolutional motion module aligns all frames be identical but does not incorporate any motion compared to the Transformer architecture.
Efficiency of Mot.onLoRA. The efficiency of MotionLoRA in AnimateDiff was examined in terms of parameter efficiency and daia efficiency. Parameter efficiency is crucial for efficient model train- ing and sharing among users, while data efficiency is essential for real-world applications where collecting adequate number of reference videos for specific motion patterns may be challenging.
To investigate these aspects, we trained multiple MotionLoRA models with varying parameter scales and reference video quantities. In Fig. 7, the first two samples demonstrate that MotionLoRA is ca- pable of learning new camera motions (e.g., zoom-in) with a small parameter scale while maintain- ing comparable motion quality. Furthermore, even with modest number of reference videos (e.g., N = 50), the model successfully learns the desired motion patterns. However, when the number of reference videos is excessively limited (e.g., N 5), significant degradation in quality is observed,
8

---

Published as a conference paper at ICLR 2024
scaler 1.0
scaler
Figure 6: Ablation domain adapter. We adjust the scaler of the adapter from 1 0 to gradually remove its effects. In this figure, we show the first frame of the generated animation.
mak=2 (~1
rank=128 (~36M)
N=5
N=50
N= 1000
Figure 7: Ablation on NotionLoRA's efficiency. Two samples the left: with different network rank; Three samples on the right: with different numbers of reference videos. Best viewed with Acrobat Reader. Click the images play the animation clips.
suggesting that MotionLoRA encounters difficulties in learning shared motion patterns and instead relies on capturing texture information from the reference videos.
5.4 CONTROLLABLE GENERATION.
The separated learning of visual content and motion priors in AnimateDiff enables the di- rect application of existing content control approaches for controllable generation. To demonstrate this capability, we combined An- imateDiff with ControlNet (Zhang et al., 2023) to control the generation with extracted depth map sequence. In contrast to recent video edit- ing techniques (Ceylan et al., 2023; Wang et al., 2023a) that employ DDIM (Song al., 2020) inversion to obtain smoothed latent sequences, generate animations from randomly sam- pled noise. As illustrated in Figure 8, our re-
city street, neon, fog, closeup portrait photo of young woman dark clothes...
Figure 8: Controllable generation. Best viewed with Acrobat Reader. Click the images to play the animation clips.
sults exhibit meticulous motion details (such hair and facial expressions) and high visual quality.
6 CONCLUSION
In this paper, present AnimateDiff, a practical pipeline directly turning personalized text-to- image (T2I) models for animation generation once and for all, without compromising quality or losing pre-learned domain knowledge. To accomplish this, we design three component modules in AnimateDiff to learn meaningful motion priors while alleviating visual quality degradation and en- abling motion personalization with lightweight fine-tuning technique named MotionL.RRA. Once trained, our motion module can be integrated into other personalized T21s to generate animated images with natural and coherent motions while remaining faithful to the personalized domain. Ex- tensive evaluation with various personalized T2I models also validates the effectiveness and gener- alizability our AnimateDiff and Moti Furthermore, we demonstrate the compatibility of method with existing content-controllin approaches, enabling controllable generation without incurring additional training costs. Overall, AnimateDiff provides an effective baseline for person- alized animation and holds significant potential for wide range of applications.

---

Published as a conference paper at ICLR 2024
7 ETHICS STATEMENT
We strongly condemn the misuse of generative AI to create content that harms individuals or spreads misinformation. However, we acknowledge the potential for our method be misused since primarily focuses on animation and can generate human-related content. It is also important to highlight that our method incorporates personalized text-to-image models developed by other artists. These models may contain inappropriate content and can be used with our method.
To address these concerns, we uphold the highest ethical standards in our research, including ad- hering to legal frameworks, respecting privacy rights, and encouraging the generation of positive content. Furthermore, we believe that introducing an additional content safety checker, similar to that in Stable Diffusion (Rombach et al., 2022), could potentially resolve this issue.
REPRODUCIBILITY STATEMENT
We provide comprehensive implementation details for the training and inference of our method in supplementary materials, aiming to enhance the reproducibility of our approach. We also make both the code and pre-trained weights open-sourcen facilitate further investigation and exploration.
ACKNOWLEDGEMENT
This project is funded in part by Shanghai AI Laboratory (P23KS00020, 20222D0160201), CUHK Interdisciplinary AI Research Institute, and the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (ITC)'s InnoHK.
REFERENCES
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1728-1738 2021.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, al. ediffi: Text-to-image diffusion models with an ensemble expert denoisers. arXiv preprint 2rXiv:22/1.0133 2022.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion mod- els. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 02563-22575. 2023.
Duygu Ceylan, Chun-Hao Paul Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. arXiv preprint orXiv:2303. 12688, 2023.
Civitai. Civitai.
2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-8794, 2021.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34: 19822299835.... 2021.
Shuangrui Ding, Maomao Li, Tianyu Yang, Rui Qian, Haohang Xu, Qingyi Chen, Jue Wang, and Hongkai Xiong. Motion-aware contrastive video representation learning via foreground- background merging- In Proceedings of the JEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9716-9726. 2022.
Ziyi Dong, Pengxu Wei, and Liang Lin. Dreamartist: Towards controllable one-shot -onagto- generation via contrastive prompt-tuning. arXiv preprint arXiv:2211. 2022.
10

---

Published as a conference paper at ICLR 2024
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Ger- manidis. Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv:2302.03011, 2023.
Robert French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128-133) 135, 1999.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth word: Personalizing text-to-image generation using textual inversion, arXiv preprint arXiv:2208.01611 2022.
Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen- Or. Designing an encoder for fast personalization of text-to-image models. arXiv preprint 1rXiv:2302.122285 2023.
Gen-2. Gen-2: The next step forward for generative ai. https::/researee runwayml. com/gen2/ 20233
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10696-10706, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint wx.iv:2200.0.::00 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv preprint arXin:2204.0345.. 03458, 2022b.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pre- training for vext-to-video generation via transformers. arXiv preprint arXiv:2205. 2022.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.00 09685, 2021.
Hugging Face. Huggingface. https::/
co/, 2022.
Xuhui Jia, Yang Zhao, Kelvin CK Chan, Yandong Li, Han Zhang, Boqing Gong, Tingbo Hou, Huisheng Wang, and Yu-Chuan Su. Taming encoder for zero fine-tuning image customization with text-to-image diffusion models. arXiv preprint arXiv:2304.02642 2023.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion mod- els are zero-shot video generators. IEEE International Conference Computer Vision (ICCV), 2023.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinske. et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521-322 2017.
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun Yan Zhu. Multi-concept customization of text-to-image diffusion. Proceedings of the IEEE/CVF Conference on Com- puler Vision and Pattern Recognition, pp. 1931-1941, 2023.

---

Published as a conference paper at ICLR 2024
Wei Li, Xue Xu, Xinyan Xiao, Jiachen Liu, Hu Yang, Guohao Li, Zhanpeng Wang, Zhifan Feng, Qiaoqiao She, Yajuan Lyu, et al. Upainting: Unified cot-to-image diffusion generation with cross-modal guidance. arXiv preprint arXiv:2210.1603// 2022.
Haoming Lu, Hazarapet Tumanyan, Kai Wang. Shant Navasardyan, Zhangyang Wang, and Humphrey Shi. Specialist diffusion: Plug-an Plug-and-play sample-efficient fine-tuning of text-to-image diffusion models to learn any unseen style. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 14267-14276, 2023.
Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 10209-10218, 2023.
Aniruddha Mahapatra, Aliaksandr Siarohin, Hsin- Ying Lee, Sergey Tulyakov, and Jun- Zhu. Text-guided synthesis of eulerian cinemagraphs, 2023.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6038-6047, 2023.
Chong Mou, Xintao Wang, Lianghin Xie, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. 12i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint AxXi:2300208855. 2023.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models, arXiv preprint arXiv:2112.10774) 2021.
Pika Labs. Pika labs. https:://// http::://www..i.c. art/ 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(55:545--551, 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning, pp. 8821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference on Computer Vision and Patlern Recognition, pp. 10684-10695, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation, 2015.
Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 02119-10228. 2023.
12

---

Published as a conference paper at ICLR 2024
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuming text-to- eext-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22500- 22510, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems, 35:36479- -6494,, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022.
Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint urXiv:2304.03411, 2023.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. aiieeo-- Text-to-video generation without text-video data. arXiv preprint 2rXiv:2209.14992 2022.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint wwXiv:2010.02502. 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Wen Wang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen. Zero- shot video editing using off-the-shelf image diffusion models, arXiv preprint arXiv:2303.1 17599, 2023a.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang. Yuwei Guo, Tianxing Wu, Chenyang Si, Yuming Jiang, Cunjian Chen, Chen Change Loy, Bo Dai, Dahua Lin, Yu Qiao, and Ziwei Liu. Lavie: High-quality video generation with cascaded latent diffusion models, 2023b.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang. Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for texi-to-video generation. IEEE International Conference on Computer Vision (ICCV), 2023.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint N.XXi:22308.662211 2023.
Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Drag- nuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308. 08088, 2023a.
Shengming Yin, Chenfei Wu, Huan Yang, Jianfeng Wang. Xiaodong Wang. Minheng Ni, Zhengyuan Yang, Linjie Li, Shuguang Liu, Fan Yang, et al. Nuwa-xl: Diffusion over diffusion for extremely long video generation. arXiv preprint arXiv:2303. /2346, 2023b.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. IEEE International Conference on Computer Vision (ICCV), 2023.
Daquan Zhou, Weimin Wang. Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.110ff 2022a.
Yufan Zhou, Ruiyi Zhang, Changyou Chen, Chunyuan Li, Chris Tensmeyer, Tong Yu, Jiuxiang Gu, Jinhui Xu, and Tong Sun. Towards language- free training for text-to-image generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17907- 17917, 2022b.
13

---

reference video on Published as a conference paper at ICLR 2024
APPENDIX
A IMPLEMENTATION DETAILS
Training. We utilize the WebVid-IOM dataset (Bain et al., 2021), a large-scale video dataset con- sisting of approximately 10.7 million text-video data pairs to train the motion module. This dataset offers diverse motion categories, which significantly facilitates the learning process of the motion module. We adopt training resolution of 256 256 to balance training efficiency and motion quality. To train the domain adapter, we randomly sample static frames and resize them to the target resolution. For the motion module and MotionLoRA, we uniformly sample the videos stride of 4 to get video clips at a length of 16. We use a learning rate of 1 x 10 and train the motion module with 16 NVIDIA A100s for 5 epochs.
Inference. As described in the main paper, at inference, we first inflate a personalized text-to- text-to-image model and insert the pre- pre-trained motion module to constitute the corresponding animation generator. In our experiment setup, we generate animations at a resolution of 512 x 512 using a DDIM (Song al., 2020) sampler with classifier-fre: guidance. We referred the model's official web page to determine the denoising hyperparameters (guidance scale, LoRA scaler, etc.) and generally adopted the same settings.
Table 2: Community models for evaluation.
Model Name
Domain
Type
Toon
2D Cartoon
T21 Base Model
MeinaMix
2D Anime
T21 Base Model
Lyriel?
Stylistic
T2I Base Model
RCNZ Cartoon 3d
3D Cartoon
T21 Base Model
epiC Realism"
Realistic
T2I Base Model
Realistic Vision
Realistic
T21 Base Model
Oil painting"
Stylistic
LoRA
MoXin TUSUN
Stylistic
LoRA
Concept
LoRA
Models for evaluation. To ensure comprehensive benchmark, we selected nine representative personalized T21 models from Civitai (2022), a model-sharing platform that enables artists to upload their creations. As illustrated in Table 2, these models encompass diverse domains such as 2D anime, stylistic painting, and realistic photographic images. They also cover wide range of subjects, including portraits, animals, landscapes, etc.. This selection ensures comprehensive evaluation of our approach across various domains and subjects.
Baselines adaptation. To adapt the two academic baselines for personalized animation genera- tion, we followed the recommended best practices in the respective papers and performed parameter tuning
by-see basis. For Tune-A- Videeo (Wu et al., 2023), the project's 's webpage and fine-tune the network after replacing the T21 backbone with a person- alized one, as suggested in the paper. Regarding Text2Video-Zenn (Khachatryan et al., 2023), directly generate video clips upon the personalized T2Is without any modifications. In addition, we conducted qualitative comparisons with two commercial tools for video generation and image animation, namely Gen-2 (2023) and Pika Labs (2023). For Gen2, we employed personalized T21 images as image prompts to generate the corresponding videos. As for Pika Labs, we utilized it to animate still images generated by the personalized T21s.
-ora
14

---

Published as a conference paper at ICLR 2024
(SD1.5) Close up of grapes on table.
(SD1.5) Sunset time-lapse the beach.
(SD1.5) astronaut
(SD1.5) bigfoot walking.
Jlying in space.
Figure 9: Results on base T21 backbone. Best viewed with Acrobat Reader. Click the images to play the animation clips.
User study. To ensure a fair comparison between our method and the baselines, we generated 20 animations for each method without annerry-pccking.. resulting in 20 sets of triple pairs. Subsequently, conducted
study involving ten participants. Each participant is presented with the three samples generated by different methods at one time and asked to rank them based on three specific aspects: text alignment, domain similarity, and motion smoothness. To evaluate text alignment, provide the corresponding text prompt to the users and request them to rank the samples accordingly. To assess domain similarity, we initially generate reference images using the same personalized T2I. These reference images are then presented to the users, who are asked to rate the animations based on their perceived similarity the reference images. Regarding motion smoothness, users are instructed to rank the animations based on the consistency of the motion.
CLIP metric. Using the generated animations, we first extract the CLIP image embeddings each frame and then compute the cosine similarity under different settings. To assess text alignment, we computed the average similarity between the prompt embedding and the embeddings of individual frames. For evaluating domain similarity, computed the CLIP score between the reference im- ages and the frames of the animations. To measure motion smoothness, we calculated the similarity between all pairs of video frames and reported the average number.
B ADDITIONAL DISCUSSIONS
B. VISUAL QUALITIES ON BASE T2I.
By integrating the motion module with the base T2I that the motion module is pre-trained upon, i.e., Stable Diffusion V1.5, AnimateDiff demonstrates capabilities in general T2V generation. We show- case such ability by generating videos with commonly used textual prompts previous works (Zhou et al., 2022a; Blattmann et al., 2023). As illustrated in Fig. 9, without the enhancement from per- sonalized T2I models, the domain of the synthetic videos corresponds closely with the pre-training dataset WebVid-10k Bain et al. (2021).
B.2 DOMAIN ADAPTER VISUALIZATION
To further validate the effectiveness of the domain adapter, we conduct an additional ablative study where the motion module trained with the domain adapter entirely removed from the pipeline. We qualitatively compare the personalized T2I animation results upon three baselines: (1) training without adapter; (2) full pipeline with scaler a set 1; (3) full pipeline with scaler a set to
As shown in Fig. 10, when the domain adapter is completely removed from the training pipeline, visual attributes inherent to the training dataset, specifically watermarks, emerge in the synthetic animations (Ist row). This arises due to the intertwining of visual appearance and motion learning during the motion module's training phase, resulting in the watermark pattern being learned by the motion module and subsequently transferred to other personalized T2I backbones. Similarly, watermarks appear when the adapter exerts its full impact (2nd row). In contrast, by fitting the visual distribution separate domain adapter and eliminating during inference, our full pipeline
15

---

Published as a conference paper at ICLR 2024
adapter
full pipeline, a
full pipeline, a
Figure 10: Abaltions on domain adapter. Best viewed with Acrobat Reader.
Click the images to
play the animation clips.
w/o scale-up training
w/ scale-up training
w/o scale-up training
w/ scale-up training
Figure 11: Ablations scale-up training. Best viewed with Acrobat Reader. Click the images to play the animation clips.
(3rd row) achieves superior quality devoid of watermarks. This implies that the visual distribution within the training dataset can be effectively eliminated by merely dropping the adapter.
B.3 BENEFITS FROM SCALE-UP TRAINING
In practice, we find that the overall quality of the generated animations benefits from scale-up train- ing. This involves training with larger batch sizes, video resolution, and the number of total optimiz- ing iterations. Fig. 11, we present two pairs qualitative comparisons between motion modules trained with standard and scale-up training. Under the scale-up training setting, train the mo- tion module on the resolutions of 320 x 512, with 8x larger batch size compared to the standard setting. The result indicates that considerable enhancement in motion amplitude and diversity can be achieved through an increase in the training scale. For instance, the camera involves view angle changes (2nd row) in contrast mere zooming (Ist row). The character's head displays turning movements (4th row) rather than solely facing forward (3rd row).
C LIMITATIONS
C.I MOTTON PRIORS IN ANIMATEDIFF
The transferable motion priors AnimateDiff learned from large-scale video dataset Web Vid- 10M (Bain et al., 2021) that encompasses mainly real-world footage. Supported by the richness and diversity of the dataset, the motion module can learn real-world motions (Ding al., 2022) like sea waves, vehicular movement, and human actions, which are modeled by the temporal self-attention mechanism. Therefore, the motion priors largely depend on the dataset coverage and accuracy, which introduces potential limitations discussed follows.
Motion diversity and complexity. In practice, we find that the motion module pre-trained on Vid-id110M performs well non-violent motions such fluid (e.g., waves, fog, etc.), rigid objects (e.g., cars, boats), and simple human movements (e.g., walking, facial expressions). This aligns with our observation that the training dataset predominantly encompasses these motions.
16

---

Published as a conference paper at ICLR 2024
Stable Diffusion V1.5
Stable Diffusion XL
Figure 12: Improvements on the more powerful backbone. Best viewed with Acrobat Reader. Click the images to play the animation clips.
However, the module struggles with complex motions that are infrequent in the training dataset and challenging to represent via short video clips during training, e.g., dance movements and drastic scene changes. These instructions often result in static synthetic outcomes or unnatural deforma- tions. Potential solutions could involve enriching the training set's 's motion diversity training with larger resolution and extended clip length, which will help to better model motion patterns.
Text-motion alignment. In the pre-training dataset, Web 10M, most text labels primarily de- scribe visual content while overlooking detailed motion descriptions. Consequently, this leads to the animations generated by Animate exhibiting little response to the motion descriptions. Notwith- standing, this phenomenon does not imply that the motion module does not acquire corresponding motion priors. For instance, zoom-in/out effects frequently appear in the pre-training videos. How- ever, their text labels typically contain only broad "zooming" tags, making it difficult for the motion module to distinguish the difference between zoom-in and zoom-out accurately. As result, utilizing a "zoom in" prefix alongside the common text prompt generates both zoom-in and zoom-out effects, indicating the need for a video dataset with more accurately labeled motion tags. This also suggests that MotionLoRA does not learn new motion patterns entirely from scratch but refines and enhances the pre-existing motion priors (regardless of whether they can triggered by text) obtained during pre-training, enabling the motion module to express such priors as desired during inference.
C.2 DEPENDENCY ON IMAGE BACKBONE
Under the decoupled training strategy, the motion and visual content in the generated animations originate from the pre-trained motion module and the underlying image backbone, respectively. Consequently, the performance of the entire pipeline of AnimateDiff is heavily reliant on the un- derlying T2I models. If the base model struggles respond appropriately fails to generate accurate content, the additional motion module is unlikely to compensate for this weakness. Conversely, superior image backbones can enhance the synthetic results. To demonstrate this, we implement AnimateDiff on Stable Diffusson XL (Podell et al., 2023) and compare general T2V results on rare semantic compositions against the Stable Diffusion V1.5 version, as depicted in Fig. 12. The figure illustrates that the synthetic video based on SDXL achieves better visual composition and semantic alignment.
the text prompt and
Practically, to mitigate potential limitations introduced by the foundational T2I models, employing off-the-shelf modules such as IP-adapter (Ye et al., 2023) for additional style/content reference, ControlNet (Zhang et al., 2023) for spatial composition corrections, could be beneficial.
D MORE VISUAL RESULTS
In Fig. 13, show more visual results of AnimateDiff and the results of further combing Animate- Diff with MotionLoRA to achieve shot type control. In Fig. 14, we show more qualitative compar- isons between AnimateDiff and four academic and commercial baselines. In Fig. 15, compare two motion module architectures, i.e., the full convolution one and its Transformer counterpart.
17

---

Published as a conference paper at ICLR 2024
Oil painting
Realistic Vision
Realistic Vision
Realistic Vision
city, rainy day, wet, a bustling street, oil painting. Oil painting
photo of 18 y.o woman dress, night city street, motion blur.
photo of cyberpunk city street, night time, dark atmosphere.
b&w photo of 42 yo man in black clothes, bald, face, half body.
Lyriel
Oil painting
epiC Realism
oil painting, black portrait halo, sun- glasses, blue eyes, tar- tan scarf,
oil painting, mountain, lake water, boat, forest, masterpiece, ...
landscape, rocky mountain with milky way, nighttime, ...
pearl pirate ship, wind, waves, night time,
epiC Realism
epiC Realism
Toon You
Realistic Vision
(zoom-in) universe,
(rolling) landscape of a aesthetically Belgium and wildflower, detailed, colorfal, ...
nebula highly
(rolling) Iboy, dark skin. playing guitar, concert, stage lights,
(zoom-in) cabins forest, water, aurora in the sky, fog. ...
Oil Painting
MoXin
RCNZ Cartoon 3d
Lyriel
(panning) oil painting, house, grass, wheat field laboring crowd. ...
(panning) tic composition, Chinese town,
fantas- old
(panning) golden labrador, warm vibrant colours.
(tilting) waters, canyon, sunlight, traveler, high quality....
Figure 13: Additional qualitative results. play the animation clips.
Best viewed with Acrobat Reader. Click the images to
18

---

Published as a conference paper at ICLR 2024
Tune-A A-ie-A-Ven
A-ne-A--ideo Video
AnimateDiff
T2
AnimateDiff
man is playing guitar, dramatic lighting, Pika Labs (2023)
girl is playing guitar, wavy hair, upper body,
AnimateDiff
Gen-2 (2023)
AnimateDiff
LBBT
cabins in the forest, water, aurora in the fog, Pika Labs (2023)
taxi, rear view, New York city at night,
AnimateDiff
Gen-2 (2023)
AnimateDiff
sunset, orange sky, fishing boats, ocean waves,
a woman standing on the road at night,
Figure 14: Qualitative comparison. Best viewed with Acrobat Reader. Click the images to play the animation clips.
Convolution
Transformer
Convolution
Transformer
Figure 15: Module design comparison.
19


Reviews:
[{"review_id": "ttDjPXOGQl", "reviewer": "Reviewer_Jp9x", "summary": "The paper introduces a simple method for generating animations using a customized text-to-image model adapted for video. It innovates by transforming a text-to-image model to produce a sequence of frames, simulating animation. This is achieved through frame-by-frame processing with original diffusion model and the integration of a transformer-based motion module that processes temporal information across patches from all frames. A domain adapter is introduced to overcome the challenge of dataset-specific artifacts, enhancing the quality of the video output.  In the end, the authors also demonstrate the model's ability to efficiently learn new motion given the unified representation including zoom in and rolling.  Quantitative and qualitative comparisons with  a broad spectrum of existing models validated the effectiveness of the proposed approach.", "questions": "Please see my comments in the weakness section.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "S1: The paper's foremost strength lies in its elegant balance between simplicity and performance. With only minor change to the base stable diffusion model—principally, the addition of a few transformer layers—the method yields robust quantitative outcomes, positioning it as a strong baseline for text-to-video synthesis within the research community.\n\nS2: The simplicity of the approach belies its technical depth. The integration of new domain adaptation layers effectively mitigates artifact issues typical in video datasets. Furthermore, the use of LORA for streamlined learning of new motions, coupled with transformer layers for temporal data synthesis, represents straightforward yet impactful innovations, all substantiated by thorough ablation studies.\n\nS3: The paper presents good qualitative and quantitative results.\n\nS4: The paper is well written. It combines clear, concise text with instructive visuals, making the methodology and results accessible and understandable.\"", "weaknesses": "W1: Some modules are not that novel from a technical perspective. E.g. the transformer block is also explored in GEN-2 [1]. LORA and adapter are also commonly used in various personalization papers. \n\nW2: The paper somewhat overlooks a thorough discussion of the method's limitations. Specifically, it would be beneficial for the authors to clarify the learning dynamics of the motion module when pre-trained on the WebVid dataset—whether motions like zoom-ins and rolls require explicit training (using LORA with curated dataset) or can be inferred from textual cues alone. Additionally, a systematic categorization of which motions are effectively learned and which are not could significantly enhance the reader's understanding of the model’s practical applications and boundaries.\n\n\n[1] Esser, Patrick, et al. \"Structure and content-guided video synthesis with diffusion models.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uWLzLtVVJU", "reviewer": "Reviewer_31Uu", "summary": "The paper introduces AnimateDiff, a novel pipeline designed to convert static Text-to-Image (T2I) models into animation generators without the need for model-specific fine-tuning. The process uses a plug-and-play motion module named MotionLoRA that learns motion priors from video datasets and can be integrated directly into personalized T2Is to produce smooth animations. Training involves fine-tuning a domain adapter on the base T2I, introducing a motion module, and then adapting this pre-trained module to specific motion patterns using Low-Rank Adaptation (LoRA). Evaluation was performed on various T2I models yielding promising results, demonstrating that a Transformer architecture is effective for capturing appropriate motion priors.", "questions": "Why does the scale of the domain adaptor in Figure 6 appear to be visually linked with camera movement?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The proposed concept offers a robust and user-friendly plugin to embed motion priors into general T2I models. The model designs are sensible and meaningful. For instance, the domain adapter plays a critical role in mitigating adverse impacts from training data, and MotionLoRA efficiently adjusts to new motion patterns.\n\n- The experiments demonstrate promising applications, including the animation of diverse-style T2I models and controllable dynamic generation using ControlNet.", "weaknesses": "- In certain animation results, the motion amplitudes are minimal and flickering between frames is noticeable. This suggests room for improvement in the motion prior. These issues may be related to the size of the video dataset and model design, and a thorough analysis of these factors should be included in the limitations section.\n\n- Qualitative comparisons are not enough. Please provide visual results for more cases like in Figure 5.\n\n- For image animation, the identity preservation is poor, in other words, the characters in the animation results are not very similar to the one in the input image.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fy70nKV9h0", "reviewer": "Reviewer_ih9K", "summary": "This paper introduces \"AnimateDiff,\" a revolutionary framework designed to infuse motion dynamics into personalized text-to-image (T2I) diffusion models without necessitating model-specific tuning. AnimateDiff features a versatile, plug-and-play motion module at its core, enabling the generation of animated images. The motion module is trained to adaptively learn and apply transferable motion priors from real-world videos, allowing for seamless integration into various personalized T2Is, and fostering the creation of a personalized animation generator.\n\nAdditionally, the paper unveils \"MotionLoRA,\" a novel, lightweight fine-tuning technique engineered for AnimateDiff. MotionLoRA facilitates the adaptation of pre-trained motion modules to new motion patterns, such as diverse shot types, ensuring adaptability with minimal training and data collection efforts.", "questions": "### 1. Domain Adapter Visualization:\nCould the authors provide visualization results that specifically exclude the domain adapter training state? To clarify, I am interested in viewing the outcomes when the domain adapter training (stage 1) is entirely omitted from the process, not just when the parameter $\\alpha$ is set to zero at the inference state.\n\n### 2. Visual Quality of AnimateDiff in Base T2I Models:\n\nRegarding the visual quality, how does AnimateDiff perform when applied directly to the base T2I model that you used to train the motion module on WebViv? I am particularly interested in understanding the performance and visual outcomes of AnimateDiff when integrated with the foundational T2I models, excluding any enhancements or personalizations from external models such as Civitai.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. AnimateDiff empowers personalized text-to-image (T2I) models with animation generation capabilities, eliminating the need for specific fine-tuning. Besides, it demonstrates the efficacy of Transformer architectures in modeling motion priors, contributing valuable insights to the field of video generation. \n\n2. The authors propose \"MotionLoRA,\" an ingenious, lightweight fine-tuning technique enabling pre-trained motion modules to adapt to new motion patterns seamlessly. \n\n3. AnimateDiff and MotionLoRA are rigorously tested against representative community models, academic baselines, like Gen2 and Tune-A-Video.", "weaknesses": "This paper is well-composed and presents a notable contribution to the field of text-to-image (T2I) diffusion models.\n\nHowever, one aspect that could be further refined or discussed is the dependency of AnimateDiff on the underlying T2I models. It appears that the success of AnimateDiff is inherently tied to the performance and reliability of the existing T2I models it seeks to enhance. Specifically, the methodology might face challenges if the base T2I models struggle or fail to accurately generate images based on user-provided prompts. In such cases, AnimateDiff might encounter difficulties in achieving its animation objectives, potentially limiting the overall effectiveness and applicability of the proposed solution.\n\nIt might be beneficial for the paper to address this dependency, possibly discussing strategies or considerations to mitigate potential challenges arising from limitations in the foundational T2I models. This would strengthen the robustness and adaptability of AnimateDiff, ensuring its broader success and applicability in various T2I contexts and scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K0CiY5Bw5R", "reviewer": "Reviewer_eMMP", "summary": "This paper proposes a practical plug-in-play solution for a personalized T2I video generation model. It consists of three parts, the first pipeline uses lora layer to adapt domain from image pre-training to video training domain due to visual quality and motion blur issues. The second part is a motion module trying to learn the motion prior with a temporal transformer model. The last part is optional, which is a motion-lora design to adapt to a personalized video domain. At inference time, the general-purpose motion module can be used in any pre-trained personalized T2I model to animal the image model. Overall, this paper is well-written and the solution is very practical. Empirically, the method has competitive performance compared to other general-purpose text-to-video models.", "questions": "-- I do have some concerns about the generalization of the motion module. It is not clear to me how reliable to use LoRA for domain adaption, and if the motion module does not generalize well, should we further fine-tune the motion module on the task domain data?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "– This paper is trying to solve a novel and practical problem in the text-to-video generation domain. How to turn a pre-trained personalized T2I model into a video model without training is attractive and practically useful.\n– Qualitatively, this method shows impressive result and demonstrate the generalization ability of the motion module to different T2I model\n– It also has an extensive ablation study showing the effectiveness of the domain adapter and motion module design choices.", "weaknesses": "– Lack of technical novelty, the proposed domain adapter is based on LoRA and the motion module is following standard feature inflation and the architecture is following timesformer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)