Bidirectional Temporal Diffusion Model for Temporally Consistent Human Animation
Authors: Tserendorj Adiya
Jae Shin Yoon
JUNGEUN LEE
Sanghun Kim
Hwasup Lim
Venue: ICLR 2024 poster (2024)
Keywords: Human Motion Transfer
Diffusion Model
Human Animation
Generative model.
Primary area: generative models

Abstract: We introduce a method to generate temporally coherent human animation from a single image, a video, or a random noise.
This problem has been formulated as modeling of an auto-regressive generation, i.e., to regress past frames to decode future frames.
However, such unidirectional generation is highly prone to motion drifting over time, generating unrealistic human animation with significant artifacts such as appearance distortion. 
We claim that bidirectional temporal modeling enforces temporal coherence on a generative network by largely suppressing the appearance ambiguity.
To prove our claim, we design a novel human animation framework using a denoising diffusion model: 
a neural network learns to generate the image of a person by denoising temporal Gaussian noises whose intermediate results are cross-conditioned bidirectionally between consecutive frames. 
In the experiments, our method demonstrates strong performance compared to existing unidirectional approaches with realistic temporal coherence.

Published as a conference paper at ICLR 2024
BIDIRECTIONAL TEMPORAL DIFFUSION MODEL FOR TEMPORALLY CONSISTENT HUMAN ANIMATION
Tserendorj Adiya3. Jac Shin Yoon?
Jungeun Leet
Sanghun Kim
Hwasup Lim
Korra Institute of Science and Technology
Adobe
3 AA Center, CJ Corporation
adiyaeey .ett, jacyooneadobe com,
(092599, ke197103, he hewin
ABSTRACT
We introduce method to generate temporally coherent human animation from a single image, a video, or a random noise. This problem has been formulated modeling of auto-regressive generation, i.e., regress past frames de- code future frames. However, such unidirectional generation is highly prone to motion drifting over time, generating unrealistic human animation with signifi- cant artifacts such as appearance distortion. We claim that bidirectional temporal modeling enforces temporal coherence generative network by largely sup- pressing the appearance ambiguity. To prove our claim, we design novel hu- man animation framework using denoising diffusion model: neural network learns to generate the image of person by denoising temporal Gaussian noises whose intermediate results are cross- cross-conditioned bidirectionally between consec- utive frames. In the experiments, our method demonstrates strong performance compared to existing unidirectional approaches with realistic temporal coherence.
INTRODUCTION
Humans express their own space-time continuum in the form of appearance and motion. While existing generative models [sola et al. (2017); |Sarkar et al. (202T) have been successful to restore the space, í.e., high-quality image generation with diverse human appearance, they often fail to decode the time, e.g., temporally incoherent human motion. In this paper, we introduce a method for temporal modeling of a generative network to synthesize temporally consistent human animations. Our method generate human animation from three different modalities: random noise, single image, and a single video as shown in Figure. I Such generated human animations enable number of applications including novel content creation for non-expert media artists and pre- visualization of human animation that can be further refined by professional video creators.
The temporal modeling for human animation has been often formulated as video auto-regression problem: using past frames as a condition to decode future frames. While such unidirectional gen- eration (forward auto-regression) has shown smooth animation results, often suffers from texture drifting, e.g., the texture on the clothing of a person such as skirt in Figure 2 is largely distorted along its dynamic movements. This is mainly due to the sigmificant motion-apy -aploo--ppe ambiguity where there exist infinite solutions to decide the future state of human appearance even with the same motion, which amplifies the artifacts (e.g., distortion) over time.
To suppress such apprion-appeaaacce ambiguity, we model a human appearance bidirectionally: generative network decodes the human appearance the context of both forward and backward im- age regression whose intermediate features are cross-conditioned over time. Our key observation is that the bidirectional temporal consistency feature space highly suppresses the motion-appearance ambiguity, which prevents from the texture drifting while maintaining its temporal smoothness.
We realize the idea of bidirectional temporal modeling by utilizing a generative denoising diffusion model Ho et al. (2020). A denoising network learns to iteratively remove temporal Gaussian noises to generate the human animation guided by conditioning poses and appearance style. Inspired by message passing algorithms in dynamic programming Felzenszwalb & Zabih (2010); Arora et al.
WWrk done during internship Korea Institute of Science and Technology.

---

Published as a conference paper at ICLR 2024
Noise
Anunation
Imuage
Aniiation
Figure 1: Our method generates temporally coherent human animation from various modalities.
2009), we recursively cross-condition the intermediate results between consecutive frames a bidirectional way as shown in Figure 3] where the temporal context of human appearance is locally consistent for consecutive frames at the first denoising step, and is progressively refined at every denoising iteration be globally coherent for entire frames.
In the experiments, we demonstrate that our bidirectional denoising diffusion model generates human animations from single image with strong temporal coherence, outperforming the results from unidirectional generative models. We also show that learning from multiple frames, ie.. pweson--speeiiee video, can further improve the phys- ical plausibility of the generated human animation. Fi- nally, we showcase that our method can generate hu- man animations with diverse clothing styles and identities without any conditioning images.
Results from a unidirectional generative model with texture drifting over time.
Figure
Contribution (I) We propose bidirectional temporal diffusion model that can generate tempo-
rally coherent human animation from random noise, single image, or a video. (2) Inspired by dynamic message passing algorithms, we introduce the feature cross-conditioning between consec- utive frames with recursive sampling, which allows embedding the motion context on the iterative denoising process in locally and globally consistent way. (3) We quantitatively and qualitatively demonstrate that our method shows a strong temporal coherence compared to existing unidirectional methods. For an accurate evaluation, we newly create high-quality synthetic data of people in dy- namic movements using graphics simulation. which provides ground-truti data, i.e., different people the perfectly same motion.
RELATED WORKS
Human Motion Transfer Given a sequence of guiding body poses and the style of human appear- ance, it aims to generate the human animation that satisfies the conditioning motion and style. Many existing pose transfer methods have utilized 2D keypoints as conditioning body pose maps Chan eet al. (2019); Balakrishnan et al. (2018); [Esse et al. (2018): Liu et al. (2019a). However, these approaches often fail to extract the physical implications from the keypoints maps, resulting in tem- porally unnatural human animation.
To address this motion consistency issue, methods such as EDN Chan et al.) 2019), V2V Wang et al. (2018). and DIW WWang al. (202I) leveraged Markovian independence generate auto-regressive frames. These approaches utilize Densepose Güler et al. (2018) as 2D pose conditioning and learn motion-dependent appearance for specific person, producing realistic animation results for unseen motions. Recent advancements in this area involve embedding 3D velocities from the SMPL Loper al. (015)) model as pose conditioning Yoon et al.) (2022), leading the better generation of complex transformations. However, these methods require extensive training on the videos of single individual, limiting their generalizability to diverse people.
To synthesize human animations of diverse people using single model, several works have studied human motion transfer from single image. Solutions include applying affine transformations Bal- akrishnan et al. (2018): Zhou et al. (2019), flow-based warping Wang et al. (2019). Zhao & Zhang (202)); Starohin et al. 2019), (01...)... or assuming a base 3D human model and texture mapping with DensePose Neverova al. (2018); {); Huang et al. (202I) or the SMPL model et al. (2019); Liu al. (2019b). However, these methods struggle to represent diverse surface transformations in clothing, Le., the clothing texture looks static under the pose changes, resulting in unnatural animations.
2

---

Published as a conference paper at ICLR 2024
k=l
-=---
t=2
Unidirectional Temporal Diffussin Model
Bidirectional Temporal Diffusion Model
Figure 3: The left illustration represents a unidirectional diffusion model, and the right one provides an overview of our proposed bidirectional temporal diffusion model (BTDM). The dotted arrows indicate the direction of conditioning, and i and t represent the denoising step and time interval, respectively.
Generative Diffuusio Models Recently, diffusion models have demonstrated outstanding per- formance in high-quality image generation Ho al. 2020); Nichol & Dhariwal (2021): ); Song
et al. (2020b), toxt-to-image translation Rombach et al. (2022); Preechakul et al. (2022); Saharia
al. (2022a): Ramesh al. (202T), image super-resolution Saharia al. (2022b). image restora-
tion Kawar et al. (2022): Waan et al. (2022) view synthesis Watson et al. (2022). Compared to
generative adversarial networks (GANs) Isola et al. (201)); Karras et al.] (2019), diffusion models
enable more stable training and reduced mode collapse, leading diverse and high-quality genera-
tion results.
The initial diffusion model was based on Song's [ong & Ermon (2019) score-matching approach, which estimates gradients using Langevin dynamics to infer data distributions. Subsequently, the DDPM al. (2020) method was introduced, leveraging weighted variational bounds and becom- ing widely adopted. Later, NCSN Song & Ermon (2019) and its equivalent from ODE Song et al. (0220) emerged, presenting a more general form.
One notable drawback of Markov Chain Monte Carlo (MCMC) based inference in diffusion models the longer inference time compared GANs. DDIM Song al. (2020a) addresses this issue by interpreting the diffusion process as an implicit function that significantly reducing sampling time while preserving generation quality.
Diffusion models, as referenced in recent studies such as Ho et al. (202bb) : Yang et al. (2023) : Ho
2023): Guo et al. (2023), have become increasingly popular in the field of video generation. These models often utilize techniques
al. (2022a): Singer et al. (2022); ; Zhou et al.) (2022); Esser al.
like cross-attention and 3D U-Nets ensure videos remain consistent over time. Despite their potential, most of these methods face challenges in generating longer videos without encountering issues like shape drifting and appearance jitters. This paper presents a new and practical approach to overcome these limitations. Our method, which is distinct from previous work, employs 2D-Unet based framework to create temporally coherent animations. Notably, our approach effective in generating animations of humans and is not constrained by video length.
METHOD
Conventional Denoising Diffusion Probabilistic Models (DDPM) work by gradually diffusing isotropic Gaussian noise onto data sample DD across K steps along Markovian chain. The process is reversed, such that is approximated from the (0, 1) distribution.
One can extend these conventional DDPM generate human animation driven by sequence of human pose maps S {(,,,.., ST) (e.g., densepose Güler et al. (001)) in an auto-regressive way. For example, network is designed generate future frames dependent on previous frames by gradually diffusing isotropic Gaussian noise onto the training sample yt E Y = (yi...,, yr under the conditional Markovian independence, i.e., p(Y) I p(pulye-1: S). However, such autoregressive models often suffer from texture drifting due to the motion-appearan ambiguity that is inherent in unidirectional prediction.
3

---

Published as a conference paper at ICLR 2024
To suppress the motion-appearance ambiguity, design bidirectional temporal diffusion model (BTDM) as shown in Figure 3 BTDM learns motion- deetonnde appearances in both forward and backward directions along the time axis. The denoising results from each step in either time direc- tion serve as mutual conditions for generating human animation. Our model can generate realistic animations unconditionally, well as conditionally from single image video.
3.1 BIDIRECTIONAL TEMPORAL DIFFUSION MODEL.
Given pose sequence S and its corresponding image sequence Y, modeling their mapping bidirec- tionally along the time axis that follows Markovian independence results in:
1 Pf PS = II p(..)) ps (Y = I1 plyc-1l9te sc-1)
(I)
In this setup. represents the forward direction along the time axis, and ps signifies the backward direction. We define marginal distribution with isotropic Gaussian process that gradually adds
increasing amounts of noise to the data sample as the signal-to-noise A(-) decreases, follow- ing (Salimans & Ho (2021):
q(y 1:K (b0)
II gl glviloz 1
aivilot =) -(( ((-x((())))
where a--) is the sigmoid function, K is the number of diffusion step, and denotes the identity.
Both the motion-dependent appearance distribution in Equation and the difffusion process in Equa- tion follow a Markovian chain. Ideally. we should predict Yr and 8t-1 using perfectly denoised Vi-1 (in the forward direction) 2 (in the backward direction) conditions. However, such per- fectly denoised images are not available during inference, which leads to overfitting to the training data and amplifies the error from motion- aationaappeaancce ambiguity. For this reason, we integrate these two independent Markovian chains as follows:
T P)(Y) = II plu 1
Po (o = II plu llt st-1),
(3)
where by utilizing the noisy yk as condition, we concurrently diminish the reliance of motion- dependent appearance generation the preceding frame and avert overfitting the condition, thereby alleviating artifacts when generating unseen conditions and improving the model's gener- alization performance. This approach also yields more temporally consistent animations by highly limiting the motion diversity between consecutive frames.
Although Py and ps are independent, p(u(lyr-1) and p(y--1 are concurrently defined the time axis t. This allows us to optimize both probabilities simultaneously. Therefore, the objective function for training defined as follows:
L = Be.. () dy do,c 2 (1l v/ X(k), c, dy) v0
(4)
+11fo(w/ (0k,,(K),,,))) 81-1, do) wo
where fo neural network whose task to denoise the frame 1ーー v given different noisy frame yi- vi= and given pose $t, St-1. The À the log signal-to- toraal-to-ooi---aueee noise-ratio function dependent on k. and is single image condition that determines the appearance of target person. The notation dy, dp are learnable positional encoding vectors for distinguishing temporal direction. Following the method used in Ramesh al. 2022), adapt our model to predict clean images instead of noise.
4

---

Published as a conference paper at ICLR 2024
fot a sm-)) To T
Multi-head Attention
Weight sharing
f Prediction
Noise ratio
Ep
Eg
St
4. Direction condition
Appennance
Spatioiznpponl
(a) Overall architecture of BTU-Net
(b) Bidirectional Attention Block
Figure 4: The illustration of (a) our BTU-Net and (b) bidirectional attention block. The dotted squares in (a) represents bidirectional attention block. The small blue and pink squares in (a) indicate the intermediate feature of E, and Eg, respectively.
3.2 BIDIRECTIONAL TEMPORAL U-NET
To enable BTDM, we construct a Bidirectional Temporal U-Net (BTU-Net) by modifying the U-Net architecture, shown in Figure This architecture consists of network, E., that encodes single image condition c; another network, Flp. that encodes poses corresponding to L- and I; and a pair of U-Nets, fe. that accept أاا and y as input and predict the denoised human images temporally in both forward and backward directions. The multi-scale intermediate features are modulated by pose features, noise ratio X, and temporal direction vectors (dy and dy) using existing Feature-wise Linear Modulation layer (FiLM) Perezz et al. (2018). This pair of U-Nets shares weights and applies attention between the features encoded by Eg and intermediate features of fe. as shown in the bidirectional attention block Figure 4(b), which composed of appearance and spatiotemporal block. Each block utilizes multi head attention Vaswani et al. (017))
Perez
Appearance Block The yellow box in Figure 4b), the appearance attention block is composed of parallel multi-head attention mechanisms with weight sharing. For the multi-head attention, fea- tures from times and 1 of fa are used as the query, while the appearance features of E, are employed as the key and value. This block is specifically designed to learn the correlation between the appearance features encoded by E, and the intermediate features of fe. Through the multi-head attention mechanism, it attends to global context correlations, facilitating the effective transfer of appearance features to So even during rapid actions (those with substantial pose changes relative to frame intervals). This, in turn, enables the generation of novel poses effectively.
Spatiotemporal Block The spatiotemporal attention block takes the output feature pairs from the appearance attention block inputs and, illustrated in Figure performs cross attention. Such a structure effectively enables the learning of the temporal correlations of spatial features between times and Furthermore, by making the features for generating t dependent on f 1 and vice versa, temporally bidirectional structure is achieved. This design facilitates the efficient learning of temporal correlations.
We adopt the bidirectional attention block for the feature at specific resolutions, ie.,, 32 16x x 16, and More details on the architecture can be found in Appendix B.3
3.3 TRAINING AND INFERENCE FOR VARIOUS TASKS
Single Image Animation Our BTDM, trained on multiple videos, can be directly applied to gen- erate realistic human animation results for unseen people and poses. Similar to existing one-shot generation methods Wang et al. 2018),
further fine-tune our BTU-Net the given single image to enhance the visual quality. For this, the conditioning image c is set as a single image sequence Y = [c} and the pose sequence S = {g(c)), where g(-) is pose estimation function (e.g. Dense- Pose). This setup aligns with the training process outlined in Equation 4
Person-Specific Animation Our method be applied to the task of generating novel animations by training a single person's video. To adapt our method to this task, we train our BTDM framework using the objective function from Equation excluding the image condition
5

---

Published as a conference paper at ICLR 2024
Unconditional animation Moreover, our method facilitates the creation of temporally consistent an- imations without any appearance-relatee conditions. For such unconditional generation, we trained model with the condition set to @ of Equation 4
During the inference stage, to effectively utilize our BTDM, we employ a bidirectional recursive sampling method across all tasks. More details about this method can be found in Appendix
EXPERIMENTS
We validate our bidirectional temporal diffusion model on two tasks: generating human animation from single image and generating human animation by learning from a person-specine video. We also show that our model can generate diverse human animation with an unconditional setting (i.e., generating human animation from random noise).
4.1 SINGLE IMAGE ANIMATION
Dataset We use two datasets that can effectively validate the quantity and quality of temporal co- herence in the generated human animation. 1) Graphics simulation: for quantitative evaluation, we construct a high-quality synthetic dataset using graphics simulation tool for soft 3D clothing an- imation Reallusion (b) which provides perfect ground truth data for the motion transfer task (i.e, different people in the exact same motion, which does not exist from real-world videos) with phys- ically plausible dynamic clothing movements, The dataset includes a total of 80 training videos and 19 testing videos, each of which lasts 32 seconds at 30 FPS. We customize the 3D human appearance using CharacterCreator Reallusion (a). and we use Mixamo motion data Adobe for an- imation. The pose map is obtained by rendering the IUV surface coordinates of 3D body model (i.e., SMPI[Loper et al. (2015)) Please see the appendix [ for more details of our graphics simu- lation data. 2) UBC Fashion dataset /Zablotskaia et al. (2019): it consists of 500 training and 100 testing videos of individuals wearing various outfits and rotating 360 degrees. Each video lasts ap- proximately 12 seconds at 30 FPS. We apply DensePose Güler et | (2018) to obtain pose UV maps. We use this dataset for the qualitative demonstration on real images since it does not provide ground truth data with the exact same motion.
Baselines We compare our method to existing unidirectional temporal models: Thin-Plate Spline Motion Model for Image Animation (TPSMM) Zhao & Zhang (2022) and Motion Representations for Articulated Animation (MRAA) Siarohin et al. (2021) are designed to predict forward optical flow to transport the pixel from a source to target pose, following a rendering network. Both methods were trained on each dataset from scratch using the provided scripts and recommended training setup. All methods are trained at a resolution of 256x256.
Metric To evaluate the quality of the generated human animations, we employ five key metries: 1) SSIM (Structural Similarity Index) Waang et al.] (2004): quantifies the structural similarity be- tween the generated and ground truth images based on local patterns of pixel intensities and con- trast spaces. 2) LPIPS (Learned Perceptual Image Patch Similarity) Zhang et al. (2018): cogni- tive similarity between synthesized images and ground truth images by comparing the perceptual features extracted from both, utilizing a pre-trained deep neural network. 3) tLPIP (Tempo- ral Learned Perceptual Image Patch Similarity) Chu et al. (2020): extends the LPIPS measure to temporal domain, evaluating the plausibility of change across consecutive frames. It is defined as tLPIPS ||LPIPS( 11) LPIPS 91 where y and g represent the synthesized and ground truth images, respectively. 4) tOF Chu et al. (2020): pixel-wise difference of the es- timated optical flow between each sequence and the ground truth. 5) FID (Fréchet Inception Dis- tance) Heusel et al. (2017): measures the distance between the distributions of synthesized and real images the feature space of pre-trained Inception network.
Result The quantitative results for the graphics simulation data are presented in Table Our BTDM method outperforms other methods in all metrics. As can be seen in Figure our method closely resembles the source image, and the appearance changes depending on the movement are more realistic than other baseline methods. TPSMM and MRAA undergo significant artifacts such as texture distortion and blur due to errors in the forward optical flow prediction. In particular, the models from baseline methods highly confuse on the motion with large dynamics. The same trend is observed in the UBC Fashion data. Specifically, when the appearance of driven video significantly

---

Published as a conference paper at ICLR 2024
Table 1: Quantitative results for single image animation tested simulation data.
Methods
SSIMt
LPIPSL
tLPIPS
tOF
FID
MRAA Siarohin et al. 2021)
0.894
0.140
0.011
13.58
67.68
TPS Zhao & Zhang 2022)
0.915
0.077
0.005
11.92
48.76
Ours
0.958
0.036
0.003
8.93
11.14
Source Image
Source Image
Ground Truth
Driving Pose
MRAA
MRAA
TPSMM
TPSMM
Ours
Ours
Figure 5: Qualitative comparisons for the single image animation task on graphics simulation (left)
and UBC Fashion data (right).
differs from the source image in TPSMM or MRAA methods, abnormal artifacts often occur such as the loss of identity. Moreover, our method is found to preserve fine details considerably better.
4.2
ANIMATION
Dataset To evaluate the performance of our method in the task of person-specific animation, we use five videos from Yoon et al. (2022). Each video comprises between 6K and 15K frames, featur- ing person performing a diverse range of dynamic actions. The pose UV map obtained using DensePosdGüler et al. (2018).
Baseline We compare our method to V2V Wang et al. (0018), EDN Chan et al. (2019), HFMT Кappel et al.] (2021), DIW WWan et al. (2021). and MDMT Yoon et al. (2022), which utilized generative network in a temporally unidirectional way. All methods were trained on the training set of each video and evaluated on the test set.
Metries We use SSIM, LPIPS, and tLPIPS as used in Section
Result The evaluation results for method and the baselines on the test sequences from the five videos are displayed in Table 2 Our approach exhibits a performance that is either comparable to or surpasses that of other state-of-the our model can synthesize the background, we only evaluate the quality of the foreground synthe- sis for consistent and fair comparison across baseline methods where existing segmentation
methods in the LPIPS and tLPIPS metrics. Note that, while

---

Published as a conference paper at ICLR 2024
Table 2: Quantitative results of person-specific animation. Each of the three values in the order of ) 10?, x 10', and SSIM(1)x 10, respectively. The number of images used for training indicated in parentheses, e.g., (6K).
Methods
Data (6K)
Data 2 (10K)
Data 3 (10K)
Data 4 (15K)
Data 5 (15K)
Average
V2V
1.84/2.5/99699
3.03/3.83/9.66
11.51/3.80/9 80//3880/9.05
2.06/2.98/9440
4.01/4.04/9.44 499
4.69/3.52/9.44
EDN
2.74/3.86/9.55
3,98/5.400 40/9.40/
13. 12/44.5//8..9
4.90/5.09 21550099.22 .22
5.00/4.82/9 01/44.82/9.34 .34
5.95/4.74/9.33
HFMT
3.68/4.41/9.48
6.39/8.54/9.26
13.27/4.62/8.91
6.08/3.22/9.11
0.86/4
7.26/5.06/9 06665.06/9.19
DIW
1.83/2.88/9.68
2.70/4.11/9.6
11.89/4.09/9.00
2.83/4.66/9.45
4. 4.14/5. 01/5.2009.44
4.68/4.19/9
MDMT
1.76/2.58/9.73
2.68/3.77/9.0 65
1..48/3.1//9111
2.81/2.86/9 45
3.81/4.12/9.50
4.31/3.29/9
Ours
1.90/2.83/972
2.91/3.76/9.65
10.32/2.52/9.28
2.78/2.85/9.57
4.07/4.26/9.51
4.39/3.22/9.55
Dota
Dana
Dana 3
Data
V2V
EDN
HFMT
MDMT
Oars
Figure 6: Qualitative comparison between the state-of- sine-off-hhe tteh-arrtt methods and our approach for the task of generating person-specifie human animation.
method Gong et al. (208)) to remove the background. for evaluation. Specifically, our method out- shines all others the SSIM evaluation with diffusion-basse generative framework. The highest average score implies that our method performs consistently better than other methods in terms of temporal coherence and visual plausibility across assorted appearance and motion styles.
Further qualitative results are demonstrated Figure 6 where the baseline methods often lose con- text become blurred in complex poses, leading physically implausible human animation. Our method demonstrates robustness to dynamic movements and strong temporal coherence, yielding clear and stable results. Please also refer the demo video.
4.3 ABLATION STUDY
To evaluate the effect of the module in method, perform an ablation study. The quantitative results are in Table 3 and please refer to the supplementary materials for visual comparison.
Unidirectional vs. Bidirectional We compare our bidirectional temporal diffusion model with the unidirectional one. For this, we trained the same BTU-Net a unidirectional manner using the same loss. The bidirectional approach demonstrates far more spatiotemporal consistency based on ILPIPS in Table 3 Our main observation is that due to significant motion-appearance ambiguity, the generated texture sometimes diverges at the end of the frame under highly dynamic human
8

---

Published as a conference paper at ICLR 2024
Table 3: Quantitative results ablation study with graphics simulation data.
Methods
SSIM
LPIPS
tLPIPS
Ours w/ Unidirectional Model
0.937
0.052
0.005
Ours
0.958
0.036
0.003
y
Figure 7: Qualitative results unconditional animation generation with UBC Fashion data. The top row shows various people generated unconditionally, while the bottom row displays samples of the generated sequences.
movements. Based on the improvements in LPIPS, we can notice such strong temporal coherence helps with improving the visual quality as well.
Number of Images for Minn-Tunngg For single-image animation, we fine-tune the model on single image. We found that the quality improves the number of fine-tuning images increases. In the supplementary material, we introduce the experiments about the impact of the number of fine-tuning images on rendering quality.
we demonstrate that our method can erate human animation with diverse appearances without conditioning any images or videos.
Unconditional Human Animation Generation Figure 7
CONCLUSION
We introduce new method to synthesize temporally coherent human animation from single im- age, video, random noise. We address the challenge of temporal incoherence from isting generative networks that decode future frames in an auto-regressive way. We argue that such unidirectional temporal modeling of generative network involves significant amount of motion- appearance ambiguity, leading to the artifacts such as texture drifting. We suppress the motion- appearance ambiguity by newly designing bidirectional temporal diffusion model (BTDM): de- noising network progressively removes temporal Gaussian noises whose intermediate results are cross-conditioned over consecutive frames, which allows conditioning locally and globally coherent motion context on our video generation framework. We perform the evaluation on two different tasks, i... human animation from a single image and person person-specifer human animation, and demon- strate that BTDM shows strong temporal coherence, which also helps to improve the visual quality, compared to existing methods.
Limitation While BTDM produces temporally coherent human animations, there exist several lim- itations. Since our model generates the video as a function the estimated body poses, the errors in the pose estimation affect the rendering quality, e.g., the misdetection of hands produces some appearance distortion around the hand. Due to the inherent ambiguity of 2D pose representation, method sometimes shows weakness in the sequence with 3D human rotations. Our potential future work to improve 3D awareness and completeness by utilizing a complete 3D body model, e.g., SMPL. Loper al. 2015), in our bidirectional temporal diffusion framework.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENT
This work was supported by Institute of Information communications Technology Planning Evalua- tion (ITTP) grant funded by the Korea governmen((MSIT) NN..RS2023-002256300 Development of Artificial Intelligence for Text-based 3D Movie Generation).
REFERENCES
Adobe. Adobe mixamo. Mttpss//www.mixxeo.ccom www.:mmixcc..ommmmm 47
Sanjeev Arora, Constantinos Daskalakis, and David Steurer. Message passing algorithms and im- proved Ip decoding. In Proceedings of the forty-first annual ACM symposium on Theory of com- puting, pp. 3-12, 2009.
Guha Balakrishnan, Amy Zhao, Adrian V Dalca, Fredo Durand, and John Guttag, Synthesizing images of humans in unseen poses. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8340-8348, 2018.
Caroline Chan, Shiry Ginosar, Tinghui Zhou, and Alexei A Efros. Everybody dance now. In Pro- ceedings of the IÉFF/CVE international conference on computer vision, pp. 5933-5942, 2019.
Mengyu Chu, You Xie, Jonas Mayer, Laura Leal-Taixé, and Nils Thuerey. Learning temporal coher- ence via sulf-supervision for gan-based video generation. ACM Transactions on Graphics (TOG), 39(4):75-1, 2020..
Patrick Esser, Ekaterina Sutter, and Bjöm Ommer. A variational u-net for conditional appearance and shape generation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8857-8866, 2018.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germani- dis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 7346-7356, 2023.
Pedro Felzenszwall and Ramin Zabih. Dynamic programming and graph algorithms in computer vision. IEEE transactions on pattern analysis and machine intelligence, 33(4):721-740, 2010.
Ke Gong, Xiaodan Liang, Yicheng Li, Yimin Chen, Ming Yang, and Liang Lin. Instance-level hu- man parsing via part grouping network. In Proceedings of the European conference on computer vision (ECCV), pp. 770-785, 2018
Riza Alp Güler, Natalia Neverova, and lasonas Kokkinos. Densepose: Dense human pose estimation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7297-7306 2018.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint N0Kv::2307.04755, 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6857 2020.
Jonathan Ho. William Chan, Chitwan Saharia, Jay Whang. Ruigi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint 10.02322 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David Fleet. Video diffusion models. arXiv preprint arXiv:2204.034588 2022h.
10

---

Published as a conference paper at ICLR 2024
Zhichao Huang, Xintong Han, Jia Xu, and Tong Zhang. Few-shot human motion transfer by person alized geometry and texture modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2297-2306, 2021.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134 2017.
Moritz Kappel, Vladislav Golyanik, Mohamed Elgharib, Jann- -le Henningson, Hans-Peter Seidel, Susana Castillo, Christian Theobalt, and Marcus Magnor. figh-fidelity neural human motion transfer from monocular video. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 1541-1550, 2021.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings the recognition, pp. 4401-4410, 2019.
the IEEE/CVF conference on computer vision and pattern
Bahjat Kawar, Michael Elad, Stefano Ermon, and Jiaming Song. Denoising diffusion restoration models. Advances in Neural Information Processing Systems, 35:23593-23606, 2022.
Yining Li, Chen Huang, and Chen Change Loy. Dense intrinsic appearance flow for human pose transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 3693-3702, 2019.
Lingji Liu, Weipeng Xu, Michael Zollhoefer, Hyeongwoo Kim, Florian Bermard, Mare Habermann, Wenping Wang. and Christian Theobalt. Neural rendering and reenactment of human actor videos. ACM Transactions on Graphics (TOG), 38(5):1-114)) 2019a.
Wen Liu, Zhixin Piao, Jie Min, Wenhan Luo, Lin Ma, and Shenghua Gao. Liquid warping gan: A unified framework for human motion imitation, appearance transfer and novel view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5904-5913, 2019b.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM transactions on graphics (TOG), 34(6):1-16, 2015.
Natalia Neverova, Riza Alp Guler, and lasonas Kokkinos. Dense pose transfer. In Proceedings of the European conference on computer vision (ECCV), pp.
2018.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8111, 2021.
Nvidia. Nvidia omniverse. https: http://wwwwwwweeececccooo//o///a
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2.18.
Sony Pictures. Lucasfilm and sony pictures imageworks release alembic 1.0. Sony Pictures Image- works, Lucasfilm (August 9, 2011).
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasom Suwajanakom. Dif- fusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10619-10629, 2022.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine PMLR, 2021.
Learning. pp.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arX1v:2204.06125, 2022.
Reallusion, Character creator. http:://wwpp......aaaaaaaaaaaaaa

---

Published as a conference paper at ICLR 2024
Reallusion. http:///w.wp//....aaaaaaooooooo.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- tion Processing Systems, 00266479-64494 2022a.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Puttern Analysis and Machine Intelligence, 2022b.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations, 2021.
Kripasindhu Sarkar, Lingji Liu, Vladislav Golyanik, and Christian Theobalt. Humangan: A genera- tive model of human images. In 2021 International Conference on 3D Vision (3DV), pp. 258-267. IEEE, 2021.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in Nearul Information Processing Systems, 32, 2019.
Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion rep- resentations for articulated animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 13653-13662, 2021.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209. 14792, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffiusinn implicit models. In Interna- tional Conference on Learning Representations, 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representattins, 2020b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention you need. Advances in neural informa- fion processing systems, 30, 2017.
Ting- Chun Wang, Ming- Yu Liu, Jun Yan Zhu, Guilin Liu, Andrew Tao, Jan Kautz, and Bryan Catan- zaro. Video-to-video synthesis. arXiv preprint arXiv: 06601, 2018
Ting-Chun Wang, Ming- Yu Liu, Andrew Tao, Guilin Liu, Bryan Catanzaro, and Jan Kautz. Few- shot video-to-video synthesis. Advances in Neural Information Processing Systems, 32, 2019.
Tuanfeng Y Wang. Duygu Ceylan, Krishna Kumar Singh, and Niloy Mitra. Dance in the wild: Monocular human animation with neural dynamic appearance synthesis. In 2021 International Conference on 3D Vision (3DV), pp. 268 688-277. IEEE, 2021.
Yinhuai Wang, Jiwen Yu, and Jian Zhang. Zero-shot image restoration using denoising diffusion null-space model. In The Eleventh International Conference on Learning Representations, 2022.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4))600- 612, 2004.
12

---

Published as a conference paper at ICLR 2024
Daniel Watson, William Chan, Ricardo Martin Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view synthesis with diffusion models. In The Eleventh International Conference on Learning Representations, 2022.
Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video generation. Entropy, 25(10):1469, 2023.
Jae Shin Yoon, Duygu Ceylan, Tuanfeng Wang, Jingwan Lu, Jimei Yang, Zhixin Shu, and Hyun Soo Park. Learning motion-dependent appearance for high-fidelity rendering of dynamic humans from a single camera. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3407-3417, 2022.
Polina Zablotskaia, Aliaksandr Siarohin, Bo Zhao, and Leonid Sigal. Dwnet: Dense warp-based network for pose-guue gose-guided human video generation. arXiv preprint 190.0999, 2019.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3657-3666, 2022.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018 2022.
Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara Berg. Dance dance generation: Motion transfer for internet videos. In Proceedings of the IEEE/CVF International Conference Computer Vision Workshops, pp. 0-0, 2019.
A BIDIRECTIONAL RECURSIVE SAMPLING
To effectively utilize our Bidirectional Temporal Diffusson Model (BTDM) during the inference stage, we have employed a bidirectional recursive sampling method, which proceeds as follows.
Algorithm Bidirectional Recursive Sampling
Input: Initial noisy inputs y {ur .. VE }, driven pose sequence S (81, 5T] ST Output: Denoised animation y (vf.... ...
for l K 1 to 0 step -1 do
K - 1 is odd then
Direction: Forward for 1 = to do
y1 folut 1
end for
else
Direction: Backward
for to step -1 1 do 5ー fo(yt vt A(k), s.-1 s,
end for end if end for
Although it's possible reverse the entire sequence (starting in the backward direction and then moving to the forward, followed by backward again), we observed no significant differences in the outcomes between these two cases.
13

---

Published as a conference paper at ICLR 2024
B IMPLEMENTATION DETAILS
B.1 SINGLE IMAGE ANIMATION
Our method is trained resolution of 256x256, similar to all other methods. We generates 64x64 animations via the BTU-Net, which are subsequently upscaled to 256x256 using the SR3 Saharia et al. (2022b). We trained both the BTU-Net and SR3 from scratch the entire training dataset, for 50k and T0Ok iterations with batch size of 32, respectively. We set the denoising step to K = 1000 and the learning rate to le-5, During testing, we fine-tune model with appearance condition for 300 iterations with learning rate of le-5, It should be noted that we employ K 50 at test time for expedited generation.
B.2 PERSON SPECIFIC ANIMATION
The training settings for the BTU-Net and SR3 Saharia et al. (2022b) are identical those used in the Single image animation setup, with the exception that both the BTU-Net and SR3 Saharia et al. 2022b) are trained for 100 epochs each without fine-tuning.
B.3 BTU-NET ARCHITECTURE
The detailed structural information of the BTU-Net's layers is illustrated in Figure 8 Due to the complexity the arrows, the input directions for Ea and Ep have been omitted. Directions are provided in Figure of main manuscript. The notation 'xdigit" below the dashed block indicates how many times that block structure is repeated.
TRADEOOR
EP
F
Resnet block
Spotideeppral block
FiLM
Appesrance Aftertion block
Figure 8: Architecture of BTU-Net.
B.4 TRAINING INFERENCE SPEED ANALYSIS
In partieular, training the BTU-Net and SR3 models using the UBC fashion dataset requires 15 and 30 epochs, respectively, on a setup of four A100 GPUs, completed within 67 hours. Fine-tuning these models for 300 iterations takes approximately 210 seconds. During inference, processing each frame takes roughly 1.4 to 1.9 seconds on single A100 GPU.
C ABLATION STUDY
C.1 COMPARISON OF BIDIRECTIONAL AND UNIDIRECTIONAL APPROACHES
Figure 9 demonstrates the qualitative results of bidirectional and unidirectional temporal training via our BTU-Net. The unidirectional approach struggles to generate images fitting the pose con- dition, tending to replicate the texture of the front image input the condition instead. Unlike the unidirectional approach, the bidirectional model successfully creates images that meet the pose condition.
14

---

Published as a conference paper at ICLR 2024
Bidirectional
Unidirectional
Figure 9: Qualitative comparison of Bidirectional and Unidirectional methods.
UPIPS
ELPIPS
0.58
0.065
0.0032
0.95
0.003
0.96
0.95
0.05
0.0028
0.54
0.55
0.0025
0.93
0.04
0.81
0.01
0.025
0.89
0.0018
30
Gopphii
Ce-Crophi -HHLIRC
Figure 10: Quantitative comparison of the number of images used for fine-tuning, "Graphic" and "UBC" notates graphic simulated data and UBC fashion data results respectively.
C.2 NUMBER OF IMAGES FOR FINE-TUNING
We also evaluate the performance depending on the number of images used for fine-tuning. The performance comparison results are shown in Figure [0 and Figure I1] As the number of images used for fine-tuning increases, performance improves across various metrics. Notably, the trend between the increase in image count and metric scores is not linear but shows signs of convergence.
C.3 USER STUDY
We conduct quantitative results of an user study in which people evaluated videos generated by our method and baseline methods. A total of 42 participants took part in this study, which involved tasks for single image animation and person specific animation. Each evaluation required participants to watch comparison videos at least twice and make selections based on two questions: "Which video preserves the identity best?" and "Which video looks most realistic to you?". For the person specific animation task, the experiment can be seen that our BTMD results are much more realistic and maintain identity better when evaluated by people, compared other baseline methods.
conducted excluding the first question. As shown in Figure 12]
15

---

Published as a conference paper at ICLR 2024
AN
AN
Ground Truth
Image
Images
5 Images
10 Images
Figure 11: Qualitative comparison of the number of images used for fine-tuning.
Which video preserves the identity best?
Which video looks realistic you?
Which video looks realistic to
16
79.64%
27.03M
11.84%
18:53%
16.
144%
MRAA
TPSMM
Cuss
MRAA
TPSMM
Ous
V2V
EDN
HFMT
DIW
MDMI
Chrss
(a)
(b)
(c)
Figure 12: Quantitative result human evaluations. Graphs (a) and (b) represent the results for the single image animation task, showing the proportion of choices made for two different questions. Graph (c) shows the results for the person-specific animation task.
C.4 UNCONDITIONAL ANIMATION GENERATION
We demonstrate that method generate human animations featuring diverse clothing styles and identities, even without any image conditions. Results from unconditional generation experi- ments both datasets illustrated in Figures and 14
16

---

Published as a conference paper at ICLR 2024
Figure 13: Sample sequences from unconditional generation on UBC fashion data.
D GRAPHIC SIMULATED DATASET
Graphic simulated dataset comprised of approximately 98,000 images, each rendered reso- lution of 512x512. These images illustrate various dynamic movements (such as dance, exercise, etc.) of 3D human models with a total of 99 different appearances. The 3D human models this dataset are created using Character Creator 4 |Reallusion (a), and we simulate the soft cloth motion in iClone8 Reallusion (b). Mixamo Adobe human motions, are exported as an Alembic Pictures file. For realistic rendering, employ Ray Tracing Texel (RTX) rendering and the Nvidia Omni- verse Nvidia as the rendering tool. Figure 15 shows a few samples from our graphically simulated data.
17

---

Published as a conference paper at ICLR 2024
Figure 14: Samples from unconditional generation on Graphic simulated data.
E MORE VISUAL RESULTS
E. SINGLE IMAGE ANIMATION
Additional visual results conducted with graphic simulated and UBC fashion data for the single image animation task are shown in Figures 17 and T8
E.2 PERSON-SPECIEE ANIMATION
For fair comparison evaluation the person-specific human animation task, we evaluate the results using only the foreground. However, our method is capable of background synthesis, and the visual results are shown Figure T6
18

---

Published as a conference paper at ICLR 2024
Figure 15: Samples of Graphic Simulation Dataset
Figure 16: Samples from person-specific animation results with background.
19

---

Published as a conference paper at ICLR 2024
Figure 17: More single image animation results on Graphic simulated data.
20

---

Published as a conference paper at ICLR 2024
Figure 18: More single image animation results on UBC fashion data.
21


Reviews:
[{"review_id": "eMevanpDZL", "reviewer": "Reviewer_VxE3", "summary": "The paper introduces a method that generates animated human videos. Depending on the training protocol, the proposed method can generation human animations either unconditionally, or conditionally from images (single image animation) or videos (Person-specific animation). Compared to previous works that relies on autoregressive generation of frames, this paper proposes a bidirectional diffusion model, achieving better quality. The bidirection model takes two noisy frames as input, denoising one of them according to the input embedding that controls the direction.\nTo facilitate the evaluation of the model, a synthetic human animation dataset containing different characters having the same motion is built.", "questions": "* The BTU-Net described in the paper has a temporal window of 2. Would increasing the number of frames further improve the performance? \n* How is the proposed model inferenced? Do you take the average of the results from both directions?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The proposed method outperforms prior art on a significant number of benchmarks.\n* The paper also designed a new, high quality synthetic dataset for better evaluation of the model. This dataset, if released, can be of great help to future works.", "weaknesses": "* The proposed bidirectional model is not new. It closely resembles a video diffusion model with temporal cross attention -- a standard technique used in video generation.\n* The writting of the paper could be improved. For example, it is unclear how the two directions of the bidirectional model is combined during inferenced. This information is very important for the understanding of the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N9FkmsruDB", "reviewer": "Reviewer_mGPz", "summary": "This paper proposes a novel method for human animation from a single image, a video or a random noise. \n- The main contribution is bidirectional temporal modeling which can generate temporally coherent videos.\n- Both qualitative and quantitative results show the effectiveness of proposed method.", "questions": "- How does the paper compare with first order motion model (Neurips 2019) which has been a strong baseline in image animation?\n- How long does it take to generate a video in the experiments? Is it time consuming using a bidirectional diffusion model?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper is well written and easy to understand.\n+ The bidirectional temporal modeling is novel and effective in improving the quality of generated videos.\n+ The proposed bidirectional diffusion process to overcome overfitting and the bidirectional attention block are both reasonable.\n+ The experiments are sufficient and demonstrate the effectiveness of the proposed method.", "weaknesses": "- The datasets used for experiments are simple because the background is white. It is not clear how the proposed method works on real world scenario where the background is complicated.\n- Table 1, tOF does not follow the format of other metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JB0EDAt1GV", "reviewer": "Reviewer_rYhA", "summary": "This paper aims to generate temporally coherent human animation from a single image, a video or a random noise. The authors propose the bidirectional temporal modeling to improve the temporal consistency, which is different from existing unidirectional formulation. Specifically, a diffusion based generative network is designed to decode the human appearance in both forward and backward direction, where  intermediate features are cross-conditioned over time. Experimental results on different generation tasks show better performance of the proposed method compared with existing unidirectional based methods.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposes a novel human animation framework, which models the human appearance bidirectionally to improve the appearance consistency within a video sequence.\n2. The proposed method achieves better performance and generalization capacity compared with existing works.", "weaknesses": "1. The results in Table 2 are not consistent with the reported results in MDMT. It would be better to explain the reason.\n2. The main objective of the paper is to improve the temporal appearance coherence of the generated video sequence. The reviewer is wondering how is the proposed bidirectional modeling compared to the texture inversion [A, B] technique in keeping the appearance consistent. \n[A] Nataniel Ruiz et al. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation.\n[B] Rinon Gal et al. An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion.\n3. Related works on diffusion model based video generation are missing, such as NUWA, Make-A-Video, MagicVideo and so on, especially discussions on how these works enforce the temporal consistency.  \n4. The intuition on how the proposed bidirectional temporal modeling solves the motion-appearance ambiguity problem needs to be further explained. \n5. Typo in the line above Eq.(4), should be p(y_(t-1)/y_t) p(y_(t-1)/p_t)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TsIzNvnhTU", "reviewer": "Reviewer_xCTh", "summary": "In this paper, the authors design a new human animation framework using a denoising diffusion probabilistic model, leveraging bidirectional temporal modeling and a denoising diffusion model to enhance temporal coherence and reduce artifacts in the generated animations. They proposed a bidirectional temporal diffusion model that can generate temporally coherent human animation from random noise, a single image, or a single video. They introduced a feature cross-conditioning mechanism between consecutive frames using recursive sampling to enable local Embed action background information in a globally consistent manner.", "questions": "Please see the questions above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The structure of the article is reasonable, clear, and easy to understand. The design of pictures and tables is rigorous.\n2. The method section and experiment settings are introduced in the details.", "weaknesses": "- Most of the references are before 2022, and there are no latest research results in 2023.\n- Too many arXiv papers are cited in the references. The publication information of the article should be indicated.\n- The paper mentions three tasks of generating human animation from a single noise, a single image, and a single video, but the quantitative results only include the single image animation.\n- More quantitative results and visual examples should be added.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)