DisPose: Disentangling Pose Guidance for Controllable Human Image Animation
Authors: Hongxiang Li
Yaowei Li
Yuhang Yang
Junjie Cao
Zhihong Zhu
Xuxin Cheng
Long Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion Model; Controllable Video Generation; Human Animation
Primary area: generative models
TL;DR: We present DisPose, a plug-and-play module for improving human image animation, which aims to provide efficient conditional control without additional dense inputs.

Abstract: Controllable human image animation aims to generate videos from reference images using driving videos. Due to the limited control signals provided by sparse guidance (e.g., skeleton pose), recent works have attempted to introduce additional dense conditions (e.g., depth map) to ensure motion alignment. However, such strict dense guidance impairs the quality of the generated video when the body shape of the reference character differs significantly from that of the driving video. In this paper, we present DisPose to mine more generalizable and effective control signals without additional dense input, which disentangles the sparse skeleton pose in human image animation into motion field guidance and keypoint correspondence. Specifically, we generate a dense motion field from a sparse motion field and the reference image, which provides region-level dense guidance while maintaining the generalization of the sparse pose control. We also extract diffusion features corresponding to pose keypoints from the reference image, and then these point features are transferred to the target pose to provide distinct identity information. To seamlessly integrate into existing models, we propose a plug-and-play hybrid ControlNet that improves the quality and consistency of generated videos while freezing the existing model parameters. Extensive qualitative and quantitative experiments demonstrate the superiority of DisPose compared to current methods. Project page: https://github.com/lihxxx/DisPose.

Published as a conference paper at ICLR 2025
DISPOSE: DISENTANGLING POSE GUIDANCE FOR CONTROLLABLE HUMAN IMAGE ANIMATION
Hongxiang Li', Yaowei Li", Yuhang Vang",Junjj Cao", Zhihong Zhu', Xuxin Cheng', Long Chen
!Peking University TTiinghua University
University of Science and Technology of China 4 Hong Kong University of Science and Technology
Figure 1:
Our method demonstrates its ability to produce diverse animations and preserve consis- tency of appearance.
ABSTRACT
Controllable human image animation aims to generate videos from reference im- ages using driving videos. Due to the limited control signals provided by sparse guidance (c.g., skeleton pose), recent works have attempted to introduce addi- tional dense conditions (c.g., depth map) ensure motion alignment. However, such strict dense guidance impairs the quality of the generated video when the body shape of the reference character differs significantly from that of the driving video. In this paper, we present DisPose to mine more generalizable and effec- tive control signals without additional dense input, which disentangles the sparse skeleton pose in human image animation into motion field guidance and keypoint correspondence. Specifically, we generate dense motion field from a sparse mo- tion field and the reference image, which provides legion-level dense guidance while maintaining the generalization of the sparse pose control. We also extract diffusion features corresponding to pose keypoints from the reference image, and then these point features are transferred to the target pose to provide distinct iden- tity information. To seamlessly integrate into existing models, we propose plug- and-play hybrid ControlNet that improves the quality and consistency of gener- ated videos while freezing the existing model parameters. Extensive qualitative and quantitative experiments demonstrate the superiority of DisPose compared to current methods. Project page: httss//gihubb.co///ixxxx//////
INTRODUCTION
Controllable video generation (Zhang et al.) 2023b Yin et 2023 Wang al. 2024c) has gained increasing attention for its ability to customize videos based on user preferences. In particular,
Corresponding author.

---

Published as a conference paper at ICLR 2025
controllable human image animation (Hu et al. 2023 Wang et al. 2024b) has attracted significant interest due to its vast potential applications in art creation, social media, and digital humans Ma et al. (0118); Xu et al. (2024b); Ma et al.] (2017); Yang et al.] (2024b);
Xu et al. (2024c). It aims to animate static images into realistic videos based on driving videos. In contrast to other control- lable video generation methods (e.g., camera control (He et al. 2024
Wang al. (024c), motion control (Wu et al.] 2024] Waan et al. 2024c Li at. al. [024] Jiang et al. 2024 Niu et al. (024)), con-
trollable human image animation allows for more flexible motion regions, diverse motion paradigms, and complex character appearances. Introducing precise pose control in existing video generation methods is challenging, but has significant value in achieving the desired results. There are two chal-
lenges: (1) following the motion of the driving video, and (2) preserving the appearance information of the reference image.
The motion control signal is critical to drive the animation. Controllable human image animation usually utilizes the skeleton pose (Yang et al. 2023) as the control signal. Besides the fact that the skeleton pose is easy to obtain, the more important reason is that it is easier to adapt to different body shapes when the target body shape is significantly different from the reference image. However, the skeleton pose as sparse expression provides limited guidance information. To provide more trol signals, recent work (Zhu et al.] 2024: Xu et al., (2024a) has attempted to express human body geometry and motion regions by extracting various dense signals from the driving video, includ- ing DensePose (Güler et al., 2018). 3D human motion (Loper et al.] 2013) and depth map (Yang) ctt al., 20244)), etc. Unfortunately, these dense signals impose striet shape constraints on the gener- ated characters, they are more difficult to adapt to reference images with different body shapes (Jin al. 2024). Morcover, extracting dense signals accurately in complex motion videos inherently difficutt (Zhang et al.) 2024). These overly dense guidance techniques exacerbate pose estimation errors and thus impair generation quality. Therefore, the existing methods (Chang et al. 2023 Hu eet al. 2023; Zhu et al., 2024 Wang et al.) 2024b) are struggling to trade off generalizability and effectiveness between sparse and dense controls. would beneficial to mine more generalizable and effective control signals from the skeleton pose map instead of dense control inputs.
On the other side, preserving appearance consistency from complex motion is also extremely chal- lenging. Image-driven generation methods (Zhang et al. 2023a; Ye et al. 2023 Wang et al. 2024a) typically employ the CLIP (Radford et al. 2021) image encoder as a substitute for the text encoder to introduce low-level details of the image. Inspired by dense reference image conditioning, recent works (Hu et al.] 2023; Xu et al., (024a) opt to train an additional reference network that uses the same initialization the denoising network. The feature maps from the reference network are in- jected into the denoising network through the attention mechanism. This dual U-Net architecture significantly increases the training cost. Moreover, such dense reference image conditioning is in- effective for actions with body shape changes. Existing works neglect the fact that the keypoints of the sparse skeleton pose correspond to appearance characteristics. We argue that considering sparse skeleton pose keypoints as correspondences can provide effective appearance guidance while relaxing shape constraints.
To this end, we propose DisPose, a plug-and-play guidance module to disentangle pose guidance, which extracts robust control signals from only the skeleton pose map and reference image without additional dense inputs. Specifically, we disentangle pose guidance into motion field estimation and keypoint correspondence. First, we compute the sparse motion field using the skeleton pose. We then introduce reference-baste based dense motion field to provide region-level motion signals through con- dition motion propagation on the reference image. To enhance appearance consistency, we extract diffusion features corresponding to key points in the reference image. These point features trans- ferred to the target pose by computing multi-scale point correspondences from the motion trajectory. Architecturally, we implement these disentangled control signals in ControlNet-like (Zhang al.) 2023a) manner to integrate them into existing methods. Finally, motion fields and point embedding are injected into the latent video diffusion model resulting in accurate human image animation as shown in Figure.
[ The contribution of this paper can be summarized as:
We propose plug-and-play module for controllable human animation.
We innovatively disentangle motion field guidance and keypoint correspondence from pose con-
trol to provide efficient control signals without additional dense inputs.
Extensive qualitative and quantitative experiments demonstrate the superiority and generality of the proposed model.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Latent Imaag//Viee Diffusion Models. Diffusion-based models (Ho et al. 2020; Song et al. 2020
Rombach et al. 2022 Zhang et al. 2023a Zeng al. 2024) have achieved remarkable success in the fields of image generation and video generation. Due to reasons such as computational intensity and information redundancy, diffusion models directly on the pixel space are hard to scale up. The latent diffusion model (LDM) (Rombach et al. 2022) introduces a technique for denoising in the latent space, which reduces the computational requirements while preserving the generation quality. In contrast to image generation, video generation requires more accurate modeling for temporal motion patterns. Recent video generation models (Blattmann et al.] 2023b) [Ge et al.] 2023; Guo al. 2023) utilize pre-trained image diffusion models to enhance the temporal modeling capability by inserting temporal mixing layers of various forms.
Diffusion Models for Human Image Animation. Recent advancements in latent diffusion models have significantly contributed to the development of human image animation. Previous human image animation models (Wang al. 2024b Xu et al. 2024a Chang al. (023) followed the same two- stage training paradigm. In the first stage, the pose-driven image model is trained on individual video frames and corresponding pose images. In the second stage, the temporal layer is inserted to capture temporal information while keeping the image generation model frozen. Based on this stage training paradigm, Animate Anyone (Hu al. [023) utilizes ReferenceNet with UNet architecture to extract appearance features from reference characters. With the development of video diffusion modeling, recent work (Zhang et al., 2024) has directly fine-tume munet Stable Video Diffusion (SVD) (Blattmann) et al. 2023a) to replace two-stage training. To prove the effectiveness of the proposed method, we integrate DisPose on both paradigms.
Pose Guidance in Human Image Animation. Human image animation typically uses the skeleton pose (e.g., OpenPose (Cao et al., (017)) as the control guide. pose (Yang et al.] (023) stands out as an augmented alternative to OpenPose (Cao al., 2017) since it provides more accurate and expressive skeletons. Recent work has focused on introducing dense conditions to enhance the quality of the generated video. MagicAnimate (Xu et al. 2024a) uses DensePose (Güler et al.) 2018) instead of skeleton pose, which establishes a dense correspondence between RGB images and surface-based representations. Champ (Zhu al., 2024 renders depth maps, normal maps, and semantic maps from SMPI. (Loper et al. 20155) to provide detailed pose information. However, these overly dense guidance techniques rely too much the driving video and generate inconsistent results when the target identity is significantly different from the reference. In this paper, we pro- pose a reference-based dense motion field that provides dense motion signals while avoiding strict geometric constraints.
3 PRELIMINARY
We choose Stable Diffusion (SD) as the base diffusion model in this paper since it is the most popular open-source model and has a dell-deeeloped community. SD performs the diffusion process in the latent space of pre-trained autoencoder. The input image I is transformed into a latent representation 2e E(T using a frozen encoder E(). The diffusion process involves applying a variance preserving Markov process to zo, where noise levels increase monotonically to generate diverse noisy latent representations:
Z, Va + v1 &Ge, (0,
where / -= 1,... T denotes the time steps within the Markov process, where T is commonly con- figured to 1000, and ae represents the pre-defined noise intensity at each time step. The denoising network (- learns to reverse this process by predicting the added noise, encouraged by the mean squared error (MSE) loss:
= ==E E.( [IE Eg (ze, t, )rexx))12
where Ctext is the text embedding corresponding to I. The denoising network ca(-) )) is typically implemented U-Net (Ronneberger et al. 2015) consisting of pairs of down/up sample blocks at four resolution levels, as well as a middle block. Each network block consists of ResNet (He et al. 2016), spatial self-attention layers, and cross- aateenatntnionn layers that introduce text conditions.
3

---

Published as a conference paper at ICLR 2025
Notion Field Estimation
Gai User Input
(b) Existing Aalmation Midel
Proose
Lids 243
SEAR
Denocsing
Touk
Ribbring
CMP
ponn
XOube bxtue
Feature Extrctionn
(1) Print Festure Extradion
"oetinn Encoder
F
Eccode:
Apace tastiis
Bcok ENEAO
poupe
Hotrovid
Coced
Point Eccodes
MAMM
Pood Enbodinn
Suaee
Figure 2: The overview of proposed DisPose.
4 DISPOSE
Given a reference image Lro E X and a driving video V X The core of our method is disentangle efficient control guidance from only skeleton poses and reference images as shown in Figure which can be applied to existing human animation methods without additional dense inputs. We first introduce sparse and dense motion field guides in Sec.[4.] Then, we introduce reference-basee based keypoint correspondence in Sec. 4.2 Finally, we introduce the pipeline of hybrid ControlNet in 4.3
4.1 MOTION FIELD GUIDANCE
Sparse Motion Field. We first estimate the skeleton pose by DWpose (Yang et al. 2023) obtain each frame's s human key point coordinates. Subsequently, the key points of the reference image are used starting points to track the motion displacement of all frames and represented
Ptref ((ah vh) k 1.. K, n O...N N}. where Prraj denotes the trajectory map of the key point k overall N frames and n = denotes the reference image. We calculate the track matrix Pa as follows:
P. = -
- 2 - In
(3)
where K denotes the number of keypoints, N denotes the number of frames, P. denotes the trajec- tory map of keypoint k over all N framess and n = 0 denotes the reference image. To avoid training instability caused by overly sparse trajectory matrice, we then apply Gaussian filtering to enhance P. to obtain the sparse motion field (N)
Dense Motion Field. Considering that sparse control provides limited guidance and dense control hard to obtain during inference, we transform dense guidance into the motion propagation from the reference frame the target pose, instead of the dense signal of the target pose. Specifically. in the inference, we reconstruct the trajectory map P, as the reference-basse sparse optical flow Pa from the reference frame each target pose as:
P ( 2h V. vo) 1 1==1...]]
We then predicted the reference-basee dense motion filed F. CMP Piraj. Pa. Tres) E
(XN=1))2xHXX by condition motion propagation (CMP) (Zhan et al. 2019 based on the sparse optical flow Pa and the reference image Iref- CMP Zhan et al. 2019) is a self cuf-ssuerveied learning- from-motion model that acquires image and sparse motion field and estimates the corresponding dense motion field. Notably, the dense motion field F of each frame starts with the reference image, which avoids geometric constraints during inference.
4

---

Published as a conference paper at ICLR 2025
To ensure motion estimation accuracy during training, first extract the forward optical flow from the driving video using existing optical flow estimation models (Teed & Deng 2020; Xu et al. 2023). We then use watershed-based approach (Zhan et al. 2019) to sample the sparse optical flow Pa from the forward optical flow. See Appendix. A for details.
To leverage motion field as guidance, we introduce motion encoder specifically designed for the optional flow, which includes sparse motion encoder &g. dense motion encoder ε and feature fusion layer Fe. &d and & have the same structure and multi-scale convolutional encoders with each stage built by Conv SArr-SiUU-A Zer roConv (Zhang et al., (203a)) as the basic block. The feature fusion layer Fe a 2D convolution for fusing sparse motion features & (F,) and dense motion features Ed (Fa). Finally, we compute the motion field guidance Fm:
Motion Encoder.
F (F...(F.) (Fa))
(5)
4.2 KEYPOINT CORRESPONDENCE
Point Feature Extraction. To maintain a consistent appearance, it is crucial to correspond the con- tent of the reference image with the motion trajectory. Specifically, we first extract the DIFT (Tang 2023 features D of the reference image using the pre-trained image diffusion model.
et al.
Subsequently, the keypoint embedding in the reference is obtained D yo), where (2 yo is retrieved from the reference pose. Next, we initialize the keypoint correspondence map F, with zero vectors and assign point embeddings according the trajectory coordinates as:
D (1r v8), if i i= = = yn: otherwise.
(6)
O,
Finally, we obtain the keypoint correspondence map Fp ( ER for all frames, where Dp is the feature dimension of the point embedding.
To utilize the content correspondence of key points as guidance, generate multi- scale correspondences of sparse point feature maps and make them compatible with the U-Net en- coder of the Hybrid ControlNet (Sec.4.3). We introduce the multi-scale point encoder &p to main- lain the key point content F, from the reference image. The point encoder Ep consists of a series of learnable MLPs. To seamlessly integrate into existing models, we extract intermediate features of the encoder of the hybrid Controlnet. The multi-scale intermediate features of the Controlnet encoder are denoted E where I denotes each U-Net block E L]. To match the spatial size of E' we apply downsampling to the feature map between the encoder layers. We compute the multi-scale keypoint correspondence as follows:
Point Encoder.
F (OF, ,W)),,
where (H', W') are denote the spatial dimension of the 1-th U-Net block and & means downsampling operation. Therefore, F'. shares the same size as E E Finally, F. are added elementwisely the intermediate feature E of the U-Net encoder as guidance: E E EEne + F!
4.3 PLUG-AND-PLAY HYBRID CONTROLNET
After obtaining motion field guidance and keypoint correspondence, we aim to integrate these con- trol guidance seamlessly into the U-Net architecture of existing animation models. Inspired by ControlNet (Zhang et al. 2023a). We design a hybrid ControlNet F to provide additional control signals for the existing animatio model as shown in Figure ([c). Specifically, given an animation diffusion model based on the U-Net architecture, we freeze its all modules while allowing the mo- tion encoder, point encoder and hybrid ControlNet to be updated during training. Subsequently, F,, is added to the noise latent before being input into the hybrid ControlNet. Considering the high sparsity of the point feature F., we correspondingly add F, to the input of the convolutional layer. Notably, the U-Net encoder intermediate feature Eene in Sec. 4.2 is from hybrid ControlNet rather than denoising U-Net. Finally, the control condition is computed as:
r = F(z, m, F.,
where r is a set of condition residuals added to the residuals for the middle and upsampling blocks in the denoising U-Net.
5

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparisons on Tiktok dataset.
VBenchf
Method
FID-FVDL FVD, CD-FVD)
Temporal Subject
Background Consistency Smoothness
Motion
Dynamic Imaging Degree
Flickering Consistency
Quality
Stable Diffusion
MagicPose Chang et al
2023
96.65
95.12
94.55
98.29
22.70
63.87
15.53
1015.04
693.24
Moore Mobre levads
96.86
95.18
95.37
98.01
25.51
69.14
11.58
924.40
687.88
MusePose Long et al. 2024
97.02
95.27
95.16
98.45
27.31
71.56
11.48
866.36
626.59
MuscPose+Uurs
97.63
95.70
71.89
622.64
Stable Video Diflusion
ControlNeXi Peng et
2024
97.55
94.58
95.60
98.75
27.58
70,40
10.49
496.87
624.51
MinicMotion Zhing et
2724
97.56
94.95
95.36
98.67
28.42
68.42
10.50
598.41
621.90
MimicMotion--Oam
97.73
95.72
95.90
98.89
29.51
71.33
10.24
466.93
603.27
EXPERIMENTS
5.1 IMPLEMENTATIONS
Baseline Models. To demonstrate the effectiveness of DisPose, we integrate proposed modules into two open source human image animation models: MusePose (Tong et al. 2024) and Mimic- Motion (Zhagg et al. 2024). MusePose (Tong et al. 2024) is reimplementation of AnimateAny- one (Hu et al. 2023) by optimizing Mloore-Animate-Anyooe
[MooreThreads: (024), which imple- ments most of the details of AnimateAnyone (Hu et al. 2023) and achieves comparable performance. MimicMotion (Zhang et al. 2024) is the state-of-the-art human animation model based on Stable Video Diffusion (Blattmann al. 2023a
Implementation Details. Following (Hu et al. 2023 Zhang et al. 2024), We employed DW- Pose (Yang et al.) (2023) to extract the pose sequence of characters in the video and ren- der it as pose skeleton images following OpenPose (Cao et al., (017). We collected 3k hu- man videos from the internet train model. For MusePose (Tong et al. (024). stable-dae molle-dinn Efusiinvv----i each video and center cropped resolution of 512x512. Training was conducted for 20.000 steps with a batch size of 32. The learning rate was set to 1e-5. For MimicMotion (Zhang et al. 2024), we initialized our hybrid ControlNet using stable Sttablevve vieeeeevvii-iiiii fusiifuiiiii inniiiiiinnnnnnm img2vid-xx sampled 16 frames from each video and center crop conducted for 10,000 steps with a batch size of 8. The learning rate was set to 2e-5.
used to initialize our hybrid ControlNet. We sampled 16 frames from
We
resolution of 768x1024. Training was
Evaluation metrics. The video quality is evaluated by calculating the Frechet Inception Distance with Fréchet Video Distance (FID-FVD) (Balaji et al., 2019), Fréchet Video Distance (FVD) (Un- terthiner et al.) 2018) and Content-Debiasee between the generated video and the grounded video. Considering that these metrics are incon- sistent with human judgment (Huang et al. 2024), introduce metries in VBench (Huang al., 2024) to comprehensively assess the consistency of the generated video with human perception, in- cluding temporal flickering, aesthetic quality, subject consistency, background consistency, motion smoothness, dynamic degree, and imaging quality.
Fréchet Video Distance (CD-FVD) (Ge et al. 2024)
5.2 QUANTITATIVE COMPARISON
Evaluation on Tik Tok dataset. We compare our method to the state-of-the-art human image anima- tion methods, including MagicPose (Chang et al.
2023), Moore Moore-Anymater AnymateAnyone (MooreThreaal
2024). MusePose (Tong et al.] 2024). ControlNeX Xt (Peng et al.] (024) and MimicMotion (Zhang al. 2024). Following previous works (Zhang al. 2024 Wang et al. 2024b),
sequences 335 to 340 from the Tok (Jafarian & Park] (021) dataset for testing. Table T presents quantita- tive analysis of the various methods evaluated on the TikTok dataset. The proposed methods achieve significant improvements across different bascline models. Our method achieves higher scores on VBench (Huang et al. 2024) while reducing FID-FVD and FVD scores, which indicates that the proposed method generates high-quality videos that align with human perception.

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative comparisons unseen dataset.
Temporal Flickering
Subject Consistency
Background Consistency
Motion Smoothness
Dynamic Degree
Imaging Quality
Aesthetic
Method
Quality
Stable Diffision 1.5
MagicRose Chang
93.71
98.51
25.67
Moore Moore Threads
20A4
92.83
92.42
98.
27.43
65,32
94,61
47.23
MusePuse long et a
93.12
93.97
98.58
28.72
65.26
96.41
49.34
MusePoseFOurs
93.43
94.22
98.76
29.61
65.48
96.63
49.39
Stable Video
ControlNeXi Peng et al
2024
93.25
94,27
98. 70
28.42
64.36
97.42
49.10
MimicMotion Zliang or
2024
93.32
94.12
98.
29.81
64.51
97.45
49.03
MimicMotion--Uum
93.59
94.35
98.75
30.02
65.56
97.80
49.93
STHOIR
STHEU
BINO
STNDIR
OTHOUE
Referenee
Moore AnimatcAnyose
MusePose
MusePose
ControlNeXt
MimicMotion
Mimic Motion Ours
Ground Truth
Image
Figure 3: Qualitative comparisons between our method and the state-of-the-ard models on the Tikkook dataset.
Evaluation on unseen dataset. Training videos collected from the Internet may exhibit domain proximity with the TikTok (Jafarian & (02T) test set. We construct an unseen dataset further compare the generalizability of various methods. We collect 30 high-quality human videos and gen- erate reference images with diverse styles using InstanID (Wang et al. 2024a). Due to the unavail- ability of the ground truth corresponding to the generated reference images, we use VBench (Huang ctt al. 2024) as the quantitative metric as shown Table
5.3 QUALITATIVE RESULTS
Comparison with state-of-the-art methods. Figure Jllustatsss the qualitative results between the various models the TikToo dataset. Thanks to the motion field guidance and keypoint correspon- dence, our method can produce reasonable results with significant pose variation.
Comparison with dense condition. To compare the proposed method with the existing dense con- dition, we conduct qualitative experiments in Figure 4 Champ (Zhu et al.) (024)) represents human body geometry and motion features through rendered depth images, normal maps, and semantic maps obtained from SMPL (Loper et al., 2015) sequences. Since rendering an accurate human body model for an arbitrary reference character during inference is virtually impossible, Champ achieves rough shape alignment by the parametric human model. This leads to dense conditional distortions some human body regions (e.g., face and hands) thus degrading the video quality. Moreover, para- metric alignment may fail when there significant differences in the shape and layout between the reference image and the driving video resulting in erroncous results as shown the last case in Fig- ure In contrast to the previous dense condition, introduce reference-based dense motion field through the motion propagation the skeleton pose as shown in Figure which provides dense signals while avoiding the strict constraints of the target pose.
Cruss-identity animation. Beyond animating each reference character with the corresponding mo- tion sequence, further investigate the cross-identity animation capability of DisPose shown in

---

In this paper, we present DisPose, a plug-and-play module for improving human image animation, which aims to provide efficient conditional control without additional dense inputs. To achieve this, obtain Published as a conference paper at ICLR 2025
Reference Image
Champ
Ours
Reference Image
Champ
Ours
Figure 4: Qualitative comparison of our approach with the dense control-based method.
Table 3: Ablation study different control guidance. "w/o Motion" denotes the model configura- tion that disregards motion filed guidance. "w/o Point" indicates the variant model that removes the keypoint correspondence.
VBencht
Method
FID-EVD.
FVDJ
Temporal Flickering
Subject Consistency
Background Consistency
Motion Smoothness
Dynumic Degree
Imaging Quality
w/o Motion
97.66
95.04
95.31
98.75
29.42
69.53
10.31
478. 71
w/o Point
97.47
95.57
95.43
98.42
29.14
70.14
10.28
498.
Full Model
97.73
95.72
95.90
98.89
29.51
71.33
10.24
466.93
Figure 6
Our method generates high-quality animations for the reference image that are faithful to the target motion, proving its robustness. See Appendix, B more qualitative results.
5.4 ABLATION STUDY
Quantitative results. As shown in Table 3 the full configuration of the proposed method out- performs the other variants in metrics. The motion field guidance provides region-level control signals that enhance video consistency, resulting in lower FID-FVD and FVD. The keypoint corre- spondence creates the feature map of the target pose by localizing the semantic point features of the reference image, which makes the generated video more consistent with human perception.
Semantic correspondence. To better understand the performance keypoint correspondences, we visualize the semantic correspondences of the variant models in Figure 7 Specifically, we select human region (e.g., hand) from the source image and query the target image using the corresponding DIFT features. The keypoint correspondence can localize the correct semantic region from the various characters.
CONCLUSION
we disentangle pose control into motion field guidance and keypoint correspondence. the motion field guidance, we first construct the tracking matrix from the skeleton pose, and then obtain the sparse and dense motion fields by Gaussian filtering and conditional motion diffusion, respectively. Moreover, we introduce the keypoint correspondence of diffusion features explore the semantic correspondence in image animation. Finally, we integrate the extracted guidance fea- tures into hybrid control network. Once trained, model can be integrated into existing human
8

---

Published as a conference paper at ICLR 2025
Reference Image
Driving Video
Reference Image
Driving Video
Dense Condition (Champ)
Dense Coadition (Champ)
Motion Field (Ours)
Motion Field
(Ours)
Figure 5: Comparison of our reference-based dense motion field and existing dense conditions.
Driving Video
Driving Video
Reference Image
Generated Video
Reference Image
Generated Video
Image
Geverated
Reference Image
Figure 6: The demonstration of cross ID animation from the proposed method.
image animation models. Extensive evaluation of various models also validates the effectiveness and generalizability of our DisPose.
LIMITATIONS AND FUTURE WORKS
Despite our DisPose enhances motion guidance and appearance alignment, the ability to synthesize unseen parts for characters is still limited. As shown in Figure 8 attempt generate multi-view

---

Published as a conference paper at ICLR 2025
Source Image
Target Image
Ours
Figure 7: Qualitative analysis of semantic correspondence. Given a red source point in an image (far left), we use its diffusion feature to retrieve the corresponding point in the image on the right.
Reference Image
Genernted Video
Reference Image Generated Video
Figure 8: Qualitative results of our method for multi-viww video generation.
videos for the single-view reference image. In the future, will explore camera control multi- view synthesis models for capturing multi-view reference information. Moreover, introducing the 3DD sparse pose control condition also address this issue.
ETHICS STATEMENT
We clarify that all characters in this paper are fictional except for the Tik (Jafarian & Park 2021 dataset. We strongly condemn the misuse of generative artificial intelligence to create content that harms individuals or spreads misinformation. However, we acknowledge the potential for misuse of our approach. This because it focuses on human-centered animation generation. We uphold the highest ethical standards in our research, including adherence to legal frameworks, respect for privacy, and encouragement to generate positive content.
ACKNOWLEEDGEMENTT
This work was supported by the Hong Kong SAR RGC Early Career Scheme (26208924), the Na- tional Natural Science Foundation of China Young Scholar Fund (62402408), Huawei Gift Fund, and the HKUST Sports Science and Technology Research Grant (SSTRG24EG04).
REFERENCES
Yogesh Balaji, Martin Renqiang Min, Bing Bai, Rama Chellappa, and Hans Peter Graf. Conditional gan with discriminative filter generation for text-to-video synthesis. IJCAI, volume 1, pp. 2, 2019.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Macicj Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127 2023a.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis, Align your latents: High-resolution video synthesis with latent diffusion mod-
10

---

Published as a conference paper at ICLR 2025
els. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22563-22575, 2023b.
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh, Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7291-7299 2017.
Di Chang, Yichun Shi, Quankai Gao, Hongyi Xu, Jessica Fu, Guoxian Song, Qing Yan, Yizhe Zhu, Xiao Yang. and Mohammad Soleymani. Magicpose: Realistic human poses and facial expressions retargeting with identity- diffusion. In Forty-first International Conference Machine Learning, 2023.
Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming-Yo Yu Liu, and Yogesh Balaji. Preserve your own correlation: noise prior for video diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22944, 2023.
Songwei Ge, Aniruddha Mahapatra, Gaurav Parmar, Jun-Yar Zhu, and Jia-Bin Huang. On the content bias in fréchet video distance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 7277-7288, 2024.
Riza Alp Güler, Natalia Neverova, and lasonas Kokkinos. Densepose: Dense human pose estimation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7297-7306, 2018.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffu- sion models without specific tuning. arXiv preprint arXiv:2307.06 04425.44725 2023.
Hao He, Yinghao Xu, Yuwei Guo, Gordon Wetzstein, Bo Dai, Hongsheng Li, and Ceyuan Yang. Cameractrl: Enabling camera control for text-to-video generation. arXiv preprint 21Xiv:2404.02101, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. arXiv preprint arXiv:211.171117 2023.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianx- ing Wu, Qingyang Jin, Nattapol Champaisit, et al. Vbench: Comprchensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 21807-21818, 2024.
Yasamin Jafarian and Hyun Soo Park. Learning high fidelity depths of dressed humans by watching social media dance videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paitern Recognition, pp.
2021.
Ziqi Jiang, Zhen Wang, and Long Chen. Combing text-based and drag- based editing for precise and flexible image editing. arXiv preprint arXiv:2410.03097, 2024.
Xiaoyu Jin, Zumnan Xu, Mingwen Ou, and Wenming Yang. Alignment is all you need: A training- free augmentation strategy for pose-guided video generation. arXiv preprint arXiv:2408. 16506, 2024.
Yaowei Li. Xintao Wang, Zhaoyang Zhang, Zhouxia Wang, Ziyang Yuan, Liangbin Xie, Yuexian Zou, and Ying Shan. Image conductor: Precision control for interactive video synthesis. arXiv preprint 4rXiv:2406./533 2024.
11

---

Published as a conference paper at ICLR 2025
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. ACM Transactions on Graphics, (Proc. SIGGRAPH Asia), 34(6):24:1)2224445 1-28::66 16, October 2015.
Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool. Pose guided person image generation. Advances in neural information processing systems, 30, 2017.
Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc Van Gool, Bernt Schiele, and Mario Fritz. Dis- entangled person image generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 99-108, 2018.
MooreThreads. Morrchrradsmmoor---iiaaaaaaaaa 2024. URL SMoorerthheassss/oorreeeataaaaar
com/
Muyao Niu, Xiaodong Cun, Xintao Wang, Yong Zhang, Ying Shan, and Yinqiang Zheng. Mofa- video: Controllable image animation via generative motion field adaptions in frozen image-to- video diffusion model. In European Conference on Computer Vision, pp. 111-128. Springer, 2024.
Bohao Peng, Jian Wang, Yuechen Zhang, Wenbo Li, Powerful and efficient control for image and video generation. arXiv preprint arXiv:2408.000 06070, 2024.
Yang, and Jiaya Jia. Controlnext:
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominík Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Olaf Ronneberger. Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical image computing and computer-assisted intervention- MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceed- ings, part III 18, pp. 234-241. Springer, 2015.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 2.t.:20/11.3355:5 2020.
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems, 36: 1363-1389,202
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Springer, 2020.
Part 11 16. pp.
Zhengyan Tong, Chao Li, Zhaokang Chen, Bin Wu, and Wenjiang Zhou. Musepose: a pose-driven image-to-video framework for virtual human generation, 2024. URL/ittps://githhut com/
IMBlyralab//busePsss
Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv: 1rXiv:1812.01717, 2018
Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot identity- preserving generation in seconds. arXiv preprint arXiv:2401.07519 2024a.
Tan Wang, Linjie Li, Kevin Lin, Yuanhao Zhai, Chung-Ching Lin, Zhengyuan Yang. Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Disco: Disentangled control for realistic human dance generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- 2024b.
nition, pp.
12

---

Published as a conference paper at ICLR 2025
Zhouxia Wang, Ziyang Yuan, Xintao Wang, Yaowei Li, Tianshui Chen, Menghan Xia, Ping Luo, and Ying Shan. Motionctrl: A unified and flexible motion controller for video generation. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-1 11.11.222.. 2024c.
Wejia Wu, Zhuang Li, Yuchao Gu, Rui Zhao, Yefei He, David Junhao Zhang, Mike Zheng Shou, Yan Li, Tingting Gao, and Di Zhang. Draganything: Motion control for anything using entity representation. arXiv preprint AXXi:24403.042200 2024.
Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. Unifying flow, stereo and depth estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 1481-1490, 2024a.
Zunnan Xu, Yukang Lin, Haonan Han, Sicheng Yang. Ronghui Li, Yachao Zhang, and Xiu Li. Mambatalk: Efficient holistic gesture synthesis with selective state space models. In The Thirty- eighth Annual Conference on Neural Information Processing Systems, 2024b.
Zunnan Xu, Yachao Zhang, Sicheng Yang, Ronghui Li, and Xiu Li. Chain of generation: Multi- modal gesture synthesis via cascaded conditional control. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 6387-6395, 2024c.
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10371-10381, 2024a.
Yuhang Yang, Wei Zhai, Hongchen Luo, Yang Cao, and Zhun Zha. Lemon: Learning 3d
human-object interaction relation from 2d images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 16284-16295, 2024b.
Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. Effective whole-body pose estimation with two-stages distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4210-4220, 2023.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.0677 2023.
Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Drag- nuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308.0800 2023.
Ailing Zeng, Yuhang Yang, Weidong Chen, and Wei Liu. The dawn of video generation: Preliminary explorations with sora-like models. arXiv preprint arXiv:2410.05227, 2024.
Xiaohang Zhan, Xingang Pan, Ziwei Liu, Dahua Lin, and Chen Change Loy. Self-supervised learn- ing via conditional motion propagation. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 1881-1889 2019.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3836-3847, 2023a.
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Con- trolvideo: Training-free controllable text-to-video generation. arXiv preprint arXiv:2305. 13077, 2023b.
Yuang Zhang, Jiaxi Gu, Li-Wer Wang, Han Wang, Junqi Cheng, Yuefeng Zhu, and Fangyuan Zou. Mimicmotion: High-quality human motion video generation with confidence-aware pose guid- ance. arXiv preprint arXiv:2406. / 19680, 2024.
Shenhao Zhu, Junming Leo Chen, Zuozhuo Dai, Yinghui Xu, Xun Cao, Yao Yao, Hao Zhu, and Siyu Zhu. Champ: Controllable and consistent human image animation with 3d parametric guidance. In European Conference on Computer Vision (ECCV), 2024.
13

---

Published as a conference paper at ICLR 2025
Figure 9: More Qualitative Comparisons.
A SAMPLING FROM WATERSHED
During training, the sparse optical flow Pa is sampled from the target optical flow. For effective propagation, those guidance vectors should be placed at some keypoints where the motions are representative. We adopt watershed-baate (Zhan al., 2019) method to sample such keypoints. Given the optical flow of image, we first extract motion edges using Sobel filter. Then we assign each pixel a value watershed map. Finally, watershed map to obtain the keypoints. We can adjust Ky to control the average number of sampled points. A larger K, results in sparser samples. Points image borders are removed. With the watershed sampling strategy, all the keypoints are roughly distributed on the moving objects.
the distance to its nearest edge, resulting the topological-distance apply Non-Maximum Suppression (NMS) with kernel size K on the
B MORE QUANTITATIVE RESULTS
Figures 9 and Figure 10 illustrate more qualitative results.
C MORE ABLATION ANALYSES
the region-level guidance provided by our motion field guidance facilitates the enhancement consistency across body regions. The proposed keypoints correspondence im- proves generation quality by aligning DIFT features of the skeleton pose, e.g., facial consistency.
As shown in Figure 11
D MORE DETAILS OF MOTION FIELD GUIDANCE
There is gap between the inference and the training optical flow. (1) During inference, we do not propose extracting the forward optical flow directly from the driving video, as it ignores the gap between the reference character and the driving video. As shown in Figure [4[a)and Figure 15 directly using the forward optical flow as motion guidance clearly inconsistent with the reference image. (2) When there is large difference between the reference image and the driving video, is
14

---

Published as a conference paper at ICLR 2025
Figure 10: More Qualitative Comparisons.
w/o Motion
w/ Motion
Ground Truth
w/o Point
w/ Point
Ground Truth
TA3B
HaIKY
AUHOIU
STHAIPa
Figure 11: Qualitative results about motion field guidance and keypoints correspondence.
impossible get the corresponding motion field by the existing optical flow estimation model, as shown in Figure b5).). Therefore, we have to compute Pd differently during inference.
Although the dense motion field we proposed in Section 4.1 can adapt different body variations during inference. However, there are two limitations of this dense motion field: (1) there is a gap with the motion field extracted from real videos, and (2) the low computational efficiency is not suitable for use during training. Considering that pairs of training data have no body changes, to utilize accurate control signals during training and to improve computational efficiency, we approx- imate the optical flow during inference by sampling sparse optical flow before prediction as shown in Figure 14(c).
E MORE ABLATION STUDY
E. THE IMPACT OF HYBRID CONTROLNET ARCHITECTURE
We show the impact of bydd ControlNet architecture in Table Specifically, we design two variant architectures, (1) Expl: inserting the motion field into the denoising network instead of the hybrid
15

---

Published as a conference paper at ICLR 2025
Table 4:
The impact of hybrid ControlNet.
Table 5: The impact of CMP.
Mcthods
subject consistencyt background consistency!
Methods
FID-FVDL
FVD]
Full Model w/o CMP
93.94
97.83
Expl
10.43
514.83
Full Model
94.35
98.75
Exp2
10.94
551.32
Full Model
10.24
466.93
Table 6: Performance comparisons for image-level metrics.
Methods
SSIM f
PSNRT
LPIPS
L14
MusePose
0.788
19.14
0.263
2.46E-05
MusePose+Ours
0.811
19.36
0.238
2.26E-05
MimicMotion
0.749
18.32
0.272
2.71E-05
MimieMotion+Ours 0.781
19.58
0.242
2.42E-05
controller as shown in Figure 12 a), and (2) Exp2: removing the hybrid ControlNet and inserting the motion field guidance and keypoint correspondence into the denoising network as shown in Figure b). Expl shows that the motion field needs to jointly optimized with U-Net provide the correct representation. Exp2 shows that complex motion information and appearance features cannot be modeled with only two shallow encoders.
E.2 THE IMPACT OF CMP.
We provide the ablation analysis of CMP in Table 5 which shows that CMP can improve the sistency of the generated video.
Motion Encoder
Denoising U-Net
Motion Encoder
Denoising U-Net
Point Encoder
Hybrid ControlNet
Point Encoder
a
(b)
Figure 12: Different hybrid ControlNet architectures.
F MORE PERFORMANCE COMPARISONS.
To further evaluate the generated results, we provide performance comparisons for image-level met- rics in Table 6 Compared the baseline model, our method achieves significant improvements in all metrics.
G TRAINABLE PARAMETERS AND INFERENCE TIME
We compare the trainable parameters and inference time of the different models in Table For fair comparison, the size of the generated video is set to 576x 1024. Our method requires fewer trainable
16

---

Published as a conference paper at ICLR 2025
Table 7: Performance comparisons for image-level metrics.
Methods
trainable Parameters(MB)
infer time (sec/frame)
MusePose
2072.64
3.37
MimicMotion
1454.19
1.61
MimicMotion+
653.40
2.36
Reference Image
Driving Pose
Motion Field
Reforence
Driving Pose
Motion Field
w/ Background
wio Rackground
Figure 13: Analysis of background noise.
parameters based the baseline model. During inference, our method estimates the motion field for the reference image, which increases inference time a little.
H ANALYSIS OF BACKGROUND NOISE.
Since our motion fields are not extracted directly from the driving video, some noise due to esti- mation errors may be introduced. As shown Figure [3] the motion field of the reference image without the background is more accurate than the complex background.
17

---

Published as a conference paper at ICLR 2025
Reference
Driving Video
(a): frame-by- frame optical flow
(b): motion field during training (w/o sample)
(c): motion field during training (w/ sample)
(d): motion field during inference
Figure 14: Body matched motion field visualization.
18

---

Published as a conference paper at ICLR 2025
Reference
Driving Video
(a): frame-by- frame optical flow
(b): motion
field during training (w/ sample)
(c): motion field during inference
Figure 15: Body mismatched motion field visualization.
19


Reviews:
[{"review_id": "m9hUM3COpJ", "reviewer": "Reviewer_BkAE", "summary": "This paper introduces a novel method for animating static human images into realistic videos using driving videos, without relying on additional dense inputs such as depth maps or DensePose. The authors present DisPose, which disentangles pose guidance into motion field estimation and keypoint correspondence. Specifically, they generate a dense motion field from a sparse motion field derived from skeleton poses and the reference image, providing region-level dense guidance while maintaining generalization. Simultaneously, they extract diffusion features corresponding to keypoints in the reference image and transfer these features to the target pose, enhancing appearance consistency and identity preservation. This approach is implemented as a plug-and-play hybrid ControlNet, allowing integration into existing human animation models without additional modifications to model parameters. Extensive qualitative and quantitative experiments demonstrate that DisPose outperforms current methods in both motion alignment and appearance consistency, effectively balancing generalizability and control effectiveness in pose-driven human image animation.", "questions": "See the weakness section.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The strengths of this paper lays in the following several three folds:\n\n1. The paper provides a clear and detailed explanation of the proposed method, including the construction of the tracking matrix from skeleton poses, Gaussian filtering, conditional motion diffusion, and the hybrid control network architecture. As well, the paper is well-organized, guiding the reader logically through the introduction, related work, methodology, experiments, and conclusions. This structure facilitates comprehension and highlights the contributions of the work.\n\n2. The paper introduces DisPose, a novel framework that disentangles pose control into motion field guidance and keypoint correspondence. This innovative approach departs from traditional methods that rely heavily on dense control inputs, offering a new perspective on pose-guided human animation.\n\n3. The DisPose module is designed as a plug-and-play component that can be seamlessly integrated into existing human image animation models. This versatility enhances the performance of baseline models without the need for extensive retraining.", "weaknesses": "The weakness of this paper lays in the following several three folds. My final rating is based on the discussion, and I would like to raise my score if some of the major concerns can be solved.\n\n1. The experimental evaluation is primarily conducted on the TikTok dataset and an additional unseen dataset comprising 30 human videos. The reliance on these datasets raises concerns about the method's generalizability. e.g. the full body, side view.\n\n2. The impact of the hybrid control network architecture, conditional motion diffusion process, and other auxiliary modules is not analyzed.\nThe figures shows promising visual results, such as the complete fingers and vivid facial expressions, so what proposed components lead to such enhanced visual details and visual improvements?\n\n3. The evaluation employs VBench metrics and Fréchet Video Distance (FVD), which may not fully capture perceptual quality. The paper does not report other standard metrics such as Structural Similarity Index (SSIM) or Peak Signal-to-Noise Ratio (PSNR) for a more objective assessment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0RnIshUb3u", "reviewer": "Reviewer_ZwXk", "summary": "The paper presents DisPose, a ControlNet utilizing optical flow/motion fields to condition Latent Diffusion Models (LDM) for image animation and motion retargeting. Instead of using dense, template-based body pose signals like SMPL and OpenPose, DisPose leverages sparse keypoints and dense optical flow fields to better handle body shape variations between the reference and the target. \n\nExperimental results show improvement over prior work on VBench metrics.", "questions": "My main concerns regarding this submission is the rigorousness of the qualitative and quantitative results, as stated above in the weakness section.\n\nPlease discuss these issues accordingly, and I will update my scores accordingly if the problems are sufficiently addressed or discussed. Note that the method/DisPose does not need to be perfect, but it is important to provide a comprehensive view of the proposed method.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper has the following strength:\n- Simple, intuitive design to disentangle motion representation, and achieves better synthesis quality and appearance consistency.\n- Plug-and-play potential: DisPose, when combining with previous approaches, improves the performance on the evaluated metrics.\n\nOverall, the paper is well-written and easy to follow.", "weaknesses": "The paper has the following weaknesses:\n- Lack of diversity in qualitative results. Both the figures, and the supp. videos focus mainly on re-animating generated/synthetic characters. These synthetic characters often have simplified, smoothed-out details, with very similar appearances and aesthetics. It is therefore unclear how the proposed method works on enhancing/improving realistic details. Particularly, since the paper already includes TikTok videos (that have real humans) for training and evaluation, it would be great if the paper can include those results.\n- Using optical flow is a good idea, but it may only work for static background. For noisy background, such as swaying trees and leaves, the motion field would likely capture those motion, and thus the model may incorrectly associate these background noises with the input motion.\n- Evaluation section can be further improved. It makes sense to use VBench due to the generative nature of DisPose. But at the same time, it is possible to check the appearance consistency by using the TikTok videos. One can split the video for the same subject into training and testing part, and examine whether DisPose/baselines can transfer the testing pose while preserving the subject identity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HOSx4nuK1k", "reviewer": "Reviewer_PDfA", "summary": "This paper proposes a plug-and-play method to disentangle pose guidance, which extracts robust control signals from only the skeleton pose map and reference image without additional dense inputs. The proposed method generates a dense motion field from a sparse motion field and the reference image, and extracts diffusion features corresponding to pose keypoints from the reference image. Extensive qualitative and quantitative experiments validate the effectiveness of the proposed method.", "questions": "Please refer to the questions and suggestions in the “Weaknesses” part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The topic of controllable human animation is valuable and the control signals without additional dense input is promising to improve the generalizability of generation model.\n\n2. The authors have conducted extensive and thorough experiments of different baselines on several datasets to validate the effectiveness of the proposed method.", "weaknesses": "1.\tThe key idea of the paper is to disentangle motion field guidance and keypoint correspondence from pose control. However, the authors just combine motion field and keypoint extraction to empower the generation ability of the original animation diffusion model. The detailed disentanglement operation is missing. \n\n2.\tFor the dense motion field of Section 4.1, the authors choose to sample the sparse optical flow Pd from the forward optical flow obtained from existing optical flow estimation models, and then predict the dense motion field using CMP. However, why not directly use the dense motion field obtained by existing methods? Please give a detailed explanation of the reasons for sampling before predicting and the benefits of this way. In addition, why do authors calculate Pd differently for training and inference?\n\n3.\tThe point correspondence is extracted according to the keypoint coordinates of pose and how can this kind of correspondence be helpful to maintain a consistent appearance? The authors are suggested to provide more ablation study results and visual comparison results to validate the effectiveness of multi-scale point correspondence.\n\n4.\tFor the part of Point Encoder of Section 4.2, the descriptions of the U-Net encoder and multi-scale intermediate features are confusing. The authors are suggested to emphasize that the intermediate feature is from hybrid ControlNet in this part.\n\n5.\tThe reference image used of the paper is realistic, however, the provided demo video is a little cartoon-style and not realistic enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dqOnerkTHg", "reviewer": "Reviewer_c9da", "summary": "This paper proposed DisPose, a human pose retargeting framework with high-quality motion control. The core of the proposed method is to disentangle control guidance from skeleton poses and reference images.", "questions": "Please see the weakness section.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to understand. The motivation for extracting sparse and dense motion fields makes much sense for this task and the DIFT feature provides content correspondence of key points as guidance. The plug-in module follows the ControlNet architecture and provides flexibility for using the pre-trained weight for other customized SD-UNet. I found Figure 7 extremely helpful for understanding the keypoint correspondences.", "weaknesses": "1. Is there any discussion or comparison of the efficiency, e.g., trainable parameters and inference time for a single batch?\n\n2. Metric for video generation evaluation. I understand the authors follow previous works and adopt FVD as the video evaluation metric. However, this metric has recently been widely criticized by the community because of its inaccuracy in reflecting the overall quality. I wonder what the performance comparison would be if debiased FVD is used for evaluation. From [here](https://content-debiased-fvd.github.io/)\n\n3. I appreciate the authors for providing video visualizations in the supp. Are there any side-by-side video comparisons between this work and recent baselines? It would be better to judge the temporal consistency of the video quality and the effectiveness of motion control.\n\n4. How does the model generalize to out-of-domain real human identities? E.g. Old people?\n\n5. (Minor) For the input of the reference image, did the authors utilize ReferenceNet for the SD-1.5-based model and the same setting of SVD for comparison to MimicMotion?\n\n6. (Minor) L78 There's a mistake in the cited works with an additional reference network. DisCo didn't adopt this architecture but MagicPose and Champ did.\n\nI'm more than **willing to raise** my score if my concerns are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)