MimicMotion: High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance
Authors: Yuang Zhang
Jiaxi Gu
Li-Wen Wang
Han Wang
JunqiCheng
Yuefeng Zhu
FangYuan Zou
Venue: ICML 2025 poster (2025)
Keywords: Human Motion Video Generation
Primary area: deep_learning->generative_models_and_autoencoders

Abstract: In recent years, while generative AI has advanced significantly in image generation, video generation continues to face challenges in controllability, length, and detail quality, which hinder its application. We present MimicMotion, a framework for generating high-quality human videos of arbitrary length using motion guidance. Our approach has several highlights. Firstly, we introduce confidence-aware pose guidance that ensures high frame quality and temporal smoothness. Secondly, we introduce regional loss amplification based on pose confidence, which reduces image distortion in key regions. Lastly, we propose a progressive latent fusion strategy to generate long and smooth videos. Experiments demonstrate the effectiveness of our approach in producing high-quality human motion videos. Videos and comparisons are available at [https://tencent.github.io/MimicMotion](https://tencent.github.io/MimicMotion).

MinicMotion: High-Quality Human Motion Video Generation with
Confidence-aware Pose Guidance
Yuang Zhang 12
Jiaxi Gu
Li-Wen Wang
Han Wang 12 Junqi Cheng'
Yuefeng Zhu
Fangyuan Zou'
1. Introduction
Dancing video
Talking video
Video generation has advanced alongside image generation. However, video generation is more challenging due to its higher inherent complexities, including the need for high- quality imagery and seamless temporal smoothness. In addition, controlling the generated content and extension to significant lengths is essential for real-world use. In this paper, we focus on pose-guided video generation with reference image. Our goal is to generate a video that contains rich imagery details and adheres the reference image and the pose guidance.
Long video generation
355
X-1 Slice
Currently, there are plenty of works focusing this task, such as DisCo (Wang et al., 2023), AnimateAny- one (Hu, 2024), MagicDance (Chang et al., 2023), Mag- icAnimate (Xu et al., 2024), etc. Though various techniques have been studied, the results are unsatisfactory in several as- pects. Imagery distortion especially on the regions of human hands is still common issue which is particularly evident in videos containing large movements. Besides, to achieve good temporal smoothness, imagery details are sometimes sacrificed resulting in videos of blurred frames. In the pres- ence of diverse appearances and motions in videos, accurate pose estimation is inherently challenging. This inaccuracy not only creates conflict between pose alignment and tem- poral smoothness but also hinders the model scaling on the training schedule due overfitting noisy samples. In addition, due to computational limitations and model capabilities, there are still significant challenges in gener- ating high-quality long videos containing a large number of frames. To solve these problems, we propose a series of approaches for generating long but still smooth videos based pose guidance and image reference.
Figure I: Pose-guided dancing and talking videos generated by MimicMotion showcase its capability to produce diverse human motions and long videos.
Abstract
In recent years, while generative has advanced significantly in image generation, video genera- tion continues face challenges in controllabil- ity, length, and detail quality, which hinder its application. We present MimicMotion, frame- work for generating high-quality human videos of arbitrary length using motion guidance. Our approach has several highlights. Firstly, we in- troduce confidence-aware pose guidance that en- sures high frame quality and temporal smooth- ness. Secondly, introduce regional loss am- plification based on pose confidence, which re- duces image distortion in key regions. Lastly, we propose a progressive latent fusion strategy to generate long and smooth videos. Experi- ments demonstrate the effectiveness of our ap- proach in producing high-quality human motion videos. Videos and comparisons are available https:/ttencent, wwtttub.io/iimii... io/MimicMotion.
To alleviate the negative impact of inaccurate pose estima- tion, we propose an approach of confidence-awert -wwrre pose guidance. By introducing the concept of confidence to the pose sequence representation, better temporal smoothness can be achieved and imagery distortion can also eased. Confidence-bart regional loss amplification can make the hand regions more accurate and clear. In addition, we pro- pose progressive latent fusion method for achieving long but still smooth video generation. Through generating video segments with overlapped frames with the proposed pro-
Tencentt FShanghai Jiao Tong University. Correspondence to: Jiaxi Gu <imjiaxie @ comm..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MimicMotion; High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance
gressive latent fusion, our model can handle arbitrary-length pose sequence guidance. By merging the generated video segments, the final long video can be of good cross-frame smoothness and imagery richness at the same time (see Fig- ure 1). For model training, to keep the cost of model training within an acceptable range, our method is based on a gen- erally pre-trained video generation model. The amount of training data not large and no special manual annotation is required.
generative capabilities for videos (Yan et al., 2021; Yu et al., 2023; Ma et al., 2024a; Bao al., 2024). Stable Video Diffusion (SVD) (Blattmann et al., 2023) is one of the most popular open-souree models built upon LDM. It offers a straightforward and effective method for image-based video generation and serves as a powerful pre-trained model for this task. Our approach extends SVD for pose-guided video generation, leveraging its pre-trained generative capabilities.
2.2. Pose-Guided Human Motion Transfer
In summary, there are three key contributions of this work:
cose-to-appearance mapping aims to transfer motion from the source identity to the target identity. Methods based paired keypoints from source and target images employ local affine transformations (Siarohin et al., 2019; 2021) or Thin-Plat Plate Spline transformations (Zhao & Zhang, 2022) to warp the source image to match the driving image. These techniques aim to minimize distortion by applying weighted affine transformations, thereby generating poses in the out- put image that closely resemble those in the driving image. Similarly, methods such as (Chan et al., 2019; Tu al., 2024; Ma et al., 2024b; Hu, 2024; Feng et al., 2023; Peng et al., 2024; Li et al., 2024) utilize pose stick figures obtained from off-the-shelf human pose detectors as motion indica- tors and directly generate video frames through generative models. Depth information (Feng et al., 2023) or 3D human parametric models, such as SMPL (Skinned Multi- Peetonn Linear) (Zhu et al., 2024; Wang et al., 2024b), can also be used to represent human geometry and motion characteris- tics from the source video. Nevertheless, these overly dense guidance techniques can rely too much on the signal from the source video, such as the outline of the body, leading to a degradation in the quality of the generated videos, especially when the target identity differs significantly from the source. Our approach, leveraging off-the-shelf human pose detec- tors, is capable of capturing the motion of the human body in driving videos without introducing excessive extrancous information, thereby ensuring the overall quality of the gen- erated video. In addition, we introduce confidence-aware pose guidance, which effectively mitigates the influence of inaccurate pose estimation in training and inference. In this way, we achieve superior portrait frame quality, especially in the hand regions.
1. We improve the pose guidance by employing confidence-aware strategy. In this way, we alleviate the negative impact of inaccurate pose estimation. This approach not only reduces the influence of noisy sam- ples during training but also corrects erroneous pose guidance during inference.
2. Based on the confidence-awant strategy, we propose hand region enhancement alleviate hand distortion by strengthening the loss weight of the region of human hands with high pose confidence.
3. While overlapped diffusion a standard technique for generating long videos, we advance it with position- aware progressive latent fusion that improves temporal smoothness at segment boundaries. Extensive experi- ments show the effectiveness of the proposed approach.
2. Related Work
2.1. Diffusion Models for Image/Video Generation
Diffusion-based models have demonstrated promising re- sults in the fields of image (Ho et al., 2020; Song et al., 2020; Rombach et al., 2022a; Zhang et al., 2023; Wang et al.,
2024a) and video generation (Singer et al., 2022; Guo et al., 2023; Ho et al., 2022; He et al., 2022; Chen et al., 2023; 2024; Blattmann et al., 2023), renowned for their capacity in generative tasks. Diffusion models operating in the pixel domain encounter challenges in generating high-resolution images due information redundancy and high computa- tional costs. Latent Diffusion Models (LDM) (Rombach et al., 2022a) address these issues by performing the diffu- sion process in now-dimensional latent spaces, significantly enhancing generation efficiency and quality while reduc- ing computational demands. Recently, efforts have been made to improve training efficiency through noise sched- ule (Zheng et al., 2024a;b). Compared image generation, video generation demands more precise understanding of temporal motion patterns. Recent video generation mod- els leverage diffusion models by adding temporal layers to pre-trained image generation models (Singer et al., 2022; Guo et al., 2023; Wu et al., 2023; Wang et al., 2024c), or utilizing DIT (Peebles Xie, 2023) structures to enhance
2.3. Long video generation
Recent diffusion- based video generation algorithms are con- strained producing videos with durations of only a few seconds, significantly limiting their practical applications. As a result, substantial research efforts have been dedicated to extending the duration of generated videos, leading to the proposal of various approaches to overcome this limi- tation. Methods like (He et al., 2022; Voleti et al., 2022) autoregressively predict successive frames, enabling the gen- eration of infinitely long videos. However, these methods
2

---

MimicMotion; High-Quality Human Motion Vidco Generation with Confidence-aware Pose Guidance
Video frrms Ruferere Image
often face quality degradation due error accumulation and the lack of long-term temporal coherence. Hierarchical approach (Yin et al., 2023; Wang et al., 2024c) are proposed for generating long videos in a coarse-to-fine manner. It first creates a coarse storyline with keyframes using a global diffusion model, then iteratively refines the video with local diffusion models to produce intermediate frames.
Irage Encodor
WIE Brooder
U-Nat
Pyysingg Jish
Noisy Iatest Bagguur
Kordiam 4ee0g
Up sumplrr
Denna wed
latent foaturna
with contienca
MultiDiffusion (Bar-Tal et al., 2023) combines multiple pro- cesses that use pre-trained text-to-image image diffusion models to create high-quality images with user-defined controls. It works by applying the model to different parts of an image and using an optimization method to ensure all parts blend seamlessly. This allows users to generate images that meet specific requirements, like certain aspect ratios or spatial layouts, without needing additional training or fine-tuning. Lumiere (Bar-Ta Tal et al., 2024) extends MultiDiffusion to video generation by dividing the video into overlapping tem- poral segments. Each segment is independently denoised, and an optimization algorithm then combines these denoised segments. This approach ensures high coherence in the gen- erated video, effectively maintaining temporal smoothness across segments. However, our experiments reveal that abrupt transitions can still occur at segment boundaries.
Posetlit (Mut lapora)
MSE Loss
Amplly los on Highes
VIDGO Frurnes
Figure 2: MimicMotion integrates an image-to-video diffu- sion model with confidence-aware pose guidance, utilizing spatiotemporal U-Net and a PoseNet for introducing pose sequence as the condition. Our design features adaptive adjustment of pose influence via keypoint confidence scores and prioritizes high-confidence hand regions in the loss func- tion.
where e is the condition. We then train co until convergence.
3.2. Data Preparation
Building upon this principle, we introduce a position-awar -wwree progressive latent fusion strategy that enhances temporal smoothness near segment boundaries. We adaptively assign fusion weight based on the temporal position, ensuring a smooth transition at the segment boundaries.
To train a pose-guided video diffusion model, we collect a video dataset containing various human motions. The dataset need not be excessively large, as the pre-trained model already has a good prior.
Given video from our dataset, the training sample is con- structed with three parts: reference image (denoted as Let). a sequence of raw video frames, and the corresponding poses. Basic pre-processing operations of resizing and crop- ping are applied to the raw video to get a sequence of video frames. For given video, a fixed number of frames are randomly sampled at equal intervals as input video frames to the diffusion model. The input reference image is inde- pendently sampled from the video and pre-processed in the same way as the video frames. Another input of the model is the pose sequence, which is extracted from the video frames with DWPose (Yang et al., 2023) frame by frame.
3. Method
3.1. Preliminaries
A Diffusion Model (DM) learns diffusion process that generates probability distribution for a given dataset. In the case of visual content generation tasks, a neural network of DM is trained to reverse the process of adding noise to real data so new data can be progressively generated starting from random noise. For data sample ~ Paata from specific data distribution Pdata the forward diffusion process is defined as a fixed Markov Chain that gradually adds Gaussian noise to the data, following: qxx, x(- = N(x;; vT - -ix,-11 1, I)II for f = 1,... 1....T where T is the number of perturbing steps and x represents noisy data after adding / steps of noise on the real data Xo. This process is controlled by a schedule 3e. The above diffusion process can be reformed as:
3.3. Pose-Guided Video Diffusion Model
The goal of MimicMotion is to generate high-quality, pose- guided human videos from a single reference image and a sequence of poses to mimic. This task involves the syn- thesis of realistic motion that adheres to the provided pose sequence while maintaining visual fidelity the reference image. We exploit the ability of specific pre-trained video diffusion model to reduce the data requirement and com- putational cost of training video diffusion model from scratch. Stable Video Diffusion (SVD) (Blattmann et al., 2023) is an open-source image-to-videe diffusion model trained on a large-scale video dataset. It shows good perfor-
qxx: x0) N(xi Vaxxo (1- az)
(L)
where ar I 1 ag and at = 1 -r. Following DDPM (Ho et al., 2020), a denoising function eo with pa- rameter 0 is trained by minimizing the mean squared error:
(e |||e- co.: 1)
3

---

MimicMotion; High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance
mance both video quality and diversity compared with the other contemporary models. The model structure of MimicMotion is designed to integrate pre-trained Stable Video Diffusion (SVD) model to leverage its image-to-video generation capabilities.
capability of the pose estimation model, like DWPose (Yang et al., 2023), is only one aspect of the reason. The more significant cause is the inherent uncertainty of pose from dynamic appearances and motions. Specifically, incorrect pose guidance signals can mislead the model, resulting in the generation of inaccurate or distorted outputs, as illustrated in Figure 7. Moreover, noisy pose guidance signals can lead to overfitting on samples with incorrect poses, potentially causing training instability. This in turn may hinder the model's 's training.
Learning a diffusion process in pixel space is costly, es- pecially for high-definitin videos involving many frames. We follow the Latent Diffusion Model (LDM) (Rombach et al., 2022b) to encode pixels into lower-dimensional latent space. LDM adopts pair of autoencoders, consisting of an encoder & and a decoder D. Given a data sample x, it is encoded into the latent space as z = {xx). Con- versely, the latent vector z can be decoded into pixel space via x = D(z).
For this problem, we propose confidence-aware pose guid- ance, which leverages the confidence scores associated with each keypoint from the pose estimation model. These scores reflect the likelihood of accurate detection, with higher val- ues indicating higher visibility, less self-occlusion from other body parts, and motion blur. Instead of applying fixed confidence threshold to filter the keypoints, as com- monly adopted in prior works (moo, 2024; Chang et al., 2023), we utilize brightness of the keypoints and limb to rep- resent the confidence level of pose estimation. Specifically, we multiply the color assigned to each keypoint and limb by its confidence score. Consequently, keypoints and corre- sponding limbs with higher confidence scores will appear more significant on the pose guidance map. This method enables the model to prioritize more reliable pose informa- tion in its guidance, thereby enhancing the overall accuracy of pose-guided generation. In this way, the uncertainty of pose estimation can be conveyed through the pose guidance, making pose guidance more informative.
Figure 2 shows the structure of our model. The VAE encoder on input video frames and the corresponding decoder for getting denoised video frames are both adopted from SVD and these parameters are frozen. The VAE encoder is ap- plied independently to each frame of the input video and the conditional reference image. In contrast, the VAE decoder processes the latent features that undergo spatiotemporal in- teraction from U-Net. To enhance the temporal smoothness of low-level details, the VAE decoder incorporates temporal layers alongside the spatial layers.
The reference image and the sequence of poses are two other inputs of the model. The reference image follows two pathways: First, it is processed through CLIP (Radford et al., 2021) to extract features that are fed into each U-Net block's cross-attention. Second, the image is encoded by the frozen VAE encoder to obtain its latent representation, which is then temporally duplicated to match the video frame dimensions. This latent representation is concatenated with the video frame features along the channel dimension before entering the U-Net.
Hand region enhancement Moreover, we employ pose estimation and the associated confidence scores to allevi- ate region-specific artifacts, such as hand distortion, which are prevalent in the diffusion-based image and vidco gen- eration models. Specifically, we identify reliable regions via thresholding keypoint confidence scores. By setting a threshold, we can distinguish between keypoints that are confidently detected and those that may be unreliable due to self- ooclusicciinn or motion blur. Keypoints with confidence scores above the threshold are considered reliable. We im- plement masking strategy that generates masks based confidence threshold. We unmask areas where confidence scores surpass a predefined threshold, thereby identifying reliable regions. When computing the loss of the video diffusion model, the loss values corresponding to the un- masked regions are amplified by a certain scale so they can have more effect on the model training than other masked regions.
For pose guidance, PoseNet consisting of multiple convolu- tion layers is designed to extract features from input pose sequences. The reason for not using the VAB encoder is that the pixel value distribution of the pose sequence is different from that of common images on which the VAE autoencoder is trained. The features of poses are then element- wisely added to the output of the first convolution layer of U-Net. Pose guidance is not added to every U-Net block due to: a) the sequence pose being extracted without temporal interac- tion may confuse the spatio-temporal layers within U-Net; b) excessive invol vement of the pose sequence may degrade the performance of the pre-trained image-to-video model.
3.4. Confidence-Awarr Pose Guidance
Specifically, to mitigate hand distortion, we compute masks using a confidence threshold for keypoints in the hand re- gion. Only hands with all keypoint confidence scores ex- ceeding this threshold are considered reliable, as a higher score correlates to higher visual quality. We then construct
Inaccurate pose estimation has negative impact on the model's training and inference. However, accurate pose estimation is challenging in dynamic videos. The limited
4

---

MimicMotion; High-Quality Human Motion Vidco Generation with Confidence-aware Pose Guidance
J. lotally denaising for "o'T'ste T'stpps
Algorithm Progressive latent fusion for long videos.
Progressively fusing latent festures overiapped frames every denolsing step.
1: Input:
Sugmerit
2: Tee: Reference image;
3: P.: Pose of j-th frame in i-th video segment;
4: z/: The latent of j-th frame in i-th video segment;
1. Denoising naisy letent features wilhin each video segmentinn every denoising step.
5: N: The number of frames in each video segment;
6: C: The number of overlapped frames.
7: Output: A long sequence of latent video frames.
Figure 3: Overview of our approach for long video gen- eration. The colored boxes represent latent video frames. A darker color means higher weight. The dashed boxes represent video frame features involved in latent fusion.
8: z (0,
9: Aiusion f- 1/(C + 1)
10: for 1 = T to 1 do
11: for i = 1.2.... do
12:
2, < DM Ine. P.,
13: end for
14:
for i 1...... do
a bounding box around the hand by padding the boundary of these keypoints, and the enclosed rectangle is designated
15:
for j 1 to N do
16:
if < 1 and j S C then
as unmasked. This region is subsequently assigned a larger
17:
z x- -JAiusion 2 + (1 (1- AArusion )2;11
N-04
weight in the loss calculation during the training of the video diffusion model. This selective unmasking and weighting process biases the model's 's learning towards hands, espe- cially hands with higher visual quality, effectively reducing distortion and improving the overall quality.
18:
else if j > N C then
19:
2 + (N + 1 - Afasion) 2 + +(C (C'-N -0-N- +
C-N+1
j) Ausion
end if 21: end for
20:
22: end for
3.5. Progressive Latent Fusion for Long Videos
23:
end for
Limited by computation resources, generating long videos containing& large number of frames is challenging. For this problem, prior works like MultiDiffusion (Bar-Tal et al., 2023) successfully use latent fusion for panoramic image generation. A similar idea can be applied to the video gener- ation task. A straightforward approach is directly applying MultiDiffusion in the time domain, as in Lumiere (Bar- Tal et al., 2024). However, unlike spatial discontinuities between image tiles, temporal discontinuities are more no- ticeable to viewers, resulting in flickering or abrupt content changes. We address this by a progressive approach for generating long videos with high temporal continuity.
24: return (2!:NC) 1=122,.
tails of progressive latent fusion. As inputs, the reference image is denoted as Let, and the pose frame corresponding to j-th frame in i-th video segment is denoted as P/-- We use xf to denote the latent feature of j-th frame in i-th video segment. The denoising process starts from the maximum time step T and the latent features are initialized with a normal distribution AOO I). Within each denoising step at time step l, the reversed diffusion process defined by the trained model (DM) is applied to the latent features of each video segment numbered i separately, with z, Int, P, and inputs. During the latent fusion stage, for every two adjacent video segments, the involved video frames are then fused. To avoid the corruption of temporal smoothness near video segment boundaries after latent fusion, we propose progressive latent fusion. For a video frame involved in latent fusion, its fusion weight is determined by its relative position in the video segment belongs. Specifically, if frame is close to the segment belongs to, will be as- signed heavier weight. For implementation, a fusion scale is pre-defined as Atusion 1/(C + 1) for controlling the level of latent fusion.
Progressive latent fusion is training-free and is integrated into the denoising process of the latent diffusion model during inference. Figure 3 shows overview of this pro- cess. We omit the VAE for brevity. The denoising process is done in latent space in our method. In general, there are T' denoising steps in total and our latent fusion is ap- plied within each step. For a long given pose sequence, we use a pre-defined strategy for splitting the whole sequence into segments, consisting of a fixed number of frames per segment (denoted as N), with a certain number (C)) of over- lapped frames between every two adjacent segments. For the sake of generation efficiency, it is common to assume that C' < N. During each denoising step, video segments are firstly denoised separately with the trained model, condi- tioning on the same reference image and the corresponding uub-sequence of poses. Algorithm| shows the specific de-
After applying T iterative denoising steps, we aggregate the final long sequence by combining the denoised overlapped video segments in latent space. The resulting aggregated
5

---

MimicMotion; High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance
long video in latent space is then fed the temporal VAE decoder to reconstruct the long video.
Ref. image
Ref. pose
MagicPose
Moore
MuseVV
Ours
4. Experiments
4.1. Implementation Details
Due to the lack of suitable high-quality academic datasets, we collect 4,436 human dancing videos from the internet for training. The average length is 1s. We adopt the pre-trained weights from SVD (Blattmann al., 2023). The PoseNet is trained from scratch. We train our model on 8 NVIDIA A100 GPUs for 20 epochs, with a batch size of 8 and 16 frames per clip. The loss weight of the hand region
Figure 4: Qualitative comparison to the state-of-the-art methods on Tik test set. Our method achieves better hand generation quality and adheres better to the reference pose. Note that our method is not trained on the TikTok dataset.
is 10. The learning rate is 10-6 with a linear warmup of 500
MagicPose
Abru Abrupleter wupprransitonn
iterations. We tune all parameters in the UNet and PoseNet.
4.2. Comparison to State-of-the-Art Methods
We compare our method with latest state-of-the-art pose- guided human vidco generation methods, including Mag- icAnymate (Xu et al., 2024), MagicPose (Chang et al., 2023), Mlorr-AnyymteeAng
Moore
Phickering to to
(moo, 2024), and Mu- seV (Xia et al., 2024). For the testing protocol of previous works (Wang et al., 2023; Chang et al., 2023), we adopt the TikTok (Jafarian & Park, 2021) dataset and use sequence 335 to 340 for our evaluation.
Musev
Flickerring &ext
Ours
We conduct qualitative and quantitative comparisons, com- plemented by user study. Each method has a different input aspect ratio. To ensure fair comparison, we only con- sider the central square region of the videos. Specifically, we individually apply center crop to the reference image and pose sequence to match each method's s input aspect ra- tio. Then, we extract the center squares from the generated videos for comparison across different methods.
Figure 5: Comparison of temporal smoothness via frame differences. MagicPose exhibits abrupt transitions, whereas Moore and MuseV show flickering texture and text. In contrast, our method maintains stable frame differences.
differences between consecutive frames. While Magic- Pose (Chang et al., 2023) shows abrupt transitions, Moore- AnymateAnyone (moo, 2024) exhibits unstable clothing textures, and MuseV (Xia et al., 2024) struggles with con- sistent text rendering, our method maintains stable frame-to- frame transitions without visible artifacts. Videos in the sup- plementary material further demonstrate this improvement. This enhancement is enabled by our confidence aware pose guidance, which mitigates the impact of inaccurate pose in- puts and the associated temporal noise. By weighting pose signals by confidence, our method ensures high temporal smoothness in the presence of pose guidance noise.
Qualitative evaluation We conduct qualitative compar- isons between the selected baselines and our method. In Figure 4, we showcase sample frames to highlight the supe- rior quality of individual frames produced by our method. Additionally, in Figure 5, we illustrate the temporal differ- ences, demonstrating the enhanced temporal smoothness of our approach compared to existing methods.
Figure 4 presents a comparison of the generated frames, where each row represents a distinct example. The first row demonstrates the superior hand quality achieved by our approach, while the second row showcases the improved adherence to pose guidance. These improvements directly result from our confidence-aware pose guidance and hand region enhancement design.
Quantitative evaluation Our method outperforms exist- ing approaches across all evaluation metrics, as shown in Table 1. We evaluated performance on test sequences from the TikTok dataset (Jafarian & Park, 2021) using both video- level metrics FID-VID (Balaji et al., 2019) and FVD (Un- terthiner et al., 2018), as well as frame-level metrics PSNR and SSIM ( (Wang et al., 2004) following (Wang et al., 2023).
Importantly, our method shows superior temporal smooth- ness, characterized by smooth motion and minimal flick- ering. Figure demonstrates this through pixel-wise
6

---

MimicMotion; High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance
Method
FID-VID4
FVD4
SSIM
PSNR
Hand
Conf.
Prog.
FID-VID
FVD4
SSIM
PSNR1
MagicAnymate
16.2
848
0.740
17.5
14.6
776
0.760
18.0
MagicPose
13.3
916
0.776
18.8
15.0
678
0.758
17.9
Moore
12,4
728
0.758
18.7
7
12.2
623
0.787
18.4
MuseV
14,6
754
0.766
17.6
4
9.3
594
0.795
20.1
MimicMotion (ours)
9.3
594
0.795
20.1
Table 2: Ablation studies of hand region augmentation (hand), confidence-aware pose guiding (conf.), and pro- gressive latent fusion (prog.).
Table I: Quantitative comparison to state-of-the-art meth- ods: MagicAnymate (Xu et al., 2024), MagicPose (Chang et al., 2023), Moore (moo, 2024), MuseV (Xia et al., 2024). We evaluate on the TikTok dataset test split. MimicMotion the best in all metrics.
this improvement. We analyze each component below.
Ours
96.3
MagicPose
Confidence-aware pose guiding Figure 7 shows the ef- fectiveness of confidence-aware pose guidance. Each row corresponds to one sample. The left side shows three im- ages to extract the pose. On the right side, we plot the corresponding pose guiding signals, both with and without confidence-awe
Ours
98.6
Moore
Ours
75.5
MuseV
0
25
50
75
100
Preference (%)
pose guiding. From the guiding signals, we can see that there are errors in the pose estimated by DW- Pose. Nevertheless, our confidence-awaud design minimizes the impact of incorrect pose estimation in guidance signals.
Figure 6: Preference of videos generated by MimicMotion (ours) over baseline methods on the TikTok dataset test split. Users consistently prefer MimicMotion over other methods.
Specifically, in the case of Pose 1, the estimation exhibits duplicate detection issue, which leads to the inclusion of duplicate keypoints. In the case of Pose 2, there is one hand obscured, and the keypoints of this hand are incorrectly estimated on the other hand; In the case of Pose 3, the right elbow is obscured, but it is still detected with confidence above the threshold thus falsely remains in the guidance signal. These problems lead to confusing hand guidance signals and ultimately lead distortions such as deformed hands or wrong spatial relationships in the generated frames.
The results demonstrate our method's superior performance compared to current state-of- ffate-of-the-are the-art approaches.
User study To supplement our quantitative and qualitative evaluations, we conduct a user study to assess the subjective preferences of participants regarding the generated videos on the TikTok dataset test split. The study involves show- ing two video clips one generated by our method and the other by one of the baseline methods-to to a diverse group of users. Participants are instructed to select the video that they perceived as having higher quality, considering fac- tors including image quality and temporal smoothness of characters and clothing. We collected data from 36 partici- pants, with each participant evaluating 6 video pairs for our method against each baseline method. As shown in Figure 6, the results indicate a strong preference for MimicMotion over the baseline methods. In comparison to MagicPose and Moore, the participants almost favored videos produced by our method. Despite MuseV showing higher image qual- ity compared to other baselines, the preference for videos produced by our method still reached 75.5%. These find- ings align with our qualitative and quantitative evaluation, reinforcing the effectiveness of our method in meeting user expectations for high-quality human video generation.
In contrast, by integrating confidence scores into pose guid- ance, our method effectively mitigates these issues. The confidence scores provide a measure of reliability for each keypoint, allowing our model to weigh the guidance signals accordingly. Specifically, keypoint with lower confidence, which typically correspond to inaccurate keypoints caused by self-oeclusion motion blur, will be of less significance in the guidance. This approach leads to clearer and richer pose guidance, as the influence of potentially erroneous keypoints is reduced. The corresponding generation results demonstrate how our method enhances the robustness of generation against false guiding signals (Pose and Pose 2) and offers visibility hints to resolve the front-back ambiguity of 2D pose estimation (Pose 3).
Hand region enhancement In conjunction with confidence-aware pose guiding, we further improve hand generation quality by assigning higher weight to the hand region in the training loss. Figure 8 compares the generation result with and without hand region enhancement, using the same reference image and pose guidance. All experiments
4.3. Ablation Study
As shown in Table 2, confidence-awand pose guiding and hand region enhancement improve all metrics. Progressive latent fusion notably improves FVD scores, indicating better temporal coherence, though other metrics may not reflect
7

---

MimicMotion; High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance
Y-T Slice
Segment
Segment
Ref
w/o pose confidence
pose pose confidence
Guidance
Generated
Guidance
Generated
Pose
Without progressive latent fusion.
Y-T Slice
Segment
Segment
Pose 2
(b) With progressive latent fusion.
Pose
Figure 9: Progressive latent fusion ensures smooth segment transitions by avoiding temporal discontinuities of weights in simple averaging, which reduces artifacts in Y-T slices.
Figure 7: Confidnce-awware pose guiding enhances gener- ation robustness to false guiding signals (Pose 1&2) and provides hints to handle self-occlusion (Pose 3).
employs simple averaging of frames within the overlap region. This method assigns equal weight to all frames in the overlap region, irrespective of their temporal position (whether they are closer to the preceding or subsequent seg- ment). It lacks a gradual transition from one segment to another, causing noticeable flickers at denoising window boundaries. This is evident in the y-t slice shown on the left of Figure 9a, where the transition at segment boundaries abrupt. The right side of the figure shows four frames at the segment boundary. Note that the background in the top-left corner (enlarged) is initially clear in segment 1. It suddenly becomes blurry in the overlapped region and then suddenly reverts to clearer in the main part of segment 2. This artifact is not observed when progressive latent fusion is applied.
Hand enhrncce Baselime
Figure 8: Hand region enhancement consistently reduces hand distortion and improves visual appeal, using the same reference image and pose guidance.
incorporate confidence-awarc pose guidance. The hands in the first row cropped from the generated video frames of model trained without hand region enhancement, which exhibits noticeable distortions, such as irregular and misplaced fingers. In contrast, the results of the model trained with hand region enhancement (the second row) show consistent improvements in hand generation quality and reduction in hand distortion. These results show the effectiveness of the proposed hand region enhancement design, which substantially mitigates hand distortion, which models. As
The proposed progressive latent fusion approach (see Fig- ure 9b) mitigates these issues. The left y-t slice shows smooth transitions across segment boundaries, avoiding abrupt changes. The right side of the figure demonstrates that this approach eliminates sudden blurring, mitigating flicking artifacts, thus improving the overall visual temporal coherence for long video generation.
5. Conclusion
is prevalent challenge in diffusion-basee humans often focus on hands, this design aligns the training process with human preferences, thereby enhancing the overall visual quality of the results. There is no noticeable degradation the quality of the remaining parts.
In this study, we introduce MimicMotion, pose- mose-guided human video generation model that leverages confidence- aware pose guidance and progressive latent fusion for pro- ducing high-quality, long videos with human motion guided by pose. Through extensive experiments and ablation stud- ies, we show that our model achieves superior adaptation to noisy pose estimation, enhancing hand quality and suring temporal smoothness. The integration of confidence scores into pose guidance, the enhancement of hand region loss, and the implementation of progressive latent fusion are crucial in achieving these improvements, resulting in more visually compelling and realistic human video generation.
Progressive latent fuusion To achieve seamless transitions between video segments, introduce progressive latent fusion, technique that gradually blends frames in the over- lapped regions of consecutive video segments. Figure 9 sample transition with an overlap
presents a comparison of 6 frames between two 16- 16-frame segments. We visualize 4 sample frames. The original MultiDiffusion approach
8

---

MimicMotion; High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance
Impact Statement
Chen, H., Xia, M., He, Y., Zhang, Y., Cun, X., Yang, S., Xing, J., Liu, Y., Chen, Q.. Wang, X., et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprint arXiv:2310.19511 2023.
MimicMotion advances human motion video generation technology with potential applications in creative content production, digital human, and entertainment. The frame- work enables the creation of high-quality, controllable hu- man motion videos. However, like other video generation technologies, there are potential risks of misuse for creating deceptive content that could be used for misinformation. Re- sponsible development and deployment of such technology should include appropriate safeguards, such as watermark- ing, detection methods, and clear usage guidelines. We encourage the development of technical capabilities and ethical frameworks for video generation technologies.
Chen, H., Zhang, Y., Cun, X., Xia, M., Wang. X., Weng, C., and Shan, Y. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprint 1.xii:2401.0904: 2024.
Feng, M., Liu, J., Yu, K., Yao, Y., Hui, Z., Guo, X., Lin, X., Xue, H., Shi, C., Li, X., al. Dreamoving: A human video generation framework based on diffusion models, 2023.
Guo, Y., Yang. C., Rao, A., Wang, Y., Qiao, Y., Lin, D., and Dai, B. Animatediff: Animate your personalized text-to text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.0472. 2023.
References
URL MonreTTheaakMMooo---nnnttttttt
May 2024. com/MooreThheadds original-date: 2024-01-
Aooore-AiimateAaee 12T07:55:21Z.
He, Y., Yang, T., Zhang, Y., Shan, Y., and Chen, Q. La- tent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211. 2022.
Balaji, Y., Min, M. R., Bai, B., Chellappa, R., and Graf, H. P. Conditional gan with discriminative filter generation for text-to-video synthesis. In IJCAI, volume 1, pp. 2, 2019.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Bao, F., Xiang, C., Yue, G., He, G., Zhu, H., Zheng, K., Zhao, M., Liu, S., Wang. Y., and Zhu, J. Vidu: a highly consistent, dynamic and skilled text-to-video generator with diffusion models. arXiv preprint arXiv:2405.04233, 2024.
Ho, J.. Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., and Fleet, D. J. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8646, 2022.
Bar-Tal, O., Yariv, L., Lipman, Y., and Dekel, T. Multi- diffusion: Fusing diffusion paths for controlled image generation. arXiv preprint arXiv:2302.08/1 13, 2023.
Hu, L. Animate anyone: Consistent and controllable image- to-video synthesis for character animation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8153-88 8163, 2024.
Bar-Tal, O., Chefer, H., Tov, O., Herrmann, C., Paiss, R., Zada, S., Ephrat, A., Hur, J., Li, Y., Michaeli, T., et al. Lu- miere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401.1 12945, 2024.
Jafarian, Y. and Park, H. S. Learning high fidelity depths of dressed humans by watching social media dance videos. Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), pp. 12753- 12762. June 2021.
Blattmann, A.. Dockhorn, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z.,, Voleti, V., Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127. 2023.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the Design Space of Diffuusion Based Generative Mod- els, October 2022. URL httl: a /tt/vviivvve///ev//ee/////aaaeva wxw..oorrrr orgg /abs/ 2206. 00364. arXiv:2206.0000
Berrusdoo-BBeedd
[cs, stat].
Chan, C., Ginosar, S., Zhou, T., and Efros, A. A. Everybody dance now. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pp. 5933-5942, 2019.
Li, H., Li, Y., Yang, Y., Cao, J., Zhu, Z., Cheng, X., and Long, C. Dispose: Disentangling pose guidance for controllable human image animation. arXiv preprint arXiv:2412.09344 2024.
Chang, D., Shi, Y., Gao, Q. Xu, H., Fu, J., Song, G., Yan, Q., Zhu, Y., Yang. X., and Soleymani, M. Magicpose: Realistic human poses and facial expressions retargeting with identity aware diffusion. In Forty-furs, first International Conference on Machine Learning, 2023.
Ma, X., Wang. Y., Jia, G., Chen, X., Liu, Z., Li, Y.F., Chen, C., and Qiao, Y. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401 03048, 2024a.

---

MimicMotion; High-Quality Human Motion Video Generation with Confidence-awwre Pose Guidance
Ma, Y., He, Y., Cun, X., Wang, X., Chen, S., Li, X., and Chen, Q. Follow your pose: Pose-guided text-to-video generation using pose-free videos. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4117-4125, 2024b.
Unterthiner, T., Van Steenkiste, S., Kurach, K., Marinier, R., Michalski, M., and Gelly, S. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv: 181.01171 2018.
Voleti, V.,
A., and Pal, C. Mevd- masked conditional video diffusion for prediction, gener- ation, and interpolation. Advances in neural information processing systems, 15:23371-23382 2022.
Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pp. 4195 2023.
Wang, Q. Bai, X., Wang, H., Qin, Z., and Chen, A. Instan- tid: sero-shot identity-preserring generation in seconds. arXiv preprint arXiv:2401.07519, 2024a.
Peng, B., Wang. J., Zhang, Y., Li, W., Yang, M.-C., and Jia, J. Controlnext: Powerful and efficient control for image and video generation. arXiv preprint arXiv:2408.0600 2024.
Wang. Q., Jiang, Z., Xu, C.,, Zhang, J., Wang, Y., Zhang, X., Cao, Y., Cao, W., Wang. C., and Fu, Y. Vividpose: Ad- vancing stable video diffusion for realistie human image animation. arXiv preprint arXiv:2405. 181566. 2024b.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G. Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp.
PMLR, 2021.
Wang, T., Li, L.., Lin, K., Zhai, Y., Lin, C.-C., Yang, Z., Zhang, H., Liu, Z., and Wang, L. Disco: Disentan- gled control for realistic human dance generation. arXiv preprint vrXiv:2307.00044 2023.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffuuson models. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition. pp- 10684-10695, 2022a.
Wang, W., Liu, J., Lin, Z., Yan, J., Chen, S., Low, C., Hoang, T., Wu, J., Liew, J. H., Yan, H., et al. Magicvideo- v2: Multi-stage high-aesthetic video generation. arXiv preprint N1v:22401.044666 2024c.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 00684-10695, 2022b.
Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. Image quality assessment from error visibility struc- tural similarity. IEEE transactions on image processing, 13(4):600-61 610000612222000 2004.
Siarohin, A., Lathuilière, S.. Tulyakov, S.. Ricci, E.. and Sebe, N. First order motion model for image animation. Advances in neural information processing systems, 32, 2019.
Wu, J. Z.,, Ge, Y., Wang, X., Lei, S. W., Gu, Y., Shi, Y., Hsu, W., Shan, Y., Qie, X., and Shou, M. Z. Tune-a- video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7623- 7633, 2023.
Siarohin, A., Woodford, O. J., Ren, J., Chai, M., and Tulyakov, S. Motion representations for articulated imation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13653- 13662, 2021.
Xia, Z., Chen, Z., Wu, B., Li, C., Hung. K.-W., Zhan, C., He, Y., and Zhou, W. Musev: Infinite-length and high fidelity virtual human video generation with visual conditioned parallel denoising. arxiv, 2024.
Li,
Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni, O., et al. Make-a- video: Text-to-video generation without tixt-video data. arXiv preprint arXiv:2209.1477 14792, 2022.
Xu, Z.., Zhang. J., Liew, J. H., Yan, H., Liu, J.-W., Zhang, C., Feng, J., and Shou, M. Z. Magicanimate: Temporally consistent human image animation using diffusion model. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 1481-1490, 2024.
Song. Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-hased generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13355 2020.
Tu. S., Xing, Z., Han, X., Cheng, Z.Q.,, Dai, Q. Luo, C., and Wu, Z. Stableanimator: High-quality identity- preserving human image animation. arXiv preprint arXiv:2411. 2024.
Yan, W., Zhang, Y., Abbeel, P., and Srinivas, A. Videogpt: Video generation using vqe and transformers. arXiy preprint arXiv:2 200Xx:2204./// 10/57, 2021.
10

---

MimicMotion; High-Quality Human Motion Video Generation with Confidence-aware Pose Guidance
Yang, Z., Zeng, A., Yuan, C., and Li, Y. Effective whole- body pose estimation with two-stages distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4211-222000 4220, 2023.
Yin, S., Wu, C., Yang. H., Wang, J., Wang, X., Ni, M., Yang, Z., Li, L., Liu, S., Yang, F., et al. Nuwa-xl: Diffusion over diffusion for extremely long video generation. arXiv preprint ar ArXiv:2303. 1ete22333...2444 12346, 2023.
Yu, L., Cheng, Y. Sohn, K., Lezama, J., Zhang, H., Chang, H., Hauptmann, A. G., Yang, M.-H., Hao, Y., Essa, I. et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10459-10469, 2023.
Zhang, L.,, Rao, A., and Agrawala, M. Adding conditional control text-to-image diffusion models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 3836-3847, 2023.
Zhao, J. and Zhang, H. Thin-plate spline motion model for image animation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 3657-3666, 2022.
Zheng, T., Geng, C., Jiang. P., Wan, B., Zhang, H.. Chen, J., Wang, J., and Li, B. Non-uniform timestep sam- pling: Towards faster diffusion model training. In Cai, J.. Kankanhalli, M. S., Prabhakaran, B., Boll, S., Subra- manian, R., Zheng, L., Singh, V. K., César, P., Xie, L.,, and Xu, D. (eds.), Proceedings of the 32nd ACM Intera- tional Conference on Multimedia, MM 2024, Melbourne, VIC, Australia, 28 October 2024- November 2024, pp. 7036-7045 ACM, 2024a.
Zheng. T., Jiang, P., Wan, B., Zhang, H., Chen, J., Wang, J., and Li, B. Beta-tuned timestep diffusion model. Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sat- tler, T., and Varol, G. (eds.), Computer Vision ECCV 2024 18th European Conference, Milan, Italy, Septem- ber 29-October 4, 2024, Proceedings, Part III, volume 15061 of Lecture Notes in Computer Science, pp. 114 130. Springer, 2024b.
Zhu, S., Chen, J. L., Dai, Z., Xu, Y., Cao, X., Yao. Y., Zhu, H., and Zhu, S. Champ: Controllable and consistent human image animation with 3d parametric guidance, 2024.

---

MimicMotion; High-Quality Human Motion Video Generation with Confidencc--wware Pose Guidance
A. Training Details
MimicMotion is tuned from Stable Video Diffusion (Blattmann et al., 2023). The video samples are resized to the resolution of 576 1024. The training dataset contains 4,436 human dancing videos collected from the internet. We train the model for 20 epochs at batch size of 8. We follow Stable video diffusion and adopt the noise distribution, i.c. log [ P ) proposed by Karras et al (Karras al., 2022) with parameter Pmean 0.5 and Psu 1.4. We train model 8 NVIDIA A100 GPUs with batch size of 8 and 16 frames per clip. The loss weight assigned to the hand region is set at 10, while the remaining part is left to 1. The learning rate is 10 with a linear warmup schedule of 500 iterations. We tune all parameters in the UNet and PoseNet.
B. PoseNet Model Architecture
The PoseNet consists of convolutional and activation layers. The detailed architecture is listed in Table 3.
Table 3: Detailed Architecture of PoseNet
#
Type
Channels
Kernel size
Stride
Pad
1
Input
3
2
Conv2d
3
3x3
3
SiLU
4
Conv2d
16
4x4
2
1
5
SiLU
6
Conv2d
16
3x3
1
1
7
SiL.U
8
Conv2d
32
4x4
2
9
SiLU
10
Conv2d
32
3x3
11
SiLU
12
Conv2d
64
4x4
2
13
SiLU
14
Conv2d
64
3x3
15
SiLU
16
Conv2d
128
1
1
17
SiLU
18
Conv2d
320
1xl
0
C. Additional Details on User Study
The user study was designed to evaluate the subjective preferences participants regarding the quality of videos generated by our method, MinicMotion, compared to several baseline methods on the Tok dataset test split. Participants were instructed to select the video they perceived as having higher quality. The data collection process online interface, as depicted in Figure 10, which allowed participants to easily view and compare the video clips and submit their preferences. The results of our method will randomly appear in the video on the left or the right, with the other video being the comparative method.
conducted through an
D. Cross-Domain Results: Cartoon and Animal Videos
To demonstrate the versatility of MimicMotion, evaluate its performance cross-domain scenarios, including 4 cartoon and 4 animal dancing videos. The videos are available on anonymous project page. As shown in Figure 11, the model generates cartoon dancing videas while maintaining the artistic style of the characters. For animal subjects (Figure 12), our method produces reasonable results despite their significantly different appearance from humans.
These results highlight MimicMotion's generalizability, which stems from two key factors: first, the pose feature space of humans, cartoons, and animals may share common subspace that preserved in our training, human control signals can also guide cartoons and animals. Second, map the limb lengths of the pose template the reference character following MuseV (Xia et al., 2024), making the pose guidance signal closer to the reference character. This helps the model maintain the character's appearance while adapting poses across varying body proportions.
12

---

MimicMotion; High-Qualit Human Motion Video Generation with Confidence-aware Pose Guidance
Play all
Please pay attention the image quality, whether there flickering, the temporal consistency of the characters and clothing, and select the higher quality video (you can rate the video after it finlshes playing).
Loft
Right
Figure 10: Snapshot of the user preference selection interface that prompts users to evaluate image quality and temporal smoothness. Note that the two videos play automatically, with method randomly appearing on one side.
13

---

MimicMotion; High-Qualit, Human Motion Video Generation with Confidence--wwree Pose Guidance
Reference images
Reference pose
Figure 11: Generated cartoon dancing videos. Our method successfully preserves the artistic style of cartoon characters
while generating natural motions that follow the input poses. Top: reference frame; Left: reference pose guidance; Bottom: generated frames at different timestamps.
14

---

MimicMotion; High-Quality Human Motion Video Generation with Confidence--wwree Pose Guidance
Reference images
iff
Reference pose
ift
Figure 12: Generated animal dancing videos. MimicMotion can generalize to animal dancing zero-shot, producing plausible
motions despite the significant appearance differences from humans. Top: reference frame; Left: reference pose guidance; Bottom: generated frames at different timestamps.
15


Reviews:
[{"review_id": "efCTKHNVl7", "reviewer": "Reviewer_NDx5", "summary": "This research focuses on the task of video generation, aiming to produce videos where the subject replicates a reference pose based on a single input image and specified pose conditions. \n\nThe paper introduces three significant advancements over existing baseline methods: \n1. Integrating confidence awareness to reduce the influence of uncertain poses, thus mitigating the detrimental impact of incorrect pose conditions during model training.\n2. Loss amplification for hand regions to enhance training weights in high-confidence situations.\n3. Leveraging progressive latent fusion to ensure smooth transitions across multiple video segments, enabling the generation of longer and more cohesive videos.\n\nComprehensive evaluations reveal that the proposed approach surpasses baseline methods both qualitatively and quantitatively on established public benchmarks.\n\n\n## update after rebuttal\nI have reviewed the rebuttal text provided by the authors. Since there was a clear response to my concerns and my initial score was to recommend acceptance, I will maintain my score unchanged.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **1. Essential References Not Discussed: DiT**\nThank you for your effort in reviewing our paper. We will revise our manuscript and add a discussion about DiT. Compared with U-Net, DiT is considered a more novel and promising foundation model, especially in video generation tasks, where it demonstrates scalable capabilities.\n\n### **2. Is the image encoder in Figure 2 CLIP?**\nYes, it is a typical CLIP image encoder.\n\n### **3. The authors mention in line 206 that \"Pose guidance is not added to every U-Net block\". What kind of degradation effects would occur if pose control were introduced in multiple layers of the UNet structure? Can comparative examples be provided?**\n\nWe apologize for the confusion. Here, we mean pose guidance is _only_ added to the output of the first convolution layer of U-Net, as we found this is sufficient for pose control in experiments.\n\nThis design choice is based on two hypotheses: First, introducing sequence pose information extracted without temporal interaction could potentially disrupt the spatio-temporal representations within subsequent U-Net layers. Second, since our image-to-video base model was pre-trained without posture information, excessive integration of pose sequence data may degrade pertained performance.\n\nTo ensure clarity, we will update the relevant section accordingly.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No Theoretical Claims.", "experimental_designs_or_analyses": "The paper presents a thorough and meticulous set of experiments, with a particular focus on the ablation studies examining the three proposed improvements. Both the qualitative and quantitative findings strongly validate the authors' claims, showcasing the effectiveness of the proposed approach.", "relation_to_broader_scientific_literature": "None", "essential_references_not_discussed": "- Scalable Diffusion Models with Transformers (DiT) [ICCV2023]\n    - Although using different architectures, U-Net vs. DiT, I still hope the authors mention or discuss this important foundational model and its derivatives based on DiT for controllable video generation in the Related Work section.", "other_strengths_and_weaknesses": "The article presents a clear motivation and outlines its proposed improvements in a well-defined manner. Each improvement point is effectively articulated, and the methodology is introduced coherently. Overall, the article is well-crafted, with a strong focus throughout, and the experimental results are highly persuasive. No significant weaknesses are apparent.", "other_comments_or_suggestions": "None", "questions_for_authors": "- Is the image encoder in Figure 2 CLIP?\n- The authors mention in line 206 that \"Pose guidance is not added to every U-Net block\". What kind of degradation effects would occur if pose control were introduced in multiple layers of the UNet structure? Can comparative examples be provided?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XcdXxRHvNd", "reviewer": "Reviewer_eXfG", "summary": "Based on the confidence-aware strategy, this work propose hand region enhancement to alleviate hand distortion, which improves video generation performance. Additionally, the supplementary materials provide generation results in other scenes (such as animals and other styles), verifying the generalization ability of the algorithm.\nI believe this work has a good practicability and the whole paper is easy to read, more ablation can be conducted to validate each part of the contribution.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **1. Essential References Not Discussed:** StableAnimator, ControlNeXt\n\nThank you for your effort and valuable feedback during the review process. We will add a discussion of these two related works and revise our manuscript accordingly.\n\n### **2. The influence of hand region enhancement.**\n> Does the hand region enhancement suppress the generation of other areas to some extent? I hope to see whether videos without hand enhancement could achieve better results, and whether the non-hand regions in videos with enhanced hands still maintain good quality.\n\nThank you for your valuable suggestion. We have updated the supplementary website with full-frame and video comparisons (see Sec. Ablation). As demonstrated by multiple examples, we observe no significant quality degradation in non-hand regions while maintaining enhanced hand generation.\n\n### **3. The claim of long-video generation.** \n> How long can other methods generate high-quality videos? It would be useful to show how the performance of the proposed method changes as the video length increases, both quantitatively and qualitatively.\n\nThe video lengths of video generation base models are as follows. AnimateDiff: 2s, SVD: 3.6s, CogvideoX-5B: 6s, LTX-Video: 10.6s.\nWe discussed long-video generation in a sub-section of *Related Work*.\nExisting methods adopt a windowed generation approach to produce longer videos, which can be extended to infinitely long videos; however, there are still issues with abrupt changes at the window boundaries. In contrast, our proposed progressive latent fusion mitigates this problem.\n\nTo further show how performance changes as the video length increases, we have added comparative videos to the website. This method has an independent good case rate for each window. While bad cases may occur over time, the quality of video clips will not deteriorate as the total video duration increases.", "claims_and_evidence": "The claim for \"long-video\" should be compared with others.", "methods_and_evaluation_criteria": "yes", "theoretical_claims": "yes", "experimental_designs_or_analyses": "The main result validates the SOTA performance, while more ablations on long-video and effect of hand should be disscuss (see weakness & strength)", "relation_to_broader_scientific_literature": "This work provide a strong potential for human-based video generation and has a good practicability.", "essential_references_not_discussed": "-StableAnimator: High-Quality Identity-Preserving Human Image Animation\n\n-ControlNeXt: Powerful and Efficient Control for Image and Video Generation\n\nThese similar works need to be discussed.", "other_strengths_and_weaknesses": "1.Based on the confidence-aware strategy, the proposed hand region enhancement alleviate the hand distortion. However, I would like to see more detailed discussion beyond the ablation study, such as:\n* Does the hand region enhancement suppress the generation of other areas to some extent?\n* I hope to see whether videos without hand enhancement could achieve better results, and whether the non-hand regions in videos with enhanced hands still maintain good quality.\n\n2.There are many similar works that are not mentioned and should be discussed in the Related Work section. These works should be compared with the proposed approach in the experiment. A discussion and comparison with these methods is needed.\n\n-StableAnimator: High-Quality Identity-Preserving Human Image Animation\n\n-ControlNeXt: Powerful and Efficient Control for Image and Video Generation\n\n\n3. Regarding the claim of long-video generation, I would like to see a comparison with other methods. For example:\n* How long can other methods generate high-quality videos?\n* It would be useful to show how the performance of the proposed method changes as the video length increases, both quantitatively and qualitatively.", "other_comments_or_suggestions": "no", "questions_for_authors": "see weakness", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NbvObBCtKw", "reviewer": "Reviewer_a5kM", "summary": "This paper proposed an SVD-based human video animation framework. The major contributions of this work are 1) a confidence-aware strategy to reduce the negative impact of inaccurate pose skeleton detection 2) a local hand region enhancement strategy to alleviate hand motion blur 3) a long video generation strategy with weighted frames fusion.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **1. Out-of-Domain Generalization**\n\nThe model is trained exclusively on human dancing videos. There are two reasons for this generalizability:\n\nFirst, the SVD backbone can generate cartoon and animal videos. Our fine-tuning enhances the model with pose controllability, and since the pose feature space of humans, cartoons, and animals may share a common subspace that is preserved in the fine-tuned model, human control signals can also guide cartoons and animals. The results in AnimateAnyone and MagicPose also verify this point.\n\nSecond, we map the limb lengths of the pose template to the reference character following MuseV, making the pose guidance signal closer to the reference character. This helps the model preserve the character's appearance and also helps to adapt one pose to another with different body proportions, enhancing generalization.\n\nWe thank the reviewer for raising these concerns and will add these explanations in the revision.\n\n### **2. Missing Baseline Comparisons**\n\nWe have enhanced our evaluation by adding comprehensive comparisons with Champ both quantitatively (see table below) and qualitatively (refer to the Comparisons section on our website). Regarding DisPose, we respectfully note that it was released within 4 months of our submission date, which falls outside the comparison requirements according to the conference guidelines. For VividPose, despite our efforts, its unavailability as open-source presents reproduction challenges.\n\nOur method specifically focuses on 2D DWPose-guided video generation, which offers practical advantages by eliminating SMPL and other inference overhead, and better handling out-of-domain characters where SMPL-based approaches face significant limitations.\n\n||FID-VID↓|FVD↓|SSIM↑|PSNR↑|\n|-|-|-|-|-|\n|Champ|17.3|736|0.686|15.0|\n|Ours|**9.3**|**594**|**0.795**|**20.1**|\n\n### **3. Supplementary Website Demos**\n\nThank you for the suggestion. We have replaced all demos on the website with their raw versions without post-processing. It needs to be clarified that, for the cases of *comparison* and *ablation*, no post-processing was applied in the original web page. We used post-processing operations in the original web page only to showcase better visual effects in demos, without affecting the contributions of this paper.\n\n### **4. Ablation Analysis of SVD and SD Backbone**\n\nThank you for your suggestion. We have added videos corresponding to Fig. 9's progressive latent fusion on the website.\n\nRegarding the SD Backbone, we believe that progressive latent fusion would not provide significant improvement. Videos generated with the SD backbone exhibit high-frequency flickering within a single window (see baseline comparison on the website). Progressive latent fusion only enhances transition smoothness between denoising windows, without improving smoothness within windows. Therefore, the SD backbone would likely still suffer from flickering. We have added a Moore AnimateAnyone (SD backbone) + proposed strategy result on the website, which verifies this point.\n\nLike other research in this field, our paper deliberately focuses on a specific backbone (SVD), which is both well-pretrained and open-source. The issues with the SD Backbone fall beyond the scope of our current research objectives, as it is standard practice to focus on optimizing a single backbone network.", "claims_and_evidence": "The contributions summarized in the introduction sections are clear and supported by experimental evaluation of baseline comparison and ablation analysis of each module.", "methods_and_evaluation_criteria": "Different from prior works with ReferenceNet architecture and AnimateDiff motion modules, MimicMotion adopted SVD as the diffusion backbone model, which makes sense for human video animation, especially for the motivation of ensuring the temporal smoothness of generated videos. The benchmark dataset, the TikTok dataset, is broadly used in previous works in this area.", "theoretical_claims": "The major theoretical claim in this paper, progressive latent fusion for long video generation, is supported by clear visualization in Figure 3 and detailed explanation in Algorithm 1.", "experimental_designs_or_analyses": "I appreciate the authors' efforts in providing baseline comparisons and visualizations in both the main paper and the anonymous website. However, I have several major concerns regarding the experimental design:\n\n1) Out-of-Domain Generalization: Section 4.1 states that the model is trained exclusively on human dancing videos, with the UNet backbone fine-tuned. For the out-of-domain visualizations featuring cartoon and animal identities, was the MimicMotion pipeline also fine-tuned on such data? If so, this should be explicitly mentioned in the Cross-Domain Results section. If not, is there any explanation for the observed generalizability?\n\n2) Missing Baseline Comparisons: The authors highlight contributions such as local hand pose enhancement and pose guidance using a confidence-aware strategy. However, a more thorough discussion and evaluation against prior works with similar motivations are needed. Comparisons with methods like DisPose (Sparse and Dense Motion Field), Champ (SMPL with depth/normal/semantic maps), and VividPose (SMPL-X-based pose) would strengthen the claims.\n\n3) Supplementary Website Demos: The website notes that videos are post-processed with face swapping and frame interpolation. However, these post-processing steps are not mentioned in the main paper. This should either be clearly discussed, or the authors should provide original videos generated by MimicMotion to demonstrate the out-of-domain real-human animations.\n\n4) Ablation Analysis of SVD and SD Backbone: MimicMotion employs SVD instead of the widely used ReferenceNet + AnimateDiff architecture in the baselines. This raises uncertainty about the effectiveness of the proposed progressive latent fusion. Is the performance gain attributable to the proposed strategy, or is it primarily due to the choice of video backbone? I appreciate the visualization in Figure 9, but this should be visualized as a video comparison between 1) RefenceNet+proposed strategy, 2) SVD, and 3) SVD+proposed strategy on the supplementary website to observe the difference better.", "relation_to_broader_scientific_literature": "The key contribution of this paper is the SVD-base long video generation pipeline with smooth motion consistency. This work explores the potential usage of a pre-trained video diffusion backbone for human video animation, which is a slightly novel architectural design compared to previous works.", "essential_references_not_discussed": "Please see the \"Experimental Designs Or Analyses\" section.", "other_strengths_and_weaknesses": "Please see the \"Experimental Designs Or Analyses\" section.", "other_comments_or_suggestions": "Please see the \"Experimental Designs Or Analyses\" section. I'm more than willing to **raise** my score if my concerns are addressed.", "questions_for_authors": "Please see the \"Experimental Designs Or Analyses\" section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4iY4aySH9t", "reviewer": "Reviewer_1Stn", "summary": "This paper presents MimicMotion, a generative approach for animating a human in a single image using a 2D pose sequence. It employs a fixed pre-trained video diffusion model while training select layers, including PoseNet and U-Net modules. The PoseNet processes the 2D pose sequence, and its output is integrated into the U-Net to combine pose information with the human appearance from the image. The paper introduces several technical contributions: 1) confidence-aware pose guidance to account for the reliability of estimated 2D poses, 2) hand region enhancement to emphasize loss around hand regions, and 3) progressive latent fusion for generating long videos.\n\n## update after rebuttal\n\nThe authors addressed most of my concerns. I raised my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **1. Is confidence-aware pose guidance the optimal approach?**\n\nWe evaluated depth information as an additional input and found it does not significantly improve the resolution of front-back ambiguity or incorrect pose issues beyond what our confidence-aware approach already achieves. Depth estimation introduces its own prediction errors and computational overhead without commensurate benefits.\nWhile alternative guiding modalities have been explored in prior work (e.g., Champ), each presents inherent limitations—depth varies with camera distance, and part segmentations may compromise action template transferability across different contexts.\nThe reviewer's suggestion, though theoretically plausible, lacks empirical substantiation. We maintain that our approach represents an effective balance of performance and efficiency within the scope of our research objectives.\n\n### **2. Why hand region enhancement improves all evaluation metrics?**\n\nWe respectfully note that our approach does not solely assign higher loss weights to hand regions. Rather, it strategically enhances weights for clear hand regions, encouraging the model to generate more defined hands. Given the iterative nature of the diffusion process, we observe that improved hand clarity appears to serve as a quality signal that positively influences overall video generation quality across successive denoising steps.\n\n### **3. Progressive latent fusion results in lower quantitative performance.**\n\nTo address this, we have supplemented the website with comparison videos corresponding to Fig. 9's progressive latent fusion to better illustrate the qualitative differences.\n\nThe discrepancy in quantitative metrics stems from inherent limitations in evaluation metrics: SSIM and PSNR are inadequate for evaluating our progressive latent fusion approach, as they assess image-level quality rather than transition smoothness between denoising windows. Similarly, Fréchet distance-based metrics rely on video classification models pre-trained on datasets like Kinetics, which primarily leverage spatial rather than temporal features for classification. These datasets can often be classified accurately from single frames, indicating their limited sensitivity to temporal consistency issues.\n\n### **4. Post-processing**\n\nThank you for raising this concern. We have addressed this by replacing all videos on our website with versions without post-processing.\n\nWe emphasize that the referred Figure 4 and all other _comparison_ and _ablation_ results in the paper and website are presented without post-processing. The identity preservation challenges visible in our results and baselines represent the actual performance of these methods, not artifacts of face swapping or other post-processing techniques.\nThe clothing color shift observed in Figure 4 occurs to varying degrees across multiple methods, including ours, confirming the fairness and transparency of our evaluation.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "They can be related to various human-related computer vision tasks.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths\n\nThe paper addresses practical challenges in diffusion-based human animation methods. The three proposed techniques—confidence-aware pose guidance, hand region enhancement, and progressive latent fusion—effectively enhance the quality of animated human videos.\n\nWeaknesses\n1. Is confidence-aware pose guidance the optimal approach? While incorporating confidence information can mitigate issues like depth ambiguity and incorrect poses, alternative approaches might be more effective. For instance, integrating depth maps or part segmentations as additional inputs to PoseNet could address both depth ambiguity and incorrect pose estimation more comprehensively. With the great success of vision foundation models, we can relatively easily obtain depth maps and part segmentations from a single image using Depth Anything v2 and Sapiens. Since 2D poses lack depth information, relying solely on confidence-aware guidance may not be sufficient. \n\n2. Confusing impact of hand enhancement on overall metrics. In Table 2, the hand region enhancement improves all evaluation metrics, including those unrelated to hands. Since this method only assigns higher loss weights to hand regions without affecting other body parts, it is unclear why overall metrics such as PSNR see significant improvement. While Figure 8 demonstrates enhanced focus on hand regions, the broader impact on all human parts needs further clarification. \n\n3. Progressive latent fusion results in lower quantitative performance. Table 2 indicates that this technique performs worse in three out of four metrics. However, the paper lacks a detailed analysis of why this occurs. \n\n4. Post-processing methods may obscure actual performance. The supplementary material states that face swapping and frame interpolation were applied to the generated videos, which can significantly influence the authenticity and temporal smoothness of the results. The paper should explicitly state which face-swapping and frame-interpolation techniques were used. Furthermore, relying on post-processing might hide weaknesses of MimicMotion, such as difficulties in maintaining identity consistency—an inherent limitation of diffusion-based human animation. The authors should provide extensive qualitative results without any post-processing to ensure a fair evaluation. Additionally, the main manuscript's figures may also be affected by these techniques. For example, Figure 4 shows clothing color changes (white turning into blue), highlighting potential challenges in preserving identity.", "other_comments_or_suggestions": "Please see weaknesses.", "questions_for_authors": "Will the collected training and evaluation videos be public?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents MimicMotion, a framework for high-quality human motion video generation using confidence-aware pose guidance, hand region enhancement, and progressive latent fusion. Reviewers found the method well-motivated with good results. While there were initial concerns about identity preservation, ablation completeness, and comparisons, the authors provided comprehensive responses and updated demos without post-processing. Most reviewers raised their scores finally.
Decision: Accept (poster)