Title: TPC: Test-time Procrustes Calibration for Diffusion-based Human Image Animation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Human Image Animation

Diffusion model

Procrustes Analysis
Abstract: Human image animation aims to generate a human motion video from the inputs of a reference human image and a target motion video. Current diffusion-based image animation systems exhibit high precision in transferring human identity into targeted motion, yet they still exhibit irregular quality in their outputs. Their optimal precision is achieved only when the physical compositions (i.e., scale and rotation) of the human shapes in the reference image and target pose frame are aligned. In the absence of such alignment, there is a noticeable decline in fidelity and consistency. Especially, in real-world environments, this compositional misalignment commonly occurs, posing significant challenges to the practical usage of current systems. To this end, we propose Test-time Procrustes Calibration (TPC), which enhances the robustness of diffusion-based image animation systems by maintaining optimal performance even when faced with compositional misalignment, effectively addressing real-world scenarios. The TPC provides a calibrated reference image for the diffusion model, enhancing its capability to understand the correspondence between human shapes in the reference and target images. Our method is simple and can be applied to any diffusion-based image animation system in a model-agnostic manner, improving the effectiveness at test time without additional training.

TPC: Test-time Procrustes Calibration for Diffusion-based Human Image Animation

Abstract

1 Introduction

Recent advancements [33, 31] of human image animation systems have demonstrated notable precision in adapting human identity into target motion. Despite advancements, these systems continue to suffer from irregular quality of image animation when the compositions ( i . e ., scale and rotation) of human shapes are not aligned between reference image and target motion. To be specific, Figure 1 (a) presents exploratory experiments on the compositional misalignment of human shapes between the reference and target. As shown in the left experiments, for a given target pose, adjusting the composition of the same human in the reference image ( i . e ., via scaling or rotating) results in inconsistent and low-fidelity output images, especially in terms of clothes and faces of the human.

(a) Predictions on samples of compositional misalignment of human shapes between target and reference: misalignment by reference image variation (left) and misalignment by target pose variation (right)

Original

Scaling

Rotating

Reference

Target

pose

MA [

23

]

Ours

(Walking motion)

Scaling

Rotating

𝑡

𝜃

Target

Reference

Relative

angle

Pelvis

Nose

PSNR according to relative angle between target and reference

(PSNR)

30

26

(Relative angle

𝜽

)

-

𝝅

𝝅

0

𝝅/𝟐

-

𝝅/𝟐

Robust

Pelvis

Nose

Sensitive

PSNR according to scaling between target and reference

30

20

1.0

0.6

0.2

(

IoU

)

Target

Reference

(scale = 0.7)

IoU

Target

Reference

(PSNR)

Robust

Sensitive

Reference

Target

motion

𝑡

DisCo [

21

]

Ours

Sensitive

(b) Sensitivity analysis according to scaling (left) and rotating (right)

Furthermore, in the right experiments, when providing motion sequences that display various dynamic movements for a given reference image, the human image animation outputs consistently show low fidelity, especially evident in target poses ( e . g ., bending or approaching close to viewpoint) that cause significant differences in the composition of the human shape. To quantitatively investigate these, Figure 1 (b) presents a sensitivity analysis evaluating how current image animation systems [33, 31] respond to varying degrees of the compositional misalignment of human shape. The left shows the fidelity ( i . e ., PSNR) of the resulting human according to the relative scale difference of input human shapes between the target and reference. We employ the Intersection of Union (IoU) of bounding boxes of the shapes for relative scale. The right shows fidelity according to the variations of relative angle θ 2 of human shapes between the target and reference. Current systems demonstrate significant vulnerability to the variations of the compositions, indicating that output fidelity forms a robust region only in the areas with compositionally aligned conditions ( i . e ., -π 6 < θ < π 6 , IoU > 0 . 7 ).

In fact, diffusion-based systems are inevitably susceptible to this compositional misalignment. The diffusion model uses a reference human image as a condition to generate controlled output from noise based on the target pose. Here, the conditioning is performed through crossattention based on a visual similarity between patch-wise features of the target pose frame and reference image. To be specific, Figure 2 shows the attention map about a single patch of the target frame ( i . e ., shoulder denoted by blue point) from the reference image during the denoising process. The upper section displays a sample where the human shapes are relatively aligned between the target and the reference, while the lower section shows a case where this alignment is not present. Initially, the attention maps were blurry in both cases. However, as denoising con-

961

79

Denoising step

Target pose

Attention maps on reference image

high

Low

high

Low

Denoising step

961

79

tinued, it became clear that samples with aligned human shapes correctly established a correspondence between the target frame and the reference image. ( i . e ., Attention to the shoulder on the target frame is focused on the shoulder on the reference image.) However, when the shape is misaligned, attention to the target frame's shoulder is incorrectly focused on unrelated areas in reference, even up to the

Enc

UNet

(Conditioning)

Reference

image

Target pose

(

𝒊

-th frame)

Animation

(

𝒊

-th frame)

Latent feature

Diffusion model

(a) Diffusion-based human image animation

Enc

TPC

Calibration branch

Diffusion model

UNet

(Conditioning)

,

Figure 3: Illustration of (a) current diffusion-based human image animation systems and (b) Test-time Procrustes Calibration (TPC) on top of these systems. The TPC can be applied to diffusion-based models in a model-agnostic manner, enhancing the fidelity and consistency of the output video.

last stages of denoising. Consequently, the compositional misalignment between the target and the reference image hinders accurate attention correspondence throughout the denoising process.

To this end, we propose a diffusion guidance referred to as Test-time Procrustes Calibration (TPC). As shown in Figure 3 (a), the existing diffusion-based human image animation system takes inputs of reference image x and i -th target pose frame p i , where it generates i -th output animation frame y i . Here, the x is encoded into the latent feature z , which serves as a conditioning input for the denoising diffusion model ( i . e ., UNet). As depicted in Figure 3 (b), our proposed TPC incorporates an auxiliary branch

Style variation

Fidelity

Low

Shape variation

High

(a) Style and shape without TPC

Reference

Target

Output

TPC

Shape variation

Calibration

(b) Style and shape with TPC

into this diffusion conditioning process. This branch is defined as a calibration branch that guides denoising UNet to properly capture visual correspondence between target and reference. To be specific, the TPC provides a calibrated reference image latent c i that aligns with the human shape in the target pose p i based on statistical shape analysis referred to as Procrustes analysis [7]. Figure 4 offers a qualitative understanding of the influence of this c i . Conceptually, in Figure 4 (a), when the humans in reference image x and the target pose sequence p = { p i -1 , p i , p i +1 , p i +2 } are projected onto the ideal 2D shape-style space, they show distinct locations in terms of style axis, where the system aims to generate image animation frame y = { y i -1 , y i , y i +1 , y i +2 } with the style of x and the shape of p . However, in cases ( e . g ., x and p i +2 ) where significant gaps exist along the shape axis ( i . e ., compositional misalignment), the current diffusion model struggles to preserve the original style. This leads to low-fidelity outputs ( e . g ., y i +2 ) and results in temporal inconsistencies due to unstable fidelity across frames. Thus, as illustrated in Figure 4 (b), we bridge this gap in shapes between reference and target by providing correspondence guidance latent c by our designed TPC. With this guidance condition, diffusion-based animation systems achieve robustness to fidelity variations and maintain temporal consistency among frames. The TPC is simple and works in a model-agnostic manner without additional training and validates its effectiveness on human image animation benchmarks ( i . e ., TikTok[14], TED-talks [25]) and even in unseen domain data of real environment scenarios.

2 Related Work

2.1 Diffusion-based Human Image Animation

Human image animation aims to provide a video about an animated version of an input human image. As a human-centric application of image-to-video technology, the human image animation systems [25, 24, 45] have previously been developed based on generative adversarial networks. The recent emergence of denoising diffusion models [27, 26, 10] has presented a new paradigm for generative models, where image animation has also faced fundamental innovations. The diffusionbased framework generates an image animation from noise using pre-trained denoising capabilities based on input human image and target motion video, where the motion video can be extracted

from various pose estimation models [4, 8]. Early work of diffusion-based image animation was made in DreamPose [15], which leveraged the pre-trained text-to-image diffusion model ( e . g ., Stable Diffusion [23]) by conditioning on human image embeddings instead of text, rendering videos of humans in various outfits performing simple walking motions. AnimateAnyone [13] introduces a UNet-style reference image encoder that enhances the layered conditioning of the reference image following the encoding-decoding process of UNet. Furthermore, ControlNet [43] has been a popular choice with the diffusion model by offering more controlled guidance about target motion. To enhance background fidelity, DisCo [31] segments the background of the reference image and integrates it into ControlNet, along with the target motion. For the temporal consistency of output video, MagicAnimate [33] introduces temporal attention by inflating the original 2D UNet to 3D temporal UNet. However, current systems still suffer from quality irregularity issues when the human shapes are not aligned between the reference image and the target motion. Such misalignments frequently occur in real-world scenarios, prompting our proposed Procrustes calibration to address this challenge.

3 Preliminary

3.1 Procrustes Analysis

Procrustes 3 analysis (PA) [6, 7] is a statistical shape analysis technique used to compare the shapes of objects [11, 19]. The PA involves finding the best alignment between two shapes by scaling, translating, and rotating one shape to match the other as closely as possible. To formulate the process of PA, we are given two sets of n points as X = { x 1 , · · · , x n } ∈ R n × d and Y = { y 1 , · · · , y n } ∈ R n × d , where d is the dimension of the point. We perform Procrustes transformation composed of scaling factor s ∈ R 1 , rotation matrix r ∈ R d × d , and translation vector t ∈ R 1 × d as given below:

where the vector t is added with broadcasting to all n points. The objective is to find the optimal transformation parameters s, r, t to minimize the sum of squared differences between ˆ Y and the Y .

where || · || F is Frobenius norm. The optimal value r ∗ is typically obtained via singular value decomposition and the s ∗ and t ∗ are calculated after the optimal rotation r ∗ is found. Here, we apply this PA to align the human shapes of a reference image and target motion frame.

4 Method

Given a human reference image R and a target pose sequence P = [ P 1 , · · · , P L ] , a human image animation system generates image animation video V = [ V 1 , · · · , V L ] which follows the pose sequence by the human in the reference image, where L is the number of frames. Figure 5 shows the application of Test-time Procrustes Calibration (TPC) into the general diffusion-based human image animation system. The TPC aims to improve the quality of resulting animation by consistently ensuring the compositional alignment of human shapes between the reference image and the target poses. At each denoising step t , TPC takes an input reference image R and target poses P , and produces calibrated image C and its embedded latent c for conditioning in diffusion denoising. The calibrated latent c guides the conditioning module ( i . e ., cross-attention) in denoising UNet with precise correspondence about human shapes between the reference and the target poses. To perform this, the TPC follows the sequential process of R → C → c , and it comprises two main modules: (1) Procrustes Warping (Sec 4.1) and (2) Iterative Propagation (Sec 4.2). Using the input reference image R and target poses P , Procrustes warping produces a calibrated reference image C optimized to align the human shape with each target pose. After embedding this calibrated image C into calibrated latent feature c , Iterative Propagation iteratively refines the c during denoising step to enhance temporal consistency among the features by applying our designed feature propagation method. This calibrated latent c is finally given to diffusion model's conditioning module ( i . e ., cross-attention) as a condition by concatenating with the original latent feature r of the reference image.

Test-time Procrustes Calibration

Diffusion model

𝐫, 𝐜

Reference image (

𝑅

)

Target pose (

𝑃

)

Image animation (

𝑉

)

Q

K

V

Conditioning

Attention

Enc

Enc

Enc

Noise

Image encoder

Pose encoder

Calibrated image (

𝐶

)

Iterative Propagation

Propagation

89

-1C

4.1 Procrustes Warping

Procrustes Warping (PW) aims to align human shapes between reference and target pose. Thus, the PW takes a reference image R as input and produces a calibrated reference image C = [ C 1 , · · · , C L ] to the target human pose P = [ P 1 , · · · , P L ] . To construct C , we first extract keypoint sets from both the reference and target humans. Then, we apply Procrustes analysis 4 between the two sets, determining transformation parameters: scaling, rotation, and translation. Using these parameters, we transform the reference image to align it with the target. To be specific, as shown in Figure 6 (a), we define 17

Image

Keypoints

Common (

𝑿

)

Reference

Target

1

3 2

4

5

6

7

8

9

10

11

12

13

14

15

16

17

Numbering

Keypoints

Optimal (

x

∗

)

1

2

3

4

5

6

7

8

10

9

12

13

14

15

1

2

3

4

5

6

7

8

10

9

12

13

1

2

3

4

5

6

7

8

12

1

2

3

4

5

6

7

8

10

9

11

12

13

14

15

1

2

3

4

5

6

7

8

10

9

12

13

1

2

3

4

5

6

7

8

14

15

14

15

13

12

13

(a) 17 keypoints

(b) Keypoints of reference and target, common point set (

X

), and optimal subset (

x

∗

)

keypoints 5 in 2-dimensional space for human body and face using keypoint extractor ( e . g ., OpenPose [4]). Figure 6 (b) illustrates that we obtain these keypoints from the reference image R and i -th target pose frame P i . From these, we filter out commonly visible N points, defining X = { x 1 , · · · , x N } as reference set and Y = { y 1 , · · · , y N } as target set. Following Procrustes analysis ( i . e ., Eq. (1,2)), we obtain optimal transformation parameters s ∗ ∈ R 1 , r ∗ ∈ R 2 × 2 , t ∗ ∈ R 1 × 2 and warp all pixels [ u, v ] ∈ R n × 2 ( n is the number of pixels) in the reference image by mapping as below:

Therefore, we obtain the i -th calibrated image C i = f P i M ( R ) using Procrustes warping f P i M , which aligns R with human shape in P i using mapping M . However, considering all N common visible points for warping is unreasonable since the reference and target shapes cannot perfectly overlap due to differing poses ( e . g ., the arms in Figure 6 (b) cannot overlap). To address this, we apply Procrustes warping to subset x ⊂ X of the common points and select the most effective subset x ∗ based on our defined alignment score function h as below (subset y is also defined corresponding to the x ):

where C x i denotes calibrated image using keypoints of subset x . The h is alignment score function that computes pixel-wise IoU 6 of human shapes between pose P i and calibrated image C x i . Thus, the final i -th calibrated image is defined as C i = C x ∗ i with optimal subset x ∗ .

4.2 Iterative Propagation

Conceptually, as shown in Figure 5, our proposed TPC aims to mitigate compositional misalignment by feeding calibrated image latent features c = [ c 1 , · · · , c L ] into the conditioning module ( i . e ., cross-attention) of denoising diffusion along with the original reference latent feature r . 7 Thus, the c is designed to bridge the correspondence of the human shapes between the reference image and target pose. However, the pose variation within the target pose frames affects the degree of calibration with the reference image, which reduces temporal consistency in the output. To address this, we introduce Iterative Propagation (IP) to enhance consistency among calibrated latent features. Figure 7 illustrates the IP process during the diffusion denoising. The IP forms M groups of sequential features in calibrated features c , randomly selects a feature within each group, and updates all features in the group with the selected one. This method enhances temporal consistency among calibrated latent features while maintaining compositional alignment with the target pose due to the continuity of target pose variation. The random selection of features ensures all features have an equal chance of being chosen during the denoising process.

4.3 Plug-and-Play Test-time Procrustes Calibration

We integrate calibrated latent feature c into human image animation systems by applying it into a conditioning module ( i . e ., cross-attention) of video diffusion UNet. For i -th frame m patch-wise image feature p i ∈ R m × d , reference feature r ∈ R m × d , and calibrated latent feature c i ∈ R m × d the cross attention is formulated as Q ← Softmax ( QK T /d ) V , where it satisfies Q = p i and K = V = [ r , c i ] ∈ R 2 m × d is concatenated latent condition.

5 Experiments

5.1 Experimental Settings

Implementation Details. SAM [16] is used for screening out the background in calibrated images. VQ-VAE [30] is used for encoding images of the video. The number of groups in iterative propagation is chosen as M = 30 under ablation studies in Table 2. The average number of video frames is about 120. We use Stable Diffusion 1.5 [23] for all baselines on 4 NVIDIA A100 GPUs. We follow the same pose encoders and image encoders of baseline models.

Data and Baselines. We validate human image animation on two popular benchmarks ( i . e ., TikTok [14], TED-talks [25]) about a test split. Due to no validation splits, we provide valid sets matching the test set sizes for the ablation study. We further collected 114 samples 8 from TikTok and TED-talks as another test split. These samples contain compositional misalignment about rotation and scaling between human shapes in reference images and motion videos. The criteria for misalignment include relative angle and scaling differences, with samples having a relative angle θ > π/ 6 or relative scale IoU < 0.7. As shown in Figure 1 (a), the relative angle measures an angle between the straight lines from the pelvis to the nose in humans of reference and target poses using keypoints estimator [4]. The relative scaling measures the IoU (Intersection of Union) between bounding boxes of humans in the reference and the target. Procrustes Calibration is validated on recent diffusion-based human image animation models including MagicAnimate [33], DisCo [31], AnimateAnyone 9 [13], DreamPose [15] on their public codes and papers.

Denoising step

Propagation

𝐜𝐿

𝐜7

𝐜8

𝐜6

𝐜1

𝐜2

𝐜3

𝐜4

𝐜5

900

Group 1

Group 2

Group

𝑴

𝐜𝐿-1

𝐜𝐿

𝐜7

𝐜8

𝐜6

𝐜1

𝐜2

𝐜3

𝐜4

𝐜5

600

Group 1

Group 2

Group

𝑴

𝐜𝐿-1

𝐜𝐿

𝐜7

𝐜8

𝐜6

𝐜1

𝐜2

𝐜3

𝐜4

𝐜5

300

Group 1

Group 2

Group

𝑴

𝐜𝐿-1

5.2 Evaluation Metrics

We evaluate videos in terms of single-frame quality and video quality. For the single-frame, we measure Peak Signal-to-Noise Ratio (PSNR) [12], Structural Similarity Index Measure (SSIM) [32], Learned Perceptual Image Patch Similarity (LPIPS) [44], FID (Fréchet Image Distance) [9], and L1 error between output and ground-truth images. For the video quality, we measure Fréchet Video Distance (FVD) [29] and FID-VID [1]. All automatic metrics are averaged over 10 runs with different seeds. We also analyze human preferences for results from the baselines with and without our TPC.

5.3 Experimental Results

Quantitative Comparisons. Table 1 presents evaluations of image animation on two benchmark datasets ( i . e ., TikTok, TED-talks) using recent diffusion image animation baselines (MagicAnimate, DisCo, AnimateAnyone, DreamPose) with TPC across three assessments ( i . e ., image, video, human). Evaluations are conducted on two test splits: the original set and the compositional misalignment set. Initially, all baselines use a single reference image latent for conditioning all poses. After integrating TPC, they use calibrated latents corresponding to each pose. Consistent improvements in image and video quality are observed in both test sets. All baselines struggle with the compositional misalignment set, but when integrated with TPC, they achieve quality close to the original test set. This demonstrates that morphological similarity affects the current diffusion model's ability to map human shapes from reference to target.

Qualitative Comparisons. To validate our proposed TPC, we applied it to four recent baselines and compared the original results. We used the same types of human pose inputs (e.g., OpenPose, DensePose) for each baseline to prepare target motion videos. Figure 8 shows predictions on four different samples exhibiting compositional misalignment. The top left results show predictions on temporal misalignment between the reference and target due to the target's bending motion. MagicAnimate's fidelity diminishes with increased bending (red box in the last frame), whereas the model with TPC maintains high fidelity. The top right results display predictions on temporal misalignment due to a walking motion towards the front, with TPC similarly enhancing DisCo's performance. The bottom left results show consistent misalignment across all frames. DreamPose struggles with low fidelity, causing unwanted stripes on the pants (red box), which are clearly removed with TPC. The results in the bottom right exhibit consistent misalignment due to a scale difference. In AnimateAnyone, the reference human's yellow pants were incorrectly mapped onto the target human's arms, making them appear yellow. However, the TPC completely mitigates this incorrect mapping. This occurs because the calibrated image filters out the pants and enhances the correspondence of each body part. (Please, refer to the calibrated images also in the Appendix.)

DisCo

GT

Reference

image

Target

motion

AA

AA + TPC

GT

Reference

image

Target

motion

MA

MA + TPC

GT

Reference

image

Target

motion

DP

DP + TPC

GT

Reference

image

Target

motion

人大

Figure 10 shows the results on reference images from an unseen domain, generated by the T2I model [21], applied in a compositional misalignment scenario. The left displays a temporal misalignment sample due to bending motions, with improved fidelity in both baselines, especially in DisCo. The right side displays a consistent misalignment sample, where the baselines show low precision in transferring identity when the reference is flipped upside down. However, with the integration of TPC, their performance is significantly improved.

(

𝜽

)

-

𝝅

0

𝝅/𝟐

-

𝝅/𝟐

(PSNR)

30

26

30

20

(PSNR)

0.6

0.3

(

IoU

)

Baselines

Baselines + TPC

Baselines

Baselines + TPC

PSNR according to rotating

between target and reference

PSNR according to scaling

between target and reference

Robustness analysis To assess the robustness of baselines with TPC, we measured fidelity (PSNR) by varying the scale and rotation of the human reference, as shown in Figure 9. In the case of scaling, the baselines show a drop in performance when the size difference between the reference and target human shapes falls below an IoU of 0.6. However, TPC maintains optimal performance even down to an IoU of 0.4. Notably, TPC enhances baseline robustness to all rotational variations by ensuring the calibrated image aligns well with shape differences caused by rotation.

DP

DP + TPC

Reference

image

Target

motion

DisCo

Reference

image

Target

motion

MA

MA + TPC

AA + TPC

AA

Table 3: Inference time of baselines with TPC. DC: DisCo.

Procrustes

Ablation Study. Table 2 presents ablation studies on transformation methods and iterative propagation. Transformations are categorized as shape-preserving ( e . g ., Linear, Procrustes) and shapedistorting ( e . g ., Affine). Linear transformation performs scaling and rotation without translation, based on the object's bounding box coordinates. The Procrustes method is the most effective, transforming by selecting the optimal keypoint. The affine transform is less effective due to shearing, which can cause distortion and information loss by moving the subject out of the frame, as shown in Figure 11. The second section presents an ablation study on iterative propagation, showing significant enhancements in foreground and background, with M =30 being the most effective. Table 3 provides inference time on baselines with TPC. Our method requires only minimal additional time because it processes all frames batch-wise, generating the calibrated image in a single iteration.

5.4 Broader Impacts and Ethic Statements

Visual generative models present a range of ethical dilemmas, including the creation of unauthorized counterfeit content, potential privacy breaches, and challenges related to fairness. Due to our reliance on the architecture of these models, our work inherently adopts these ethical vulnerabilities. Addressing these concerns is imperative and requires the establishment of comprehensive regulations and technical countermeasures. We are exploring advanced measures such as learning-based digital forensics and digital watermarking to ethically navigate the complexities of visual generative models.

5.5 Limitation and Future work

We provide an overview of the various limitations and potential development directions identified during this study. Human image animation systems transfer reference images to target poses. However, some frames in a target pose video may lack proper specification, leading to flicker or low fidelity in model predictions. Thus, integrating high-quality pose estimation is crucial. Another limitation is that significant differences in body shape between the reference and target poses result in awkward transfers, such as transferring a skinny person to a fat target pose. To achieve natural results, a system or module that aligns body shapes is required. For the future extension of human image animation system, it is essential to ensure robust operations under multiple individuals. The system should ensure consistent identity transfer across multiple individuals and accurately adapt to those appearing at specific times. To achieve this, video technologies about recognition [22, 46] and perception [37, 40] can be further incorporated. Currently, conditioning relies solely on images, but it is anticipated that various other modalities, such as text and audio, could also be incorporated. Especially, the integration of the text modality with emerging large language models [3, 28] is expected to drive innovative industrial advancements, with the introduction of video/image-based conversational systems [36, 42, 38] serving as a compelling utility. Lastly, integrating technologies focused on speed [18, 2], resource efficiency [34], and test-time calibration [35] is expected to improve human image animation systems' applicability into real-world environments.

6 Conclusion

Current diffusion-based human image animation systems are facing challenges on samples of compositional misalignment of human shapes between a reference image and target pose frames. To this end, this paper presents Test-time Procrustes Calibration (TPC) which improves the robustness of image animation models on the compositional misalignment samples in a model-agnostic manner. Extensive experiments demonstrate the effectiveness of TPC.

References

Yogesh Balaji, Martin Renqiang Min, Bing Bai, Rama Chellappa, and Hans Peter Graf. Conditional gan with discriminative filter generation for text-to-video synthesis. In IJCAI , volume 1, page 2, 2019.

Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461 , 2022.

Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 , 2020.

Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7291-7299, 2017.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems , 34:8780-8794, 2021.

John C Gower. Some distance properties of latent root and vector methods used in multivariate analysis. Biometrika , 53(3-4):325-338, 1966.

John C Gower. Generalized procrustes analysis. Psychometrika , 40:33-51, 1975.

Rıza Alp Güler, Natalia Neverova, and Iasonas Kokkinos. Densepose: Dense human pose estimation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7297-7306, 2018.

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems , 30, 2017.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems , 33:6840-6851, 2020.

Ji Woo Hong, Sunjae Yoon, Junyeong Kim, and Chang D Yoo. Joint path alignment framework for 3d human pose and shape estimation from video. IEEE Access , 11:43267-43275, 2023.

Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition , pages 2366-2369. IEEE, 2010.

Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. arXiv preprint arXiv:2311.17117 , 2023.

Yasamin Jafarian and Hyun Soo Park. Learning high fidelity depths of dressed humans by watching social media dance videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12753-12762, 2021.

Johanna Karras, Aleksander Holynski, Ting-Chun Wang, and Ira Kemelmacher-Shlizerman. Dreampose: Fashion image-to-video synthesis via stable diffusion. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 22623-22633. IEEE, 2023.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4015-4026, 2023.

Gwanhyeong Koo, Sunjae Yoon, Ji Woo Hong, and Chang D Yoo. Flexiedit: Frequency-aware latent refinement for enhanced non-rigid editing. arXiv preprint arXiv:2407.17850 , 2024.

Gwanhyeong Koo, Sunjae Yoon, and Chang D Yoo. Wavelet-guided acceleration of text inversion in diffusion-based image editing. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 4380-4384. IEEE, 2024.

Julieta Martinez, Rayat Hossain, Javier Romero, and James J Little. A simple yet effective baseline for 3d human pose estimation. In Proceedings of the IEEE international conference on computer vision , pages 2640-2649, 2017.

Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov. Grad-tts: A diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning , pages 8599-8608. PMLR, 2021.

Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning , pages 8821-8831. Pmlr, 2021.

Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 , 2024.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in neural information processing systems , 32, 2019.

Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion representations for articulated animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13653-13662, 2021.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 , 2023.

Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: A new metric & challenges. arXiv preprint arXiv:1812.01717 , 2018.

Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems , 30, 2017.

Tan Wang, Linjie Li, Kevin Lin, Yuanhao Zhai, Chung-Ching Lin, Zhengyuan Yang, Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Disco: Disentangled control for realistic human dance generation. arXiv preprint arXiv:2307.00040 , 2023.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4):600-612, 2004.

Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Jia-Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. arXiv preprint arXiv:2311.16498 , 2023.

Xingyi Yang, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Diffusion probabilistic model made slim. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition , pages 22552-22562, 2023.

Hee Suk Yoon, Joshua Tian Jin Tee, Eunseop Yoon, Sunjae Yoon, Gwangsu Kim, Yingzhen Li, and Chang D Yoo. Esd: Expected squared difference as a tuning-free trainable calibration measure. arXiv preprint arXiv:2303.02472 , 2023.

Hee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Kang Zhang, Yu-Jung Heo, Du-Seong Chang, and Chang D Yoo. Bi-mdrg: Bridging image history in multimodal dialogue response generation. arXiv preprint arXiv:2408.05926 , 2024.

Sunjae Yoon, Ji Woo Hong, Eunseop Yoon, Dahyun Kim, Junyeong Kim, Hee Suk Yoon, and Chang D Yoo. Selective query-guided debiasing for video corpus moment retrieval. In European Conference on Computer Vision , pages 185-200. Springer, 2022.

Sunjae Yoon, Dahyun Kim, Eunseop Yoon, Hee Yoon, Junyeong Kim, and Chang Yoo. Hear: Hearing enhanced audio response for video-grounded dialogue. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 11911-11924, 2023.

Sunjae Yoon, Gwanhyeong Koo, Ji Woo Hong, and Chang D Yoo. Dni: Dilutional noise initialization for diffusion video editing. arXiv preprint arXiv:2409.13037 , 2024.

Sunjae Yoon, Gwanhyeong Koo, Dahyun Kim, and Chang D Yoo. Scanet: Scene complexity aware network for weakly-supervised video moment retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 13576-13586, 2023.

Sunjae Yoon, Gwanhyeong Koo, Geonwoo Kim, and Chang D Yoo. Frag: Frequency adapting group for diffusion video editing. arXiv preprint arXiv:2406.06044 , 2024.

Sunjae Yoon, Eunseop Yoon, Hee Suk Yoon, Junyeong Kim, and Chang Yoo. Information-theoretic text hallucination reduction for video-grounded dialogue. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 4182-4193, 2022.

Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3836-3847, 2023.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3657-3666, 2022.

Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, and Yichen Wei. Flow-guided feature aggregation for video object detection. In Proceedings of the IEEE international conference on computer vision , pages 408-417, 2017.

A Appendix

A.1 Algorithm for Iterative Propagtion

A.2 More qualitative results

Explanation

Figure 12 shows calibrated reference images corresponding to the samples in Figure 8.

Figure 13 shows results on identical motion with different reference images.

Figure 14 and 15 show results on identical references with different motion videos in the TikTok dataset.

Figure 16 shows results on multiple humans in reference and target motion.

As in Figure 1, here we visualize the target pose in a format of DensePose for the visibility of target motions. The video samples in Figure 8 are publicly available from:

https://www.pexels.com/video/a-woman-stretching-5510095/

https://www.pexels.com/video/modeling-wedding-dresses-7305163/

https://www.youtube.com/watch?v=8S0FDjFBj8o

https://www.tiktok.com/@dbstjswo505/video/7371773469899476231

Reference

image

Target

motion

Reference

image

Target

motion

Reference

image

Target

motion

Reference

image

Target

motion

人大

DP

DP + TPC

Reference

image

Target

motion

DisCo

AA

AA + TPC

MA

MA + TPC

AA

AA + TPC

Reference

image

Target

motion

AA

AA + TPC

Reference

image

Target

motion

AA

AA + TPC

Reference

image

Target

motion

MA

MA + TPC

Reference

image

Target

motion

MA

MA + TPC

Reference

image

Target

motion

MA

MA + TPC

Reference

image

Target

motion

MA

MA + TPC

Reference

image

Target

motion

MA

MA + TPC

Reference

image

Target

motion

00

iii

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Yes, in the line 15-17 in the Abstract and last paragraph in the Introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: It is provided as a Limitation section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers DisCover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Method section contains the details of our proposed framework.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We submit the source links for data used in our experiment and also a demo video.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: It is explained in the Implementation Details section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: It is explained in the evaluation metrics.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide information about GPU resources in the Implementation Details section for implementing our framework.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We read the NeurIPS Code of Ethics and follow all of it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We provide a broader impact including societal negative impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [Yes]

Justification: The broader impact contains information about safeguards.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We explicitly provide all the references we used including url links.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes diffusion guidance (Test-time Procrustes Calibration, TPC) for human image animation. TPC incorporates an auxiliary branch into the diffusion conditions process and provides a calibrated reference image latent. Experimental results show the effectiveness of the provided method.\n\nSTRENGTHS:\n1.\tThis paper studies an interesting problem of human image animation, that is, when the given human image and pose image have shape differences, the performance of the original model is poor.\n2.\tThe structure of this paper is well-organized and easy to follow.\n3.\tThe experimental results show the effectiveness of the proposed method.\n\nWEAKNESSES:\n1.\tIn Figure 1a, I acknowledge that there may be a large shape difference between a given image that needs to be animated and a guided image. In this case, I think the most direct and useful way is to find a way to align the two images, and I don't think it is very difficult to align the two images. So why do we need the proposed TPC method?\n2.\tIn line 54, the method of injecting guided images into the model should be more than just cross-attention. Would TPC work if the human image animation methods do not use cross-attention to guide image injection into the model?\n3.\tThe last paragraph of the introduction introduces the proposed TPC approach. However, this part is to analyze the gains that TPC can bring, and I think this part is more suitable for methods or experiments.\n4.\tAs for the TED man in the demo of the experimental results, why does the animated video only show the man's upper body?\n5.\tFor the unseen domain shown in the provided demo, I think in human image animation, we'll give the model images that need to be animated and guided images. Also, the base model is trained on a lot of images of human bodies, and the model should have seen something like this before, so calling it an unseen domain isn't accurate. For this kind of scene, there is a big gap between the character identity in the generated video and the given image, and TPC seems to have no way to help the original model maintain the character identity.\n\nQUESTIONS:\nPlease see the above.\n\nLIMITATIONS:\n1. The necessity of TPC\n2. The experimental results do not help the original model retain the details of the input image.\n3. Introduction needs to be reorganized", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose TPC, an alignment algorithm for human image animation systems. optimal precision is currently achieved only when the physical compositions of the human shapes in the reference image and target pose frame are aligned. Misalignment leads to a significant decline in fidelity and consistency. To address this issue, the authors propose Test-time Procrustes Calibration (TPC). TPC provides a calibrated reference image for the diffusion model, enhancing its ability to understand the correspondence between human shapes in the reference and target images. This technique can be applied to any diffusion-based image animation system in a model-agnostic manner, improving effectiveness at test time without requiring additional training.\n\nSTRENGTHS:\n(1) TPC bridges the gap in shapes between reference and target by providing correspondence guidance latent. With this guidance condition, diffusion-based animation systems achieve robustness to fidelity variations and maintain temporal consistency among frames.\n\n(2) The TPC is simple and works in a model-agnostic manner without additional training.\n\nWEAKNESSES:\n(1)\tThe implementation details of Iterative Propagation are not relatively comprehensive, and I suggest the authors make an algorithm table to depict the concrete process of the proposed Iterative Propagation.\n\n(2)\tThere are some skeleton alignment algorithms in the previous works, such as Everybody Dance Now. The authors should conduct additional quantitative and qualitative comparison experiments between TPC and other previous skeleton alignment algorithms. \n\n(3)\tAccording to my observations of your provided demos, the backgrounds of some rotation cases encounter considerable changes during the animation process. The background details don’t align with the given reference image, such as the layout and color distribution. For instance, in the galaxy case, the positions of the stars in the background of the animated video are not the same as those in the original given reference image.\n\nQUESTIONS:\n(1)\tIn some particular situations, the protagonist in the given reference image only shows half of the whole body, while the driven target pose shows the whole body parts. Can TPC handle this sort of situation? Could you please give some quantitative and qualitative experiment results regarding this situation? \n\n(2)\tI am unsure how you can estimate or ensure the accuracy of the subset X. Could you please provide some visual comparison results in terms of different subsets with different numbers of key points? Additionally, could you provide the details of the common filtering process?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper starts from an interesting problem that what happens when the motion condition and the reference image are not well aligned. It analyses the robustness of an existing human animation network given different levels of misalignment, and tries to find the underlying cause through attention maps. Then this paper propose TPC, a test-time calibration module to mitigate this issue. This module provides a calibrated latent feature as an additional condition to the diffuser. The calibrated latent is optimized via Procrustes Warping, and Iterative Propagation is proposed to improve temporal consistency among calibrated features.\n\nSTRENGTHS:\nThis paper is among the most innovative work I've ever seen in human animation area. It is a curiosity-driven work starting with an interesting problem. The solution is neat and elegant.\n\nThe presentation is clear and easy to follow. The analysis in Fig 1 and Fig 2 is very interesting and insightful.\n\nTechnically, using keypoints as the evidence to align shapes is quite direct and reasonable. Using Procrustes analysis to optimize a transformation is sound.\n\nWEAKNESSES:\nI did not find obvious weakness, except that the demo results still contain some artifacts. But it is mainly due to the baseline MA or Disco. I recommend the authors to try some new methods as the baseline that have higher quality.\n\nThe method is insightful, but only for human animation, a small area. It would greatly enhance the value of this paper if it can be applied to some more general tasks.\n\nQUESTIONS:\nWhat if directly calculating the transformation between the reference image and the target pose via some heuristical evidences? For example, considering the torso part, i.e., keypoints set {6,7,13,12}, since torso keeps rigid to some extend, compared to arms and legs. Then using the vector (mid(6,7) , mid(13, 12)) to compute the rotation angles, the perimeter for scales, the center coordinate for shifting. I know this design is not optimal. But how is the result with such simple idea? I noticed Table 2, the \"Linear\" and \"Affine\" setting, but seems not exactly the same one.\n\nCan this method be potentially applied to a broader tasks that require spatial alignment between some input variables?\n\nLIMITATIONS:\nThe Broader Impacts and Ethic Statements have been adequatedly addressed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

comments: SUMMARY:
This work proposes a method that combines exsiting human image animation method and  Procrustes Warping, which improves the robustness of image animation approach. In addition to the explicit image warping, this work also proposes an interative propagation method to improve the temporal consistency. Experiments show that the proposed method is effective in enhancing the animation performance.

STRENGTHS:
1. The proposed method is well motivated, the observation of the misalignment between reference image and target pose is straightforward and insightful.
2. The experiments in this work are solid, both the comparisons and ablations are extensive and detailed. Compared with the baseline methods, the improvements are noticeable.
3. The writing is clear and easy to follow.

WEAKNESSES:
1. As far as I know, the Procrustes alignment is popular in the evaluation of 2D/3D human pose estimation. So, this work introduces this simple technique into the human image animation but technically this contribution is not novel enough.
2. There is no detailed description about the training process of this method, it could be better to introduce more about the training process in the main text.
3. From the Table 2 we can see that the best M is 20, but the evaluation metrics shown in the table are SSIM and FVD, it could better to show more results to help reader understand the ablation comprehensively. 
4. Are there any trade-off boserved between the temporal smoothness and motion precision? For example, if M=1, the temporal consistency is the best while the pose precision is the worst?

QUESTIONS:
How do you apply TPC in DisCo and MagicAnimate? I am a little bit confused since the Figure 5 only depicts the structure of Anymate Anyone.

LIMITATIONS:
Discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12