TPC: Test-time Procrustes Calibration for Diffusion-based Human Image Animation
Authors: Sunjae Yoon
Gwanhyeong Koo
Younghwan Lee
Chang D. Yoo
Venue: NeurIPS 2024 poster (2024)
Keywords: Human Image Animation
Diffusion model
Procrustes Analysis
Primary area: diffusion_based_models
TL;DR: This paper proposes Test-time Procrustes Calibration, which enhances the robustness of diffusion-based image animation systems in the real-world unseen domain by guiding correspondences of human shapes between reference image and target poses.

Abstract: Human image animation aims to generate a human motion video from the inputs of a reference human image and a target motion video. Current diffusion-based image animation systems exhibit high precision in transferring human identity into targeted motion, yet they still exhibit irregular quality in their outputs. Their optimal precision is achieved only when the physical compositions (i.e., scale and rotation) of the human shapes in the reference image and target pose frame are aligned. In the absence of such alignment, there is a noticeable decline in fidelity and consistency. Especially, in real-world environments, this compositional misalignment commonly occurs, posing significant challenges to the practical usage of current systems. To this end, we propose Test-time Procrustes Calibration (TPC), which enhances the robustness of diffusion-based image animation systems by maintaining optimal performance even when faced with compositional misalignment, effectively addressing real-world scenarios. The TPC provides a calibrated reference image for the diffusion model, enhancing its capability to understand the correspondence between human shapes in the reference and target images. Our method is simple and can be applied to any diffusion-based image animation system in a model-agnostic manner, improving the effectiveness at test time without additional training.

TPC: Test-time Procrustes Calibration for Diffusion-based Human Image Animation
Sunjae Yoon Gwanhyeong Koo Younghwan Lee Chang D. Yoo"
Korea Advanced Institute of Science and Technology (KAIST) {sunjae. yoon, yo yoa.ssooooais.a.......
Abstract
Human image animation aims to generate human motion video from the inputs reference human image and a target motion video. Current diffusion-basee image animation systems exhibit high precision in transferring human identity into targeted motion, yet they still exhibit irregular quality their outputs. Their optimal precision is achieved only when the physical compositions (i.e., scale and rotation) of the human shapes in the reference image and target pose frame are aligned. In the absence of such alignment, there noticeable decline in fidelity and consistency. Especially, in real-world environments, this compositional misalignment commonly occurs, posing significant challenges to the practical usage of current systems. To this end, we propose Test-time Procrustes Calibra- tion (TPC), which enhances the robustness of diffusion-basse image animation systems by maintaining optimal performance even when faced with compositional misalignment, effectively addressing real-world scenarios. The TPC provides a calibrated reference image for the diffusion model, enhancing its capability to understand the correspondence between human shapes the reference and target images. Our method is simple and can be applied to any diffusion- image animation system model-agnostic manner, improving the effectiveness at test time without additional training.
Introduction
Denoising diffusion models [5] 27 26 [0 have transformed the generative landscape of artificial
intelligence, leading to groundbreaking achievements [17 20 39 in image, speech, and video generation. We explore the application of diffusion model in the specific context of image-to-vides generation, focusing on the task of human image animation. The technology of image animation holds great promise, enabling immersive and interactive experiences in entertainment, virtual reality, and digital communication. The human image animation systems 33
are designed to work with referential human image and target motion video, where they transfer the human identity into the target motion, ensuring seamless and unobtrusive integration. This process requires understanding the correspondence between human shapes in the reference image and the target pose frame.
31
[5| 13
Recent advancements 33 3ll of human image animation systems have demonstrated notable preci- sion in adapting human identity into target motion. Despite advancements, these systems continue to suffer from irregular quality of image animation when the compositions (i.e., scale and rotation) of human shapes are not aligned between reference image and target motion. To be specific, Figure [] (a) presents exploratory experiments on the compositional misalignment of human shapes between the reference and target. As shown in the left experiments, for given target pose, adjusting the composition of the same human in the reference image (i.e., via scaling or rotating) results in in- consistent and low-fidelity output images, especially in terms of clothes and faces of the human.
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Original
Scaling
Scaling
Rotating
Rotating
[Bending motion)
(Walking motion)
NA 22
DeCn
Chrrs
(a) Predictions on samples of compositional misa sigamettmentt human shapes betw een target and reference:
misalignment by reference image variation (left) and misalignment by target pose variation (right)
according scating target
according to rolative anglo between target and reference
Robust
Robust
Nine
Sansition
Sensttive
Retative
(loU)
Target
Referonce
angle
(b) analysás according scaling and rotating (right)
02
64
0/2
Figure 1: Illustration of compositional misalignment: (a) Results current human image animation models [33] (3] on samples in compositional misalignment of human shapes between reference and target. (b) Sensitivity analysis according to variation of compositional misalignment by scaling and rotating (MA: MagicAnimate). Best viewed with zoom.
Furthermore, in the right experiments, when providing motion sequences that display various dynamic movements for given reference image, the human image animation outputs consistently show low fidelity, especially evident target poses (e.g., bending or approaching close viewpoint) that cause significant differences in the composition of the human shape. To quantitatively investigate these, Figure (b) presents sensitivity analysis evaluating how current image animation systems [3 respond to varying degrees of the compositional misalignment of human shape. The left shows the fidelity (i.e., PSNR) of the resulting human according to the relative scale difference of input human shapes between the target and reference. We employ the Intersection of Union (IoU) of bounding boxes of the shapes for relative scale, The right shows fidelity according to the variations of relative angle 0 human shapes between the target and reference. Current systems demonstrate significant vulnerability to variations the compositions, indicating that output fidelity forms robust region only in the areas with compositionally aligned conditions (i.e.,
6
x IoU 0.7).
In fact, diffusion-baast systems are inevitably susceptible to this compositional misalignment. The diffusion model uses a reference human image condition to generate controlled out- put from noise based on the target pose. Here, the conditioning is performed through cross- attention based on a visual similarity between patch-wise features of the target pose frame and reference image. To be specific, Figure shows the attention map about single patch of the tar- get frame (i.e., shoulder denoted by blue point) from the reference image during the denoising process. The upper section displays a sample where the human shapes relatively aligned between the target and the reference, while the lower section shows case where this alignment not present. Initially, the attention maps were blurry in both cases. However, as denoising con- tinued, it became clear that samples with aligned human shapes correctly established a correspondence between the target frame and the reference image. (i.e., Attention the shoulder on the target frame is focused on the shoulder on the reference image.) However, when the shape is misaligned, attention the target frame's 's shoulder incorrectly focused on unrelated areas in reference, even up to the
Target pose
Attention maps on reference image
961 Denoising step 79
high
Low
Danoising stop
Figure 2: Attention maps the reference im- age corresponding to the target human shape (e.g., shoulder blue point) according to denoising.
We define the axis of the torso of a human from the pelvis to nose using estimated human body key points 4 and measure the relative angle of axes between target and reference.

---

Calibration branch
TPC
L Reference
Enc
Latend feature
Encl 2, C2
(onniiiionig)
(Conditioning)
pi
0
Yi (i-thfrrame)
Pi
|00 UNet
yi
Target pass framme)
UNet Diffusion model
Diffusion model
(b) Diffusion- human image animation with Test-time Procrustes Calibration (TPC)
(a) Diffusion-based human image animation
Figure 3: Illustration of (a) current diffusion- human image animation systems and (b) Test-time Procrustes Calibration (TPC) on top of these systems. The TPC can be applied to diffusion-based models in a model-agnostic manner, enhancing the fidelity and consistency of the output video.
last stages of denoising. Consequently, the compositional misalignment between the target and the reference image hinders accurate attention correspondence throughout the denoising process.
To this end, propose diffusion guidance referred to as Test-time Procrustes Calibration (TPC). As shown in Figure 3 (a). the existing basfe human image animation system takes inputs of reference image and i-th target pose frame Pt. where it generates i-th output animation frame y/. Here, the x is encoded into the latent feature 2, which serves as condi- tioning input for the denoising diffusion model (ie., UNet). As depicted in Figure 3 (b), our proposed TPC incorporates an auxiliary branch into this diffusion conditioning process. This branch is defined as calibration branch that guides denoising UNet to properly capture visual correspondence between target and reference. To be specific, the TPC provides calibrated reference image latent c; that aligns with the human shape in the target pose p. based on statistical shape analysis referred to as Procrustes analysis 77 Figure offers qualitative understanding of the influence of this C, Conceptually, in Figure 4 (a), when the humans in reference image x and the target pose sequence onto the ideal 2D shape-style space, they show distinet locations terms of style axis, where the system aims to generate image animation frame y = {yi-1, yr; V,+1, W with the style of a and the shape of p. However, in (e.g., g-. and p:+2) where significant gaps exist along the shape axis (i.e., compositional misalignment), the current diffusion model struggles to preserve the original style. This leads to low-fidelity outputs (e.g., 2/++2) and results in temporal inconsistencies due unstable fidelity across frames. Thus, as illustrated in Figure (b). we bridge this gap in shapes between reference and target by providing correspondence guidance latent by our designed TPC. With this guidance condition, diffusion-baan based animation systems achieve robustness to fidelity variations and maintain temporal consistency among frames. The TPC is simple and works without additional training and validates its effectiveness on human image animation benchmarks (i.e., TikTok/[4) TED-talks
Shape variation
Shape veriation
Raference
Target Dugpae
Collbratio
TPC
Style vareevaaaaa varialion
Cr
Fidelity
34-11
Mi-11
V6+22
Styleard shape wilhodd TPC
(b) Slyleaad shapp with TPC
Figure 4: Conceptual illustration of the effective- ness of TPC in terms of style and shape variation.
{Pi-1- Ps. Pi+1, Pi+-2/ are projected
model-agnosee manner
255 and even in unseen domain data of real environment scenarios.
2 Related Work
2.1 Diffusion-based Human Image Animation
Human image animation aims to provide video about an animated version of an input human image. As human- centric application of image-to-vided technology. the human image animation have previously been developed based on generative adversarial networks. The 26 10 has presented a new paradigm for generative models, where image animation has also faced fundamental innovations. The diffusion- based framework generates an image animation from noise using pre-trained denoising capabilities based input human image and target motion video, where the motion video can be extracted
systems 25 24 45
recent emergence of denoising diffusion models 22
3

---

from various pose estimation models [4 [8. Early work of liffusion-based image animation was made in DreamPose [TS], which leveraged the pre-trained text-to-image diffusion model (e.g., Stable Diffusion (23 by conditioning on human image embeddings instead of text, rendering videos of humans in various outfits performing simple walking motions. Animate [3] introduces a UNet-style reference image encoder that enhances the layered conditioning of the reference image following the incoding-decoding process of UNet. Furthermore, ControlNet 44 has been a popular choice with the diffusion model by offering more controlled guidance about target motion. To enhance background fidelity, DisCo [3] segments the background of the reference image and integrates it into ControlNet, along with the target motion. For the temporal consistency of output video, MagicAnimate [33] introduces temporal attention by inflating the original 2D UNet to 3D temporal UNet. However, current systems still suffer from quality irregularity issues when the human shapes are not aligned between the reference image and the target motion. Such misalignments frequently occur in real-world scenarios, prompting our proposed Procrustes calibration to address this challenge.
3 Preliminary
3.1 Procrustes Analysis
Procrustes Jnnalyis (PA) 6 is statistical shape analysis technique used to compare the shapes of objects 11 19]. The PA involves finding the best alignment between two shapes by scaling, translating, and rotating one shape to match the other as closely as possible. To formulate the process of PA, we are given two sets of n points as X = {21,-- ... Inn E Rxd and Y = {V1, yo E Raxd, where d is the dimension of the point. We perform Procrustes transformation composed of scaling factor E R,, rotation matrix E Rdxd, and translation vector t R xd as given below:
y = Xr+t.
(1)
where the vector t is added with broadcasting to all n points. The objective is to find the optimal transformation parameters r, to minimize the sum of squared differences between Y and the Y.
argmin||y - YYIIF, 8,r,t
(2)
where || ||I is Frobenius norm. The optimal value r- is typically obtained via singular value decomposition and the s* and l" are calculated after the optimal rotation r* is found. Here, we apply this PA to align the human shapes of a reference image and target motion frame.
4 Method
Given a human reference image R and a target pose sequence P P [P1.... .P). a human image animation system generates image animation video V [Vi...., Vz] which follows the pose sequence by the human in the reference image, where L is the number of frames. Figure 5 shows the application of Test-time Procrustes Calibration (TPC) into the general diffusion-based human image animation system. The TPC aims to improve the quality of resulting animation by consistently ensuring the compositional alignment of human shapes between the reference image and the target poses. At each denoising step f, TPC takes an input reference image R and target poses P, and produces calibrated image C' and its embedded latent с for conditioning in diffusion denoising. The calibrated latent guides the conditioning module (i.e., cross-attention) in denoising UNet with precise correspondence about human shapes between the reference and the target poses. To perform this, the TPC follows the sequential process of R + C + c, and it comprises two main modules: (1) Procrustes Warping (Sec and (2) Iterative Propagation (Sec 4.2 Using the input reference image R and target poses P. Procrustes warping produces calibrated reference image C' optimized to align the human shape with each target pose. After embedding this calibrated image C into calibrated latent feature c, Iterative Propagation iteratively refines the during denoising step to enhance temporal consistency among the features by applying our designed feature propagation method. This calibrated latent is finally given diffusion model's conditioning module (i.e., cross-attention) as condition by concatenating with the original latent feature r of the reference image.
'The name "Procrustes" comes from Greek mythology, where Procrustes was a bandit who would stretch or cut people to fit his bed, reflecting the idea of adjusting data to fit common form.

---

Test-time Procrustes Calibration
Procrustes Warping
Iterative Propagation
C (R)
Calibrated image (C)
r,
C
Image encoder
Conditioning
Reference image (R)
Noise
Q
K
|00 Diffusion model
Enc
Pose encoder
Target pose (P)
Image animation (V)
Figure 5: Illustration of Test-time Procrustes Calibration (TPC) on diffusion-basee human image animation. TPC provides calibrated latent feature to enhance shape correspondence between the reference image and target poses. Procrustes Warping aligns the reference image with the target pose shape, while Iterative Propagation improves temporal consistency among calibrated features.
4.1 Procrustes Warping
Procrustes Warping (PW) aims to align hu- man shapes between reference and target pose. Thus, the PW takes reference image R as in- put and produces calibrated reference image C' = Cc. C1] to the target human pose P P [PI. Pr... To construct C, we first ex- tract keypoint sets from both the reference and target humans. Then, we apply Procrustes analy- sis between the two sets, determining transfor- mation parameters: scaling, rotation, and trans- lation. Using these parameters, we transform the reference image to align with the target. To be specific, shown in Figure (a), define 17 keypoints in 2-dimensional space for human body and face using keypoint extractor (e.g., OpenPose [4). Figure (b) illustrates that obtain these keypoints from the reference image R and i-th target pose frame P:. From these, we filter out commonly visible points, defining X {x1,
Image
Keypoints
Common (X) Optimal(x'
Reference
ta
Torget
Keypontta Namberiig
Keypoints reference and target, point set (X), and optimal subset (x)) Figure 6: Illustration of (a) pre-defined 17 key- points and (b) keypoints reference and target human, their common point set, and optimal set.
(a) 17 keypoints
JN
reference and Y = {yr,... [NN] target set. Following Procrustes analysis (i.e., Eq. 2
we obtain optimal transformation parameters s* E RR. r" E R222, l" E R1xx and warp all pixels (n is the number of pixels) in the reference image by mapping below:
[u, v] E R
M: : [u, u] > (",,, v]
tt.
Therefore, we obtain the i-th calibrated image C; = (P) using Procrustes warping L which aligns R with human shape in P. using mapping M. However, considering all N common visible points for warping is unreasonable since the reference and target shapes cannot perfectly overlap due to differing poses (e.g., the arms in Figure (b) cannot overlap). To address this, apply Procrustes warping to subset x C X of the common points and select the most effective subset x" based on our defined alignment score function below (subset y is also defined corresponding to the x):
x argmaa h(P,, )
(4)
x
where C denotes calibrated image using keypoints of subset x. The is alignment score function that computes pixel-wise loU To of human shapes between pose P; and calibrated image C Thus, the final i-th calibrated image is defined as C, = Cx with optimal subset x
RReason for choosing the Procrustes transform: Transformations are largely categorized into (1) shape- preserving (e.g.. linear, Procrustes) and (2) shape-distorting (e.g., affine). To put the conclusion first, the Procrustes transform was the most effective. Shape-distorting methods were less effective than shape-preserving due to the loss of visual information by distortion. Detailed analysis is available in Table and Figure [T] *ee place more points on informative regions human identity (face: 5, torso: arms: 4, legs: 4),
SAM [16] is used for segmenting each human shape measure pixel-wise IoU.
5

---

4.2 Iterative Propagation
Conceptually, as shown in Figure our proposed TPC aims to mitigate compositional misalignment by feeding calibrated image latent features c = cc,... cr.] into the conditioning module (i.e., cross-attention) of denoising dif- fusion along with the original reference latent feature r Thus, the c is designed to bridge the correspondence of the human shapes between the reference image and target pose. However, the pose variation within the target pose frames affects the degree of calibration with the reference image, which reduces temporal consistency in the output. To ad- dress this, we introduce Iterative Propagation (IP) to enhance consistency among calibrated latent features. Figure ( iillus- trates the IP process during the diffusion denoising. The IP forms M groups of sequential features in calibrated features c, randomly selects a feature within each group, and updates all features in the group with the selected one. This method enhances temporal consistency among calibrated latent fea- tures while maintaining compositional alignment with the target pose due to the continuity of target pose variation. The random selection of features ensures all features have an equal chance of being chosen during the denoising process.
Group
Group C. G
Group M G o
900
Group 6
Group 2
Group M
Ge 6
Group
Group 2
Group
OL-1
300
Denoising step
Figure 7: Illustration of iterative prop- agation on calibrated latent features. It shows M groups on L frame cal- ibrated features and updates features in each group with randomly selected ones during the denoising process.
4.3 Plug-and-Pae Play Test-time Procrustes Calibration
We integrate calibrated latent feature into human image animation systems by applying it into conditioning module (i.e., cross-attention) of video diffusion UNet. For i-th frame m pasch-wise image feature P, E R reference feature Rmxd and calibrated latent feature c, E mxd
the cross attention is formulated as Q < Softmax (QKT) /dVV, where it satisfies Q = =P; and jr, c.] E 222 is concatenated latent condition.
K = V
5
Experiments
5.1
Experimental Settings
Implementation Details. SAM [16] is used for screening out the background in calibrated images. VA-V [30] is used for encoding images of the video. The number of groups in iterative propagation chosen as M = 30 under ablation studies in Table The average number of video frames is about 120. We use Stable Diffusion 1.5
23 for all baselines on 4 NVIDIA A100 GPUs. We follow the same pose encoders and image encoders of baseline models.
Data and Baselines. We validate human image animation on two popular benchmarks (i.e., TikTook 14] TED-talks (25) about test split. Due validation splits, we provide valid sets matching the test set sizes for the ablation study. We further collected 14 samples from Tok and TED-talks as another test split. These samples contain compositional misalignment about rotation and scaling between human shapes in reference images and motion videos. The criteria for misalignment include relative angle and scaling differences, with samples having relative angle 0 T relative scale IoU < 0.7. As shown in Figure T (a), the relative angle measures angle between the straight lines from the pelvis the nose humans of reference and target poses using keypoints estimator [4) The relative scaling measures the IoU (Intersection of Union) between bounding boxes of humans in the reference and the target. Procrustes Calibration is validated on recent diffusion-baser human image animation models including MagicAnimate 3 DisCo 31, AnimateAnyone
13, DreamPose
15 on their public codes and papers.
"The dimensions are E xxxx and r E gg by d-dimensional patch-wise image encoder Enc(-). where is the number of image patches and the I the number of frames. *Supplementary provides all the video links of these samples.
As the code is not available,
the work at: http://ittabcamoooooeeeeeeee/eed

---

Table 1: Quantitative evaluations of Test-time Procrustes Calibration (TPC) with recent diffusion- based human image animation models. A-Anyone: AnimateAnyone, M- M-Aniaate: MagicAnimate. It reported in a format of (original test set compositional misalignment test set).
Image
Vidco
Method
PSNR 1
SSIM
LPIPS
FID
HD-VID
1
Human
TiKTok 14
DreamPose 7.22/9.66 64
27.31/25.
0.532/0.481
0.449/0.529
55.4/87.2
61.1/93.1
568/738
0.04
DreamPose- + TPC
5,15/5.31
28.47/28.01
0.620006133
0,406/0.412
48.4/49.3
54.7/56.3
426/441
0.96
DisCo
4.09/5.23
28.43/24.97
0.641/0.512
0.312/0.992
37.17144
58.3/82.1
339/522
0.28
DisCo + TPC
3,49/3.82
28.92/28.89 87
0,689/0.673
0,283/0,287
34.3/36.2
$1,2/52.4
281/297
0.72
A-Anyone
3.774482
29.00/26.32
0.670/0.884
0,28970,392
32,9165.2
$4,2/59.1
296442
0.24
A-Anyone + TPC
3.32/3.53
29.27/29.01
0,705/0.688
0.264A12275
31.3/52.6
48.7/49.3
254/269
0.76
M-Animate
3,174,36
29.11/27.82
0.71770.641
0.241A0.321
31.849.2
22.4/52.3
182562
0.34
M-Animate TPC
2.98/3.19
29.43/29.21
0.75300731
0.232/02499
29.2/30.4
21.02199
158/164
0.66
TED-tals (25
DreamPose
6.65/7.41
27.63/26.11
0,559/0.482
0.421/0.521
63.4/86.2
43.6/64.2
411/532
0.24
DreamPuse TPC
6.17/6.31
0.593/0.589
0.393/0.404
53,2/55.2
38.2/38.9
DisCo
3.52/6.91
28.51/26.71
0.661/0.511
0.309/0.451
34,5/7111
28.2/52.6
332/471
0.24
DisCo + TPC
3.18/3.23
28. 28.93/28.88 87
0.704/0.692 692
0,283/0.22 294
31.8/32.6
25.1/25.8
298/304
0.76
Anynye
3.12/6.61
28.93/27.05
0.712/0.613
0,267/0.361
29.3/54.2
20.3/39.5
A-Anyone + TPC
2
29.31/29.25
0,7530.742
0,254/0.269
27. 27.8/28.6
18.8/19.5
173/78 78
0.76
M-Amimate
2,9244.11
29.17/27.47
0.734/0.661
0.239/0.312
25.7/47.1
20.2/39.1
136/331
0.24
M-Animate TPC
2.77/2.77
29.52/29.41
0.782/0.764
0.233/0.249
24.3/25.6
19.2/20.4
128/137
0.76
5.2
Evaluation Metrics
We evaluate videos in terms of single-frame quality and video quality. For the Lingle-frame, we measure Peak Signal-to-Noise Ratio (PSNR) [12], Structural Similarity Index Measure (SSIM) [2] Learned Perceptual Image Patch Similarity (LPIPS) 44 FID (Fréchet Image Distance) (9. and L. error between output and ground-truth images. For the video quality, we measure Fréchet Video Distance (FVD) (29] and FID-VID [I] All automatic metrics are averaged over 10 runs with different seeds. We also analyze human preferences for results from the baselines with and without our TPC.
5.3 Experimental Results
Quantitative Comparisons. Table presents evaluations of image animation on two benehmark datasets (i.e., TikTok, TED-talks) using recent diffusion image animation baselines (MagicAnimate, DisCo, Animate DreamPose) with TPC across three assessments (i.e., image, video, human). Evaluations are conducted on two test splits: the original set and the compositional misalignment set. Initially, all baselines use a single reference image latent for conditioning all poses. After integrating TPC, they use calibrated latents corresponding to each pose. Consistent improvements in image and video quality are observed in both test sets. All baselines struggle with the compositional misalignment set, but when integrated with TPC, they achieve quality close to the original test set. This demonstrates that morphological similarity affects the current diffusion model's s ability to map human shapes from reference target.
Qualitative Comparisons. To validate our proposed TPC, we applied it to four recent baselines and compared the original results. We used the same types of human pose inputs (c.g.. OpenPose, DensePose) for each baseline to prepare target motion videos. F shows predictions on four different samples exhibiting compositional misalignment. The top left results show predictions on temporal misalignment between the reference and target due to the target's bending motion. MagicAnimate's fidelity diminishes with increased bending (red box in the last frame), whereas the model with TPC maintains high fidelity. The top right results display predictions on temporal misalignment due to a walking motion towards the front, with TPC similarly enhancing DisCo's performance. The bottom left results show consistent misalignment across all frames. DreamPose struggles with low fidelity, causing unwanted stripes on the pants (red box), which are clearly removed with TPC. The results in the bottom right exhibit consistent misalignment due to scale difference. In AnimateAnyone, the reference human's 's yellow pants were incorrectly mapped onto the target human's 's arms, making them appear yellow. However, the TPC completely mitigates this incorrect mapping. This oceurs because the calibrated image filters out the pants and enhances the correspondence of each body part. (Please, refer to the calibrated images also in the Appendix.)
7

---

Target
Target motion
Reference image
Reference image
DisCo
MA
MA + TPC
DisCo TPC
GT
Target motion
Target motion
Reference image
Reference image
DP
AA
ee
TPC
DP
GT
Figure 8: Qualitative results about applying TPC on diffusion human image animation systems (i.e.. MA: MagicAnimate, DisCo, AA: AnimateAnyone, DP: DreamPose) cases about compo- sitional misalignment human shape between reference image and target motion: Temporal misalignment by motion affecting factor rotation (top left) and scale (top right). Consistent misalign- ment affecting rotation (bottom left) and scale (bottom right). Calibrated images and predictions of compositional aligned samples are available in Appendix. Please also video in the supplementary.
Figure 10 shows the results on reference images from an unseen domain, generated by the T2I model [21]. applied in compositional misalignment scenario. The left displays a temporal misalignment sample due bending motions, with improved fidelity in both baselines, especially in DisCo. The right side displays consistent misalignment sample, where the baselines show low precision in transferring identity when the reference flipped upside down. However, with the integration of TPC, their performance is significantly improved.
PSNR accordind scaling beitween target and reference
PSNR according to ronating between target and reforence
(PNN))
Dussirns Deaeksss-11 1PC
Saselins
-2/2
6/2
(0)
Figure 9: Robustness analysis of variations in compositional misalignment. Measurements of scale and rotation are in [
Robustness analysis To assess the robustness of baselines with TPC, we measured fidelity (PSNR) by varying the scale and rotation of the human reference, shown in Figure In the case of scaling, baselines show a drop performance when size difference between the reference and target human shapes falls below an loU of 0.6. However, TPC maintains optimal performance even down to an IoU of 0.4. Notably, TPC enhances baseline robustness all rotational variations by ensuring the calibrated image aligns well with shape differences caused by rotation.

---

Target motion
Target mation
Reference image
Reference image
DisCo
DP
DisCo TPC
TPC
MA
AA
MA+
AA+TPO
Figure 10: Qualitative results about unseen domain reference image. TPC enhances human image animation models' robustness on samples about compositional misalignment in the unseen domain.
Table 2:
Ablation studies on transformation methods for reference image calibration and iterative propagation (IP) on TED-talks and TikTok. (validation splits, average score com- positional aligmmen/mmialignmeenn)
Table 3: Inference time of baselines with TPC. DC: DisCo.
Procrustes Aftino
Meibod
sec/frame
Foreground
Background
DF
18.3
Methed
SSIMT
IVD1
SSIMI
IVD.
TPC
18.9
0.702
191
170
DC
5.8
Affine
0.704
193
0.754
171
DC + TPC
6.3
Procrustes
0.734
162
0.782
142
AA
2.3
Figure 11: Comparisons of calibrated images using an affine transform and Pro- crustes transform.
w/o IP
0.709
184
0.728
162
AA+ TPC
2.6
(M=20)
w/ TP (M=30)
0.734
162
0.782
142
MA
1.4
(M=40)
0.728
165
0.777
145
Ablation Study. Table 2 presents ablation studies transformation methods and iterative prop- agation. Transformations are categorized as shape-preserving (e.g., Linear, Procrustes) and shape- distorting (e.g., Affine). Linear transformation performs scaling and rotation without translation, based on the object's bounding box coordinates. The Procrustes method is the most effective, trans- forming by selecting the optimal keypoint. The affine transform less effective due shearing, which can cause distortion and information loss by moving the subject out of the frame, as shown in Figure IT] The second section presents an ablation study on iterative propagation, showing significant enhancements in foreground and background, with M-=0 being the most effective. Table provides inference time on baselines with TPC. Our method requires only minimal additional time because processes all frames batch-wise, generating the calibrated image in a single iteration.
5.4 Broader Impacts and Ethic Statements
Visual generative models present a range of ethical dilemmas, including the creation of unauthorized counterfeit content, potential privacy breaches, and challenges related to fairesss. Due to our reliance on architecture these models, our work inherently adopts these ethical vulnerabilities. Addressing these concerns is imperative and requires the establishment of comprehensive regulations and technical countermeasures. We are exploring advanced measures such as learning- digital forensics and digital watermarking ethically navigate the complexities of visual generative models.

---

5.5 Limitation and Future work
We provide an overview of the various limitations and potential development directions identified during this study. Human image animation systems transfer reference images to target poses. However, some frames in target pose video may lack proper specification, leading to flicker low fidelity in model predictions. Thus, integrating high-quality pose estimation is crucial. Another limitation is that significant differences in body shape between the reference and target poses result awkward transfers, such as transferring a skinny person to fat target pose. To achieve natural results, system or module that aligns body shapes is required. For the future extension of human image animation system, it is essential to ensure robust operations under multiple individuals. The system should ensure consistent identity transfer across multiple individuals and accurately adapt to those appearing at specific times. To achieve this, video technologies about recognition [22] 46 and
perception [37 40] can be further incorporated. Currently, conditioning relies solely on images, but it is anticipated that various other modalities, such as text and audio, could also be incorporated. Especially. the integration of the text modality with emerging large language models [ 28 is expected to drive innovative industrial advancements, with the introduction of video//mage-basee
conversational systems 36 42 38) serving as compelling utility. Lastly, integrating technologies focused speed 18
resource efficiency [34], and test-time calibration R5 is expected to
improve human image animation systems' applicability into real-worke environments.
Conclusion
Current diffusion-based human image animation systems facing challenges samples of compo- sitional misalignment of human shapes between a reference image and target pose frames. To this end, this paper presents Test-time Procrustes Calibration (TPC) which improves the robustness of image animation models on the compositional misalignment samples in a model model-agnostic manner. Extensive experiments demonstrate the effectiveness of TPC.
Acknowledgements
This work was supported by Institute for Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government(MSIT) (No.RS-2021-1221338 Develop- ment of Causal AI through Video Understanding and Reinforcement Learning, and Its Applications to Real Environments) and partly supported by Institute of Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea MSV)) Ro--S2222--122000 2022-0-00188 00440,0441110 Development and Study of AI Technologies Inexpensively Conform Evolving Policy on Ethies)
References
Yogesh Balaji, Mantin Renqiang Min, Bing Bai, Rama Chellappa, and Hans Peter Graf. Conditional gan with discriminative filter generation for text-to-video synthesis. IJCAI, volume I, page 2, 2019.
[2] Daniel Bolya, Cheng- Cheng-Yang Fu, Xiaoliang Dai. Peizhao Zhang, Christoph Feichtenhofer. and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arX1v:2210.04411, 2022.
[3] Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005. 10t::2055.1415,,,, 2020.
[4] Zhe Cao, Tomas Simon, Shih-En En Wei, and Yaser Sheikh. Realtime multi-person part affinity fields. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 7291-7299, 2017.
person 2d pose estimation using
[5] Prafulla Dhariwal and Alexander Nichol, Diffusion models beat gans on image synthesis. Advunces in neural information processing systems, 34:8780- 1448888--779, 2021.
[6] John C Gower. Some distance properties of latent root and vector methods used in multivariate analysis. Biometrika, 53(3-4):325-335 1966.
[7] John C Gower. Generalized procrustes analysis. Psychometrika, 40:33-51 01, 1975.
[8] Riza Alp Güler, Natalia Neverova, and lasonas Kokkinos. Densepose: Dense human pose estimation in the wild. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7297-7306. 2018
10

---

[9] Martin Heusel. Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by time-scale update rule converge a local nash equilibrium. Advances neural information processing systems, 30, 2017.
[10] Jonathan Ho, Ajay Jain, and Pieter Abheel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:684006855 2020.
[11] Ji Woo Hong. Sunjae Yoon, Junyeong Kim, Chang D Yoo. Joint path alignment framework for 3d human pose and shape estimation from video. IEEE Access, 11:43267 43275, 2023.
[12] Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pages 2366-2369. IEEE, 2010.
[13] Li Hu, Xin Gao, Peng Zhang, Ke Sun, Bang Zhang, and Liefeng Bo. Animate anyone: Consistent and controllable image-to video synthesis for character animation. arXiv preprint urXiv:2311.17111 2023,
[14] Yasamin Jafarian and Hyun Soo Park. Learning high fidelity depths of dressed humans by watching social media dance videos. In Proceedings of the IEEF/CVF Conference Computer Vision and Pottern Recognition, pages 12753-12762, 2021.
[15] Johanna Karras, Aleksander Holynski, Ting- Chun Wang, and Ira Kemelmacher Shiizerrrn-Shizmaannn Dreampose: Fashion image-to-videe synthesis via stable diffusion. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 22623-22633, IEEE, 2023.
[16] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloc Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015-4026, 2023.
[17] Gwanhyeong Koo, Sunjae Yoon, Ji Woo Hong, and Chang D Yoo. Flexiedit: Frequency-awee aware latent refinement for enhanced non-rigid editing. arXiv preprint arXiv:2407.17887 2024.
[18] Gwanhyeong Koo, Sunjae Yoon, and Chang D Yoo. Wavelet-guided acceleration of text inversion in diffusion-based image editing. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 4380-4384. IEEE, 2024.
[19] Julieta Martinez, Rayat Hossain, Javier Romero, and James J Little. simple yet effective baseline for 3d human pose estimation. In Proceedings of the IEEE international conference on computer vision, pages 2640-2649, 2017.
[20] Vadim Popov, Ivan Vovk, Vladimir Gogoryan, Tasnima Sadekova, and Mikhail Kudinov, Grad tts: diffusion probabilistic model for text-to-speech. In International Conference on Machine Learning, pages 8599 8608. PMLR, 2021.
[21] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot lext-to-image generation. In International conference on machine learning. pages 8821-8831 Pmlr, 2021.
[22] Nikhila Ravi, Valentin Gabeur, Yuan- Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam Segment anything in images and vidcos. arXiv preprint
2024.
[23] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEED/CVF conference on computer vision and pattern recognition, pages 10684-10699 2022.
[24] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. Advances in neural information processing systems, 32, 2019.
[25] Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion represen- tations for articulated animation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13653 13662,, 2021.
[26] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models, arXiv preprint urXiv:2010.02502, 2020.
[27] Yang Song. Jascha Sohl- Sohl-Dickstein, Diederik P Kingma. Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-basee generative modeling through stochastic differential equations. arXiv preprint arX1v:2011.13456, 2020.
11

---

[28] Hugo Touvron. Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv: 13971, 2023.
[29] Thomas Unterthiner, Sjoerd Van Steenkiste, Karol Kurach, Raphael Marinier, Marcin Michalski, and Sylvain Gelly. Towards accurate generative models of video: new metric & challenges. arXiv preprint arXiv:1812.01717. 2018.
[30] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning, Advances in neural information processing systems, 30, 2017.
[31] Tan Wang. Linjie Li, Kevin Lin, Yuanhao Zhai, Chung-Ching Lin, Zhengyuan Yang. Hanwang Zhang, Zicheng Liu, and Lijuan Wang. Disco: Disentangled control for realistic human dance generation. arXiv preprint 237712207.000400 2023.
[32] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. JEEE transactions on image processing, 13(4):600-61 1610000--,,,-0 2004.
[33] Zhongcong Xu. Jianseng Zhang, Jun Hao Liew, Hanshu Yan. Jea Liu, Chenxu Zhang. Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. arXiv preprint arXiv:2311. 164998, 2023.
[34] Xingyi Yang. Daquan Zhou, Jiashi Feng, and Xinchao Wang. Diffusion probabilistic model made slim. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 22552-22566 2023.
[35] Hee Suk Yoon, Joshua Tian Jin Tee, Eunseop Yoon, Sunjae Yoon, Gwangsu Kim, Yingzhen Li, and Chang D Yoo. Esd: Expected squared difference as a tuning-free free trainable calibration measure. arXiv preprint arXiv:2303. 02477, 2023.
[36] Hee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Kang Zhang, Yu-Jung Heo, Du-Seong Chang, and Chang D Yoo. Bi-mdrg: Bridging image history in multimodal dialogue response generation. arXiv preprint arXiv:2408.0550 2024.
[37] Sunjae Yoon, Woo Hong, Eunseop Yoon, Dahyun Kim, Junyeong Kim, Hee Suk Yoon, and Chang Yoo. Selective query-guided debiasing for video corpus moment retrieval. In European Conference on Computer Vision, pages 185- 25--000.65 Springer, 2022.
[38] Sunjae Yoon, Dahyun Kim, Eunseop Yoon, Hee Yoon, Junyeong Kim, and Chang Yoo. Hear: Hearing enhanced audio response for indoo-gronded dialogue. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11911-11922 2023.
[39] Sunjae Yoon. Gwanhyeong Koo, Woo Hong. and Chang Yoo. Dni: Dilutional noise initialization for diffusion vidco editing. arXiv preprint ar arXiv:2499 13037, 2024.
[40] Sunjae Yoon, Gwanhyeong Koo, Dahyun Kim, and Chang D Yoo. Scanet: Scene complexity aware network for weakly- supervee video moment retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13576-13588 2023.
[41] Sunjae Yoon, Gwanhyeong Koo, Geonwoo Kim, and Chang D Yoo. Frag: Frequency adapting group for diffusion video editing. arXiv preprint arXiv:2406. 06044,, 2024.
[42] Sunjae Yoon, Eunseop Yoon, Hee Suk Yoon, Junyeong Kim, and Chang Yoo. eeformation-theoreed text hallucination reduction for video- radeo-grounded dialogue. In Proceedings of the 2022 Conference Empirical Methods Natural Language Processing, pages 4182 4193, 2022.
[43] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-lo-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3836-3847, 2023.
[44] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 386-595. 2018.
[45] Jian Zhao and Hui Zhang. Thin-plate spline motion model for image animation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 3657-3666, 2022.
[46] Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, and Yichen Wei. Flow-guided feature aggregation for video object detection. In Proceedings of the IEEE international conference on computer vision, pages 408 417, 2017.
12

---

A
Appendix
A.1 Algorithm for Iterative Propagtion
Algorithm Iterative Propagation
1: Input: Calibrated L frame features [c1: ... CL).
Number of frame groups M
2: Output: Denoising t-step propagated features = ccj. cf.
3: Initialize the number of frames in each group: m M
4: for t T do
T is total denoising step.
5: for i = to .M do
6: <- ((--1) 1) m
7: Sample ~ U11, m)
b bu1,, m) is a discrete uniform distribution between 1 and m. Update list.
8: + of n+22 + f [Cntr Catri ... Cnarl
9: end
c² [cf ... m mm+11" c2ml (c ,c,l
> Concatenation
11:
end
A.2 More qualitative results
Explanation
Figure
2
shows calibrated reference images corresponding the samples in Figure 8.
Figure
shows results on identical motion with different reference images.
Figure
14 and
15 show results on identical references with different motion videos in the TikTok
dataset.
Figure shows results on multiple humans in reference and target motion.
As in Figure 1, here we visualize the target pose in a format of DensePose for the visibility of target motions. The video samples in Figure 8 are publicly available from:
13

---

Target motion
Reference image
Target motion
Reference image
Target motion
Reference Image
Target motion
Reference image
Figure 12: Qualitative results about calibrated images on the samples in Figure g
14

---

Target motion
Reference image
DP
DP TPC
DisCo
DisCo TPC
AA
AA TPC
MA
MA+ TPC
Figure 13: Qualitative results about applying TPC on recent diffusion- iiffusiin-bbsed human image animation
systems (i.e., DreamPose (DP), MagicAnimate (MA), DisCo, AnimateAAyone(AA on identical motions with different reference images. The images are obtained from T2I image generation model.
15

---

Target motion
Reference image
AA
AA TPC
Target motion
Reference image
AA TPC
INC
Target motion
Reference image
AA
AA + TPC
ISC
Figure 14: Qualitative results about applying TPC on recent diffusion based AnimateAnyone (AA) on identical reference image with different motions. The motion videos are obtained from TikTok dance video dataset. (test-split)
16

---

Target motion
Reference
image
MA
MA MA+ TPC
Target motion
Reference
image
MA
MA TPC
Target motion
Reference image
MA
MA+ TPC
Figure 15: Qualitative results about applying TPC on recent diffusion based MagicAnimate (MA) on identical reference image with different motions. The motion videos are obtained from TikToo dance video dataset. (test-split)
17

---

Target motion
Reference
image
MA
MA TPC
Target motion
Reference image
MA
MA+ TA+TPC TPC
Figure 16: Qualitative results about applying TPC recent diffusion based MagicAnimate (MA) on multiple humans of reference and motion.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, in the line 15-17 in the Abstract and last paragraph in the Introduction. Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: is provided as Limitation section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers DisCover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
19

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Method section contains the details of our proposed framework.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
20

---

Answer: [Yes]
We submit the source links for data used in our experiment and also demo video.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (ht cttpuscccclllliiicc hetplliccggiidee/COO// wtwtieeee//oeeubiisssssssssse ssionPolicy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: It is explained in the Implementation Details section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
is explained the evaluation metrics.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
21

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide information about GPU resources in the Implementation Details section for implementing our framework.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We read the NeurlPS Code of Ethics and follow of
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: provide a broader impact including societal negative impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
The broader impact contains information about safeguards.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We explicitly provide all the references used including url links.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, |paper swi thcode. com/ conasaatssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "u5pL46YaUs", "reviewer": "Reviewer_mDg5", "summary": "This work proposes a method that combines exsiting human image animation method and  Procrustes Warping, which improves the robustness of image animation approach. In addition to the explicit image warping, this work also proposes an interative propagation method to improve the temporal consistency. Experiments show that the proposed method is effective in enhancing the animation performance.", "questions": "How do you apply TPC in DisCo and MagicAnimate? I am a little bit confused since the Figure 5 only depicts the structure of Anymate Anyone.", "limitations": "Discussed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is well motivated, the observation of the misalignment between reference image and target pose is straightforward and insightful.\n2. The experiments in this work are solid, both the comparisons and ablations are extensive and detailed. Compared with the baseline methods, the improvements are noticeable.\n3. The writing is clear and easy to follow.", "weaknesses": "1. As far as I know, the Procrustes alignment is popular in the evaluation of 2D/3D human pose estimation. So, this work introduces this simple technique into the human image animation but technically this contribution is not novel enough.\n2. There is no detailed description about the training process of this method, it could be better to introduce more about the training process in the main text.\n3. From the Table 2 we can see that the best M is 20, but the evaluation metrics shown in the table are SSIM and FVD, it could better to show more results to help reader understand the ablation comprehensively. \n4. Are there any trade-off boserved between the temporal smoothness and motion precision? For example, if M=1, the temporal consistency is the best while the pose precision is the worst?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We have addressed your concerns to the best of our ability. \n\n[**Q1**] Procrustes alignment is popular in evaluating 2D/3D human pose estimation. This work introduces simple technique into the human image animation but technically contribution is not novel enough.\n\n[**A1**] We are not claiming novelty for the Procrustes alignment but instead for the test-time calibration concept for diffusion model that aligns the target subject in the reference image to the pose in the pose video. To the best of our knowledge, our study is the first work to address the issue of human image animation systems performing poorly in real-world scenarios, identifying compositional misalignment as the cause. When incorporated with any diffusion-based image animation system, our calibration significantly enhances its robustness in real-world applications.\n\nThe Procrustes approach is one of our experimental attempts to implement this calibration concept. Table 2 in our manuscript also provides results from other attempts, including affine and linear transformation approaches. The Procrustes approach proved to be the most effective and our technical contributions (i.e., optimal keypoints selection and iterative propagation) focus on optimizing the calibration.\n\n[**Q2**] There is no detailed description about the training process of this method, it could be better to introduce more about the training process in the main text.\n\n[**A2**] The proposed test-time method does not have a separate training procedure, generating video during inference. No training is required as mentioned in Line 18 and 99.\n\n[**Q3**] From the Table 2 we can see that the best $M$ is 20, but the evaluation metrics shown in the table are SSIM and FVD, it could be better to show more results to help reader understand the ablation comprehensively.\n\n[**A3**] As mentioned in Line 288 and also shown in Table 2, $M$=30 provides the best setting. Table A below provides the ablation studies with additional metrics for both single frames (i.e., PSNR, SSIM, LPIPS, FID, L1 error) and videos (i.e., FID-VID, FVD). We will incorporate this into Table 2 of our manuscript. Thank you.\n\n**Table A.** Ablation study of Iterative Propagation (IP) according to the number of group $M$, reported in a format of foreground/background (validation splits, average score of compositional alignment/misalignment).\n|||||||||\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n||L1↓$_{\\times \\textrm{E-04}}$|PSNR↑|SSIM↑|LPIPS↓|FID↓|FID-VID↓|FVD↓|\n|w/ IP ($M$=1)|3.92/3.25|28.61/29.32|0.709/0.767|0.281/0.249|29.1/26.1|36.7/27.6|177/153|\n|w/ IP ($M$=10)|3.46/3.06|28.88/29.37|0.715/0.769|0.279/0.247|28.6/25.9|35.2/27.4|171/151|\n|w/ IP ($M$=20)|3.12/2.83|29.06/29.48|0.731/0.776|0.274/0.244|27.3/25.6|31.5/26.9|164/145|\n|w/ IP ($M$=30)|**3.08**/**2.79**|**29.12**/**29.50**|**0.734**/**0.782**|**0.273**/**0.241**|**26.9**/**25.1**|**31.2**/**26.4**|**162**/**142**|\n|w/ IP ($M$=40)|3.10/2.81|29.08/29.46|0.728/0.777|0.275/0.243|27.0/25.3|31.5/26.8|165/145|\n|w/ IP ($M$=50)|3.09/2.79|29.09/29.43|0.731/0.777|0.274/0.242|26.9/25.2|31.8/27.2|170/149|\n|w/o IP (w/ IP $M$=video length)|3.11/2.80|29.09/29.47|0.731/0.778|0.275/0.243|27.0/25.2|33.2/27.1|169/150|\n\n[**Q4**] Are there any trade-off observed between the temporal smoothness and motion precision? For example, if $M$=1, the temporal consistency is the best while the pose precision is the worst?\n\n[**A4**] Yes, we observed a slight trade-off between consistency and precision within an effective operating region for iterative propagation (IP). Figure A in our attached PDF presents a sensitivity analysis of the IP according to $M$ in terms of synthesizing precision (LPIPS) and temporal consistency (FVD). We identified the effective operating region as 20 < $M$ < 50 (for an average 120-frame video), where the IP shows a slight trade-off.\n\nHowever, when we selected $M$ = 1, both precision and consistency were at their lowest. This outcome is attributed to the insufficiency of calibrated images, as only a single calibrated frame is selected to propagate all the target frames at each denoising step. The selected calibrated image failed to provide accurate calibration to the target frame due to significant temporal misalignment. Consequently, this reduced precision, and the series of inaccuracies also negatively impacted consistency. Therefore, the trend in consistency changes led us to determine that the optimal choice is $M$ = 30. \n\n[**Q5**] How do you apply TPC in DisCo and MagicAnimate? \n\n[**A5**] In Figure B of our attached PDF, we illustrate the applications of TPC on top of DisCo and MagicAnimate. For DisCo, the image encoder  in Figure 5 is the CLIP Encoder, and the pose encoder is the pose ControlNet of DisCo. For MagicAnimate, the image encoder corresponds to the Appearance Encoder of MagicAnimate, and the pose encoder corresponds to ControlNet.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4dSELypVu3", "reviewer": "Reviewer_rFbR", "summary": "This paper starts from an interesting problem that what happens when the motion condition and the reference image are not well aligned. It analyses the robustness of an existing human animation network given different levels of misalignment, and tries to find the underlying cause through attention maps. Then this paper propose TPC, a test-time calibration module to mitigate this issue. This module provides a calibrated latent feature as an additional condition to the diffuser. The calibrated latent is optimized via Procrustes Warping, and Iterative Propagation is proposed to improve temporal consistency among calibrated features.", "questions": "What if directly calculating the transformation between the reference image and the target pose via some heuristical evidences? For example, considering the torso part, i.e., keypoints set {6,7,13,12}, since torso keeps rigid to some extend, compared to arms and legs. Then using the vector (mid(6,7) , mid(13, 12)) to compute the rotation angles, the perimeter for scales, the center coordinate for shifting. I know this design is not optimal. But how is the result with such simple idea? I noticed Table 2, the \"Linear\" and \"Affine\" setting, but seems not exactly the same one.\n\nCan this method be potentially applied to a broader tasks that require spatial alignment between some input variables?", "limitations": "The Broader Impacts and Ethic Statements have been adequatedly addressed.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper is among the most innovative work I've ever seen in human animation area. It is a curiosity-driven work starting with an interesting problem. The solution is neat and elegant.\n\nThe presentation is clear and easy to follow. The analysis in Fig 1 and Fig 2 is very interesting and insightful.\n\nTechnically, using keypoints as the evidence to align shapes is quite direct and reasonable. Using Procrustes analysis to optimize a transformation is sound.", "weaknesses": "I did not find obvious weakness, except that the demo results still contain some artifacts. But it is mainly due to the baseline MA or Disco. I recommend the authors to try some new methods as the baseline that have higher quality.\n\nThe method is insightful, but only for human animation, a small area. It would greatly enhance the value of this paper if it can be applied to some more general tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your recognition of our work. We address your concerns below.\n\n[**Q1**] Demo results still contain some artifacts. But it is mainly due to the baseline MA or Disco. I recommend the authors to try some new methods as the baseline that have higher quality.\n\n[**A1**] Yes, as shown in Figure F of our attached PDF, the application of TPC with more recent models, such as Champ (to be published in ECCV'24) and MimicMotion (Arxiv'24), also enhances their inferences, resulting in better quality. We will incorporate this into our manuscript. Thank you!\n\n[**Q2**] It would greatly enhance the value of this paper if it can be applied to some more general tasks. Can this method be potentially applied to broader tasks that require spatial alignment between some input variables?\n\n[**A2**] Yes, as shown in Figure C, we confirmed that our proposed calibration concept can also be applied to other image-to-image tasks (e.g., virtual try-on) and video-to-video tasks (e.g., video editing), enhancing their quality. We will incorporate this as an additional section for plug-and-play applications in our manuscript.\n\n[**Q3**] What if directly calculating the transformation between the reference image and the target pose via some heuristical evidences? For example, considering the torso part, i.e., keypoints set {6,7,13,12}, since torso keeps rigid to some extend, compared to arms and legs. Then using the vector (mid(6,7) , mid(13, 12)) to compute the rotation angles, the perimeter for scales, the center coordinate for shifting. I know this design is not optimal. But how is the result with such simple idea? I noticed Table 2, the \"Linear\" and \"Affine\" setting, but seems not exactly the same one.\n\n[**A3**] Figure D illustrates the resulting frames based on the set of keypoints on the torso. This method is as effective as our original method when the torso is clearly visible in the target motion (i.e., first frame). However, it shows vulnerabilities when the torso is not visible (i.e., third frame). In fact, our initial approach also used keypoints of rigid body parts (e.g., torso) for transformations, but visibility issues arose with various movements. To address this, we proposed finding the optimal keypoints for each frame, considering commonly visible keypoints between the target and reference.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ibvM52z3bu", "reviewer": "Reviewer_c8u6", "summary": "In this paper, the authors propose TPC, an alignment algorithm for human image animation systems. optimal precision is currently achieved only when the physical compositions of the human shapes in the reference image and target pose frame are aligned. Misalignment leads to a significant decline in fidelity and consistency. To address this issue, the authors propose Test-time Procrustes Calibration (TPC). TPC provides a calibrated reference image for the diffusion model, enhancing its ability to understand the correspondence between human shapes in the reference and target images. This technique can be applied to any diffusion-based image animation system in a model-agnostic manner, improving effectiveness at test time without requiring additional training.", "questions": "(1)\tIn some particular situations, the protagonist in the given reference image only shows half of the whole body, while the driven target pose shows the whole body parts. Can TPC handle this sort of situation? Could you please give some quantitative and qualitative experiment results regarding this situation? \n\n(2)\tI am unsure how you can estimate or ensure the accuracy of the subset X. Could you please provide some visual comparison results in terms of different subsets with different numbers of key points? Additionally, could you provide the details of the common filtering process?", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) TPC bridges the gap in shapes between reference and target by providing correspondence guidance latent. With this guidance condition, diffusion-based animation systems achieve robustness to fidelity variations and maintain temporal consistency among frames.\n\n(2) The TPC is simple and works in a model-agnostic manner without additional training.", "weaknesses": "(1)\tThe implementation details of Iterative Propagation are not relatively comprehensive, and I suggest the authors make an algorithm table to depict the concrete process of the proposed Iterative Propagation.\n\n(2)\tThere are some skeleton alignment algorithms in the previous works, such as Everybody Dance Now. The authors should conduct additional quantitative and qualitative comparison experiments between TPC and other previous skeleton alignment algorithms. \n\n(3)\tAccording to my observations of your provided demos, the backgrounds of some rotation cases encounter considerable changes during the animation process. The background details don’t align with the given reference image, such as the layout and color distribution. For instance, in the galaxy case, the positions of the stars in the background of the animated video are not the same as those in the original given reference image.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We address your concerns below.\n\n[**Q1**] I suggest an algorithm table of the proposed Iterative Propagation.\n\n[**A1**] Figure E of our attached PDF presents the algorithm table for Iterative Propagation. We will incorporate this into our manuscript. Thank you.\n\n[**Q2**] The authors should conduct additional quantitative and qualitative comparison experiments between TPC and other previous skeleton alignment algorithms (e.g., Everybody Dance Now [1]).\n\n[**A2**] Our investigation identified several transformation (i.e., alignment) methods, which can be categorized into two types: (1) shape-distorting (e.g., Affine [2,3], Perspective [4,5]) and (2) shape-preserving methods (e.g., TPC (Ours), Global Pose Normalization (GPN) [1]). Qualitative results in Figure G show that shape-distorting transformations can accurately align to the targeted region but tend to cause significant information loss in other areas due to distortion, leading to low fidelity. Shape-preserving methods generally performed better, but GPN struggles with rotational motions due to its inability of rotation. Quantitative analysis in Table B confirms that TPC is the most effective. We will include these comparisons in our manuscript. Thank you.\n\n**Table B.** Ablation study of various transformation methods for calibration, reported in a format of foreground/background (validation splits, average score of compositional alignment/misalignment). PS: Perspective.\n|||||||||\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n||L1↓$_{\\times \\textrm{E-04}}$|PSNR↑|SSIM↑|LPIPS↓|FID↓|FID-VID↓|FVD↓|\n|w/o calibration|4.51/3.57|27.81/29.15|0.689/0.758|0.304/0.259|38.3/27.7|37.2/29.4|184/162|\n| Affine|3.92/3.25|28.61/29.32|0.709/0.767|0.281/0.249|29.1/26.1|36.7/27.6|177/153|\n|PS|3.46/3.06|28.88/29.37|0.715/0.769|0.279/0.247|28.6/25.9|35.2/27.4|171/151|\n|GPN|3.12/2.83|29.06/29.48|0.731/0.776|0.274/0.244|27.3/25.6|31.5/26.9|164/145|\n|TPC (Ours)|**3.08**/**2.79**|**29.12**/**29.50**|**0.734**/**0.782**|**0.273**/**0.241**|**26.9**/**25.1**|**31.2**/**26.4**|**162**/**142**|\n\n[1] Everybody Dance Now, ICCV'19\n\n[2] FedMed-ATL: Misaligned Unpaired Cross-Modality Neuroimage Synthesis via Affine Transform Loss, ACM MM'22\n\n[3] Adaptive Affine Transformation: A Simple and Effective Operation for Spatial Misaligned Image Generation, ACM MM'22\n\n[4] Shape-Preserving Half-Projective Warps for Image Stitching, CVPR'14\n\n[5] Image stitching with perspective-preserving warping, ISPRS'16\n\n[**Q3**] The backgrounds of some rotation cases don’t align with the reference image. For instance, in the galaxy case, the positions of the stars in the background of the animated video are not the same as those in the original reference image.\n\n[**A3**] Thank you for your observation. The galaxy case issue was mistakenly generated without applying a background mask in our method, which led to the observed artifacts in background preservation. As noted in our implementation details (Line 214), we eliminate background intervention in calibrated images. When the background mask is correctly applied, it ensures alignment of background details with the reference image. Figure H shows the corrected sample with proper background preservation. We will update this and apologize for any confusion it caused.\n\n[**Q4**]  In some particular situations, the protagonist in the given reference image only shows half of the whole body, while the driven target pose shows the whole body parts. Can TPC handle this sort of situation? Give some quantitative and qualitative experiment results.\n\n[**A4**] Yes, Figure 13 in the Appendix of our paper illustrates the resulting frames in such scenarios (specifically, the sample of a woman wearing white clothes). Existing models generate unnatural lower bodies from the upper body reference. However, integrated with TPC, they seamlessly blend the lower body with the upper body's style. For quantitative analysis, we collected such samples as another test set where the reference and target human shapes are not aligned in terms of rotation or scale, indicating compositional misalignment. Table 1 provides a quantitative evaluation of compositional misalignment samples. TPC significantly improves the qualities of these samples, comparable to those of composition-aligned samples.\n\n[**Q5**] I am unsure how you can estimate the accuracy of the subset X. Provide some visual comparison results in terms of different subsets with different numbers of key points and provide the details of the common filtering process.\n\n[**A5**] Yes, Figure I in our attached PDF shows visual comparisons with different numbers of keypoints, demonstrating optimal performance with 6 keypoints. Using 4 or 5 keypoints was also effective in this sample.\n\nThe selection of optimal keypoints subset involves two steps (code available in our supplementary material).\n\n1. Filtering Keypoints: Keypoints from the reference and target images are filtered to retain only commonly visible ones. Points with a low prediction score (< 0.3, invisible poinsts are usually lower than 0.3) are removed (using OpenPose for keypoint prediction). The remaining points common to both images form the common sets $X$ and $Y$, representing the same body parts.\n\n2. Selecting Optimal Subset: From the common sets, we generate all possible subsets with four or more points (for computational efficiency). Using Procrustes Analysis, we obtain transformation parameters (i.e., scaling, rotation, translation) from each subset $\\mathbf{x} \\subset X$ to corresponding subset $\\mathbf{y} \\subset Y$ and transform the reference image into the calibrated image. We then measure the overlap accuracy of human shapes between the calibrated and target images using the pixel-wise IoU score (with SAM for segmentation). The subset with the highest score is selected as the optimal subset $\\mathbf{x}^*$. For computational efficiency, this process is also available with a batch-wise implementation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OPyuwZyfa6", "reviewer": "Reviewer_KKQ2", "summary": "This paper proposes diffusion guidance (Test-time Procrustes Calibration, TPC) for human image animation. TPC incorporates an auxiliary branch into the diffusion conditions process and provides a calibrated reference image latent. Experimental results show the effectiveness of the provided method.", "questions": "Please see the above.", "limitations": "1. The necessity of TPC\n2. The experimental results do not help the original model retain the details of the input image.\n3. Introduction needs to be reorganized", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.\tThis paper studies an interesting problem of human image animation, that is, when the given human image and pose image have shape differences, the performance of the original model is poor.\n2.\tThe structure of this paper is well-organized and easy to follow.\n3.\tThe experimental results show the effectiveness of the proposed method.", "weaknesses": "1.\tIn Figure 1a, I acknowledge that there may be a large shape difference between a given image that needs to be animated and a guided image. In this case, I think the most direct and useful way is to find a way to align the two images, and I don't think it is very difficult to align the two images. So why do we need the proposed TPC method?\n2.\tIn line 54, the method of injecting guided images into the model should be more than just cross-attention. Would TPC work if the human image animation methods do not use cross-attention to guide image injection into the model?\n3.\tThe last paragraph of the introduction introduces the proposed TPC approach. However, this part is to analyze the gains that TPC can bring, and I think this part is more suitable for methods or experiments.\n4.\tAs for the TED man in the demo of the experimental results, why does the animated video only show the man's upper body?\n5.\tFor the unseen domain shown in the provided demo, I think in human image animation, we'll give the model images that need to be animated and guided images. Also, the base model is trained on a lot of images of human bodies, and the model should have seen something like this before, so calling it an unseen domain isn't accurate. For this kind of scene, there is a big gap between the character identity in the generated video and the given image, and TPC seems to have no way to help the original model maintain the character identity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We address your concerns below.\n\n[**Q1**] I think the most direct and useful way is to find a way to align the two images, and I don't think it is very difficult to align the two images. So why do we need the proposed TPC method?\n\n[**A1**] The direct way to input an aligned reference image is sub-optimal, as it fails in maintaining temporal consistency and preserving the background. Figure J of our attached PDF shows the experimental results for this. For right-to-left walking motions, the single reference image cannot aligned with both target positions, resulting in temporal inaccuracies, and at the same time, the background is not preserved. \n\nThe TPC method addresses these issues by performing frame-wise calibrations to match all target frames and providing these as additional conditions for the diffusion model. To preserve the background, TPC filters out backgrounds in the calibrated frames (Line 214). The Procrustes approach used in TPC is more effective for calibration than other transformations, as demonstrated in Figure G and Table C, which show qualitative and quantitative superiority over methods like Affine [1,2], Perspective [3,4], and Global Pose Normalization (GPN) [5].\n\n**Table C.** Ablation study of various transformation methods for calibration, reported in a format of foreground/background (validation splits, average score of compositional alignment/misalignment). PS: Perspective.\n|||||||||\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n||L1↓$_{\\times \\textrm{E-04}}$|PSNR↑|SSIM↑|LPIPS↓|FID↓|FID-VID↓|FVD↓|\n|w/o calibration|4.51/3.57|27.81/29.15|0.689/0.758|0.304/0.259|38.3/27.7|37.2/29.4|184/162|\n| Affine|3.92/3.25|28.61/29.32|0.709/0.767|0.281/0.249|29.1/26.1|36.7/27.6|177/153|\n|PS|3.46/3.06|28.88/29.37|0.715/0.769|0.279/0.247|28.6/25.9|35.2/27.4|171/151|\n|GPN|3.12/2.83|29.06/29.48|0.731/0.776|0.274/0.244|27.3/25.6|31.5/26.9|164/145|\n|TPC (Ours)|**3.08**/**2.79**|**29.12**/**29.50**|**0.734**/**0.782**|**0.273**/**0.241**|**26.9**/**25.1**|**31.2**/**26.4**|**162**/**142**|\n\n[1] FedMed-ATL: Misaligned Unpaired Cross-Modality Neuroimage Synthesis via Affine Transform Loss, ACM MM'22\n\n[2] Adaptive Affine Transformation: A Simple and Effective Operation for Spatial Misaligned Image Generation, ACM MM'22\n\n[3] Shape-Preserving Half-Projective Warps for Image Stitching, CVPR'14\n\n[4] Image stitching with perspective-preserving warping, ISPRS'16\n\n[5] Everybody Dance Now, ICCV'19\n\n[**Q2**] In line 54, the method of injecting guided images into the model should be more than just cross-attention. Would TPC work if the human image animation methods do not use cross-attention to guide image injection into the model?\n\n[**A2**] Yes, it works. Figure K demonstrates that even when using feature addition as a simple injection method, it is effective, though not as effective as the cross-attention method. Especially, the addition method is effective when the shapes are positionally aligned between the calibrated frame and the target frame. Furthermore, Table D demonstrates the quantitative effectiveness of these.\n\n**Table D.** Ablation study of different injection methods for applying TPC, reported in a format of foreground/background (validation splits, average score of compositional alignment/misalignment).\n|||||||||\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n||L1↓$_{\\times \\textrm{E-04}}$|PSNR↑|SSIM↑|LPIPS↓|FID↓|FID-VID↓|FVD↓|\n|w/o TPC|4.51/3.57|27.81/29.15|0.689/0.758|0.304/0.259|38.3/27.7|37.2/29.4|184/162|\n|w/ TPC (addition)|3.16/2.92|29.01/29.37|0.727/0.771|0.283/0.249|28.2/25.9|32.6/26.9|171/152|\n|w/ TPC  (cross-attention) |3.08/2.79|29.12/29.50|0.734/0.782|0.273/0.241|26.9/25.1|31.2/26.4|162/142|\n\n[**Q3**] The last paragraph of the introduction is more suitable for methods or experiments.\n\n[**A3**] Yes, we will move it to the experiment section.\n\n[**Q4**] As for the TED man in the demo (Figure 8) of the experimental results, why does the animated video only show the man's upper body?\n\n[**A4**] This is because the input target motion only includes the upper body. The skeleton's green and blue branches, connected to the pelvis, are part of the upper body. For reference, please see the full-body skeletons in the upper sample of the woman walking.\n\n[**Q5**] For the unseen domain shown in the provided demo, I think in human image animation, we'll give the model images that need to be animated and guided images. Also, the base model is trained on a lot of images of human bodies, and the model should have seen something like this before, so calling it an unseen domain isn't accurate. For this kind of scene, there is a big gap between the character identity in the generated video and the given image, and TPC seems to have no way to help the original model maintain the character identity.\n\n[**A5**] The unseen domains we aim to address are samples containing compositional misalignment (i.e., samples of human shape misalignments in terms of scale and rotation between the reference and target images). Human image animation datasets (e.g., TikTok, TED-talks) ensure this alignment, but real-world scenarios often involve compositional misalignments, making current systems vulnerable to these samples. To address this problem, we introduce a calibration branch for the diffusion model, demonstrating its effectiveness at test-time without additional training. To quantitatively validate our approach, we have collected an additional test set specifically for compositional misalignment samples, alongside the original test set (as mentioned in Line 221, with source links available in our supplementary material).\n\nWe acknowledge the redundant use of the term \"unseen\" in Figure 10 and the demo. This term actually refers to the reference image obtained from the Text-to-Image (T2I) model. We will update the terminology to \"T2I-synthesized\" to avoid confusion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We have uploaded a PDF file containing figures. Please refer to this PDF along with our rebuttal for a clear understanding.

Meta-review: This paper was reviewed by four experts in the field. The initial reviews were mixed, as reviewers are concerned about the validation of the experiment. The main concern was: 1) the novelty of the proposed approach, as alignment is a common solution in 2D-3D human pose estimation, 2) the motivation of Procrustes Calibration is not clear, as a pre-calibration is also a possible alternative solution.

After discussion, the second concern is mostly resolved and reviewers remain concerned about the novelty of this work. Still, all reviewers agreed that this is well-presented work, the idea is simple and effective, and the performance is good, even though it is not a fully novel approach. Therefore, the area chair would suggest acceptance.

We recommended the authors to carefully read all reviewers’ final feedback, and revise the manuscript as suggested in the final camera-ready version. We congratulate the authors on the acceptance of their paper!
Decision: Accept (poster)