Large Displacement Motion Transfer with Unsupervised Anytime Interpolation
Authors: Guixiang Wang
Jianjun Li
Venue: ICML 2025 poster (2025)
Keywords: Image animation
action generation
Unsupervised Interpolation
Primary area: applications->computer_vision

Abstract: Motion transfer is to transfer pose in driving video to object of source image, so that object of source image moves. Although great progress has been made recently in unsupervised motion transfer, many unsupervised methods still struggle to accurately model large displacement motions when large motion differences occur between source and driving images. To solve the problem, we propose an unsupervised anytime interpolation based large displacement motion transfer method, which can generate a series of anytime interpolated images between source and driving images. By decomposing large displacement motion into many small displacement motions, difficulty of large displacement motion estimation is reduced. In the process, we design a selector that can select optimal interpolated image from generated interpolated images for downstream tasks. Since there are no real images as labels in the interpolation process, we propose a bidirectional training strategy. Some constraints are added to optimal interpolated image to generate a reasonable interpolated image. To encourage network to generate high-quality images, a pre-trained Vision Transformer model is used to design constraint losses. Finally, experiments show that compared with the large displacement motion between source and driving images, small displacement motion between interpolated and driving images is easier to realize motion transfer. Compared with existing state-of-art methods, our method has significant improvements in motion-related metrics.

Large Displacement Motion Transfer with Unsupervised Anytime Interpolation
Guixiang Wang'
Jianjun Li?
Abstract
1. Introduction
In recent years, motion transfer task has become one of the research hotspots in computer vision because it has been widely used in film animation, game production, face exchange and other ficlds(Siarohin et al., 2019a; Oquab et al., 2021; Wang et al., 2021; Zakharov et al., 2020). In the work, we aim to animate an object in a still image by transferring the pose in the driving video to generate a video with the same pose as the driving video. To make the generated video more vivid, it is necessary not only to accurately transfer motion patterns in driving video but also to ensure that the identity information of the generated video frame and source image are consistent.
Motion transfer is to transfer pose in driving video to the object of the source image so that the ob- ject of the source image moves. Although great progress has been made recently in unsupervised motion transfer, many unsupervised methods still struggle to accurately model large displacement motions when large motion differences occur be- tween source and driving images. To solve the problem, we propose an unsupervised anytime interpolation- large displacement motion transfer method, which can generate series of any time interpolated images between source and driving images. By decomposing large displace- ment motion into many small displacement mo- tions, the difficulty of large displacement mo- tion estimation is reduced. In the process, we design selector to select optimal interpolated images from generated interpolated images for real images
At present, motion transfer tasks can be divided into su- pervised and unsupervised methods. Supervised methods perform motion transfer by using prior knowledge of target object, such as landmarks(Chan et al., 2019; Ha et al., 2020; Ren et al., 2020; Siarohin al., 2018; Zakharov et al., 2019) and 3D models(Doukas et al., 2021; Liu al., 2019; Thies et al., 2020; Blanz & Vetter, 1999). However, these methods usually rely on pre-trained models to extract object-specific representations, and often fail to obtain satisfactory results for data objects that do not appear in training. Recently, re- searchers have proposed some unsupervised motion transfer methods that do not require prior knowledge. These meth- ods use different transformation methods to model motion to improve the accuracy of motion transfer. For example, both FSMM(Siarohin et al., 2019b) and MRAAoSiarohin et al., 2021) methods used a local linear affine transfor- mation to model motion. However, in the real world, the motion of objects is usually not locally linear, which makes it difficult for affine transformation to represent complex motions accurately. To this end, TPSMM(Zhao & Zhang, 2022) introduced more flexible nonlinear transformation (Thin-plate splines) to approximate motions. However, the set of keypoints detected by TPSMM is messy, and accuracy is not high, which limits the motion representation ability of thin-plate spline transformation. To solve the problem, CPABMM(Wang et al., 2024) utilized continuous piece- wise affine transformation to model motion. However, these motion models are limited in their ability to simulate finer motions and often produce artifacts around local regions. To this end, Tao et al. et al., 2023) proposed a motion re- finement module to compensate for previous motion models,
downstream tasks. Since there as labels in the interpolation process, we propose a bidirectional training strategy. Some constraints are added to the optimal interpolated image to generate reasonable interpolated image. To en- courage the network to create guggh-ualtty images, pre-trained Vision Transformer model is used to design constraint losses. Finally, experiments show that compared with the large displacement motion between source and driving images, the small displacement motion between interpolated and driving images makes it easier to realize mo- tion transfer. Compared with existing state-f--tee- art methods, our method significantly improves motion-related metrics.
'School of Computer Science and Engineering, Hangzhou Dianzi University, Zhejiang, China "Scoool of Information Science and Technology, Hangzhou Normal University, Zhejjang, China Correspondence to: Jianjun Li @
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Large Displacement Motion Transfer with Unsupervised Anytime Interpolation
achieving finer motion modeling in local regions.
constraint losses encourage the network to generate images with good quality.
Although these methods focusing accurate motion esti- mation have achieved good performance, it is often difficult to obtain accurate animation results when the motion dif- ference between source and driving images is large. It is well known that the motion difference between two im- ages is smaller, then the pose of the driving image more straightforward to transfer accurately. In the case of large displacement motions, previous methods attempt to model complex large displacement motions directly, which will be a serious challenge. To solve the problem of large dis- placement motion, CoP(Fu et al., 2023) proposed a method based on chain-of- poose. However, in the real world, when the source and driving images have different identity infor- mation, it is difficult to obtain a segment of the pose chain between the source and driving images. In some datasets, there may also be large displacement motion between two adjacent frames in chain-of-pose. To solve the above problems, novel large displacement motion transfer model based on unsupervised anytime interpolation is proposed.
Our main contributions can be summarized as follows:
To solve the problem of large displacement motion, we propose a keypoint-hased anytime interpolation method, which can realize unsupervised interpolation between source and driving images at any time and decompose complex large displacement motion into many small displacement motions.
A motion transfer method is proposed based on the optimal interpolated image. We design a selector for optimal interpolation, which requires that the optimal interpolated image not only preserves the identity in- formation of the source image well but also has small motion to the drive image. Compared with large dis- placement motion from the source image to the drive image, small displacement motion from the optimal interpolated image to the driving image is easier to estimate so that the generated target image can learn a more accurate pose.
Specifically, by inserting a series of intermediate images between the source and the driving images, complex large displacement motion is decomposed into many small dis- placement motions to improve the performance of the mo- tion transfer model in the case of large displacement motion. As shown in Figure 1, we propose an unsupervised keypoint- based interpolation method to realize interpolation between source and driving images at any time. Firstly,, the keypoints of source and driving images are obtained, and the keypoint information of interpolated images at different times is esti- mated according to the keypoints of the two images. Then, a dense motion network is used to predict motion flow from the source image to interpolated images at different times. An optimal interpolation selector is designed to obtain a bet- ter transfer effect for the downstream motion transfer task. The selector requires that the optimal interpolated image should meet the following two requirements: 1) maintain the integrity of identity information; 2) The motion between interpolated and driving images should be small. Since an optimal interpolated image is generated unsupervised, a bidirectional training strategy is proposed to complete the interpolation without paired data, and reasonable constraints are designed. Compared with large displacement motion estimation from source image to driving image, small dis- placement motion estimation from interpolated image to driving image will be more accurate. Finally, we design structural consistency loss using a pre-trained Vision Trans- former (VIT) model(Caron et al., 2021; Tumanyan et al., 2022) as an external semantic before extract structural infor- mation from both generated target image and driving image. The appearance consistency loss is designed by extracting appearance information from pairwise optimal interpolated images generated during the bidirectional training. These
A bidirectional training strategy is proposed, which uses pre-trained VIT model to construct constraint terms, such as structural consistency loss and appear- ance consistency loss, to reduce the solution space of interpolated images and improve the quality of gener- ated images.
The experiments and analysis are carried out on three datasets. Compared with other ofate-of-the-art motion transfer models, the proposed method can achieve the best performance in motion-related metrics.
2. Related Work
2.1. Supervised Motion Transfer Methods
Supervised motion transfer methods are designed to handle specifie objects (e.g., faces, bodies, etc.) that exhibit strong regularity in shape and structure. The video can be gener- ated more stably by introducing the corresponding struc- tural prior as an intermediate representation. For example, HeadG ANDouuass et al., 2021) used 3DMM & Vet- ter, 1999) as intermediate representation and SPaDEs(Park et al., 2019) to fuse identity features of source image, and finally renders 3D image back animation frame. GANi- mation et al., 2018) used Facial Action Coding System(Ekma) & Rosenberg, 1997) to describe facial ex- pressions. With the help of these structural priors, super- vised motion transfer methods can generate videos more accurately. However, these methods must rely on explicit structural representations, annotated or extracted by pre-
2

---

Large Displacement Motion Transfer with Unsupervised Anytime Interpolation
D
Kps
Keypoint
Detector
1
11,
(Kp Te
Motion
Interpolation Network
Transfer
KPp
(1)())
Network
Dense
E()
Source
Expansion
Network
Occlusion
r,o
GO
Generted torget image
B-) Fnender
Generated Image
Kp, Ky..KPo-AA 1Pp.l Keypoints
GC):
U. 1....... Interpoaaionn
Deformed C'eture
Figure 1. Overview of our model. The model consists of two modules. The first module is arbitrary interpolation method based on keypoints. At Eq. (1), we visually describe the motion of keypoints of interpolated images. The blue circles represent the corresponding keypoints time circles represent keypoints of the interpolated image at time & under linear motion assumption. After obtaining series of anytime keypoints, input into the interpolation network generate the corresponding interpolated images. The second module is the motion transfer method based on the optimal interpolated image. To improve the motion transfer effect, we designed selector to select an optimal interpolated image from series interpolated images for motion transfer. These two modules use the same network architecture with shared woights, where f is the result of deformed features of the input image.
(source image) and ' (driving image). the red circles represent true keypoints time f. and the orange
trained models, to perform motion transfer for specific ob- jects. Due to the limitations of these structural priors, these supervised methods do not generalize well to new objects.
improve keypoint detector. Although these methods have achieved good transfer performance, they are often unsatis- factory in the case of large displacement motion. Therefore, we propose a large displacement motion transfer method based on unsupervised anytime interpolation to improve motion estimation accuracy by decomposing large displace- ment motion into multiple small displacement motions.
2.2. Unsupervised Motion Transfer Methods
In recent years, many unsupervised motion transfer methods have been proposed. MoSiay-Ne et al., 2019a) is a general object image animation framework, which consists of a keypoint detector, a motion prediction network, and an image generation network. Based on the assumption of local linear motion around each keypoint, FOMM(Siarohin al., 2019b) used local affine transformation to model mo- tion. Zhao et al. et al., 2021) transferred motion from sparse landmarks to face images and combined global and local motion estimation into a unified model that can pro- duce not only global motion but also subtle local motion. MRAA(Siarohin al., 2021) used an inference algorithm based PCA(Kurita, 2014) to calculate affine parame- ters, which improves stability and performs better for joint movements. TPSMM(Zhao & Zhang, 2022) predicted sev- eral sets of keypoints from source and driving images and used each set of keypoint sets to compute a thin-plate spline (TPS) transformation to increase the flexibility of the motion model. CPABMM((Wnng et al., 2024) used the inference method of gradient descent estimate transformation based continuous piecewise affine (CPA), and to overcome the problem of chaotic keypoint prediction in TPSMM, key- point semantic loss based on SAM model is designed to
3. Methods
3.1. Motivation
Given a source image S and a driving video D = d with n frames, our goal is to have the object of the source image imitate motion in the driving video generate realistic video. However, when the pose difference between the source image and some frames of driving video is large, it is often difficult to accurately estimate the motion of two images, resulting in high-quality video that cannot be gen- erated. To solve this problem, we propose motion transfer model based on unsupervised anytime interpolation. Our model consists of two modules: keypoint- -based anytime interpolation and motion transfer based on the optimal inter- polated image. First, predict interpolated images for moments unsupervised for given source image and driving images. A selector is then set to select the most suitable image from m interpolated images for the downstream mo- tion transfer task. We use the optimal interpolated image to replace the source image and realize motion transfer from
3

---

Large Displacement Motion Transfer with Unsupervised Anytime Interpolation
the interpolated image to the driving image. We use the same network architecture for these two modules: a key- point detection network, dense motion network, and an image generation network. An optimal interpolated image and final target image are generated end-to-end. Compared with traditional motion transfer methods, the target image generated by the proposed method has a more accurate pose.
motion from S D, but they often cannot obtain satis- factory results in the case of large displacement motion.
Therefore, we propose a motion transfer method based on optimally interpolated images. Because keypoints of the interpolated image are obtained through keypoints of the source image and keypoints of the driving image, it makes the pose of the interpolated image be in the state between the source and driving images. Then, compared with large displacement motion estimation ---> from S to D, small displacement motion estimation - D from 4, to D will predict more accurately.
3.2. Large displacement motion transfer with unsupervised anytime interpolation
Keypoint-based anytime interpolation method. In FOMM(Siarohin et al., 2019b), it is assumed that motion consists of series of linear transformations. Inspired by it, a keypoint-based anytime interpolation method is pro- posed. In Figure 1, we visually describe the motion of keypoints of interpolated image, where when T = 0, it can be represented as keypoints (Kps) of source image, and when T = 1, it can be represented as keypoints (Kpp) of driving image. In the unsupervised case, prior motion information of each keypoint at time E 10, cannot be obtained. Therefore, we assume that the motion of each keypoint is linear, then anytime keypoints within / E 1) can be expressed as follows:
Then we use I, image to replace S for motion transfer af- ter obtaining interpolated image It. However, the quality of I image will seriously affect the motion transfer's per- formance. Therefore, we design a selector for optimal in- terpolated image, which selects image from a series of generated interpolated images that simultaneously satisfies the following requirements: 1) the identity information of the interpolated image should be intact; 2) The motion of the interpolated image to driving image should be as small as possible. The selector set by the following:
[I((1)) - A(S)
(1) Taking the interpolated image at time as an example, here we apply the TPS transformation to estimate the motion from S to I such that T(S)= = It. Like TPSMM(Zhao & Zhang, 2022), we get K + 1 transformations (K TPS trans- formations and an affine transformation), where affine trans- formation is used to model the motion of the background. Then, we feed S, Kps, and Kp'j into dense motion network to predict K + 1 contribution maps { which are combined with K + transformations to approximate the motion I:
Kp} = (1- - t)Kps + tKpD, E (0
Lapt arg min(
174 [|(TT)) - A(S)
(4)
|Kp] -Kppl
Σ 1=1 [Ky] KPoll
where topt represents the index of optimal interpolated im- age, and A(-) represents appearance features of image ex- tracted by pre-trained Vision Transformer (ViT) model, the introduction of VIT is shown in subsection 3.4. We cal- culate the appearance loss of all interpolated images and source images, and if the appearance information of the interpolated image is kept more complete, then the selection weight is larger. Morcover, distance loss from keypoints of all interpolated images to keypoints of the driving image is calculated. if the motion from the interpolated image the driving image is small displacement, then the selection weight is larger. The appearance loss and keypoint distance loss are normalized, and the optimal interpolated image is selected from interpolated images with the minimum sum of the loss, that is, the interpolated image with appearance information intact and small displacement from the driving image. After obtaining an optimal interpolated image, we use motion transfer from the optimal interpolated image to the driving image (Interpolation to Driving) to replace the traditional motion transfer (Source to Driving). Compared with the traditional methods, the pose of the transferred im- ages obtained by the proposed method is more accurate. To learn high-quality optimal interpolated images, we propose a bidirectional training strategy.
[p)= (p)
Σ Ma(p)(a(P))
(2)
where L'og (p) is prediction of background motion and (p) is prediction of target motion by K TPS transformations. After obtaining motion flow I from S to It, we use I to deform& S and feature map fs of S, and then imput them into the image generation network to obtain interpolated image I at time t:
7 = G(S )
(3)
where G(-) Denote as image generation network, S is im- age after deformation of S, and J is feature after deforma- tion of fs. Similarly, we can obtain a series of interpolated images {4)), where m is the number of interpolations.
Motion transfer based on optimal interpolated image. Traditional motion transfer methods usually directly model
4

---

Large Displacement Motion Transfer with Unsupervised Anytime Interpolation
Egt3) Lewx
lation and motion transfer models utilize the shared-weight image generation network. If the image generation network in the motion transfer model exhibits strong generative ca- pabilities, it can significantly enhance the quality of the optimal interpolated image generated.
Input: S. Kp, Kp
1.. Kpr Kpo
Outguts
Anytime
interpolateal Model
3.4. Vision Transformers
In the motion transfer task, we aim to keep the source im- age's identity information and learn the driving image's pose. To make the generated image meet this requirement, we use pre-trained Vision Transformers(Vil) model(Caron et al., 2021: Tumanyan et al., 2022) to represent appearance information and structure information of image in the ViT feature space. With respect to appearance, we employ [CLS] to indicate global image representation aimed at captur- ing both the informational content and stylistic attributes of global appearance. Regarding structure, the utilization of depth space features extracted by ViT, along with their self- similarity as structural representation, not only facilitates a robust representation of local texture but also ensures the preservation of the spatial layout and shape of the object and its surroundings.
Impot: D. Kp Kp"
Kel Kps
Outpur:
(8) Low
Figure Bidirectional training strategy. worth noting that the interpolation module and the motion transfer module in both pipelines use the same optimal interpolation keypoints, as shown in red boxes.
3.3. Bidirectional Training Strategy
As shown in Figure 1, the pipeline is designed to generate an optimal interpolated image to decompose large displace- ment motion into small displacement motion. We can pro- vide supervision at the end the pipeline by extracting two frames from the video as input. However, only such su- pervision cannot generate satisfactory optimal interpolated images. Without monitoring the intermediate results, all sub- nets are treated task. In our framework, if there is no paired data to super- vise, we should design some constraints to guarantee that the optimal interpolated image is meaningful. Otherwise, as long as the downstream motion transfer model can interpret the interpolated image, then the generator of the interpola- tion model can generate arbitrary interpolation results. If an effective constraint is found, the solution space can be reduced and a meaningful solution with the properties em- phasized by the constraint can be obtained. To ensure the effectiveness of optimal interpolated images, we designed bidirectional training strategy that does not require paired data.
In the bidirectional training strategy, we make use of the abil- ity of ViT to extract appearance representation and structure representation and add appearance consistency and struc- ture consistency constraints generate optimal interpolated images (Irope and Itoge) ) to reduce their solution space. In addition, structure consistency constraint is added to the generated target image and driving image to promote the network and generate a better quality target image.
whole to complete the motion migration
3.5. Loss Functions
Reconstruction loss Lyee : To make the generated images D and Š more realistic, we use the pre-trained VGG-19 network(Johnson et al., 2016) to calculate the perceptual loss of multi-resolltion images:
Lree = ΣΣ vi(D,) -v,))))((D)))
(5)
As shown in Figure 2, the bidirectional training strategy consists of two pipelines: the upper pipeline is the motion transfer from S to D. and the lower pipeline is the motion transfer from to S. In this process, the two pipelines generate optimal interpolation images Itoge and Ie respec- tively by giving the same optimal interpolation keypoints Kpper In theory, given the same interpolation keypoints, source and driving images with the same identity often obtain the same interpolation image by interpolation model. Therefore, we add constraints to the optimal interpolated images < y > such that they are consistent.
+ ΣΣ> Σ - J
where Vú is the ith layer of the pre-trained VGG-19 network, and indicates that the image has been downsampled times.
Feature consistency loss Lfea : The encoder in Figure used to extract multi-scale features of optimal interpolated images (L and I,,,.) in the bidirectional training process. The Tefaa loss encourages optimal interpolated images be consistent. The loss is expressed as:
Our experiments demonstrated that the image generation network is crucial. Within our framework, both the interpo-
Lsea Σ Ei(Ttope) Ei(Tenpe)
(6)
5

---

Large Displacement Motion Transfer with Unsupervised Anytime Interpolation
Tai-Chi-HD
Fashion
UvA-Nemo
TedTalks
T
(AKD, MKR)
AED
L
(AKD, MKR) AED
T
AKD
AED
L
(AKD, MKR)
AED
X2Face
0.080
(17.65, 0.109)
0.270
0.031
3.539 0.221
FOMM
0.057
(6.65, 0.036)
0.172
0.013 (1.131, 0.006)
0.059
0.021
1.408
0.067
0.033
(7.07, 0.014)
0.163
MRAA
0.048
(5.41, 0.025)
0.149
0.017
1.323 0.060
0.026
(3.75, 0.007)
0.114
DAM
0.044
(4.79, 0.021)
0.146
0.011
(1.041, 0.004)
0.054
MTIA
0.045
(4.67, 0.021)
0.148
0.026
(3.46, 0.007)
0.113
TPSMM
0.045
(4.57, 0.018)
0.151
0.011
(0.845, 0.005) 0.056
0.011
1.177 0.050
0.027
(3.39, 0.007)
0.124
CPABMM
0.041
(4.61, 0.021)
0.117
0.022
(3.21, 0.008)
0.085
Ours
0.047
(4.21, 0.014)
0.157
0.011
(0.800, 0.004)
0.056
0.010
1.155
0.051
0.028
(3.15, 0.005)
0.136
Ours- V2
0.046
(3.67, 0.013)
0.149
0.011
(0.771, 0.004) 0.058
0.010
0.853
0.050
Table 1. Quantitative comparison of video reconstruction task on four different datasets.
Source
Driviag
POMM
MRAA
DAM
MITA
TPSMM
Olus
Ous V2
Figure 3. Some bad comparison methods cases on the Tai-Chi Chi-HDD dataset, while our method shows high quality on video reconstruction task.
Driving
FOMM
DAM
TPSMM
Source
Figure Qualitative comparison image animation task on Tai-Chi-HD (above) and Fashion (helow).
6

---

Large Displacement Motion Transfer with Unsupervised Anytime Interpolation
Scaice
T:033
T
Tol?
Diiing
Tuls
Tul
Diviig
Figure 5. Interpolation results generated different moments der the same identity.
Figure 6, Interpolation results generated at different moments un- der the different identity.
where E, represents layer ith of encoder.
Metries: Same as previous work, in the video reconstruc- tion task with the same identity, the first frame D1 of video is used as source image to reconstruct {Da) The quantitative indicators (Siarohin et al., 2019b) used are LI, Average Keypoint Distance (AKD). Missing Keypoint Rate (MKR), and Average Euclidean Distance (AED).
Appearance consistency loss Lapp: The pre-trained ViT (Caron et al., 2021; Tumanyan et al., 2022) is used to extract the appearance features of the optimal interpo- lated images I. and T..p. in bidirectional training process respectively, denoted as to and A A(ft.p:), where A(-) represents appearance features of images extracted by the ViT model. The appearance the optimal interpolated im- age generated by Lopp loss is encouraged be consistent, and loss is expressed as:
Implementation Details: The method is implemented on Py PyTorch(Paszke et al., 2019), all experiments conducted NVIDIA 4090 GPU with resolution of 256 x 256 for all datasets and 100 epochs of training. We use Adam pptimizer(Kingma & Ba, 2015) to update our model and set the learning rate to 0.0001, which dropped by a factor of 10 at the end of the 70th epoch and the 90th epoch. We set the training hyperparameters to: Ap = 10. Ay = 10, Aa 10. and X, 10. The interpolated images are generated {0.3, 0.5, 0.7) respectively, and then optimal interpolated images that meet the conditions are selected.
Lapp- Allrou)) - A(trop)
(7)
Structural consistency loss Lisruue: The self-similarity matrix of Itage S, S, D, and D is extracted using the pre-trained ViT (Cren et al., 2021; Tumanyan al., 2022), denoated as & (Lopp)) ). 4 (.))) (SS), (S)). ((D), and Φ D)) respectively, where (b)) represents structural information of image extracted by ViT model. The structural consistency loss is calculated as:
4.2. Comparison
We compare the proposed method with the state-of-the art unsupervised motion transfer methods (X2Face Wiles et al., 2018), FOMM, MRAA, DAM(Tao et al., 2022b), MTiA(Tao et al., 2022a), TPSMM, and CPABMM) for video reconstruction and image animation tasks.
(J(t.pe)) - (T(Tepp)) 1 + ((8)))(($))
Listrue
+ (D() (T(D F
(8)
Video reconstruction:
The quantitative results of Video reconstruction shown in Table 1. Our approach achieved significant improvements in motion-related metrics on four datasets. Compared with TPSMM, on the Tai-Chi-HD dataset, AKD and MKR improved by 7.88% and 22.2%, respectively, and on the Fashion dataset, AKD and MKR increased by 5.3% and 20%, respectively. on the TedTalks dataset. Compared with CPABMM, AKD and MKR in- creased by 1.87% and 37.5%. It shows that our model can obtain pose closer to the real image.
In general, the total training loss function is:
Lioss = XpLrrec + A + AaLapp + AgLettuue
where \r, Xy, das and As are hyperparameters.
4. Experiments
4.1. Benchmarks
Datasets: We trained on multiple types of datasets, in- cluding faces and human bodies. The datasets are as fol-
As shown in Figure 3, we show qualitative results of video reconstruction, which prove that the interpolation method between source and driving images can obtain more accu- rate estimation of large displacement motion. However,
lows: UvA-Nemo(Dibeelloglu et al., 2012; 2015), Fash- ion(Zablotskaia et al., 2019),
et al.,
2019b), and (Saaoons al., 2021)
7

---

Large Displacement Motion Transfer with Unsupervised Anytime Interpolation
LI
(AKD, MKR)
performance but also decreases the model's performance. 0.151
AED
TPSMM
0.045
(4.57,0.018)
This is because the interpolation method results in poor generation of interpolated images without constraint infor- mation, which affects the downstream motion transfer task. For Model2, when we add the bidirectional training strategy, the generation of interpolated images is appropriately con- strained, improving motion transfer performance. For the Full model (Ours), we add appearance consistency loss and structure consisteney loss so that the quality of the gener- ated images is improved and optimal in the motion-related metrics.
+ +TPSMM, INT (Modell)
0.053 (5.56,0.026)
0.169
+Model1, BT (Model2)
0.048
(4.31,0.016) 0.158
+Mode12, Lapp. Latrue (Ours) 0.047 (4.21,0.014) 0.157
Table 2. Ablating the key components of the proposed method. INT is denoted as adding an unsupervised anytime interpolation method, and BT is denoted as adding bidirectional training strat- egy.
4.4. Method Extension
the interpolation method also causes a loss of appearance information. As shown in Figures 5 and 6, we show the in- terpolation at different moments and final generated results. Compared with the source image, the generated interpolated image already has the problem of blurring, which leads to the appearance effect of the target image generated by the downstream task is not satisfactory. In addition, for the Chi-HD),, the interpolation
The proposed method is extended to multi-view tasks ac- cording to the optimal interpolation method. Specifically, source image and optimally interpolated images are used to constitute multi-view data. The source image can provide better appearance information, and the motion displacement between the optimal interpolated image and the driving im- age is smaller, which can learn the pose information more accurately. Therefore, in the motion transfer network, we fuse the deformation features of the source image and the optimally interpolated image, and the experimental results are shown in Ours- V2 of Table 1. Compared with the pro- posed single-view motion transfer method (Ours), Ours-Vi V2 achieves significant improvements on most datasets. Among them, the motion-related metric AKD is increased by 12.8%. 3.6% and 26.1% on Tai- Tai-Chi-HD, Fashion and UvA-Nemo, respectively. As shown in Figures 3 and 4, the appearance of the generated images is significantly improved.
dataset with background method causes occlusion region to expand, as shown by the background in the second row of Figure 6. This is also why the L1 and AED metrics in Table are slightly lower than the existing methods.
Image Animation: Figure 4 shows the comparison results of motion transfer between the proposed method and com- parison methods under different identities. Experimental results show that the proposed method has better motion transfer performance for large displacement motion of the human body (Tai-Chi-HD and Fashion). However, the abil- ity to maintain the details of images, such as clothing and face, is slightly worse. As shown in the second row of Fig- ures 5 and 6, the clothing color of the interpolated image generated has a color difference with the source image, re- sulting in a large difference between the appearance of the image generated by downstream motion transfer and the appearance of the source image.
5. Conclusion
To solve large displacement motion, we propose a large displacement motion transfer model based on unsupervised anytime interpolation. Compared with the previous method of directly predicting the motion from the source image to the driving image, the proposed method interpolates the source image and driving image at any time, decomposes the original large displacement motion into multiple small displacement motions, and improves the accuracy of mo- tion estimation. To achieve unsupervised interpolation, we propose a bidirectional training strategy, which adds con- straints to the intermediate interpolated images to narrow the generation range of interpolated images. The pre-trained Vit model adds appearance and structure consistency to the images so that the generated images have better qual- ity. Experiments show that our method achieves optimal performance on motion-related metrics compared to other state-of-art methods.
4.3. Ablations
In this section, we conduct comprehensive ablation experi- ments the Tai-Chi-HD dataset to systematically evaluate the contribution of key components to the final performance of the method. We successively added our proposed com- ponents; the results are shown in Table 2. The second row of Table represents the results of the TPSMM method, Modell represents the addition of an unsupervised anytime interpolation method to the TPSMM method, Model2 is the addition of a bidirectional training strategy based on Model1, and Full model is our proposed model.
For the experimental results of Model1, we find that adding an unsupervised anytime interpolation method to the TPSMM method not only does not improve the model's
8

---

Large Displacement Motion Transfer with Unsupervised Anytime Interpolation
Acknowledgements
unseen targets. In In Proceedings of the Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI, pp. 10893-10900, 2020.
This work was partly supported by the Key Research and De- velopment Plan of Zhejiang: No. No.2021C0313 National Nat- ural Science Fund of China 66.62471170 & No.61871170; Zhejiang Province Fund: LQ21F020005.
Johnson, J., Alahi, A., and Fei-Fei, L. Perceptual losses for real-time style transfer and super-resolution. In In Proceedings of the Computer Vision ECCV 2016 14th European Conference, volume 9906, pp- 694-69911711 114-711, 2016.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In In Proceedings of the 3rd International Conference on Learning Representations, ICLR, 2015.
Kurita, T. Principal component analysis (PCA). In Com- puter Vision, A Reference Guide, pp. 636-639. 2014.
References
Liu, W. Piao, Z., Min, J., Luo, W., Ma, L., and Gao, S. Liquid warping GAN: unified framework for human motion imitation, appearance transfer and novel view synthesis. In In Proceedings of the 2019 IEEF/CVF International Conference on Computer Vision, ICCV, pp. 5903-5912, 2019.
Blanz, V. and Vetter, T. A morphable model for the synthesis of 3d faces. In In Proceedings of the 26th Annual Confer- ence on Computer Graphics and Interactive Techniques, SIGGRAPH, pp. 187-194, 1999.
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. In In Proceedings of the 2021 IEEE/CVF International Conference on Com- puter Vision, ICCV, pp. 9630-9640, 2021.
Oquab, M., Stock, P., Haziza, D., Xu, T., Zhang, P., Celebi, O., Hasson, Y., Labatut. P., Bose-Kolanu, B., Peyronel, T., and Couprie, C. Low bandwidth video-chat com- pression using deep generative models. In In Proceed- ings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, CVPR Workshops, pp. 2388 2388-2397, 2021.
Chan, C., Ginosar, S., Zhou, T., and Efros, A. A. Everybody dance now. In In Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision, ICCV, pp- 5932-5941, 2019.
Park, T., Liu, M., Wang, T., and Zhu, Semantic image synthesis with addptive normalization. In In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 2337-2346, 2019.
Dibeklioglu, H., Salah, A. A., and Gevers, T. Are you really smiling at me? spontancous versus posed enjoyment smiles. In In Proceedings of the Computer Vision ECCV 2012 12th European Conference on Computer Vision volume 7574, pp. 525-538, 2012.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang. E. Z., Vito, Z., Rai- son, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, high-performance deep learning library. In In Proceed- ings of the Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS, pp. 0024-8035, 2019.
Dibeklioglu, H., Salah, A. A., and Gevers, T. Recognition of genuine smiles. IEEE Trans. Multim., 17(3):279-294 2015.
Doukas, M. C., Zafeiriou, S., and Sharmanska, V. Headgan: One-shot neural head synthesis and editing. In In Pro- ceedings of the 2021 IEEE/CVF International Conference on Computer Vision, ICCV, pp. 143877 2021.
Pumarola, A., Agudo, A., Martínez, A. M., Sanfeliu, A., and Moreno-Noguer, F. Ganimation: Anatomically facial animation from a single image. In In Proceedings of the Computer Vision ECCV 2018 15th European Conference, volume 11214, pp. $35-851. 2018.
Ekman, P. and Rosenberg. E. L. What the face reveals: Basic and applied studies of spontaneous expression using the Facial Action Coding System (FACS). 1997.
Fu, X., Wang, X., Liu, J., Wang, S., Dai, J., and Han, J. Cop: Chain-of-pose for image animation in large pose changes. In In Proceedings of the s1 ACM International Conference on Multimedia, MM, pp. 6969-6999999 (609966977200 2023.
Ren, Y., Yu, X., Chen, J., Li, T. H., and Li, G. Deep image spatial transformation for person image generation. In In Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp. 7687-7696, 2020
Ha, S., Kersner, M., Kim, B., Seo, S., and Kim, D. Mari- onette: Few-shot face reenactment preserving identity of

---

Large Displacement Motion Transfer with Unsupervised Anytime Interpolation
Siarohin, A., Sangineto, E., Lathuilière, S., and Sebe, N. Deformable gans for pose-based human image genera- tion. In In Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR, Pp. 3408-3416, 2018.
Conference on Artificial Intelligence, AAAI, pp. 5427- 5435, 2024.
Wang. T., Mallya, A., and Liu, M. One-shot free-view neural talking-head synthesis for video conferencing. In In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, CVPR, pp. 10039-10049, 2021.
Siarohin, A., Lathuilière. S., Tulyakov, S.. Ricci, E., and Sebe, N. Animating arbitrary objects via deep motion transfer. In In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, CVPR, pp. 2377-2386, 2019a.
Wiles, O., Koepke, A. S., and Zisserman, A. X2face: network for controlling face generation using images, audio, and pose codes. In In Proceedings of the Computer Vision ECCV 2018 15th European Conference, volume 11217, pp. 690-706, 2018.
Siarohin, A., Lathuilière, S., Tulyakov, S., Ricci, E., and Sebe, N. First order motion model for image animation. In In Proceedings of the Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS, pp. 7135- 7145, 2019b.
Zablotskaia, P., Siarohin, A.,. Zhao, B., and Sigal, L. Dwnet: Dense warp-based network for pose-guided human video generation. In In Proceedings of the 30th British Machine Vision Conference 2019, BMVC, pp. 51, 2019.
Siarohin, A., Woodford, O. J., Ren, J., Chai, M., and Tulyakov, S. Motion representations for articulated an- imation. In In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, CVPR, pp. 13653-13662, 2021.
Zakharov, E., Shysheya, A.. Burkov, E., and Lempitsky, S.
Few-shot adversarial learning of realistic neural talking head models. In In Proceedings of the 2019 IEEE/CVF International Conference Computer Vision, ICCV, pp. 9458-9467, 2019.
Tao, J., Wang, B., Ge. T., Jiang, Y., Li, W., and Duan, L. Motion transformer for unsupervised image animation. In In Proceedings of the Computer Vision ECCV 2022 17th European Conference, volume 13676, pp. 702-719, 2022a.
Zakharov, E., Ivakhnenko, A., Shysheya, A., and Lempitsky, V. S. Fast bi-layer neural synthesis of one-shot realistic head avatars. In In Proceedings of the Computer Vision ECCV 2020 16th European Conference, volume 12357, pp. 540, 2020.
Tao, J., Wang. B., Xu, B., Ge, T., Jiang, Y., Li, W., and Duan, L. Structure-aware motion transfer with deformable an- chor model. In In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, CVPR, pp. 3627-3636, 2022b.
Zhao, J. and Zhang, H. Thin-plate spline motion model for image animation. In In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp. 3647-3656, 2022.
Zhao, R., Wu, T., and Guo, G. Sparse to dense motion transfer for face image animation. In In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, ICCVW, pp. 1991-2000, 2021.
Tao, J., Gu, S., Li, W., and Duan, L. Learning motion re- finement for unsupervised face animation. In In Proceed- ings of the Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS, 2023.
Thies, J., Zollhöfer, M., Stamminger, M., Theobalt, C., and Nießner, M. Face2face: Real-time face capture and reenactment of RGB videos. CoRR, abs/2007.1480s 2020.
Tumanyan, N., Bar-Tal, O., Bagon, S., and Dekel, T. Splic- ing vit features for semantic appearance transfer. In In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp. 10738-10747, 2022.
Wang, H., Liu, F., Zhou, Q., Yi, R., Tan, X., and Ma, L. Con- tinuous piecewise affine based motion model for image animation. In Proceedings of the Thirty-Fight AAAI
10


Reviews:
[{"review_id": "LAsPbjIQpn", "reviewer": "Reviewer_LsVD", "summary": "This paper proposes a novel method for transferring large motion from a driving to a source image. The proposed method is to find a middle step which essentially adds non linearity to the motion transfer. The method generates a set of interpolated in between images based on key point transfers and selects the optimal interpolation point. Then transfers the motion from the interpolated image to the driving image, in effect takes a shorter final step. In order to constrain the optimal interpolations the method adopts a bidirectional training scheme where both the (source -> driving) and (driving -> source) is considered and the hypothesis is to have the same optimal interpolation point between the two directions.\nThe results show improvements in terms of keypoint accuracy.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments and suggestions.\n\nQ1: The experiments are missing results on TedTalks and VoxCeleb.\n\nA1: The unsupervised optimal interpolation method proposed in this paper aims to address the large motion problem in motion migration. To validate the effectiveness of this method, we selected two datasets—TaiChiHD and Fashion—that exhibit significant motion amplitudes. \n\nWhile the face dataset typically demonstrates a smaller range of motion compared to larger human datasets, we included the UvA-Nemo dataset to assess the method’s performance in scenarios with small motion amplitudes. Although VoxCeleb is another face dataset, its size—approximately 308 GB—restricts its usability. \n\nThe Ted Talks dataset has a resolution of 384×384, which is higher than the resolution of the datasets we used. However, the motion of the human body in the Ted Talks videos is minimal, as illustrated in https://github.com/ICML2025Anonymity/Anonymity/tree/main/TedTalks, by the double-blind policy. Due to the constraints of the rebuttal date, we only conducted experiments using the Ted Talks dataset.\n\nQ2: There is not much discussion why the L1 and AED metrics are not on par with SoTA, the quality of the generated images might be suffering from the interpolation.\n\nA2: Although the interpolated image's quality significantly impacts the final target image's quality, it helps reduce the substantial motion between the source and driving images to a smaller motion between the interpolated and driving images, significantly improving the AKD metric. \n\nTo address the issues of L1 and AED degradation, we have included enhancements to the model in the manuscript and added relevant content to subsection 4.4. Specifically, the method based on optimal interpolation provides better poses for the model, while the motion transfer technique from the source image to the driving image enhances the appearance. By combining the advantages of these two approaches, we achieved significant improvements demonstrated in Ours-V2 in Table 1, linked to https://github.com/ICML2025Anonymity/Anonymity/tree/main/Table%201. \n\nQ3: This paper does not compare with the recent paper CPABMM\n\nA3: Although CPABMM performed well in the motion migration task, the paper's authors did not provide pre-training checkpoints, and the training process is quite time-consuming. In our experimental setting, reproducing the original results was challenging. As a result, we utilized the official results. Thanks for your suggestion again. Since the rebuttal time limitation, we will try to reproduce the results when we have the time. On the other hand, our method still achieves optimal performance in the AKD metric compared to CPABMM on the TaiChiHD dataset.\n\nQ4: What is the difference between features used in the appearance loss and structure loss? Both of them seems to be extracted from the same network. Are they from different layers? Which layers?\n\nA4: Appearance loss involves comparing two images' [CLS] token. The [CLS] token serves as the global semantic descriptor in Vision Transformers (ViT), akin to the “full-image feature” found in Convolutional Neural Networks (CNNs). This token is derived from the last layer of ViT and emphasizes the overall semantics of the images.\n\nOn the other hand, structural loss focuses on comparing the self-similarity of two images using the Key matrix from the 11th layer of ViT. The self-similarity matrix generated by the Keys at this deeper layer reveals the global correlation patterns among different image regions. As such, it can be utilized to assess the structural similarity between the two images.\n\nQ5: TPSMM on taichiHD is usually 4.57 AKD, why is it higher here? It would be better to report the official AKD for TPSMM and add CPABMM to the table.\n\nA5: For TPSMM and FOMM, the experimental results presented in Table 1 are those we reproduced under the existing experimental conditions. We have updated the results to reflect the official findings and included several recent methods, such as CPABMM, in the experiments, as shown in Table 1.\n\nQ6: Page 6, 2nd paragraph, [21 - 22] seems to be a citation syntax error. Eq. 5: break so that it doesn't go over text width.\n\nA6: The manuscript has been revised, and the errors have been corrected.", "claims_and_evidence": "The claim is that the interpolation helps with large motion transfer. Overall the accuracy of the keypoints are improved  but there are no ablations on large or excessively large motion that is not usually considered in other works. If any some of the typical datasets such as Ted-talks and voxceleb are missing.", "methods_and_evaluation_criteria": "yes, the method makes sense for motion transfer and the evaluation criteria are well established.", "theoretical_claims": "no theoretical claim.", "experimental_designs_or_analyses": "The experiments are missing results on TedTalks and VoxCeleb. The current evaluations show that the bidirectional training improves the accuracy of the keypoints significantly. There is not much discussion why the L1 and AED metrics are not on par with SoTA, specially considering CPABMM it seems that the quality of the generated images might be suffering from the interpolation.", "relation_to_broader_scientific_literature": "The notion of using interpolation and incremental steps is already well established. But the idea of finding an optimal interpolation point and using bidirectional training to fix that point is novel and seems to be significantly improving the keypoint accuracy.", "essential_references_not_discussed": "none", "other_strengths_and_weaknesses": "The idea is novel and significantly improves the results in terms of keypoint accuracy. \n\nThe write up has many syntax and grammatical errors but it still reads fine. \n\nThe main weakness is not quantifying what a \"large\" motion is, selecting datasets that show \"larger\" motion than typical ones and reporting results on Ted Talks.", "other_comments_or_suggestions": "TPSMM on taichiHD is usually 4.57 AKD, why is it higher here? It would be better to report the official AKD for TPSMM and add CPABMM to the table as well. \n\nThe paper is not in review format. \n\nPage 6, 2nd paragraph, [21 - 22] seems to be a citation syntax error.\n\neq. 5: break so that it doesn't go over text width.", "questions_for_authors": "what is the difference between features used in the appearance loss and structure loss? both of them seems to be extracted from the same network. are they from different layers? which layers?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fAZ0auoJbt", "reviewer": "Reviewer_usGM", "summary": "This paper propose a unsupervised motion transfer algorithm that can transfer pose in the driving video to the object of the source image so that the source image can copy the movement of the driving video. To be exact, the method decompose complex large displacement motion into many small displacement motions, and improve the accuracy of motion estimation. A bi-directional training strategy is used to constrain the intermediate interpolated images.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuabe comments. We have incorporated your feedback into the maniscript. We believe it will help strengthen the work and present it better.\n\nQ1: The experimental results are not convincing, such as when comparing with state-of-the-art qualitatively, it only shows one state-of-the-art method (TPSMM) in figure 3, and figure 5. It did not compare with X2Face, FOMM, MRAA, I wonder why?\n\nA1: Thank you for the valuable feedback. We have provided more comparative results in the following link: https://github.com/ICML2025Anonymity/Anonymity. This is fully compliant with the double-blind policy. In response to the reviewer's question, our decision to compare the proposed method primarily with TPSMM is based on existing literature, which shows that TPSMM outperforms X2Face, FOMM, and MRAA. Therefore, we have chosen to validate the superiority of our method by comparing it only to TPSMM.\n\nQ2: X2Face is not referred. Why not compare with any more recent papers such as Wang, H., et al. Continuous piecewise-affine based motion model for image animation.\n\nA2: Thank you for your valuable comments, and the manuscript has been revised and cited more recent literature papers such as Wang, H., et al. Also,  we added comparison results with CPABMM, as the answer to Reviewer 1 LsVD's A3.\n\nQ3: Qualitative comparison seems not convincing, as shown in Figure 3-6, the proposed method's results look quite blurry, and images are of low resolution. Also since the paper claims to deal with large displacement motion transfer, I find that in the figures, a lot of examples have small displacement motion transfer, such as Figure 3 (first row), Figure 4 (second and third rows), Figure 6 (second the third rows). Why not showing more results that have large motion displacement to highlight the paper's main claim?\n\nA3: Thanks to the reviewers' insightful feedback, we show more results with large motion displacements to highlight the paper's main points in this link (https://github.com/ICML2025Anonymity/Anonymity). We modify the presentation of Figures 3-6, and the process of motion transfer is all large displacement motion, such as https://github.com/ICML2025Anonymity/Anonymity/tree/main/Figues%203-6.\n\nQ4: Why are datasets such as \"TED-talks\", \"VoxCeleb\", \"MGif\" not used when quantifying experiments?\n\nA4: Thank you for your comments, and Reviewer 1 LsVD also raised a similar question in Q1, and we addressed it in our response A1 to Reviewer LsVD. The unsupervised optimal interpolation method proposed in this paper aims to address the large motion problem in motion migration. To validate the effectiveness of this method, we selected two datasets—TaiChiHD and Fashion—that exhibit significant motion amplitudes. \n\nWhile the face dataset typically demonstrates a smaller range of motion compared to larger human datasets, we included the UvA-Nemo dataset to assess the method’s performance in scenarios with small motion amplitudes. Although VoxCeleb is another face dataset, its size—approximately 308 GB—restricts its usability. \n\nThe Ted Talks dataset has a resolution of 384×384, which is higher than the resolution of the datasets we used. However, the motion of the human body in the Ted Talks videos is minimal, as illustrated in https://github.com/ICML2025Anonymity/Anonymity/tree/main/TedTalks, by the double-blind policy. Due to the constraints of the rebuttal date, we only conducted experiments using the Ted Talks dataset.\n\nRegarding the MGif cartoon animal dataset, we were unable to evaluate motion-related metrics like AKD, which means that the dataset is not appropriate for validating our claim that our method enhances postural accuracy.\n\nQ5: The main strength is that the paper claims to deal with a challenging tasks of large displacement motion transfer. The weakness is the lack of experimental results to support the claim.\n\nA5: In this paper, we deal with the problem of large displacements between source and driver images by interpolating between them. As in Fig. 4 and Fig. 6, we show the interpolated image under the same identity and the interpolated image under different identities, respectively. Visually, it is observed that the movements of the interpolated image are closer to those of the driving image than those of the source image, which significantly alleviates the large displacement problem in the motion migration task. More experimental results will be shown in the following link (https://github.com/ICML2025Anonymity/Anonymity).\n\nQ6: There seems to be a grammar issue in section 3.2 \"Inspired by it, a keypoint-based anytime interpolation method\", is this an un-finished sentence?\n\nA6: Thank you for your careful review. The error has been revised: \"Inspired by it, a keypoint-based anytime interpolation method is proposed\".", "claims_and_evidence": "The experimental results are not convincing, such as when comparing with state-of-the-art qualitatively, it only shows one state-of-the-art method (TPSMM) in figure 3, and figure 5. It did not compare with X2Face, FOMM, MRAA, I wonder why?", "methods_and_evaluation_criteria": "Evaluation follow criteria that used by state-of-the-art. But the experiment lack a few key datasets.", "theoretical_claims": "looks correct.", "experimental_designs_or_analyses": "When comparing with state-of-the-art, it's essential to compare with the most up to date and relevant methods. X2Face is not referred, so I don't know which paper it is referring to. FOMM was published in 2019. MRAA was published in 2021, TPSMM was published in 2022. Why not compare with any more recent papers such as \nWang, H., Liu, F., Zhou, Q., Yi, R., Tan, X., and Ma, L. Continuous piecewise-affine based motion model for image animation. In In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI, pp. 5427–5435, 2024.\nFrom this paper's experiment results, it seems that this paper has much better performance.\n\n\nQualitative comparison seems not convincing, as shown in Figure 3-6, the proposed method's results look quite blurry, and images are of low resolution. Also since the paper claims to deal with large displacement motion transfer, I find that in the figures, a lot of examples have small displacement motion transfer, such as Figure 3 (first row), Figure 4 (second and third rows), Figure 6 (second the third rows). Why not showing more results that have large motion displacement to highlight the paper's main claim?\n\nWhen conducting the quantitative experiment, the proposed methods leave out a few key evaluation datasets, such as \"TED-talks\" , \"VoxCeleb\",  \"MGif\". I wonder why this paper did not compare with state-of-the-art methods on these datasets.", "relation_to_broader_scientific_literature": "The proposed method address a small research area of motion transfer.", "essential_references_not_discussed": "In the experiment part, the paper compares with a few state-of-the-art algorithms including X2Face, but X2Face is not mentioned/referred in the reference. It's difficult to understand what is exactly X2Face.", "other_strengths_and_weaknesses": "The main strength is that the paper claims to deal with a challenging tasks of large displacement motion transfer. The weakness is the lack of experimental results to support the claim,", "other_comments_or_suggestions": "There seems to be a grammar issue in section 3.2 \"Inspired by it, a keypoint-based anytime interpolation method\", is this an un-finished sentence?", "questions_for_authors": "My question are mainly related to the experimental design and comparison with more recent/relevant papers.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WuDhuz4XFj", "reviewer": "Reviewer_cLLC", "summary": "The proposed method advances unsupervised motion transfer by addressing the challenge of large displacement motions through interpolation and strategic training. While it excels in pose accuracy, it faces minor challenges in maintaining appearance details, particularly in complex scenarios. This work provides a robust framework for applications in image animation, with potential for further refinement in appearance preservation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments and suggestions you gave. We have incorporated your feedback into the paper. \n\nQ1: The method is reasonable but assumes linear motion, which may not hold for complex cases.\n\nA1: In the interpolation method, we assume that the local keypoints' motion is linear to obtain keypoints that do not exist between the source and driver images. The corresponding interpolated images are then generated from these interpolated keypoints. The pose of these interpolated images is between the source and driver images. The motion from the interpolated image to the driver image is smaller than the magnitude from the source image to the driver image. And for modeling the motion, we use the nonlinear transform (TPS) from the literature [1] to approximate the motion.\n\n[1] Zhao J, Zhang H. Thin-plate spline motion model for image animation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022: 3657-3666.\n\nQ2: The datasets are appropriate, but lack comparisons with newer methods.\n\nA2: We have added the 3 most recent comparison methods (DAM [2], MTIA [3], and CPABMM [4]) as shown in Table 1 in A2 of Reviewer 1 LsVD. Relevant qualitative results have also been added, as shown in the link (https://github.com/ICML2025Anonymity/Anonymity), which complies with the double-blind policy.\n\n[2] Tao J, Wang B, Xu B, et al. Structure-aware motion transfer with deformable anchor model[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 3637-3646.\n\n[3] Tao J, Wang B, Ge T, et al. Motion transformer for unsupervised image animation[C]//European conference on computer vision. Cham: Springer Nature Switzerland, 2022: 702-719.\n\n[4] Wang H, Liu F, Zhou Q, et al. Continuous piecewise-affine based motion model for image animation[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(6): 5427-5435.\n\nQ3: In ablation study, the effectiveness of the proposed module is not clearly proved according to Table.2.\n\nA3: In the ablation experiment, we mainly validate three modules, the first is the interpolation method. We added the interpolation method on the baseline (TPSMM), as shown in Model 1 in Table 2. In the unsupervised case, the lack of constraints on the interpolated image makes the generated interpolated image very poor, which seriously affects the quality of the downstream target image generation. To solve this problem, we add a second module, the two-way training strategy, as shown in Model 2 in Table 2. By adding consistency constraints to the intermediate generated interpolated image pairs, the generation range of the interpolated images is narrowed down, significantly improving the generation quality of the target images. In order to improve the generation quality of the target image, we added a third module to add appearance consistency loss and structural consistency loss to the image using the pre-trained Vit model, as shown in “Ours” in Table 2, which is our complete model.", "claims_and_evidence": "The linear motion assumption is too strict and not empirically validated.\n\nThe lack of comparison with recent works makes it uncertain if the method is truly state-of-the-art.\n\nThe ablation study does not clearly prove the necessity of each module.", "methods_and_evaluation_criteria": "The method is reasonable but assumes linear motion, which may not hold for complex cases. The datasets are appropriate, but lack comparisons with newer methods.", "theoretical_claims": "Since the paper primarily relies on empirical validation, no formal proof of correctness check is required.", "experimental_designs_or_analyses": "Comparisons should include recent methods (2023-2024) to validate performance.\n\nAblation studies need more detailed analysis to confirm the contribution of each module.", "relation_to_broader_scientific_literature": "The paper extends prior motion transfer work with a two-step strategy and ViT-based consistency enforcement, but its linear motion assumption, outdated baselines, and missing perceptual metrics limit its broader impact.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strength:\n\n1.\tThe author proposed to decompose the traditional motion transfer pipeline into two steps to solve the difficulty when there is large displacement of motion between S and D.\n\n2.\tA ViT is introduced to ensure the consistency of appearance and motion.\n\nWeakness:\n\n1.\tThe assumption that motion between S and D of each keypoint is linear seems to be strict, especially in cases such as non-linear human actions in Tai-Chi-HD.\n\n2.\tIn quantitative comparison, the newest work is from 2022. More recent works are better be included.\n\n3.\tIn ablation study, the effectiveness of the proposed module is not clearly proved according to Table.2.\n\n4.  Some related works [1-2] are discussed and compared. \n\n[1]. Structure-aware motion transfer with deformable anchor model. CVPR 2022.\n\n[2]. Motion Transformer for Unsupervised Image Animation. ECCV 2022.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QX5Bnwqzug", "reviewer": "Reviewer_swRi", "summary": "This paper presents an anytime interpolation framework for flexible and accurate motion-driven frame generation. Specifically, during training, the model searches for an optimal intermediate time step that produces the highest-quality interpolated frame for training. To ensure valid motion transfer, the authors design an unsupervised bidirectional training strategy, effectively preserving appearance and structural consistency in the generated frames.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the valuable comments. \n\nQ1: The authors provide several experimental details to validate the algorithm, but some ablation analyses are missing.\n\nA1: Thank you for your comments. The Reviewer 3 LLC asked the same question in Q3. \nIn the ablation experiment, we mainly validate three modules, the first is the interpolation method. We added the interpolation method on the baseline (TPSMM), as shown in Model 1 in Table 2. In the unsupervised case, the lack of constraints on the interpolated image makes the generated interpolated image very poor, which seriously affects the quality of the downstream target image generation. To solve this problem, we add a second module, the two-way training strategy, as shown in Model 2 in Table 2. By adding consistency constraints to the intermediate generated interpolated image pairs, the generation range of the interpolated images is narrowed down, significantly improving the generation quality of the target images. In order to improve the generation quality of the target image, we added a third module to add appearance consistency loss and structural consistency loss to the image using the pre-trained Vit model, as shown in “Ours” in Table 2, which is our complete model.\n\n\nQ2: The use of L1/L2-based reconstruction loss may lead to blurry interpolated results. Consider integrating GAN-based refinement or a perceptual loss to improve sharpness.\n\nA2: Thank you for your comments. In the unsupervised scenario, we aimed to improve the quality of the interpolated images through bidirectional training by incorporating additional loss functions to constrain their generation. Initially, we used L1 constraints on the generated pairs of interpolated images; however, our experiments showed that these L1 constraints were too strict. As a result, minimizing the loss on these image pairs often led to the generation of all-white images. We applied L1 loss to their multiscale features to address this issue, as demonstrated in Equation (6).\n\nTo further enhance image quality, we utilized a pre-trained Vision Transformer (ViT) model to introduce appearance consistency loss and structure consistency loss to the interpolated image pairs, as outlined in Equations (7) and (8). Additionally, we attempted to refine the images using a super segmentation network, but the improvements were minimal.\n\nQ3: The design of the interpolation selector lacks sufficient justification. A more detailed discussion on the impact of different weighting strategies on final performance would strengthen the argument.\n\nA3: The interpolation network generates multiple interpolated images that progressively adopt the poses of the driving image as time (t) increases from 0 to 1. However, we observe that as the pose gets closer to that of the driving image, the visual quality of the interpolated images tends to deteriorate. To address this issue, we have designed an interpolation selector aimed at identifying the optimal interpolated image from the set of generated images. This optimal image should not only have a pose that closely matches the driving image but also maintain visual consistency with the source image. In motion migration, preserving appearance is just as crucial as learning the pose; therefore, we assign equal weights in Equation (4). In our future studies, we plan to explore how different weighting strategies might affect overall performance.\n\nQ4: The model requires pre-generating multiple interpolated frames and selecting the best one, which may introduce significant computational overhead.\n\nA4: Thank you for your valuable comments, multiple interpolated frames and selecting the optimal interpolated frame impose a significant computational overhead on the model. Since we are unsupervised interpolation method, there is no guarantee that the generated interpolated images can satisfy the subsequent downstream tasks without labeling. In my future research, I will improve the proposed model as a way to reduce the computational overhead.\n\nQ5: The paper should discuss and compare conditional diffusion-based approaches, which are gaining popularity in motion transfer. Highlighting the advantages of the proposed method over such alternatives would further clarify its contributions.\n\nA5: Thank you for your valuable comments. In recent years, diffusion-based motion migration methods have shown satisfactory performance. However, these methods are supervised and often rely on a pre-trained model to extract a priori conditions for the target image, such as human body key points. In the case of the Taichi dataset, accurately extracting human body poses can be challenging due to the low resolution of the images. Additionally, the diffusion process requires extensive computational resources, which may limit the model's performance in environments with restricted resources.", "claims_and_evidence": "The majority of the content is clear, but some detailed design aspects need further justification.", "methods_and_evaluation_criteria": "The evaluation criteria appear to be appropriate and sufficient.", "theoretical_claims": "No theoretical claims", "experimental_designs_or_analyses": "Yes, the authors provide several experimental details to validate the algorithm, but some ablation analyses are missing.", "relation_to_broader_scientific_literature": "Provide insights into unsupervised large motion transfer, such as the search for optimal interpolated time steps and constraints on appearance and structural consistency for effective decomposition.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Strengths:\n\nThe motion decomposition strategy alleviates the challenge of directly transferring large displacement motions.\nTo enhance supervision, the authors propose an optimal interpolation selector, which identifies the interpolated frame with the best identity consistency and minimal warping distance.\nThe method incorporates appearance and structural consistency constraints, improving the overall motion transfer quality.\n\nWeaknesses & Suggested Revisions\n\nThe use of L1/L2-based reconstruction loss may lead to blurry interpolated results. Consider integrating GAN-based refinement or a perceptual loss to improve sharpness.\nThe design of the interpolation selector lacks sufficient justification. A more detailed discussion on the impact of different weighting strategies on final performance would strengthen the argument.\nThe model requires pre-generating multiple interpolated frames and selecting the best one, which may introduce significant computational overhead. It would be beneficial to provide a comparison of inference/training costs with other baseline methods.\nThe paper should discuss and compare conditional diffusion-based approaches, which are gaining popularity in motion transfer. Highlighting the advantages of the proposed method over such alternatives would further clarify its contributions.", "other_comments_or_suggestions": "See the ``Other Strengths And Weaknesses\"", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_flag": "Flag this paper for an ethics review.", "ethics_expertise_needed": ["Privacy and Security"], "ethical_review_concerns": "The visual examples and the training/evaluation datasets mayraise concerns regarding privacy and security."}}]

Author rebuttal:

Meta-review: This paper is about transferring large motion from a driving video to a source image. 

The paper initially received three weak accept and one weak reject. The main strength lies in the novel bi-directional training strategy and optimal interpolation selector. The reviewers appreciate the novelty of these components and the improvement over the baselines. The main weaknesses are mainly on the lack of comparisons with more recent methods and missing results on the common datases like TedTalks and VoxCeleb. 

In the authors' rebuttal, the authors provided additional experimental comparisons with three recent methods (DAM, MTIA, and CPABMM) in Table 1 in the provided link, as well as results on TedTalks. Overall, the three reviewers LsVD, cLLC, and swRi maintain their positive views about this paper while Reviewer usGM did not respond or acknowledge the rebuttal. The AC read the reviews and the rebuttal and found that the additional results sufficiently addressed the raised concerns. Consequently, the AC recommends to accept.
Decision: Accept (poster)