Implicit Warping for Animation with Image Sets
Authors: Arun Mallya
Ting-chun Wang
Ming-Yu Liu
Venue: NeurIPS 2022 Accept (2022)
Keywords: image animation
attention
motion transfer
video synthesis

Abstract: We present a new implicit warping framework for image animation using sets of source images through the transfer of motion of a driving video. A single cross-modal attention layer is used to find correspondences between the source images and the driving image, choose the most appropriate features from different source images, and warp the selected features. This is in contrast to the existing methods that use explicit flow-based warping, which is designed for animation using a single source and does not extend well to multiple sources. The pick-and-choose capability of our framework helps it achieve state-of-the-art results on multiple datasets for image animation using both single and multiple source images.

Implicit Warping for Animation with Image Sets
Arun Mallya amallya@nvidia..cmm com
Ting-Chun Wang tingchunw@nvidiaa com NVIDIA
Ming-Yo Yu Liu
mingyul@nvidia.
Abstract
We present a new implicit warping framework for image animation using sets source images through the transfer of the motion of driving video. A single cross- modal attention layer used to find correspondences between the source images and the driving image, choose the most appropriate features from different source images, and warp the selected features. This is in contrast to the existing methods that explicit flow-based warping, which is designed for animation using a single source and does not extend well to multiple sources. The pick-and-choose capability of our framework helps it achieve ofute-of-thh-an results on multiple datasets for image animation using both single and multiple source images.
Introduction
We study the task of generating videos by animating a set source images of the same subject using the motions of driving video possibly containing different subject. This is more general than the setting of animating a single source image studied in the prior works [26 27 41 (00 as it a special when the set contains just image. Moreover, having the capability of animating set of images, especially those representing different views when available, of great practical value. A single image often cannot fully describe the subject due to occlusions, limited pose information, Diverse source images provide more appearance information and reduce the burden hallucination that an image generator has perform. As shown in Fig. D multiple source images provide more complete information, such as the color of the eyes, the texture of the background, etc. This allows for potentially generating output image that is more faithful to the source setting.
The single-soune source-uusssseeee based prior works 226
50 ubiquitously rely on explicit flow-based warping of the source image conditional the pose of the driving image. Due this architectural choice, they often have to be modified in ad-hoe ways to take advantage of multiple source images. One scheme is to train additional pre-processing network to select the most appropriate source image for the given driving image. This would, however, not allow for the use of features from multiple source images at time. The other possibility is warp each source image to the driving pose and then average the now-aligned warped features for the generator input. But as is visible in Fig. and later measured in Section 4 this leads sub-optimal results due to the misalignment of warped features and inconsistent predictions across views.
In this work, introduce novel implicit warping mechanism that overcomes the above drawbacks (I) It operates on a set of images and naturally scales from a single source image to multiple source images; and (2) Without predicting explicit per-source flow, single layer finds correspondences between the source images and the driving image, chooses the aptest features from different source images, and performs warping of the selected features. Our novel layer and the ability pick-and- choose appropriate features from a set of source images allows our method to achieve state-of-the-ae image animation quality in the multiple source image setting, as well as the single source image setting. The benefits of our framework are obvious in Fig. D in the image reconstruction and cross-identity motion transfer tasks. In the top row, the inability of prior works of FOMM (26) and face-vid2vid 41 to pick-and-choose features results in an incorrect output eye color. Only our
36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

Sources
Driving video
FOMM 26
fv2v 41
Ours
Sources
Driving vidco
Ours (one souree)
FOMM 26
Ours
Fig. 1: Our image animation model uses information from one more souree images by picking and choosing from available features. This allows our model output for e.g., the correct eye color, and the correct background as highlighted by the orange ovals, in both same- and cross-identity motion transfer. Prior works fail to correctly align and utilize multiple source images, leading to blurry and incorrect outputs. Click on any result to play the video. Please view with Adobe Acrobat Reader.
method produces the correct background appearance the lower row when given both source images. The orange ovals highlight areas of major differences amongst predicted outputs.
2
Related Work
Image animation. Recent works on image animation can be roughly categorized into subject- dependent and subject-agnostic works. For subject-dependent, the framework is trained on specific subject and can only animate that specific person 2
K33 36 46] For example, NerFACE g trains neural radiance field to model target face using short clip of the subject. On the other hand, subject-agnostic frameworks only need a single 2D image to perform the animation on the target person, so they are usually more general and applicable
g. 30
5
M
26
37
40, 3J1 44! learned latent keypoints. Zakharov et al. (50) focus on inference speed and propose decompose high and www-freuueccy components for better efficiency. Wang et al. [41 learn 3D latent keypoints and their decomposition to predict warping flows and gain better controllability. Doukas al. [6] adopt 3D morphable models to generate dense flow field to warp the source image and then take the warped input along with audio features to synthesize the final output. While these frameworks achieve promising results in image animation, none of them are designed for using multiple reference source images as input. In contrast, our framework designed to take advantage complementary information available in different input images for high-quality image animation.
50, 55 (54). For example. Siarohin et al. 2 predict local affine transformations using

---

Attention in computer vision. The self- olf-attention layer, which uses the same input as the query, key, and value, was popularized by the work of Vaswani et al. [35] for the task of machine translation. Since then, number of works have explored using the self-attention layer in conjunction with convolutional layers for the tasks of image and video recognition [33 38
47 A few works have also explored cross-modal attention in which the queries, keys, and values are computed from two different domains, such as vision and text 4S 49], audio and text 191, etc. In our modll attention layer, the queries and keys are from the domain of compact image representation, such as keypoints, while the values are extracted from the dense image features.
f5
Networks fully composed of self- attentttntion layers, also known as transformers, have recently obtained state-of the-art numbers in the tasks of image classification [ 20
34 12 as well as generation 7 15 17 22 [4! [5]] Our generator uses hybrid architecture, with single cross-moda attention layer for warping the features of the source image conditional on the pose of the driving image. This is
followed by decoder made up of convolutional residual blocks to produce an output image.
While we do not predict explicit flows to warp feature maps, related work by Xu al. 47 has used efficient I-D attention layers for predicting explicit optical flow. We use global 2-D attention, but such factored I-D attention layers and spatial-reduction attention layers +2] can be used reduce the run-time further. To the best of our knowledge, we are the first to propose a way of finding correspondences and warping features via a single cross-mode attention layer for image animation.
3 Image Animation with Implicit Warping
The idea behind image animation using a source image is to warp features of the source image conditional on the given driving image so to reconstruct the driving image transfer the pose of the driving image in the case of cross-identity motion transfer. Such image animation methods typically consist of 4 components: (1) Source image representation extraction, (2) Driving image representation extraction, (3) Source image feature warping, and (4) Warped image feature decoding. In this work, we focus the third component-sourre
image feature warping. The other components In the following sections, we describe
in our method are largely based on prior works 226 27
our framework and contributions in more detail.
Explicit flowe warping. Prior works on image animation, such as Monkey-Nee 2 First Order Motion Model (FOMM) (26), and face-vid2vid 411, use flow-based warping technique to deform the features extracted from the source image. They generate a warping field using the compact representations extracted from the source and driving images, usually in the form of keypoints and additional related information such as Jacobian matrices. This field is used to warp the source image features, which are subsequently mapped to the output image space by a decoder network. The prior works, specifically FOMM (6 and face-vid2vid [41), perform warping using the scheme illustrated in Fig. 2a Given the locations of K unsupervisedly-leaneed keypoints in the source image S and driving image D. they first generate K per-keypoint flows W1,-. .KK By using the source image (or extracted features) warped with each of the K flows, motion estimator network predicts how to combine the individual flows in the flow composition mask m. Combining the К per-keypoint flows with this mask gives the final composite flow mask w, which is applied the source image features fs to obtain the warped feature w(fs)) Due to the per-keypoint warping steps, the time complexity of these methods is directly proportional the number of keypoints K.
As mentioned in Section I the prior works have significant drawbacks when extending them to multiple source images. We either have average the per-source image warped features employ a heuristic to choose the most appropriate source image given the driving image. Neither of these two solutions ideal as the former can lead to blurry outputs and the latter leads to flickering when switching between sources. A more desirable way is to have an end-to-end trainable pick-and-mix feature combination mechanism that can aggregate information from multiple source images.
In this work, we introduce 'Implicit Warping' -a new and simpler way to warp source features. This results in image animation method that is easily and directly extensible to using multiple source images. Our method relies on a cross cross-modal modal attention layer illustrated in Fig, 26 neither produces explicit per-keypoint warping flows nor requires per-source image warping of features.
Implicit Warping using attention. The process of warping the source image features conditional on the driving image consists of two parts: (1) Identifying dense correspondences between the source
3

---

Sturce taage
Source Amaqu/ featurrs
FLONS
Par Nar-keypsint FLous
Source keyposints
Source
Source
Flom compon ptippcttion #ask
X
co Flon a
estinattr
feature WCF,
keypoints
(a) Warping mechanism used in First Order Motion Model (FOMM) 26
and face-vid2vid 41
inage(s)
Source image(s) and keypcints
co feature ko.
feature
enceder
keypalint encoder
Value(s)
Key(s)
Query
attertion
aff.)
keyyolit
seypsint feature
encoder
kou
Delwing
keypstnts
(b) Our warping mechanism based cross modal attention.
Fig. 2: Comparison of image warping mechanisms of the prior works against Our method is based on cross- modal attention and simpler. Moreover, it can be easily extended to using multiple source images. Our method does not require per-keypoint warping of the source image(s) or features unlike FOMM 26 and face-vid2vid [41] and can pick and mix features from multiple images.
veypoint feature KR.
Briving
Source Heypoonnt feature
feature
Source
Source
Flatten and Linearly posject
Flattio
Latten
Arage
(002))
(avd)
Tuee
Query
Quusy
petd-pp atcention
Way
Cqpie3
X
Cqpli
X
Casl (313))
Concatannae
TLP
Cyykk)
Restdual contection
Reshape
Cross-nodal
attantion module
Kasped
(qed')
Fig. 3: Coss Coss-modal attention used for warping. The features from the driving keypoints and source keypoints are treated as the query and the key respectively. The computed attention applied on the source features, the value. This reconfigures the source features correspond to the pose of the driving image. The attention is also applied to the source image pixels and the key, which then concatenated with the query. After processing with MLP, these added to the output values. Reshaping the outputs back to 2D produces the warped feature. This scheme can be naturally extended multiple source images. Adding new source image increases the number of keys and values, resulting in an appropriately larger attention map A (specifically dimension k in above figure).
and driving images, and (2) Warping the source image features based on the dense correspondences. The key intuition behind our method is that these steps be represented using single scaled dot-product attention layer popularized by Vaswani et al. 35
Q KT
Attention Q, K, V) = softmax
X xV=5xV,
(I)

---

where Q. K, and V are made up of vector-valued queries, keys, and values, respectively, and c is scaling factor. For a given query vector Q of size (1 d), the softmaxed value of the product Q x KT can be interpreted as the similarity S of the given query vector with the set of all available keys K of size (k: x d). After the softmax normalization, the key with the highest dot-product with the query will obtain the highest similarity. After computing the similarity S of size (1 x k) and given values V of size (k x )"), the product S x V gives a query-conditions weighted average of values of size (1 x d'). This scheme naturally extends to multiple queries Q of size (g d). In our particular application, the queries Q come from the driving keypoints, and the keys K and values V come from the source image(s) keypoints and features, respectively. Using additional source images only leads to an increase in the number of available keys and values.
a
We refer to the above procedure as implicit warping, as illustrated in Fig. 26 This is opposed to explicit warping shown in Fig. 2a where an explicit flow map is first computed from the source and driving keypoints and then applied to source features to produce warped outputs. There is no explicit flow map in our scheme; the similarity between the queries and keys produces an implicit flow map, which is then applied to the values to produce the warped outputs. In the following, we describe how we learn queries, keys, and values for the novel implicit warping based on cross-modal attention.
Learning queries, keys, and values. Similar to the prior works 26 and associated information as a compact representation of the source and driving images. Our image encoder that predicts keypoints reuses the architecture from FOMM [2 While reconstructing the driving image, we only have access the driving keypoints but have access to both the source image(s) and the source keypoints. Given the K keypoints predicted from an image, we create a spatial representation of K channels where each channel represents keypoint. For the 1.th keypoint, we place Gaussian of a fixed mean and variance in channel k. centered at the predicted keypoint location. In addition to the K keypoints, we predict scalar per-keypoint with values in [0, 1J. serving as the keypoint strength score. We multiply this predicted scalar with the channel corresponding to the keypoint. This allows us to modulate the keypoint strength, which is related to its visibility, in the given source and driving images. The predicted keypoint strengths are later discussed in Section 4 and visualized in Fig. 6 Both the keypoint locations and the scalar factors are learned in an unsupervised manner. Unlike FOMM (26]. we do not predict the Jacobian matrix per keypoint in addition to the keypoint location, making our intermediate representation more compact (2 + 1 = 3 values per 6 values per keypoint in the prior work). Unlike vid2vid [1] we
we use learned keypoints
keypoint instead of 2 do not assume rigid motion and find applications beyond talking-head animation.
We use a U-net to encode the above spatial driving keypoint representations, with final point-wise projection layer to map it to the dimension selected for the queries and the keys. A similar U-net is used to encode the source image, except that its input consists of both the source image and the spatial source keypoint representation. Both these networks accept inputs and produce outputs at 1/4 resolution of the source image, i.e. 64 for 256 256, 96 for 384 384, etc. Assuming source image size of 256 x 256, the two U-net networks produce a total of (64 x 64 x d). or (4096 x d) queries and keys, for each driving and source image, respectively. Since the output of these U-nets is of fixed dimension d, the design of the subsequent components is independent of the number of keypoints used. In order to obtain values from the source image, we pass it through two downsampling convolution layers and obtain values of size (4096 x d'). Note that there is a one-to-one spatial correspondence between the keys and the values. Additional details can be found in the supplementary material.
However, there is one possible pitfall to the scheme described so far. If none of the keys have a high dot-product with the query, the key with the highest dot-produce will still be assigned the high similarity after the softmax, ensuring that the outputs sum up to 1. While it is the key most similar to the query in the given set of keys, it may not be similar enough to produce good output. For example, suppose the source image has a face with lips closed, while the driving image has one with lips open and teeth exposed. In this case, there will be no key (and value) in the source image appropriate for the mouth region of the driving image. We overcome this issue by allowing our innge-independentt key-value pairs, which can be used in the case of
method to learn additional missing information in the source image. These additional keys and values are concatenated to the keys and values obtained from the source image. In our experiments, we add 400 extra keys and values, approximately 10% of the number of keys and values obtained from a single source image of size 256 x 256. We use semantic dropout while training our framework encourage the learning and use of the added keys and values. This is achieved by randomly dropping out keys and values corresponding to different facial regions, such as eyebrows, eyes, lips, teeth, etc. This possible as
5

---

an attention layer treats its input keys and values as a set. In contrast, convolutional layers assume specific spatial order, and spatial locations cannot be randomly dropped out. As shown later in Section and Table 4 these extra keys and values help improve output quality.
The cross-modal attention module. After obtaining queries, keys, and values, the cross-modal attention module computes the warped source features conditional on the driving image via implicit warping. As shown in Fig. ] the queries kpa obtained from the driving image are of size (g d) after flattening. The keys kp, and values fa obtained from the one or more source images are of size (k d) and (k x d') respectively. We add learnable position embeddings to the queries kp and keys kp., The dot product Q K" after scaling and softmax gives the attention matrix A of size (g x k). By multiplying A with values V of size (k x d'), we obtain the warped output feature of size (q x d').
Additionally, we concatenate the image pixels with the keys and align them with the driving image by multiplying them with the computed attention map, which is further concatenated with the queries and passed through an MIP (purple-shaded part of Fig. 5 The intuition behind using the warped keys and queries is to recover any information, such as skew or rotation, which a weighted average of values may not easily capture. Further, including the pixel values helped by aiding color consistency. Adding this residual connection helped improve the quality of outputs, as shown in Section f and Table
While our end-to-end framework is reasonably fast (~ 10 FPS on 512 x 512 images), it is possible to reduce the run-time of the attention operations by utilizing the factored 1-D attention layers 477 or the
attention (SRA) layer 1.) which we leave for the future work. After training, observed that the learned attention masks very sparse. This indicates that we can further reduce the inference time by selecting just the top-few values per row from the product A = Q x KT, and multiplying them with corresponding values of V, thereby saving computation.
We show results on multiple datasets in the single and multiple source image settings with comparison to strong baselines in the next section, and try to provide insights into the workings of the model.
4 Experiments
Datasets. We perform our ablations on the TalkingHead-1 KKH [1 dataset the 256 256 resolution and compare with baselines at the full 512 x 512 resolution. We also report results on the 256 x 256 VoxCeleb2 2 dataset. Additionally, to demonstrate the generality of our method, we report results on the more challenging TED Talk
[XS) dataset of moving upper bodies at a resolution of 384 x 384.
Metrics. We measure the fidelity of driving image reconstruction using PSNR, L1, LPIPS 5k and
FII 12) The quality of motion transfer is measured using average keypoint distance (AKD) between keypoints predicted using MTCNN ( for faces and OpenPose 4 for upper bodies. For upper-body motion transfer, we also report the missing keypoint ratio (MKR), at chosen keypoint prediction confidence threshold C, together denoted as (AKD, MKR) @ C. For human evaluation, MTurk users were shown videos synthesized by two different methods and asked to choose the more realistic one.
Baselines. On the face datasets, we compare our method against two statc-of-the-an methods: First-Order Motion Model (FOMM) (26), and face-vid2vid (fv2v) 41. On the upper-body dataset, we compare against FOMM PTC and the state-of-the the-tr AA-PCA
28 We use the pretrained model provided for AA-PCA, while we train all other methods from scratch. As previously mentioned in Section I prior works are unable to handle multiple source images, despite the immense practicality of the setting. We adapt these methods, where possible, to multiple source images by warping features from each provided source image, and then averaging them to obtain a single warped feature map. All methods use K- = 20 keypoints in the following experiments. Additional details about datasets, metrics, and training hyper-parametert for all methods are provided in the supplementary material.
Driving image reconstruction with single source image. This is the standard evaluation setting used in prior work [26-28 AT The first frame of each video is chosen as a source image, and the full-length videos are reconstructed. Quantitative evaluation of predictions is presented in Table Our method outperforms competing methods in almost all metrics on both datasets. We obtain a lower FID and average keypoint distance (AKD) on both datasets, indicating good visual quality as well as motion reproduction in the predictions. Fig. El highlights a case where our attention- able to provide the correct reconstruction while other methods fail. As attention is non-local and global in nature, it can borrow image features from regions that are spatially far away, e.g. the left and right hair and ears. Prior works based on explicit flow prediction would have to predict large values of flow for such newly disoccluded regions. Fig. 5 compares results between the various methods
method
6

---

Table 1: Comparisons with prior work on image reconstruction using a single source image.
i larger is better, 1. smaller is better. See section for details about metrics.
TalkingHead-II8e eT
VoxCeleb2 26
TED Talk 28
512 x G12 faces
256
256 faces
384
3841 upper bodies
FOMM 26 fv2v H1
Ours
FOMM
(26 fv2v [1]
Ours
FOMM (6
AA-PCA (28)
Ours
PSNR
23.23
23.27
23.32
24.15
23.66
23.54
24.17
25.14
25.24
C 1
12.86
12.30
12.86
10.99
11.10
11.40
8.22
6.87
7.69
LPIPS 1
0.16
0.16
0.15
0.12
0.12
0.11
0.16
0.13
0.12
AKD (MTCNN) 1
4.25
3.83
3.48
1.69
1.70
1.75
(AKD, @KR)@0.1
(7,44, 0.01)
(5.10, 0.005)
(4.39, 0.005)
(AKD, @K5)@0.5
(5.93, 0.06)
(4.13, 0.030)
(3.31, 0.023)
FID
18.15
18.09
16.44
9.53
5.76
4.68
24.69
19.17
18.27
Table 2: Comparisons with prior work on image reconstruction using multiple source images.
TalkingHead KH (512 x 512 faces)
#Source frames
2
3
FOMM 26
fv2v 41
Ours
FOMM 26
fv2v [4]
Ours
FOMM 26
fv2v 41
Ours
PSNR C1 1 LPIPS 1
24.260
24.273
24.276
25.433
25.557
26.094
26.044
25.843
26,768
11.075
10.338
11.152
9.023
8.292
7,990
8.238
8.330
7.347
0.139
0.135
0.130
0.116
0.110
0.089
0.109
0.118
0.081
AKD (MTCNN) J FID 1
5.039
4.792
4.370
4.662
4.384
4.029
4.483
6,201
3.957
17.454
17.174
15.658
19.647
16.846
9,850
19.495
19.216
9.097
TED Talk (384
384 upper bodies)
#Source frames
3
FOMM 26
Ours
FOMM 26
Ours
FOMM 26
Ours
PSNR
24.096
25.188
24.546
26.528
24.835
26.884
Ci
8.290
7.737
7.726
6.597
7.398
6.319
LPIPS
0.156
0.119
0,143
0.088
0.137
0.081
(AKD, MKR)@0.1 J
(7.618, 0.010)
(4.438, 0.005)
(6.898, 0.010)
(4.182, 0.004)
(6.518 0.010)
(3.989, 0.004)
(AKD, MKR)@nn.5 (6.041, 0.060)
(3.349, 0.025)
(5.325, 0.067)
(3.264, 0.021)
(5.064, 0.067)
(3.173, 0.021)
FID 1
24.324
17.746
29.053
13.822
30.838
13.128
when using a single source image on the TED Talk dataset. Our method demonstrates better hand and face reconstruction than prior works.
Driving image reconstruction with multiple source images. We now compare results in the case of driving image reconstruction when multiple source images provided. This is setting of great practical value as a single source image often does not have all the details necessary to reconstruct driving image with any arbitrary pose and expression. We present results obtained when using multiple source images in Table 2 Here, we use sequences of length at most 180 frames and select regularly-spaced source images addition the first frame. Similar to the single source image setting, our method outperforms all prior work. As the number of source images increases, our method obtains better reconstructions as indicated by the improving scores on all metries. However, reconstructions by prior work get worse as the number of source images increases, contrary to expectation. Upon viewing the qualitative results shown in Fig. [ the issue with prior works becomes clear-even though features from each source image are warped to the same driving pose, there exist misalignments between the warped features. Further, they do not have the ability to choose features from only specific source image. Our method does not suffer from this issue due to global attention over all available images and their features. As a result, our method is able to predict the correct eye color and sharp background, as seen in Fig. D Fig. g shows regions of the source image that the highlighted regions of the output borrow features from. This demonstrates our picking-and and-coss--- mechanism- regions {1, 2, 4, 7,99 visible in both sources contribute equally to the output, while the rest, only visible in one of the sources, is used exclusively. User preference scores are provided in Table Users prefer our method in all settings, and the preference increases further when more source images are used. Additional comparisons are available in the supplementary material.
A closer look at the architecture. Having shown the superiority of our method on image reconstruc- tion in the single and multi-source image settings, we now examine the contribution of architectural choices to the output quality. As shown in Table [ adding the residual connection visualized by the purple region in Fig- 3 improves all metrics. This indicates that the warped image and keys, along
with the queries provide additional information to the values. Adding extra learnable keys and values, as described in Section g further improves all metrics by allowing the network to share features when information is missing in the source image(s). Fig, visualizes predicted keypoints and strengths for
7

---

Our output
Source image
Our output
Driving
Ours
FOMM 26
Iv2v
Source image
Source image 2
Fig. 4: Benefits of non-local attention. In the 2 top-left images, we visualize the locations in the source image assigned the highest attention score for correspondingly marked regions in our output. Note that the features for the newly disoccluded right hair and right ear of the output are borrowed from the left side the source image. The images on the bottom-left side show zoomed-in regions of the driving image, and reconstructions by 3 different methods. Due to global attention the source, our method produces the correct hair color for the disoceluded regions, while the rest fail to do To generate the output image the top-right, both or one of the source images on the bottom-right, depending on the visibilities of various features, e.g. region is used from both images, while region 8 only from source image 2.
method picks-and-choose features from
Source image
Driving image
FOMM 26
AA-PCA 23
Ours
Fig. 5: Visual comparison of results the TED Talk dataset. Our method produces better motion, face, and hand quality. Click on any result to play video. Please view with Adobe Acrobat Reader:
various configurations of a person's face. For the same arrangement of keypoints (both or single eye closed), we see different keypoint strengths based on the appearance. We also see large changes in keypoint strengths when certain keypoints are occluded, e.g. when the face is rotated to an extreme angle. This single scalar in place of a 4-valued Jacobian matrix per keypoint helps us achieve more compact representation useful for applications such as video compression and conferencing
Discussion
In this work, presented the novel implicit warping framework and showed its application to image animation. Through the use of a single cross-modal attention layer, our method is able to deform source image features conditional on the driving image. and can scale to multiple source images as-is. Unlike prior work, our method does not predict explicit per-source image flows, and pick-and-chooss from multiple source image features. It convincingly beats prior state-of-the art methods on multiple datasets in both the single and multiple source image settings, while using a more compact intermediate representation of keypoint locations and strengths.
Limitations. Our method can fail in cases where is expected to hallucinate a of missing information. For example, given just source image of the back of someone's 's head, it cannot generate

---

Table 3: User preference scores. Users prefer method even when using one source, with increased preference as more sources are used.
Table 4: Architecture ablations on image re- construction with single source image TalkingHedd- IKH at 256 256 resolution.
TalkingHead- IKH
VS FOMM
vs fv2v
Residual connection
Extra key-value
source
64.1% 65.8%
53.5%
sources
59.9%
PSNR
23.253
23,483
23.582
L1
12.741
12.721
12.632
TED Talk
vs FOMM vs AA-PCA
LPIPS 1
0.122
0.114
0.113
source
81.5% 91.4%
63.3%
AKD (MTCNN) ! FID |
1.969
1.914
1.895
sources
19.978
18.462
18.252
Reference Roth eyes open Mouth closed
Both eves closed
Left eye closed
Rightt eve closed
Mouth open
Left facing
Right Geing
Reference keypoint strengths
Fig. 6: On the left, we show reference image overlayed with predicted keypoints, and their strengths. On the right, we show different input images and their relative keypoint strengths w.r.. reference keypoint strengths (Please zoom in to numbered keypoints the face). The network latently learns to associate different values of the strength to different keypoint visibilities and facial configurations.
the correct frontal face. Since it is a data-driven approach, it might fail for extreme expressions not present in the training dataset. Additional data and augmentations might help alleviate such issues.
Societal impact. Our method has the potential for negative impact if used to create deepfakes. Via the of cross-identics transfer and speech synthesis, malicious actor can create faked videos of a person, resulting in identity theft or dissemination of fake news. However, in controlled settings, the same technology low-bandwidth video compression the same-identity video reconstruction setting. This can help improve video quality, especially for regions with poor connectivity. As our method generates outputs using source image features, we have not observed and do not expect it to demonstrate any racial bias.
also be used for entertainment purposes. Our method can be used in

---

References
[1] Hadar Averbuch-Elor, Daniel Cohen-Or, Johannes Kopf, and Michael F Cohen. Bringing portraits life. TOG, 2017.
[2] Aayush Bansal, Shugao Ma, Deva Ramanan, and Yaser Sheikh. Recycle-GAN, Unsupervised video retargeting. ECCV, 2018.
[3] Egor Burkov, Igor Pasechnik, Artur Grigorev, and Victor Lempitsky. Neural head reenactment with latent pose descriptors. CVPR, 2020.
[4] Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-Fn Wei, and Yaser Sheikh. OpenPose: realtime multi-person
21D pose estimation using Part Affinity Fields. TPAMI, 2019.
[5] Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Hierarchical cross coss-modal talking face generation with dynamic pixel-wise loss. In CVPR, 2019.
[6] Michail Christos Doukas, Stefanos Zafciriou, and Viktoriia Sharmanska. HcadGAN: One-shot neural head synthesis and editing. In ICCV, 2021.
[7] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021.
[8] Guy Gafni, Justus Thies, Michael Zollhofer, and Matthias Nicßner. Dynamic neural radiance fields for monocular 4D facial avatar reconstruction. In CVPR, 2021.
[9] Jiahao Geng. Tianjia Shao, Youyi Zheng, Yanlin Weng, and Kun Zhou. Warp-guided GANs for single-photo facial animation. TOG, 2018
[10] Kuangxiao Gu, Yuqian Zhou, and Thomas S Huang. FLNet:
Landmark driven fetching and learning network for faithful talking facial animation synthesis. In AAAI, 2020.
[1I] Sungjoo Ha, Martin Kersner, Beomsu Kim, Seokjun Seo, and Dongyoung Kim. MarioNETle: Few-shot face reenactment preserving identity of unseen largets. In AAAI, 2020.
[12] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to local Nash equilibrium. In NeurlPS, 2017.
[13] Jie Hu, Li Shen, and Gang Sun. Squecze-and-excitation networks. In CVPR, 2018.
[14] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. CCNet: Criss-cross attention for semantic segmentation. In ICCV, 2019.
[15] Drew A Hudson and Larry Zitnick. Generative adversarial transformers. In ICMI, 2021.
[16] Amir Jamaludin, Joon Son Chung, and Andrew Zisserman. You said that?: Synthesising talking faces from audio. IJCV, 2019.
[17] Yifan Jiang, Shiyu Chang, and Zhangyang Wang. TransGAN: Two pure transformers can make one strong gan, and that can scale up. NeurlPS, 2021.
[18] Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Unterthiner, and Xiaohua Zhai. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[19] Hang Li, Wenbiao Ding, Yu Kang, Tianqiao Liu, Zhongqin Wu, and Zitao Liu. CTAL: Pre-training cross-modal transformer for audio-and-langee representations. In EMNLP, 2021.
[20] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[21] Yuval Nirkin, Yosi Keller, and Tal Hassner. FSGAN: Subject agnostic face swapping and reenactment. In ICCV, 2019.
[22] Niki Parmar, Ashish Vaswani. Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer. Alexander Ku, and Dustin Tran. Image transformer. In /CML, 2018
[23] Albert Pumarola. Antonio Agudo, Aleix M Martinez, Alberto Sanfeliu, and Francese Moreno-Noguer GANimation: Anatomically aware facial animation from single image. In ECCV, 2018.
10

---

[24] Xuanchi Ren and Xiaolong Wang. Look outside the room: Synthesizing a consistent long-term 3D scene video from a single image. arXiv preprint urXiv:2203.09457, 2022.
[25] Robin Rombach, Patrick Esser, and Björ Ommer. Geometry-free view synthesis: Transformers and no 3D priors. In ICCV, 2021.
[26] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In NeurIPS, 2019.
[27] Aliaksandr Siarohin, Stéphane Lathuilière. Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. Animating arbitrary objects via deep motion transfer. CVPR, 2019.
[28] Aliaksandr Siarohin, Oliver Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion representa- tions for articulated animation. In CVPR, 2021.
[29] Yang Song, Jingwen Zhu, Dawei Li, Andy Wang, and Hairong Qi. Talking face generation by conditional recurrent adversarial network. In IJCAI, 2019.
[30] Supasorn Suwajanakom, Steven M Seitz, and aamellmacherSShlizemmee Synthesizing Obama: learning lip syne from audio. TOG, 2017.
[31] Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: Image synthesis using neural textures. TOG, 2019.
[32] Justus Thies, Michael Zollhöfer, Matthias NieBner, Levi Valgaerts, Mare Stamminger, and Christian Theohalt. Real-time expression transfer for facial reenactment. TOG, 2015.
[33] Justus Thies, Michael Zollhofer, Mare Stamminger, Christian Theobalt, and Matthias Nießner. Face2Face: Real-time face capture and reenactment of rgb videos. In CVPR, 2016.
[34] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In JCML, 2021.
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrlPS, 2017.
[36] Daniel Vlasic, Matthew Brand, Hanspeter Pfister, and Jovan Popovic. Face transfer with multilinear models. TOG, 2005.
[37] Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic. Realistic speech-driven facial animation with GANs. IJCV, 2019.
[38] Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang. Residual attention network for image classification. In CVPR, 2017.
[39] Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan Yuille, and Liang Chich Chen. Axial- Liang-Chieh DeepLab: Stand-alous axial-attention for panoptic segmentation. ECCV, 2020.
[40] Ting- Ting-Chun Wang. Ming-Yu Yu Liu, Andrew Tao, Guilin Liu, Jan Kautz, and Bryan Catanzaro. Few-shot video-to-video synthesis. In NeurlPS, 2019.
[41] Ting-Chun Wang, Arun Mallya, and Ming-Yi Yu Liu. One-shot free-view neural talking-head synthesis for video conferencing. In CVPR, 2021.
[42] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transforer:; A versatile backbone for dense prediction without convolutions. In ICCV, 2021.
[43] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018.
[44] Olivia Wiles, A Sophia Koepke, and Andrew Zisserman. X2Face: A network for controlling face generation using images, audio, and pose codes. In ECCV, 2018.
[45] Sanghyun Woo, Jongchan Park, Joon- Young Lec, and In So Kweon. CBAM: Convolutional block attention module. In ECCV, 2018.
[46] Wayne Wu, Yunxuan Zhang, Cheng Li. Chen Qian, and Chen Change Loy. ReenactGAN: Learning to reenact faces via boundary transfer. In ECCV, 2018.

---

[47] Haoféi Xu, Jiaolong Yang, Jianfei Cai, Juyong Zhang, and Xin Tong. High-resolution optical flow from ID attention and correlation. In ICCV, 2021.
[48] Xing Xu, Tan Wang. Yang Yang. Lin Zuo, Fumin Shen, and Heng Tao Shen. Cross modal attention with semantic consistence for image- text matching. IEEE transactions on neural networks and learning systems, 2020.
[49] Linwei Ye, Mrigank Rochan, Zhi Liu, and Yang Wang. Cross-modal self-attention network for referring image segmentation. In CVPR, 2019.
[50] Egor Zakharov, Aleksei Ivakhnenko, Aliaksandra Shysheya, and Victor Lempitsky, Fast bi-layer neural synthesis of one-shot realistic head avatars. In ECCV, 2020.
[51] Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-shot adversarial learning of realistic neural talking head models. In ICCV, 2019.
[52] Kaipeng Zhang. Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 2016.
[53] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018
[54] Hang Zhou, Yu Liu, Ziwei Liu, Ping Luo, and Xiaogang Wang. Talking face generation by adversarially disentangled audio-visual representation. In AAAI, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
section 5
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No] The datasets used are publically available and URLs are in the supplementary materials. We will release the code after acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See provided supplementary material.
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We have reported standard deviations for ablations in the supplementary material.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] See provided supplementary material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes], when and where provided.
(c) Did you include any new assets either in the supplemental material or as a URL? [No]
No new assets are being used/released as part of this work.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
12

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] All datasets we use were obtained from public YouTube videos under the Creative Commons license.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes] See provided supplementary material.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes] See provided supplementary material.
13


Reviews:
[{"review_id": "z7U1Bh286uI", "reviewer": "Reviewer_U2o1", "summary": "This paper presents a novel method for transferring the motion of a driving video to a subject via an image set of the subject. By using a cross-model attention layer for directly generating warped features, the proposed method differs from most previous methods which find explicit correspondences and warp images accordingly. By using the proposed mechanism, the proposed method can benefit from multiple source images without resorting to ad hoc approaches and suffering from sub-optimal results such as flicker or blur.", "questions": "Could you please provide more details regarding the procedure for adding additional keys and values?\n\nIs there a reason why the ablation study was conducted at a reduced resolution?\n", "limitations": "The paper discusses the societal impacts adequately. The method may fail if it is required to fill in a large amount of missing information. Additionally, it may encounter problems when dealing with extreme expressions not present in the training data.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the paper appears to be very promising. The paper describes a simple yet effective method of generating an animation from a set of images. In the case of multiple images, the proposed method clearly outperforms other methods. Though the proposed attention layer is commonly used in many applications, its use in this problem setting and the overall architecture design are novel.\n\n### Strengths ###\n* The proposed method is both simple and effective. The core of the algorithm is the cross-model attention layer. Although the idea has been used extensively in many applications, its use in this setting is novel to my knowledge.\n\n* The results of the experiments are promising and convincing. The proposed method was compared with a couple of SOTA methods using multiple datasets. A major advantage of the proposed method is its ability to utilize multiple images simultaneously and effectively.\n\n* The paper is well written and enjoyable to read. The supplementary material, in particular the video, is very well done.\n\n### Weaknesses ###\n* The cross-modal attention layer does not present much technical innovation. However, the overall flow and architecture design are novel.\n\n* It is not entirely clear how to add additional keys and values. It would be helpful if a more detailed description was provided.\n\n* It is not clear why the ablation study was conducted at a reduced resolution.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XY8bArHRtY_", "reviewer": "Reviewer_hL38", "summary": "This paper concerns the task of animating a set of one or more source images driven by a target (or driving) video; applications include efficient video compression (e.g. for video calls) or retargeting a video.\n\nOut of a four stage pipeline, the authors note that their work focuses on the warping of image features from the source image(s) to the output. Previous work provides feature representation extraction from the source and driving images, we well as decoding from from the warped feature representation to the final output image.\n\nThe main contribution is the direct application of a dot-product attention based transformer to perform implicit warping between the driving and source image(s); this is an elegant solution that requires no special treatment for different numbers of source images. This is contrasted against previous approaches that only consider a single source image without an obvious method to extend to multiple sources images that avoids undesirable image operations (e.g. averaging over output images or hard transitions between source images).\n\n\n\n\n\n\n", "questions": "Additional keys: The use of the additional keys seems relatively critical to the method (particularly some of the outputs where occlusion means there are no suitable source regions to use). The ablation study for this is relatively simple (as a binary consideration) whereas there are a number of parameters for the additional keys. At the same time, the ablation study results are not very conclusive (Table 5 in the supplement); the improvements for the majoritiy of the metrics are well within the associated standard error values provided. It would seem that a further investigation of this area is waranted - as the authors point out, it should be vital to the success but this doesn't seem to follow through to the ablation study metrics - please could the authors comment on this? Is there not a need for further study here and how were the parameters (e.g. the number of additional keys) derived? It would also seem that these additional keys become very application specific (e.g. synthesising eyes); does this limit the general applicability of the results across datasets?\n\nError bars/Variance: Given the contrast between Table 4 and Table 5 (where std errors are provided) it seems necessary to provide the std errors to the other results tables - it is difficult to judge the significance of the improvement and we would suspect, from the Table 5 std dev, that these improvements may not be particularly significant given the dataset (e.g. table 1 and 2). Please can these values be provided and the authors comment on this - if this is not important, please could the authors indicate why it should be neglected in the analysis of the quantitative results?\n\nMismatch in the results: Please could the authors explain why the first columns of Table 2 don't match the corresponding columns in Table 1 (i.e. the results for single source images)? Sorry if I have missed something.\n\nApplication Level Evaluation: Please could the authors comment on the intended application for such work? For example, is the goal retargetting or also efficiency for (e.g.) video calls? It seems the information required per frame (the A matrix) exceeds the original image dramatically therefore there would not be a reduction in bandwidth if the result is to be sent over a communications channel? How would this compare with the need to send the warping field (which might be readily compressible)? The authors specifically cite this in lines 257-258 as an application and it seems to me that no data compression has been achieved? Or is it the case that we don't need to send q x d and that the driving keypoint network is fixed so only the keypoint input data can be sent?", "limitations": "The authors flag that operating under high levels of occlusion will be very challenging (this makes sense and the arguments that multiple images help with this is reasonable - perhaps a consideration of how the source images should be chosen would be helpful)? There is also discussion that computational/memory complexity could be improved (under the observation that there is perhaps significant sparsity that has not been exploited).\n\nComments that might want to be considered by the authors but I don't believe warrant ethical review:\n\nIn consideration of societal factors, the authors point out that the final video is generated from the source image and therefore they would not expect racial bias in the results; whilst this makes sense for one part, I think this misses the consideration that the method relies on the key point, representation networks and feature decoders to operate well and these are all trained on data - if these datasets contain significant bias would we not expect the performance to vary as a result?\n\nThe authors included the evaluation protocol and payment for the human evaluation study; I noted that the estimated hourly rate provided ($5/hour) is demonstrably lower than the US minimum wage (around $7.5/hour).\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nWhilst the direct application of a transformer, the resulting approach is an elegant solution that prioritises simplicity for which I commend the authors. I found the presentation of the method to be straight-forward and clear - I believe the use of the different components is logical and well justified. \n\nThe proposed method offers a number of advantageous properties over previous approaches whilst removing complexity. I believe it is fair to say that the empirical evaluation should be of critical consideration in determining the merits of the approach as it determines whether or not the the hand specified architectural and modelling decisions (or implicit priors) in previous approaches can be relaxed in favour of a data driven approach. For this reason, this is the main focus of the review.\n\nIn terms of the quality of the results, I have a number of questions regarding the quantitative evaluation. This also speaks to the significance of the results in terms of how well we expect the method to generalise and how well it would perform at downstream tasks. There are a number of points that I currently see to be weaknesses in the evaluation; please see the questions below for my comments on this which include specific questions for the authors to clarify.\n\nCaveat: This is not my main area of research; whilst I am familiar with the methods and concepts being brought to bear, I am not familiar with the literature in the application area and cannot speak to the novelty or otherwise for the application; I cannot guarantee that there is not related work that I have missed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LCCFpbT_bTa", "reviewer": "Reviewer_BA5v", "summary": "The paper presents a method for warping-based image animation using multiple source images (i.e., images of the scene to be animated) and a driving video (i.e., image frames from the same or similar scene used to drive the animation). Since previous methods have focused mostly on animating scenes with a single source image, including multiple source images in these frameworks is non-trivial. Yet, it makes sense that multiple source images could improve performance since they may provide textures from occluded regions that can be used in the animation. The key technical contribution is to combine existing keypoint-detection methods with an attention-based warping. This allows the model to attend to image features across all input frames in order to produce a dense grid of warped output features. The method is evaluated for face and body animation and demonstrates state-of-the-art results for both single and multiple input source images.\n", "questions": "- The main paper doesn't seem to describe how the outputs of the cross-modal attention are upsampled to the final output resolution. While this is clear from the supplement, I think a sentence should be added for completeness.\n\n- I wasn't entirely sure what the keypoint representation is. In the FOMM paper, the keypoint predictor produces a heatmap that is passed through a softmax to produce a probability distribution whose mean is used to calculate a pixel location. Here, the keypoints seem to be passed directly into a convolutional layer so are the heatmaps (pre- or post-softmax) used for the keypoint representation?\n\n- How much does the number of keypoints matter in this case? I see that 20 keypoints are used for all experiments, but I'm curious how much this hyperparameter matters.\n\n- How are the extra/added key-value pairs implemented? Are these simply learnable parameter vectors of dimension d? Or are they conditioned somehow on the input source images?\n\n- I was curious why the FOMM model was selected for baseline comparisons rather than the more recent work from the same authors which addresses some of the shortcomings of the original model (Siarohin et al., ref. below). If I look at the quantitative results from that paper, they seem close to the proposed method, e.g., for AKD on the TED Talk dataset. Is there a reason this baseline wasn't included?\n\n    - Siarohin, Aliaksandr, et al. \"Motion representations for articulated animation.\" Proc. CVPR. 2021.\n", "limitations": "- The paper briefly touches on computational efficiency, but I think this limitation could be clarified a bit more. How does the proposed attention-based model compare to the baseline methods in terms of efficiency? I would guess that FOMM or FV2V have a much higher framerate, and the proposed model achieves better quality, but with some penalty to efficiency. Additional clarification on this point would be helpful.\n\n- There are obvious ethical concerns with the proposed method, especially with respect to potential misuse for deep fakes. Still, I think the authors address this point reasonably well in the societal impact section while highlighting positive benefits of this class of technique.\n\nTypos\nFig. 3: \"Coss-model\" -> \"Cross-modal\"\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strenghts*\n\n- The paper is well-written and the technical descriptions are clear and fairly easy to follow.\n\n- This method seems to elegantly and sensibly address the challenge of warping multiple input frames into the single output using information from the driving video. The implicit warping formulation using the attention mechanism helps to solve some limitations of previous methods (which could only support explicit, linear warping). I could see this approach being relevant to other problems in image-based rendering or sensor fusion.\n\n- The method clearly outperforms the other baselines and the additional results (included in the videos and webpage) are compelling.\n\n*Weaknesses*\n\n- While the paper is generally easy to follow, there are some technical details missing from the main paper (see questions below).\n\n- Some aspects of the related work section could be improved. For example the \"Image Animation\" section notes that \"none of [these frameworks] is designed to take advantage of complementary information available in different input images.\" Yet, there are a number of papers that use multiple input images, e.g., for facial animation [30-33]. In these works, multiple input images can be used to synthesize a dense texture map of the face, which can then be animated using a 3DMM style model. Some rewording here could help make this section more accurate.\n\n- One other limitation of the method seems to be scalability. While there are results demonstrated on 512x512 images, achieving higher resolution seems difficult since the attention-based mechanism happens at a relatively low resolution (the 64x64 resolution is mentioned in the paper). The features predicted after the attention module are then upsampled afterwards to produce the a higher resolution output, but I suspect there's a limit to how much upsampling can be done while retaining good quality.\n\nOverall I think these are relatively minor weaknesses; the method appears sufficiently novel and original, and the results seem quite strong. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept