Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper concerns the task of animating a set of one or more source images driven by a target (or driving) video; applications include efficient video compression (e.g. for video calls) or retargeting a video.

Out of a four stage pipeline, the authors note that their work focuses on the warping of image features from the source image(s) to the output. Previous work provides feature representation extraction from the source and driving images, we well as decoding from from the warped feature representation to the final output image.

The main contribution is the direct application of a dot-product attention based transformer to perform implicit warping between the driving and source image(s); this is an elegant solution that requires no special treatment for different numbers of source images. This is contrasted against previous approaches that only consider a single source image without an obvious method to extend to multiple sources images that avoids undesirable image operations (e.g. averaging over output images or hard transitions between source images).







strengths_and_weaknesses: 
Whilst the direct application of a transformer, the resulting approach is an elegant solution that prioritises simplicity for which I commend the authors. I found the presentation of the method to be straight-forward and clear - I believe the use of the different components is logical and well justified. 

The proposed method offers a number of advantageous properties over previous approaches whilst removing complexity. I believe it is fair to say that the empirical evaluation should be of critical consideration in determining the merits of the approach as it determines whether or not the the hand specified architectural and modelling decisions (or implicit priors) in previous approaches can be relaxed in favour of a data driven approach. For this reason, this is the main focus of the review.

In terms of the quality of the results, I have a number of questions regarding the quantitative evaluation. This also speaks to the significance of the results in terms of how well we expect the method to generalise and how well it would perform at downstream tasks. There are a number of points that I currently see to be weaknesses in the evaluation; please see the questions below for my comments on this which include specific questions for the authors to clarify.

Caveat: This is not my main area of research; whilst I am familiar with the methods and concepts being brought to bear, I am not familiar with the literature in the application area and cannot speak to the novelty or otherwise for the application; I cannot guarantee that there is not related work that I have missed.
questions: Additional keys: The use of the additional keys seems relatively critical to the method (particularly some of the outputs where occlusion means there are no suitable source regions to use). The ablation study for this is relatively simple (as a binary consideration) whereas there are a number of parameters for the additional keys. At the same time, the ablation study results are not very conclusive (Table 5 in the supplement); the improvements for the majoritiy of the metrics are well within the associated standard error values provided. It would seem that a further investigation of this area is waranted - as the authors point out, it should be vital to the success but this doesn't seem to follow through to the ablation study metrics - please could the authors comment on this? Is there not a need for further study here and how were the parameters (e.g. the number of additional keys) derived? It would also seem that these additional keys become very application specific (e.g. synthesising eyes); does this limit the general applicability of the results across datasets?

Error bars/Variance: Given the contrast between Table 4 and Table 5 (where std errors are provided) it seems necessary to provide the std errors to the other results tables - it is difficult to judge the significance of the improvement and we would suspect, from the Table 5 std dev, that these improvements may not be particularly significant given the dataset (e.g. table 1 and 2). Please can these values be provided and the authors comment on this - if this is not important, please could the authors indicate why it should be neglected in the analysis of the quantitative results?

Mismatch in the results: Please could the authors explain why the first columns of Table 2 don't match the corresponding columns in Table 1 (i.e. the results for single source images)? Sorry if I have missed something.

Application Level Evaluation: Please could the authors comment on the intended application for such work? For example, is the goal retargetting or also efficiency for (e.g.) video calls? It seems the information required per frame (the A matrix) exceeds the original image dramatically therefore there would not be a reduction in bandwidth if the result is to be sent over a communications channel? How would this compare with the need to send the warping field (which might be readily compressible)? The authors specifically cite this in lines 257-258 as an application and it seems to me that no data compression has been achieved? Or is it the case that we don't need to send q x d and that the driving keypoint network is fixed so only the keypoint input data can be sent?
limitations: The authors flag that operating under high levels of occlusion will be very challenging (this makes sense and the arguments that multiple images help with this is reasonable - perhaps a consideration of how the source images should be chosen would be helpful)? There is also discussion that computational/memory complexity could be improved (under the observation that there is perhaps significant sparsity that has not been exploited).

Comments that might want to be considered by the authors but I don't believe warrant ethical review:

In consideration of societal factors, the authors point out that the final video is generated from the source image and therefore they would not expect racial bias in the results; whilst this makes sense for one part, I think this misses the consideration that the method relies on the key point, representation networks and feature decoders to operate well and these are all trained on data - if these datasets contain significant bias would we not expect the performance to vary as a result?

The authors included the evaluation protocol and payment for the human evaluation study; I noted that the estimated hourly rate provided ($5/hour) is demonstrably lower than the US minimum wage (around $7.5/hour).
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a novel method for transferring the motion of a driving video to a subject via an image set of the subject. By using a cross-model attention layer for directly generating warped features, the proposed method differs from most previous methods which find explicit correspondences and warp images accordingly. By using the proposed mechanism, the proposed method can benefit from multiple source images without resorting to ad hoc approaches and suffering from sub-optimal results such as flicker or blur.
strengths_and_weaknesses: Overall, the paper appears to be very promising. The paper describes a simple yet effective method of generating an animation from a set of images. In the case of multiple images, the proposed method clearly outperforms other methods. Though the proposed attention layer is commonly used in many applications, its use in this problem setting and the overall architecture design are novel.

### Strengths ###
* The proposed method is both simple and effective. The core of the algorithm is the cross-model attention layer. Although the idea has been used extensively in many applications, its use in this setting is novel to my knowledge.

* The results of the experiments are promising and convincing. The proposed method was compared with a couple of SOTA methods using multiple datasets. A major advantage of the proposed method is its ability to utilize multiple images simultaneously and effectively.

* The paper is well written and enjoyable to read. The supplementary material, in particular the video, is very well done.

### Weaknesses ###
* The cross-modal attention layer does not present much technical innovation. However, the overall flow and architecture design are novel.

* It is not entirely clear how to add additional keys and values. It would be helpful if a more detailed description was provided.

* It is not clear why the ablation study was conducted at a reduced resolution.

questions: Could you please provide more details regarding the procedure for adding additional keys and values?

Is there a reason why the ablation study was conducted at a reduced resolution?

limitations: The paper discusses the societal impacts adequately. The method may fail if it is required to fill in a large amount of missing information. Additionally, it may encounter problems when dealing with extreme expressions not present in the training data.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents a method for warping-based image animation using multiple source images (i.e., images of the scene to be animated) and a driving video (i.e., image frames from the same or similar scene used to drive the animation). Since previous methods have focused mostly on animating scenes with a single source image, including multiple source images in these frameworks is non-trivial. Yet, it makes sense that multiple source images could improve performance since they may provide textures from occluded regions that can be used in the animation. The key technical contribution is to combine existing keypoint-detection methods with an attention-based warping. This allows the model to attend to image features across all input frames in order to produce a dense grid of warped output features. The method is evaluated for face and body animation and demonstrates state-of-the-art results for both single and multiple input source images.

strengths_and_weaknesses: *Strenghts*

- The paper is well-written and the technical descriptions are clear and fairly easy to follow.

- This method seems to elegantly and sensibly address the challenge of warping multiple input frames into the single output using information from the driving video. The implicit warping formulation using the attention mechanism helps to solve some limitations of previous methods (which could only support explicit, linear warping). I could see this approach being relevant to other problems in image-based rendering or sensor fusion.

- The method clearly outperforms the other baselines and the additional results (included in the videos and webpage) are compelling.

*Weaknesses*

- While the paper is generally easy to follow, there are some technical details missing from the main paper (see questions below).

- Some aspects of the related work section could be improved. For example the "Image Animation" section notes that "none of [these frameworks] is designed to take advantage of complementary information available in different input images." Yet, there are a number of papers that use multiple input images, e.g., for facial animation [30-33]. In these works, multiple input images can be used to synthesize a dense texture map of the face, which can then be animated using a 3DMM style model. Some rewording here could help make this section more accurate.

- One other limitation of the method seems to be scalability. While there are results demonstrated on 512x512 images, achieving higher resolution seems difficult since the attention-based mechanism happens at a relatively low resolution (the 64x64 resolution is mentioned in the paper). The features predicted after the attention module are then upsampled afterwards to produce the a higher resolution output, but I suspect there's a limit to how much upsampling can be done while retaining good quality.

Overall I think these are relatively minor weaknesses; the method appears sufficiently novel and original, and the results seem quite strong. 
questions: - The main paper doesn't seem to describe how the outputs of the cross-modal attention are upsampled to the final output resolution. While this is clear from the supplement, I think a sentence should be added for completeness.

- I wasn't entirely sure what the keypoint representation is. In the FOMM paper, the keypoint predictor produces a heatmap that is passed through a softmax to produce a probability distribution whose mean is used to calculate a pixel location. Here, the keypoints seem to be passed directly into a convolutional layer so are the heatmaps (pre- or post-softmax) used for the keypoint representation?

- How much does the number of keypoints matter in this case? I see that 20 keypoints are used for all experiments, but I'm curious how much this hyperparameter matters.

- How are the extra/added key-value pairs implemented? Are these simply learnable parameter vectors of dimension d? Or are they conditioned somehow on the input source images?

- I was curious why the FOMM model was selected for baseline comparisons rather than the more recent work from the same authors which addresses some of the shortcomings of the original model (Siarohin et al., ref. below). If I look at the quantitative results from that paper, they seem close to the proposed method, e.g., for AKD on the TED Talk dataset. Is there a reason this baseline wasn't included?

    - Siarohin, Aliaksandr, et al. "Motion representations for articulated animation." Proc. CVPR. 2021.

limitations: - The paper briefly touches on computational efficiency, but I think this limitation could be clarified a bit more. How does the proposed attention-based model compare to the baseline methods in terms of efficiency? I would guess that FOMM or FV2V have a much higher framerate, and the proposed model achieves better quality, but with some penalty to efficiency. Additional clarification on this point would be helpful.

- There are obvious ethical concerns with the proposed method, especially with respect to potential misuse for deep fakes. Still, I think the authors address this point reasonably well in the societal impact section while highlighting positive benefits of this class of technique.

Typos
Fig. 3: "Coss-model" -> "Cross-modal"
review_title: null

Meta-review: metareview: Consistent reviews, both in content and in score.

The cross-identity motion transfer is a good test of the paper's capability -- it would improve the paper to provide more such examples, which are clearly more challenging than the same-identity case.

The concerns about the limited diversity of example subjects mentioned by R1 are indeed relevant.  The video examples are all male, with quite light skin tone.  Please include examples with female subjects, darker (Fitzpatrick 6+) skin tone, and other ethnicities.  To be clear: the rebuttal's current response "we will emphasize that training on a diverse dataset is a must" does not go far enough.  It is very important that qualitiative examples are shown, even more important than that the test datasets are diverse.  If the results are less good, efforts should be made before NeurIPS to improve them (e.g. by retraining), and if improvement is not possible, this should be very clearly stated in the limitations of the final copy and NeurIPS presentation/poster.

confidence: Certain
Decision: Accept