Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper looks into the topic of generating gesture videos from given speech and images. The proposed approach learns the representations of gesture motions as a codebook, and learns the mapping from the speech to the codebook. With the learned representation and the speech-to-codebook mapping, the approach can animate the given image with gestures based on the given speech. The speech representation for the image is based on previous work “Motion Representations for Articulated Animation”. Since the codebook provides quantized representation and can miss some details, the approach includes an additional residual learning approach to further construct the image details from speech. User studies showed improvement compared to previous work. Ablation studies are provided to show the effectiveness of the proposed techniques.
strengths_and_weaknesses: Strength:
The approach generates good quality gesture videos based on the given image, and showed improvement compared to previous work.

Ablation studies demonstrated the effectiveness of the proposed techniques.


Weaknesses:
The idea of learning representations and learning the mapping from utterances to the representations are not new, and the motion representations used in the work is proposed in the previous work. The main novelty is applying the idea on gesture generations.

It is not clear how the approach performs on speakers that are not in the training data.

questions: How well does the approach generalize to speakers that are not in the training data?
limitations: The authors have properly discussed the limitations.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper presents ANGIE, an approach for audio-driven image animation with specific focus on upper body gestures. ANGIE leverages an implicit motion model (MRAA) as the intermediate motion representation instead of operating on a landmark-based skeleton representation which allows it to work directly in the image domain. The problem is divided into 3 parts: a motion representation learning module, a motion prediction module, and a motion refinement module. First, a VQVAE is trained to discretize the motion into a fixed set of codes. This model learns a position-independent codebook of co-speech gestures. Next, a Co-Speech Gesture GPT learns to map audio onset features to the discrete codes of motion extracted from training videos. This GPT network is also trained with a residual loss which focuses on refining the motion predicted from the discrete codes.

The authors compare their approach to previous state-of-the-art audio-driven gesture generation networks and evaluate using objective metrics and subjective user studies. Overall, the proposed method outperforms the baselines. One of the baselines: HA2G tends to have better or similar beat consistency than the proposed method. The authors also perform ablations studies to compare different vector quantization strategies and confirming the benefit of the motion refinement step. The authors also analyze the quantized motion representations.
strengths_and_weaknesses: Strengths:

- This paper is the first to animate face and torso images without using a skeleton representation of the images and demonstrates improved performance over those approaches. 
- The method is another interesting application of VQ-VAE followed by a prediction model. The novelty also lies in the design of the VQVAE. Instead of naively learning a codebook for the motion features of MRAA, the authors transform their data to address constraints in their problem, such as learning a codebook from the lower triangular decomposition of their covariance matrix and using relative motion parameters instead of absolute parameters. Their decisions are also ablated in their evaluation showing the benefits of these transformations.

Weaknesses:

There are some minor issues with the paper in terms of details. 
- The subjective evaluation involves some kind of mean opinion score (MOS). The authors do not add 95% confidence intervals or a standard deviation to their scores. This is important to see the spread in the scores given to each system.
- Some details regarding audio feature extraction are missing in the paper. Please refer to the questions section.

questions: - L140: Shouldn't the output frames be from I^hat_(2:N) if I_1 is given? If that is the case, perhaps the authors can just say I_0 is given and then the rest of the paper need not be changed.
- How is the onset feature strength computed? What is the window size or hop size for spectral flux computation? I don't understand what the 426 dimensions are in the onset feature. Onset strength should be a one dimensional feature per frame. Do you mean to say that the training videos are a fixed length and have 496 audio frames? It does seem like the videos are fixed length (96 frames @ 25 fps) but then you also need to mention some technical specifications: sample rate of the audio.
- Similarly, how are the mfcc features computed? Again, I don't understand the dimensions of the MFCCs. Are you using 12 mfccs or 28 mfccs. In any case, if the onset strength length is 426, why is the temporal dimension of the mfccs so small? And you mention that they are computed with a window size of 10 ms. What about hop size? What about fft block size?
- Does the network also infer mouth movement? It would seem that way since there is no special treatment of head/face from the body gestures, but then I am really surprised by the quality of the mouth motion in the sample videos shared. Why have the authors not also evaluated using audio-driven face animation metrics?

limitations: The authors adequately discuss the limitations of their work along with ethics considerations of such algorithms.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper concerns synthesis of co-speech gestures, gestures are coded in terms of entries in a VQ code book, and a GPT-like model is used to generate the sequence of symbols corresponding to input audio. In the generation, first the broad gestures are created, and then these are refined to improve the overall fidelity.
strengths_and_weaknesses: + To construct the training data, an existing dataset was augmented with new features, and this will be made available.
+ The generated image sequences look compelling.
+ The approach was compared against, and beat, several baselines.

- The paper refers to gestures as “common motion patterns” and “rhythmic dynamics”.  What do these mean?  Use standard terminology from the co-speech gesture literature.
- It looks like the models are over fit to the specific sequences.  For example, I think it’s the Kubinec sequence: when the speaker is referring to information on the screen to the speaker’s left, the speaker points and gestures towards the screen.  Why?  Is the screen always to the speaker’s left?  There is no language model and there is no explicit capturing of the speech content, so it is not clear to me why we would see this behavior.
- I believe you are training a separate model for each speaker, but you criticized prior work for limited generalization ability.
- The paper considers gesture only in a very limited context - talk show host speaking monologue to camera.  How well do you expect this to generalize, e.g., to conversational gestures that are likely more subtle and drawn from a larger lexicon?
- There are broad categories of gesture that this approach will fail to capture.  Specifically without the use of language, iconics, metaphorics, and deictics, are unlikely to be generated.  You ought to clarify that the focus is specifically only on beats.

questions: The audio clips are variable length, are these clipped to be a fixed size?

How do you ensure consistency when generating longer sequences? Only short segments are shown in the demo video, it would be nice to see longer generated sequences.

I was confused by “position-irrelevant” because position is important as position gives the sense of scale in gesture space. It was cleared up in later text that you mean image location invariant.

In the demo video, the speaker refers to “four quadrants” and it looks like the model produces gestures that seem to point to four quadrants. This seems suspicious since the “common motion patterns” are generated from just the speech envelope, and the refined motion from the MFCCs.

limitations: Clarify that the approach is specifically targeting only beat gestures.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper works on audio-driven gesture image generation, which receives a still image and audio to generate a sequence of images (video). Compared to previous works, it does not rely on skeletal pose annotation to prevent error accumulation caused by cascading of synthesis networks. The proposed system (named as ANGIE) is consisted of VQ-motion extractor and Co-speech GPT, each mapping the image to vector quantized representation and transformer network autoregressively predict next frame's quantized representation, respectively.
strengths_and_weaknesses: Strenghts
- By vector-quantizing the images, the proposed network could enjoy powerful capability of the Transformer. This kind of framework might be also applied at various video generation networks.
- The motion refinement network to aid missing details of reconstructed images from quantized representation seems novel.
- Ablation studies (Table 3) clearly justifies the choice of the components.

Weaknesses
- The related works shown in section 2 is limited in the application viewpoint. It would be much better if authors could make a connection to slightly other domains that the proposed method could be also applied. In overall, the scope of the paper is too specific on the co-speech gesture image generation. To meet broader NeurIPS readership, I wish there would be some implications outside the main problem.
- Please do not cite Wikipedia article as a reference (line 303, Fleiss's Kappa statistic). Please briefly describe about the concept in the appendix.
questions: - Regarding the line 272 - "The motion estimator is pretrained for knowledge distillation" - please elaborate on this. The authors might want to describe such implementation details on the appendix.
- What is a new dataset that the authors have collected, as described in line 92? Is it simply an extension of PATS dataset by some post-processing steps?
limitations: It would be better if the authors could provide specific examples of the limitation of ANGIE (discussed in line 344).
review_title: null

Meta-review: metareview: This paper enjoyed a reasonable interaction between the authors and the reviewers, with the authors addressing the reviewers' concerns about the novelty of the proposed method, its specificity to the "talking head" scenario, the fact that the model is used in a speaker-dependent fashion, and some concerns about specific details in the writing. Three of the four reviewers responded to the authors during the discussion period, and the fourth reviewer acknowledged having read the rebuttal during the discussion between the AC and reviewers.

In the end, all reviewers recommend acceptance of the paper, citing the good performance of the model, the novelty of co-speech gesture generation in the image domain, and the nice design of the model (specifically, the VQ motion plus residual structure).

confidence: Certain
Decision: Accept