Audio-Driven Co-Speech Gesture Video Generation
Authors: Xian Liu
Qianyi Wu
Hang Zhou
Yuanqi Du
Wayne Wu
Dahua Lin
Ziwei Liu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Audio-Visual
Co-Speech Gesture
Video Generation

Abstract: Co-speech gesture is crucial for human-machine interaction and digital entertainment. While previous works mostly map speech audio to human skeletons (e.g., 2D keypoints), directly generating speakers' gestures in the image domain remains unsolved. In this work, we formally define and study this challenging problem of audio-driven co-speech gesture video generation, i.e., using a unified framework to generate speaker image sequence driven by speech audio. Our key insight is that the co-speech gestures can be decomposed into common motion patterns and subtle rhythmic dynamics. To this end, we propose a novel framework, Audio-driveN Gesture vIdeo gEneration (ANGIE), to effectively capture the reusable co-speech gesture patterns as well as fine-grained rhythmic movements. To achieve high-fidelity image sequence generation, we leverage an unsupervised motion representation instead of a structural human body prior (e.g., 2D skeletons). Specifically, 1) we propose a vector quantized motion extractor (VQ-Motion Extractor) to summarize common co-speech gesture patterns from implicit motion representation to codebooks. 2) Moreover, a co-speech gesture GPT with motion refinement (Co-Speech GPT) is devised to complement the subtle prosodic motion details. Extensive experiments demonstrate that our framework renders realistic and vivid co-speech gesture video. Demo video and more resources can be found in: https://alvinliu0.github.io/projects/ANGIE

Audio-Driven Co-Speech Gesture Video Generation
Xian Liu', Qianyi Wu', Hang Zhou', Yuanqi Du², Wayne Wu', Dahua Lin Ziwei Liu
Multimedia Laboratory, The Chinese University of Hong Kong "Monash University Cormell University *Shanghai Al Laboratory
SS-Lab, Nanyang Technological University
Abstract
Co-speech gesture is crucial for human-machine interaction and digital enter- tainment. While previous works mostly map speech audio to human skeletons (e.g.. 2D keypoints), directly generating speakers' gestures in the image domain remains unsolved. In this work, we formally define and study this challenging problem of audio-driven co-speech gesture video generation, i.e. using unified framework to generate speaker image sequence driven by speech audio. Our key insight is that the co-speech gestures can be decomposed into common motion patterns and subtle rhythmic dynamics. To this end, we propose novel framework, Audio-driveN Gesture vIdeo gEneration (ANGIE), to effectively capture the reusable co-speech gesture patterns well as fine-grained rhythmic movements. To achieve high-fidelity image sequence generation, we leverage an unsupervised motion representation instead of structural human body prior (e.g. 2D skeletons). Specifically, 1) we propose a vector quantized motion extractor (VQ-Motion Extractor) to summarize common co-speech gesture patterns from implicit motion representation to codebooks. 2) Moreover, a co-speech gesture GPT with motion refinement (Co-Speech GPT) is devised to complement the subtle prosodic motion details. Extensive experiments demonstrate that our framework renders realistic and vivid co-speech gesture video. Demo video and more resources be found www.lli..ggggttii github. ioriii//ooor wtoprrjjcts////OO/
in:
Introduction
During daily conversation among humans, speakers naturally emit co-speech gestures to complement the verbal channels and express their thoughts [17, 35, 56]. Such non-ven verbal behaviors ease speech comprehension [10, 58] and bridge the communicator's 's gap for better credibility [7, 54]. Therefore, equipping the social robot with conversation skills constitutes a crucial step to interaction. To achieve it, researchers delve into the task of co-speech gesture generation [21, 39, 62), where audio-coherent human gesture sequences are synthesized the form of structural human representation (e.g., skeletons). However, such representation contains appearance information of the target speaker, which is crucial for human perception. As demonstrated in audio-driven talking head synthesis [34, 65], generating real-world subjects in the image domain highly desirable. To this end, we explore the problem of audio-driven co-speech gesture video generation, i.e., using unified framework to generate speaker image sequence driven by speech audio (illustrated in Fig. 1).
Conventional methods require exhaustive human efforts to pre-define the speech-gesture pairs and connection rules for coherent result [11, 12]. With the development of deep learning, neural networks are leveraged to learn the mapping from encoded audio feature to human skeletons in a data-driven manner [21, 39, 62]. Notably, one category of approaches relies small-scale MoCap datasets in co-speech setting [16, 18, 48), which contributes to specific models with limited capacity and robustness. To capture more general speech-gesture correlations, another category of methods builds large training corpus by exploiting off-the-shelf pose estimators [9, 15] to label enormous online videos pseudo ground truth [21, 63]. However, the inaccurate pose annotations induce error
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Co-Speech Gesture Image Generator
Figure I: Illustration of Problem Setting. In this paper, we focus on audio-driven co-speech gesture video generation. Given an image with speech audio, we generate aligned speaker image sequence.
accumulation in the training phase, which makes the generated results unnatural. Besides, most previous works ignore the problem of co-speech gesture video generation. Only few works [21, 39] animate in the image domain as an independent post-processing step, which borrows from the existing pose-to-image generators [5, 13] to train on the target person's 's images. How to design a unified framework generate speaker image sequence driven by speech audio remains unsolved.
To effectively learn the mapping from audio to co-speech gesture video, we pinpoint two important observations from current studies: 1) hand-crafted structural human priors like 2D/3D skeletons would eliminate articulated human body region information. Such a zeroth-order motion representation fails to formulate first-order motion like local affine transformation image animation [44]. Besides, the error in structural prior labeling impairs cross-modal audio-to-gesturg learning [33]. 2) Motivated by previous linguistic studies [27, 47], the co-speech gestures could be dccomposed into common motion patterns and rhythmic dynamics, where the former ones refer to large-scale motion templates (e.g., periodically put hands up and down), while the latter ones play a refinement role to complement subtle prosodic movements and synchronize with speech audio (e.g., finger flickers).
We take inspiration from the above observations and propose novel framework Audio-driveN Gesture vIdeo gEneration (ANGIE) to generate co-speech gesture video. The key insight is to summarize common co-speech gesture patlerns from motion representation quantized codebooks and further refine subtle rhythmic details by motion residuals for fine fine-grained results. In particular, two modules are designed, namely VQ-Motion Extractor and Co-Speech GPT. In VQ-Motion Extractor, we utilize an unsupervised motion representation to depict the articulated human body and first-order gestures [45]. The codebooks are established to quantize the reusable common co-speech gesture patterns from unsupervised motion representation. To guarantee the validity of gesture patterns, we propose cholesky decomposition based quantization scheme to relax the motion component constraint. The position-irrelevant motion pattern is extracted as final quantization target to represent the relative motion. In this way, the quantized codebooks naturally contain rich common gesture pattern information. With the quantized motion code sequence, in Co-Speech GPT we use a GPT-like [40] structure to predict discrete motion patterns from speech audio. Finally, motion refinement network is used to complement subtle rhythmic details for fine-graine fine-grained results.
To summarize, our main contributions are three-fold: 1) We explore a challenging problem of audio- driven co-speech gesture video generation. To the best of our knowledge, we are the first to generate co-speech gesture in image domain with unified framework without any structural human body prior. 2) We propose the VQ-Motion Extractor to quantize the motion representation into common gesture patterns and the Co-Speech GPT to refine subtle rhythmic movement details. The codebooks naturally contain reusable motion pattern information. 3) Extensive experiments demonstrate that the proposed framework ANGIE renders realistic and vivid co-speech gesture video generation results.
2 Related Work
Co-Speech Gesture Generation. Synthesizing co-speech gesture has gained research interest in vision [3, 21, 29], graphies [4, 60, 62] and robotics [23, 25, 63] domains. Recent researches resort mapping in data-driven manner, with major focuses on below perspectives: 1) Dataset. One strand of methods use small-scale MoCap datasets to learn specific models [16, 18, 30, 42, 48, 51, 55], while another strand of works exploit off- -he-hh estimator to label enormous videos as structural prior [1-3, 21, 33, 39, 61-63]. The dataset scale pose annotation accuracy often acts as a trade-off in this task: A large amount of speech-gesture pairs facilitate the training of more general models with better capacity and robustness, yet error accumulation in annotations induces unnatural results. 2) Framework architecture. CNN-based [21]]
to deep neural networks to learn the

---

arganón [Cap dapll
Quantized Ay
Motion
Ap Decodde
Relative Diffecence
Bepresenarion
Codabook
Recover Skit
A.
Quantized dL
L.
Recover
Recover
Cholesay Decompositon
Relative Difference
Factoria
Covariance
argmin ca daill
VQ-Motion Extractor
Ca-Speech GPT
Mirion Estimator
Linear Layse
Sofoklas
Ap/SI. Dicodler
Frand
Trans
Selection
Insgee
M
Driving Aucho
R(x) Recidual Motion Refinement
Ridirectional
ff
R(L)
Ca-Sperch Mation Refinement
Figure 2: Overview of the Audio-driveN Gesture vIdeo gEneration (ANGIE) framework. In VQ-Motion Extractor, the cholesky decomposition with position-irrelevan design transforms the shift-translation μ and covariance to relative motion pattern representation of Ap and AL, which further quantized by codebooks to extract the common motion patterns. Given the driving audio and starting gesture codes, the Co-Speech GPT predicts the future motion fields. A Motion Refinement network further learns motion residuals to complement the subtle rhythmic dynamics.
RNN-based [62] and Transformer-base [6] frameworks show promising results. To further improve the diversity of generated gestures and grasp the fine-grained cross-modal associations, components like adversarial loss [21], VAE sampling [30, 59] and hierarchical encoder-decoder design [33] are proposed. 3) Input modality. Some approaches
single modality of speech audio [2, 19, 21- 23, 30, 39] or text transcription [3, 6, 25, 63] as input to drive the co-speech gesture, while some others use both modalities as stimuli for generation [1, 33, 62]. To ease the learning of implicit cross-modal mapping from speech to gesture and create more stable results, recent works involve auxiliary input modality such speaker style [2], pose mode [59] and motion template [39].
In this work, we take step further in the above three aspects: 1) For the dataset, we collect a new co-speech gesture dataset in image domain, where unsupervised motion representation is used to model articulated human body and bypass the inaccuracy from structural prior annotations. 2) For the architecture, a vector quantized (VQ) network with novel discretization scheme is proposed to extract valid relative motion patterns. We further devise a motion refinement network to complement subtle rhythmic dynamics. 3) For the input modality. we explicitly decouple the common motion pattern from co-speech gestures, which serves a similar role as motion template [39] to provide auxiliary input. However, our discrete codebook design more suitable for finite gesture patterns than continuous representation, which is also proven in recent cross-modal generation tasks [36, 41, 46]. Furthermore, we propose novel vector quantization network with cholesky decomposition scheme to extract the valid motion patterns. We improve the quantization scheme to encode the relative motion representation that is position (absolute location) irrelevant. motion refinement network is further devised to complement subtle rhythmic dynamics. Notably, our approach gives idea on how to deal with the constraints vector quantization and how to complement sequential results with missing details. Such design could prospectively provide insights for relevant domains like constrained vector guantization problem, cross-modall learning [32] and video generation tasks [46].
Video Generation. Traditional video-driven approaches for image animation can be categorized into supervised and unsupervised, where the supervised methods typically involve structural human body prior such as landmarks [8, 64] and 3D parametric models [20, 50]. while the unsupervised approaches design self supervieee tasks to animate unlabeled images 45, 57]. To facilitate broader applications, researchers explore audio-driven video generation, where one of the most relevant tasks is talking face generation [14, 38). Different from the strong correlations between audio and mouth shape, the mapping from audio to complicated co-speech motion is multi-modal and harder to learn. Most co-speech gesture studies synthesize human skeletons as final results (e.g., 2D keypoints), while only few works [21, 39] generate co-speech images in post- post-processing manner.

---

3 Our Approach
We present ANGIE that generates audio-driven co-speech gesture in image domain, where the speakers' image sequence is driven by speech audio as shown in Fig. 1. The whole pipeline is illustrated Fig. To make the content self-contained and narration clearer, we first introduce the preliminaries and problem setting in Sec. 3.1. Then, we present the VQ-Motion Extractor which extracts common co-speech gesture patterns as quantized codebooks in Sec. 3.2. Finally, we elaborate the Co-Speech GPT to complement subtle rhythmic dynamics for fine-grained results in Sec. 3.3.
3.1 Preliminaries and Problem Setting
Unsupervised Motion Representation. To achieve high-fidelity image animation, we take inspiration from MRAA [45] that uses an unsupervised motion representation to drive articulated objects. MRAA first estimates coarse motion representation from the source and driving frames, then predicts dense pixel-wise flow for image generation. Specifically, an encoder-decoder keypoint predictor produces K different heatmaps H' HH,,, H where K is region number and H" denotes the k-th image region. Afterwards, each heatmap is normalized by softmax operation, i.e. Σeez (2) = 1, where is the image pixel location and is the set of all pixels. The key insight behind MRAA is to represent each region's motion by affine transformation with a thift-translation component 1z R
component. The and the distribution C* of the k-th part can be calculated as:
Σ [H²(2)2 C* = (=)(= - T ZE2
(1)
ZEZ
where T is matrix transpose, C 22 measures the covariance of heatmap value. It naturally depicts the size and shape of an articulated region. To represent the affine transformation A E R222 of the k-th region, we apply singular value decomposition (SVD) C* and derive A as:
C -
A =
(2)
where unitary matrices UK, V and diagonal matrix E* are the SVD result of covariance matrix Ck The representation M extracted by motion estimator the concatenation of [u; C; A] E (x(+4++4) for K distinct regions, an image generation module with dense pixel-wise flow predictor synthesizes the final generation results. In this work, the motion representation M and image generation module G generally follow MRAA. We suggest the readers referring to [45] for more details.
We collect training data of massively- available speaking videos with clear co-speech gestures for natural self-reconstruution supervision. Specifically, given an (N + 1)-frame video clip V = {H ..... [(v)): the goal of our framework at the training stage is to predict the motion representation M(1:N) based on the first image frame (()) and vidco's accompanying audio sequence a = {(qi)...... ((v)) }. Further, the image generation module G reconstructs the video frames T(I.N)- At the inference stage, an arbitrary reference image with speech audio clip is provided to generate subsequent image frames. According to the observation in Sec. 1, we decompose the co-speech gestures into common motion patterns and subtle rhythmic dynamics. The overall training setting can be formulated as:
Problem Setting for Co-Speech Gesture Image Generation.
[ G(I[),, (a))
Mrhsshmicc M(LNN) Marter + MMuieyy (1:N)
(3)
where Mpallem denotes the gesture pattern (Sec. 3.2) and Mrhytlmic is rhythmic movement (Sec. 3.3).
3.2 Vector Quantized Motion Pattern Extractor
To decompose the co-speech gestures, we propose firstly extract the common motion patterns. However, three problems remain: 1) The gesture sequences are different from each other. While some motion sequences share the same action pattern, the dynamic details may vary lot. How to extract the major motion pattern despite the influence from minor prosodic movements? 2) The covariance matrix C' is symmetric positive definite (Eq. 1), which further constrains the range of affine matrix A. How to preserve such characteristic for valid gesture patterns? 3) Since the unsupervised motion representation is extracted in the image pixel space, is affected by the absolute location of each articulated region. How to represent the cosition-irrelewv motion pattern information?

---

Vector Quantized Motion Pattern Learning. Our solution to the first problem is to quantize the common motion pattern into a codebook. Since the gesture pattern is finite, it could be summarized to discrete codebook entries. Besides, each codebook entry refers to certain type of gesture pattern, which matches our goal to extract the common and reusable co-speech gesture patterns.
A naive way is to quantize the motion representation M separately as shift-translation covariance matrix C' and affine transformation A. Specifically, for T-frame co-speech gesture sequence I(I:T), transform it into (a:; C; (1.)) (Rxxx((++++) where [au; A] = . denotes motion representation of regions as in Sec. 3.1. We first build three codebooks D,, = {d / Dc = {dc.m I m=1 and DA = Am /m=mm== for each motion component respectively, where is codehook size. du.m. dc.. and A E R are the m-th entry of Ê-channel for each codehook. Then, three separate encoders E,., Ec and EA are utilized encode the corresponding context information into latent features of e {err>) i=1, ec = (ecr) and е {e IT" R xt, where T is the temporal dimension and is the channel dimension. Notably, we denote the i-th temporal feature of each motion component as Gn,i ec,, and e,,i- The feature encoding process can be formulated as:
E((((T))) Cj: (c(C(ii))) cc. EA(A(I)) CA-
Following the pipeline of VQ-VAE [53], we individually quantize eμ ec and e by substituting each temporal feature p,,i, eC,, and e,1,i to the nearest codebook entry da,, dc and A. as:
arg min || dull c ED,.
arg min |lec dell, c arg min ||eA dalb dxEDA
(5)
doEDe
quantige shift iiffttrrmssattonn J
quanttize covariance matris C
quantize offine transfonmation A
where eg (e (i. eg (eg (
R'x' are the quantized code sequence of length T" for each motion component. The i-th quantized code of each motion component is denoted as H... og and c,, respectively. Finally, three separate decoders D,, De and DA are leveraged to reconstruct the motion representations of each component as:
)T and e) (e
P(i.T) Dx(c))) Č(T.Ty = Dc(eg),
(6)
Such discrete representation also case the audio-to- seltii-to-gesttnn to-gesture learning (Sec. 3.3): Previous methods predict continuous output a harder regression problem. While we only need to predict features nearer to the correct codebook entry, which in essence resembles an easier classification problem.
Quantization Design for Valid Motion Representation. To extract valid gesture patterns, we have to preserve certain characteristics of motion representation. Especially, the covariance matrix C should be symmetric positive definite (Eq. 1), and the affine transformation A is determined by C through SVD (Eq. 2). Therefore, instead of naively quantize each component in Eq. 5, we propose to only quantize the shift-translation and covariance matrix C, while derive the affine transformation A with SVD. The only constraint is to guarantee that the covariance matrix C' is symmetric positive definite. To satisfy such requirement, we use the unique cholesky decomposition theorem [52]:
Theorem 1. For any real symmetric positive definite matrix C' E S there exists umique lower triangular matrix L with positive diagonal entries, such that C LLT.
0
In this way, we turn to quantize the lower triangular matrix L
L/2 [s )=) where the constraint is
much simpler 4 13 0. The updated quantization scheme with cholesky decomposition is:
e arg min l,,, d,lbb et. arg min ||e dull,
ED
dLEDL
quantize shifi-tramslation
quentize the lower triangular matris I,
where CL2 c. Dr and d, denote the encoded feature, quantized feature, codebook and codebook entry for factorial covariance L, respectively. A simple transformation of 4133 ReLU((1,3) + guarantees the diagonal entries to be positive, where e is a small positive number. The motion component C and can be further obtained by LLT and SVD calculation, respectively.
Motion Pattern. Another problem arises when we inspect the value of the motion representation: Since M is extracted in the image pixel space, the object location will affect
5

---

the element in M. For example, if a person poses the same gesture at different image regions, the motion component differs yet the underlying motion pattern remains the same. Thus we focus on image location invariant motion pattern representation. In particular, due to the linear additiveness, the relative shift-translation js between adjacent frames can represented as 4 = M4 - 11j-1, and the relative change of the lower triangular matrix is AL, I.j Tj-1 for any that with the uniqueness of cholesky decomposition, (L + Á corresponds to the sole covariance matrix C' (L + (L) + AL)", which further determines affine matrix by SVD. In this way, the term AL is sufficient to represent any relative affine transformation between two frames. We accordingly update the quantization scheme with position-irrelese irrelevant motion pattern representation as:
2, ... Note
App = arg min ||esa dapll: dapEDaP
q CAL
=
arg min |lCAl daill dAL.
(8)
quantize relotive sifftstrassaaton Ap
quantize relative lower triangular matris change AL
where e AL), AAp P(AnAL)) and dIAn,A are the encoded feature, quantized feature, codebook and entry for the relative shift-translation Au and factorial covariance AT,, respectively.
Overall Quantized Motion Pattern Learning. With the position-irrelevant motion pattern, the codebook naturally contains reusable common co-speech gesture patterns Mouutemm The encoders Eays EsL and the decoders Da Da, are jointly learned with the codebooks Da and DaL via:
Lve
Apl
|g||cn/l اللl | 31lea, sgic
AL
AL [sg|ea]] - + 82 fel sgle
(9)
where sg denotes the stop gradient operation, 31 and B3 are two weight balancing coefficients.
3.3 Co-Speech Gesture GPT with Motion Refinement
Co-Speech Gesture GPT Network. With the position-irrelevant motion pattern of valid quantization design, each co-speech gesture clip can be transformed into discrete representation. We then learn speech gesture GPT network to map from speech audio a(:T) to quantized code sequences (1.(1) and c -..,(1:77- Specifically, we extract audio features aonser with onset strength information, which more suitable for cross-modal pattern learning [46, 49]. Then, a feature embedding layer with positional embedding is leveraged to obtain the tokens for audio onset features, quantized relative frlt--translation and qunatized relative factorial covariance. Further, we encode cross- coossaattnttion information with series of transformer layers. Finally, followed by linear transformation with softmax activation, the M-ddimensional output denotes the probability of each quantization code at that time step. The whole co-speech gesture GPT is trained with cross-entropy loss Leg. Such design enables us to predict and sample future quantization code e2, and eA, with speech audio.
Motion Refinement by Learning Residuals. Now that
reconstruct the relative shift- translation Åp and factorial covariance change AL by VQ- VAE decoding. Given μ4 and Lı extracted from the initial image frame [(), the absolute shift-transse translatinn and factorial covariance for the j-th
frame can be calculated as ju, = μ1 + E/=, Ayi; and L, = L1 + E/=, ÁL respectively. However, since the quantized codebook is designed to only represent the large- common motion pattern information, while fine-gre grained rhythmic details are omitted. Therefore, we propose to refine the co-speech movements by learning residual terms. Concretely, extract the audio mfcc features mifec
(1rr) encode more contextual audio cues for prosodic dynamics learning. Then bi-directional LSTM is used to predict the per- framee motion representation residuals the main motion pattern result P(I and LQI:T). i.e., Mrlyyhmic (1:T)
(R))( a ((::)): ): R(L(I.T); a (1.r)) By adding residual terms, the overall co-speech gesture GPT with motion refinement learning can be formulated as:
LReerdual [IM(UTT) - M(.rM where Marr (a) Muutte (a onee) + M (a (10)
In this way, we capture both major gesture patterns and subtle rhythmic dynamics for vivid results.

---

Table 1: The quantitative results on PATS Image Dataset. We compare the proposed Audio-driveN Gesture vIdeo gEneration (ANGIE) against recent SOTA methods [21, 33, 39, 62] and ground truth on four speakers' subsets. For FGD the lower the better, and the higher the better for other metrics.
Oliver
Seth
Kubinec
Jon
Methods
FGD
BC
Div.
FGD
BC
Div.
FGD
BC
Div.
FGD
BC
Div.
GT
0.00
0.76
54.6
0.00
0.71
49.3
0.00
0.84
38.9
0.00
0.73
62.8
S2G [21]
8.57
0.59
46.1
5.75
0.62
38.2
4.76
0.67
31.6
6.07
0,51
47.3
HA2G [33]
3,28
0.75
49.2
4.06
0.72
40.1
2.98
0.79
32.3
3.74
0.64
50.2
SDT [39]
1.04
0.61
52.9
1.97
0.58
46.7
1.15
0.77
36.1
1.63
0.60
57.4
TriCon [62]
3.63
0.53
48.3
3.79
0.52
40.3
3.27
0.77
35.7
3.98
0.61
49.7
ANGIE
0.88
0.72
53.5
1.83
0.69
46.7
1.10
0.81
36.5
1.57
0.65
60.9
4 Experiments
4.1
Experimental Settings
Dataset and Preprocessing. Pose, Audio, Transcript, Style (PATS) is large-scale dataset of 25 speakers with aligned pose, audio and transcripts [1, 2, 21]. The training corpus contains 251 hours of data with around 84,000 intervals of mean length 10.7s. Notably, the PATS dataset contains three modalities of audio log-mel spectrograms, speech transcripts and per-frame skeletons labeled with OpenPose [9]. To bypass the error accumulation in pose annotation and facilitate co-speech gesture image generation task, we extend PATS with more features: 1) preprocessed image frames and 2) onset strength audio features which are more suitable for co-speech gesture pattern learning.
We conduct the experiments on four speakers' co-speech video subsets, including Oliver, Seth, Kubinec and Jon. Concretely, 2D skeletons of the image frames are obtained by OpenPose [9] for baseline methods training. The frames are cropped to make the speaker locate at the image center. Since the time span of meaningful co-speech gesture unit sequence ranges from 4s to 14s [27, 47], we trim invalid videos and finally obtain 1306, 990, 1294 and 1284 clips for four subsets, respectively, The overall mean clip length is 9.8s. We randomly split the segments into 90% for training and 10% for evaluation. The image frames are sampled at 25 fps and further resized to 256 x 256.
Comparison Methods. We compare with recent SOTA works: 1) Speech to Gesture (S2G) [21], GAN-based pipeline that maps audio to 2D keypoints with a U-Net; 2) Hierarchical Audio to Gesture (HA2G) [33] which captures the hierarchical associations between multi-level audio features and tree- like human skeletons; 3) Speech Drives Template (SDT) [39] which relieves the one-to-many mapping ambiguity by set of continuous gesture template vectors; 4) Trimodal Context (TriCon) [62], representative framework that considers the trimodal context of audio, text and speaker identity. Note that all methods could drive 2D human skeletons with speech audio. We train baselines on the PATS image dataset and tune the hyper--aramene parameters by grid search for the best evaluation result. In particular, also show direct evaluations the Ground Truth (GT) skeletons for clearer comparison.
Implementation Details. We sample = 96 frame clips with stride 32 for training. 1) For the VQ- Motion Extractor: the co-speech gesture pattern codebook size M for both relative shift-translation Ap and factorial covariance change AL are set to 512. The encoders Es ESL and the decoders Da Dau are based on ID- ID-convolution structure. The channel dimension E of each codebook entry مرمs daL as well as the encoded latent features Cafи CAL are 512, while the temporal dimension T' is set T/8- 12 to encode contextual features with downsampling rate of 8. The is set as 1 x 10 10-5 to guarantee the positiveness of diagonal entries in factorial covariance L. The commit loss trade-offs in Lvo empirically set as 3, B2 0.1. We optimize the gesture pattern VQ- VAE with Adam optimizer [28] of learning rate 3 x 10-5 2) For the Co-Speech GPT: the Transformer channel dimension is 768, and the attention layer is implemented in 12 heads with dropout probability of 0.1. The onset strength audio features a onsef E 121 are extracted by Librosa, while the audio mfcc features mice E IR 28 are computed with the window size of 10 ms. During the GPT training, the e (1111: 9 (1.(1:111 are used as input while e (2.(2:12)> (2.(2):1) serve as supervision labels. 3) For the motion representation M and image generator G: we implement as MRAA [45] to use K = 20 regions. The motion estimator is pretrained for knowledge distillation. The overall framework is implemented in Py PoTch [37] and trained on one 16G Tesla V100 GPU for three days.
7

---

Table 2: User study results on co-speech gesture generation quality. The rating scale is 1-5, with the larger the better. We compare the Realness, Synchrony and Diversity to baselines [21, 33, 39, 62].
Methods
GT
S2G [21]
HA2G [33]
SDT [39]
TriCon [62]
ANGIE (Ours)
Realness
4.29
3.27
3,92
4.01
3.74
4.08
Synchrony
4.36
3,48
4.01
3.97
3,85
4.11
Diversity
3.97
2.49
3.31
3.88
3.02
3.95
Figure 3: Image sequence results ANGIE. We show the co-speech gesture image generation results of Kubinec, Seth and Jon, respectively. More qualitative results can be found in demo video.
4.2 Quantitative Evaluation
Evaluation Metrics. We adopt 1) Fréchet Gesture Distance (FGD) [62] to evaluate the distance between the real and synthetic gesture distribution. We train an auto-encoder on the PATS image dataset and use the encoder to compute fréchet distance between the real and synthetic gesture in feature space. We also use the 2) Beat Consistency Score (BC) and 3) Diversity (Div.) [31, 33] to account for the speech-motion alignment and the diversity among generated gestures. Specifically, BC is computed as the average temporal distance between each audio beat and its closest gesture beat, and Diversity indicates the difference of multiple audios' corresponding gestures in the latent space. Note that since all metrics are skeleton- akeleton-based, we downgrade our method to operate on skeleton data for fair comparison, i.e., we use VQ- VAE w/o cholesky scheme to create 2D skeletons for evaluation.
Evaluation Results. The results are reported in Table 1. It can be seen that the proposed ANGIE achieves the best evaluation results most metrics. Since our method summarizes reusable co- speech gesture patterns into quantized codebooks and complements subtle rhythmic dynamics, we can cover richer gesture patterns and create diverse results. Note that HA2G [33] tends generate over-expressive gestures with multi-level audio features, which makes their results on BC even better than the ground truth some cases. Despite of this, we perform comparable to ground truth on BC metric with stable motion results, showing that we can generate audio-aligned gestures. Besides, we can find that both SDT [39] and ours perform better on FGD and Diversity metrics than other methods due to the explicit modeling of co-speech gesture patterns. However, since the gesture pattern is finite and discrete, our quantized codebook design more suitable than continuous representation.
4.3 Qualitative Analysis
User Study. We further conduct a user study to reflect the quality of audio-driven gestures. Concretely, we sample 25 audio clips from the PATS image test set for all methods to generate skeleton results, then involve 18 participants for user study. The Mean Opinion Scores protocol is adopted, which requires the participants to rate three aspects: (1) Realness; (2) Synchrony; (3) Diversity. The rating scale to 5, with I being the poorest and being the best. The results are reported in Table 2, where method performs the best on all three aspects. Notably, with the help of motion pattern codebook.
8

---

Table 3: Ablation study of VQ-Motion Extractor and Co-Speech GPT with Motion Refinement.
Ablation Settings
FGD /
BC
Diversity 1
L1 4
Perceptual 4
AED
w/o Quantization Quantize μa, L Quantize Ajr, L Quantize μr, AL w/o Motion Refinement ANGIE (Ours)
5.86
0.54
35.6
0.071
79.2
0.095
0.058
67.4
0,086
0.043
52.3
0.069
0.052
63.6
0.083
1.39
0.58
48.3
0.041
48.1
0.063
1.35
0.72
49.4
0.037
42.9
0.063
Drive the Same Image with Different Patterns
Drive Different Images with the Same Pattern
Figure 4: Codebook Analysis. We validate that the codebooks contain meaningful motion patterns.
we achieve comparable diversity result to the ground truth, demonstrating that the quantization design helps capture common gesture patterns.
Video Generation Results. With the Co-Speech GPT and motion refinement network, we could predict the future quantization code and complement rhythmic motion details based on audio features, which enables us to generate video results. As shown in Fig. 3, the synthesized image sequence results from left to right contain diverse and meaningful gestures that are aligned with speech audio.
Codebook Analysis for Co-Speech Gesture Pattern. We analyze the meaning the quantized codebooks by asking the question: Does each codebook entry really correspond to certain type of motion pattern? To answer this, we validate two things in Fig. 4: 1) Different entries represent diverse gesture patterns, hence driving the same image with different quantized codes leads to different gestures (left). 2) Each codebook entry denotes fixed motion pattern, hence driving different images with the same quantized code show same motions (right). Please refer to demo video for more results.
4.4 Ablation Study
In this section, present ablation study of two modules in framework. Note that except for the skeleton-based metrics, we also use video reconstruction accuracy as a proxy for image quality, including L1 and perceptual loss [26] between the reconstructed and GT image; Average Euclidean Distance (AED) evaluates identity preservation by pretrained re-identification networks [24, 45].
VQ-Motion Extractor. We conduct ablation experiments five settings: 1) w/o Quantization, where we directly infer motion representation with audio; 2) Quantize μ., L; 3) Quantize Api, L; 4) Quantize μr, A L,, where we quantize the absolute or relative difference of the motion components; 5) w/o Cholesky Decomposition, which means the covariance matrix is not guaranteed to be symmetric positive definite. The results are shown in Table 3, which proves that the quantization design with position-irrelevant relative motion pattern could improve the performance. We also find that quantizing Ap is more effective than AL, since the shift translation is more correlated with region position. Note that when motion component invalid, the pipeline fails to generate image due to the numerical instability in calculating affine matrix. Hence the results for the setting 5 are not reported.
Motion Refinement Module. To verify the efficacy of motion residuals, we eliminate the motion refinement for ablation. The results in Table 3 suggest that this module improves beat consistency by capturing subtle rhythmic movements, while the improvement on FGD majorly derives from the quantization design. Such module complements the motion pattern for fine-grained results.

---

5 Discussion
Conclusion. In this paper, we propose a novel framework ANGIE to generate audio-driven co-speech gesture video in the image domain. To summarize valid common co-speech gesture patterns, we propose VQ-Motion Extractor with cholesky decomposition based quantization scheme and position- irrelevant design represent relative motion patterns. Then we propose Co-Speech GPT to refine subtle rhythmic movement details for fine-grain tne-grained results. Extensive experiments demonstrate that our framework renders realistic and vivid co-speech gesture video generation results.
Ethical Consideration. Generating co-speech gesture images facilitates applications such as digital human. However, it could be misused for malicious purposes like forgery generation. We believe that the proper use of this technique will enhance the machine learning research and digital entertainment.
Limitation and Future Work. As an early work towards audio-driven co-speech video generation without structural prior, we notice that our method fails for some challenging cases. For example, the source image is in large pose, it is hard to generalize well in such an out-of-domain data. We will explore how to develop a model with higher generalization ability in future work.
6
Acknowledgment
This work is in part supported by GRF 14205719, TRS T41-603/20-R, Centre for Perceptual and Interactive Intelligence, and CUHK Interdisciplinary Al Research Institute; and in part supported by NTU NAP, MOE AcRF Tier 2 (T2EP20221-0033) and under the RIE2020 Industry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s).
References
[1] Chaitanya Ahuja, Dong Won Lee. Ryo Ishii, and Louis-Philippe Morency. No gestures left behind: Learning relationships between spoken language and freeform gestures. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 1884-1895, 2020. 2,
3.7
[2] Chaitanya Ahuja, Dong Won Lee, Yukiko I Nakano, and Louis-Philippe Morency. Style transfer for
co-speech gesture animation: A multi-speaker conditional-mixture approach. In European Conference on Computer Vision, pages 248-265, Springer. 2020. 3.7
[3] Chaitanya Ahuja and Louis-Philippe Moreney. Language2pose: Natural language grounded pose forecasting. In 2019 International Conference on 3D Vision (3DV), pages 719-728. IEEE, 2019.
[4] Simon Alexanderson, Gustav Eje Henter, Taras Kucherenko, and Jonas Beskow. cotercontrollable speech- driven gesture synthesis using normalising flows. In Computer Graphics Forum, volume 39, pages 487-496.
Wiley Online Library, 2020. 2
[5] Guha Balakrishnan, Amy Zhao, Adrian V Dalca, Fredo Durand, and John Guttag. Synthesizing images of humans in unseen poses. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8340-8348. 2018. 2
[6] Uttaran Bhattacharya, Nicholas Rewkowski, Abhishck Banerjee, Pooja Guhan, Aniket Bera, and Dinesh Manocha. Text2gestures: transformer-basoe network for generating emotive body gestures for virtual agents. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR), pages 1-10. IEEE, 2021.
[7] Judee Burgoon. Thomas Birk, and Michael Pfau. Nonverbal behaviors, persuasion, and credibility.
Human communication research, 17(1):140-1e) 16900-66.1111611 1990. 1
[8] Chen Cao, Qiming Hou, and Kun Zhou. Displaced dynamic expression regression for real-time facial tracking and animation. ACM Transactions on graphics (TOG), 33(4)1-10, 20144 3
[9] Zhe Cao, Gines Hidalgo, Tomas Simon, Shih- En Wei, and Yaser Sheikh. Openpose: realtime multi- person 2d pose estimation using part affinity fields. IEEE transactions on pattern analysis and machine intelligence, 43(1):172- 186, 2019. 1.7
[10] Justine Cassell, David McNeill, and Karl-Frik McCullough. Speech-gesture mismatches: Evidence for one underlying representation of linguistic and nonlinguistic information. P Prugmatics & cognition, 7(1):133,
1999. 1
[11] Justine Cassell, Catherine Pelachaud, Norman Badler, Mark Steedman, Brett Achorn, Tripp Becket, Brett Douville, Scott Prevost, and Matthew Stone. Animated conversation: rule-based generation of facial expression, gesture& & spoken intonation for multiple conversational agents. In Proceedings of the s annual conference Computer graphics and interactive techniques, pages 413-420, 1994. 1
[12] Justine Cassell, Hannes Högni Vilhjålmsson. and Timothy Bickmore. Beat: the behavior expression animation toolkit. In Like Characters, pages (63-185, Springer, 2004. 1
10

---

[13] Caroline Chan, Shiry Ginosar. Tinghui Zhou, and Alexei A Efros. Everybody dance now. In IEEE International Conference on Computer Vision (ICCV). 2019. 2
[14] Lele Chen, Ross K Maddox, Zhiyao Duan, and Chenliang Xu. Hicrarchical cross- modal talking face generation with dynamic pixel-wise loss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7832-7841, 2019. 3
[15] Vasileios Choutas, Georgios Pavlakos, Timo Bolkart, Dimitrios Tzionas, and Michael J. Black. Monocular expressive body regression through body body-driven attention. In European Conference on Computer Vision
(ECCV), 2020.
[16] Daniel Cudeiro, Timo Bolkart, Cassidy Laidlaw, Anurag Ranjan, and Michael J Black. Capture, learning, and synthesis of 3d speaking styles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 10101-10111, 2019. 1,
[17] Jan P De Ruiter, Adrian Bangerter, and Paula Dings. The interplay between gesture and speech in the production of referring expressions: Investigating the tradeoff hypothesis. Topics in cognitive science,
4(2):232-248, 2012.
[18] Ylva Fersil and Rachel McDonnell. Investigating the use of recurrent motion modelling for speech gesture generation. In Proceedings of the 18th International Conference on Intelligent Virtual Agents, pages 93-98,
2018. 1, 1.2
[19] Ylva Ferstl, Michael Neff, and Rachel McDonnell. Adversarial gesture generation with realistic gesture
phasing. Computers & Graphics, 89:1 17-130, 2020. 3
[20] Zhenglin Geng, Chen Cao, and Sergey Tulyakov. 3d guided fine-grained, face manipulation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 9821-9830) 2019, 3 [21] Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, and Jitendra Malik. Learning individual styles of conversational gesture. In Proceedings of the IEEE/CVF Conference on Computer
Vision and Puttern Recognition, pages 3497-3506, 2019. 1. 2, 3. 7.8
[22] Ikhsanul Hahibie, Weipeng Xu, Dushyant Mehta, Lingjie Liu, Hans-Peter Seidel, Gerard Pons-Moll, Mohamed Figharib, and Christian Theobalt. Learning speech-driven 3d conversational gestures from video.
arXiv preprint arXiv:2 102.06777 2021.
[23] Dai Hasegawa, Naoshi Kaneko, Shinichi Shirakawa, Hiroshi Sakuta, and Kazuhiko Sumi. Evaluation of speech-to- to- gesturettr generation using bi-directional Istm network. In Proceedings of the 18th International
Conference on Intelligent Virtual Agents, pages 79-86, 2018, 2,
[24] Alexander Hermans, Lucas Beyer, and Bastian Leibe. In defense of the triplet loss for person re-
identification. arXiv preprint arXi:: 07737, 2017. 9
[25] Carlos T. 1shi, Daichi Machiyashiki, Ryusuke Mikata, and Hiroshi Ishiguro. A speech-driven hand gesture generation method and evaluation in androdd robots. IEEF Robotics and Automation Letters, 3(4):37573776, 2018. 2,
[26] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pages 694-711. Springer, 2016. 9
[27]
Adam Kendon. Gesture: Visible action as utterance. Cambridge University Press, 2004, 2,
[28]
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego
CA. USA, May 7-9, 2015, Conference Truck Proceedings, 2015. 7
[29] Taras Kucherenko, Patrik Jonell, Sanne van Waveren, Gustav Eje Henter, Simon Alexandersson, Iolanda Leite, and Hedvig Kjellström. Gesticulator: A framework for semantically-wware speech-driven gesture generation. In Proceedings of the 2020 International Conference Multimodal Interaction, ICMI '20, page 242-250, New York, NY, USA. 2020. Association for Computing Machinery.
[30] Jing Li, Di Kang. Wenjie Pei, Xuefei Zhe, Ying Zhang, Zhenyu Hc, and Linchao Bao. Audio2gestures: Generating diverse gestures from speech audio with conditional variational autnencoders. arXiv preprint
arXiv:2 06720, 2021. 2,3
[31] Ruilong Li. Shan Yang. David A Ross, and Angjoo Kanazawa. Learn to dance with aist++: Music conditioned 3d dance generation. arXiv preprint arXiv:2 00779777,, 2021. 8
[32] Xian Liu, Rui Qian, Hang Zhou, Di Hu, Weiyao Lin, Ziwei Liu, Bolei Zhou, and Xiaowei Zhou. Visual sound localization in the wild by cross-modal interference erasing, arXiv preprint 2202.0::00006400000 2022.
[33] Xian Liu, Qianyi Wu, Hang Zhou, Yinghao Xu, Rui Qian, Xinyi Lin, Xiaowei Zhou, Wayne Wu, Bo Dai. and Bolei Zhou. Learning hierarchical cross-modal association for co-specch gesture generation. arXiv preprint Xiv:22203 13161, 2022. 2, 3, 7,
[34] Xian Liu, Yinghao Xu, Qianyi Wu, Hang Zhou, Wayne Wu, and Bolei Zhou. Semantic-aware implicit neural audio-driven video portrait generation. arXiv preprint MXXi:22011007780 2022. 1
[35]
David McNeill. Hand and
De Gruyter Mouton, 2011.
[36]
Evonne Ng. Hanbyul Joo, Liwen Hu, Hao Li, Trevor Darrell, Angjoo Kanazawa, and Shiry Ginosar.
Learning to listen: Modeling non-deteriinistic dyadic facial motion. arXiv preprint arXiv:2204.08..
2022. 3
[37] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026-8037, 2019. 7
[38] KR Prajwal, Rudrabha Mukhopadhyay, Vinay P Nanboodiri, and CV Jawahar. A lip sync expert is all you need for speech lip generation in the wild. In Proceedings of the 28th ACM International Conference on
11

---

Multimedia, pages 184-492 2020. 3
[39]
Shenhan Qian, Zhi Tu, YiHao Zhi, Wen Liu, and Shenghua Gao. Speech drives templates: Co-speech
gesture synthesis with learned templates. arXiv preprint arXiv:2108.0m 080200 2021. 3, 7.8
[40] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language
models are unsupervised multitask learners. OpenAI blog. 1(8):9. 2019. 2
[41] Alexander Richard, Michael Zollhöfer, Yandong Wen, Fernando De la Torre, and Yaser Sheikh. Meshtalk:
3d face animation from speech using cross- mross-to disentanglement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1173-1 182, 2021. 3
[42] Najmeh Sadoughi, Yang Liu, and Carlos Busso. Msp-avatar corpus: Motion capture recordings to study the role of discourse functions in the design of intelligent virtual agents. In 2015 1 11th JEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), volume 7, pages 1 1-6. IEEE,
2015. 2
[43] Aliaksandr Siarohin, Stéphane Lathuilière. Sergey Tulyakov. Elisa Ricci, and Nicu Sebe. Animating arbitrary objects via deep motion transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 2377-2386. 2019.
[44]
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion
model for image animation. Advances in Neural Information Processing Systems, 32, 2019. 2
[45] Aliaksandr Siarohin, Oliver Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion representations for articulated animation. In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pages 33653-13662, 2021. 2, 3, 4. 7.7.
[46] Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. arXiv preprint arXiv:2203. 13055, 2022. 6
[47]
Michael Studdert- Kennedy. Hand and mind: What gestures reveal about thought. Language and Speech,
37(2):203-209, 1994. 2.7
[48] Kenta Takeuchi. Souichirou Kubota, Keisuke Suzuki. Dai Hasegawa, and Hiroshi Sakuta. Creating gesture-speech dataset for speech- based automatic gesture generation. In International Conference on Human-Compp
Interaction, pages 198-202. Springer, 2017. 1.2
[49] Taoran Tang. Jia Jia, and Hanyang Mao. Dance with melody: An aunoocooonee approach to music- oriented dance synthesis. In Proceedings of the 26th ACM international conference on Multimedia, pages
1598-1606, 2018. 6
[50] Justus Thies, Michael Zollhofer. Mare Stamminger, Christian Theobalt, and Matthias NieBner. Face2face: Real-time face capture and reenactment of rgb videos. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2387-2395, 2016.
[51] Jackson Tolins, Kris Liu, Yingying Wang. Jean E Fox Tree. Marilyn Walker, and Michael Neff. A multimodal motion-captured corpus of matched and mismatched extravert-introvent conversational pairs. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC' 16),
pages
2016. 2
[52]
Lloyd N Trefethen and David Bau III. Numerical linear algebra, volume 50. Siam, 1997. 5
[53]
Aaron Van Den Oord, Oriol Vinyals. et al. Neural discrete representation learning. Advances in neural
information processing systems. 30, 2017. 5
[54] Susanne Van Mulken, Elisabeth Andre, and Jochen Müller. The persona effect: how substantial is it? In
People and computers XIII. pages 53-66. Springer, 1998. 1
[55] Ekaterina Volkova, Stephan De La Rosa, Heinrich H Bülthoff, and Betty Mohler. The mpi emotional body expressions database for narrative scenarios. PloS one, 9:3413 2014. 2
[56] P. Wagner, 7. Malisz, and S. Kopp. Gesture and speech in interaction: An overview. Speech Communication,
17:209-232. 2014.
[57] Olivia Wiles, A Koepke, and Andrew Zisserman. X2face: A network for controlling face generation using images, audio, and pose codes. In Proceedings of the European conference on computer vision (ECCV), pages 370-686, 2018. 3
[58] Jason R Wilson, Nah Young Lee, Annie Saechao, Sharon Hershenson, Matthias Scheutz, and Linda Tickle- Degnen. Hand gestures and verbal acknowledgments improve numan-robor rapport. In International Conference on Social Robotics, pages 334-344. Springer, 2017.
[59] Jing Xu, Wei Zhang, Yalong Bai, Qibin Sun, and Tao Mci. Freeform body motion generation from speech. arXiv preprint arXiv:2203.0 022911. 2022. 3
[60] Yanzhe Yang. Jimei Yang, and Jessica Hodgins. btade motion synthesis for social conversations.
In Computer Graphics Forum, volume 39, pages 201-212 Wiley Online Library, 2020. 2
[61] Payam Jome Yazdian, Mo Chen, and Angelica Lim. Gesture2vec: Clustering gestures using representation
learning methods for co-speech gesture generation, 2022. 2
[62] Youngwoo Yoon, Bok Cha, Joo-Haeng Lee,, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee. Speech gesture generation from the trimodal context of text, audio, and speaker identity. ACM Transactions on Graphics (TOG), 39(6):1-16, 2020. 1, 2, 7,
[63] Youngwoo Yoon, Woo-Ri Ko, Minsu Jang, Jacyeon Lee, Jaehong Kim, and Geehyuk Lee. Robots learn social skills: End-to-end learning of co-speech gesture generation for humanoid robots. In 2019 International Conference on Robotics and Automation (ICRA). pages 4303 (003-4309..0 IEEE, 2019, 1, 2.
[64] Egor Zakharov, A Aiaksandra Shysheya, Egor Burkov, and Victor Lempitsky. Few-shot adversarial learning of realistic neural talking head models. In Proceedings of the IEEF/CVF international conference on
12

---

compuler vision, pages 9459-9468, 2019.
[65] Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang. and Ziwei Liu. Pose- cobtrolllll talking face generation by implicitly modularized audio visual representation. In Proceedings of the IEED/CVF Conference Computer Vision and Puttern Recognition, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] The limitations are discussed in
Section 5. More details and analysis are provided in the supplemental document.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] The potential negative socictal impacts are discussed in Section 5. More ethical considerations with potential measures provided in the supplemental document.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] The assumptions of the unique cholesky decomposition theorem are the real symmetric positive definite characteristic of the matrix, which are stated in the Theorem
(b) Did you include complete proofs of all theoretical results? [Yes] The proofs are included the supplemental document.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [N/A] We include very detailed implementation instructions in Section 4.1 help reproduce the main experimental results. Besides, although the code and data are not included, as promised we will make the code, models and data publicly available.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] They are specified in the Implementation Details. Please see Section 4.1.
(c) Did you report eπror bars (e.g.. with respect to the random seed after running experiments multiple times)? [Yes] Please see the supplemental document for evaluation results with error bars.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type GPUs, internal cluster, or cloud provider)? [Yes] They are specified in the Implementation Details. Please Section 4.1.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] The citations are all included in the reference, and the detailed licenses are included in the supplemental document.
(b) Did you mention the license of the assets? [Yes] The citations all included in the reference, and the detailed licenses are included in the supplemental document.
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] The citations are all included in the reference, and the detailed licenses are included in the supplemental document.
(d) Did you discuss whether and how consent obtained from people whose data you'ne using/curating? [Yes] Our work and data are based on publicly available dataset and used for the academic
exclusively.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] The discussions are elaborated in the supplemental document, where all the data and models are for the academic use only.
13

---

5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes] The detailed user study instructions with details are provided in the supplemental document.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [Yes] Since the user study merely requires the participants to watch and evaluate the co-speech gesture generation results, there is no potential participant risk.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [Yes] We pay the user study participants for their efforts. The details are provided in the supplemental document.
14


Reviews:
[{"review_id": "uSSf9VZAo-k", "reviewer": "Reviewer_FuNA", "summary": "The paper presents ANGIE, an approach for audio-driven image animation with specific focus on upper body gestures. ANGIE leverages an implicit motion model (MRAA) as the intermediate motion representation instead of operating on a landmark-based skeleton representation which allows it to work directly in the image domain. The problem is divided into 3 parts: a motion representation learning module, a motion prediction module, and a motion refinement module. First, a VQVAE is trained to discretize the motion into a fixed set of codes. This model learns a position-independent codebook of co-speech gestures. Next, a Co-Speech Gesture GPT learns to map audio onset features to the discrete codes of motion extracted from training videos. This GPT network is also trained with a residual loss which focuses on refining the motion predicted from the discrete codes.\n\nThe authors compare their approach to previous state-of-the-art audio-driven gesture generation networks and evaluate using objective metrics and subjective user studies. Overall, the proposed method outperforms the baselines. One of the baselines: HA2G tends to have better or similar beat consistency than the proposed method. The authors also perform ablations studies to compare different vector quantization strategies and confirming the benefit of the motion refinement step. The authors also analyze the quantized motion representations.", "questions": "- L140: Shouldn't the output frames be from I^hat_(2:N) if I_1 is given? If that is the case, perhaps the authors can just say I_0 is given and then the rest of the paper need not be changed.\n- How is the onset feature strength computed? What is the window size or hop size for spectral flux computation? I don't understand what the 426 dimensions are in the onset feature. Onset strength should be a one dimensional feature per frame. Do you mean to say that the training videos are a fixed length and have 496 audio frames? It does seem like the videos are fixed length (96 frames @ 25 fps) but then you also need to mention some technical specifications: sample rate of the audio.\n- Similarly, how are the mfcc features computed? Again, I don't understand the dimensions of the MFCCs. Are you using 12 mfccs or 28 mfccs. In any case, if the onset strength length is 426, why is the temporal dimension of the mfccs so small? And you mention that they are computed with a window size of 10 ms. What about hop size? What about fft block size?\n- Does the network also infer mouth movement? It would seem that way since there is no special treatment of head/face from the body gestures, but then I am really surprised by the quality of the mouth motion in the sample videos shared. Why have the authors not also evaluated using audio-driven face animation metrics?\n", "limitations": "The authors adequately discuss the limitations of their work along with ethics considerations of such algorithms.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- This paper is the first to animate face and torso images without using a skeleton representation of the images and demonstrates improved performance over those approaches. \n- The method is another interesting application of VQ-VAE followed by a prediction model. The novelty also lies in the design of the VQVAE. Instead of naively learning a codebook for the motion features of MRAA, the authors transform their data to address constraints in their problem, such as learning a codebook from the lower triangular decomposition of their covariance matrix and using relative motion parameters instead of absolute parameters. Their decisions are also ablated in their evaluation showing the benefits of these transformations.\n\nWeaknesses:\n\nThere are some minor issues with the paper in terms of details. \n- The subjective evaluation involves some kind of mean opinion score (MOS). The authors do not add 95% confidence intervals or a standard deviation to their scores. This is important to see the spread in the scores given to each system.\n- Some details regarding audio feature extraction are missing in the paper. Please refer to the questions section.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "lNwj0Lrufb", "reviewer": "Reviewer_S12n", "summary": "This paper concerns synthesis of co-speech gestures, gestures are coded in terms of entries in a VQ code book, and a GPT-like model is used to generate the sequence of symbols corresponding to input audio. In the generation, first the broad gestures are created, and then these are refined to improve the overall fidelity.", "questions": "The audio clips are variable length, are these clipped to be a fixed size?\n\nHow do you ensure consistency when generating longer sequences? Only short segments are shown in the demo video, it would be nice to see longer generated sequences.\n\nI was confused by “position-irrelevant” because position is important as position gives the sense of scale in gesture space. It was cleared up in later text that you mean image location invariant.\n\nIn the demo video, the speaker refers to “four quadrants” and it looks like the model produces gestures that seem to point to four quadrants. This seems suspicious since the “common motion patterns” are generated from just the speech envelope, and the refined motion from the MFCCs.\n", "limitations": "Clarify that the approach is specifically targeting only beat gestures.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ To construct the training data, an existing dataset was augmented with new features, and this will be made available.\n+ The generated image sequences look compelling.\n+ The approach was compared against, and beat, several baselines.\n\n- The paper refers to gestures as “common motion patterns” and “rhythmic dynamics”.  What do these mean?  Use standard terminology from the co-speech gesture literature.\n- It looks like the models are over fit to the specific sequences.  For example, I think it’s the Kubinec sequence: when the speaker is referring to information on the screen to the speaker’s left, the speaker points and gestures towards the screen.  Why?  Is the screen always to the speaker’s left?  There is no language model and there is no explicit capturing of the speech content, so it is not clear to me why we would see this behavior.\n- I believe you are training a separate model for each speaker, but you criticized prior work for limited generalization ability.\n- The paper considers gesture only in a very limited context - talk show host speaking monologue to camera.  How well do you expect this to generalize, e.g., to conversational gestures that are likely more subtle and drawn from a larger lexicon?\n- There are broad categories of gesture that this approach will fail to capture.  Specifically without the use of language, iconics, metaphorics, and deictics, are unlikely to be generated.  You ought to clarify that the focus is specifically only on beats.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "N6A451n51Ed", "reviewer": "Reviewer_WZ3S", "summary": "This paper works on audio-driven gesture image generation, which receives a still image and audio to generate a sequence of images (video). Compared to previous works, it does not rely on skeletal pose annotation to prevent error accumulation caused by cascading of synthesis networks. The proposed system (named as ANGIE) is consisted of VQ-motion extractor and Co-speech GPT, each mapping the image to vector quantized representation and transformer network autoregressively predict next frame's quantized representation, respectively.", "questions": "- Regarding the line 272 - \"The motion estimator is pretrained for knowledge distillation\" - please elaborate on this. The authors might want to describe such implementation details on the appendix.\n- What is a new dataset that the authors have collected, as described in line 92? Is it simply an extension of PATS dataset by some post-processing steps?", "limitations": "It would be better if the authors could provide specific examples of the limitation of ANGIE (discussed in line 344).", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts\n- By vector-quantizing the images, the proposed network could enjoy powerful capability of the Transformer. This kind of framework might be also applied at various video generation networks.\n- The motion refinement network to aid missing details of reconstructed images from quantized representation seems novel.\n- Ablation studies (Table 3) clearly justifies the choice of the components.\n\nWeaknesses\n- The related works shown in section 2 is limited in the application viewpoint. It would be much better if authors could make a connection to slightly other domains that the proposed method could be also applied. In overall, the scope of the paper is too specific on the co-speech gesture image generation. To meet broader NeurIPS readership, I wish there would be some implications outside the main problem.\n- Please do not cite Wikipedia article as a reference (line 303, Fleiss's Kappa statistic). Please briefly describe about the concept in the appendix.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BSAJ6QD7gaS", "reviewer": "Reviewer_SBef", "summary": "This paper looks into the topic of generating gesture videos from given speech and images. The proposed approach learns the representations of gesture motions as a codebook, and learns the mapping from the speech to the codebook. With the learned representation and the speech-to-codebook mapping, the approach can animate the given image with gestures based on the given speech. The speech representation for the image is based on previous work “Motion Representations for Articulated Animation”. Since the codebook provides quantized representation and can miss some details, the approach includes an additional residual learning approach to further construct the image details from speech. User studies showed improvement compared to previous work. Ablation studies are provided to show the effectiveness of the proposed techniques.", "questions": "How well does the approach generalize to speakers that are not in the training data?", "limitations": "The authors have properly discussed the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe approach generates good quality gesture videos based on the given image, and showed improvement compared to previous work.\n\nAblation studies demonstrated the effectiveness of the proposed techniques.\n\n\nWeaknesses:\nThe idea of learning representations and learning the mapping from utterances to the representations are not new, and the motion representations used in the work is proposed in the previous work. The main novelty is applying the idea on gesture generations.\n\nIt is not clear how the approach performs on speakers that are not in the training data.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept