Gesture2Vec: Clustering Gestures using  Representation Learning Methods for Co-speech Gesture Generation
Authors: Payam Jome Yazdian
Mo Chen
Angelica Lim
Venue: ICLR 2022 Submitted (2022)
Keywords: representation learning
gesture generation
vector quantization
machine translation

Abstract: Co-speech gestures are a principal component in conveying messages and enhancing interaction experiences between humans. Similarly, the co-speech gesture is a key ingredient in human-agent interaction including both virtual agents and robots. Existing machine learning approaches have yielded only marginal success in learning speech-to-motion at the frame level. Current methods generate repetitive gesture sequences that lack appropriateness with respect to the speech context. In this paper, we propose a Gesture2Vec model using representation learning methods to learn the relationship between semantic features and corresponding gestures. We propose a vector-quantized variational autoencoder structure as well as training techniques to learn a rigorous representation of gesture sequences. Furthermore, we use a machine translation model that takes input text and translates it into a discrete sequence of associated gesture chunks in the learned gesture space. Ultimately, we use translated quantized gestures from the input text as an input to the autoencoder’s decoder to produce gesture sequences. The resulting gestures can be applied to both virtual agents and humanoid robots. Subjective and objective evaluations confirm the success of our approach in terms of appropriateness, human-likeness, and diversity. 

Under review as a conference paper at ICLR 2022
VESTURE2VEC: CLUSTERING GESTURES USING REPRESENTATION LEARNING METHODS FOR CO-SPEECH GESTURE GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
Co-speech gestures principal component in conveying messages and enhanc- ing interaction experiences between humans. Similarly, the co-speech gesture is a key ingredient in human-agent interaction including both virtual agents and robots. Existing machine learning approaches have yielded only marginal success in learning speech-to-moticn
the frame level. Current methods generate repet- itive gesture sequences that lack appropriateness with respect to the speech con- text. In this paper, we propose Gesture2 Vec model using representation learning methods to learn the relationship between semantic features and corresponding gestures. We propose new conceptual framework that considers gestures as a non-verbal language itself. Our approach first converts gesture sequences into symbolic chunks, using frame and sequence level autoencoders and rigorous train- ing techniques to learn the vocabulary. Using this higher-level representation, we then take advantage of machine translation model to learn translations of text to discrete sequences of associated gesture chunks in the learned gesture space. Ul- timately, we use these quantized gestures as input to the autocncoder's decoder to produce gesture sequences. The resulting gestures can be applied to both virtual agents and humanoid robots. Ablation studies support that our gesture chunking approach, fixed decoder weights, and vector quantization are the main drivers of diversity in objective gesture diversity measures. Further subjective and objec- tive evaluations confirm the success of our approach in terms of appropriateness. human-likeness, and diversity.
INTRODUCTION
Non-verbal behaviour is indispensable part of our daily communication. Prior research stated that 70-33% of communication non-verbal, including facial expression, hand gesture, body pose, and vocal tones (Mehrabian 2017 Lapakko 2007). People spontaneously gesticulate to complement
verbal channels during the speech to convey messages (McNeill 2011 De Ruiter et al. 2012 Cassell let al. 1999). Hence, integrating non-verbal communication skills into social robots and virtual agents is crucial for compelling interactions (Minato et af.) 2004 Woods et al. [2004) Breazeal et al., 2005). Gestures originate from semantic features(Chut & Kita (016) and are characterized by speech context (Lücking et al. 2013). Therefore, co-speech gesture generation has been typically addressed using textual and acoustic features generate relevant gestures.
Gesture synthesis can be categorized into deterministic and probabilistic models. Deterministic models predict a single output for given input, while probabilistic models estimate a plausible output probability distribution conditioned on the given input. The limitation of deterministic data- driven methods is that they neglect span the variation of many dimensions data space (Fragki- adaki et al.) 2013 Ferstl et al. 2019). Probabilistic models are potentially able to capture a broader gesture space, but probabilistic models also suffer from the posterior-collapse problem (Bowman) et al. 2015) which results in repetitive gestures close to the average. The problem arises when the one-to-one mapping assumption disregards the one-to-man, mony relationship between speech and ges- ture. This is common problem among generative models for gestures (Yoon et al. 2019; Ginosar jet al., 2019). as well as in other domains, e.g. image generation models that produce a limited set of blurry and similar images (Lucas et al. 2019a), and early attempts for natural language genera-

---

Under review as a conference paper at ICLR 2022
tion (Bowman al. 2013 Kingma al. 2016 He et al. 2019). Literature attempted address
this posterior collapse problem using different techniques such as adversarial training (Goodfellow)
eet al.
2014: Ginosar et al., 2019 Arjovsky et al. 2017 Srivastava et al. 2017). variational autoen-
coders (Higgins et al. 2016, Mi et al. [0188] Ling et al. 2020), normalizing flow (Alexanderson
eet al.
2020),
vector quantization (Oord al. 2017) (Razavi et al. 2019), weakened decoders (Yang Semeniuta et al. 2017). However, as reported, prior work (Yoon et al.) 2019; Ferst]
ict al.
2017:
al.
2019; Kucherenko et al., (020b)) were not successful in capturing long-term dependencies.
Indeed, the model parameters focus on imperceptible and local features such as continuity among consecutive frames, most likely due to mode collapse.
Inspired by advances in other machine learning fields, especially natural language processing, we model gestures as a language that contains a vocabulary, and then perform text-to-gestun tootogetrrrreeeeee as a lan- guage translation task. Specifically, we integrate unsupervised representation learning methods and a machine translation algorithm. First, we reduce body pose dimensionality using a Denoising Au- toencoder at the frame level. Then, we perform discretized motion representation learning to cluster similar motion sequences to a symbol. This vector quantization method is form of cluster pattern recognition where each motion sequence is assigned to particular word from codebook. Finally, we train a machine translation model to translate between utterances and their accompany- ing gestural motion symbols. This method is effective since it mitigates the complexity of gesture space and focuses on longer dependencies. In addition to traditional evaluation measurements, we also present new objective and subjective metries to evaluate the diversity of gestures.
2 RELATED WORK
We first review co-speech gesture generation methods. Next, we discuss recent gesture generation approaches and their limitations. Finally, we introduce the deep neural network that we used and its advantages over prior work.
Early attempts for gesture generation use blending smooth feasible motion clips selected from a database. Rule-based algorithms (Cassell et al., 2004 Huang & Mutlu 2012). hidden Markov models (Levine et al., 2009), conditional random fields Levine et al. 2010). and hybrid systems
(Kipp) Neff et al. 2008) (Chiu et al.) 2013)) were used to select proper gestures conditioned on given input.
now recogmized that these kind of systems require extensive efforts to annotate data and cannot generate gestures for unseen inputs. Also, they provide single prediction for given input and lack variation of generated movement. Furthermore, scheduling the gestures with speech is challenging since they may not be precisely synchronized (Butterworth & Hadar 1989. Kendon 2004) despite originating from the same source (McNeill 2011).
Recent deep generative models, e.g. VAEs (Kingma & Welling 2013 Higgins et al. 2016). GANs
(Goodfellow et al., 2014 Abdal et al. 2019). and transformers /Vaswani et al. 2017 7 Brown et al. 2020 Dehghani et al. 2018), achieved notable results on different tasks. A generative model com- prises the joint probability of given data and output. Therefore, it can generate new plausible in- stances by taking samples from the learned distribution (Hasegawa al. 2018). Generative models
have been used for human motion generation (Yan et al. [218 Hermandez et al. 2019 Ling et al.,
2020 as well as co-speech gesture generation (Ginosar et 2019 Yoon et al. 2019 Ferstl al.,
2021 Li ct al. 2021). However, co-speech gesture generation is more challenging since the rela-
tionship between speech and motion is complex (Butterworth Hadar 1989.
Input to the generative system can be supplied from either speech-text, audio, or both modalities with a broad range of semantic and acoustic features (Kucherenko et al., 2021c). Systems that use acoustic features led to generated beat gestures according to the speech rhythm (Hasegawa et al.) 2018 Ferstl et al. 2020 Ginosar et al.] 2019 Kucherenko) 2018 Pouw & Dixon 2019). and text- based systems (Yoon al.) 2019; Ishi et al. 2018) produced more semantically aware gestures. Although text-based models capture communicative features, they lack the strong effect of speech acoustics, i.e. intonation, prosody, and loudness, expressed gestures (Pouw et al.] 2020). Recent work benefiting from both modalities generated more compelling co-speech gestures in terms of appropriateness and naturalness (Yoon et al., 2020).
Typically, co-speech gesture generation systems build upon an encoder and decoder architecture (asegaaw al. 2018 Kucherenko et al. 2019 Yoon al. 2019 Kucherenko et al. 2020a Ferstl
2

---

Under review as a conference paper at ICLR 2022
Recurrent Neural Networks (RNN) structures have been used extensively for the encoder and decoder (aseeawa et al., 2018). However, RNN based models suffer from the error accumulation problem and are not good capturing long-term human motion dependencies (Her- handez et a 2019). While Convolutional Neural Network (CNN) based gesture generation models not vulnerable to accumulation problems, CNNs prone to regress to the average motion and generate repetitive gestures (Li et al., [021]. Variational Autoencoders were also proposed for the co-speech gesture generation task to generate more realistic and diverse gestures, e.g. (Rezende et al. 2014 Li et al., [022]] Kucherenko 2018). VAEs with strong decoders tend to ignore the latent variable and learn the mode of the oulput data. This problem, known as "posterior collapse" causes the model to generate slightly similar outputs close to the average (Lucas et al. 2019b)
& McDonnell 2018).
Literature considerably explored adversarial training to generate more realistic and diverse gestures (Ginosar et al., 2019 AAhuja et al. 2020; Yoon et al.) 2020, Li et al. 2021 Ferstl et al. 2020 2021]. Although they brought attractive results, GANS are hard to train (Lucie et al. 2017) and suffer from mode collapse (Tulyakov et al. 2018). Normalizing-fol
based models have GANs advantages while replacing adversarial loss by classical likelihood maximization training (Kingma & Dhariwal 2018) and efficient probabilistic inference of VAEs (Henter et al., 2020) (Kingma & Dhariwal 2018). Furthermore, autoregressive models on discrete data achieved impressive results in many sequences-tO-ssqueeeee tasks such as machine translation (Wang et al. [2019), image gener- ation(Salimans et al., 2017) (Razavi et al.] 2019), and speech synthesis (Oord et al. (016)) (Gárbaceal ct al. 2019). VQ-VAE models learn discrete latents space. enable us use autoregressive models on the posterior, and does not suffer from "posterior collapse" (Oord et al. 2017). Ordinarily, ges- ture generation systems use generator network that produces gestures from latent code. To better regress the data and cover a broad range of motion, (Li et al. [2021] Kucherenko et al. 2021b uses a motion-spectio code space that represents motion attributes.
All of the state-of- tfae-of-the-ane art models mentioned above generate gestures mame-by-frame.. We suggest that this architecture drains model capabilities on local dependencies at the expense of global fea- tures and diversity. We therefore reformulate the problem as a machine translation task. Inspired by VQ-VAE, we propose a method that combines VQ-VAE, Denoising Autoencoders and weakened decoders to learn discretized latent space. Finally, we use autoregressive model on the quantized motions to produce co-speech gestures from the input.
3 METHOD
We considered Yoon et al. (2019) as our baseline since also focus on word embedding textual
features (Bojanowski et al. 2017) as the control signal for gesture generation. The authors in Yoon architecture with soft attention mechanism (Bahdanau et al. 2014 Cho et al. 2014). The encoder extracts textual features, and the decoder produces co-speech gestures frame-by by-mam. We extend the baseline model by utilizing represen- lation learning approaches, Our proposed system has the following steps.
cet al., 2019) used RNN based Encoder-Decodee
Pose representation learning the frame-level
Discrete motion representation learning at the sequence-level.
Translation from text to the learned discrete motion representation space.
Learning powerful representations without supervision is of utmost importance to reduce problem complexity. Autoencoders are unsupervised models that learn significant data features and discard spurious patterns by minimizing the reconstruction error (Kingma & Welling 2013). They consist of an encoder network followed by a decoder network. An autoencoder's bottleneck finds shared data representation (Goodfellow et al.) 2009] [Bngio et al. (007) by learning the correlations between input dimensions and reconstructs them from a low-dimensionality representation. We used two different autoencoders with different components, both at the frame-level and sequence-level, as explained in the following sections.
3.1 FRAME-LEVEL AUTOENCODER
Inspired from Kucherenko et al. 2019 202 we used a Denoising Autoencoder (DAE) archi- tecture (Vincent al. 2010 Goodfellow et al. 2016) to reduce dimensionality at the frame level.
3

---

Under review as a conference paper at ICLR 2022
Encoder
Decorie
Latert
Figure 1: Pose DAE: Representation learning at the frame-level.
DAE learns lower-dimensionalityy representation of data while corrupting the data intentionally through additive isotropic Gaussian noise. The bottleneck in the middle of DAF's encoder and de- coder is generated from the noisy input, and the DAE Iearns to reconstruct the original input from it. DAES can capture fundamental structures the input distribution while preventing it from simply learning the identity function as each feature is encoded and decoded independently to the others (Vincent et al. 2010). The frame-level DAE includes an encoder and decoder as follows. The DAE Encoder maps a noise injected input m from pose space to the representation 2f and decodes the representation 2 back single frame n in motion space.
2f = Encoder ( my = my + N(O, I)
m = Decoder
(2)
As the DAE minimizes the reconstruction mean square error (MSE) loss, it learns a more informative representation to resemble the original input closely. Figure. Lillustrates our representation learning for poses at the frame level.
3.2 SFQUENCE-LEVER AUTOENCODER
In this step, we aim to create a vocabulary over the gesture space in which we will have a specific to- ken for a of similar motion sequences in the real world. We use variational autoencoder (VAE) framework armed with discrete latent representation (Oord et al. 2017). Given set of observa- tions, a Vector Quantized Variational Autoencoder (VQ-VAE) learns to create a motion vocabulary by parameterizing the posterior distribution of discrete latents.
In more detail, the VAE defines the posterior distribution as p(z[æ) x p(rl=)p(z). Typically, the prior p(z) has been considered
(o: T) on the latent variable E IR where D) is the bottle- neck's 's dimensionality. Accordingly, the VAE encoder creates a posterior distribution ((=[x) over the latent representation of input Meanwhile, for chosen approximate posterior ((2(z), the decoder
trained on the reparametrized sample ((=(<) instead of deterministic encoded = <Kingma & Welling 2013). The VAE arranges the latent space such that motions with similar movements are projected close to each other while reducing the reconstruction loss (Kingma & Welling 2013
Stewart et al. 2021).
Additionally, we consolidated a denoising characteristic to the variational autoencoder framework (DVAE), presenting more flexible and robust posterior distribution approximation than the standard VAE (Im Im et al. 2017). It can be considered as a standard VAE with the denoising criterion, which samples noise injected input â ~ p()(z) rather than x itself. Afterward, it samples 2 ((x[²) using an encoder network, and samples the reconstructed input from the p(z|2).
Furthermore, we discretize the latent space by decomposing it into set of embedding vectors (Oord al., (017). Previous work has shown that VQ led better representation learning, pre- vented mode collapse, and provided high reconstruction resolution (Oord et al. 2017 Razavi et al.) 2019 Chorowski et al., 2019 Baevski et al., 2019). Additionally, discretization allows employ- ing algorithms from the NLP community to model long-range temporal dependencies rather than imperceptible details at the frame level.
In this model, shown in Figure 2 the posterior g(=(ā) and prior p(z) distributions are categorical. Indeed, samples from these distributions map to tokens from a codebook of K embedding vec- tors R², P 1, 2, ..., K. The discrete latent variables z is determined from the continuous VAE's encoder output z. VQ-DVAE finds the x nearest neighbour embedding vector e; in the K embedding vectors of RDD As shown in equation 3. the posterior categorical distribution g (=(xx)
4

---

Under review as a conference paper at ICLR 2022
Codebook
A(O.D)
Prender
Decoder
pExIe))
STEE
Reconstaaionn
Figure 2: Motion VQ-DVAE: Discretized representation learning at the sequence-level.
probabilities represent a one-hot. Afterward, the corresponding embedding ez is fed to the decoder for the reconstruction process.
g² k|() (1
for k arg min y||= -eyll Otherwise
(3)
Despite the standard VAE, the prior is uniform distribution over the K elements in the codebook. Therefore, the KL-divergence term usually incorporated into the ELBO constant and safely re- moved. equation Jeefines the loss term for the VQ-DVAE from/Oord et al. (017). Since equation
not differentiable, similar to the trright-through estimator [Bengio et al. 2013 we consider gra- dients from decoder inputs ex for encoder outputs z.
C = log p(>|2)) + ||>g[z] =] - ell2 B||= sg [e])
(4)
The first term stands for the reconstruction loss and trains the encoder and decoder. Although the velocity does not appear on the loss function explicitly, we feed the input concatenated with its derivative input to the VQ-DVAE. In fact, encourage the model to reconstruct not only the input sequence frame by frame, but also its derivative. This can be counted as proxy for motion dynamics (Yoon et al.) 2019). The second term encourages the embeddings to move closer to the encoder output. On the other hand, the third loss term, named commitment loss, encourages the en- coder generate latents close the assigned embeddings from equation 3 Meanwhile, it prevents the volume of the embedding space from growing arbitrarily.
Note that sg means "stop-gradient", so do not propagate the gradient w.r.t that term. There- fore, the first term updates both the encoder and decoder, the second term updates the codebook embeddings, and the third term updates the encoder to commit to embedding.
3.3 TRANSLATION FROM TEXT TO GESTURE VOCABULARY
Finally, having a gesture vocabulary, the co-speech gesture generation resembles a machine transla- tion task of English to the Gesture domain. Consequently, we apply purely autoregressive model of sequence-tar sequencceto-sequence known as a machine translation model. For instance, n-gram models (Juraf-] Sky & Martin 2009), convolutional neural language models (Dauphin al. 2017 Bai et 2018) attention-based models (Vaswani et al., 2017 Wang et al. 2019), etc. yielded attractive results. We use two-layered bidirectional gated recurrent neural network (GRU) (Cho et al. 2014), with a soft
attention mechanism (Bahdanau et al. 2014). After the translation task of input text to gesture to- kens is complete, we use the VQ-DVAE s decoder to reproduce the entire gesture sequence followed by further prot-processes such as the Savitzky- smoothing filter (Savitzky & Golay 1964).
EXPERIMENT
This section describes our implementation and experiment details, such as the specifics the train- ing data, model, and the training techniques used. We obtained our training set from the Trinity dataset (Ferstl & McDonnell 2018), which contains 23 clips, each approximately 10 minutes in length, with 244 minutes of aligned speech text, audio, and gesture training data in total. Using the GENEA challenge train and test set (Kucherenko et al.) (2020b) also enables us to benchmark results provided test cases, including baselines and submitted models. Since this research
5

---

Under review as a conference paper at ICLR 2022
focuses gaining the most semantic information from text inputs, do not use the provided audio data in our approach.
The dataset is captured with a 53 marker setup and 20 Vicon cameras at 59.95 frames per second (FPS). The motion data was stored as a time series Euler rotations for each joint in the BioVis Hierarchy format (BVH) with 59.95 frames per second. Since we focus on the upper body, obtain the corresponding 15 upper-body joints out of the available 69 body joints. Moreover, Euler angles were converted to joint positions in 3D space and normalized regarding shoulder length. We also down-sampled gestures to the frame rate of 20 FPS and removed finger motions due to the low accuracy of recordings. Although the dataset provides aligned word transcription in JSON format, it is inaccurate and might mislead the model. We hence applied the Gentle forced aligner algorithm 2017) to obtain exact timing information for each word.
(Ochshorn & Hawkins
4.1 DAE
We train our system on 15 joints from the upper body, including: Spine, Spinel, Spine2, Spine3, Neck, Neck1, Head, RightShoulder, RightArm, RightForeAmm, RightHand, LeftShoulder LeftArm, Left and LeftHand. Furthermore, we used 3x3 rotational matrices instead of 3D coordi- nates of each 15 joints to train the DAE network. Thus, the size of input and output vectors for pose representation at frame-level was 3x3x15=135.. We also standardized each dimension to a mean of zero and a maximum absolute value of one for fast convergence training.
The DAE consists of one linear layer of input size to the bottleneck dimensionality followed by Tanh activation as an encoder and a linear layer of bottleneck to input size as a decoder and learning rate of 0.001. We injected Gaussian noise of standard deviation 0.1 to input data and the dropout rate was set to 0.2. To decide proper dimensionality for the bottleneck, we trained the model with different hidden dimensions fo 20 epochs. Similar to TThngthhii et al. (202T); Kucherenko et al. (019)) we picked 40 dimensions to represent every single motion frame.
4.2 VQ-DVAE
In this section, we explain further details of the VQ-DVAE model as well as the training process. At this stage, aimed to map a sequence of s; to latent representation. We selected sequences with length of 30 frames and a stride size of 10 frames. Also, considering the joint velocity as a
proxy of motion dynamics (Yoon et al. 2019: Kucherenko al., 2020a 2021a), we concatenated
the derivative of poses representations to the input. Therefore, we feed 40+40 = 80 features per frame to the autoencoder encode and reproduce VQ-DVAE consists of an encoder, quantized latent space embeddings as bottleneck, and decoder. In order to learn sequential data, by stacking Bi-directional GRU networks on top of each other, we defined a multi-layer bi-directional recurrent neural network architecture for both encoder and decoder networks with the hidden size of 200. Consequently, we compress 80 * 30 2400 dimensions to the continuous latent space of 200 2 400 dimensions. Afterward, we quantize this variable into the nearest neighbour embedding e, E R D = 400, i E 1,2, .... K and feed it to the decoder as discussed before. We empirically chose the number of embedding vectors A equal to 300.
4.3 WEAKENED DECODERS
As mentioned earlier, VAEs suffer from posterior collapse when strong decoder network is used. One simple yet effective solution to this problem could be weakened decoders (Yang et al.) 2017] Semeniuta et al. 2017). A conditional RNN decoder receives the last generated output frame as input to the current step. In our problem, continuity, short-range correlation, is a strong assumption (Alexanderson et al., 2020, Srivastava et al. 2015). For instance, the joint positions of frame are the same as the previous one with subtle changes. A conditioned decoder simply determines these correlations and neglects extremely subtle movements requiring long-term dependencies (Srivastaval et al. 2015). An unconditioned decoder that does not receive that input enforces the encoder to find this information and put it into the encoded vector. To learn a more robust and informative represen- tation, weaken the decoder by freezing its weights while minimizing the ELBO loss. Training the model with no gradient affecting the decoder enforces the encoder to learn informative and ro- bust representation. In this setup, the decoder only propagates the encoder output through time, and

---

Under review as a conference paper at ICLR 2022
the encoder carries reconstruction loss. However, after 20 epochs, we also trained the decoder in conditioned fashion for another 20 epochs concerning smooth reproducibility at inference time.
EVALUATION
In this section, we elucidate our comprehensive experiments validate the effectiveness of the proposed method. We present an ablation study in the Appendix to substantiate the effectiveness of all the components of VQ-DVAE. We describe the baselines systems as well as the objective and subjective measures used assess method.
5.1 BASELINE SYSTEMS
We assessed our system against the ground-truth, and three recently published co-speech gesture generation systems. Our ground-truth taken from the natural motions of the actor for given speech segment. Baseline Text (BT) from Yoon et al. (2019) is the most similar work to ours since both used Fasttext word vectors (Bojanowski et al. (017)) our features from the input text transcript. Also, we both used the Bahaduna RNN Encoder-Decoder architecture with a soft at- tention mechanism (Bahdanau al., 20144 Cho al. 2014). The encoder extracts text features, model maps
and the decoder generates poses frame-by-frame. However, our sequence-to-sequeen extracted features from the text into series of motion symbols, with each symbol representing 30 frames of motion. Baseline BA proposed by Kucherenko et al. (2019) used audio as input to generate co-speech gestures composed of encoder-decoder structure. In this model, the encoder maps audio input to a sequence of learned pose representations while the decoder projects them back to poses. The third baseline model (BTA) (Korzun et al., (020) achieved impressive results KKcherenko et al. 2020b) using both audio and text modalities. In order to extract acoustic and textual features, BTA combined two separate encoders, one for each modality. This work was in- teresting since it used both modalities and scored the highest in human liknan-likensss and sconnd-highest in terms of appropriateness among (Yoon et al.] 2019; Kucherenko et al. 2019 Alexanderson et al. et al. 2021 Thangthai et al.) 2021).
2020
5.2
OBJECTIVE MEASURES
Although improving subjective measures is our ultimate goal, they are costly, time-consuming, and require human labour. Moreover, evaluating generative models is tricky since there is not one-and- only true motion sequence for a given speech utterance. Accordingly. we support our experimental results with numerical evaluations recently proposed unveil gesture qualities.
Average jerk and velocity metrics have been used in prior work to quantitatively assess gesture systems (Kucherenko al. (0206)). We used average jerk and velocity metrics proposed by Kucherenko et al. (2019). Jerk, the third derivative of joint positions, characterizes smoothness by calculating the rate of acceleration joints as an objective metric to compare systems. We also posit that plausible generated gestures should follow similar velocity characteristics to the ground truth. Note that the gestures were con- verted from joint rotational angles to 3D joint positions. Hellinger distance (Hellinger 1909) has extensively been used to compare two distributions. Accordingly, we compare each system to the ground truth to see how close the gestures are to the natural motion w.r.t that structural aspect.
movement. We used the average over third derivative of
Additionally, we introduce an objective metric to evaluate gesture diversity over the long term, thanks to proposed discretized latent space at the sequence level. Indeed, used our trained VQ-DVAE model to cluster predicted gestures for each condition at the sequence level. We adopted the Hellinger metric to evaluate how system's 's output distribution follows the real-world gesture vocabulary distribution. The Hellinger distance metric was applied between each system and the ground truth evaluate its closeness in terms of diversity.
5.3 SUBJECTIVE MEASURES
Our ultimate purpose is to generate realistic gestures that are as valid and natural as the ground truth. Therefore, we conducted human study evaluate our performance subjectively. Comparable to
7

---

Under review as a conference paper at ICLR 2022
Table 1: Objective Evaluation Results
Table 2: Subjective Evaluation Results
System
Average Jerk
Valocity
Hellinger Distance Acceleration
Diveritty
System
Meon
Human- - Liveness
57D
Meau
Appropriatenes
371
Medi
Diversity
577
GT
1588 633 61
0
GT
0.25
0.21
795,00
0,0988
0.1651
0:20
BA
1273.4 43
21
0,0783
0.1 0.1582
0.6208
BA
0.30
0.20
0.92
0.21
0.45
0.19
BTA
904 10 + 81 61
0.070
0.1268
0.5715
RTA
0.50
0.22
0.53
0.22
0.45
0.20
Proposed
1601
1 104.03
0.0511
0.06665
0.59181
Proposed
0.50
0.20
0.45
0.22
0.53
0.18
the prior work in this area (Ginosar et al. 2019
Salvi et al. [200 Kucherenko et al. 2020b]] Alexan
derson et al. 20200] Thangthai al. 2021 Lu al. 2021 Yoon et al. 2019 Korzun et al. [020)]
Kucherenko et al. 2019). we eva evaluated generated gestures regarding perceived human-likened and appropriateness the motion given the speech context. These two measures indeed disentangle motion quality from the relevancy to the speech context. In addition, we also assessed systems in terms of diversity as opposed repetitive movements over longer intervals. Specifically, we asked the following questions in our human-study: "How human-like does the motion appear?", "How appropriate are the gestures for the speech?" and "How diverse does the motion appear?".
For the appropriateness evaluation, we selected 40 speech segments each contains complete sen- tence or phrase with an average of 10 seconds. Similarly, we selected 40 muted segments randomly for the human-likeness study and diversity study with fixed length of 10 seconds and 20 seconds, respectively.
5.4 EXPERIMENTAL PROCEDURE
We recruited participants from the Amazon Mechanical Turk (MTurk) to evaluate the results sub- jectively. MTurk is crowdsoureing website proven effective in recruiting more diverse participants than in college experiments (Keith et al.) 2017). MTurk provides the option to set requirements for the intended population based on different factors, i.e. age, gender, nationality, HIT approval rate. The HIT approval rate is defined as the percentage of completed work that other requestors have approved for that specific person. We posted a Human Intelligence Task (HIT) on MTurk containing an external link to our web interface. More details about our human-study interface are provided in the Appendix section. Participants were asked to complete the HIT, obtain a survey completion code, and enter it in MTurk to be compensated based on the minimum wage per hour in their coun- try. We restricted our intended population to collect high-quality results by setting the minimum HIT approval rate 95% and approved HIT's greater than 5000. We also required participants to be located in Canada due to ethical review board policies.
RESULTS
We recruited 24 participants and filtered out 6 judgments based on attention checks and ratings the ground-truth gestures as a superior system over all conditions. The average age was 33.1 (STD=8.9) years with 12 men and women, all English native speakers living in Canada. Among accepted judgments, the average experiment duration was approximately 41 minutes. The mean and standard deviation (STD) of ratings for the three studies are presented in Table
Objective results on the 20 minute test set for all conditions is summarized in Table 3 We report the mean and standard deviation of the average jerk. Table also presents the Hellinger distance of velocity and acceleration histograms to the ground truth. The diversity column shows the Hellinger distance on vocabulary frequency based upon assigned labels to constituent sequences (30 frames) obtained from VQ-DVAE.
DISCUSSION
Gesture generation is challenging, especially in terms of appropriateness for the speech. Conse- quently, the difference between systems is subtle and long way from the natural gestures. Objec- tive metrics are consistent with the subjective results; however, we can see that different systems performed differently evaluation metrics.
The subjective study shows that our model is preferred over all the systems in terms of human like- ness. This is also aligned with similarity metries in objective evaluation, where the proposed method
8

---

Under review as a conference paper at ICLR 2022
ranked first in fundamental factors, ie. jerk, acceleration, and velocity. Indeed, that our representation learning method efficiently captured essential features and reproduced more natural gestures. Prior research has also shown that vector quantization led to more diverse and high-quality outputs. The BTA is the best in terms of appropriateness and our method is ranked sec- ond. We only used text features similar the BT, while BTA used more advanced semantic features alongside acoustic features. We interpret it as the effectiveness of quantization against continuous regression resulted in better performance. Furthermore, raters perceived our model outputs more diverse, showing our model's 's capability to overcome the mode collapse problem. As shown in Table 3] the proposed gesture generation method significantly closer the ground truth than the BT and approximately similar to BA and BTA in terms of the appearance of gesture sequences distribution.
conclude
Subjective and objective results suggest that we avoided the average gesture problem and generated more diverse and natural outputs than repetitive gestures. Vector quantization of motion representa- tion narrowed the gesture space complication down to a set of vocabulary in codebook that clusters similar sequences into specific motion symbol. Although we significantly reduced the dimension- ality with discrete encoding, both objective and subjective measurements show that generated ges- tures maintain high quality in terms of naturalness. Quantization of motion representation space promotes the gesture generation model to focus on longer dependencies rather than local correlation. Consequently, we can apply machine translation algorithms from the NLP community to perform co-speech gesture generation tasks using entropy family loss instead of regression type.
We found that weakening the decoder strongly impacts the learned representation during the training process, resulting in better clusterings. However, its reproducibility was not smooth, and we later trained the decoder for 20 epochs while other parameters were fixed.
7.1 LIMITATIONS AND FUTURE WORK
The first limitation of our work was that the dataset used in this research was limited to person speaking in a monologue situation. Different persons have their own gesticulating style and different environments lead to complex speech and gestures. In future work we should consider datasets with mixed speakers in different environments. Another limitation of our research is that we trained our model on uppe.r body while fingers were excluded. Lower body is also important e.g. stepping forward and backward motions, standing still, approaching, facial expressions and fingers motions.
Although our system showed a significant improvement over baselines, it is still far behind human- generated motions. Kucherenko et al. (2020b) reported that raters were inclined to rate mismatched speech gestures from the ground truth, higher than synthesized gestures for given input. Therefore, it is possible that we achieved a high appropriateness rate, especially compared to the Baseline Text (BT), due to the higher numan-likenee quality of our system instead of appropriateness.
In this study, we did not involve audio and only used speech text. Punctuations such as question marks were also not provided in the source corpus. Therefore, generated gestures may lack move- ments relevant to acoustic features such intonation at the end of sentence to indicate question (Pouw et al. 2020). Current work can be improved by adding punctuation using an automatic punc- tuation restoration system (Courtland et al. 2020). The proposed method can be extended by looking for correspondences between motions and audio features.
8 CONCLUSION
We proposed a fully unsupervised co-speech gesture generation system that combines representa- tion learning methods and machine translation algorithm. To the best of our knowledge, this is the first method that uses a representation learning algorithm at the sequence-level for the text-to- gesture generation task and provides a new state-of-the-art baseline in this area. We connected pose (frame-level) representation learning method and quantized motion (sequence-level) rep- resentation learning, each trained separately. We also introduced a new objective evaluation metric that calculates the Hellinger distance of motions occurrence distributions as a measure for diversity. Finally, we trained a machine translation model to translate English sentences to gesture vocabulary. We found that the discretized motion space causes the model to focus on longer dependencies and generate more diverse and human-like like gestures. The main limitation of our work was that we did not include audio features. In future work, we will consider more advance semantic features, audio fea-

---

Under review as a conference paper at ICLR 2022
tures as well emotional features, i.e. valence and arousal (Lim et al. 2011). The dataset was also not large enough to generalize a seeh-pefrrrancee model, especially in terms of appropriateness. This be addressed by creating proper dataset through automated process from currently available datasets. Furthermore, we will investigate hierarchical vector quantization (Razavi et al, 2019), to generate more natural and diverse gestures.
REPRODUCIBILITY
Towards reproducibility of our results, we include here link our code implementing the proposed model as well as statistical analysis used in this paper:
We also provide the following link to visualizations and the human-study interface described in our Experiments section:
Finally, the Appendix includes more details regarding the human study interface, in order for others to reproduce the subjective results.
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4432 4441, 2019.
Chaitanya Ahuja, Dong Won Lee, Yukiko I Nakano, and Louis-Philippe Morency. Style transfer for co-speech gesture animation: multi-speaker conditional-mixture approach. European Conference on Computer Vision, pp. 248-265. Springer, 2020.
Simon Alexanderson, Gustav Eje Henter, Taras Kucherenko, and Jonas Beskow. Style-controllable speech-driven gesture synthesis using normalising flows. In Computer Graphics Forum, vol- ume 39, pp. 487-496. Wiley Online Library, 2020.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 213-223. PMLR, 2017.
Alexei Baevski, Steffen Schneider, and Michael Auli. wq-wav2vec: Self-supervised learning of discrete speech representations. arXiv preprint arXiv:1910005453, 2019.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. arXiv preprint 100.v::449904755 2014.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv: 2018.
Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle. Greedy layer-wise training of deep networks. In Advances neural information processing systems, pp. 153-160, 2007.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 300..4328 2013.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146, 2017.
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowiez, and Samy Ben- gio. Generating sentences from continuous space. arXiv preprint arXiv: /5/ 1.06449, 2015.
Cynthia Breazeal, Cory D Kidd, Andrea Lockerd Thomaz, Guy Hoffman, and Matt Berlin. Effects of nonverbal communication on efficiency and robustness in human-roke monont teamwork. 2005 IEFF/RSJ international conference intelligent robots and systems, pp. 708-713. IEEF, 2005.
10

---

Under review as a conference paper at ICLR 2022
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 2020.
Brian Butterworth and Uri Hadar. Gesture, speech, and computational stages: A reply to mcneill. 1989.
Justine Cassell, David McNeill, and Karl-Erik McCullough. Speech- seeech-gesture mismatches: Evidence for one underlying representation of linguistic and nonlinguistic information. Pragmatics & cog- nition, 7(1):1-34, 1999.
Justine Cassell, Hannes Högni Vilhjálmsson, and Timothy Bickmore. Beat: the behavior expression animation toolkit. In Life-Like Characters, pp. 163-185. Springer, 2004.
Chung-Cheng Chiu, Kouis-Philippe Morency, and Stacy Marsella. Predicting co-verbal gestures: a deep and temporal modeling approach. In International Conference on Intelligent Virtual Agents, pp. 152-166 122.166.66 Springer, 2015.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv: 1078, 2014.
Jan Chorowski, Ron J Weiss, Samy Bengio, and Aäron van den Oord. Unsupervised speech repre- sentation learning using wavenet autoencoders. JEEE/ACM transactions on audio, speech, and language processing, 27(12):2041-2055 2019.
Mingyuan Chu and Sotaro Kita. Co-thought and co-speech gestures are generated by the same action generation process. Journal of Experimental Psychology: Learning, Memory, and Cognition, 42 (2):257, 2016.
Blender Online Community. Blender - a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018.
wwww hwt.:w..ww......mm.mmwm org
Maury Courtland, Adam Faulkner, and Gayle McElvain. Efficient automatic punctuation restoration using bidirectional transformers with robust inference. In Proceedings of the 17th International Conference on Spoken Language Translation, pp. 272-279, 2020.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International conference on machine learning, pp. 933-941. PMLR, 2017.
Jan P De Ruiter, Adrian Bangerter, and Paula Dings. The interplay between gesture and speech in the production referring expressions: Investigating the tradeoff' hypothesis. Topics in Cognitive Science, 4(2):232-248) 2012.
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. arXiv preprint arXiv:1807.03813 2018
Ylva Ferstl and Rachel McDonnell. Investigating the use of recurrent motion modelling for speech gesture generation. In Proceedings of the 18th International Conference Intelligent Virtual Agents, pp. 93 98, 2018
Ylva Ferstl, Michael Neff, and Rachel McDonnell. Multi-objective adversarial gesture generation. In Motion, Interaction and Games, pp. 1-10. 2019.
Ylva Ferstl, Michael Neff, and Rachel McDonnell. Adversarial gesture generation with realistic gesture phasing. Computers & Graphics, 89: 19:117, 2020.
Ylva Ferstl, Michael Neff, and Rachel McDonnell. Expressgesture: Expressive gesture generation from speech through datahase matching. Computer Animation and Virtual Worlds, pp. e2016, 2021.
Katerina Fragkiadaki, Sergey Levine, Panna Felsen, and Jitendra Malik. Recurrent network models for human dynamics. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4346-4354, 2015.
11

---

Under review as a conference paper at ICLR 2022
Cristina Gârbacea, Aäron van den Oord, Yazhe Li, Felicia SC Lim, Alejandro Luebs, Oriol Vinyals, and Thomas C Walters. Low bit-rate speech coding with vq-vae and a wavenet decoder. In ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp- 735-739. IEEE, 2019.
Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, and Jitendra Malik. Learn- ing individual styles of conversational gesture. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 3497-3506, 2019.
lan Goodfellow, Honglak Lee, Quoc Le, Andrew Saxe, and Andrew Ng. Measuring invariances in deep networks. Advances in neural information processing systems, 22:646-655 2009.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. http:
//www. www..eeeppearninooooooo.k org
John Haas. A history of the unity game engine. 2014.
Dai Hasegawa, Naoshi Kaneko, Shinichi Shirakawa, Hiroshi Sakuta, and Kazuhiko Sumi. Evalua- tion of speech-to-gesture generation using bi-directional Istm network. In Proceedings of the 18th International Conference on Intelligent Virtual Agents, pp. 79-86, 2018.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick. Lagging inference networks and posterior collapse variational autoencoders. arXiv preprint arXiv:1901.05534 2019.
Ernst Hellinger. Neue begründung der theorie quadratischer formen von unendlichvielen veränderlichen. Journal für die reine und angewandte Mathematik, 1909(136)211-271,, 1909.
Gustav Eje Henter, Simon Alexanderson, and Jonas Beskow. Moglow: Probabilistic and controllable motion synthesis using normalising flows. ACM Transactions on Graphics (TOG), 39(6):1-14, 2020.
Alejandro Hernandez, Jurgen Gall, and Francese Moreno-Noguer. Human motion prediction via spatio-temporal inpainting. In Proceedings of the JEEE/CVF International Conference Com- puter Vision, pp. 7134-7143, 2019.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. 2016.
Chien-Ming Huang and Bilge Mutiu. Robot behavior toolkit: generating effective social behaviors for robots. In 2012 7th ACM/IEEE International Conference on Human-Roban Interaction (HRI), pp. 25-32. IEEE, 2012.
Daniel Im Im, Sungjin Ahn, Roland Memisevic, and Yoshua Bengio. Denoising criterion for vari- ational auto-encoding framework. In Proceedings of the AAAI Conference Artificial Intelli- gence, volume 31, 2017.
Carlos T Ishi, Daichi Machiyashiki, Ryusuke Mikata, and Hiroshi Ishiguro. A speech-driven hand gesture generation method and evaluation android robots. IEEE Robotics and Automation Letters, 3(4):3557-3664, 2018.
Daniel Jurafsky and James H. Martin. Speech and Language Processing (2Nd Edition). Prentice- Hall, Inc., Upper Saddle River, NJ, USA, 2009. ISBN 0131873210.
Melissa G Keith, Louis Tay, and Peter Harms. Systems perspective of amazon mechanical turk for organizational research: Review and recommendations. Frontiers in psychology, 8:1359, 2017.
Adam Kendon. Gesture: Visible action as ulterance. Cambridge University Press, 2004.
12

---

Under review as a conference paper at ICLR 2022
Diederik P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1xl convolutions. arXiv preprint arXiv: arXiv:1807.03039, 2018.
Diederik Kingma and Max Welling.
Auto-encoding variational
bayes. arXiv preprint
arXiv: 14, 2013.
Durk P Kingma, Tim Salimans, Rafal Jozefowiez, Xi Chen, lly Sutskever, and Max Welling. Im- proved variational inference with inverse autoregressive flow. Advances in neural information processing systems, 29:4743 4751, 2016.
Michael Kipp. Gesture generation by imitation: From human behavior to computer character ani- 2005.
mation. Universal-Publise
Vladislav Korzun, Ilya Dimov, and Andrey Zharkov. The finemotion entry the genea challenge 2020. In Proc. GENEA Workshop. https://doi. org/10, volume 5281, 2020.
Taras Kucherenko. Data driven non-verbal behavior generation for humanoid robots. In Proceedings of the 20th ACM International Conference Multimodal Interaction, pp. 520-523, 2018.
Taras Kucherenko, Dai Hasegawa, Gustav Eje Henter, Naoshi Kaneko, and Hedvig Kjellström. An- alyzing input and output representations for speech-driven gesture generation. In Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents, pp. 97-104, 2019.
Taras Kucherenko, Patrik Jonell, Sanne van Waveren, Gustav Eje Henter, Simon Alexandersson, Iolanda Leite, and Hedvig Kjellström. Gesticulator: A framework for semantically-aww speech- driven gesture generation. Proceedings of the 2020 International Conference on Multimodal Interaction, pp. 242-250, 2020a.
Taras Kucherenko, Patrik Jonell, Youngwoo Yoon, Pieter Wolfert, and Gustav Eje Henter. The genea challenge 2020: Benchmarking gesture-generation systems on common data. 2020b.
Taras Kucherenko, Dai Hasegawa, Naoshi Kaneko, Gustav Eje Henter, and Hedvig Kjellström. Moving fast and slow: Analysis of representations and post-processing speech-driven au- tomatic gesture generation. International Journal of Human-Compute Interaction, pp. 1-17, 2021a.
Taras Kucherenko, Rajmund Nagy, Patrik Jonell, Michael Neff, Hedvig Kjellström, and Gustav Eje Henter. Specet2propcttis22estteesss Gesture- property prediction as a tool for generating repre- sentational gestures from speech. In Proceedings of the 11th ACM International Conference on Intelligent Virtual Agents, pp. 145-147, 2021b.
Taras Kucherenko, Rajmund Nagy, Michael Neff, Hedvig Kjellström, and Gustav Eje Henter. Multi- modal analysis of the predictability of hand-gesture properties. arXiv preprint arXiv:2108.05762, 2021c.
David Lapakko. Communication is 93% nonverbal: urban legend proliferates. Communication and Theater Association of Minnesota Journal, 34(1):2, 2007.
Sergey Levine, Christian Theobalt, and Vladlen Koltun. time-time prosody-driver synthesis of body language. In ACM SIGGRAPH Asia 2009 papers, pp. 1-10. 2009.
Sergey Levine, Philipp Krähenbühl, Sebastian Thrun, and Vladlen Koltun. Gesture controllers. In
ACM SIGGRAPHI 2010 papers, pp. 1-11. 2010.
Jing Li, Di Kang. Wenjie Pei. Xuefei Zhe, Ying Zhang, Zhenyu He, and Linchao Bao. Au- dio2gestures: Generating diverse gestures from speech audio with conditional variational au- toencoders. arXiv preprint arXiv:2 06720, 2021.
Angelica Lim, Tetsuya Ogata, and Hiroshi G Okuno. Converting emotional voice to motion for robot telepresence. In 2011 11th IEEE-RAS International Conference Humanoid Robots, pp. 472-479. IEEE, 2011.
Hung Yu Ling, Fabio Zimno, George Cheng, and Michiel Van De Panne. Character controllers using motion vaes. ACM Transactions Graphics (TOG), 39-4),40-1, 2020.
13

---

Under review as a conference paper at ICLR 2022
JinHong Lu, TianHang Liu, ShuZhuang Xu, and Hiroshi Shimodaira. Double-deccae: Estimation of body gestures from speech waveform. ICASSP 2021-2021 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 900-904. IEEE, 2021.
James Lucas, George Tucker, Roger Grosse, and Mohammad Norouzi. Understanding posterior collapse in generative latent variable models. 2019a.
James Lucas, George Tucker, Roger Grosse, and Mohammad Norouzi. Don't blame the elbo! linear vae perspective on posterior collapse. Advances in Neural Information Processing Systems, 32:9408-948 -9188, 2019b.
Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. arXiv preprint arXiv:1711. 2017.
Andy Lüccknng Kirsten Bergman, Florian Hahn, Stefan Kopp, and Hannes Rieser. Data-based anal- ysis of speech and gesture: The bielefeld speech and gesture alignment corpus (saga) and its applications. Journal on Multimodal User Interfaces, 7(1):5-18, 2013.
David McNeill. Hand and mind. De Gruyter Mouton, 2011.
Albert Mehrabian. Nonverbal communication. Routledge, 2017.
Lu Mi. Macheng Shen, and Jingzhao Zhang. A probe towards understanding gan and vae models. 2018.
arXiv preprint
Takashi Minato, Michihiro Shimada, Hiroshi Ishiguro, and Shoji Itakura. Development of an android robot for studying human-robot interaction. In International conference Industrial, engineer- ing and other applications of applied intelligent systems, pp. 424 434. Springer, 2004.
Michael Neff, Michael Kipp, Irene Albrecht, and Hans-Pete Seidel. Gesture modeling and anima- tion based on a probabilistic re-creation of speaker style. ACM Transactions on Graphics (TOG), 27(1):1-24, 2008
RM Ochshorn and Max Hawkins. Gentle forced aligner. github. com//owerqualltygeentte 2017.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: 1609.03499 2016.
Aaron den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learn- ing. arXiv preprint arXiv:1711.00937. 2017.
Wim Pouw and James A Dixon. Quantifying gesture-speech synchrony. In the 6th gesture and speech in interaction conference, pp. 75-80. Universituctsbibliothek Paderborn, 2019.
Wim Pouw, Steven Harrison, and James A Dixon. Gesture-speech physics: The biomechani- basis for the emergence gesture-speech synchrony. Journal of Experimental Psychology: General, 149(2):391, 2020.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. In Advances in neural information processing systems, pp. 14866-14876, 2019.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and ap- proximate inference in deep generative models. In International conference on machine learning, pp. 1278-1286. PMLR, 2014.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517, 2017.
Giampiero Salvi. Jonas Beskow, Samer Al Moubayed, and Bjöm Granström. sefface--spech-- driven facial animation for virtual speech-reading support. EURASIP journal on audio, speech, and music processing, 1-09: 2009.
14

---

Under review as a conference paper at ICLR 2022
Abraham Savitzky and Marcel JE Golay, Smoothing and differentiation of data by simplified least squares procedures. Analytical chemistry, 36(8):1627-1639, 1964.
Stanislau Semeniuta, Aliaksei Severyn, and Erhardt Barth. A hybrid convolutional variational au- toencoder for text generation. arXiv preprint urXiv:/702.002: 2017.
B Series, Method for the subjective assessment of intermediate quality level of audio systems. International Telecommunication Union Radiocommunncaation Assembly, 2014.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 3310-3320, 2017.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised learning of video rep- resentations using Istms. In International conference on machine learning, pp. 843-852. PMLR, 2015.
Kenneth Stewart, Andreea Danielescu, Lazar Supic, Timothy Shea, and Emre Neftei. Gesture sim- ilarity analysis on event data using hybrid guided variational auto encoder. arXiv preprint arXiv:2104.00165. 2021.
Ausdang Thangthai, Kwanchiva Thangthai, Arnon Namsanit, Sumonmas Thatphithakkul, and Sit- tipong Saychum. Speech gesture generation from acoustic and textual information using Istms. In 2021 18th International Conference on Electrical Enginneriing/Electooiic Computer, Telecom- munications and Information Technology CoanoologyyyCC (ECTI-CON), pp. 718-723. IEEE, 2021.
Sergey Tulyakov, Ming-Yo Yu Liu, Xiaodong Yang, and Jan Kautz. Mocogan: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1526-1 1535, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pierre-Antoin Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in deep network with a local denoising criterion. Journal of machine learning research, 11(12), 2010.
Qiang Wang, Bei Li, Tong Xiao, Jingbo Zhu, Changliang Li, Derek F Wong, and Lidia S Chao. Learning deep transformer models for machine translation. arXiv preprint (.XXv:1906.0111 2019.
Sarah Woods, Kerstin Dautenhahn, and Joerg Schulz. The design space of robots: Investigating children's ' views. In RO-MAN 2004. 13th IEEE International Workshop on Robot and Human Interactive Communication (IEEE Catalog No. 04TH8759), pp. 47-52. IEEE, 2004.
Xinchen Yan, Akash Rastogi, Ruben Villegas, Kalyan Sunkavalli, Eli Shechtman, Sunil Hadap. Ersin Yumer, and Honglak Lee. Mt-vae: Learning motion transformations to generate multimodal human dynamics. In Proceedings of the European Conference on Computer Vision (ECCV), pp- 265-281, 2018
Zichao Yang. Zhiting Hu, Ruslan Salakhutdinov, and Taylor tional autoencoders for text modeling using dilated convolutions. In International conference on machine learning, pp. 3881-3890. PMLR, 2017.
Improved varia-
Youngwoo Yoon, Woo-Ri Ko, Minsu Jang, Jaeyeon Lee, Jaehong Kim, and Geehyuk Lee. Robots leam social skills: End-to-end learning of co-speech gesture generation for humanoid robots. In 2019 International Conference Robotics and Automation (ICRA), pp. 4303-4309. IEEE, 2019.
Youngwoo Yoon, Bok Cha, Joo-Haeng Lee, Minsu Jang, Jacyeon Lee, Jaehong Kim, and Geehyuk Lee. Speech gesture generation from the trimodal context of text, audio, and speaker identity. ACM Transactions on Graphics (TOG), (9(6))19111)) 2020.
15

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
A.1 INTERFACE
The human-study interface, shown in Fig. 3 was implemented in Unity3D HHas 2014) and pub- lished in WebGL format. We used Blender software (Community 2018) to convert Bio Vision Hierarchy (BVH) files to Filmbox (fbx) format that Unity3D can import humanoid animation. The avatar has 15 joints, excluding fingers in order to be consistent with our benchmarks.
The landing page of the experiment interface presented a consent form and instructions on how to use the interface with quick guide tour. We randomly picked 10 clips from each of the three afore- mentioned segment pools to keep the experiment within 30 minutes and avoid exhausting raters. The first 10 clips shown to participants were from the human-likenes: study. Next, raters were asked to use their headphone and test the audio settings to rate appropriateness stimuli. In the end, they were asked to rate 10 muted long clips from the diversity pool. To evaluate several systems simultane- ously, we used a methodology similar to the (Kucherenko et al. 2020b) inspired by the Multiple Stimuli with Hidden Reference and Anchor test (Series/2014: We used page- wise strategy such that on each page, we included motions from all conditions corresponding to a specific segment assigned that page. Therefore, in parallel. We selected the 100-point rating scale and labelled them: "Bad", "Poor", "Fair", and randomized the order of clips on each page. After rating 30 stimuli, participants were asked to complete demo- graphic questionnaire. To if participant was paying enough attention and eliminate inattentive raters, we included an attention check on each page where we asked the participant to rate a specific value for a stimulus, On each page, the attention video and corresponding answer were selected randomly. Afterward, we excluded collected data with more than four failures in attention checks.
could employ pairwise statistical tests since stimuli were rated
"Excellent" within intervals of 20 points (Kucherenko et al., (0206). Meantime,
Please watch all videos and rate each clip according to the question bellow.
How appropriate are the gestures for the speech?
00% Bad
00% Bad
00% Bad
00% Bad
Video 5
00% Bad
00% Bad
Page
Press Video button play
Figure A screenshot of page with stimuli from the human-stur study interface.
A.2 ABLATION STUDY
To evaluate the essence of the contributions of each component in our method, we assess the latent representation space as well as the final output within an ablation study. To evaluate the latent representation, we analyze how two shifted sequences related to each other in latent space. We defined Neighbour Sample Distance (NSD) metric, which measures the average distance between sequence and its shifted version. We assume that in good representation space, shifted samples' distances is smaller concerning the average distance in that space. We also apply the Fréchet gesture distance (FGD) Yoon et al. (2020) to compare distributions the latent gesture space between
16

---

Under review as a conference paper at ICLR 2022
Table 3: Ablation Study Results
Syeee
MA
TOAME
FoB
110
00
51.0
14:03
Pepood
042 41
21-0
4.3
15:05
836
089
010
real and generated gestures. The more generated motions similar to the ground truth on the latent distribution, the smaller FGD value is.
17


Reviews:
[{"review_id": "zRD0C1JpL5E", "reviewer": "Reviewer_RQVL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n1. The idea of representing gesture sequences as vector-quantized features is technically sound and clearly presented.\n2. The experimental results show reasonable benefits over prior work, especially in objective evaluation metrics.\n\n### Weaknesses\n1. Related work appears incomplete. There is at least one other work on generating gestures from input text transcripts:\nBhattacharya, U., Rewkowski, N., Banerjee, A., Guhan, P., Bera, A., & Manocha, D. (2021, March). Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agents. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR) (pp. 1-10). IEEE.\n\n2. Some technical aspects of the proposed method are unclear to me:\n    1. For the DAE (Section 4.1), how do the authors ensure that the decoder outputs are valid rotation matrices?\n    2. What reconstruction loss is used for the DVAE that works with gesture sequences? Is it a mean squared error (MSE) similar to that used by the frame-level DAE? Since MSE forces the mean values of the poses in the input and the output gestures to be close, it may cause the outputs to be temporally smoothed out. If that is the case, then how does the proposed method provide the largest average jerk (Table 1, last row)? If not, do the authors perform any additional steps to safeguard against the smoothing?\n    3. Yoon et al. 2020, as referred to by the authors, have already proposed an objective measure for gesture diversity, dubbed Fréchet Gesture Distance (FGD). Since the authors propose their own metric, the Hellinger distance, for the same purpose, it would have been appropriate to have at least some discussion on how the two metrics compare. More importantly, what is the motivation to propose a new metric over the existing one?\n    4. Before the user study, were the users shown any good and bad examples (or something similar), as determined by the authors, for the questions they were subsequently asked?\n    5. Did the authors perform any inter-rater agreement, e.g., using Fleiss' Kappa measure, to verify the reliability of the responses?\n3. In Section 7.1, the authors mention that they were limited to one person in their work. However, multi-person datasets are already available, such as the TED Gesture Dataset proposed by Yoon et al. 2019, and used by both Yoon et al. 2019 and Yoon et al. 2020. Are there any specific reasons why the authors did not work with this dataset? Moreover, in Section 4 (page 5, last line), the authors mention that they only focus on upper body gestures. Again, were they any specific reasons to eliminate the lower-body gestures from the training and the testing processes?", "summary_of_the_paper": "The authors present a method to generate co-speech gestures by learning a vector-quantized representation space of gestures coupled with a machine translation from natural language sentences to that representation space. They design a denoising variational autoencoder (DVAE) to learn the encoding space corresponding to gesture sequences, and then quantize that encoding space using nearest neighbor embeddings. Once the vector quantized representations are learned, they use a sequence-to-sequence network to translate the text transcript of the input speech to those representations, which they subsequently decode into the output co-speech gestures. They train and test their method on the Trinity Gesture Dataset and show the improvements achieved by their method on both objective and subjective evaluations.", "summary_of_the_review": "The proposed method is technically sound and attempts to solve the challenging problem of co-speech gesture generation using an insightful approach, that of vector-quantizing the latent representation space of gestures and mapping from natural language sentences to it. However, a number of technical details were unclear to me, which makes the overall method and evaluations hard to follow. I invite the authors to respond to my comments under \"weaknesses\" before I can strongly recommend the paper for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "q6_xCAwvitM", "reviewer": "Reviewer_aN3z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- I appreciate that the related work section which discusses each prior art very well with their cons and pros.\n- The proposed method brings some improvements over the baseline methods for metric jerk, velocity and accelerarion when tested on Trinity dataset.\n\nWeaknesses:\n-  The technical novelty of the proposed method is questionable. It is important to highlight that this is not the first time variational autoencoders are adapted for this task. I found the proposed method incremental, such as adding the denoising property to the existing Vector Quantized Denoising Variational Autoencoder architecture.\n- The paper should include ablation study showing the effectiveness of the components compared to the other possibilities. For example, the authors claim that “….we consolidated a denoising characteristic to the variational autoencoder framework (DVAE), presenting a more flexible and robust posterior distribution approximation than the standard VAE”. However the correctness of this argument was not empirically justified.\n - I found the experimental analysis limited. The proposed method should be validated also on other datasets and following that, it should be compared with other state of the arts too (e.g., \"MoGlow: Probabilistic and Controllable Motion Synthesis Using Normalising Flows, ACM Trans. Graph., code available\"). \nI am aware that prior art, for example, Yoon et al. 2019 was used TED gesture dataset, which is a larger scale dataset compared to Trinity. Another dataset is Gesture-speech dataset collected by Takeuchi et al. Gesture-speech dataset include audio in which one can obtain the transcriptions. Authors should include these datasets into their analysis, to show that the findings generalize across various conditions.\n- It is also an important limitation that the proposed method was tested on a dataset having only monologues. The interaction scenarios is perhaps much more challenging and also more realistic, assuming that such a system can be deployed to social robots which interact with other social agents.\n- “Although our system showed a significant improvement over baseline….” this argument should be justified by a significance test.\n-  The applied human study should be better described in the main paper, it is not clear to me what is the reliability score of the human annotations.\n- Figure captions require a better explanation. E.g., in Fig 1, x, x\\tilde, x\\head should be described as well as Loss should be precisely mentioned as MSE Loss. The similar comment applies for Fig.2.\n- My understating is that the proposed method is not end-to-end. Can you confirm? If so, this fact should be included to the discussions as some of the baseline methods are end-to-end.", "summary_of_the_paper": "This paper tackles with co-speech gesture generation. First a denoising autoencoder is applied to the body pose detected at each frame. Then, a Vector Quantized Denoising Variational Autoencoder learns to create a motion vocabulary by parameterizing the posterior distribution of discrete latents. As the third step,  a two-layered bidirectional gated recurrent neural network with a soft attention mechanism is used for translation from text to gesture vocabulary. The decoder of Vector Quantized Denoising Variational Autoencoder is used to reproduce the entire gesture sequence.", "summary_of_the_review": "- limited technical novelty\n- limited experimental analysis", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "_BCw7d9PgLI", "reviewer": "Reviewer_mN3D", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### **Strengths**\n\n- The choice of learning a representation of a gesture sequence instead of a single frame can indeed provide the model with the ability to generate plausible gesture sequences. This is corroborated by somewhat better subjective evaluation scores.\n\n### **Weaknesses and Suggestions**\n- The novelty of the contributions is overplayed in the paper. Most of the discussion is about prior work on VAE, VQVAE, crossmodal conditional VQVAE decoding [1]. Learning representations for gesture sequence is an incremental contribution (it is also quite similar to Hierarchical VQVAE[6]), but ablation studies discussing its importance is missing. Hence it is hard to judge whether these representations really helped with the performance. \n\n- As the task is to generate a sequence of upper body gestures, it would have been more convincing to see some demo videos. While I appreciate the subjective evaluation studies, it is hard to judge the output of a generative model without actually having seen any examples. \n\n- Method: The overall method explained in Sections 3.1 and 3.2 is effectively paraphrasing the VAE and VQVAE papers. Only a small portion on sequence level representation learning seems to be novel. It would have been better to see significantly more descriptions about the contributions instead of the prior work. \n\n- A big chunk of the paper talks about the representation learning of co-speech gestures, yet there is no analysis of these representations. There are several challenges that could occur with a sequence level codebook, two of which are as follows:\n  - How do two sequences, one of which is a shifted version of the other (i.e. the two sequences are x[0:30] and x[10:40]), appear in the latent space? Are they represented by the same code in the codebook? If yes, how does the decoder figure out where to start the gesture from, 0 or 10? \n  - What is the optimal number of codes in the codebook that is able to represent the manifold of all possible gestures? How is the value of K=300 chosen? Often, if the value of K is higher than optimal, there can be codes that are very close to each other. This is similar to the challenge of choosing more cluster centers than the data requires. Here, what could happen is that the language encoder (in the third step) will now be uncertain about which code to choose from.\n\n- Section 3.3 - In step 3, the text features are used to choose a sequence of codes which are later decoded into gesture sequences. An interesting metric here could be measuring the classification accuracy of these codes. This could also help measure the uncertainty of prediction of the gestures which is not possible in the system with implicit generative models like GANs.\n\n- Section 4.2 VQ-DVAE - The derivatives (i.e speed/velocity) of the pose representations of the input were also fed in with the pose representations as a proxy for motion dynamics. This makes an assumption that transformations such as addition or subtraction in pose representations are equivalent to the same transformations in the original pose. This is in general not true as the network used here is non-linear. \n\n- It is not clear when the decoder is frozen (Section 4.3 on Weakened decoders)? Is it during step 1, 2 or 3?\n\n- Section 5.1 Baselines: Some of the baselines are missing including [2] (text + audio input with a gesture as output), [4](audio input and gesture as output) and [5] (text input and human body motion as output with a seq2seq architecture).\n\n- Hellinger distance: The authors propose the use of Hellinger distance as an objective metric to measure the distance between the distributions of the generated gestures and ground truth gestures. While measuring the distribution distance is a good idea, it has been proposed for the same task in [2] and [3] with the use of Wasserstein Distance (i.e FID[2] or FGD[3]). There is a tradeoff between Hellinger distance and Wasserstein distance. They both have similar properties when the domains of the two distributions have some overlap. But as soon as the overlap between the distributions becomes zero, Hellinger distance fails to give any information on how far the two distributions are. Wasserstein distance on the other hand will still be able to give information on how far the two distributions are.\n\n- Table 1: What is the average jerk of the ground truth gestures so high ($1588.63 \\pm 2899.61$)? The interval covered by the GT jerk is [$-1310.98, 4400.24$] which encompasses the jerk vaues of all the models. Hence, this experiment can't definitively say much about the performance of baselines and the proposed model. \n\n- Ablations: There are many hyperparameters in this approach, including codebook size K, gesture sequence length (which was taken as 30 arbitrarily), DAE bottleneck. In section 3.2, the description of the model indicates that it also predicts the velocity of the output poses. Why is that necessary? It was also not clear how that affects the learnt representations.\n\n- The paper is fairly confusing, and hence unclear at times. The clarity of the paper can definitely be improved a lot with a significant re-write. For example:\n  - In Section 4, it is mentioned that the Euler angles were converted to joint positions. But in Section 4.1, it is stated that 3x3 rotational matrices were used instead of 3D coordinates. These two statements are inconsistent.\n  - In section 4.1, it is stated that the model is trained with differently sized latent embedding bottlenecks to find the optimal size. But the next statement mentions that a dimension size of 40 is chosen similar to Thangthai et. al. 2021 and Kucharenko et. al. 2019. These statements are contradictory.\n  - It wasn't very clear how that learn representations of the DAE were used as part of the second VQ-DVAE. I had to infer this from the 40+40=80 dimensional vector in Section 4.2. \n  - The terms DAE and VAE have been used interchangeably. It would be good to stick to one of them.\n\n\nReferences\n- [1] Richard, Alexander, et al. \"MeshTalk: 3D Face Animation from Speech using Cross-Modality Disentanglement.\" ICCV 2021\"\n- [2] Ahuja, Chaitanya, et al. \"No Gestures Left Behind: Learning Relationships between Spoken Language and Freeform Gestures.\" Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings. 2020.\n- [3] Yoon, Youngwoo, et al. \"Speech gesture generation from the trimodal context of text, audio, and speaker identity.\" ACM Transactions on Graphics (TOG) 39.6 (2020): 1-16. \n- [4] Ginosar, Shiry, et al. \"Learning individual styles of conversational gesture.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n- [5] Ahuja, Chaitanya, and Louis-Philippe Morency. \"Language2pose: Natural language grounded pose forecasting.\" 2019 International Conference on 3D Vision (3DV). IEEE, 2019.\n- [6] Razavi, Ali, Aaron van den Oord, and Oriol Vinyals. \"Generating diverse high-fidelity images with vq-vae-2.\" Advances in neural information processing systems. 2019.", "summary_of_the_paper": "In this paper, the authors propose an approach to perform co-speech gesture generation. A model Gesture2Vec is proposed to first learn the latent representation of gesture sequences. A vector quantized variational autoencoder is used to learn a codebook of different kinds of gesture sequences speaker's gesture manifold. Finally, the pre-trained decoder and codebook representations are conditioned on a sequence of language tokens to generate relevant co-speech gestures. This approach is tested on a public dataset containing a single speaker objective and subjective metrics. ", "summary_of_the_review": "While the motivation behind the learning representations of a gesture sequence instead of single frames is interesting, it is not executed very well. A lot of time has been spent explaining prior work instead of the contributions. The experiments to analyse the learnt representations is also lacking. Some of the details in the paper are confusing and hence needs significant revisions. And finally, being a generative modelling task, it would have been good to see some example videos. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Meta-review: PAPER: This paper describes a method to generate visual gestures by learning an intermediate representation based on gesture sequences. This proposed method builds from previous work on VAE and vector quantized VAE.
DISCUSSION: The reviewers wrote some detailed reviews about the paper, bringing some valid concerns and asking some questions to the authors. Unfortunately, no responses were posted from the authors.
SUMMARY: After looking at all reviews, there was a general consensus among the reviewers that this paper was not ready for publication. We hope that the reviews will be helpful for the authors in revising their work for future submission.
Decision: Reject