Gesture2Vec: Clustering Gestures using  Representation Learning Methods for Co-speech Gesture Generation

1 INTRODUCTION . Non-verbal behaviour is an indispensable part of our daily communication . Prior research stated that 70−93 % of communication is non-verbal , including facial expression , hand gesture , body pose , and vocal tones ( Mehrabian , 2017 ; Lapakko , 2007 ) . People spontaneously gesticulate to complement verbal channels during the speech to convey messages ( McNeill , 2011 ; De Ruiter et al. , 2012 ; Cassell et al. , 1999 ) . Hence , integrating non-verbal communication skills into social robots and virtual agents is crucial for compelling interactions ( Minato et al. , 2004 ; Woods et al. , 2004 ; Breazeal et al. , 2005 ) . Gestures originate from semantic features ( Chu & Kita , 2016 ) and are characterized by speech context ( Lücking et al. , 2013 ) . Therefore , co-speech gesture generation has been typically addressed using textual and acoustic features to generate relevant gestures . Gesture synthesis can be categorized into deterministic and probabilistic models . Deterministic models predict a single output for a given input , while probabilistic models estimate a plausible output probability distribution conditioned on the given input . The limitation of deterministic datadriven methods is that they neglect to span the variation of many dimensions in data space ( Fragkiadaki et al. , 2015 ; Ferstl et al. , 2019 ) . Probabilistic models are potentially able to capture a broader gesture space , but probabilistic models also suffer from the posterior-collapse problem ( Bowman et al. , 2015 ) which results in repetitive gestures close to the average . The problem arises when the one-to-one mapping assumption disregards the one-to-many relationship between speech and gesture . This is a common problem among generative models for gestures ( Yoon et al. , 2019 ; Ginosar et al. , 2019 ) , as well as in other domains , e.g . image generation models that produce a limited set of blurry and similar images ( Lucas et al. , 2019a ) , and early attempts for natural language genera- tion ( Bowman et al. , 2015 ; Kingma et al. , 2016 ; He et al. , 2019 ) . Literature attempted to address this posterior collapse problem using different techniques such as adversarial training ( Goodfellow et al. , 2014 ; Ginosar et al. , 2019 ; Arjovsky et al. , 2017 ; Srivastava et al. , 2017 ) , variational autoencoders ( Higgins et al. , 2016 ; Mi et al. , 2018 ; Ling et al. , 2020 ) , normalizing flow ( Alexanderson et al. , 2020 ) , vector quantization ( Oord et al. , 2017 ) ( Razavi et al. , 2019 ) , weakened decoders ( Yang et al. , 2017 ; Semeniuta et al. , 2017 ) . However , as reported , prior work ( Yoon et al. , 2019 ; Ferstl et al. , 2019 ; Kucherenko et al. , 2020b ) were not successful in capturing long-term dependencies . Indeed , the model parameters focus on imperceptible and local features such as continuity among consecutive frames , most likely due to mode collapse . Inspired by advances in other machine learning fields , especially natural language processing , we model gestures as a language that contains a vocabulary , and then perform text-to-gesture as a language translation task . Specifically , we integrate unsupervised representation learning methods and a machine translation algorithm . First , we reduce body pose dimensionality using a Denoising Autoencoder at the frame level . Then , we perform a discretized motion representation learning to cluster similar motion sequences to a symbol . This vector quantization method is a form of cluster pattern recognition where each motion sequence is assigned to a particular word from a codebook . Finally , we train a machine translation model to translate between utterances and their accompanying gestural motion symbols . This method is effective since it mitigates the complexity of gesture space and focuses on longer dependencies . In addition to traditional evaluation measurements , we also present new objective and subjective metrics to evaluate the diversity of gestures . 2 RELATED WORK . We first review co-speech gesture generation methods . Next , we discuss recent gesture generation approaches and their limitations . Finally , we introduce the deep neural network that we used and its advantages over prior work . Early attempts for gesture generation use blending to smooth feasible motion clips selected from a database . Rule-based algorithms ( Cassell et al. , 2004 ; Huang & Mutlu , 2012 ) , hidden Markov models ( Levine et al. , 2009 ) , conditional random fields ( Levine et al. , 2010 ) , and hybrid systems ( Kipp , 2005 ; Neff et al. , 2008 ) ( Chiu et al. , 2015 ) were used to select proper gestures conditioned on a given input . It is now recognized that these kind of systems require extensive efforts to annotate data and can not generate gestures for unseen inputs . Also , they provide a single prediction for a given input and lack variation of generated movement . Furthermore , scheduling the gestures with speech is challenging since they may not be precisely synchronized ( Butterworth & Hadar , 1989 ; Kendon , 2004 ) despite originating from the same source ( McNeill , 2011 ) . Recent deep generative models , e.g . VAEs ( Kingma & Welling , 2013 ; Higgins et al. , 2016 ) , GANs ( Goodfellow et al. , 2014 ; Abdal et al. , 2019 ) , and transformers ( Vaswani et al. , 2017 ; Brown et al. , 2020 ; Dehghani et al. , 2018 ) , achieved notable results on different tasks . A generative model comprises the joint probability of given data and output . Therefore , it can generate new plausible instances by taking samples from the learned distribution ( Hasegawa et al. , 2018 ) . Generative models have been used for human motion generation ( Yan et al. , 2018 ; Hernandez et al. , 2019 ; Ling et al. , 2020 ) as well as co-speech gesture generation ( Ginosar et al. , 2019 ; Yoon et al. , 2019 ; Ferstl et al. , 2021 ; Li et al. , 2021 ) . However , co-speech gesture generation is more challenging since the relationship between speech and motion is complex ( Butterworth & Hadar , 1989 ) . Input to the generative system can be supplied from either speech-text , audio , or both modalities with a broad range of semantic and acoustic features ( Kucherenko et al. , 2021c ) . Systems that use acoustic features led to generated beat gestures according to the speech rhythm ( Hasegawa et al. , 2018 ; Ferstl et al. , 2020 ; Ginosar et al. , 2019 ; Kucherenko , 2018 ; Pouw & Dixon , 2019 ) , and textbased systems ( Yoon et al. , 2019 ; Ishi et al. , 2018 ) produced more semantically aware gestures . Although text-based models capture communicative features , they lack the strong effect of speech acoustics , i.e . intonation , prosody , and loudness , on expressed gestures ( Pouw et al. , 2020 ) . Recent work benefiting from both modalities generated more compelling co-speech gestures in terms of appropriateness and naturalness ( Yoon et al. , 2020 ) . Typically , co-speech gesture generation systems build upon an encoder and decoder architecture ( Hasegawa et al. , 2018 ; Kucherenko et al. , 2019 ; Yoon et al. , 2019 ; Kucherenko et al. , 2020a ; Ferstl & McDonnell , 2018 ) . Recurrent Neural Networks ( RNN ) structures have been used extensively for the encoder and decoder ( Hasegawa et al. , 2018 ) . However , RNN based models suffer from the error accumulation problem and are not good at capturing long-term human motion dependencies ( Hernandez et al. , 2019 ) . While Convolutional Neural Network ( CNN ) based gesture generation models are not vulnerable to accumulation problems , CNNs are prone to regress to the average motion and generate repetitive gestures ( Li et al. , 2021 ) . Variational Autoencoders were also proposed for the co-speech gesture generation task to generate more realistic and diverse gestures , e.g . ( Rezende et al. , 2014 ; Li et al. , 2021 ; Kucherenko , 2018 ) . VAEs with strong decoders tend to ignore the latent variable and learn the mode of the output data . This problem , known as “ posterior collapse ” causes the model to generate slightly similar outputs close to the average ( Lucas et al. , 2019b ) . Literature considerably explored adversarial training to generate more realistic and diverse gestures ( Ginosar et al. , 2019 ; Ahuja et al. , 2020 ; Yoon et al. , 2020 ; Li et al. , 2021 ; Ferstl et al. , 2020 ; 2021 ) . Although they brought attractive results , GANs are hard to train ( Lucic et al. , 2017 ) and suffer from mode collapse ( Tulyakov et al. , 2018 ) . Normalizing-flow based models have GANs advantages while replacing adversarial loss by classical likelihood maximization training ( Kingma & Dhariwal , 2018 ) and efficient probabilistic inference of VAEs ( Henter et al. , 2020 ) ( Kingma & Dhariwal , 2018 ) . Furthermore , autoregressive models on discrete data achieved impressive results in many sequences-to-sequence tasks such as machine translation ( Wang et al. , 2019 ) , image generation ( Salimans et al. , 2017 ) ( Razavi et al. , 2019 ) , and speech synthesis ( Oord et al. , 2016 ) ( Gârbacea et al. , 2019 ) . VQ-VAE models learn a discrete latents space , enable us to use autoregressive models on the posterior , and does not suffer from “ posterior collapse ” ( Oord et al. , 2017 ) . Ordinarily , gesture generation systems use a generator network that produces gestures from a latent code . To better regress the data and cover a broad range of motion , ( Li et al. , 2021 ; Kucherenko et al. , 2021b ; a ) uses a motion-specific code space that represents motion attributes . All of the state-of-the-art models mentioned above generate gestures frame-by-frame . We suggest that this architecture drains model capabilities on local dependencies at the expense of global features and diversity . We therefore reformulate the problem as a machine translation task . Inspired by VQ-VAE , we propose a method that combines VQ-VAE , Denoising Autoencoders and weakened decoders to learn a discretized latent space . Finally , we use an autoregressive model on the quantized motions to produce co-speech gestures from the input . 3 METHOD . We considered Yoon et al . ( 2019 ) as our baseline since we also focus on word embedding textual features ( Bojanowski et al. , 2017 ) as the control signal for gesture generation . The authors in ( Yoon et al. , 2019 ) used an RNN based Encoder-Decoder architecture with a soft attention mechanism ( Bahdanau et al. , 2014 ; Cho et al. , 2014 ) . The encoder extracts textual features , and the decoder produces co-speech gestures frame-by-frame . We extend the baseline model by utilizing representation learning approaches . Our proposed system has the following steps . • Pose representation learning at the frame-level • Discrete motion representation learning at the sequence-level . • Translation from text to the learned discrete motion representation space . Learning powerful representations without supervision is of utmost importance to reduce problem complexity . Autoencoders are unsupervised models that learn significant data features and discard spurious patterns by minimizing the reconstruction error ( Kingma & Welling , 2013 ) . They consist of an encoder network followed by a decoder network . An autoencoder ’ s bottleneck finds a shared data representation ( Goodfellow et al. , 2009 ; Bengio et al. , 2007 ) by learning the correlations between input dimensions and reconstructs them from a low-dimensionality representation . We used two different autoencoders with different components , both at the frame-level and sequence-level , as explained in the following sections .

Summary: The authors present a method to generate co-speech gestures by learning a vector-quantized representation space of gestures coupled with a machine translation from natural language sentences to that representation space. They design a denoising variational autoencoder (DVAE) to learn the encoding space corresponding to gesture sequences, and then quantize that encoding space using nearest neighbor embeddings. Once the vector quantized representations are learned, they use a sequence-to-sequence network to translate the text transcript of the input speech to those representations, which they subsequently decode into the output co-speech gestures. They train and test their method on the Trinity Gesture Dataset and show the improvements achieved by their method on both objective and subjective evaluations.