Gesture2Vec: Clustering Gestures using  Representation Learning Methods for Co-speech Gesture Generation

1 INTRODUCTION . Non-verbal behaviour is an indispensable part of our daily communication . Prior research stated that 70−93 % of communication is non-verbal , including facial expression , hand gesture , body pose , and vocal tones ( Mehrabian , 2017 ; Lapakko , 2007 ) . People spontaneously gesticulate to complement verbal channels during the speech to convey messages ( McNeill , 2011 ; De Ruiter et al. , 2012 ; Cassell et al. , 1999 ) . Hence , integrating non-verbal communication skills into social robots and virtual agents is crucial for compelling interactions ( Minato et al. , 2004 ; Woods et al. , 2004 ; Breazeal et al. , 2005 ) . Gestures originate from semantic features ( Chu & Kita , 2016 ) and are characterized by speech context ( Lücking et al. , 2013 ) . Therefore , co-speech gesture generation has been typically addressed using textual and acoustic features to generate relevant gestures . Gesture synthesis can be categorized into deterministic and probabilistic models . Deterministic models predict a single output for a given input , while probabilistic models estimate a plausible output probability distribution conditioned on the given input . The limitation of deterministic datadriven methods is that they neglect to span the variation of many dimensions in data space ( Fragkiadaki et al. , 2015 ; Ferstl et al. , 2019 ) . Probabilistic models are potentially able to capture a broader gesture space , but probabilistic models also suffer from the posterior-collapse problem ( Bowman et al. , 2015 ) which results in repetitive gestures close to the average . The problem arises when the one-to-one mapping assumption disregards the one-to-many relationship between speech and gesture . This is a common problem among generative models for gestures ( Yoon et al. , 2019 ; Ginosar et al. , 2019 ) , as well as in other domains , e.g . image generation models that produce a limited set of blurry and similar images ( Lucas et al. , 2019a ) , and early attempts for natural language genera- tion ( Bowman et al. , 2015 ; Kingma et al. , 2016 ; He et al. , 2019 ) . Literature attempted to address this posterior collapse problem using different techniques such as adversarial training ( Goodfellow et al. , 2014 ; Ginosar et al. , 2019 ; Arjovsky et al. , 2017 ; Srivastava et al. , 2017 ) , variational autoencoders ( Higgins et al. , 2016 ; Mi et al. , 2018 ; Ling et al. , 2020 ) , normalizing flow ( Alexanderson et al. , 2020 ) , vector quantization ( Oord et al. , 2017 ) ( Razavi et al. , 2019 ) , weakened decoders ( Yang et al. , 2017 ; Semeniuta et al. , 2017 ) . However , as reported , prior work ( Yoon et al. , 2019 ; Ferstl et al. , 2019 ; Kucherenko et al. , 2020b ) were not successful in capturing long-term dependencies . Indeed , the model parameters focus on imperceptible and local features such as continuity among consecutive frames , most likely due to mode collapse . Inspired by advances in other machine learning fields , especially natural language processing , we model gestures as a language that contains a vocabulary , and then perform text-to-gesture as a language translation task . Specifically , we integrate unsupervised representation learning methods and a machine translation algorithm . First , we reduce body pose dimensionality using a Denoising Autoencoder at the frame level . Then , we perform a discretized motion representation learning to cluster similar motion sequences to a symbol . This vector quantization method is a form of cluster pattern recognition where each motion sequence is assigned to a particular word from a codebook . Finally , we train a machine translation model to translate between utterances and their accompanying gestural motion symbols . This method is effective since it mitigates the complexity of gesture space and focuses on longer dependencies . In addition to traditional evaluation measurements , we also present new objective and subjective metrics to evaluate the diversity of gestures . 2 RELATED WORK . We first review co-speech gesture generation methods . Next , we discuss recent gesture generation approaches and their limitations . Finally , we introduce the deep neural network that we used and its advantages over prior work . Early attempts for gesture generation use blending to smooth feasible motion clips selected from a database . Rule-based algorithms ( Cassell et al. , 2004 ; Huang & Mutlu , 2012 ) , hidden Markov models ( Levine et al. , 2009 ) , conditional random fields ( Levine et al. , 2010 ) , and hybrid systems ( Kipp , 2005 ; Neff et al. , 2008 ) ( Chiu et al. , 2015 ) were used to select proper gestures conditioned on a given input . It is now recognized that these kind of systems require extensive efforts to annotate data and can not generate gestures for unseen inputs . Also , they provide a single prediction for a given input and lack variation of generated movement . Furthermore , scheduling the gestures with speech is challenging since they may not be precisely synchronized ( Butterworth & Hadar , 1989 ; Kendon , 2004 ) despite originating from the same source ( McNeill , 2011 ) . Recent deep generative models , e.g . VAEs ( Kingma & Welling , 2013 ; Higgins et al. , 2016 ) , GANs ( Goodfellow et al. , 2014 ; Abdal et al. , 2019 ) , and transformers ( Vaswani et al. , 2017 ; Brown et al. , 2020 ; Dehghani et al. , 2018 ) , achieved notable results on different tasks . A generative model comprises the joint probability of given data and output . Therefore , it can generate new plausible instances by taking samples from the learned distribution ( Hasegawa et al. , 2018 ) . Generative models have been used for human motion generation ( Yan et al. , 2018 ; Hernandez et al. , 2019 ; Ling et al. , 2020 ) as well as co-speech gesture generation ( Ginosar et al. , 2019 ; Yoon et al. , 2019 ; Ferstl et al. , 2021 ; Li et al. , 2021 ) . However , co-speech gesture generation is more challenging since the relationship between speech and motion is complex ( Butterworth & Hadar , 1989 ) . Input to the generative system can be supplied from either speech-text , audio , or both modalities with a broad range of semantic and acoustic features ( Kucherenko et al. , 2021c ) . Systems that use acoustic features led to generated beat gestures according to the speech rhythm ( Hasegawa et al. , 2018 ; Ferstl et al. , 2020 ; Ginosar et al. , 2019 ; Kucherenko , 2018 ; Pouw & Dixon , 2019 ) , and textbased systems ( Yoon et al. , 2019 ; Ishi et al. , 2018 ) produced more semantically aware gestures . Although text-based models capture communicative features , they lack the strong effect of speech acoustics , i.e . intonation , prosody , and loudness , on expressed gestures ( Pouw et al. , 2020 ) . Recent work benefiting from both modalities generated more compelling co-speech gestures in terms of appropriateness and naturalness ( Yoon et al. , 2020 ) . Typically , co-speech gesture generation systems build upon an encoder and decoder architecture ( Hasegawa et al. , 2018 ; Kucherenko et al. , 2019 ; Yoon et al. , 2019 ; Kucherenko et al. , 2020a ; Ferstl & McDonnell , 2018 ) . Recurrent Neural Networks ( RNN ) structures have been used extensively for the encoder and decoder ( Hasegawa et al. , 2018 ) . However , RNN based models suffer from the error accumulation problem and are not good at capturing long-term human motion dependencies ( Hernandez et al. , 2019 ) . While Convolutional Neural Network ( CNN ) based gesture generation models are not vulnerable to accumulation problems , CNNs are prone to regress to the average motion and generate repetitive gestures ( Li et al. , 2021 ) . Variational Autoencoders were also proposed for the co-speech gesture generation task to generate more realistic and diverse gestures , e.g . ( Rezende et al. , 2014 ; Li et al. , 2021 ; Kucherenko , 2018 ) . VAEs with strong decoders tend to ignore the latent variable and learn the mode of the output data . This problem , known as “ posterior collapse ” causes the model to generate slightly similar outputs close to the average ( Lucas et al. , 2019b ) . Literature considerably explored adversarial training to generate more realistic and diverse gestures ( Ginosar et al. , 2019 ; Ahuja et al. , 2020 ; Yoon et al. , 2020 ; Li et al. , 2021 ; Ferstl et al. , 2020 ; 2021 ) . Although they brought attractive results , GANs are hard to train ( Lucic et al. , 2017 ) and suffer from mode collapse ( Tulyakov et al. , 2018 ) . Normalizing-flow based models have GANs advantages while replacing adversarial loss by classical likelihood maximization training ( Kingma & Dhariwal , 2018 ) and efficient probabilistic inference of VAEs ( Henter et al. , 2020 ) ( Kingma & Dhariwal , 2018 ) . Furthermore , autoregressive models on discrete data achieved impressive results in many sequences-to-sequence tasks such as machine translation ( Wang et al. , 2019 ) , image generation ( Salimans et al. , 2017 ) ( Razavi et al. , 2019 ) , and speech synthesis ( Oord et al. , 2016 ) ( Gârbacea et al. , 2019 ) . VQ-VAE models learn a discrete latents space , enable us to use autoregressive models on the posterior , and does not suffer from “ posterior collapse ” ( Oord et al. , 2017 ) . Ordinarily , gesture generation systems use a generator network that produces gestures from a latent code . To better regress the data and cover a broad range of motion , ( Li et al. , 2021 ; Kucherenko et al. , 2021b ; a ) uses a motion-specific code space that represents motion attributes . All of the state-of-the-art models mentioned above generate gestures frame-by-frame . We suggest that this architecture drains model capabilities on local dependencies at the expense of global features and diversity . We therefore reformulate the problem as a machine translation task . Inspired by VQ-VAE , we propose a method that combines VQ-VAE , Denoising Autoencoders and weakened decoders to learn a discretized latent space . Finally , we use an autoregressive model on the quantized motions to produce co-speech gestures from the input . 3 METHOD . We considered Yoon et al . ( 2019 ) as our baseline since we also focus on word embedding textual features ( Bojanowski et al. , 2017 ) as the control signal for gesture generation . The authors in ( Yoon et al. , 2019 ) used an RNN based Encoder-Decoder architecture with a soft attention mechanism ( Bahdanau et al. , 2014 ; Cho et al. , 2014 ) . The encoder extracts textual features , and the decoder produces co-speech gestures frame-by-frame . We extend the baseline model by utilizing representation learning approaches . Our proposed system has the following steps . • Pose representation learning at the frame-level • Discrete motion representation learning at the sequence-level . • Translation from text to the learned discrete motion representation space . Learning powerful representations without supervision is of utmost importance to reduce problem complexity . Autoencoders are unsupervised models that learn significant data features and discard spurious patterns by minimizing the reconstruction error ( Kingma & Welling , 2013 ) . They consist of an encoder network followed by a decoder network . An autoencoder ’ s bottleneck finds a shared data representation ( Goodfellow et al. , 2009 ; Bengio et al. , 2007 ) by learning the correlations between input dimensions and reconstructs them from a low-dimensionality representation . We used two different autoencoders with different components , both at the frame-level and sequence-level , as explained in the following sections .

Summary: This paper tackles with co-speech gesture generation. First a denoising autoencoder is applied to the body pose detected at each frame. Then, a Vector Quantized Denoising Variational Autoencoder learns to create a motion vocabulary by parameterizing the posterior distribution of discrete latents. As the third step,  a two-layered bidirectional gated recurrent neural network with a soft attention mechanism is used for translation from text to gesture vocabulary. The decoder of Vector Quantized Denoising Variational Autoencoder is used to reproduce the entire gesture sequence.