Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning

1 INTRODUCTION . Arguably , dancing to music is one of human ’ s innate abilities , as we can spontaneously sway along with the tempo of music we hear . The research in neuropsychology indicates that our brain is hardwired to make us move and synchronize with music regardless of our intention ( Chen et al. , 2008 ) . Another study of archaeology also suggests that dance is a social communication skill among early humans connected to the ability of survival long time ago ( Adshead-Lansdale & Layson , 2006 ) . Nowadays , dance ( to music ) has become a means to the cultural promotion , a method of emotional expression , a tool for socialization and an art form to bring aesthetic enjoyment . The neurological mechanism behind dancing behavior and the unique value of dance to society motivate us to explore a computational approach to dance creation from a piece of music in artificial intelligence research . Such work is potentially beneficial to a wide range of applications such as dance creation assistant in art and sports , character motion generation for audio games and research on cross-modal behavior . In literature , the music-conditioned dance generation is a relatively new task that attracts increasing research interests recently . Early works ( Fan et al. , 2011 ; Lee et al. , 2013 ) synthesize dance sequences from music by retrieval-based methods , which show the limited creativity in practice . Recently , Lee et al . ( 2019 ) formulate the task from the generative perspective , and further propose a decompositionto-composition framework . Their model first generates basic dance units from the music clips and then composes them by using the last pose of current unit to initialize the first pose of the next ∗Equal contribution . †Work done during the internship at Microsoft STCA . ‡Corresponding author . unit . Although this approach shows better performance than the retrieval-based methods , several challenges still remain . First , existing generative methods synthesize new human motion sequences through autoregressive models like RNN , which tend to result in short sequences . In other words , the generated sequences often quickly freeze within a few seconds due to an accumulation of prediction errors that are fed back into the neural network . This problem becomes even more severe in long motion sequence generation . For instance , composing a dance for a 1-minute music clip under 15 Frame Per Second ( FPS ) means generating 900 poses at one time . In practice , we need novel methods which can effectively generate long motion sequences . Besides , how to enhance the harmony between the synthesized dance movements and the given music is a largely unexplored challenge . Inutitively , the movements need to be consistent with the music in terms of style , rhythm and beat . However , to achieve this goal is non-trivial , which requires the generation model to have the capability to capture the fine-grained correspondence between music and dance . In this paper , we formalize music-conditioned dance generation as a sequence-to-sequence learning problem where the fine-grained correspondence between music and dance is represented through sequence modeling and their alignment is established via mapping from a sequence of acoustic features of the music to a sequence of movements of the dance . The model consists of a music encoder and a dance decoder . The encoder transforms low-level acoustic features of an input music clip into high-level latent representations via self-attention with a receptive field restricted within k-nearest neighbors of an element . Thus , the encoder can efficiently process long sequences of music features , e.g. , a sequence with more than 1000 elements , and model local characteristics of the music such as chord and rhythm patterns . The decoder exploits a recurrent structure to predict the dance movement frame by frame conditioned on the corresponding element in the latent representations of music feature sequence . Furthermore , we propose a curriculum learning ( Bengio et al. , 2009 ) strategy to alleviate error accumulation ( Li et al. , 2017 ) of autoregressive models in long motion sequence generation . Specifically , it gently changes the training process from a fully guided teacher-forcing scheme using the previous ground-truth movements , towards a less guided autoregressive scheme which mostly utilizes the generated movements instead . This strategy bridges the gap between training and inference of autoregressive models , and thus alleviates error accumulation at inference . The length of each video clip in the dataset released by Lee et al . ( 2019 ) is only about 6 seconds , which can not be used by other methds to generate long-term dance except for their specially designed decomposition-to-composition framework . To facilitate the task of long-term dance generation with music , we collect a high-quality dataset consisting of 1-minute video clips , totaling about 12 hours . And there are three representative styles in our dataset : “ Ballet ” , “ Hiphop ” and “ Japanese Pop ” . Our contributions in this work are four-fold : ( 1 ) We formalize music-conditioned dance generation as a sequence-to-sequence learning problem and devise a novel seq2seq architecture for the long-term dance generation with music . ( 2 ) We propose a novel curriculum learning strategy to alleviate error accumulation of autoregressive models in long motion sequence generation . ( 3 ) To facilitate long-term dance generation with music , we collect a high-quality dataset that is available with our code1 . ( 4 ) The extensive experiments show that our approach significantly outperforms the existing state-of-the-arts on both automatic metrics and human judgements . The demo video in the supplementary material also exhibits our approach can generate diverse minute-length dances that are smooth , natural-looking , style-consistent and beat-matching with the musics from test set . 2 RELATED WORK . Cross-Modal Learning . Most existing works focus on the modeling between vision and text , such as image captioning ( Lu et al. , 2017 ; Xu et al. , 2015 ) and text-to-image generation ( Reed et al. , 2016 ; Zhang et al. , 2017 ) . There are some other works to study the translation between audio and text like Automatic Speech Recognition ( ASR ) ( Hinton et al. , 2012 ) and Text-To-Speech ( TTS ) ( Oord et al. , 2016 ) . While the modeling between audio and vision is largely unexplored and the music-conditioned dance generation is a typical cross-modal learning problem from audio to vision . 1https : //github.com/stonyhu/DanceRevolution Human Motion Prediction . Prediction of human motion dynamics has been a challenging problem in computer vision , which suffers from the high spatial-temporal complexity . Existing works ( Chan et al. , 2019 ; Wang et al. , 2018 ) represent the human pose as 2D or 3D body keyjoints ( Cao et al. , 2017 ) and address the problem via sequence modeling . Early methods , such as hidden markov models ( Lehrmann et al. , 2014 ) , Gaussian processes ( Wang et al. , 2006 ) and restricted boltzmann machines ( Taylor et al. , 2007 ) , have to balance the model capacity and inference complexity due to complicated training procedures . Recently , neural networks dominate the human motion modeling . For instance , Fragkiadaki et al . ( 2015 ) present LSTM-3LR and Encoder-Recurrent-Decoder ( ERD ) as two recurrent architectures for the task ; Jain et al . ( 2016 ) propose a structural-RNN to model human-object interactions in a spatio-temporal graph ; and Ghosh et al . ( 2017 ) equip LSTM-3LR with a dropout autoencoder to enhance the long-term prediction . Besides , convolutional neural networks ( CNNs ) have also been utilized to model the human motion prediction ( Li et al. , 2018 ) . Audio-Conditioned Dance Generation . In the research of audio-conditioned dance generation , most existing works study 2D dance motion generation with music since the training data for paired 2D pose and music can be extracted from the huge amount of dance videos available online . Various methods have been proposed to handle this task , such as adversarial learning based methods ( Lee et al. , 2019 ; Sun et al. , 2020 ; Ferreira et al. , 2021 ) , autoencoder methods ( Tang et al. , 2018 ) and sequence-to-sequence methods ( Lee et al. , 2018 ; Ren et al. , 2019 ; Yalta et al. , 2019 ; Ye et al. , 2020 ) . While these works mainly focus on exploring the different neural architectures and overlook the freezing motion issue in dance motion synthesis . In this work , we first propose a novel seq2seq architecture to model the fine-grained correspondence between music and dance , and then introduce a novel curriculum learning strategy to address the freezing motion issue caused by error accumulation ( Li et al. , 2017 ) in long-term dance motion generation . 3 APPROACH . In this section , we present our approach to music-conditioned dance generation . After formalization of the problem in question , we elaborate the model architecture and the dynamic auto-condition learning approach that facilitates long-term dance generation according to the given music . 3.1 PROBLEM FORMALIZATION . Suppose that there is a dataset D = { ( Xi , Yi ) } Ni=1 , where X = { xt } nt=1 is a music clip with xt being a vector of acoustic features at time-step t , and Y = { yt } nt=1 is a sequence of dance movements with yt aligned to xt . The goal is to estimate a generation model g ( · ) from D , and thus given a new music input X , the model can synthesize dance Y to music X based on g ( X ) . We first present our seq2seq architectures chosen for music-conditioned dance generation in the following section . Later in the experiments , we empirically justify the choice by comparing the architectures with other alternatives . 3.2 MODEL ARCHITECTURE . In the architecture of g ( · ) , a music encoder first transforms X = ( x1 , ... , xn ) ( xi ∈ Rdx ) into a hidden sequence Z = ( z1 , ... , zn ) ( zi ∈ Rdz ) using a local self-attention mechanism to reduce the memory requirement for long sequence modeling , and then a dance decoder exploits a recurrent structure to autoregressively predicts movements Y = ( y1 , ... , yn ) conditioned on Z . Music Encoder . Encouraged by the compelling performance on music generation ( Huang et al. , 2018 ) , we define the music encoder with a transformer encoder structure . While the self-attention mechanism ( Vaswani et al. , 2017 ) in transformer can effectively represent the multi-scale structure of music , the quadratic memory complexity O ( n2 ) about sequence length n impedes its application to long sequence modeling due to huge of GPU memory consumption ( Child et al. , 2019 ) . To keep the effectiveness of representation and control the cost , we introduce a local self-attention mechanism that modifies the receptive field of self-attention by restricting the element connections within k-nearest neighbors . Thus , the memory complexity is reduced toO ( nk ) . k could be small in our scenario since we only pursue an effective representation for a given music clip . Therefore , the local patterns of music are encoded in zt that is sufficient for the generation of movement yt at time-step t , which is aligned with the common sense that the dance movement at certain time-step is highly influenced by the nearby clips of music . Yet we can handle long sequences of acoustic features , e.g. , more than 1000 elements , in an efficient and memory-economic way . Specifically , we first embed X = ( x1 , ... , xn ) into U = ( u1 , ... , un ) with a linear layer parameterized by WE ∈ Rdx×dz . Then ∀i ∈ { 1 , ... , n } , zi can be formulated as : zi = F ( ai ) , ai = i+bk/2c∑ j=i−bk/2c αij ( ujW V l ) , U = XW E , ( 1 ) where F ( · ) : Rdv → Rdz is a feed forward neural network . Each uj is only allowed to attend its k-nearest neighbors including itself , where k is a hyper-parameter referring to the sliding window size of the local self-attention . Then attention weight αij is calculated using a softmax function as : αij = exp eij∑j+bk/2c t=j−bk/2c exp eit , eij = ( uiW Q l ) ( ujW K l ) > √ dk , ( 2 ) where for the the l-th head , WQl , W K l ∈ Rdz×dk and WVl ∈ Rdz×dv are parameters that transform U into a query , a key , and a value respectively . dz is the dimension of hidden state zi while dk is the dimension of query , key and dv is the dimension of value . Dance Decoder . We choose a recurrent neural network as the dance decoder in consideration of two factors : ( 1 ) the chain structure can well capture the spatial-temporal dependency among human movement dynamics , which has proven to be highly effective in the state-of-the-art methods for human motion prediction ( Li et al. , 2018 ; Mao et al. , 2019 ) ; ( 2 ) our proposed learning strategy is tailored for the autoregressive models like RNN , as will be described in the next section . Specifically , with Z = ( z1 , ... , zn ) , the dance movements Y = ( ŷ1 , ... , ŷn ) are synthesized by : ŷi = [ hi ; zi ] W S + b , ( 3 ) hi = RNN ( hi−1 , ŷi−1 ) , ( 4 ) where hi is the i-th hidden state of the decoder and h0 is initialized by sampling from the standard normal distribution to enhance variation of the generated sequences . [ · ; · ] denotes the concatenation operation . WS ∈ R ( ds+dz ) ×dy and b ∈ Rdy are parameters where ds and dy are the dimensions of hi and ŷi , respectively . At the i-th time-step , the decoder predicts the movement ŷi conditioned on hi as well as the latent feature representation zi , and thus can capture the fine-grained correspondence between music feature sequence and dance movement sequence .

Summary: The authors present a seq2seq model with a sparse transformer encoder and an LSTM decoder. They utilize a learning curriculum wherein the autoregressive decoder is initially trained using teacher forcing and is gradually fed its past predictions as training progresses. The authors introduce a new dataset for long term dance generation. They utilize both subjective and objective metrics to evaluate their method. The proposed method outperforms other baselines for dance generation. Finally they conduct ablation studies demonstrating the benefits of using a transformer encoder over other architectures, and the benefits of the proposed curriculum learning scheme.