DANCE REVOLUTION: LONG-TERM DANCE GENERA-
Year: 2021

Abstract:
Dancing to music is one of human’s innate abilities since ancient times. In machine learning research, however, synthesizing dance movements from music is a challenging problem. Recently, researchers synthesize human motion sequences through autoregressive models like recurrent neural network (RNN). Such an approach often generates short sequences due to an accumulation of prediction errors that are fed back into the neural network. This problem becomes even more severe in the long motion sequence generation. Besides, the consistency between dance and music in terms of style, rhythm and beat is yet to be taken into account during modeling. In this paper, we formalize the music-conditioned dance generation as a sequence-to-sequence learning problem and devise a novel seq2seq architecture to efficiently process long sequences of music features and capture the fine-grained correspondence between music and dance. Furthermore, we propose a novel curriculum learning strategy to alleviate error accumulation of autoregressive models in long motion sequence generation, which gently changes the training process from a fully guided teacher-forcing scheme using the previous ground-truth movements, towards a less guided autoregressive scheme mostly using the generated movements instead. Extensive experiments show that our approach significantly outperforms the existing state-of-the-arts on automatic metrics and human evaluation. We also make a demo video to demonstrate the superior performance of our proposed approach at https://www.youtube.com/watch?v=lmE20MEheZ8.

heading: 1 INTRODUCTION
text: Arguably, dancing to music is one of human’s innate abilities, as we can spontaneously sway along with the tempo of music we hear. The research in neuropsychology indicates that our brain is hardwired to make us move and synchronize with music regardless of our intention (Chen et al., 2008). Another study of archaeology also suggests that dance is a social communication skill among early humans connected to the ability of survival long time ago (Adshead-Lansdale & Layson, 2006). Nowadays, dance (to music) has become a means to the cultural promotion, a method of emotional expression, a tool for socialization and an art form to bring aesthetic enjoyment. The neurological mechanism behind dancing behavior and the unique value of dance to society motivate us to explore a computational approach to dance creation from a piece of music in artificial intelligence research. Such work is potentially beneficial to a wide range of applications such as dance creation assistant in art and sports, character motion generation for audio games and research on cross-modal behavior.
In literature, the music-conditioned dance generation is a relatively new task that attracts increasing research interests recently. Early works (Fan et al., 2011; Lee et al., 2013) synthesize dance sequences from music by retrieval-based methods, which show the limited creativity in practice. Recently, Lee et al. (2019) formulate the task from the generative perspective, and further propose a decompositionto-composition framework. Their model first generates basic dance units from the music clips and then composes them by using the last pose of current unit to initialize the first pose of the next ∗Equal contribution. †Work done during the internship at Microsoft STCA. ‡Corresponding author.
unit. Although this approach shows better performance than the retrieval-based methods, several challenges still remain.
First, existing generative methods synthesize new human motion sequences through autoregressive models like RNN, which tend to result in short sequences. In other words, the generated sequences often quickly freeze within a few seconds due to an accumulation of prediction errors that are fed back into the neural network. This problem becomes even more severe in long motion sequence generation. For instance, composing a dance for a 1-minute music clip under 15 Frame Per Second (FPS) means generating 900 poses at one time. In practice, we need novel methods which can effectively generate long motion sequences. Besides, how to enhance the harmony between the synthesized dance movements and the given music is a largely unexplored challenge. Inutitively, the movements need to be consistent with the music in terms of style, rhythm and beat. However, to achieve this goal is non-trivial, which requires the generation model to have the capability to capture the fine-grained correspondence between music and dance.
In this paper, we formalize music-conditioned dance generation as a sequence-to-sequence learning problem where the fine-grained correspondence between music and dance is represented through sequence modeling and their alignment is established via mapping from a sequence of acoustic features of the music to a sequence of movements of the dance. The model consists of a music encoder and a dance decoder. The encoder transforms low-level acoustic features of an input music clip into high-level latent representations via self-attention with a receptive field restricted within k-nearest neighbors of an element. Thus, the encoder can efficiently process long sequences of music features, e.g., a sequence with more than 1000 elements, and model local characteristics of the music such as chord and rhythm patterns. The decoder exploits a recurrent structure to predict the dance movement frame by frame conditioned on the corresponding element in the latent representations of music feature sequence. Furthermore, we propose a curriculum learning (Bengio et al., 2009) strategy to alleviate error accumulation (Li et al., 2017) of autoregressive models in long motion sequence generation. Specifically, it gently changes the training process from a fully guided teacher-forcing scheme using the previous ground-truth movements, towards a less guided autoregressive scheme which mostly utilizes the generated movements instead. This strategy bridges the gap between training and inference of autoregressive models, and thus alleviates error accumulation at inference.
The length of each video clip in the dataset released by Lee et al. (2019) is only about 6 seconds, which cannot be used by other methds to generate long-term dance except for their specially designed decomposition-to-composition framework. To facilitate the task of long-term dance generation with music, we collect a high-quality dataset consisting of 1-minute video clips, totaling about 12 hours. And there are three representative styles in our dataset: “Ballet”, “Hiphop” and “Japanese Pop”.
Our contributions in this work are four-fold: (1) We formalize music-conditioned dance generation as a sequence-to-sequence learning problem and devise a novel seq2seq architecture for the long-term dance generation with music. (2) We propose a novel curriculum learning strategy to alleviate error accumulation of autoregressive models in long motion sequence generation. (3) To facilitate long-term dance generation with music, we collect a high-quality dataset that is available with our code1. (4) The extensive experiments show that our approach significantly outperforms the existing state-of-the-arts on both automatic metrics and human judgements. The demo video in the supplementary material also exhibits our approach can generate diverse minute-length dances that are smooth, natural-looking, style-consistent and beat-matching with the musics from test set.

heading: 2 RELATED WORK
text: Cross-Modal Learning. Most existing works focus on the modeling between vision and text, such as image captioning (Lu et al., 2017; Xu et al., 2015) and text-to-image generation (Reed et al., 2016; Zhang et al., 2017). There are some other works to study the translation between audio and text like Automatic Speech Recognition (ASR) (Hinton et al., 2012) and Text-To-Speech (TTS) (Oord et al., 2016). While the modeling between audio and vision is largely unexplored and the music-conditioned dance generation is a typical cross-modal learning problem from audio to vision.
1https://github.com/stonyhu/DanceRevolution
Human Motion Prediction. Prediction of human motion dynamics has been a challenging problem in computer vision, which suffers from the high spatial-temporal complexity. Existing works (Chan et al., 2019; Wang et al., 2018) represent the human pose as 2D or 3D body keyjoints (Cao et al., 2017) and address the problem via sequence modeling. Early methods, such as hidden markov models (Lehrmann et al., 2014), Gaussian processes (Wang et al., 2006) and restricted boltzmann machines (Taylor et al., 2007), have to balance the model capacity and inference complexity due to complicated training procedures. Recently, neural networks dominate the human motion modeling. For instance, Fragkiadaki et al. (2015) present LSTM-3LR and Encoder-Recurrent-Decoder (ERD) as two recurrent architectures for the task; Jain et al. (2016) propose a structural-RNN to model human-object interactions in a spatio-temporal graph; and Ghosh et al. (2017) equip LSTM-3LR with a dropout autoencoder to enhance the long-term prediction. Besides, convolutional neural networks (CNNs) have also been utilized to model the human motion prediction (Li et al., 2018).
Audio-Conditioned Dance Generation. In the research of audio-conditioned dance generation, most existing works study 2D dance motion generation with music since the training data for paired 2D pose and music can be extracted from the huge amount of dance videos available online. Various methods have been proposed to handle this task, such as adversarial learning based methods (Lee et al., 2019; Sun et al., 2020; Ferreira et al., 2021), autoencoder methods (Tang et al., 2018) and sequence-to-sequence methods (Lee et al., 2018; Ren et al., 2019; Yalta et al., 2019; Ye et al., 2020). While these works mainly focus on exploring the different neural architectures and overlook the freezing motion issue in dance motion synthesis. In this work, we first propose a novel seq2seq architecture to model the fine-grained correspondence between music and dance, and then introduce a novel curriculum learning strategy to address the freezing motion issue caused by error accumulation (Li et al., 2017) in long-term dance motion generation.

heading: 3 APPROACH
text: In this section, we present our approach to music-conditioned dance generation. After formalization of the problem in question, we elaborate the model architecture and the dynamic auto-condition learning approach that facilitates long-term dance generation according to the given music.

heading: 3.1 PROBLEM FORMALIZATION
text: Suppose that there is a dataset D = {(Xi, Yi)}Ni=1, where X = {xt}nt=1 is a music clip with xt being a vector of acoustic features at time-step t, and Y = {yt}nt=1 is a sequence of dance movements with
yt aligned to xt. The goal is to estimate a generation model g(·) from D, and thus given a new music input X , the model can synthesize dance Y to music X based on g(X). We first present our seq2seq architectures chosen for music-conditioned dance generation in the following section. Later in the experiments, we empirically justify the choice by comparing the architectures with other alternatives.

heading: 3.2 MODEL ARCHITECTURE
text: In the architecture of g(·), a music encoder first transforms X = (x1, ..., xn) (xi ∈ Rdx) into a hidden sequence Z = (z1, ..., zn) (zi ∈ Rdz ) using a local self-attention mechanism to reduce the memory requirement for long sequence modeling, and then a dance decoder exploits a recurrent structure to autoregressively predicts movements Y = (y1, ..., yn) conditioned on Z.
Music Encoder. Encouraged by the compelling performance on music generation (Huang et al., 2018), we define the music encoder with a transformer encoder structure. While the self-attention mechanism (Vaswani et al., 2017) in transformer can effectively represent the multi-scale structure of music, the quadratic memory complexity O(n2) about sequence length n impedes its application to long sequence modeling due to huge of GPU memory consumption (Child et al., 2019). To keep the effectiveness of representation and control the cost, we introduce a local self-attention mechanism that modifies the receptive field of self-attention by restricting the element connections within k-nearest neighbors. Thus, the memory complexity is reduced toO(nk). k could be small in our scenario since we only pursue an effective representation for a given music clip. Therefore, the local patterns of music are encoded in zt that is sufficient for the generation of movement yt at time-step t, which is aligned with the common sense that the dance movement at certain time-step is highly influenced by the nearby clips of music. Yet we can handle long sequences of acoustic features, e.g., more than 1000 elements, in an efficient and memory-economic way.
Specifically, we first embed X = (x1, ..., xn) into U = (u1, ..., un) with a linear layer parameterized by WE ∈ Rdx×dz . Then ∀i ∈ {1, ..., n}, zi can be formulated as:
zi = F(ai), ai = i+bk/2c∑
j=i−bk/2c
αij(ujW V l ), U = XW E , (1)
where F(·) : Rdv → Rdz is a feed forward neural network. Each uj is only allowed to attend its k-nearest neighbors including itself, where k is a hyper-parameter referring to the sliding window size of the local self-attention. Then attention weight αij is calculated using a softmax function as:
αij = exp eij∑j+bk/2c
t=j−bk/2c exp eit , eij =
(uiW Q l )(ujW K l ) >
√ dk
, (2)
where for the the l-th head, WQl , W K l ∈ Rdz×dk and WVl ∈ Rdz×dv are parameters that transform U into a query, a key, and a value respectively. dz is the dimension of hidden state zi while dk is the dimension of query, key and dv is the dimension of value.
Dance Decoder. We choose a recurrent neural network as the dance decoder in consideration of two factors: (1) the chain structure can well capture the spatial-temporal dependency among human movement dynamics, which has proven to be highly effective in the state-of-the-art methods for human motion prediction (Li et al., 2018; Mao et al., 2019); (2) our proposed learning strategy is tailored for the autoregressive models like RNN, as will be described in the next section. Specifically, with Z = (z1, ..., zn), the dance movements Y = (ŷ1, ..., ŷn) are synthesized by:
ŷi = [hi; zi]W S + b, (3)
hi = RNN(hi−1, ŷi−1), (4)
where hi is the i-th hidden state of the decoder and h0 is initialized by sampling from the standard normal distribution to enhance variation of the generated sequences. [·; ·] denotes the concatenation operation. WS ∈ R(ds+dz)×dy and b ∈ Rdy are parameters where ds and dy are the dimensions of hi and ŷi, respectively. At the i-th time-step, the decoder predicts the movement ŷi conditioned on hi as well as the latent feature representation zi, and thus can capture the fine-grained correspondence between music feature sequence and dance movement sequence.

heading: 3.3 DYNAMIC AUTO-CONDITION LEARNING APPROACH
text: Exposure bias (Ranzato et al., 2015; Lamb et al., 2016; Zhang et al., 2019) is a notorious issue in natural language generation (NLG), which refers to the train-test discrepancy of autoregressive generative models. This kind of models are never exposed to their own prediction errors at the training phase due to the conventional teacher-forcing training scheme. However, compared to NLG, motion sequence generation suffers from the more severe exposure bias problem. In NLG, the bias of predicted probability distribution over vocabulary can be corrected by sampling strategy. For instance, we can still generate the target token with index 2 by sampling on probability distribution [0.3, 0.3, 0.4] whose groundtruth is [0, 0, 1]. While any small biases of the predicted motion at each time-step will be accumulated and propagated to the future, since each generated motion is a real-valued vector (rather than a discrete token) representing 2D or 3D body keyjoints in the continuous space. As a result, the generated motion sequences tend to quickly freeze within a few seconds. This problem becomes even worse in the generation of long motion sequences, e.g., more than 1000 movements.
Scheduled sampling (Bengio et al., 2015) is proposed to alleviate exposure bias in NLG, which does not work in long motion sequence generation. Since its sampling-based strategy would feed long predicted sub-sequences of high bias into the model at early stage, which causes gradient vanishing at training due to error accumulation of these high-biased sub-sequences. Motivated by this, we propose a dynamic auto-condition training method as a novel curriculum learning strategy to alleviate the error accumulation of autoregressive models in long motion sequence generation. Specifically, our learning strategy gently changes the training process from a fully guided teacher-forcing scheme using the previous ground-truth movements, towards a less guided autoregressive scheme mostly using the generated movements instead, which bridges the gap between training and inference.
As shown in Figure 1, The decoder predicts Ytgt = {y1, ..., yp, yp+1, ..., yp+q, yp+q+1, ...} from Yin = {y0, ŷ1, ..., ŷp, yp+1, ..., yp+q, ŷp+q+1, ...} which alternates the predicted sub-sequences with length p (e.g., ŷ1, ..., ŷp) and the ground-truth sub-sequences with length q (e.g., yp+1, ..., yp+q). y0 is a predefined begin of pose (BOP). During the training phase, we fix q and gradually increase p according to a growth function f(t) ∈ {const, bλtc, bλt2c, bλetc} where t is the number of training epochs and λ < 1 controls how frequently p is updated. We choose f(t) = bλtc in practice through the empirical comparison. Note that our learning strategy degenerates to the method proposed in Li et al. (2017) when f(t) = const. While the advantage of a dynamic strategy over a static strategy lies in that the former introduces the curriculum learning spirit to dynamically increase the difficulty of curriculum (e.g., larger p) as the model capacity improves during training, which can further improve the model capacity and alleviate the error accumulation of autoregressive predictions. Finally, we estimate parameters of g(·) by minimizing the `1 loss on D:
`1 = 1
N N∑ i=1 ||g(Xi)− Y (i)tgt ||1 (5)

heading: 4 EXPERIMENTAL SETUP

heading: 4.1 DATASET COLLECTION AND PREPROCESSING
text: Although the dataset (Lee et al., 2019) contains about 70-hour video clips, the average length of each clip is only about 6 seconds. Moreover, their dataset cannot be used by other methods to generate longterm dance except for their specially designed decomposition-to-composition framework. To facilitate the task of long-term dance generation with music, we collect a high-quality dataset consisting of 790 one-minute clips of pose data with 15 FPS, totaling about 12 hours. And there are three styles in the dataset including “Ballet”, “Hiphop” and “Japanese Pop”. Table 1 shows the statistics of our dataset. In the experiment, we randomly split the dataset into the 90% training set and 10% test set.
Pose Preprocess. For the human pose estimation, we leverage OpenPose (Cao et al., 2017) to extract 2D body keyjoints from videos. Each pose consists of 25 keyjoints2 and is represented by a 50-dimension vector in the continuous space. In practice, we develop a linear interpolation algorithm to find the missing keyjoints from nearby frames to reduce the noise in extracted pose data.
2https://github.com/CMU-Perceptual-Computing-Lab/openpose/blob/master/ doc/output.md#pose-output-format-body_25
Audio Preprocess. Librosa (McFee et al., 2015) is a well-known audio and music analysis library in the music information retrieval, which provides flexible ways to extract the spectral and rhythm features of audio data. Specifically, we extract the following features: mel frequency cepstral coefficients (MFCC), MFCC delta, constant-Q chromagram, tempogram and onset strength. To better capture the beat information of music, we convert onset strength into a one-hot vector as an additional feature to explicitly represent the beat sequence of music, i.e., beat one-hot. Finally, we concatenate these audio features as the representation of a music frame, as shown in Table 4 in Section 6.2. During extracting audio features, the sampling rate is 15,400Hz and hop size is 1024. Hence, we have 15 audio samples per second that is aligned with the 15 FPS of pose data.

heading: 4.2 IMPLEMENTATION DETAILS
text: The music encoder consists of a stack ofN = 2 identical layers. Each layer has two sublayers: a local self-attention sublayer with l = 8 heads and a position-wise fully connected feed-forward sublayer with 1024 hidden units. Each head contains a scaled dot-product attention layer with dk = dv = 64 and its receptive yield is restricted by setting k = 100. Then we set the sequence length n = 900, dimension of acoustic feature vector dx = 438, dimension of hidden vector dz = 256, respectively. The dance decoder is a 3-layer LSTM with ds = 1024 and the dimension of pose vector dy = 50. We set λ = 0.01 and q = 10 for the proposed learning approach and train the model using the Adam optimizer with the learning rate 1e-4 on 2 NVIDIA V100 GPUs.

heading: 4.3 BASELINES
text: The music-conditioned generation is an emerging task and there are few methods proposed to solve this problem. In our experiment, we compare our proposed approach to the following baselines: (1) Dancing2Music. We use Dancing2Music (Lee et al., 2019) as the primary baseline that is the previous state-of-the-art; (2) Aud-MoCoGAN. Aud-MoCoGAN is an auxiliary baseline used in Dancing2Music, which is original from MoCoGAN (Tulyakov et al., 2018); (3) LSTM. Shlizerman et al. (2018) propose a LSTM network to predict body dynamics from the audio signal. We modify their open-source code to take audio features of music as inputs and produce human pose sequences.

heading: 5 EXPERIMENTAL RESULTS
text: In this section, we conduct extensive experiments to evaluate our approach and compare it to the aforementioned baselines. Note that we do not include comparison experiment on the dataset collect by Lee et al. (2019) due to the above issue in Section 4.1.

heading: 5.1 AUTOMATIC METRICS AND HUMAN EVALUATION
text: We evaluate the different methods by automatic metrics and conduct a human evaluation on motion realism and smoothness of generated dance movements, as well as style consistency. Specifically, we randomly sample 60 music clips from the test set to generate dances, then divide the dances generated by three methods and the real dances into four pairs: (LSTM, ours), (Dancing2Music, ours), (real, ours) and (real, real). We invite 10 amateur dancers as the annotators to answer 3 questions for each pair that is blind for them (use (A, B) pair instead): (1) Which dance is more realistic regardless of music? (2) Which dance is more smooth regardless of music? (3) Which dance matches the music better with respect to style? Note that, we do not include Aud-MoCoGAN in the human evaluation, since both it and Dancing2Music are GAN based methods while the latter is the state-of-the-art.
Style Consistency Preference
Smoothness PreferenceMotion Realism Preference
Motion Realism, Style Consistency and Smoothness. We evaluate the visual quality and realism of generated dances by Fréchet Inception Distance (FID) (Heusel et al., 2017), which is used to measure how close the distribution of generated dances is to the real. Similar to Lee et al. (2019), we train a style classifier on dance movement sequences of three styles and use it to extract features for the given dances. Then we calculate FID between the synthesized dances and the real. As shown in Table 2, our FID score is significantly lower than those of baselines and much closer to that of the real, which means our generated dances are more motion-realistic and more likely to the real dances. Besides, we use the same classifier to measure the style accuracy of generated dance to the music, and our approach achieves 77.6% accuracy and significantly outperforms Dancing2Music by 17.2%.
Human evaluation in Figure 2 consistently shows the superior performance of our approach, compared to baselines on motion realism, style consistency and smoothness. We observe the dances generated by LSTM have lots of floating movements and tend to quickly freeze within several seconds due to the error accumulation, which results in lower preferences. While Dancing2Music can generate the smooth dance units, the synthesized dances still have significant jumps where dance units are connected due to the inherent gap between them, which harm its overall performance in practice. This is also reflected in the preference comparisons to our approach, in which only 35% annotators prefer Dancing2Music on motion realism and 21.7% prefer Dancing2Music on smoothness. Besides, the result on style consistency shows our approach can generate more style-consistent dances with musics. Since our approach models the fine-grained correspondence between music and dance at the methodological level and also introduces more specific music features. However, Dancing2Music only considers a global style feature extracted by a pre-trained classifier in the dance generation.
In the comparison to real dances, 41.2% annotators prefer our method on motion realism while 30.3% prefer on style consistency. Additionally, we found that 57.9% annotators prefer our approach on smoothness compared to real dances. Since the imperfect OpenPose (Cao et al., 2017) introduces the minor noise on pose data extracted from dance videos. On the other hand, in the preprocessing, we develop an interpolation algorithm to find missing keyjoints and reduce the noise of training pose data.
Besides, this also indicates the chain structure based decoder can well capture the spatial-temporal dependency among human motion dynamics and produce the smooth movements.
Beat Coverage and Hit Rate. We also evaluate the beat coverage and hit rate introduced in Lee et al. (2019), which defines the beat coverage as Bk/Bm and beat hit rate as Ba/Bk, where Bk is the number of kinematic beats, Bm is the number of musical beats and Ba is the number of kinematic beats that are aligned with the musical beats. An early study (Ho et al., 2013) reveals that kinematic beat frames occur when the direction of the movement changes. Therefore, similar to Yalta et al. (2019), we use the standard deviation (SD) of movement to detect the kinematic beats in our experiment. The onset strength (Ellis, 2007) is a common way to detect musical beats. Figure 3 shows two short clips of motion SD curves and aligned musical beats. We observed that the kinematic beats occur where the musical beats occur, which is consistent with the common sense that dancers would step on musical beats during dancing but do not step on every musical beat. As we can see in Table 2, LSTM and Aud-MoCoGAN generate dances with few kinematic beats and most of them do not match the musical beats. Our method outperforms Dancing2Music by 6.1% on beat coverage and 2.7% on hit rate, which indicates that introducing the features about musical beat into model is beneficial to the better beat alignment of generated dance and input music.
Diversity and Multimodality. Lee et al. (2019) introduce the diversity metric and the multimodality metric. The diversity metric evaluates the variations among generated dances corresponding to various music, which reflects the generalization ability of the model and its dependency on the input music. Similarly, we use the average feature distance as the measurement and these features are extracted by the same classifier used in measuring FID. We randomly sample 60 music clips from test set to generate dances and randomly pick 500 combinations to compute the average feature distance. Results in Table 2 show the performance of our approach is superior to Dancing2Music on diversity. The reason is that Dancing2Music only considers a global music style feature in generation while different music of the same style have the almost same style feature. Our approach models the fine-grained correspondence between music and dance at the methodological level, simultaneously takes more specific music features into account. In other words, our approach is more dependent on input music and thus can generate different samples for difference music clips of the same style.
Multimodality metric evaluates the variations of generated dances for the same music clip. Specifically, we generate 5 dances for each of randomly sampled 60 music clips and compute the average feature distance. As we can see, our approach slightly underperforms Dancing2Music and AudMoCoGAN since it has more dependency on music inputs as aforementioned, which make the generated dances have the limited variation patterns given the same music clip.

heading: 5.2 LONG-TERM DANCE GENERATION
text: To further investigate the performance of different methods on long-term dance generation, we evaluate FID of dances generated by different methods over time. Specifically, we first split a generated dance with 1 minute into 15 four-second clips and measure FID of each clip. Figure 4 shows FID scores of LSTM and Aud-MoCoGAN grow rapidly in the early stage due to error accumulation and converge to the high FID, since the generated dances quickly become frozen. Dancing2Music maintains the relatively stable FID scores all the time, which benefits from its decomposition-tocomposition method. While its curve still has subtle fluctuations since it synthesizes whole dances by
composing the generated dance units. Compared to baselines, FID of our method are much lower and close to real dances, which validates the good performance of our method on long-term generation.

heading: 5.3 ABLATION STUDY
text: Comparison on Encoder Structures. We compare the different encoder structures in our model with the same LSTM decoder and the same curriculum learning strategy, as shown in Table 3. The transformer encoders outperform LSTM and the encoder in ConvS2S (Gehring et al., 2017) on both FID and style consistency, due to its superior performance on sequence modeling. Although the transformer encoder with our proposed local self-attention slightly underperforms the global self-attention by 0.7 higher in FID and 1.5% lower in ACC, it has much fewer parameters in our settings. This confirms the effectiveness of the local self-attention on long sequence modeling.
Comparison on Learning Strategies. To investigate the performance of our proposed learning approach, we conduct an ablation study to compare the performances of our model using different learning strategies. As we can see the right of in Table 3, all curriculum learning strategies significantly outperform the static auto-condition strategy (Li et al., 2017). Among them, the linear growth function f(t) = bλtc performs best, which might be that increasing the difficulty of curriculum too fast would result in the model degradation. Besides, the original teacher-forcing strategy has the highest FID and the lowest style accuracy due to severe error accumulation problem, which indicates that using our proposed curriculum learning strategy to train the model can effectively alleviate this issue.

heading: 6 CONCLUSION
text: In this work, we propose a novel seq2seq architecture for long-term dance generation with music, e.g., about one-minute length. To efficiently process long sequences of music features, we introduce a local self-attention mechanism in the transformer encoder structure to reduce the quadratic memory requirements to the linear in the sequence length. Besides, we also propose a dynamic auto-condition training method as a novel curriculum learning strategy to alleviate the error accumulation of autoregressive models in long motion sequence generation, thus facilitate the long-term dance generation. Extensive experiments have demonstrated the superior performance of our proposed approach on automatic metrics and human evaluation. Future works include the exploration on end-to-end methods that could work with raw audio data instead of preprocessed audio features.

heading: 6.1 MORE DETAILS ABOUT DATA COLLECTION
text: The dance videos are collected from YouTube by crowd workers, which are all solo dance videos with 30FPS. We trim the beginning and ending several seconds for each of collected videos to remove the silence parts, and split them into one-minute video clips. Then, we extract 2D pose data from these video clips by OpenPose (Cao et al., 2017) with 15FPS setting and collect 790 one-minute clips of pose data with 15FPS, totaling about 13 hours. Finally, we extract the corresponding audio data from video clips by FFmpeg3. The code is implemented based on PyTorch framework. MIT License will be used for the released code and data.

heading: 6.2 EXTRACTED MUSIC FEATURES
text: In this section, we detail the extracted features of music that are feeded into our model. Specifically, we leverage the public Librosa (McFee et al., 2015) to extract the music features including: 20- dim MFCC, 20-dim MFCC delta, 12-dim chroma, 384-dim tempogram, 1-dim onset strength (i.e., envelope) and 1-dim one-hot beat, as shown in Table 4.

heading: 6.3 DOWNSTREAM APPLICATIONS
text: The downstream applications of our proposed audio-conditioned dance generation method include: (1) Our model can be used to help professional people choreograph new dances for a given song and teach human how to dance with regard to this song; (2) With the help of 3D human pose estimation (Ci et al., 2019) and 3D animation driving techniques, our model can be used to drive the various 3D character models, such as the 3D model of Hatsune Miku (very popular virtual character in Japan). This technique has the great potential for the virtual advertisement video generation that could be used for the promotion events on social medias like TikTok, Twitter and etc.

heading: 6.4 MUSICAL BEAT DETECTION UNDER LOW SAMPLING RATES
text: In this section, we conduct an additional experiment to evaluate the performance of Librosa beat tracker under 15,400Hz sampling rate. Specifically, we first utilize it to extract the onset beats from audio data with 15,400Hz and 22,050Hz respectively, then compare two groups of onset beats under the same time scale. Since 22,050Hz is a common sampling rate used in music information retrieval, we define the beat alignment ratio as B2/B1, where B1 is the number of beats under 15,400Hz and B2 is the number of beats under 15,400Hz that are aligned with the beats under 22,050Hz. One beat is counted when | t1 − t2 |≤ ∆t. t1 denotes the timestamp of a certain beat under 15,400Hz while t2 refers to the timestamp of a certain beat under 22,050Hz, ∆t is the time offset threshold.
In the experiment, we set ∆t = 1/15s (FPS is 15) and calculate the beat alignment ratio for audio data of three styles respectively. We randomly sample 10 audio clips from each style to calculate the beat alignment ratio and do the sampling for 10 times to take the average. As shown in Table 5, most of beats extracted with 15,400Hz are aligned with those extracted with 22,050Hz. Besides, we randomly sample an audio clip and visualize beat tracking curves of its first 20 seconds under the sampling rates of 15,400Hz and 22,050Hz. As we can see in Figure 5, most of the beats from 15,400Hz and 22,050Hz are aligned within the time offset threshold ∆t = 1/15s.
3https://ffmpeg.org/
Table 5: The beat alignment ratio B2/B1.
Category Beat Alignment Ratio (%)
Ballet 88.7 Hiphop 93.2 Japanese Pop 91.6
0 2.5 5 7.5 10 12 15 18 20
0.0
0.2
0.4
0.6
0.8
1.0 Onset Strength (22050Hz) Onset Strength (15400Hz) Musical Beats (22050Hz) Musical Beats (15400Hz)
Figure 5: Beat tracking curves for the first 20 seconds of a music clip randomly sampled from audio data under the sampling rates of 15,400Hz and 22,050Hz. Most of the beats are aligned within the time offset threshold ∆t = 1/15s.

Review summaries:
The authors present a seq2seq model with a sparse transformer encoder and an LSTM decoder. They utilize a learning curriculum wherein the autoregressive decoder is initially trained using teacher forcing and is gradually fed its past predictions as training progresses. The authors introduce a new dataset for long term dance generation. They utilize both subjective and objective metrics to evaluate their method. The proposed method outperforms other baselines for dance generation. Finally they conduct ablation studies demonstrating the benefits of using a transformer encoder over other architectures, and the benefits of the proposed curriculum learning scheme.