Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning

Abstract: Dancing to music is one of human's innate abilities since ancient times. In machine learning research, however, synthesizing dance movements from music is a challenging problem. Recently, researchers synthesize human motion sequences through autoregressive models like recurrent neural network (RNN). Such an approach often generates short sequences due to an accumulation of prediction errors that are fed back into the neural network. This problem becomes even more severe in the long motion sequence generation. Besides, the consistency between dance and music in terms of style, rhythm and beat is yet to be taken into account during modeling. In this paper, we formalize the music-driven dance generation as a sequence-to-sequence learning problem and devise a novel seq2seq architecture to efficiently process long sequences of music features and capture the fine-grained correspondence between music and dance. Furthermore, we propose a novel curriculum learning strategy to alleviate error accumulation of autoregressive models in long motion sequence generation, which gently changes the training process from a fully guided teacher-forcing scheme using the previous ground-truth movements, towards a less guided autoregressive scheme mostly using the generated movements instead. Extensive experiments show that our approach significantly outperforms the existing state-of-the-arts on automatic metrics and human evaluation. We also make a demo video to demonstrate the superior performance of our proposed approach at https://www.youtube.com/watch?v=lmE20MEheZ8.

Reviews and author responses:

Summary:
The authors present a seq2seq model with a sparse transformer encoder and an LSTM decoder. They utilize a learning curriculum wherein the autoregressive decoder is initially trained using teacher forcing and is gradually fed its past predictions as training progresses. The authors introduce a new dataset for long term dance generation. They utilize both subjective and objective metrics to evaluate their method. The proposed method outperforms other baselines for dance generation. Finally they conduct ablation studies demonstrating the benefits of using a transformer encoder over other architectures, and the benefits of the proposed curriculum learning scheme.

Comments:
1. The authors claim to introduce the local self-attention mechanism, however, it is very similar to an already proposed sparse transformer architecture [1].
2. In the music encoder section, the authors claim that they can afford to look at a small locality in the music to generate the dance sequence. This is not necessarily true. The structure of music is arguably an important feature in the choreography of a dance sequence.
3. Section 4: The experiment setup lacks important details. There is no mention of the length of the length of the music clip input to the model. Furthermore, in the appendix detailing the audio pre-processing steps, the sampling rate, window size, and hop size are not mentioned without which reproducibility greatly suffers. The mention that the audio frames are aligned with video frames. This is not ideal to extract chromagrams or onset strength. Assuming an audio sample rate of 44100Hz, the equivalent frame length will be 2940 samples (for 15 fps). 2940 samples is a long enough time for several onsets to occur within the frame. Regardless, these details are necessary within the main text of the paper and should not be relegated to the appendix.
4. There are a few minor issues: 
    - Lee et al., 2013 is cited in the text but no reference exists in the bibliography. 
    - genration -> generation
    - grammatical issues here and there
    - the paper ends abruptly. A conclusion section summarizing the key findings and discussing future direction would be nice.
    - there also exists another large dance database [2] which may be worth mentioning in the paper.

Overall the paper utilizes modern deep learning techniques well to solve an interesting problem. However, there is a lack of depth in terms of how these techniques are adapted for the particular task. Hence, I rate the paper as a 4/10.

References:
[1] Child, Rewon, et al. "Generating long sequences with sparse transformers." arXiv preprint arXiv:1904.10509 (2019).
[2] Tsuchida, Shuhei, et al. "AIST Dance Video Database: Multi-Genre, Multi-Dancer, and Multi-Camera Database for Dance Information Processing." ISMIR. 2019.

**Update**: Revised score from 4 to a 6, mainly because the authors rightfully pointed out that they did have an ablation study in their paper which demonstrates the efficacy of their proposed methods.

**Summary**: This paper presents a method for generating dances from audio in an end-to-end fashion. Specifically, they pose the problem as a sequence-to-sequence learning task from acoustic features to pose information. They demonstrate that humans prefer dances generated by their method more often than those from the prior state-of-the-art dance generation system.

Based on the human evaluation and my own observations of the qualitative results in the supplementary material, the authors' claim that this system is an improvement over the previous state-of-the-art seems reasonable. However, the technical contributions (novel Transformer architecture, a new "curriculum learning strategy) are _not_ justified experimentally (e.g., by an ablation study). Hence, the work feels lacking in substance; at best, the paper simply demonstrates performance improvements to an existing task using existing metrics.

#### Technical novelty is unsubstantiated

The biggest problem with this work is a lack of ablation study for the novel aspects of the proposed system. Specifically, the modifications to the Transformer architecture (Section 3.2) and the curriculum learning (Section 3.3). These constitute the _only_ technically novel aspects of this work (the latter is even included in the title of the paper), but neither is justified experimentally. Hence, I can only treat these ideas as implementation details rather than contributions.

#### One song -> one dance?

Based on my understanding of the proposed approach, it should only be possible to produce a single dance for a given musical input. But the "Multimodality" metric (incidentally, a strange and misleading name choice) is defined as the variation among the generated dances for the same piece of music. The multimodality score for the proposed model indicates that it _can_ generate multiple dances per song. How is this possible? The decoder doesn't model a distribution of pose information given audio, and I can't find an explanation anywhere in the paper.

Furthermore, the multimodality score for the proposed model is worse than that of the previous state of the art. For downstream applications, is it better to have a system which generates a single excellent dance for a given song, or one which can generate many lower-quality dances? I think the human evaluation is a little bit unfair in this regard, as it does not take the variety into account. But at the very least, there should be some justification from an explanation of downstream use cases.

#### Human element?

One high level question I have about this work is why generate _dances_ rather than _choreography_? Presumably it would be quite challenging to teach dances to humans from this pose information as opposed to typical choreography instruction dancers might receive. Perhaps teaching these dances to humans is not an intended downstream application, but it seems like it would be easier and more useful to generate choreography rather than 3D pose information. Can the authors comment on this?

#### Unusual methodological decisions

It is a strange decision to use a Transformers for the encoder and an RNN for the decoder of the proposed seq2seq model. The stated justification for using an RNN as the decoder is that the decoder needs to be autoregressive. But most (all?) Transformer-based decoders are also autoregressive... this justification "smells funny". I'm guessing that Transformers just didn't work as well for whatever reason; why not just say that? Or better yet compare the two experimentally?

#### Low-level comments

Missing many citations (probably many more related to dance / choreography generation):
- Citations on page 1 to Fan et al. 2011 and Lee et al. 2013 are broken (not hyperlinked) and missing (from the bibliography)
- Dance Dance Convolution (Donahue et al. 2017)
- Music-driven dance generation (Qi et al. 2019)
- Dance beat tracking from visual information alone (Pedersoli and Goto 2020)

Dear reviewer, thank you for your feedback and suggestion.

Yes, we utilize the same parameters for the beat tracker. Follow your suggestion, we conduct an additional experiment to evaluate the performance of librosa beat tracker under the sampling rate of 15,400Hz. Specifically, we use librosa beat tracker to extract the onset beats from audio data with 15,400Hz and 22,050Hz respectively, and then compare two groups of onset beats under the same time scale.

As you mentioned, 22,050Hz is one common sampling rate used in music information retrieval, thus we define the beat alignment ratio as $B_2/B_1$, where $B_1$ is the number of beats under 15,400Hz and $B_2$ is the number of beats under 15,400Hz that are aligned with the beats under 22,050Hz.
Specifically, one beat is counted when $\mid t_1 - t_2 \mid \leq \Delta t$. $t_1$ denotes the timestamp of a certain beat under 15,400Hz while $t_2$ refers to the timestamp of a certain beat under 22,050Hz, $\Delta t$ is the time offset threshold. 

In the experiment, we set $\Delta t=1/15s$ (FPS is 15) and calculate the beat alignment ratio for audio data of three styles respectively.
We randomly sample 10 audio clips from each style to calculate the beat alignment ratio and do the sampling for 10 times to take the average. As shown in the table below, most of beats extracted with 15,400Hz are aligned with those extracted with 22,050Hz. 

    Category         |     Beat Alignment Ratio (%)
     Ballet          |     88.7    
     Hiphop          |     93.2    
     Japanese Pop    |     91.6    

Besides, we randomly sample an audio clip and visualize the beat tracking curves of its first 20 seconds under the sampling rates of 15,400Hz and 22,050Hz. As we can see in the figure, most of the beats under two sampling rates are aligned within the time offset threshold.

We have revised the paper to include this additional experiment into the appendix to make this issue more clear.

Dear reviewer, thank you for your time and comments. We are glad to discuss to clarify some questions. Below, we respond to each of your comments and look forward to your further feedback.

Q1: “The authors claim to introduce the local self-attention mechanism, however, it is very similar to an already proposed sparse transformer architecture.”

A1: The sparse transformer and our local self-attention transformer are different in both the architecture and the usage. The former modifies the attention mechanism in the autoregressive transformer decode phase to generate long sequences when attending to all previously generated tokens is not necessary. Therefore, sparse transformer selectively pays attention to some positions, which are not always nearby. However, we design the local self-attention transformer in the encode phase to extract local bidirectional music features for each position.


Q2: “In the music encoder section, the authors claim that they can afford to look at a small locality in the music to generate the dance sequence. This is not necessarily true. The structure of music is arguably an important feature in the choreography of a dance sequence.”

A2: We do believe that the structure of music is important for choreography of a dance sequence. In our opinion, using local self-attention in our model would not hurt its perception of the music structure to a great extent, but can save a lot of memory cost from $O(n^2)$ to $O(nk)$ with small $k$, especially when sequence length $n$ is large, e.g., more than 1000. On the other hand, stacking multiple encoders would increase the receptive field of local self-attention. For example, the stacked blocks with $N=2$ encoders offer a receptive field of around 13 seconds under $k=100$ and FPS = 15 setting, which is enough to capture many useful music structures, such as repetition, comparison, and melodic sequence. Besides, we use local attention only in the encode phase to extract local music features, while the decoder can obtain all the music information in one direction.


Q3: “The experiment setup lacks important details……Regardless, these details are necessary within the main text of the paper and should not be relegated to the appendix.”

A3: Due to the limit of 8-page main text, we do not have enough space to pull all preprocessing details into the main text. The length of the music clip input into the model is also 1 minute. In our preprocessing, the sampling rate is 15400Hz while hop size is 1024, thus we have 15 audio samples per second. That’s to say, there are 900 audio samples in one minute, equal to the 900 frames in one-minute video (FPS is 15). Since one additional page for the main text is allowed during the rebuttal phase, we revised the paper and added these implementation details into the experimental setup section.


Q4: “There are a few minor issues….”

A4: Thanks for pointing out these minor issues. We make these issues clear in the revised version of the paper.


Q5: “However, there is a lack of depth in terms of how these techniques are adapted for the particular task.”

A5: (1) This technique can be used to help professional people choreograph new dances for a given song and teach human how to dance with regards to this song; (2) With the help of 3D human pose estimation and 3D animation driving techniques, our system can be used to drive the various 3D character models, such as the 3D model of Hatsune Miku (very popular virtual character in Japan). We have tried the these use cases in the real production and the feedback is good.


Thanks for the detailed response to my comments.

1. Sparse-transformer is indeed not exactly the same as the local attention proposed. Apologies for the confusion.

2. Thanks for additional details about the audio pre-processing parameters. I am curious if you utilize the same parameters for the beat tracker. Most methods in music information retrieval operate on audio sample rates of 22050 or 44100 Hz. I would recommend evaluating the librosa beat-tracker and checking whether it performs as expected with lower sample rate audio.

3. Thanks for adding these details.

Dear reviewer, thank you for your constructive feedback and comments. We will make the additional details of responses into the appendix to make the paper more clear.

**Response to Questions for the rebuttal period:**

Q1: “Did you find that limiting the model to k=100 caused any limitations related to long-term coherence in the dances that were generated? For example, was the model unable to repeat dance motifs over a period of time longer than 100 events?”

A1: We did not observe the obvious limitations related to long-term coherence in generated dance. Because we only use the local self-attention transformer to extract musical feature, while the dance decoder can obtain all the music information in one direction and capture the repeat dance motifs.

Q2: “In section 3.3, you mention that generating motion as a real-valued vector causes more problems with error accumulation than sampling from a discrete probability distribution. Did you consider using a Mixture Density Network to allow sampling from continuous outputs?”

A2: Yes, at the early stage of this work, we tried Mixture Density Network. The poses that sample from the learned distribution suffer from the significant distortion and are not smooth enough. Its performance is not desirable.

Thank you for the updates! It looks like not quite all of the additional details in your responses made it into the paper, likely because of space reasons. Maybe a few more could be added to the appendix? But overall, I'm happy with the responses and clarifications and think it's a solid paper.

Thanks for the updates and responses -- I'm generally happy with your answers!


Dear reviewer, thank you for your time and comments. We are glad to discuss to clarify some questions. Below, we respond to each of your comments and look forward to your further feedback.

Q1: “The biggest problem with this work is a lack of ablation study for the novel aspects of the proposed system.”

A1: Actually, in Section 5.3, we have already provided the ablation study to empirically justify our proposed Transformer architecture and curriculum learning strategy. In the left one of Table 4, we use the same LSTM decoder and the same curriculum learning strategy (growth function $f(t)=\lfloor \lambda t \rfloor$), then compared the encoders with different architectures, including the proposed transformer encoder architecture with local self-attention, the original transformer encoder (global self-attention), LSTM encoder and the encoder in ConvS2S [1]. While in the right one of Table 4, we use the same seq2seq architecture (proposed transformer encoder with local self-attention and LSTM decoder), then compared different training strategies. 

    Learning Approach   |     FID     |      ACC(%)  
     Teacher-forcing    |     61.2    |      5.3  

Besides, we also added one more experiment using original teacher-forcing (without curriculum learning strategy) to train the proposed seq2seq model. As we can see, it has high FID score and low style accuracy due to the severe error accumulation problem, which indicates that using our proposed curriculum learning strategy to train the model can effectively alleviate this issue.



Q2: “The multimodality score for the proposed model indicates that it can generate multiple dances per song. How is this possible?”

A2: “Multimodality” metric is originally introduced in [2]. To avoid confusion, we just follow it to name this metric. As is mentioned in the dance decoder part of Section 3.2, the initial hidden state $h_0$ of decoder is initialized by sampling from Standard Normal Distribution, which enables our generation model to have some randomness. Secondly, when evaluating multimodality metric, we did the generation for 5 times per song and calculate the average feature distance (please refer to the last paragraph in Section 5.1). Thus, our system can generate multiple dances per song.



Q3: “Furthermore, the multimodality score for the proposed model is worse than that of the previous state of the art. For downstream applications, is it better to have a system which generates a single excellent dance for a given song, or one which can generate many lower-quality dances?”

A3: The GAN based Aud-MoCoGAN and Dancing2Music use a global music style feature and random gaussian noise to initialize the generation process. They do not that depend on music inputs since different music of same style have the almost same style feature. While our seq2seq method considers more fine-grained music features in the generation and has more dependencies on music inputs. That’s the reason why our proposed method slightly underperforms Aud-MoCoGAN and Dancing2Music on multimodality metric. As explained in A2, our system can generate multiple excellent dances for a given song. 



Q4: “But at the very least, there should be some justification from an explanation of downstream use cases.”

A4: Downstream use cases: (1) Our system can be used to help professional people choreograph new dances for a given song and teach human how to dance with regards to this song; (2) With the help of 3D human pose estimation [4] and 3D animation driving techniques, our system can be used to drive the various 3D character models, such as the 3D model of Hatsune Miku (very popular virtual character in Japan). We have tried these use cases in the real production and the feedback is good.



Q5: “One high level question I have about this work is why generate dances rather than choreography?”

A5: In this work, we use the term “dance generation” rather than “choreography” just for the easy understanding for readers. Our system can be regarded as a machine choreographer that can generate new dances for a given song. Besides, it is not hard to teach human dance by the generated pose information since these poses are all human skeletons and it is easy for human to mimic.



Q6: “Unusual methodological decisions.”

A6: Directly utilizing transformer-based decoder to predict human poses ignores the temporal-spatial dependencies between pose key joints, which have proven to be highly effective in the state-of-the-art methods in human motion prediction. This issue has been addressed by recent works, such as [3]. But it is another research topic.


Will make low-level issues clear in the revised version.

References:
[1] Gehring et al. Convolutional sequence to sequence learning. ICML 2017.
[2] Lee et al. Dancing to Music. NeurIPS 2019.
[3] Cai et al. Learning Progressive Joint Propagation for Human Motion Prediction. ECCV 2020.
[4] Ci et al. Optimizing Network Structure for 3D Human Pose Estimation. ICCV 2019.


Dear Reviewers,

We appreciate you for your constructive feedback and comments for the improvement of the paper. Due to one additional page allowed for rebuttal, we have updated the manuscript with changes as follows:

1. We include the detailed statistics of our collected dataset, including genre information, number of clips of different genres, clip length, fps and resolution. 

2. We rearrange the tables and figures to make them on the same page with the corresponding text description, and add a conclusion summarizing the key findings and discussing future direction. Besides, the content in the appendix is also moved to the experimental setup, with the included parameters for computing spectral features of music.

3. To further validate the effectiveness of our learning approach, we conduct one more experiment to train the proposed model by the original teacher-forcing. The detailed information is included in the right of Table 4 and Section 5.3.

4. We revised the paper to clear minor issues you mentioned in comments.

Thank you again and best regards,
Authors of Paper1186

Dear reviewer, thank you for your time and comments. We are glad to discuss to clarify some questions. Below, we respond to each of your comments and look forward to your further feedback.

Q1: “The appendix is quite short, but includes vital details to understand what the data is. For example, the fact that openpose is used to extract pose from the video data. These details should be in the main text of the paper.”

A1: Thanks for this suggestion. We revised the paper to include these details in the main text of paper.


Q2: “I didn't understand the "beat coverage" evaluation. Is "standard deviation of motion" the euclidean distance between successive poses? Or something different? What constitutes a "hit" here.”

A2: In general, music has more beats than dance in a video. Beat coverage measures the ratio of kinematic beats to musical beats. The higher the beat coverage is, the stronger the rhythm of dance is. Yes, “standard deviation of motion” is the Euclidean distance between successive poses. One hit is counted when a kinematic beat and a musical beat occur at the same time. 


Q3: “The data includes genre information, but this doesn't seem to be explicitly reported on except by way of the "style match" evaluation. Are there differences in performance across styles, or do they all perform comparably?”

A3: We revised the paper to add the statistics of the dataset, including genre information, please refer to Table 1 in the revised paper. The performance of Japanese pop style is a bit better than other two types, due to the data distribution in dataset.


**Response to the minor comments:**

Q1: “On the topic of beat tracking, I'm a little unclear on what exactly is being done here on the audio side.”

A1: Sorry for the misleading. We track the musical beat by invoking librosa.beat.beat_track function that is implemented base on (Ellis, 2007). The paper have been revised to clear this issue.

Q2: “Related suggestion, many dance styles depend on the downbeat (bar lines) in addition to beats (usually quarter notes). It may be worth including downbeat estimations (eg from madmom [2]) as an input feature at some point.”

A2: Thanks for this helpful suggestion. We will try the downbeat feature as an input feature later.

Q3: “are features really extracted at 15400 frames per second?”

A3: No. We mean the audio sampling rate is 15400Hz, hop size is 1024, thus features are extracted at 15 frames per second. We revised the paper to include these preprocessing details in the experimental setup.

Q4: “If you continue this line of work, you might want to check out the recently published (2019) AIST database [3].”

A4: Thanks for the kind reminder, we will check this dataset later.


**Response to Concerns:**

Q1: “The paper mentions a new dataset and codebase will be released, but few details are given about the contents of the dataset (e.g., how many clips of each genre? how were they collected? what license will be used?), the codebase (e.g., what framework was used?), or how they will eventually be accessed.”

A1: We revised the paper to include the detailed statistics of dataset due to one additional page allowed for rebuttal. The dance videos are collected from YouTube. We will release pose data and corresponding audio data, which are extracted from collected dance videos. The code is implemented based on PyTorch framework and MIT License will be used.

Q2: “Based on the description in section 3.3, it sounds like the model never trains without a sequence $p$ of ground truth teacher forced in the output. Is my understanding correct? If this is the case, does the model exhibit any signs of struggling to generate sequences longer than the maximum length of during training? Have you tried changing the training schedule such that $p$ eventually disappears? I would like to see some more discussion/clarification around these questions.”

A2: Actually, the original code implementation of our proposed curriculum learning is to first feed the model with the autoregressive subsequence and then with the ground-truth subsequence. We are sorry to ignore this detail when drawing the model figure and writing the corresponding part in main text, due to the tight schedule of submission. The paper has been revised to correct this point. The growth function $f(t)$ for the auto-regressive length would eventually make the ground-truth subsequence disappear within the maximum length (900 frames), if the training time is long enough. Using a growth function for the auto-regressive input and a decreasing function for the ground-truth input at the same time is an interesting topic, we would try in the future work.

Q3: “I would like to see more information about the balance of the dataset by genre or other important attributes and then also see some of the evaluation statistics broken out by those attributes. Does the model perform better on some genres than others? If so, is this because of training set imbalance, audio feature differences, or other issues?”

A3: We revised the paper to include this information about dataset. Yes, the model performs a bit better on Japanese pop style due to the imbalance of training data. 

Q4: “More information should be provided about the human evaluation procedure. For example: how many raters were involved, how many questions per rater, were they dance experts or not, did they view wire renderings or 3d models, etc.”

A4: We invite 10 amateur dancers as the raters. Each rater is asked to answer 3 questions for each pair (please refer to the first paragraph of Section 5.1). Since the human evaluation is designed to evaluate the quality of visualized skeleton dances, we do not let them view 3D model.

Q5: “In section 5.1, under “Beat Coverage and Hit Rate”….. I definitely think that at least an overview of what features are used as input should be included in the main body of the paper”

A5: We are sorry about this point. Due to the limit of 8-page main text, we do not have enough space to pull all preprocessing details into the main text. We have revised the paper to include these details in the main text of paper.

Q6: “It would be nice to have a brief conclusion at the end, including a discussion of future research directions.”

A6: Thanks for the kind reminder, we have included the conclusion part in the revised paper.

Q7: “In the appendix, I think you should include much more detail about the audio features used as input. For example, what were the parameters for computing the spectral features? Did you try different sets of features in your investigations? Are both CQT and MFCC really needed?”

A7: We have revised the paper to include the parameter setting for extracting the spectral features. Yes, we tried different set of features and the current set in the paper is the one which performs best. Yes, we have tested in the experiment, CQT and MFCC are good features that have the contribution on improving the performance.

**Additional minor feedback:**

We have revised the paper to clear these issues. Thanks for the kind reminder, we will try the new method to beat detection for evaluation and feature extraction later.


Q6: "It is a strange decision to use a Transformers for the encoder and an RNN for the decoder of the proposed seq2seq model. The stated justification for using an RNN as the decoder is that the decoder needs to be autoregressive. But most (all?) Transformer-based decoders are also autoregressive... this justification "smells funny". I'm guessing that Transformers just didn't work as well for whatever reason; why not just say that? Or better yet compare the two experimentally?"

A6: First, not all transformer based decoders are autoregressive, such as "Non-Autoregressive Neural Machine Translation" (Gu et al., ICLR 2018). Second, we tried the transformer based decoder at the early stage of this work, its performance is not desirable. The reason why cannot directly apply Transformer based decoders to predicting human motions has also been studied in another recent work, "A Spatio-temporal Transformer for 3D Human Motion Prediction" (Aksan et al., arXiv 2020). 

**Summary**

This paper proposes a system for generating long sequences of dance movements conditioned on audio. Through extensive analysis, the proposed system is shown to outperform previous methods across many metrics.

**Strengths**

Extensive analysis across many metrics, both qualitative and quantitative, give solid evidence of this system outperforming others. The included demo video is also a great example.

Long-term generation analysis in section 5.2 clearly demonstrates this model handles long term sequences and shows important differences compared with other models that struggle with that. I thought showing this breakdown by time in addition to the overall FID score in Table 1 was very convincing.

Planned dataset and code release is a good community contribution and will ensure reproducibility.

A novel architecture was created to handle difficulties of long-term sequence generation of dance moves.

This work expands the field of cross-modal learning.

**Concerns**

The paper mentions a new dataset and codebase will be released, but few details are given about the contents of the dataset (e.g., how many clips of each genre? how were they collected? what license will be used?), the codebase (e.g., what framework was used?), or how they will eventually be accessed.

Based on the description in section 3.3, it sounds like the model never trains without a sequence $p$ of ground truth teacher forced in the output. Is my understanding correct? If this is the case, does the model exhibit any signs of struggling to generate sequences longer than the maximum length of $q$ during training? Have you tried changing the training schedule such that $p$ eventually disappears? I would like to see some more discussion/clarification around these questions.

I would like to see more information about the balance of the dataset by genre or other important attributes and then also see some of the evaluation statistics broken out by those attributes. Does the model perform better on some genres than others? If so, is this because of training set imbalance, audio feature differences, or other issues?

More information should be provided about the human evaluation procedure. For example: how many raters were involved, how many questions per rater, were they dance experts or not, did they view wire renderings or 3d models, etc.

In section 5.1, under “Beat Coverage and Hit Rate”, you mention for the first time that features about music beat were incorporated into the model. Prior to this, model input has just been described as audio features. I finally realized that there was an overview of the features hidden in the appendix. I definitely think that at least an overview of what features are used as input should be included in the main body of the paper.

It would be nice to have a brief conclusion at the end, including a discussion of future research directions.

In the appendix, I think you should include much more detail about the audio features used as input. For example, what were the parameters for computing the spectral features? Did you try different sets of features in your investigations? Are both CQT and MFCC really needed?

**Additional minor feedback**

Section 1, first paragraph: dance creation assistant -> dance creation assistance
Section 1, final paragraph: four-folds -> fourfold

I found the wording of the second paragraph in section 3.3 confusing. There’s a particularly awkward split between the first two sentences. I would recommend reworking this paragraph to be more clear.

I would recommend rearranging the tables and figures that start on page 6 to occur in the order they are referenced in the text. If reasonable, it would also be nice if the table/figure is on the same page as the text describing it. For example, section 5.1 starts off by talking about human evaluation, but that figure isn’t until the next page. Figure 2 is about beat tracking, but that text isn’t until the next page.

For beat detection evaluation and as feature input to the model, it might be interesting to use a more recent model such as “MULTI-TASK LEARNING OF TEMPO AND BEAT: LEARNING ONE TO IMPROVE THE OTHER” by Bock et al. (http://archives.ismir.net/ismir2019/paper/000058.pdf). There’s an open source implementation that would be easy to incorporate here: https://github.com/CPJKU/madmom

**Questions for the rebuttal period**

Did you find that limiting the model to $k=100$ caused any limitations related to long-term coherence in the dances that were generated? For example, was the model unable to repeat dance motifs over a period of time longer than 100 events?

In section 3.3, you mention that generating motion as a real-valued vector causes more problems with error accumulation than sampling from a discrete probability distribution. Did you consider using a Mixture Density Network to allow sampling from continuous outputs?

**Recommendation**

My recommendation is to accept this paper. It proposes novel techniques for music-conditioned dance generation and extensive analysis to show the success of those techniques.


This paper describes a method for generating dance movements (pose sequences) from musical audio inputs.
The proposed method combines an attention-based encoder with a recurrent decoder, and uses a curriculum learning strategy to gradually transition from teacher-forcing to autoregressive training.

I generally found this paper to be well written, thoroughly executed, and the proposed method performs well compared to prior work.  Nice job!

I do have a few suggestions for improvements, primarily to the presentation of the method and results.

1. The appendix is quite short, but includes vital details to understand what the data is.  For example, the fact that openpose is used to extract pose from the video data.  These details should be in the main text of the paper.

2. I didn't understand the "beat coverage" evaluation.  Is "standard deviation of motion"  the euclidean distance between successive poses?  Or something different?  What constitutes a "hit" here -- beat tracking evaluation usually involves a tolerance window to account for differences in analysis parameters when comparing systems (eg, in mir_eval [1]).  It would be helpful to have a bit more detail (eg an equation) here.

3. The data includes genre information, but this doesn't seem to be explicitly reported on except by way of the "style match" evaluation.  Are there differences in performance across styles, or do they all perform comparably?  I mainly ask because some of the input features (eg onset strength) will work better on some genres than other (hip hop vs ballet), and it would be good to have a sense of sensitivity to style in general.


Minor comments:

- On the topic of beat tracking, I'm a little unclear on what exactly is being done here on the audio side.  The main text refers to (Ellis, 2007) for beat tracking, but the appendix refers to (Boeck and Widmer, 2013), which uses a similarly defined (but practically quite different) onset strength function.  It'd be great to check the consistency here and report exactly what's being used in each place.

- Related suggestion, many dance styles depend on the downbeat (bar lines) in addition to beats (usually quarter notes).  It may be worth including downbeat estimations (eg from madmom [2]) as an input feature at some point.

- I think there's a typo in the appendix on audio preprocessing: are features really extracted at 15400 frames per second?

- If you continue this line of work, you might want to check out the recently published (2019) AIST database [3].


[1] https://craffel.github.io/mir_eval/

[2] https://madmom.readthedocs.io/en/latest/

[3] Tsuchida, Shuhei, Satoru Fukayama, Masahiro Hamasaki, and Masataka Goto. "AIST Dance Video Database: Multi-Genre, Multi-Dancer, and Multi-Camera Database for Dance Information Processing." In ISMIR, pp. 501-510. 2019.




Meta-review: Reviews for this paper were quite mixed (7744), and none were exactly borderline. All reviews were detailed and informative, as was the rebuttal. The main criticisms were (1) lack of detail in the experiments, and some missing evaluation (2) missing related work, (3) overall lack of polish (mentioned among positive reviews too), and (4) some unsubstantiated claims. Positively, reviewers praise the novelty, dataset, the demo, and some reviewers found the experiments mostly convincing.

Ultimately this is still a borderline decision. The rebuttal does appear to address many of the claims about missing evaluation, and the complaints about polish can be easily addressed. I think the unsubstantiated claims are reasonably rebutted too. Related work doesn't seem to be addressed in the rebuttal.

Decision: poster-presentations