You Never Stop Dancing: Non-freezing Dance Generation via Bank-constrained Manifold Projection
Authors: Jiangxin Sun
Chunyu Wang
Huang Hu
Hanjiang Lai
Zhi Jin
Jian-Fang Hu
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3D dance generation
music conditioned human motion prediction

Abstract: One of the most overlooked challenges in dance generation is that the auto-regressive frameworks are prone to freezing motions due to noise accumulation. In this paper, we present two modules that can be plugged into the existing models to enable them to generate non-freezing and high fidelity dances. Since the high-dimensional motion data are easily swamped by noise, we propose to learn a low-dimensional manifold representation by an auto-encoder with a bank of latent codes, which can be used to reduce the noise in the predicted motions, thus preventing from freezing. We further extend the bank to provide explicit priors about the future motions to disambiguate motion prediction, which helps the predictors to generate motions with larger magnitude and higher fidelity than possible before. Extensive experiments on AIST++, a public large-scale 3D dance motion benchmark, demonstrate that our method notably outperforms the baselines in terms of quality, diversity and time length. 

You Never Stop Dancing: Non-freezing Dance
Generation via Bank-constrained Manifold Projection
Jiangxin Sun'
Chunyu Wang Microsoft Research Asia chnuva@microsoft.com
Sun Yat-sen University Sunjx5@maiin wyy....sysuuuu..a.ammmmmmm
sysu. edu. en
Huang Hu
Hanjiang Lai
Peking University tonyhu@pku. edu. en
Sun Yat-sen University Laihanj3@mail. sysu. edu. cn
Zhi Jin
Jian-Fang Hu
Sun Yat-sen University
Sun Yat-sen University hu hujf5@mail- sysu. edu. cn
j jizh226mail. sysu. edu. en
Abstract
One of the most overlooked challenges in dance generation is that the auto- regressive frameworks are prone to freezing motions due to noise accumulation. In this paper, present two modules that can be plugged into the existing mod- els to enable them to generate non-freezing and high fidelity dances. Since the high-dimensional motion data easily swamped by noise, we propose learn a low-dimensional manifold representation by an auto-encoder with a bank of latent codes, which can be used to reduce the noise in the predicted motions, thus preventing from freezing. We further extend the bank to provide explicit priors about the future motions to disambiguate motion prediction, which helps the pre- dictors to generate motions with larger magnitude and higher fidelity than possible before. Extensive experiments on AIST++. public large-scale 3D dance motion benchmark, demonstrate that our method notably outperforms the baselines in terms of quality, diversity and time length.
Introduction
Dancing to music has been one of the most popular art forms since ancient days. It can vividly express human's 's emotions and fulfill social communications even before symbolic languages came along. Nowadays, there are booming number of people sharing their dance videos the popular media platforms such as YouTube and TikToo, which drives the strong need for automatic AI choreography help users create their dances. The task is related general human motion 39 42 except that it poses new challenges: i) dance generation needs to produce high-fidelity motions for about three minutes cover a music which is much longer than that in general motion prediction; ii) dance generation needs to handle more diverse and stylistic motions (e.g., ballet Jazz, hip-pop, etc.). The high spatio-tempora complexity requires more expressive models so as to generate
prediction [2 31 38
dance motions.
Thii work was done when Jiangxin Sun was an intern at Microsoft Research Asia. 1 Hu is also with Guangdong Province Key Laboratory Information Security Technology, Guangzhou, China and Key Laboratory of Machine Intelligence Advanced Computing, Ministry of Education, China.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

RefineBank
Music Feahure Extractor
Erooder
Daxxder Doa
Cross- Modall Predictor?
Dance Reature Extracton
Docoder
Manifold
TranstBaakk
Dra
Bank
Encoder
Past- Fubure
Eyg
Bank
Auto Regression for Inference
Figure 1:
The motion prediction framework. first uses the past motions to query TransitBank to retrieve the priors about the future motions, which are fed to the predictor to get high-fidelity motion predictions. Then RefineBank refines the predictions to reduce error accumulation.
The state-of-the-an methods follow a cross moossl prediction framework where the future motions are predicted based the past motions and music in auto-regressive way [II 211 30J. However, the generated motions are prone to freezing and converging to small- -aagitudde motions after only several seconds. The main reason is that the prediction error will accumulate in the generation process, and eventually can not be handled by the neural predictors. Besides, motion prediction suffers from the huge uncertainty and ambiguity because of the high spatio-temports complexity of the task. As a result, the models tend to predict mean poses [42 instead of aage-maa and high-fidelity motions if there
informative priors about the future motions.
In this paper, we present two modules that can plugged into existing auto-regressive models to achieve non-freezing sare-magnitude motion generation. Figure iillustrates overview. Firstly, to prevent from error accumulation, we present RefineBank to learn a fow-dimensional manifold representation for the high nigh-dimensional motion data. It equips auto-encoder with bank of latent codes to tightly constrain the manifold to close to the ground-trun truth (GT) motions and meanwhile remove the noise in the predicted
far from the ones with noise. This representation allows motions by projecting them to the learned manifold as illustrated in Figure 2 With RefineBank, the baseline method [20) already generate full choreography for complete musics in the dataset without freezing motions. Secondly, inspired by the fact that most dances can be coarsely constructed by number of basic motion segments, we present Transi/Bank to learn and memorize the frequently used <past, future> motion dynamics on top of the manifold. Given past motions, provides explicit priors about the future motions reduce the uncertainty and ambiguity prediction, which can effectively facilitate the higher-fidelity dance motion generation with larger magnitude than before.
We conduct extensive experiments evaluate approach on the AIST+ dataset [20). Not only does our method notably outperform the baselines on the existing metrics, but also shows better results newly introduced freezing rate metric. In addition, the user study indicates that people have obvious preferences toward the dances generated by our method. Our contributions are summarized evidence that the reddiction-based methods can generate
follows: 1) This the first time long-term dance motions on AIST+,, which paves the way for full choreography for entire music: 2) We present RefineBank and TransilBank which can be plugged into most motion-prediction based methods to achieve long-term non- con-freezing dance generation; 3) We introduce new metrics that can quantitatively evaluate the freezing situations in the motion sequences.
2 Related Work
Prediction-based Methods This line of work treats dance generation as motion prediction
problem and has achieved promising results. A number of network architectures have been proposed
including CNNs 16). RNNs 11 32 35|41] GCNs 4 26 36), GAN (30 and Transformers
[2 18-50) 20). Several works focus on the alignment of motion and music. For instance, Sun al. (30]
and Li et al. [8 use a classifier to test the authenticity of the predicted motion conditioned on the

---

du de de
Manifold Bank Byr
Transformer Encoder Fan
Linear projection
PE
Lineer projection
Transfomer Decoder Dasder
2, By soflmaa/B
PE
dz.d2.-..d dg
Figure RefineBank takes motion sequence with frames as input, projects them to low- dimensional manifold, and reconstructs the sequence with minimum noise.
music. Zhang et al. 41 and Huang et al. [12] learn the dance style embeddings to provide prior information to the predictor. Few works have studied the freezing problem. Huang et al. [1 propose a curriculum learning strategy to bridge the gap between training and inference by alternately feeding predicted and GT motions to the predictor. Li et al. (20) present future-n full-attention to replace the traditional shift-by-1 casual-attention to leverage the temporal context. However, the predicted motions are still prone to freezing after several seconds. Our approach is also a prediction-ball method. But different from the previous methods, we present two plug-in modules that can achieve longer-term non-freezing motion generation.
Retrieval-based Methods Some earlier works compose complete dance by retrieval [14 24 28) They select the closest predefined motion segments in pre-built database based music, and construct a sequence with the proper transition routines. Lee et al. (17) and Ye et al. (37] use deep networks generate future motion segments from the input music and past motion segments. Duan et al. 3 propose an attention-
MI.P to translate music phrases to motion segments. Chen al. [ propose predict dance sequences through traversing the node transition routines on motion graph and introduce the choreography-oriene sequences. However, they cannot generate new motions beyond the database. Our approach is also inspired by the retrieval-ban dynamies for the predictor. The core difference is that the bank is automatically learned from data without manual efforts and our approach can generate new motions beyond the database.
constraints to compose the final dance motion
methods in that we construct a bank/database to learn common motion
3 Preliminaries
Given the past and future music features m... || m4 1-4+-. and the past dance motions di.t. the task aims to predict the future motions de+1:t+K- The input music features E TR35 are obtained from Librosa [23) consisting of 1-dim envelope, 20-dim MFCC, 12-dim chroma, 1-dim one-hot peaks and I-dim one-hot beats. The motions are represented by SMPL (2I) model, which consists of parameters referring to body shape, human pose and translation. The shape parameters are dance-irrelevant values which mainly capture the expansion/shrink of human body such taller shorter, we only consider the prediction of parameters respecting human pose and global translation in d, 221 We follow the state-of-the-an dance generation architecture [20) as shown in Figure I has a music feature extractor E,,., dance feature extractor Eg and cross-modal predictor P.. The auto-regressive prediction process can be formulated as:
d d+ = Pa(Em (mmmmmma)) Ea(d Ea(dia))
(1)
Although previous works such as 20 have attempted to improve the long-term generation performance, the predicted motions are still prone to freezing after only several seconds.

---

Pot Firare Bani
sim;K, g) (K)Fg
Post Dast
Fimmosstoo
V softmax(sim(K,, g1)
Encodlr
Decodlr Pya
TIIMT prior representaiin ambodding future motion dynamies
past
Figure 3: TransitBank: Given first half of the motion sequence, it produces a prior representation for the second half of the sequence by computing the weighted average of the values of the bank items to reduce the uncertainty and ambiguity motion prediction.
4 Method
The dance motions are believed to lie on low-dimensional manifold since the body parts are highly correlated (9) We take advantage of the nice property and present RefineBank (RB) to reduce the noise in the motions. It learns an auto-encoder with a bank of latent codes to tightly represent the compact motion manifold. By projecting noisy or corrupted motions the manifold, can remove the noise in the motions, which prevents eπror accumulation in the auto-regressive generation. Secondly, we propose TransitBank (T B) on top of the learned manifold, which maintains past-future motion dynamics bank to provide explicit priors about the future motions. The priors narrow down the motion prediction space which facilitates high-fidelity motion generation with large magnitude. Mathematically, the prediction process can be formulated as:
d+11+K = RB( Pa( (m)4kk). Ea(dia), (B()))))
4.1 RefineBank
Figure 2 shows the components of RefineBank which has an encoder ERB, decoder DRB, and manifold bank Ba E RRCNN The bank Ba has N learnable latent codes of dimension C' which span the low-dimensional dance manifold. As will be described later, the bank is learned from the GT motions so can be interpreted as prior probabilistic distribution where the motion data with noise will have the small likelihood.
For (noisy) motion sequence dı produced by the predictor, we first transform each of them latent feature by a transformer- based encoder ERB. Concretely, we uniformly sample n features at to represent the sequence. Then we project each 2, to the manifold represented by the bank remove the prediction error. Specifically, for each latent feature zt, we compute the similarity scores between 2; and the latent codes in Bar, and use the learnable weights to project 2, onto the manifold get ž,:
different time steps z = z {00, - R- ....) }
2. B softmax (B (,))
(3)
The projected latent features ) will be fed the transformer-basee decoder DRB to generate the motions that we expect to have little noise.
Compared with previous bank-based autoencoder approaches for image generation such VQ- VAE (33), we have two unique designs in our RefineBank for dance generation. The first is to use soft-assignment instead of hard nearest- neighbor assignment prevent simply repeating existing dance movements and improve the diversity of the generated dances. The second is that we follow the multi-heat implementation in transformers encode the input motion sequence with several different latent features such that more details can be preserved.
4.2 TransitBank
The high spatio-temporal complexity of the motion space increases the uncertainty and ambiguity of motion prediction. Hence, the motion predictors are prone to generating small-magnitude motions. To address the problem, we present TransitBank which is inspired by the fact that there exist number of basic short motion segments that are frequently used in many dances. TransitBank aims to exploit such cues to estimate the prior distributions for future motion prediction.

---

Manifold Bank dance manifold D=
Past Future Bank
Isey R (past) valuu (funue)
Decndkr
Enankk EM
dance morrfold D
[kny (pasti | val (futtre)
danon marifald e
Iong (past)) jralue un (futume)
Figure 4: Constructing the past-future bank from the manifold bank.
Concretely, TransitBank works by dividing a motion sequence into past and future sub-sequences and storing them past-future pairs. Then given the past motions input, it queries the bank and reads the corresponding future motion dynamics from the bank. This prior information is fed to the cross-modal predictor reduce the uncertainty and ambiguity in motion prediction.
Figure 3 shows the structure of TransitBBank. consists of an query encoder ETp, past-future bank BpF, and read decoder DYB- The past-future bank BpF = { (K, V) V E RCXN } is directly constructed as the <past, future> extension of manifold bank B M. As for ETB and DTB. simple transformer structure for encoder and decoder, respectively. To avoid information leakage in case that the cross-modal predictor directly copies the motions in TransitBank, we design the query-read process by using the attention mechanism rather than finding closest effectively blur the future motions. Specifically, we encode input motion d1 via ErB and take the average pooling of all outputs the query vector q. Then compute the similarity between q and key features K following:
which will
sim(K, q) = =(K)²
(4)
In the read process, we compute the future motion prior r as the weighted average of the corresponding values V:
V softma q)).
Finally, decode the vector via DTB and feed it to the cross-modal predictor additional tokens.
4.3 Model Learning
This section describes how we learn the two banks from training data. It is worth noting that the two banks are not independent. Instead, TransitBank is a <past, future> extension of RefineBank (i.e., mapping motion prefixes to suffixes). In the following part, we first illustrate how we learn the bank in RefineBank and then describe how to derive the one in TransitBank. Finally, we describe the training optimization strategy.
Bank in RefineBank We first initialize the manifold bank B by clustering to obtain a proper initialization where items are representative and widely used in GT motion. Specifically, a number of motion segments by applying a sliding window with length to all training motion sequences. Then AP clustering (5) is used to obtain number of cluster centers from the segments. We apply transformer-basee encoder F.M to encode each cluster center d',, and compute the average of the encoding outputs of the frames to initialize one bank elements E B
obtain
After initialization, we train the encoder EM,, decoder Dar and the bank B to reconstruct all motion segments d using the method described Section 4.1 The only difference that we project the latent feature z to its closest item b of B as:
arg min beB
||
b1
(6)
Bank in TransitBank Constructing the fast-future bank BrF from the manifold bank Bas straightforward. As shown in Figure 4 for each element b BM, decode it to motion sequence
5

---

d1 via the learned transformer decoder DM. Then we divide the sequence into two parts: d1a/2 and d1/2+1a, which are fed the encoder EM obtain the latent codes representing the past (K) and future (V), respectively. In that way, we obtain P = (K, V)(K, V RCCN].
We train our model in three stages to ensure that the learned manifold bank can accurately reconstruct the GT motions. In the first stage, we just follow the implementation in VQ-VAE 33 and optimize the manifold bank by minimizing the following loss:
Optimization Strategy
BaaiifodBaank d1 dui ||sg[=]
B|| sg( 2
The first term minimizes the reconstruction error. The second part is the "item loss" 33 to update items in manifold bank, where sg denotes "stop gradient". This objective function moves the items close to the outputs of the encoder. The sg[-] operator is implemented by the identification function during forward computation with zero partial derivatives. The third part is "commitment loss" [33] to ensure the output of encoder commits to an item and the value of / is set as 0.2, empirically. We optimize the encoder with the first and third loss terms. The bank items are updated with only the second loss term and the decoder is trained with only the first term. Once finalize the manifold bank, we can compute the past-future bank. Note that we do not update these two banks in the following stages.
In the second stage, we train the encoder and decoders RefineBank to reconstruct all motion sequences of the training set with only reconstruction loss.
CRefneBank = ||āu: d
(8)
In the third stage, we train the whole framework in an end-to-end manner with the L2 loss between predicted motion and GT motion.
= |d - d++1++K 2
Preediction
(9)
5 Experiments
5.1 Experimental Settings
Dataset We evaluate our method the largest AIST++ (20) dance dataset that contains 60 music pieces belonging to 10 dance genres. In total, there are 992 3D pose sequences at 60 FPS. Following 20), we use 952 samples for training and the rest 40 for evaluation.
Implementation Details The model takes 240 frames of music and 120 frames of motions as input and predicts the next K 20 frames of motions. We the same network structures for the music extractor, motion extractor and cross-modal predictor as FACT [20). For the encoders and decoders in RefineBank and TransitBank, we use transformers with 4 layers and 10 attention heads with 2048 hidden size. The number of items in manifold bank and past-future bank is 256 and each item is 2048-dim latent vector. The input to RefineBank and TransitBank frames. Since K < I, we concatenate the previous - K frames of motions with the predictor output feed RefineBank. We construct separate manifold bank and past-future bank for each dance genre. In the first training stage, we adopt Adam optimizer [15) with learning rate of 1 x 10-4 to train the manifold bank for 50 epochs. In the second training stage, we pre-train the RefineBank using Adam optimizer with learning rate of 1 x 10-1 for 25 epochs. In the third stage, we train the whole framework with Adam optimizer for 55 epochs. The learning rate starts with 10-4 and decreases to {1 x 10-5, 1 x 10-6) after (30, 90) epochs, respectively. The whole training process takes about four days on four NVIDIA GeForce RTX 2080Ti GPUs
motion sequence with / 120
The previous works mainly evaluate the dance generation results from three aspects: quality, diversity and alignment. Following [20), we compute FID (8) (Frechet Inception Distances) on the kinetic features (denoted as FID;;) and geometric features (denoted as FID,), respectively, to measure quality. We use the fairmotion toolbox (6) to extract the features. For diversity, we compute the average Euclidean distance in the kinetic (denoted Dist,) and geometric (denoted as Dist,)
Metrics

---

Table 1: Comparison to the state-of-the-ar methods on the + dataset.
Quality
Diversity Dist, t Dist,
Align BeatAlign
User Study
Method
FID&
FID, Arese t
Taass t Freeze
Win Rate
GT
3.28
1.16
18.7%
9.06
7.31
0.292
31.7%
Li al.
19
86.43
43,46
1.02
59.0%
6,85
3.32
0.232
95.8%
DanceNet
43
69.18
25.49
1.25
0.80
46.8%
2.86
2.85
0.232
90.8%
Revolution
m
73.42
25.92
3.52
4.87
0.220
84.2%
FACT (20)
35.35
22.11
1.33
1.07
39.0%
5.94
6.18
0.241
86.7%
Ours
25.96
13,42
1.64
1.36
29.6% %
7.68
6.59
0.249
Table 2: Ablation study of RefineBank and TransitBank.
Quality Aees
Diversity Dist,, t Dist,
Alignment BeatAlign
Method
FIDx 4 FID,
Arrans
Freezing
Baseline
35.35
22.11
1.33
1.07
39.0%
5.94
6.18
0.241
RefineBank
28.67
16.38
1.53
1.15
32.1%
6.65
6.34
0.246
TransitBank
31.24
19.18
1.49
1.31
34.3%
7.42
6.47
0.245
Ours
25.96
13.42
1.64
1.36
29.6%
7.68
6.59
0.249
feature space across the generated motions. For dance-musie alignment, we adopt Beat Alignment Score in 20) to compute average distance between every kinematic beat and its nearest music beat. Since the freezing problem is largely overlooked previously, there are no metrics available to evaluate. We propose to compute the average values of the temporal differences of the pose and translation parameters in the whole sequence, which are termed as Aesss and Araass respectively. In addition. we also calculate the Freezing Rate of each sequence. We divide a sequence into non-overiapping sub-sequences of 60 frames, and for each sub-sequence, if Apose < A and Arans < A Itans where A s is predefined threshold statistically derived from the training set, we regard it as a freezing sub-sequence. Then we compute the percentage of freezing sub-sequences.
5.2 Comparison to the State-of-the-arts
We compare our approach number of recent methods including Li et al. [19], Dancenet 43)
DevoeRevolution [11]. and FACT [20). Our approach employs the same structure as FACT except that has RefineBank and TransitBank. For each music, generate a motion sequence with 1200 frames
(20 seconds). The experiment results are shown in Table Since DevceRevolution [1]] predicts 3D keypoint positions, we cannot compute the SMPL-based freezing metries. The approach in [19] does not predict the translation parameters, As shown, our approach outperforms the state-of-the-an methods on all metrics. Interestingly, the Araas of our method is even larger than that of GT. We visually compare our generated motions and GT motions and find that it is because GT often have stationary poses at transition moments. By contrast, seems difficult for learning- based methods to predict stationary poses.
We present detailed analysis for one sequence in Figure 5 As we can see, the motion and translation changes of FACT gradually decrease to a small number suggesting that the freezing situation occurs. In contrast, the numbers of method always above those of FACT and do not freeze. The results validate that our proposed bank-based manifold projection and past-future dynamic priors indeed improve the quality of the generated motions.
User Study We conduct user study to investigate how people think of the dances generated by our method and the other ones. We invite 20 participants and each participant asked watch 30 pairs of comparison videos. Each pair consists of our dance and one competitor's 's generated with the same music. We ask each participant determine "which person dancing better to the music" and provide the statistics in Table (last column). Our approach significantly outperforms the other methods in user study. We can keep at least 84.2% win rate to the SOTA methods including DanocRevolution and FACT. Notably, we achieve 31.7% win rate compared to GT motions. We
7

---

FACT
Seed Motion
0 urs
Ours uscan prise chanage: 1.73
Oera mose trasolation chonge:1.52 Fart moos translation chingena.7
Fact axean pass shanger d6
Oass freeding perceuta FACT trewring procentedd
Fresiing
Pow
Iramn
Prrams
Prame-level Pose Change
(b) Frame-level Translation Change
Frame-Jevel Freezing Determiiaaion
Figure The top row shows the motions generated by FACT and our method for the same music. The bottom provides frame-level statistics for the two sequences. In figure (c), 0 represents non-freezing and means freezing. Best viewed in color.
1.-
34.
Baseline
Oith RefineRank
(c) With RefineBank TransitBank
Figure 6: I-SNE visualization of the generated dances. Each dot represents a 3D pose and different colors represent different genres of the music used to generate the poses. Best viewed in color.
also collect detailed feed-backs and find that our generated dance is generally thought to be more diverse and non-freezing, The main problem with FACT is that the motions freeze frequently while the problem with DanceRevolution is that the motions are unnatural. Compared to GT motions, ours thought to lack suitable transition motions and precise beat alignment which is general problem faced by most prediction-bast methods.
5.3 Ablation Study
Ablation Study of RefineBank and TransitBank The experiment results are shown in Table 2 Our first observation is that adding RefineBank to the baseline notably improves Aoose meaning that the method reduces the chance of getting freezing motions. Meanwhile, the improvement also brings benefits to dance quality and diversity. However, the variation of the global positions of the dancers, i.e,, Trans is only slightly improved. This expected since RefineBank only guarantees that the refined motions are on the manifold. In contrast, adding TransitBank notably improves Araas- Meanwhile diversity metrics are also notably improved. The results suggest that by exploiting motion dynamic priors the method can predict high-fidelity and diverse motions with large magnitude instead of mean poses. Finally, the two modules are complementary to each other and combining them will further improve the results on all metrics.

---

Table 3: Comparison of the bank-based AE and other options.
Quality Aoss
Diversity
Align
Method
FID
FID, J
! Trass
Freezing
Dist, t Dist,
BeatAlign
AE
29.85
17.64
1.52
1.32
33.5%
7.48
6.49
0.245
VAE
29.67
17.05
1.53
1.32
33.1%
7.49
6.50
0.245
Bank-AE (Discrete)
27.48
15.19
1.60
1.34
30.4%
7.45
6.48
0.247
Bank-AE (Ours)
25.96
13.42
1.64
1.36
29.6%
7.68
6.59
0.249
Table Evaluation on the number of latent features.
Quality
Diversity Distz
Alignment BeatAlign f
FID
FID,
Aros f Arias
Freezing
Disty
n=1
27.94
15.61
1.59
1.34
31.2%
7.51
6.52
0.246
n=2
26.72
14.78
1.62
1.35
30.6%
7.63
6.57
0.248
n=3
26.15
13.95
1.64
1.36
29.9%
7.66
6.59
0.249
25.96
13.42
1.64
1.36
29.6 %
7.68
6.59
0.249
n=
26.67
13.88
1.63
1.35
30.1%
7.72
6.63
0.248
Table 5: Evaluation on the number of bank items.
Quality
Diversity Dista
Alignment
FID;
FID, 9
Arwe 1 Atran 1 Freezing
Dist, 1 BeatAlign f
N=32
29.74
17.34
1.53
1.32
32.9%
7.50
6.49
0.245
N-64
29.14
16.98
1.54
1.33
31.7%
7.56
6.52
0.246
N=128
27.31
14.86
1.61
1.35
30.2%
7.62
6.54
0.248
N=256
25.96
13.42
1.64
1.36
29.6%
7.68
6.59
0.249
N=512
26.39
14.07
1.63
1.35
29.9%
7.64
6.56
0.248
Table 6: Evaluation of module applicability to other models for dance generation.
Table 7: Evaluation of our module applicability to other models for motion prediction.
Quality FIDx
Diversity
Reconstruction error
!
Method
FID, 1
Dist,
Dist, t|
Method
10
20
30
Revolution
73.42
25.92
3.52
4.87
PHD (40)
64.4
67.1
98.5
125.9
+ Our Modules
50.83
23.75
5.13
5.69
Our Modules
62.9
64.3
77.2
93.5
117.8
We visualize the poses in the generated dances using t-SNE (34] in Figure 6 We can see that the dances generated by the baseline tend to be mixed together with other genres. means the motions may have lower fidelity, smaller magnitude and lack uniqueness. We think this is caused by the freezing problem and the high spatio-temporal complexity of the prediction space. In contrast, adding RefineBank alleviates the freezing problem which allows the poses to preserve the high-fidelity details and to be differentiable from other dances. Further adding TransitBank allows to generate more diverse dances with larger motion magnitude.
Bank-based Auto-encoder We compare bank-basch auto-encoder with other options such as vanilla AE and VAE. The experimental results are shown Table Our achieves clearly better results than the other methods. This is because the bank of latent codes provide a tight characterization of the dance manifold. The tightness requires that the generated motion sequences strictly follow the dance styles.
auto-encoder
We also compare to a discrete variant of Bank-AE. Different from our current method, it uses closest bank item instead of convex combinations of the neighboring items to reconstruct each datum similar VQ-VAE (33]. We can see that also reduces the the freezing rate. However, the quality and diversity metrics are notably worse than our method. This is because the discrete variant has

---

limited capability to reconstruct data with sufficient accuracy, compared to our approach using convex combinations of multiple bank items.
Number of Latent Features We study the impact of the number of latent features n for representing a motion segment as discussed in section 4.1 'he results are shown Table [ In general, using more latent features improves the prediction performance because it can capture more details. But keep increasing the number may lead degeneration.
Number of Bank Items We study the impact of the number of elements N in the two banks. The results are shown in 5 Initially, increasing the number of elements improves the results. This reasonable because the expressive power of the bank is improved and more details can be preserved after projection. However, keep increasing N to 512 begins to have negative effects, We think this is because using too many items may increase the risk of over-fitting to the small levels of noise in the GT data. In addition, the introduced redundancy may also bring negative effects. Nevertheless, the method is relatively robust to this parameter and achieves reasonably good results when N is between 128 and 512.
Evaluation of Applicability Our proposed two modules RefineBank & TransitBank can be plugged into other prediction-bare based methods to achieve long-term non-freezing dance generation. As shown in table 6 our proposed modules can clearly improve motion quality and generation diversity over the RNN-based prediction method Dance Revolution
Besides the adaption of other approaches for dance generation, we also try to adapt our proposed modules to the more general 3D human motion prediction task. We add our proposed modules to a strong open source method PHD 44).. Following the settings in [40). we report PA-MPJPE (mean error of per joint position after applying Procrustes Alignment [7] in mm through 50 frames and evaluate the performance on a large 3D human motion dataset Human3. 66 [13). As shown in Table
we can observe that our proposed approach can clearly improve the prediction accuracy, which demonstrates that our proposed approach can be used for other tasks related to human motion.
7
6 Conclusion
In this work, we presented two general modules that can be plugged into the existing methods to address the freezing problem in dance motion generation. This largely overlooked problem has limited motion generation to short segments of only several seconds. By reducing noise accumulation and exploiting dynamic priors, our approach can generate motions for at least 30 seconds with 60 FPS, which the maximum music length in current dataset AIST++, without freezing. Our user study also shows that our method has obvious advantages over other methods in terms of quality and diversity. The method paves the way for addressing a more valuable problem of full choreography for entire musics instead of short clips.
Broader Impact We believe that our work has values for not only dance generation but also for more general motion prediction. This benefits areas including media platforms, robotics, and autonomous driving. On the other hand, our method can have negative downstream consequences such as being extended to generate fake videos conditioned on the generated human motions with GANs. The potential limitation is that due to the relatively short duration of music pieces in AIST+ dataset, the test performance is not precise evaluation for long-tern dance generation capability.
Acknowledgments and Disclosure of Funding
Jian-Fang Hu is the corresponding author. This work was supported partially by the NSFC (U1911401. U1811461, 62076260, 61772570), Guangdong Natural Science Funds Project (2000B15151200855) Guangdong NSF for Distinguished Young Scholar 100221515500000 and Development Program of Guangzhou (202007030004), We thank Wenjun Zeng from Eastern Institute for Advanced Study and Dewen Ai from An-Ji Mixed Reality Research Institute for helpful discussions. We also thank Ruilong Li and Siyao Li for sharing their generation results.
and the Key-Area Research
10

---

References
[1] Omid Alemi, Jules Françoise, and Philippe Pasquier. Groovenet: Real-time music-driven dance movement
generation using artificial neural networks, networks, 8(17):26, 2017.
[2] Kang Chen, Zhipeng Tan, Jin Lei, Song- Hai Zhang, Yuan- Chen Guo, Weidong Zhang, and Shi-Min Mii-Min Hu. Choreomaster: saaeoorraph--orientte music-driven dance synthesis. ACM Transactions on Graphics (TOG), 40(4):1-13, 2021.
[3] Yinglin Duan, Tianyang Shi, Zhipeng Hu, Zhengxia Zou, Changjie Fan, Yi Yuan, and Xi Li. Automatic translation of susic-to-dance for in in-game characters.
[4] Joao P Ferreira, Thiago M Coutinho, Thiago L Gomes, José F Ncto, Rafael Azevedo, Renato Martins, and Erickson R Nascimento. Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio. Computers Graphics, 94:11-21, 2021. 2021.
[5] Brendan Frey and Delbert Dueck. Clustering by passing messages between data points. science,
915(5814):972 997,677777 2007.
[6] Deepak Gopinath and Jungdam Won, fairmotion - tools to load, process and visualize motion capture data. Github, 2020.
[7] John C Gower. Generalized procrustes analysis. Psychometrika, 40 1):3333,,, 1975.
[8] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[9] Daniel Holden, Jun Saito, and Taku Komura. deep learning framework for character motion synthesis and editing. ACM Transactions on Graphics (TOG), 35(1):1-11, 2016.
[10] Daniel Holden, Jun Saito, Taku Komura, and Thomas Joyce. Learning motion manifolds with convolutional autoencoders. In SIGGRAPH Asia 2015 technical briefs. pages 1-4. 2015.
[11] Ruozi Huang, Huang Hu, Wei Wu, Kei Sawada, Mi Zhang, and Daxin Jiang. Dance revolution: Long- dance generation with music via curriculum learning. In International Conference on Learning Representations, 2020.
[12] Yuhang Huang. Junjie Zhang, Shuyan Liu, Qian Bao, Dan Zeng, Zhineng Chen, and Wu Liu. Genre- conditioned long-term dance generation driven by music. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4858-4862. IEFE, 2022.
[13] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for human sensing natural environments. IEEE transactions on pattern analysis and machine intelligence, 36(7):1325-1333 2013.
[14] Jae Woo Kim, Hesham Fouad, and James K Hahn. Making them dance. In AAAI Fall Symposiun: Aurally Informed Performance, volume 2, 2006.
[15] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: Ttti:1412..660 6980, 2014.
[16] Kosmas Kritsis, Aggelos Gkiokas, Aggelos Pikrakis, and Vassilis Katsouros. Danceconv: Dance motion
generation with convolutional networks. IEEE Access, 2022.
[17] Hsin- Ying Lec, Xiaodong Yang, Ming- Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming- Hanu--Huann Yang, and Jan Kautz, Dancing music. Advances in Neural Information Processing Systems, 32, 2019.
[18] Buyu Li, Yongchi Zhao, Zhelun Shi, and Lu Sheng. Danceformer: Music conditioned 3d dance generation
with parametric motion transformer. AAAI, 2022.
[19] Jiaman Li, Yihang Yin, Hang Chu, Yi Zhou, Tingwu Wang, Sanja Fidler, and Hao Li. Learning to generate diverse dance motions with transformer. arXiv preprint arX0v:2008.08// 2020.
[20] Ruilong Li. Shan Yang. David A Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++ In Proceedings of the IEEE/CVF International Conference on Computer
Vision, pages 13401-13412 2021.
[21] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black, Smpl: skinned multi-person linear model. ACM Transactions on Graphics, 34(6), oct 2015.
[22] Wei Mao, Miaomiao Liu, and Mathieu Salzmann. Generating smooth pose sequences for diverse human motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
13309-13318, 2021.
[23] Brian McFee, Colin Raffel, Dawen Liang, Daniel P Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science conference, volume 8, pages 18-25. Citeseer. 2015.
[24] Ferda Ofli, Yasemin Demir, Yücel Yemez, Engin Erzin, A Murat Tekalp, Koray Balc,, Idil Kizoglu, Lale Akarun, Cristian Canton-Ferres Joëlle Tilmanne, et al. An audio-driven dancing avatar. Journal on Multimodal User Interfaces, 2(2):93-103, 2008
[25] Mathis Petrovich, Michael J Black, and Gul Varol. Action conditioned 3d human motion synthesis with transformer vae. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11985-1099) 2021.

---

[26]
Xuanchi Ren. Haoran Li, Zijian Huang, and Qifeng Chen. Self-supervised dance video synthesis condi- tioned music. In Proceedings the 28th ACM International Conference Multimedia, pages 46-54, 2020.
[27] Takaaki Shiratori and Katsushi Ikeuchi. Synthesis of dance performance based on analyses of human motion and music. Information and Media Technologies. 3(4):834-88) 847, 2008.
[28]
Takaaki Shiratori, Atsushi Nakazawa, and Katsushi Ikeuchi. Dancing-to-musst character animation. In Computer Graphics Forum, volume 25, pages 449-458. Wiley Online Library, 2006.
[29] Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin. Quan Wang. Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. In Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1 11050-1 11059, 2022.
[30] Guofei Sun, Yongkang Wong, Zhiyong Cheng, Mohan Kankanhalli, Weidong Geng, and Xiangdong Li. Deepdance: music-to-dane motion choreography with adversarial Iearning. IEEE Transactions on Multimedia, 23:497-509, 2020.
[31] Jiangxin Sun, Zihang Lin, Xintong Han, Jian MannFang Hu, Jia Xu, and Wci-Shi Zheng. Action-guided 3d human motion prediction. Advances in Neural Information Processing Systems, 34, 2021.
[32] Taoran Tang. Jia Jia, and Hanyang Mao. Dance with melody: An lstm- autooncoonc approach to music- oriented dance synthesis. In Proceedings of the 26th ACM international conference on Multimedia, pages 1598-1606, 2018.
[33] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation lcarning, Advances in neural information processing systems, 30, 2017.
[34] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[35] Nelson Yalta, Shinji Watanabe, Kazuhiro Nakadai, and Tetsuya Ogata. Wuakly--upervised deep recurrent neural networks for basic dance step generation. In 2019 International Joint Conference on Neural Networks (JCNN), pages 1-8. IEEE, 2019.
[36] Sijic Yan, Zhizhong Li, Yuanjun Xiong, Huahan Yan, and Dahua Lin. Convolutional sequence generation for skeleton-based action synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4394-4402, 2019.
[37] Zijie Ye, Haozhe Wu. Jia, Yaohua Bu, Wei Chen, Fanbo Meng, and Yanfeng Wang. Choreonet: Towards music to dance synthesis with choreographic action unit. In Proceedings of the 28th ACM International Conference on Multimedia, pages 744-752, 2020.
[38] Yuan and Kris Kitani. Dlow: Diversifying latent flows for diverse human motion prediction. In European Conference on Compuder Vision pages 346-364, Springer, 2020.
[39] Jason Y Zhang, Panna Felsen, Angjoo Kanazawa, and Jitendra Malik. Predicting 3d human dynamics from video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7114-7123, 2019.
[40] Jason Y Zhang, Panna Felsen, Angjoo Kanazawa, and Jitendra Malik. Predicting 3d human dynamics from vidco. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7114-7123,
2019.
[41] Xinjian Zhang, Yi Xu, Su Yang, Longwen Gao, and Huyang Sun. Dance generation with style embedding: Learning and transferring latent representations of dance styles. arXiv preprint arXiv:2104. 14802, 2021,
[42] Yi Zhou, Zimo Li, Shuangjiu Xiao, Chong Hc, Zeng Huang, and Hao Li. Auto-conditionel recurrent networks for extended complex human motion synthesis. In International Conference on Learning Representations, 2018
[43] Wenlin Zhuang, Congyi Wang, Jinxiang Chai, Yangang Wang, Ming Shao, and Siyu Xia. Music2dance: Dancenet for dniven dance generation. ACM Transactions Multimedia Computing, Communica- tions, and Applications (TOMM), 18(2):1-21, 2022.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See section
6
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
12

---

(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or URL)? [No] Code and
models will be released upon acceptance to ensure the reproducibility.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See section 5.
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] Results reported in this paper are the mean results for three repeating experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See section 5.
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See section 5.
(b) Did you mention the license of the assets? [Yes] We provide the license in the supple- mental material.
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes] See section 5.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [No]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes] Considering that user study can be completed
in less than half an hour, offer each participant payment of 10 dollars.
13


Reviews:
[{"review_id": "vctcErh9piq", "reviewer": "Reviewer_GKyZ", "summary": "The paper proposes two new modules to improve the motion generation capabilities of auto-regressive networks. The designing of modules lies in the idea of using a bank of learned features, which are used to reduce the noise in prediction motions. Both modules were tested in the task of dance motion generation. The experimental results in the AIST++ dataset show the benefits of using the proposed modules. It is worth mentioning that the modules seem to be very useful in other applications where motion generation is required.", "questions": "1. Have you tried to train the network without applying the stage-wise strategy? What would be the performance of the methods?\n2. For the sake of reproducibility, kindly consider explicitly saying the list of audio features used to train the network.\n3. After training and creating each feature bank, are the weight frozen when training the whole network?\n4. Regarding the use of one bank for each genre. Have you tried to use banks trained with all genres? I think the results in this scenario could bring interesting insights to the community.\n5. Another question related to the one bank for each genre. Why not condition the bank to the music style?\n", "limitations": "Yes. The limitations and broad impact are well discussed in the conclusion section.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths: \n    - I enjoyed reading the paper, which is well written, and structured. It is easy to read, understand, and the tables and figures are illustrative.\n    - Incorporating representation banks with learned features to leverage the learning of motion generation is a very interesting and well-formulated idea. \n    - The experiments are done thoroughly. The user study, the experimental results with the baselines, and the ablation study are in great detail and defend the finding well.\n\n\n- Weakness:\n    - I have two minor concerns. First refers to the lack of motivation in the designing choices, such as the stage-wise strategy to train the network. Clear seems to be a good choice, but why is that in the case of dance motion generation.\n    - Another possible weakness would be the limitation of requiring training the features bank for each genre. This limitation and designing decision could be better discussed in the paper since it limits the practical use of the approach.\n\nIn conclusion, the proposed method is very interesting, and I believe this is a solid paper that deserves to be communicated.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "b7eTCqPVp7", "reviewer": "Reviewer_1dJr", "summary": "A common failure mode of human motion or dance generation models (which aim to synthesize parametrized human pose sequences, possibly conditioned on music), is that when attempting to predict long sequences of motion, they can freeze. Essentially, due to accumulated prediction uncertainty, the model eventually decides to predict a static, non-moving pose, as a way to minimize prediction error. This has been raised in a number of papers, and there are a few ideas on how to solve it, but no clear definition or metric exists.\n\nThe authors address this by adapting the method, dataset, and evaluation from the FACT [18] paper, and adding two modules designed to eliminate freezing, as well as proposing a new freezing metric.\n\nThe modules, RefineBank, and TransitBank, are based on attention-style lookups from learned memory banks. Specifically, RefineBank is designed to take a predicted motion segment, and clean it up by projecting it to and from a low-dimensional motion manifold. Essentially use a transformer encoder to the pose input sequence, do memory lookup for each pose from this learned pose bank, then decode the sequence. TransitBank uses a similar technique to propose extensions of an initial pose sequence based on a learned memory bank that maps motion prefixes to suffixes.\n\nThe new freezing metric is based on how often the pose delta or global translation delta between frames is less than a threshold determined by analyzing the ground truth data. The proposed model improves upon FACT in standard FID, Diversity, and beat alignment methods, while doing significantly better at presenting freezing, and is preferred in a human quality assessment study.\n\n", "questions": "Table 1 copies the numbers from [18] Table 2, but misreports two numbers: for FACT, FIDg is 12.40 and Distg is 5.30. Using the numbers from [18], \"Ours\" would no longer surpass the original FACT on FIDg. Could the authors please explain?\n\nHow does the proposed method for presenting freezing interact with FACT's \"future-n full-attention\" scheme, also designed to prevent freezing? Do RefineBank & TransitBank eliminate the need for future-n full-attention?\n", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Quality & Clarity:** The paper is clearly written and easy to follow. In particular, the diagrams in Figures 1-4 were well done, and very helpful for understanding sections 4.1-4.3. The explanations are sound, with two exceptions detailed below.\n\nLine 84: SPML should be cited and explained. Also the phrase \"Since shapes\" should be explained more clearly, to convey that it's the shape parameters of the individual dancers you're discarding. \n\nLine 182: \"We construct a separate manifold bank and past-future bank for each dance genre.\" This implies that, at test time, the model is informed which dance genre should be used for generation. This is strictly more information than is used in FACT [18], potentially making the problem easier. The need for per-genre memory banks and the quantitative advantage obtained by doing so should be addressed further.\n\nLines 210-213 \"Interestingly, the ∆ Trans of our method is even larger than that of GT. ...\" This explanation highlights a problem with the definition of the delta trans, delta pose, and Freeze metrics. The goal shouldn't be to maximize delta pose & delta trans, and minimize Freeze... instead it should be for the predicted model to hit the same values for each sequence as observed in the ground truth.\n\nFurthermore, I cannot square the claim that \"it is difficult for learning-based methods to predict stationary poses due to the lack of sufficient data\" with the observation that the \"GT often has stationary poses at transition moments\". This suggests that there *is* sufficient stationary data, and the models are not capturing it, or specifically designed to never freeze even though occasional freezes are desirable.\n\n**Originality & Significance:** This paper is an extension of the FACT model from [18] with a small enhancement based on relatively known techniques (transformer memory) and new evaluation metric, applied to the same AIST++ dataset. As such the originality and significance is very minor. Perhaps if the memory bank methods were demonstrated to be more widely useful across a variety of human motion tasks or models, the paper would be more significant. As is, this does not meet the bar for publication based on significance and originality.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "b6PwI4Q34J5", "reviewer": "Reviewer_HLRW", "summary": "The authors look into the problem of freezing motions in dance generation frameworks.  They propose RefineBank and TransitBank modules that can be plugged on existing methods to generate high fidelity dance movements without freezing after a few frames. Refine Bank learns a low dimensional manifold representaton of motion data close to the ground truth to prevent error accumulation. TransitBank learns a motion prior for the future poses to reduce ambiguity in long-term motion prediction.The authors evaluated their method on the AIST++ dataset and outperform certain baselines. The authors also propose a new metric which measues rate of freezing sub-sequences in movements.", "questions": "- How is the commitment loss different from the item loss (Eq. 7)?\n- How is the Win Rate interpreted in the last column of Table 1?", "limitations": "- Since this is an incremental work on the existing architecture of FACT [18], it is heavily reliant on the performance of the baseline FACT model. It is unclear if or how the proposed modules can be combined with other baseline models such as [10, 17, 38].\n\n- It is also unclear if the proposed modules can be generalized to any genre of dancing.\n\n- Qualitative results also produce some self-intersections of the limbs, which is problematic for such synthesis problems.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The authors tackle a challenging problem for long-term dance motion generation and synthesize diverse and natural dance movements with respect to music. The presented qualitative results and user study results indicate the proposed approach is promising.\n- Using the bank-based auto-encoder modules in the architecture to learn a compact motion manifold is technically sound.  \n- The authors provide ablations for various experiments to strengthen their claims. \n- The paper is overall clearly written and easy to follow.\n\nWeaknesses:\n- I am not clear why the three-stage optimization is required. Why is the Manifold Bank trained separately from the Refine Bank encoder and decoder?\n- How are the different loss terms being used in the first stage of optimization (Eq. 7)? Ablations for the use of multiple losses are missing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "TKYR2CJQyBW", "reviewer": "Reviewer_53W8", "summary": "This submission specifically focuses on preventing the generation of frozen motions in long-term motion generation task.\nTo achieve this goal, the latent code (z^hat) used to generate the final motion is represented as the linear combination of codes in a learned codebook (termed as feature bank in this submission) as VQ-VAE [29]. And the weight of every code (in the codebook) is determined by the similarity between the original latent code (z) and the code.\nAnd a TransitBank is proposed to \"retrive\" future motion features according to the past motion features.", "questions": "The novelty; More thorough evaluation and analysis; Much improved writing.", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Major weaknesses**\n\n- The description is not clear enough and I could not understand the exact operations of the proposed method.\n\nL94, L156: cite the VQ-VAE [29] paper. What is the difference between the proposed RefineBank and VQ-VAE? This is very important and should be discussed as thorough as possible. Maybe you use the weighted sum of the codes while they use a certain code? Personally, I do not think this is a good choice. Pros and cons of these two choices should be evaluated. BTW, why do you need to do a clustering in L143-144? And what is the E_M, is it the same as E_RB?\n\nSince there is an average operation when calculating b^i (L145), so I guess the network extract a latent code for every frame (but with a temporal receptive field), right? Then what are the Ks/Vs in the past-future bank, I did not find the T dimension but it's said that \"TransitBank is a temporal extension of RefineBank\" in L139? Frankly, I did understand the exact use and learning related to TransitBank (Sec. 4.2 & 4.3). Many operations need to be clarified.\n\nBTW, try not to name both network module (RefineBank) and tensor (manifold bank) as banks. It is a little confusing. Personally, I would recommend using code (codebook) or word (vocabulary) to refer to the latent feature (the collection of the latent features).\n\nHow does the music feature (Fig. 1) affect the final output? Is it working? Some dedicated experiments should be conducted about this part.\n\nWhy does FACT results look that bad even for the first few seconds?\n\nL119: \"basic short motion segments\": What are they and how do you find them?\n\nUser study: Are there any professionals conducting the user study (L220)? Better in what aspect (L222)? Normally, people will compare several specific and clearly described aspects in a user study.\n\n**Minor weaknesses**\n\nL175: How many sec is 240 frames of music?\n\nL197: differences between what and what?\n\nL212: What do you mean by \"stationary poses at transition moments\"?\n\nFig. 6:How is GT looks like?\n\nWhat are prediction-based methods (L57), retrieval-based methods (L68), and bank-based auto-encoder (L249) exactly? These are not well-known concept yet.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept