Duolando: Follower GPT with Off-Policy Reinforcement Learning for Dance Accompaniment
Authors: Li Siyao
Tianpei Gu
Zhitao Yang
Zhengyu Lin
Ziwei Liu
Henghui Ding
Lei Yang
Chen Change Loy
Venue: ICLR 2024 poster (2024)
Keywords: motion generation
multi-modality
dance generation
human human-interaction
GPT
reinforcement learning
Primary area: generative models

Abstract: We introduce a novel task within the field of human motion generation, termed dance accompaniment, which necessitates the generation of responsive movements from a dance partner, the "follower", synchronized with the lead dancer’s movements and the underlying musical rhythm. Unlike existing solo or group dance generation tasks, a duet dance scenario entails a heightened degree of interaction between the two participants, requiring delicate coordination in both pose and position. To support this task, we first build a large-scale and diverse duet interactive dance dataset, DD100, by recording about 117 minutes of professional dancers’ performances. To address the challenges inherent in this task, we propose a GPT based model, Duolando, which autoregressively predicts the subsequent tokenized motion conditioned on the coordinated information of the music, the leader’s and the follower’s movements. To further enhance the GPT’s capabilities of generating stable results on unseen conditions (music and leader motions), we devise an off-policy reinforcement learning strategy that allows the model to explore viable trajectories from out-of-distribution samplings, guided by human-defined rewards. Based on the collected dataset and proposed method, we establish a benchmark with several carefully designed metrics.

Published as a conference paper at ICLR 2024
Duolando: FOLLOWER GPT WITH OFF-POLICY REIN- FORCEMENT LEARNING FOR DANCE ACCOMPANIMENT
Li Siyao'
Tianpei ou
Zhengyu Lin
Zhitao Yang*
Ziwei Liu'
Henghui Ding'
Lei Yang 3,4
Chen Change Loy! s
Ls-Lab, Nanyang Technological University Lexica Time
Shhanhaii Al Laboratory
https://lisiyao2:1 github io wttp::ooots///sollllsa.....
Figure I: Example of Duolando's 's results. The female avatar (red arrow) is driven by the proposed method to accompany real human's 's (white) dancing.
ABSTRACT
We introduce a novel task within the field 3D dance generation, termed dance accompaniment, which necessitates the generation of responsive movements from a dance partner, the "follower". synchronized with the lead dancer's 's movements and the underlying musical rhythm. Unlike existing solo or group dance generation tasks, a duet dance scenario entails a heightened degree of interaction between the two participants, requiring delicate coordination in both pose and position. sup- port this task, we first build a large-scale and diverse duet interactive dance dataset, DD1 00, by recording about minutes professional dancers' performances. To address the challenges inherent in this task, we propose GPT-based model, Duolando, which autoregressively predicts the subsequent tokenized motion condi- tioned the coordinated information of the music, the leader's 's and the follower's s movements. To further enhance the GPT's capabilities of generating stable results unseen conditions (music and leader motions), we devise off-polio rein- forcement learning strategy that allows the model to explore viable trajectories from out-of-distribution samplings, guided by human-defined rewards. Based on the collected dataset and proposed method, we establish a benchmark with several carefully designed metrics.
INTRODUCTION
Duet dancing an interactive art involving coordination between two individuals' body movements under background music. In the context of ballroom dancing, the duet dancers typically comprise leader and a follower. the leader responsible for initiating and guiding the movements, while the follower responds to the leader's cues and follows his/her lead. Developing computational model that enables virtual agents to accompany the dance with the user-controlledd leader has great potential wide range of virtual reality (VR) and augmented reality (AR) applications.
Although the dance accompaniment task has broad academic and practical value, there is no currently available duet dance data in public. To facilitate this task. we build large-scale dataset named DD1C0 (DuerDance100) Specifically. we record data of 10 different genres of ballroom dancing, including kinds of Latin (Cha Cha, Rumba, Samba, Paso Doble and Jive), 4 Moderns (Foxtrot, Tango, Quickstep and Waltz) and Pas de Deux of ballet performed by 5 pairs of professional dancers under unique background music. For each genre, the performers play 10 distinct clips, within duration of 36 to 107 seconds, resulting in total duration of around 117 minutes. As duet dances usually involve significant parts of occlusion and rotations, we collect the 3D motion data using professional MoCap equipment to ensure data quality. The final data consists of SMPL-X (Pavlakos et al. 2019) sequences, with body poses reconstructed from the point clouds by and hand gestures
Corresponding authors. * Work mostly completed UCLAA

---

Published as a conference paper at ICLR 2024
Runnba
Sambe
Chi dba
Latia
Jive
Paso Doble
Bailet
Waltz
Tampo
Modorn
Quackenp
Foemmt
Figure 2: Samples DD100 dataset. leader the follower are colored in green and red, respectively, DD100 contains 10 dance genres, featuring a diverse range of poses and interactions, with intricate hand gestures.
deduced from meta glove records. DD101 00 provides a rich context of interactive coordination between two dancers, including either physical connections (such as steps with hand holding) or strong semantic relations (such as leader-centered surrounding), which serves as fundamental basis for the study related to duet interactive dance.
Despite having a high-quality duet dance dataset, the dance accompaniment task cannot be solved by applying existing solo dance methods (Ren et al. 2020 Sun et al. 2020 Vaswani et al. 2017 Li
et al. 2021b Siyao et al. 2022 Sun al. 2022 seng al. (023). As interactive
duet dance not only requires the follower to maintain the aesthetics and the sense of rhythm of his/her movements, but also demands high level of interactive coordination with the leader. Such requirements make existing works on solo dancing inadequate for this task as they lack the scope of the partner. Meanwhile, generating a single agent's 's motion response to a pre-conditioned leader's movement presents more significant challenges than simultaneously synthesizing multiple agents (Le) al. 2023), since the latter can retrieve well-coordinated motion patterns from the training data, which simplifies the task.
In this paper, we propose two-stage framework to generate follower motion in harmony with both the background music and the leader's 'sss movements. In the first stage, we train VQ-VAFs to embed and quantize the dance movements of different body parts and the relative translation between the two dancers. Then, in the second stage, devise an autoregressively predict the next token, conditioned on the amalgamated information the music signal, the leader's motion, and the previous follower sequence. Stability issues arise when confronted with unheard music or unseen leader motion patterns. A common observation is that the lower body movement appears incompatible with the global displacement, resulting in skating artifacts. To address this challenge, we introduce an off-policy reinforcement learning strategy for GPT, enabling method handle dut-if-distribution instances robustly based human-defined rewards. The above modules contribute to Duolando, which can generate reasonable movement responding to the leader's movement and serves as a strong baseline of dance accompaniment.
GPT to
The contributions of our paper are three-fold: (1) We introduce novel multi-modal task, dance accompaniment, and provide a large-scale and diverse dataset for both training and testing purposes. Leveraging this data, we establish a new benchmark with metrics reflecting both the quality of isolated dancers and the interaction between partners.
(2) We construct a GPTT-bsed network capable of generating motion sequences, taking into account the coordination between partners, which serves as a robust baseline for this task. (3) We introduce an off-policy reinforcement learning strategy for GPT address out-of--istributte challenges, and demonstrate its successful application in our task.
2

---

Published as a conference paper at ICLR 2024
Table 1: Comparison with human-human interaction and music-to-dance datasets. HHI denotes Human- Human Interaction, where S represents Strong interaction with physical contact while W means Weak interaction like repeated motion in group dance. 5 indicates whether having accompanied music modality. O denotes whether having hand (finger-level) motions. # Subj. denotes the number of performers. T denotes average duration and T the total duration of all sequences. MV stands for capturing with multi-view cameras. Genres for human-human interaction datasct means the type of interactions, while it indicates the music and dance styles for music-to-dance datasts. n/a means that the exact information is missed in the original paper. *Interactions without physical contact appear in partial data (Showcase, Cypher and Battle) in AIST++
Dataset
HHT
O
# Genres
Suhj
T
T
Acquisition
GT
CMU-MoCap (interactive)
(cmu)
5.2s
285.5s
MoCap
3D Joints
UMPM Vaa der An
2011
W
30
222s
2.2h
MoCap
3D Joints
NTU-RGI-DD 121
LLu
2020
S
26
106
2.7s
0.47h
Kinect
3D Joints
You2Me (gg al
120s
1.4h
MV Kinect
3D Joints
InterHomman Liang
2025
S
X
X
11
m/a
3.9s
6.56h
MV
SMPL
DanseNet Zhuane cl.
1122
K
n/a
0.96h
MoCap
3D Joints
Dancing2Must (Lez
2019
m/a
71h
Pseudo
2D Joints
DanceRevolusion liang el
2021
X
nda
60s
12h
Pseudo
2D Joints
PMSD Valle-Pércz ct 2021
3.th
MoCap
3D Joints
SMVR Valis-PSrez el
2021
X
n/a
9h
VR Tracker
3D Joints
AIST++
2224b
5.2h
Pseudo
SMPL
AIOZ-GDANCE Le
2023
W
n/a
4000+
37,5s
16,7h
Pscudo
SMPL
DD100
70.2s
1.95h
MoCap
SMPL-X
DD100: A LARGE-SCALE DUET DANCE MOCAP DATASET
Data Statistics. To facilitate research in the dance accompaniment task, large-scale dataset of duet dances, named DD1 00, was collected. DD100 comprises ten distinct genres of duet dances, all featuring strong interaction between the dancers. The genres include five Latin dances (Cha Cha. Rumba, Samba, Paso Doble, and Jive), four Modern (a.k.a. Standard) dances (Foxtrot, Tango, Quick- step, and Waltz), and Pas-de Deux ballet. The dances were performed by five pairs of professional dancers, while each dance sequence was accompanied by unique background music. Each dance genre was recorded by ten distinct clips, with clip durations ranging from 49 to 96 seconds, resulting in total duration of approximately 1.9 hours (or approximately 1 15..4 minutes). In the experiment, we randomly split the dataset into the 80% training set and 20% test set, with the training set of 168,176 frames (5605.9 seconds) and the test set of 42,496 frames (1416.5 seconds). During the recording, nearly 80% sequences are performed twice. If counting these duplicated takes, the total duration will be 3.24 hours. Considering the diversity in moving direction and difference in motion details at each take, we use all these data for training and testing in practice.
Data Collection. To obtain high-quality data, we used 20 optical MoCap cameras to capture body data for two dancers at 120 FPS. The raw Mocap data consist of the 3D positions of 53 marker points on body surface in each frame. To process these data into SMPL-X (Pavlakos et al. 2019) format, we first select a clip for each dancer to fit his/her body shape parameters (beta), following the pipeline of SOMA (Ghorbani & Black 2021p. Then, parameters (theta) regression in each frame after filtering out those invisible points with confidence scores lower than threshold to keep the robustness of the regression results. During this procedure, we do not fit the hand movement. Since the hands are prone to self-occlusion or inter-occlusions between two individuals, we employed inertial motion capture gloves (meta gloves) capture these data. The raw data of meta gloves are stored in BioVision Motion Capture (BVH) format, describing hierarchical rotations of finger joints in Euler angles. We transfer those Euler angles to axis angles aligning the MANO (Romero et al. 2022) model initialized with "flat" gestures for both hands. As the "flat" pose of MANO differs from the initial one in BVH format with fingers apart in specific angles, we subtract this difference while mapping from BVH to MANO. Finally, we combine the body and hand parameters with the wrist rotations from MoCap regression. Fach dance clip data in DD100 consists of the SMPL-X (Pavlakos et al. 2019) sequences for both the leader and follower, along with the corresponding music within an average rhythmical beat of 118 beat per minute (BPM) from 72 BPM to 163 BPM.
the pre- pre-processed body shape to assist pose
Comparison to Related Datasets. As shown in Table I DD100 stands out from existing datasets by simultaneously incorporating the following distinct features: (1) Complex and Strong Interaction. Existing solo dance datasets (Zhuang et al.) [022 Lee et al. 2019 Huang et al., [202]] Valle Pérez. et al. Li et al. 2022) overlook the element of interaction, while existing group dance
2021
3

---

Published as a conference paper at ICLR 2024
INNs
Codebook
ククリリン 3D joint positions T x J x
ID CNN
グクイーンインズ 3D joint positions p T x
quantize
feature f T C
DM
C
(a) Motion VQ-VAE
Joint rotation matrix T J
Codcbook n
ID CNN
ID CNN
quantize
Relative trunslation
feature f
Relative translation
7
T'C
T
(b) Relative Translation VQ-VAE
Figure 3: (a) Structures of Motion VQ-VAEs and (b) Relative Translation VQ-VAE. The quantization to substitute a encoded feature to the most similar one 2 in the codcbook 2 such that 2 = arg minzez llf. =||-
datascts (Le et al. 2023) primarily emphasize group movement coordination with similar motions replicated to different agents with minimal physical contact. Furthermore, current -umann interaction datasets (emu Van der aa et al. 2011 Liu et al., 2020) typically feature simpler actions and weaker interactions, such as handshakes and hugs. In contrary, as depicted in Figure Z duet dances in DD1 00 involve more complex motions and intense interactions between various body parts. (2) Multimodal. Unlike datasets (Ng et al.) 2020 Liang et al. 2023) focusing on numan-hunme interaction, DD1 00 incorporates musical modality, which a crucial factor consider during the generation process to ensure synchrony with the music. (3) Extended Duration. Many existing human motion datasets feature short durations. For example, dances in AIST++ (Li et al.] (021b) typically last around 10 seconds. In contrast, each dance clip in DD1 00 extends over one minute. This not only supplies a higher-level choreography reference for model training but also poses the proposed dance accompaniment problem to be more realistic and challenging.
OUR APPROACH
To address the dance accompaniment task, we propose a baseline method, Duolando, GPT-based network enhanced with off-policy reinforcement learning to improve its generalization ability.
3.1 QUANTIZING MOTION AND RELATIVE TRANSLATION
Previous studies (Siyao et al. 2022 Ng et al. 2022 has viewed dance as a sequential combination of reusable dance positions, capturing the essence of movement in rhythmic context. To efficiently distill and quantize these basic dance components into deep feature space in an unsupervised manner, draw VQ- (Q-V (Van Den Oord et al.] 2017). This provides tokenized and interpretable representation of the dance motion sequence. Our approach incorporates a 1D CNN-based VAE to preserve the spatio-temporal continuity of dance movement. Inspired by Siyao al. (2022), use four VQ-VAEs to encode and quantize the motion of four body parts (upper half body, lower half body, left hand, and right hand) into code sequences "" z zown." z!hand and x zrhand Additionally, we employ another VQ-VAE VQ"" to model the relative translation tr between the follower and leader. This allows the GPT model to explicitly output the relative positions of the follower by subtracting the root point of the follower from that of the leader. Please see Figure 3 for details.
In the case of motion VQ- VQES,,s, CNN encoder E, where T is the frame length and J is the joint count. This is then translated to temporally downsampled deep feature f TR''xx xC where T' T/d and C' is the number of channels. Subsequently, deep feature f is quantized into code sequence z by replacing each element fi with an element z codebook Z that shows the smallest difference [| =x|l. For training the motion VQ-VAE, we use two decoders, Dp and DM. which translate the quantized code back to 3D joint positions e and rotation matrix M, respectively, allowing us to generate rotations that can directly animate avatars without the need for inverse kinematics. The training loss of the motion
X J sequence of 3D joint positions fed into a 1D-
4

---

Published as a conference paper at ICLR 2024
One-hot Cross- Enttrop Loss
10T
T
Interaction coordinated GPT
Coordinated Att.
D...
T+L
Zup
Adoum
Rihand
Rehand
Sup
Zeown
Ihhnnd Sehhnd Ztr
1
LAT
LAT
LAT
LAT
LAT
follower sequence
translation
TiL.
Look-Ahead Att.
2 Z.own
Zthond
Brhand
music
leader sequence
Figure 4: Structure of follower GPT. The GPT takes ten inputs and autoregressively predicts the subsequent tokens of follower's 's motion and the relative translation. Preconditions (music signals and leader's motion) are integrated with Look-Ahead Transformers (I.AT).
VQ-VAE is computed as follows:
LvQ = Lree + Lree + - sg(2)]l
(I)
where Lree is the sum of 22-reconstruction losses for the values, the first (velocity) and the second derivatives (acceleration) of the reconstructed motion sequence and the ground truth. p and denote the ground-truth 3D joint positions and rotation matrix, respectively, while "sg" denotes "stop gradient" (Chen & He. 2021) and À is the trade-off "commitment" parameter (Dhariwal al. 20220] Siyao et al. (022)). The training loss of the relative translation VQ- VQ-VAE follows the same formula but only includes one reconstruction loss, specifically Lree (tr,tt) tr).
3.2 INTERACTION COORDINATE GPT
Formulation. With the pretrained VQ-VAE, the 3D dancing movement can be transferred into sequence of quantized code numbers. The goal of using GPT is to generate the follower's dance sequence in the quantized code domain with the maximum likelihood:
upll
down
IhandV
chandV
arg max Pr (z/m, upp
down
thande
(2)
We use shared VQ- VAES to generate the code sequences for the dancers. The leader's movement represented in green (z with notion A while the follower's movement is represented in pink (z with notion V The generated code sequences are then decoded by DM to reconstruct the 3D joint rotation and drive the follower's motion with global root position q computed by = Dr(z"r) * where q is the root position of the leading dancer. For more details on the follower GPT structure, refer to Figure 4
Looking-Ahead Conditions. In duet dance, the follower needs not only to respond to the leader's current movements but also anticipate futurre changes the dance dynamics. To fulfill this requirement, we implement a look-ahead mechanism that allows the GPT to be aware of future conditional signals. Specifically, when sampling the conditional inputs (music m and leader motion ), extract an additional L tokens beyond the GPT's block size T, leading conditioning sequences of length (7 + L). These extended conditioning sequences are then processed through Look-Ahead Transformers (LAT) to derive deep embeddings, denoted as m LAT
and LAT(2 0....I++-1)) Within the LATs, use look-ahead attention layers propagate information from L future tokens to the current one, which is achieved through a banded mask of attention with window size of L. By incorporating future rhythms and leader movements, the GPT predicts more synchronized and stable follower dance positions.
Interaction Coordination. After the look-ahead expansion is completed, truncate the first T embeddings of m and こ and use them together with the follower motion d0 and relative translation frr indexed from 0 T - input for our model. The GPT then incrementally generates predictions
5

---

Published as a conference paper at ICLR 2024
encode
one-hot labels Rapll
|2] Fo. myy1
GPT
EST
DDMI00 Training set
(T(-(1s)
In-distribution data D
Tx 1 input
T T'x K prob.
Supervised Training
rewards Mi Py
-HHHll iil unheard music
generate
Za
GPT
{[E
GPT8
leader
motion
OOD data
x'x K prob.
(b) Off-policy Reinforcement Learning
Figure 5: GPT's supervised training vs off-policy RL stage (b), For ST, supervising labels sequences {z} quantized from the in-domain data training set. In RL stage, network weight is optimized based on human-defined rewards (scores) on GPT's generated samples (&) on OOD conditions.
for subsequent tokens, which correspond to indices ranging from T. Formally,
upp? down WhandV ",.., I.... "I... T 1I....
rrhando fr :.,,....?
(mo. T- "0.... T-...:.-11 1-11 "D..., -11 O...., T-11 20. T-.....11
coloumn athand
aehandd
(3)
2g T- "...., -11 " P...... 20. .-.....T-I) T-1).
2 down
Hhand
-hhand
tr
To effectively integrate the information of the 10 input items in Equation 3 block-wise lower-triangular matrix as the mask of interaetion-coordinated attention. Since the lower triangular matrix masks off the future information, this preserves the causal order of inference. ensuring that past tokens do not have access to future ones, and maintain the integrity and coherence of these items during the prediction process.
10-by-10 10
3.3 OFF-POLICY GPT REINFORCEMENT LEARNING
Although GPT powerful, still prone to generate unsatisfactory results when confronted with dut-o--distributtin (OOD) scenarios. In our task, specifically, when the leader presents unseen motion patterns, the follower may displace to response without proper foot steps thus yielding skating artifacts. To better adapt GPT models this challenge, finetune the network. Unlike the supervised training stage, there is no ground truth labels to regress in RL stage. As shown in Figure 5 for RI., the inputs of GPT model are code sequences (&) generated by the GPT itself on OOD conditions, either using the current network weight or the past ones. The network weight then optimized according to a loss based on human-defined rewards.
employ off policy reinforcement learning (RL) to
From an RL standpoint, the act of predicting the next token at step / is construed as an action or. which involves selecting appropriate token ž from the dictionary. Concurrently, the GPT model with weight is considered policy network xe that predicts a probability distribution
(- for each candidate & E Z step t, with the state s; perceived as the preceding sequence (2g. 21 .... 2g At present, a majority of the publicly documented RL implementations (Ouyang et al.) 2022 Siyao jet al. 2022) for GPT employ on-policy actor-critic (AC) strategy, where the update of the policy network hinges on the sequences generated by the current GPT weight. Basically, AC uses loss function as follows:
Co Σ log (To(a{(6?))
(4)
where {af} denotes the trajectory sampled by the immediate GPT weight 0. The term A represents "advantage", which reveals the increment after taking action af (see Appendix C. positive advantage increases the probability T(āel (5)) of the currently sampled action ag. whereas negative advantage reduces it. This indiscriminate mechanism of increasing or decreasing hinders the ability
of RL to reuse past data. For example, when the advantage of a previous sampling (s a. alse) is negative, the on-policy loss Lon will persist in pushing associated values to decrease, regardless of the current probability already being very small. Such misalignment is counter-logical and potentially harmful to network training due the misleading optimization target.

---

Published as a conference paper at ICLR 2024
To address this issue, we extend the RL GPT off policy by establishing explicit optimization targer for the probability To ( Drawing from ideas in energy- based policy modeling (Haarnoja ct al. 2017: 2018), equivalence can be established between the policy probability and the expected reward income, implemented by a monotonic mapping a : RR - [o. 1]. This relationship can be formulated (a] 8) (Q( a)), where the value represents the expected total future income after executing action a at state s. In light of this equivalence, if an action a results in greater future income Q(s, a), it should hold larger probability (Q a)) of being selected, which leads an explicit optimization reference to To (a)ss)) With this concept as our foundation, we construct a novel learning strategy using loss function defined as
T-1
RI
Σ T=
log (1 (1-abby abs (To (ārlá.) - a (Qt,,, (B,,))))))
(5)
The proposed learning strategy is off-policy since RR can be reused on past data to guide the GPT to learn a specific probability of oz. In practice, we set sigmoid Q + 3) and approximate Q by two-step accumulation of rewards on sampled trajectory Q r($;, âs) + yr(64+1, dc+1). Detailed algorithms pipeline is shown in the supplementary file.
Step-wise Rewards in Dance Accompaniment. In this implementation, we specifically address the skating artifacts caused by the inconsistency between predicted translation and the lower body movement. To this end, we design a step-wise reward for each predicted token. Specifically, we train an additional velocity decoding branch, namely Dy. for lower-body motion VQ-VAE in an unsupervised manner, as per implementation by Siyao et al. (022)). to decode the predicted lower body sequence doren' velocity V by V = Dv (2 whether the follower's global position 9% calculated from translation, synchronizes with the lower body movement by computing a difference 8 as d Σe=o ll i-d+u Ve-dtull (ietulldd /d, where 8 is the first derivative approximated by subtraction between adjacent items. The reward ydown for lower body at step is then defined as
We then use V as a reference to judge
1, 8 < threshold,
down
(6)
-77 -7-6 6, otherwise,
where y) is a parameter to control the punishment magnitude. 77 is set to 100 and the threshold is 0.03 in our experiment. Rewards for other components (up, Ihand, rhand and Ir) are consistent be 1.
4 EXPERIMENTS
Detailed descriptions and implementations of the VQ-VAE and GPT are in Appendix B
Evaluation Metrics. We apply series of quantitative metries evaluate the generated follower's movement in three distinct aspects: (1) the inherent quality of the follower's motion, independent of its interaction with the leader, (2) the follower's interaction with the leader, and (3) the alignment of the dance with the background music. For (1), we adopt metrics used in the solo dance benchmark AIST++ (Li et al. (02Tb). Specifically, we compute the Fréchet Inception Distance (FID) between the generated follower's motion and the real dance in the whole DD1 CC dataset on kinematic (denoted as "k=") (Onuma et al. 2008) and graphical (denoted as "g"") (Müller al.) (005) features, and compute the standard deviation between features of the generated follower sequences as the diversities Div and Div, that indicate the difference among those sequences. As to (2), the interactive quality, we first extract cross-distance (cd) feature from the duet dance sequences. Specifically, for each frame, calculate the pairwise distances between ten joints' positions, including the pelvis, both knees, feet, shoulders, the head, and two twists, of the leader and those of the follower obtain a 100-dimension (10 x 10) features. Then, we use these distances as an interactive feature and compute the FID,,t and Div Meanwhile, we compute contact frequency (CF) value explicitly explore the strength of interaction. CF is defined as the ratio of the number of frames in which two dancers are in physical contact, where physical contact is defined as two SMPL-X models having a minimum absolute mesh distance below a 2-em threshold. Finally, we define Beat Echo Degree (BED) to evaluate the consistency of dynamic rhythms of two dancers:
1
min ee ||el - tl [1²) )
exp
(7)
20
PEB
7

---

Published as a conference paper at ICLR 2024
Table Quantitative benchmark for dance accompaniment. The first place and runner- up are highlighted in bold and underlined, respectively. S denotes a solo dance generation model that does not take the leader into condition, while denotes that one does. *Since solo dance has no interaction, the cross-distance between agents are completely irregular, making the diversity particularly high.
Solo Metrics
Interactive Metries
Rhythmic
Method
FIDz
FID,(L)
Divu(t)
Div,(t)
Divea(?)
CR())
BED(T)
BAS(T)
Ground Truth
6.56
6.37
11.31
7.61
12:35
74.25
0.5308
0.1839
Bailondo Siyao et
2022
78.52
36.
11,15
7.92
6643,31
52.50
7.13
0.1831
0.1930
EDGE (Teeng et 2023
69.14
44.58
8.62
6.35
5894.45
60.62
6.82
0.1822
0,1875
8
Duslando w/o. RL fr 10
12.53
24.17
10.51
9.42
4803.20
42.72
7.04
0.1826
0,1852
Dunlando w/o. RI. fr
62.29
27.95
13.16
8:53
7970.19
54.539
7.76
0.2194
0.2002
Duolando w/o. RL
106.72
34, 10
13.88
7.03
21.68
9.33
57.43
0.2795
0.2193
Duolando
25.30
33.52
10.92
7.97
9.97
14.02
52.36
0.2858
0.2046
where B² = { and B =(1) represent the timing of beats in the leader and follower movements, respectively, while the motion beat time is calculated by finding the local minimum time of the motion velocity; is normalization parameter (a 3 in our experiments). The BED takes similar formulation as the Beat-Align Score (BAS) (Siyao et al., 2022). where the reference base is substituted with leader's 's dynamic beats instead of the music's 's rhythmic beats. For (3), we adopt the BAS defined by Siyao et al. (2022), to assess the correspondence between the generated motion and the rhythm of the background music.
Baseline Setup. In addition to our proposed Duolando, we evaluate two state-of thet solo dance generation methods, Bailando (Siyao et al., 2022) and EDGE (Tseng et al., 2023), to discern the differences between solo dance and interactive duet dance. Considering the absence of existing dance accompaniment methods conditioned on both music and a leader's motion, we also perform ablation studies using various Duolando variants. Specifically, reinforcement learning (RL), relative translation (tr), and interaction coordination (IC), through the evaluation of variants "w/o. RL", "w/o. RL tr", and "w/o. RL tr IC", respectively. For the variant without tr, we delete 2²" from the input of GPT and predict the translation via the velocity decoding branch Dy. which is trained in an unsupervised manner, as detailed in Section 333 and (Siyao et al. 2022). In the case of "w/o. RL Ir IC", we further simplify the model's input to m, z 2'p 2 down Frando and x² zzhando excluding the influence of the leading dancer, to observe the effectiveness of interactive coordination in dance accompaniment. Along with these methods, we include the scores of the ground truth test data as reference points. The quantitative benchmark is provided in Table
investigate the effectiveness of
Analysis. Given that the methods presented in Table are organized in an order with incremental addition of modules, we conduct detailed pairwise analysis of each successive pair. Upon examina- tion of the first two solo dance models. Builando achieves FID; and FID, scores of 78.52 and 36.19, respectively, while those of EDGE are 69.14 and 44.58. In contrast, the model "Duolando w/o. RL Ir IC" significantly improves upon these solo metrics, with scores increasing by 65.99 (84%) and 56.61 (72%) respectively. Compared to Bailando, this variant of Duolando includes a look-ahead (LA) mechanism, which allows for the generation of more fluent and continuous dance movements over long durations, contributing to the significant improvement of the FID scores. However, since all solo dance methods lack the condition of a leader, they do not exhibit noticeable change in poor interactive values. By taking the leader's 's movement into account and using the proposed interactive coordination (IC), "Duolando (w/o. RL tr)" achieves 20% improvement on BED, increasing from 0.1826 to 0.2194, which can be attributed to better synchronization with the leader's dynamics. Nonetheless, when compared with "Duolando w/o. RL tr IC", "Duolando w/o. RL tr" still shows suboptimal performance in terms of FID ed and Div This is because the ed feature is strongly affected by the relative translation between the two dancers. While the virtual follower can respond to the leader's 's motion pattern, it still fails with positioning itself in a reasonable location based on unsupervised velocity prediction. Moreover, the low CF value of 7.76% indicates that the generated follower rarely makes contact with the leader, contradicting the fundamental requirements of dance accompaniment. With the introduction of explicit relative translation (tr) prediction via interactive coordination, the FID cd of "Duolando w/o. RL" reduces drastically from 7970.19 to 21.68, aligning much more closer with the ground truth. Simultancously. the CF value increases to 57.43% (a 7-fold increment), and BED is raised 0.27955 (27% 1), indicating higher level of interaction with
8

---

Published as a conference paper at ICLR 2024
User study
(Dnolando wins)
Duolando wio, RL Ir
100%
80%
60%
Duolando wio, RL LT
20%
Duolando w/o. RL
A wio. Rr IC
w/o, RL
w/o. RL
A GT
Duolando
Figure 6: Qualitative results and user study (b). qualitative results, conditioning leader is colored in gray while generated followers are red. In boxplot of user study, triangles and colored lines are mean and median values, respectively. Circles are outliers beyond 1.5x interquartile range (3a in normal dist.).
the leader. However, the explicitly predicted translation can sometimes clash with the generated movements, deteriorating the quality of the dance generated. Upon closer examination, compared to "Duolando w/o. RL tr", the FID; value of "Duolando w/o. RL" drops significantly from 62.29 to 106.72 (71% f). Given that the kinetic features (Onuma et al. 2008 are computed based on motion velocity, they are particularly susceptible to unreasonable global shifts like the so-called "skating artifacts". This issue is resolved by the proposed reinforcement learning mechanism, enabling the full Duolando model to show a 81.42 (76%) improvement on FID; compared to the variant without RL. Moreover, the reinforcement learning mechanism contributes an improvement in interactive quality (11.71, 54% J) diversity (4.69, 50% 1) and motion alignment (0.063, 22% 1), demonstrating its effectiveness in enhancing the performance of the full Duolando model. In conclusion, the improvements of different metrics attest to the effectiveness of each proposed module, underscoring their potential producing more fluent, interactive, and diverse dance movements.
Qualitative Comparisons. To clearly demonstrate the effectiveness of each proposed module, we provide several visualizations of results produced by different variations in Figure In the case of "Duolando w/o. RL fr IC", the model lacks a condition for the leader, resulting in the virtual follower operating independently and not responding to the "dance hold" (the pose holding arms) initiated by the leader. Conversely, the follower in "Duolando w/o. RL tr" can pose responsively, but she still fails to accurately follow the leader's displacement, causing a unreasonable distance between the dancers. The relative translation module allows the two dancers to interact closely in "Duolando w/o. RL". However, as highlighted in the red dotted box, the follower's legs do not move appropriately when shifting with the leader, leading learning, the follower can alternate footwork, presenting proper displacement of steps backwards.
noticeable skating artifact. With the integration of reinforcement
User Study. To explore subjective judgments of quality, we conduct a user study wherein Duolando is compared against each of its variants individually. As both Bailando and "Duolando w/o. RL tr IC" are solo dance generation frameworks, we only conduct this study exclusively on the latter model, which demonstrates higher performance quantitatively. Specifically, we recruit 15 participants, while for each participant, we randomly display 40 pairs of dance clips, with each pair comprising one Duolando result and one result from the comparison method. We then ask to indicate which one dances better response to the leader and the music. As depicted in Figure bb)) the full Duolando model outperforms the "w/o. RI. Ir IC"", w/o. RL tr", and "w/o, RI." variants in 80%, 83%, and 62% comparisons respectively, demonstrating the significance each proposed module. However, when compared to the ground truth, Duolando only surpasses it around 15% of the time, indicating that dance accompaniment remains a challenging and open task for future works.
CONCLUSION
We introduce a new task named dance accompaniment. To support this task, we first collect large-scale duet dance dataset DD100, 10,, and propose a baseline framework named Duolando, which develops follower GPT with off- off-policy reinforcement learning. We establish benchmark with several metrics to evaluate the dance quality, interaction, and alignment with music. Additionally, DD1 00 has the potential to support more multi-moda human-human interaction tasks.

---

Published as a conference paper at ICLR 2024
Ethics Statement The task we're proposing dance accompaniment holds significant potential for enhancing a multitude of VR/AR applications, including the prospect of virtually dancing alongside AL. Furthermore, research into numan-humai interaction could substantially aid in improving the immersive and interactive experiences offered by VR games. However, such applications or games have a potential risk in the future that might be addicted to interacting with virtual agent instead of real-world social events, once the response of the virtual agent is developed to be charming and real-looking enough. Meanwhile, the research on real-looking motion generation in response to human may also lead to AI fraud, where the existing verifying methods could be more difficult to judge whether is fake based on response.
The data collection process (including contracts with actors) in our research was conducted ethically. The data will be released without containing any personally identifiable information. Some audio tracks in our dataset come from copyrighted musics. The inclusion of these tracks should be regarded as 'fair use' provisions due three reasons. (1) The music segments used are modified (shortened, and partially slowed down), deviating from their original form. (2) The use of these tracks is solely for model training/testing the model rather than entertainment. (3) The academic nature this work and the absence of public searchability (we won't 't provide music info beyond the audio signals) avoid potential commercial impact.
Acknowledgement This research is supported by the National Research Foundation, Singapore under This study is supported
its AI Singapore Programme (AISG Award No: AISG-PPDD/2021-11-081111 under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s). It is also supported by Singapore MOE AcRF Tier (MOE-T2PP20221-001] and AcRF Tier (M0E-T2EP20221-001)
We sincerely thank Mingyang Song, the PM in Sense'Time, for helping kick off this project and thank Judy for organizing the dancers. We also appreciate Lai Jiang, Zhijie Cao. Tinghao Liu and Quan Wang for their sigmificant help during data pre-processing. Siyao is grateful to Mr. Zhuo Sun and Dr. Chen Qian for their indispensable supports. The Mocap data are collected through services of QINGMU TECH LTD., Shanghai.
REFERENCES
Cmu mocap.
http::/mocap:: cs. cmu camuueddiy edd/
Tenglong Ao, Qingzhe Gao, Yuke Lou, Baoquan Chen, and Libin Liu. Rhythmic gesticulator: Rhythm-aware co-speech gesture synthesis with hierarchical neural embeddings. In SIGGRAPH Asia, 2022.
Nikos Athanasiou, Mathis Petrovich, Michael J Black, and Gül Varol. Teach: Temporal action composition for 3d humans. arXiv preprint arXiv:2209.04000 2022.
Zhongang Cai, Mingyuan Zhang, Jiawei Ren, Chen Wei, Daxuan Ren, Jiatong Li. Zhengyu Lin, Haiyu Zhao, Shuai Yi, Lei Yang, et al. Playing for 3d human recovery. arXiv preprint arXiv:2110.07588, 2021.
Zhongang Cai, Daxuan Ren, Ailing Zeng, Zhengyu Lin, Tao Yu, Wenjia Wang, Xiangyu Fan, Yang Gao, Yifan Yu, Liang Pan, et al. Humman: Multi-modal 4d human dataset for versatile sensing and modeling. In ECCV, 2022.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint arXiv:2005.00344 2020.
Nima Ghorbani and Michael Black. Soma: Solving optical marker-based mocap automatically. In ICCV, 2021.
Anindita Ghosh, Rishabh Dabral, Vladislav Golyanik, Christian Theobalt, and Philipp Slusallek. Imos: Intent-driven full-body motion synthesis for human-object interactions. In Computer Graphics Forum, 2023.
10

---

Published as a conference paper at ICLR 2024
Kehong Gong, Dongze Lian, Heng Chang, Chuan Guo, Xinxin Zuo, Zihang Jiang, and Xinchao Wang. Tm2d: Bimodality driven 3d dance generation via music-text integration. In ICCV, 2023.
Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2r: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In ECCV, 2022.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In ICML, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In ICML, 2018.
Fangzhou Hong, Mingyuan Zhang, Liang Pan, Zhongang Cai, Lei Yang, and Ziwei Liu, Avatarclip: Zero-shot text-driven generation and animation of 3d avatars. arXiv preprint arXiv:2005.08535, 2022.
Ruozi Huang, Huang Hu, Wei Wu, Kei Sawada, Mi Zhang, and Daxin Jiang. Dance revolution: Long-term dance generation with music via curriculum learning. ICLR, 2021.
Sumit Jain and C Karen Liu. Interactive synthesis of human-object interaction. In SIGGRAPH, 2009.
Biao Jiang. Xin Chen, Wen Liu, Jingyi Yu, Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. arXiv preprint arXiv:2306. 14795, 2023.
Andrej Karpathy. https///gitub.ccommkaaaaattinnnn
2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014.
Nhat Le, Thang Pham, Tuong Do Erman Tjiputra, Quang Tran, and Anh Nguyen. Music-driven group choreography. In CVPR, 2023.
Hsin- Ying Lee, Xiaodong Yang, Ming- Yu Liu. Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Hsuan Yang, and Jan Kautz. Dancing to music. In NeurlPS, 2019.
Sergey Levine.
Buyu Li, Yongchi Zhao, Shi Zhelun, and Lu Sheng. Danceformer: Music conditioned 3d dance generation with parametric motion transformer. In AAAI, 2022.
Jing Li, Di Kang, Wenjie Pei, Xuefei Zhe, Ying Zhang, Zhenyu He, and Linchao Bao. Audio2gestures: Generating diverse gestures from speech audio with conditional variational autoencoders. In ICCV, 2021a.
Quanzhou Li, Jingho Wang, Chen Change Loy, and Bo Dai. Task-oriented human-object interactions generation with implicit neural representations, arXiv preprint 13Xi::2303.33/22 2023.
Ruilong Li, Shan Yang, David Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In ICCV, 2021b.
Han Liang, Wenqian Zhang, Wenxuan Li, Jingyi Yu, and Lan Xu. Intergen: Diffusion-based multi-human motion generation under complex interactions. arXiv preprint arXiv:2304.0556 2023.
Jun Liu, Amir Shahroudy, Mauricio Perez, Gang Wang, Ling- Yu Duan, and Alex C Kot. Ntu rgb+d 120: A large-scale benchmark for 3d human activity understanding. IEEE TPAMI, 42(10), 2020.
Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black. Amass: Archive of motion capture as surface shapes. In ICCV, 2019.
Brian McFee, Colin Raffel, Dawen Liang, Daniel Ellis, Matt Mc Vicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in python. In Proceedings of the 14th python in science conference, 2015.
Qianhui Men, Hubert PH Shum, Edmond SL Ho, and Howard Leung. Gan-based reactive motion synthesis with class-aware discriminators for human-human interaction. CG, 102, 2022.

---

Published as a conference paper at ICLR 2024
Meinard Müller, Tido Röder, and Michael Clausen. Efficient cantent-based retrieval of motion capture data. In SIGGRAPH. 2005.
Evonne Ng, Donglai Xiang, Hanbyul Joo, and Kristen Grauman. You2me: Inferring body pose in egocentric video via first and second person interactions. In CVPR, 2020.
Evonne Ng, Hanbyul Joo, Liwen Hu, Hao Li, Trevor Darrell, Angjoo Kanazawa, and Shiry Ginosar.
Learning to listen: Modeling non-deterministic dyadic facial motion. In CVPR, 2022.
Kensuke Onuma, Christos Faloutsos, and Jessica K Hodgins. Fmdistance: A fast and effective
distance function for motion capture data. In Eurographics, 2008.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. NeurIPS, 2022.
Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands, face, and body from a single image. In CVPR, 2019.
Mathis Petrovich, Michael Black, and Gül Varol. Action-conditioned 3d human motion synthesis with transformer In ICCV, 2021.
Mathis Petrovich, Michael J Black, and Gül Varol. Temos: Generating diverse human motions from textual descriptions. In ECCV, 2022.
Xuanchi Ren, Haoran Li, Zijian Huang, and Qifeng Chen. Self-supervised dance video synthesis conditioned on music. ACM MM, 2020.
Javier Romero, Dimitrios Tzionas, and Michael J Black. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint arXiv:2201.0260 02610,, 2022.
Yonatan Shafir, Guy Tevet, Roy Kapon, and Amit H Bermano. Human motion diffusion as generative prior. arXiv preprint arXiv:2303.01418, 2023.
Li Siyao, Weijiang Yu, Tianpei Gu, Chunze Lin, Quan Wang, Chen Qian, Chen Change Loy, and Ziwei Liu. Bailando: 3d dance generation by actor-critic gpt with choreographic memory. In CVPR, 2022.
Guofei Sun, Yongkang Wong, Zhiyong Cheng, Mohan Kankanhalli, Weidong Geng, and Xiangdong Li. Deepdance: music-to-dance motion choreography with adversarial learning. TMM, 23, 2020.
Jiangxin Sun, Chunyu Wang, Huang Hu, Hanjiang Lai, Zhi Jin, and Jian-Fang Hu. You never stop dancing: Non-freezing dance generation via bank-constrained manifold projection. In NeurIPS, 2022.
Taoran Tang, Jia Jia, and Hanyang Mao. Dance with melody: An lstm-autoencoder approach to music-oriented dance synthesis. In ACM MM, 2018.
Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209. 14916, 2022.
Jonathan Tseng, Rodrigo Castellon, and C Karen Liu. Edge: editable dance generation from music. In CVPR, 2023.
Guillermo Valle-Pérez, Gustav Eje Henter, Jonas Beskow, Andre Holzapfel, Pierre-Yve Oudeyer, and Simon Alexanderson. Transflower: probabilistic autoregressive dance generation with multimodal attention. ACM TOG, 40(6), 2021.
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NeurIPS, 2017.
NP Van der Aa, Xinghan Luo, Geert-Jan Giezeman, Robby T Tan, and Remco C Veltkamp. Umpm benchmark: muulti-person dataset with synchronized video and motion capture data for evaluation of articulated human motion and interaction. In ICCVW, 2011.
12

---

Published as a conference paper at ICLR 2024
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. ArXiv, aby/1706.05762, 2017.
Edward Vendrow, Satyajit Kumar, Ehsan Adeli, and Hamid Rezatofighi. Somoformer: pose forecasting with transformers. arXiv preprint arXiv:2208. 14023, 2022.
Sicheng Yang, Zhiyong Wu, Minglei Li, Zhensong Zhang, Lei Hao, Weihong Bao, Ming Cheng, and Long Xiao. Diffusestylegesture: Stylized audio-driven co-speech gesture generation with diffusion models. In IJCAI, 2023a.
Zhitao Yang, Zhongang Cai, Haiyi Mei, Shuai Liu, Zhaoxi Chen, Weiye Xiao, Yukun Wei, Zhongfei Qing. Chen Wei, Bo Dai, et al. Synbody: Synthetic dataset with layered human models for 3d human perception and modeling. In ICCV, 2023b.
Hongwei Yi, Chun-Hao Huang, Dimitrios Tzionas, Muhammed Kocabas, Mohamed Hassan, Siyu Tang, Justus Thies, and Michael Black. Human- object placement for visual environment reconstruction. In CVPR, 2022.
Hongwei Yi, Hao P Huang, Shashank Tripathi, Lea Hering, Justus Thies, and Michael J Black. Mime: Human-aware 3d scene generation. In CVPR, 2023.
Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Shaoli Huang, Yong Zhang, Hongwei Zhao, Hongtao Lu, and Xi Shen. T2m-gpt: Generating human motion from textual descriptions with discrete representations. In CVPR, 2023a.
Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208. 15001, 2022.
Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: Retrieval-augmentte motion diffusion model. arXiv preprint 2:X 2023b.
Kaifeng Zhao, Shaofei Wang, Yan Zhang, Thabo Beeler, and Siyu Tang. Compositional human-scene interaction synthesis with semantic control. In ECCV, 2022.
Wenlin Zhuang, Congyi Wang, Jinxiang Chai, Yangang Wang, Ming Shao, and Siyu Xia. Mu- sic2dance: Dancenet for music-driven dance generation. ACM TOMM, 18(2), 2022.
A RELATED WORK
Recent years have witnessed numerous studies on conditional human motion synthesis. One one hand, it can be classified according to conditions, such as text-to-motion (Petrovich et al. 2021 Guo
al.
2022
Tevet al. 2022 Hong al. 2022 Petrovich al., [022;] Athanasiou et al. 2022
Zhang et al.
2022 2023a b). voice-to-gesturg (gg et al. [2022] Li et al. 222 Ao et al. 2022 Y Yang
al. 2023a) and music-to-daaned (Li al.
2021b 2022 Siyao al. 2022 Le et al. 2023). On the
other hand, it can be categorized based on the type of interaction, including single- single-human motion
synthesis (Huang et al.
2021 Li et al., 202Tb) Siyao et al., 2022 Zhang et al. 2022 2023b Gong 2023). group human motion synthesis (Le et al. 2023 Vendrow et al. 2022
al. 22223] Jiang et al.
human-scene interaction (Zhao et al. 2022 Yi et al. 2022 2023). human-objec interaction (Jain
Liu 2009; Ghosh et al. 2023 Li et al.) 2023). ) and human- numann-human human interaction (cmu Men et al. 2022
Shafrr et al.
2023 Liang et a.. 2023).
A.1 MUSIC-TO-DANCE
Datasets.
Music-to-dance represents a crucial category within the realm of conditional human motion synthesis, where music serves as the input generate a sequence of human body movements. Previous studies have made significant strides in this field by developing several datasets (Tang et al.
2018 Mahmood et al.) 2019 Li al. 2021b: Valle-Péres et al.) [021] Cai et al. [021] Li et al. 2022
Cai et al. 2022 Yang et al. 2023b Le et al. 2023) either compiled through motion capture (MoCap)
systems reconstructed from videos using automated algorithms. AIST++(Li al. 2021b) is
13

---

Published as a conference paper at ICLR 2024
dataset derived from multi-camera videos spanning 10 dance genres and 30 subjects, compiling total of 1408 sequences or 5.2 hours of content. Le et al. (023) consists of 16.7 hours of paired music and 3D motion gathered from in-the-wild videos, featuring 7 dance styles and 16 music genres. Despite the advantage of an automated approach in terms of large-scale dataset collection, the quality of reconstruction is often compromised, particularly when occlusions present or human poses are complex. To capture high-guality motions, professional motion capture equipment is utilized. Dance Melody (Tang et al. 2018) employs the MoCap of 3D skeletons from dancers, establishing a music-to-danee dataset with four types of dance. PMSD (Valle-Pére) et al. 202I) uses an Optitrack system comprising 17 cameras to collect 142 minutes of casual dancing and 44 minutes of street dance. Notably, DanceForrer (Li et al. 2022) employs animators to manually create high-quality dancing data. It took approximately 18 months for the animators to produce the 300 dance animations, which underscores the scalability challenges inherent in this approach. However, these datasets do not incorporate strong interactions between individuals, rendering them unsuitable for either dance accompaniment or duet dance generation tasks.
Existing Methods. With the rapid development of datasets and benchmarks, numerous new methods have emerged in this field. To generates 3D dance with input music, Dance Revolution (Huang et al. (021) formalizes music-conditioned dance generation as a seuuencceto-sqqueenn learning problem and devise a seq2seq architecture. FACT (Li et al. 202Tb) incorporates deep cross-modal transformer block with full-attention. Bailando (Siyao et al. 2022 proposes a two-stage procedure, where the first stage applies VQ-VAE to quantize dancing-style poses and the latter stage employs an on-policy actor-crtic GPT to generate dancing motions, improving temporal coherency with music. GDanceR (Le et al. 2023) is recently proposed to produce coherent group dance, with transformer music encoder and a group motion generator. The existing methods, however, focus on either synthesizing solo dance motion or simultaneously generating multiple agents with weak interactions, which is not suitable for the requirement to respond to preconditioned lead dancing motion.
In contrast to previous datasets and methods, we propose new task called dance accompaniment that requires conditioning on both music and the leader's 's dance movements. We collect a large-scale and diverse duet dance dataset named DD100, which consists of both dancing sequences and the music signals, and develop a baseline method on top of it, along with evaluation protocols.
A.2 HUMAN-HUMAN INTERACTION
Datasets. As the quality of single-person motion generation improves, researchers are shifting their focus the human-human interaction task. An early attempt in this direction is CMU-MoCap (cmu), which collects weak interactions like handshaking using motion capture equipment. Since then, various multi-person motion datasets have been developed, including NTU-RGBD (Liu et al. 2020). You2Me (Ng et al., 2020), and UMPM (Van der Aa et al. 201I). However, while these datasets contain multi-person motion data, they are limited in interaction type and strength. recent dataset, InterHuman (Liang et al., 2023), also captures interactions through MocCap, and labels the motions with textual descriptions. However, InterHuman does not provide music as a multi-modality condition, which is not comprehensive enough for the task of dance accompaniment.
Existing Methods. With the availability of these datasets, several methods have been proposed for generating human-human interaction motions. (Men et al. 2022) propose seq2seq GAN system that synthesizes the reactive motion of character given the active motion from another character. ComMDM (Shafir et al., 2023) devises communication block to infuse interaction between two generated motions, using pretrained diffusion- based model as a generative prior. InterGen (Liang et al. 2023) tailors the motion diffusion model to two-person interaction setting, via two cooperative transformer-based denoisers with mutual attention mechanism.
In contrast these human-human interaction datasets and approaches, our dance accompaniment introduces the music modality and requires adherence to both the rhythm of the music and the motion from another person. Moreover, due the unique nature of dance, involves long-term strong interaction between dancers.
14

---

Published as a conference paper at ICLR 2024
B MORE DETAILS
In this section introduce statistics of the motion speed in different dance types, the detailed network structures and training process of Duolando.
B. SPEED STATISTICS
Avg. speed per frame
Rool
Hands
Speed (crvrraee)
4-
3-
2-
Waltz
jive
Tango
Ballet
Samba
Rumba
Paso Dobal Quickstep
Foxtrot
Cha cha
Dance type
Figure 7: Statistics of average speed values (em per frame) for different dances. We present the speed values kinds of joints: root (pelvis), hands and feet.
We calculated the average speed for each dance type by computing the average difference in 3D coordinates between frames for the root (pelvis), hands, and feet, as shown in Figure 7 Typically, Ballet involves many spinning in place, resulting in a low overall global (root) speed, while Quickstep, known for its lightness and speed, exhibit a higher overall velocity. Meanwhile, some dances with relatively low overall movement speed but expressive hand movements (like Jive and Cha Cha), contrast with those that have smaller hand variations and primarily rely on footwork (such as Waltz and Foxtrot).
B.2 NETWORK STRUCTURES
VQ-VAEs The encoder and decoders in Section 3.1 of the main paper complied as ID temporal CNNs. The detailed structures are shown in Table The encoder is designed to downsample the 3D joint sequence 8 times into the encoding features, and therefore the learned quantized features are aware of contextual poses the dance sequence.
Separate VQVAEs are trained for the compositional upper-and-lower half bodies, respectively. The joint number of the upper half body is and that of the lower is while J the global- velocity branch.
GPT. We follow the fundamental structure of minGPT by Karpathy (022)). Specifically, the we employ 12 Transformer layers with the block size T of 30 and feature dimension of 768, while the LAT consists three Transformer layers with the look-ahead length of 29 (around seconds).
The music features
extracted from the audio signal using the public audio processing toolbox Librosa (McFee et al., 2015), including mel frequency cepstral coefficients (MFCC), MFCC delta, constant-Q chromagram, onset strength and onset beat, which 54-dimension in total, and mapped to the same dimension GPT via a learned linear layer.
B.3 TRAINING HYPER-PARAMETEE
In terms h hyper we set the codebook capacity K to 512 for all quantized items (2 down, Ithond x rhand and z²). During VQ-VAE training, segment motion sequences into 4-second (T 120) slices with a batch size 64. The encoder's 's temporal downsampling rate is set to 4, and the commitment trade-off À is 0.1. We adopt learning rate of 3 10 train the VQ- VAE for 500 epochs, with a decay of 0.1 after both the 200th and 300th epochs. Throughout the supervised training state of GPT, adopt cross-entropy loss with learning rate of 10-4 for
15

---

Published as a conference paper at ICLR 2024
Table Architectures of VQ-VAE encoder and decoders. (Section 3.1). "Conv" and "TransConv" represent ID temporal the convolution and wranspose-convllutii arguments represent the input channel number, the output channel number, the kernel size, the convolution stride, the padding size both ends of input data, and the dilation number in turn. RB denotes Residual Block. denotes the 3D joint number.
operations, respectively. and their
Dp
Input: 0, Argument:
RB(p = 9, = 9)
Residual Block
TransConv(512, 512, 4,
Input: O: Argument: d ReLU, Conv(512, 512, 3, 1. ReLU, Conv(512, 512,
RB(p = 3, 3)
TransConv($12, 512,
RB(p= 3, 3)
Output: +
TransConv5512, 512,
Encoder
7
Conv(512, J x 3, 3, 1, 1,
Input: 0, Argument: J
Output: 7
Conv(J 3,512,
DM
RB(p = 1,
Input: 0, Argument:
3
Con25512,512.4,22
RB(p = 9)
4 RB(p 3, =)
2
TransConv(512,, 512, 4.
Conv(512, 51 1)
RB(p = = 3)
RB(p = =
TransConv(512, 512,
Output:
RB = 3, d. 3)
6
TransConv(512,, 512, 4, Conv(512, J x 1,
7
Output: 7
500 epochs. For the reinforcement learning stage, apply C learning rate of 3 x 10-5 for 50 epochs. The OOD dataset D is compiled on the sequences generated by GPT conditioned on the music and leader motion in test set, without using any ground-truth follower information. During entire training process, we employ the Adam optimizer (Kingma Ba 2014) with 31 = 0.9 and B2 0.99. The training conducted on four NVIDIA Tesla V100 GPUs, taking approximately seven days in total.
C REINFORCEMENT LEARNING DETAILS
C.I DERIVATION TO ON-POLICY ACTOR- -CITTIC Loss
Here we derive the on-policy actor-critic loss LAC from the initial reinforcement learning objective. The derivation is mainly based the instruction of Levine
From an RL standpoint, GPT predicting the next token at step t is construed as an action at, which involves selecting appropriate token 2++1 from the dictionary. As such, the cross-entropy loss in the supervised stage can be reformulated as
T-1
Lsr(0)=
Σ 1=0
log (To(a)))))
(8)
while it can be reframed as For RL, the object J(0) is to maximize the total income E (r) [r(T]]
((0) = (...T()) = xx T)(T))(T))
(9)
all
where 8 denotes the weights of the policy making network, represents a string of actions T (ar)T=) and (T) is the probability I1:-0) Te(a, (t) that the policy network predicts to take such trajectory; r(7) is the total income taking trajectory Et-1 r(a,,
16

---

Published as a conference paper at ICLR 2024
Algorithm 1 Off-Policy RL in Duolando
Require: pertained GPT weights 8, OOD conditions {(r,, 2)}, Epoch number &
1:
4-0 0
2: @
3: while < do
4: generate sequence {29, tr < (m))
5: D Du 2, ()
Reuse past generated data
6:
for 2,, E do
7:
8 *m, T1,
8:
ā à (21.1, fri..r)
9:
Compute r1..7 described in Sec. 3.3 Compute Loff by Eq. (4) (main paper) 0+00 Fo. RT.
10:
11:
12:
end for
13:
c c+c+]
14:
end while
One approach to maximize J is to optimize the network weight 0 along the gradient VaJ
as
- 0 + aVJ. Since
Vene
VaTT
To V loggo
(10)
can rewrite VaJ into
VoJ= VoTo(T)r(T)
Σ To logg))
(11)
BT) Volog
Since TT(T) ITT=0 (a, we have
T
Vo log (T)
Σ
Vo log (arl8s))
(12)
Hence,
V ( (T) Vo (T(T)((T)
ETNSTo(T)
Vo log (a((ss))
r(as, st)
(13)
= E)
Vo log (a)))))))
Σ r(au, su)
For on-policy reinforcement learning, vnJ is estimated simultaneously sampled sectional trajec- tories [[a] } such that the equation above is approximated to be
T-1
V Pd
Vo log no (af, s!)
Σ
r(a $0) ==0
(14)
Note that the optimization of the policy making network for policy (a,, St) is not expected to be influenced by the past trajectories, i.e., the rewards before t. Therefore, Equation (14) is reframed as
VoJ
Vo log To (āf, s()
Σ rr(a 5h)
(15)
where ΣT-1 r(af, 89) is the expected total future income after taking a under state sy, to say, the expected "reward go" which is formally named the Q-value Q(a 8?).
17

---

Published as a conference paper at ICLR 2024
Table Ablation studies for music input.
X represents random input.
Solo Metries
Internctive Merries
Rhythmic
Method
(J)
FID,(L)
Div,(t)
Divy(1)
Diveu(t>>
CF(S)
BED(1)
BAS(1)
Grand Truth
6.56
6.37
11.31
7.61
3.41
12.35
74.25
0,5308
0.1842
Ground Truth (E3
6.56
6.37
11.31
7.61
3.41
12.35
74.25
0,5308
0.1760
Duolando
25.30
33.52
10,92
7.97
9.97
14.02
52.36
0.2858
0.2046
Duslandoo (85)
23.73
35.91
10.95
8.36
7.77
13.96
51.77
0.2582
0.2072
To avoid the bias of rewards, e-g. rewards are positive, the Q-value item in Equation (15) is normalized by an expected "reward to go" on state i.e., the critic value V, E,,, (Q(a,, 8t]]. such that
V.J
Ve log To (a s.) (Q(a), v(()))
(16)
Vo log to 8) (r(o), 8 + (( V(?)))
1=
where r(a,. ((5 is summarized an "advantage" A. Looking closely to Equa- tion (16), if A is positive, to increase J, the policy making network will be optimized to enhance the probability of action a under state gf. our proposed framework, the actions are within finite selections of the choreographic memory. Hence, the optimization along with Equation (T6) is equivalent to the the optimization via an weighted cross-entropy loss on the in-time self predictions of the policy making network:
T-1
L
log (T (To())(s)) A.
(17)
C.2 ALGORITHM PIPELINE OF PROPOSED OFF-POLICY RI.
The detailed pipeline of proposed off-policy RL is shown in Algorithm
D MORE ABLATION STUDIES
D. MUSIC
To explore the case when the music and the leader are not aligned. we randomize the music input during the phase. The experimental results are shown in Table + where the random music input are noted with X. Interestingly, the BAS of Duolando (%) even increases by 0.0026 (1.3%) compared to that from the correct musics, while BED score decreases by 0.0276 (10%), implying the framework.
music signal have significant impact on the rhythm of generated follower's 's motion in our proposed
D.2 LOOKING AHEAD
We conduct ablation study the looking-ahead mechanism, and train variant "Duolando w/o. RL LA" where the LAT is substituted to single linear layer to map each token's 's embedding to the same dimension LAT (see Table As the effect of looking ahead, compared with "Duolando w/o. RL", the variant without LAT shows salient drop on FID; (26.01, 20%) while the interactive metrics do not significantly change (less than 7%). We observe that lacking the ability look ahead could lean to some sudden changes in the motion flow, making the movement less smooth.
D.3 SKATING EFFECT
To assess the severity of the skating effect in generated follower motion, we introduce a new metric, the skating ratio (SR), which represents the proportion of frames exhibiting skating. Specifically, the speeds the motions of both legs (represented by the vector from the ankle to the pelvis) less than threshold X1, and the global speed root (pelvis) is greater than threshold X2,
18

---

Published as a conference paper at ICLR 2024
Table 5: Ablation study on looking ahead (LA).
Solo Metries
Interactive Metrics
Rhythmic
Method
FIDx(U)
FID,(I)
Divu(t)
Div,(t)
FID.a(I)
Divca(7)
CR())
BED(1)
BAS(1)
Ground Truth
6.56
6.37
11.31
7.61
3.41
12.35
74.25
0.5308
0.1839
Duolando w/o. LA
132.73
40.22
12.94
9,92
55.81
0.2775 0.2795
0.2121 0.2193
Duolando w/o. RL
106.72
34.10
13.88
7.05
21.68
9.33
57.43
Table 6: Skating ratio (SR) of different methods. The SR is calculated as the percentage of the number of frames with negligible leg movement but significant global displacement.
Method
SR (% 4)
Ground Truth
0.00
Bailando (Siyao et 2022
0.00
EDGE (Tseng el al. 2023
0.06
Duolando w/o. RL tr IC
0.00
Duolando w/o. RL
0.01
Duolando RL LA
1.24
Duotando w/o. RL
1.06
Duolando
0.33
consider this count skating frame. It's worthy note that in calculating global movement, plane to account since there exist reasonable vertical shifts
only consider movement on the (e.g., jumping or lifting) that don't need stepping. Based on the statisties in Figure 7 we set Xi to 1 cm per frame and to 3 em per frame. As shown in Table 6 due to the high quality of mocap collection, the ground truth statistically does exhibit skating. Meanwhile, the SRs of methods like Bailando, "Duolando w/o. RL fr IC" and "Duolando w/o. RL fr" that directly predict displacement based on steps, are also near to zero. contrast, after introducing ir, the frequency of skating steps in "Duolando w/o. RL" increases to 1.06%. Despite the follower's position being more coordinated with the leader in this variant, the skating effect becomes a significant drawback. To address this issue, employ RL to align the lower body movement global shift. As result, the SR was significantly reduced by 0.73 (69%), demonstrating the effectiveness of the proposed off- off-policy RL.
E USER STUDY SCREENSHOTS
We conduct our user study through Google Form. The screenshot shown in Figure
F DATA LICENCE
Code and data will be publicly available upon acceptance. The dataset contains the dance sequences in SMPL-X (Pavlakos et al. 2019) format stored in .py files and the corresponding music clips in mp3 format. To protect performers' facial information, do not release the original raster videos following the contracts with performers.
G VIDEO DEMO
We provide data examples and generated results in the demo.
19

---

Published as a conference paper at ICLR 2024
Duolando User Study2
accompaniment
given
of the leed dancer, to generate the movements of the back-up dancer. The left and right the results generated by different methods, which the gray lead dancer real human's dancer generated. Please watch one you think is better. Please measure from the degree of poordinstion/iteeaccto of the (when meeting similar degrees
motton
clips choose which
dance (priority) coordination).
quality
Indicates required question
Which one is better?
left better
right better
Figure 8: Screenshot of user study.
20


Reviews:
[{"review_id": "3dNmnM5nJO", "reviewer": "Reviewer_cNc4", "summary": "This paper proposes a new task and motion captured dataset related to dance generation, specifically, the task of generating dance “accompaniments”. In this task, the model’s job is to generate the follower’s motion from that of the leader, potentially enabling VR/AR experiences where users dance with virtual avatars. This paper also proposes a GPT-based method to address this task which treats dance accompaniment as a sequence-to-sequence language modeling task to convert leader dance tokens and musical context into follower dance tokens.", "questions": "- Why use copyrighted music for capturing this valuable dataset as opposed to copyright-free audio?\n- Why not pre-train models on noisy dance data?\n- There doesn’t appear to be any details about the music tokenization strategy used in this work - can the authors clarify how music features are represented?\n- I was confused by the use of lookahead - would the lookahead model actually work in a real-time dance setting?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "This paper has several strengths including **introducing a novel task and dataset**, **proposing well-designed methods with solid evaluation**, and **it is well-written**.\n\n**Novel task and dataset**. The idea of generating dance accompaniments is interesting, and the authors expend extraordinary effort and expense to create a novel dataset for this task. Based on the supplementary material, this dataset appears to my eyes to be of extremely high quality thanks to the use of fine-grained motion capture. I have no doubt that this dataset will constitute a valuable resource to the growing dance generation research community.\n\n**Well-designed methods and evaluation**. This paper poses dance accompaniment as a sequence-to-sequence “text generation” problem by learned tokenizations of dance motion. Despite the inevitable methodological complexity that comes with dealing with high-dimensional dance data, this approach is _overall_ satisfyingly straightforward and reasonable. Moreover, the authors construct reasonable quantitative evaluation metrics for their method and also conduct a user study, achieving impressive performance relative to baselines (but less impressive than the ground truth) and also effectively ablating additional elements of their approach (e.g. RL).\n\n**Well-written**. This paper is also very well-written, with remarkable clarity both in its conceptual presentation and formalized notation. I was able to follow the details quite well despite being a newcomer to working with dance and motion capture data.", "weaknesses": "This paper also has some weaknesses including **output quality issues**, **potential copyright issues with the dataset**, and **limited reusability of insights**.\n\n**Output quality issues**. While reasonable in design, the proposed method produces fairly rigid dance accompaniments that, subjectively speaking, vaguely resemble someone dancing with a lifeless mannequin (especially in contrast to extraordinary richness of the ground truth accompaniments). While the authors make a valiant effort to improve results w/ RL, the best system still falls far short. It seems like progress here is more likely to be driven by large-scale pre-training from noisier dance datasets followed by adaptation to small high-quality mocap datasets, rather than training from scratch on mocap datasets.\n\n**Potential copyright issues with the dataset**. Perhaps moreso than the proposed methods, the DD100 dataset may be the most valuable aspect of this work to the dance generation community. This paper promises to release “MP3s” associated with the dataset but fails to report details about the copyright status of the music in the dataset. Listening to the demo videos, the dataset appears to feature copyrighted material (e.g. “Charlie Puth - LA Girls”). It is likely that, even if the authors release the audio, they will likely be forced to take it down eventually, compromising the value of the dataset to the research community.\n\n**Limited reusability of insights**. There is not a lot of information in this paper that would be of interest to researchers outside of the dance generation community. Perhaps there is something reusable happening in the use of RL to refine GPT models, but this is only explored within the context of dance generation. Though ICLR does occasionally have dance generation work in its proceedings, I suspect that this paper overall will not be particularly interesting to the broader ICLR community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h9VA3kLwNn", "reviewer": "Reviewer_otBs", "summary": "- This paper proposed an interesting human-human interaction task called dance accompaniment generation, which aims to generate human dancing motions conditional on a leader dance and background music. \n- Identifying the need for a dataset for this new task, the author proposed a mo-caped duet dancing dataset, containing 1.92 hours of duet dancing, with an average time for each clip over 1 minute. The dataset is collected with 20 120-FPS optical cameras and meta gloves for motion capture, and it contains 10 distinct genres of duet dances. SMPL-X is fitted for this dataset.\n- This paper proposed a pure kinematics-based method for conditional dance generation. It first tokenizes the dancing motion with 5 VQ-VAE, then trains a GPT-like arch to generate dancing motions autoregressive. To reduce the misalignment of lower-body motion and global root translation, the author used off-policy RL to finetune the model.\n- For experimental evaluations, the author proposed a set of simple metrics for evaluating the follower's interaction with the leader and the background music. The metric is inspired by the Beat-Align Score, where the beat time of dance is defined as the local minimum of velocity. Besides metrics, a user study is also conducted. The author showed better performance of its proposed algorithm than other adopted baselines.", "questions": "- Why use 2D skeletons for user study? Why not use 3D-rendered motions, like the one in the demo video? \n- What is the training time for each of the stages? \n- A general question is how to evaluate the quality of the motion-capture? I watched the video, and it seems there are some flickering frames and some exaggerated hand motions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The algorithm proposed does not change the whole landscape of the kinematics-based motion generation, but it has some novelty, especially the second-stage off-policy tuning. Additionally, the authors also collected a high-quality duet dancing dataset, making this paper more sound. \n- For the evaluation part, it is thorough to design three separate metrics and also a user-study.\n- The writing, tables, and figures are clear and easy to follow. For example, table-1 cleary summerize the unique of the collected dataset. Figure-4 shows the autoregressive generator quite clearly. The details of the methods are well-documented in the supplementary parts.", "weaknesses": "I only have some minor comments on the weakness part.\n\n- I found a major weakness in the off-policy RL part. For a pure kinematics-based method, there might exist a couple of artifacts, such as floor-skating, penetration between bodies and ground, and other physical-infeasible dynamics. The videos in the supplementary materials also showed such artifacts. The paper only addresses the problem of misalignment of lower-body motion and full-body motion; why the author just stopped here? it seems that other artifacts might also be alleviated through similar RL tuning. Has the author tried to address such artifacts? Does the author believe such off-policy tuning combined with some heuristic reward function can reduce most artifacts?\n- The author mentioned using off-policy RL to reduce the artifacts of misalignment of lower and full-body motion. It would be beneficial to add some new metrics to evaluate this specific improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QzlPUgPj2e", "reviewer": "Reviewer_CR8y", "summary": "The paper presents a method for generating dance accompaniment with respect to a leader dancer dancing under a music piece. The method consists of a series of VQVAEs for encoding body motions, music encoder, and a GPT for autoregressive modeling the conditions and follower tokens. In addition, an off-policy RL method is employed to improve the generation quality. In terms of the dataset, the authors collect a duet dance dataset from professional dancers along with the music.", "questions": "See weakness.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The paper presents a novel task, generation of a dance follower conditioned on the other dance along with the music.\n\n2. The author contributes a dataset and the motion part is accurately captured using MoCap sensor.", "weaknesses": "1. How is music represented? Midi or spectrogram or raw waveform? I didn’t see where the paper defined the representation of it.\n\n2. An important ablation study is missing, what if the music is not fed as input. I think it still makes sense to generate a follower dancer as the rhythm of music is already embedded in the leader dancer?\n\n3. The RL setup aims to handle the OOD data for unseen dance motions, and the authors present an off-policy learning approach. However, I didn’t see ablations against on-policy RL nor ablations on the reward design, which looks very ad-hoc to me.\nThe supplementary material provides videos for original dataset and generated samples. However, I find some samples where actions are not well-aligned with music beats at all, I doubt whether it is something performed by professional dancers. Also, from the generated samples, I saw that sometimes part of a body model passing through part of another body model in an unnatural manner.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0IzM7akLsa", "reviewer": "Reviewer_JwGL", "summary": "This work introduces a novel task in 3D dance generation called dance accompaniment. The task involves generating responsive movements from a dance partner, or follower, synchronized with the lead dancer's movements and the underlying musical rhythm. The authors also introduce a large-scale and diverse duet interactive dance dataset called DD100, recorded from professional dancers' performances. A GPT-based model named Duolando is proposed, which predicts motion sequences based on coordinated information from music, leader movements, and previous follower sequences. The model is further enhanced with an off-policy reinforcement learning strategy to generate stable results in unseen conditions.", "questions": "See weaknesses. \n\n(1)\tMore details on data processing and Figure 3.\n\n(2)\tExperiment to explore the importance of signal. \n\n(3)\tQuestions in writing.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "(1)\tThe paper introduces a new task called dance accompaniment.\n\n(2)\tThe DD100 dataset containing 115.4 minutes is collected for task training and evaluation. \n\n(3)\tThe proposed Duolando model utilizes a GPT-based approach to predict follower movements based on coordinated information from music, leader movements, and previous follower sequences.\n\n(4)\tThe incorporation of an off-policy reinforcement learning strategy enhances the model's ability to generate stable results on unseen conditions.", "weaknesses": "This work introduces a new dance accompaniment task, which is interesting and promising for dance generation community. I have some questions and suggestions. \n\n(1)\tTo gain a better understanding of the differences between various dance styles within the dataset and across datasets, it would be beneficial to provide more data statistics, such as movement speed and action distribution. More details on data processing would be helpful to understand it. \n\n(2)\tIn Equation 1, it's stated that both p and M are inputs, but in Figure 3, only p is shown as an input. \n\n(3)\tIt would be interesting to explore which signal, between music and leader motion, has a more dominant influence. For instance, when swapping the music between two test cases in the test set, and using music(B) with leader motion(A) and music(A) with leader motion(B) as inputs, how does this affect the results?\n\n(4)\tThere is some ambiguity in the terminology between \"duet dance generation\" and \"dance accompaniment.\" It's important to differentiate them more clearly in the draft. Additionally, it's worth noting that this dataset potentially holds promise for duet dance generation, where only music serves as input to generate movements for two individuals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)