How Does it Sound?
Authors: Kun Su
Xiulong Liu
Eli Shlizerman
Venue: NeurIPS 2021 Poster (2021)
Keywords: audio-visual learning
video to music
applications
multi-modality

Abstract: One of the primary purposes of video is to capture people and their unique activities. It is often the case that the experience of watching the video can be enhanced by adding a musical soundtrack that is in-sync with the rhythmic features of these activities. How would this soundtrack sound? Such a problem is challenging since little is known about capturing the rhythmic nature of free body movements. In this work, we explore this problem and propose a novel system, called `RhythmicNet', which takes as an input a video which includes human movements and generates a soundtrack for it. RhythmicNet works directly with human movements by extracting skeleton keypoints and implements a sequence of models which translate the keypoints to rhythmic sounds.
RhythmicNet follows the natural process of music improvisation which includes the prescription of streams of the beat, the rhythm and the melody. In particular, RhythmicNet first infers the music beat and the style pattern from body keypoints per each frame to produce rhythm. Next, it implements a transformer-based model to generate the hits of drum instruments and implements a U-net based model to generate the velocity and the offsets of the instruments. Additional types of instruments are added to the soundtrack by further conditioning on the generated drum sounds. We evaluate RhythmicNet on large scale datasets of videos that include body movements with inherit sound association, such as dance, as well as 'in the wild' internet videos of various movements and actions. We show that the method can generate plausible music that aligns well with different types of human movements.

How Does it Sound? Generation of Rhythmic Soundtracks for Human Movement Videos
Kun Su
Xiulong Liu +
Eli Shlizerman
RhythmicNet
Video With Human Movements
Rhythmic Music
Figure I: RhythmicNet: Given an input of silent human movement video, RhythmicNet generates soundtrack for it.
Abstract
One the primary purposes of video to capture people and their unique activities. It is often the case that the experience of watching the video be enhanced by adding a musical soundtrack that is in-sync with the rhythmic features of these activities. How would this soundtrack sound? Such problem is challenging since little known about capturing the rhythmic nature of free body movements. In this work, explore this problem and propose novel system, called 'RhythmieNet', which takes as an input a video with human movements and generates soundtrack for it. RhythmicNet works directly with human movements, by extracting skeleton keypoints and implementing a sequence of models translating them to rhythmic sounds. RhythmicNet follows the natural process of music improvisation which includes the prescription of streams of the beat, the rhythm and the melody. In particular, RhythmicNet first infers the music beat and the style pattern from body keypoints per each frame to produce the rhythm. Next, it implements a transformer- based model generate the hits of drum instruments and implements U-net based model to generate the velocity and the offsets of the instruments. Additional types instruments are added the soundtrack by further conditioning on generated drum sounds. We evaluate RhythmicNet on large scale video datasets that include body movements with inherit sound association, such as dance, as well as 'in the wild' internet videos of various movements and actions. We show that the method can generate plausible music that aligns with different types of human movements.
1 Introduction
Rhythmic sounds are everywhere, from raindrops falling on surfaces, to birds chirping, to machines generating unique sound patterns. When sounds accompany visual scenes, they enhance the percep- tion of the scene by complementing it with additional cues such as semantic association events, means of communication. drawing attention to parts of the scene, and many more. For visual scenes
'These authors contributed equally.
I DDeparmment of Electrical & Computer Engineering. University of Washington, Seattle, USA. Department of Applied Mathematics, University of Washington, Seattle, USA
8 Corresponding author: Ahlizee@uw.ce
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Drum Performance
Rhythm
Rhythm
Rhythm2Drum
Drum2Music
Human Activity Video
Keypoints
RhythmicNet.
Rhythmic Music
Figure 2: System Overview of RhythmicNet. Keypoints are extracted from human activity video and are processed through Video2Rhythm stage to generate the rhythm. Afterwards Rhythm2Drum converts the rhythm to drum performance. In the last step, Drum2Music component adds additional instrument tracks on top of the drum track.
that include activity of people, rhythmical music that is in-sync with the rhythm of body movements can emphasize the actions of the person and enhance the perception of the activity (22. Indeed, to support such synchrony, usual practice is that a musical soundtrack is chosen manually in professionally edited videos.
Drum instruments serve as the fundamental part in music by generating the underlying leading rhythm patterns. While drum instruments vary in shape, form, and mechanics, their main purpose is to set the essential rhythm for any music. Indeed, drums known to have existed from around 6000 BC, and even beforehand there were instruments based on principle of hitting two objects and generating sounds 33. On top of drum patterns, additional instruments add secondary patterns and melody, creating rich multifaceted music. In modern music, in composition and improvisation, is also the case that composers would start a musical piece by designing the rhythm for the corresponding drum track. As the piece evolves, additional accompanying instruments tracks are gradually superimposed on top of the drum track produce the final music.
Inspired by the possibility of associating rhythmic soundtracks to vidcos, in this work we explore automatic generation of rhythmic music correlated with human body movements. We follow similar music composition and improvisation steps as in music improvisation by first generating the rhythm of the music that is strongly correlated with the beat and movements patterns. Such rhythm can then be then used to generate novel drums music accompanying the body movements. With the rhythm being inferred, we follow further steps of music improvisation and add new instruments (piano and guitar) tracks to enrich the music. In summary, we address the challenge of generating a rhythmic soundtrack for human movement video by proposing a novel pipeline named 'RhythmicNet', which translates human movements from the domain of video to rhythmic music with three sequential components: Video2Rhythm, Rhythm2Drum, and Drum2Music.
In the first stage of RhythmicNet, given a human movement video, we extract the keypoints from the video and use ppatio-temporal graph convolutional network 4 in conjunction with transformer encoder [5] to capture motion features for estimation of music beats. Since music beats are periodic and there are various visual changes occurring in human movements, we propose an additional stream, called the style, which captures fast movements. The combination of the two streams constitutes the movements rhythm and guides music generation in the next stage, called Rhythm2Drum. This stage includes an encoder-decoder transformer that given the rhythm, generates the drums performance hits and U-net [6] which subsequently generates drums velocities and offsets. We find that these two stages are critical for generation of quality drum music. In the last stage, called Mrum2Music, we complete the drum music by adopting an encoder-decoder architecture using transformer-Xi [ to generate music track of either piano or guitar conditioning on the generated drum performance. An overview of RhythmicNet is shown in Fig. 2 Our main contributions are: (i) To the best of our knowledge, we are the first to generate a novel musical soundtrack that is in-sync with human activities. (ii) We introduce an entire pipeline, named 'RhythmicNet', which implements three stages to complete the transformation. (iii) RhythmicNet is robust and generalizable. Experiments on datasets of large-scale dance videos and 'in the wild' internet videos show that music generated by RhythmicNet will be consistent with human body movements in videos.

---

2 Related Work
Generation of sounds for a video is a challenging problem since it aims to relate two signals that are indirectly correlated. It belongs to the class of problems of Audio- Visioo--iillll learning, which deals with exploration and leveraging of the correlation of both audio and video for tasks such as audio- visual correspondence [ 9
10, [I] video sound separation [12 13, 14 [5], audio-visual event
localization (16), transformations of audio to body movements 18 19], lips movements [20] and talking faces [21
22 23 Audio-visual systems are usually developed by using multi-modal
learning techniques which have been shown effective in action recognition [24] 25], speech question
answering 226 27 28 29 30), 3D world physical simulation 31,, and medical images analysis [32
33
34
35
36].
Several approaches were proposed for the relation of sounds to a video. A deep learning approach showed the potential of such application by proposing recurrent neural network to predict the audio features of impact sounds from videos. The approach was able to produce a waveform from these features B7. In a subsequent work, a conditional generative adversarial network was proposed to achieve cross-modal audio-visual generation of musical performances
(38) In both methods, single image was used as an input, and the network performed supervision on instrument classes to generate a low-resolution spectrogram. Concurrently, for natural sounds, Sample RNN-based method [39] has been introduced to generate sounds such as baby crying, water flowing, given a visual scene. This approach was enhanced by audio forwarding regularizer that considers the real sound as an input and outputs bottle-cccked sound features which provide stronger supervision for natural sound predictions only from visual features 44]. Compared to natural sounds with relatively simple characteristics, music contains more complex elements. While such problem is more challenging, the possibility to correlate movement and sounds was shown by rule-based sensor system which succeeded to convert sensed motion to music notes [41.
In recent years there has been remarkable progress in the generation of music from video. An interac- tive background music synthesis algorithm guided by visual content was introduced to synthesize dynamic background music for different scenarios 42). The method, however, relied on reference music retrieval and could not generate new music directly. Direct music generation approaches have been developed for videos capturing musician playing an instrument. A ResNet-based method was proposed to predict the pitch and the onsets events, given video frames of top-view vidcos of pianists playing the piano [3). Later, Audeo 44 demonstrated the possibility transcribe video to high-quality music. While the results of such methods are promising, the generation is limited to a single instrument. Thereby, Foley Music 45
network to generate Midi events from body keypoints and achieved convincing synthesized music from Midi. Further, Multi-instrumentaiiss Net (46) showed generation of music waveform of different instruments in an unsupervised way. While these approaches demonstrate the possibilities of generating music from videos, the videos need to contain solid visual cues such as instruments indicate the types of music being generated. It still remains unclear whether it is possible to generate music when such visual cues do not exist. With respect human movement, this would be extracting the characteristics of the movement and attempting to match music with them. In this regard, a recent novel approach of dance beat tracking was proposed 47 The approach is aimed at detecting the characteristics of musical beats from a video of a dance by using visual information only. Inspired by this work, we design novel methodology to estimate in a precise way musical characteristics, such as beats, from movements and utilize them to improvise new music.
proposed
There has been vital recent progress in the generation of music from its representations, such as symbolic representations, as well. In particular, Musical Instrument Digital Interface (Midi) representation has been shown to be useful in modeling and generating music. Initial works converted Midi into piano-roll representation and used generative adversarial networks (48) or variational autoencoders 44) 50 to generate new music. A limitation of the piano-roll is that may result in memory inefficiency when the length of the music is too long. In order to address this limitation, event- based representation has been proposed and was shown be useful and efficient representation in modeling music [5] 52, 53]. While the event-based representation enabled models to obtain convincing generated results, it lacks metrical structure, leading to unsteady beats in the generated samples. Thereby, recently, new representation called Remi was proposed to impose a metrical structure in the input data so that the models can include awareness of the har--ba--hrse hierarchical structure in the music [54]. In our work, we utilize the Remi representation by converting the Midi into
3

---

Cony
Prodicted Beats
ST-OON
Atlention
Fonward
Softine
Layer
Layer
Koypetmss
Velocity
Directogram
Knnmmtti Offset
STFT
Threshold
Rd-
Style
Rhythm
Figure 3: Detailed schematics of the components in the Video2Rhythm stage.
Remi the Drum2Music stage. While the methods mentioned above generate unconditional music, it was shown to be possible to constrain music generation. For example, it was proposed to constrain generative models to sample for predefined attributes [55]. Systems such as Jukebox [56] and MuseNet (57) showed the possibility of generating music based user preferences which correspond to network model specifically trained with labeled tokens as a conditioning input. Furthermore, a Transformer autoencoder has been proposed aggregate encoding of Midi data across time obtain a global representation of style from given performance. Such a global representation can be used to control the style of the music (88) Additional models have been proposed, such model capable of generating kick drums given conditional signals including beat, downbeat, onset of snare and Bass (59]. In RhythmicNet, conditioning additional music instruments on the drum track is expected to provide a richer soundtrack. For this purpose, in the Drum2music stage, we utilize the Transformer autoencoder and consider the drum track as the conditioning input and generate the track of another musical instrument, such as piano or guitar.
3 Methods
RhythmicNet includes three sequential components: 1) Association of rhythm with human movements (Video2Rhythm), 2) Generation of drum track from rhythm (Rhythm2Drum), 3) Adding instruments to the drum track (Drum22usic). We describe the details of each stage below.
Video2 Rhythm. We decompose the rhythm into two streams: beats and style. We propose novel model to predict music beats and a kinematic offsets based approach to extract style patterns from human movements.
Music Beats Prediction. Beat is a binary periodic signal determined by fixed tempo, and it obtained by music beat prediction network, which learns the beat by pairing body keypoints with ground truth music beats in a supervised way. To predict regular music beats from human body movements, we extract 2D skeleton keypoints via the OpenPose framework (6)) and perform first-order difference to obtain the velocity for each video. Motion sequences is considered as three dimensional tensor X E RVxTx2 X'xxx2 where is the number of keypoints, T is the number of frames, and the last dimension indicates the 2D coordinates. We formulate the prediction of music beats as a temporal binary classification problem: Given the skeleton keypoints X, we aim to generate the output with the same length Y E R where each frame is classified into 'beat' (y 1) or 'non-beat' (y = 0).
We encode the keypoints using spatio-temporal graph convolutional neural network (ST-GCN) [4] Such encoding represents the skeleton sequence as an undirected graph G (V, E), where each node U/ E V corresponds to a key point of the human body and edges reflect the connectivity of body keypoints. The sequence passes through a spatial GCN to obtain the features at each frame independently, and then temporal convolution is applied to the features to aggregate the temporal cues. The encoded motion features are then represented as P = AXWsWW E R XT.xx. where X is the input. A E VVVV is the adjacency of matrix of the graph defined based the body keypoints connections. Ws and Wr are the weight matrices of spatial graph convolution and temporal convolution. T, and C. indicate the number of temporal dimension and feature channels. We obtain the final motion features P E TR. T.C.C. by averaging the node features.
Given the motion feature P, we use transformer encoder that contains stack of multi-head self- attention layers to learn the correlation between different frames. Due to the periodicity of the music beats, we introduce two components to allow the model to capture them more accurately: 1) We adopt a relative position encoding [6] to allow attention to explicitly resolve the distance between two tokens in sequence instead of using common positional sinusoids to represent timing information.

---

structures in This encoding is critical for modeling the timing in music where relative differences matter more than their absolute values [52] 2) We use temporal self-similarity matrix of motion features (SSM), which has been shown effective in human action recognition in regularization of the transformer and counting the repetitions of periodic movements 62 64 SSM can be constructed by computing all pairwise similarities S = f(P,, P3) between pairs of frame-level motion features P, and P,- where f(-) is the similarity function. We use the negative of the squared euclidean distance as the similarity function, f fa, b) -||a bllk. followed by taking softmax over the time axis. SSM has only one channel and it goes through convolution layer S Conv(S) and then added to every attention head in the self attention component implemented as
Attention (Q, K, V) Softmax QKT +S+ R V
VD.
where Q, K, V are the standard query, key and value respectively. and R is the ordered relative position encoding for each possible pairwise distance among pairs of query and key on each head. We train the model using weighted binary cross-entropy loss that puts more weight toward the beat category to address imbalances.
In comparison with previous work 477, the combination of graph representation, relative self- attention and SSM components enables the model to better capture the body dynamics which allows for more accurate beat estimation.
The output of the network is the beat activation function; i.e., for each video frame, the model predicts its probability of being 'beat' frame. To obtain beat positions, we apply an algorithm based on HMM decoding proposed in 655)
Style Extraction. While beats represent the monotonic periodic pattern occurring at fixed time intervals (i.e. periodic signal), there are additional a-periodic components in the rhythm. In particular, between two music beats, there are typically various irregular movements that contribute to the rhythm. In contrast to beats, these patterns are inconsistent and it is unclear how to systematically extract such patterns from visual information. We, therefore, define an additional stream, called style, which records incidences of transitional movements of the human body, such as rapid and sudden movements. For prediction of such events, we apply a rule-based approach since the definition of style is implicit and there is no data to learn mapping from body keypoints transitional movements. The style is defined as a binary stream that indicates transition time points as and non- -on-transitional time points O. We compose the style stream by implementing several steps based spectral analysis of kinematic offsets of the motion (66]. The first step is to compute kinematic offsets, Kinematic offsets are 1D time series signal representing the average acceleration of the human body over time. To obtain kinematic offsets, we calculate the directogram of the motion by factoring it into different angles. Given F.(j.)) t) as the velocity magnitude of joint j at time i,, we formulate the directogram Dtt, 0) [67] as:
{ 0
0 Φ S 2
D(t,0) = Σ [(GG0)))0(4GG))) where Lo(d)
(1)
otherwise
The indicator function 1 (0) is used to distribute the motion of all joints into Noins angular intervals. Then the first-order difference of the directogram is calculated to obtain the acceleration of motion across different angles. The mean acceleration in the positive direction measures motion strength (i.e., the larger the value, the more remarkable in motion strength) and corresponds to the kinematic offsets.
Once kinematic offsets are obtained, in the next step we perform Short-Time- Foureer Transform (STFT) on them to identify peaks in the change of acceleration. The highest frequency bin in STFT (out of 8) represents the most profound transitions in the signal and we use the highest frequency bin to extract the style patterns from motion. The peaks are defined as 10% top magnitudes over the duration of the video. We mark the timepoints of the peaks as 1 and other timepoints as Since STFT results with low temporal resolution (due to hop-size set to 4 for efficient computation) we upsample the binary signal by the hop size to obtain a binary signal that matches the resolution of the video. The output signal is re-sampled to have the same sampling rate as the music beats.
Rhythm Composition. We obtain the rhythm by adding the streams of the beats and the style into single signal. The rhythm should correspond to the correlation of body movements with the tempo of the soundtrack.
5

---

Velocities
Encoder- Decoder Transformer
U-Net
Rhythm
Hitls oulput
Offsets
Figure 4: Detailed schematics of the components in the Rhythm2Drum stage.
Target track memory
Remi Drum Sequence
Drum Target Track
Remi Target track output
Engoder
Decoder
Drum Track Midi
Drum brack memrory
Target Track Midi output
Figure 5: Detailed schematics of the components in the Drum2Music stage.
Rhythm2Drum. The stage Rhythm2Drum interprets the provided rhythm from previous stage into drum sounds. In this stage we follow the GrooveVAE setup [50], where each drum track can be represented by three matrices: hits, velocities, and offsets. The hits represent the presence of drum onsets and is binary matrix H E IR NxT, where N is the number of drum instruments and T is the number of time steps (one per 16-th note). The velocities is continuous matrix, V. that reflects how hard drums are struck, with values in the range of [0, 1J. The offsets O is also a continuous matrix and stores the timing offsets, with values the range of [-0.5, 0.5). These values indicate how far and in which direction each note's timing lie, relative to the nearest 16-th note. The matrices V O and have the same shape.
Given the input rhythm sequence Y E IX we aim to generate the H, V and O. In contrast to Groove VAE [50), which models all three matrices simultancously with multiple losses, we model H, V. and O smoothly in two steps using the combination of an encoder-decoder transformer [5] and U-net ( In the first step, the binary rhythm is passed an input to the transformer encoder. In the decoder, the H matrix is converted into word a sequence defined by a small vocabulary set of all possible combinations of hits, and is mapped back to a binary matrix for the final output. We observe that autoregressively learning the hits H as word sequence, the transformer can generate more natural and diverse drum onsets. We train the transformer with the cross-entropy loss. In the second step, we add style patterns (velocity and offsets) to the onsets. Since H has the same shape as V and O, we can consider it as a transformation between 2 images of the same shape. To achieve such transformation, we adopt U-net (6) to take the onset matrix I/ as an input and to generate V and O. We use Mean-Square Error (MSE) loss for U-net optimization. Finally, we convert the generated matrices H, V and O to the Midi representation to produce the drum track.
Drum2Music. In this last stage we add further instruments to enrich the soundtrack. Since the drum track contains rhythmic music, we propose to condition the additional instrument stream on the generated drum track. Specifically, we propose an encoder-decoder architecture, such that the encoder receives the drum track as an input, and the decoder generates the track of another instrument. We consider the piano or guitar as the additional instruments, since these are dominant instruments. We use Remi representation (54) to represent multi-track music. Compared to the commonly-used Midi- like event representation (52), the Remi representation includes information such as Tempo changes, Chord, Position, and Bar, which allow our model to learn the dependency of note events occurring at same positions across bars. For both the encoder and decoder, we adopt the transformer- XL network model which extends the transformer by including the recurrence mechanism [7]. The recurrence mechanism enables the model to leverage the information of past tokens beyond the current training segment and to look further into the history.
The encoder contains stack of multi-head self-attention layers. Its output E, can be represented as: F; = Enc(x;, FF),, where M, is the encoder memory used for the i-th bar input and the encoder hidden state sequence computed in previous recurrent steps. Similarly, in the decoder, the prediction of j-th token of the i-th bar Vi.j is formulated as yi.j Dec(yi.e. M,P. E,), where Vi,ck are the previously generated tokens in the same bar, MP is the decoder memory used for i-th bar, and E; the corresponding encoder output of the same bar. The decoder consists of stack of layers with casual self-attention, coss-attention to the encoder output, and feed beedfforward network.

---

ModrisiMetrics
CML. (%)
CML; (%)
Cem (%)
(%6
TCN 47
44.97
45.15
48.14
63.04
TF
16.07
16.24
32,85
46.90
ST-GCN
54,89
55.45
49.23
64.78
TT
61.89
62.34
55.09
71.93
ST-GCN+TF+SSM
63.20
63.58
57.72
73.07
ST-GCN+TE-RelAuu
68.01
68.31
59.19
74.67
ST-GCN+TT+SSMARRI 71.43
(+26.46%)
71.94
(+26.79%)
61.59 (+13.45%)
75.79 (+12.75%)
Table I: Music beat prediction evaluation. The abbreviation of each component stands for: TF (transformer), ST-GCN (spatio-temporal graph convolutional network), SSM (Self-similarity Matrix), RelAttn (Relative Attention); F (F-score measure), Cem (Cemgil's 's score), CMLg (Correct metrical level continuous accuracy), CML, (Correct metrical level total accuracy). Bold font indicates the
best value.
Modrl\Metric (lower better)
NDB
MSE Velocity
MSE Offsets
Groove VAE [50]
46
0.0437
0.0402
TF multi-outputs w.o. hits sequence
44
0.0507
0.0348
TF multi-outputs w. hits sequence
39
0.0493
0.0369
TF w. hits sequence + Unet
39 (1.15%
0.0267 (140%) 0.0169 (158%)
Table 2: Rhythm2Drum performance evaluation. Abbreviations stand for: TTF (encoder-decoder
transformer), Multi-outpure (Predict the hits, velocities and offsets simultancously), w./.oo. hits sequence (whether using word tokens to represent the hits). Bold font indicates the best value.
For the training data, we split the music piece into segments with total number of bars. In the encoder, for recurrent step i, we provide the i-th bar of drum performance 2; to the transformer-X We adopt a teacher forcing strategy and feed the ground truth tokens into the decoder to generate the next tokens. We minimize the negative log- likelihoood (NLL) between generated tokens and ground truth tokens optimize the model. During inference, the drum track is given to the encoder for each bar and the tokens in the decoder are generated one by one. Finally, we use the temperrature-contoolles stochastic top-k sampling method (68 to randomly generate new music track.
Experiments & Results
Datasets. We use the AIST Dance Video Database, large-scale collection dance videos in 60fps for training and testing of Video2Rhythm (69]. This database includes 10 street dance genres, 35 dancers, camera viewpoints, and 60 musical pieces covering 12 types of tempo. For each genre, we 1080 dance videos, resulting in total of 10, 800 videos. We split the samples into train/validate/test sets by 1/0.. based on the dance genres, dancers, and camera ids.
For Rhythm2Drum, use the Groove Midi dataset [50] which contains 1150 Midi files and over 22, 000 measures of drumming. We split the data into 0.8/ of train/validate/test sets.
For Drum2Music, we extract two subsets of Lakh Midi dataset 770 to separately train Drum2Piano and Drum2Guitar models. For Drum2Piano, we select the Midi files that contain both tracks drums and acoustic piano with at least 16 bars, and we consider 16 bars to be single segment. This results 9994//1944/194 segments for train/validateetest sets respectively. For drum2guitar, we perform a similar selection to obtain 12904/717777 segments for train/validate/tsst sets respectively.
Implementation Details. We use Pytorch 7 implement all models in RhymicNet with two Titan X GPUs. For all videos, we extract 17 keypoints of body joints. In Video2Rhythm, the network contains a 10-layer ST-GCN and 2-layer transformer encoder with 2-head attention. For the style extraction part, the motion sequence is down-sample to 15fps to calculate the kinematic offsets. The number of bins used for the directogram 12, and 16-point FFT with hop-size of is applied to extract candidate styles. Each detected style is repeated 4 times to match the hop size, and then up-sampled to original time resolution of 60fps and re-indexed in unit of quarter note based on estimated tempo. In Rhythm2Drum, the Hits transformer includes 3 layers, and 4 heads in both the encoder and the decoder. The vocabulary size of the decoder input is 152, consisting of possible combinations of types of drum hits in the dataset. U-net that generates velocity and offistts contains
7

---

Metries
PC/bar
Pi
101
PCH
NLH
NLL 4
Dataset (Piano)
5.48
6.16
0.31
Drum2Piano W.O. memory
7.17
4.63
0.12
0.63
0,52
0.77
Drum2Piano
6.82
5,86
0.14
0.63
0.54
0.53
Dataset (Guitar)
5.33
5.51
0.22
Drum2Guitar w.o. memory
3.54
8,94
0.52
0.56
0.46
0.58
Drum2Guitar
5.63
5.69
0.13
0.64
0.51
0.40
Table Drum2Music evaluation. For PC/bar, PI, IOI values, the closer to the dataset the better. For PCH and NLH values, the larger. the better.
4 down-sample blocks with channel sizes of 16, 32, 64, 128. In Drum2Musie, the model consists of a recurrent transformer encoder and a recurrent transformer decoder. We set the number of encoder layers, decoder layers, encoder heads and decoder heads to 4, 8, 8, and respectively. The length of the training input tokens and the length of the memory is 256. We provide additional configuration details in the supplementary materials. Code. System setup and code are available in Github repository 5
Video2Rhythm Evaluation. Following the rubrics proposed for musical beat tracking 72 we compute the performance in terms of F-score measure, Cemgil's score (Cem), and Correct Metrical Level continuity required/not required (CML.e/) score. To compare with existing approaches, we implement a baseline temporal convolutional network (TCN) for beat prediction [47]. The comparison and ablation results are shown in Table []] The best method of Video2Rhythm (ST- GCN+TF-SSM-RRIAttm) significantly outperforms the baseline model in all metries by a large margin. In particular, the continuity scores outperform the baseline model by more than 25%, indicating the estimated beat sequence is significantly more consistent.
Rhythm2Drum Evaluation. We use several metrics to evaluate Rhythm2Drum. For measuring the diversity of the generated drum hits, we adopt the Number of Statisticall--Diffrreen Bins (NDB) metric proposed and used in 73
74 45). To compute NDB, we cluster all training examples into k = 50 Voronoi cells by K-means. The generated examples are then assigned to the nearest cell. NDB reported as the number of cells where the number of training examples statistically significantly different from the number of generated examples by a two-sample Binomial test. For each model, we generate 9000 samples from the testing set and perform the comparison. For evaluation of velocities and offsets, we compute the MqaaeS Error (MSE) for the test set. We compare our methods with the baseline model Groove VAE (50). The results are shown in Table The results show that using hits sequences to generate the drum track enables a more diverse set of samples such that the next U-net component, which generates velocities and offsets, in turn will generate more realistic drumming sounds.
Drum2Music Evaluation. To evaluate the generated piano and guitar tracks, we use objective metrics such as PC/bar (pitch count per bar), PI (average pitch interval), IOI (average inter-onset interval) described in 775) For these metrics we compare the statisties calculated on the test dataset and on the generated music. For additional metrics of PCH (pitch class histogram) and NLH (note length histogram), we calculate the overlapping area (OA) between the statisties on the test dataset and the generated music for each sample and report the average of them. In addition, we compare the NLL loss based on the validation set. The numerical results are shown in Table We compare two versions (with and without using memory) to show the effectiveness of the recurrence mechanism. Our results show that for both Drum2Piano and Drum2Guitar with recurrent encoder-decoder transformer (i.e. with memory), the NI.. loss is lower for the validation set and the statistics of the generated samples are much closer to the test dataset than the no-memory counterpart.
In the wild Experiments and Qualitative Evaluation. In Fig. 6 we show set of examples of generated soundtracks for video clips AIST dataset and 'in the wild' clips downloaded from Youeube. For the AIST dataset, we generate and compare tracks of predicted beats with the ground-truth(GT) beats. The predicted and GT tracks appear to be in close agreement. We then demonstrate the extracted style and its correspondence to frames which exhibit special movements. The beat and style tracks constitute the thythm from which the waveform of drum music is generated. To test and demonstrate the generality of RhythmicNet we apply it to videos clips of various human activity. Examples of such
8

---

Style
Beat
Beat
Beat
Beat
Style
Break Dance (AIST)
Pred
beat
Style Rhythm
House Dance (AIST)
Ice Skating (in the wild)
Pred Beat
Style
Rhythm
Drum
Football
(in the wild)
Guitar
Figure 6: Examples of generated beat and style streams and corresponding audio waveforms for dance (AIST videos) and 'in the wild' videos. Dark Blue: predicted beats, Red: ground truth beats, Green: extracted style, Purple: rhythm, Light Blue: audio waveform of generated drums. Supplementary Materials include additional examples and sounded video clips.
activitics are shown in Fig. and include Ice Skating and Playing Football. We provide additional examples and videos clips along with the soundtracks in the Supplementary Materials. The generated rhythms are well synchronized with the videos and the drum track appears to be in-sync with the activities. In addition, demonstrate the generated waveform additional instrument (guitar) in Fig. 6 Additional instruments indeed provide a richer music that accompanies the movements.
Human Perceptual Evaluation of Soundtrack Music. In addition to the objective evaluation of the different components of RhythmicNet we also performed human perceptual surveys using Amazon Mechanical Turk. These surveys were intended to evaluate the effectiveness of RhythmicNet gener- ated soundtracks to align with the movements and the extent that the generated soundtrack enhance the overall perception of the video compared with various soundtrack controls. Since RhythmicNet ultimately targets in the wild videos, for which there are no given background soundtracks we three surveys that focused on these videos. For all surveys, background RhythmicNet was given to the participants to avoid perceptual biases. We surveyed 85 participants individually, where each participant (850 segments in total) along with different generated soundtracks.
the survey
asked evaluate 10 videos each with around 10 seconds
In the first survey, we asked people (non-experts) to choose the video that they prefer, including video without soundtrack and 3 variations of soundtracks generated by our approach (drums-only or drums with another instrument). Results in Table clearly indicate preference of video

---

Soundtrack Preference
Soundtrack
Drums Only
Drums + Piano
Drums + Guitar
votes
7.3%
31.2%
32.1%
29,4%
Table 4: Soundtrack preference.
Soundtrack match to the video
Soundtrack match to the video (Ablation)
Random
Video2Rhythm
Video2Rhythm + Rhythm2Drum
Random
Shuffle
RhythmicNet
Groo VAE
+ Vro
30.8%
27.8%
41.4%
23.3%
33.3%
43.4%
Table 5: Soundtracks match movements in the video.
with soundtrack. Furthermore, interestingly, preference for which instruments are included in the track split almost equally between the provided variations, with slight preference for tracks with Drums+Piano.
In the second survey, we asked people watch the same human activity video with different soundtracks and answer the question: "In which video the sound best matches the movements?". The given options of the soundtracks were generated soundtracks with Random, Shuffle and RhythmicNet rhythms. The Random drum track was generated with Rhythm2Drum method with a random rhythm with 50% chance be ON or OFF at each time step. The chance of 50% was chosen such that there is significant probability that the a rhythm that sounds like real rhythm will be sampled. We found that sampling with lower probability would generate rhythms that do not sound well at all. The Shuffle drum track was generated with Rhythm2Drum method but the order in the rhythm is shuffled. RhythmicNet option corresponded the drum track generated with Rhythm2Drum method. From results shown in [[lff we observe clear indication that the drum tracks generated with our method are chosen to be the best match to the movements more frequently (41.4% (Ours) v.s. 30.8% (Random) and 27.8% (Shuffle)).
In an additional survey, we performed a perceptual ablation study to test how the two components, Video2Rhythm and Rhythm2Drum, influence the perception of the soundtrack compared to baseline approaches. Survey results shown in Taigh(]rright) and suggest that in comparison to the baseline these two components significantly improve the perception of the soundtrack.
5 Conclusion and Discussion
In this exploratory work, we have considered creative task of automatically generating novel rhythmic soundtracks consistent with human body movements captured in a video. Our results show that RhythmicNet pipeline is able to achieve this creative task and generate soundtracks that align with movements and enhance the perception of them when the video is being watched. At its core, RhythmicNet defines and implements a systemic approach of soundtrack generation by following the process of music improvisation in which a rhythm of movements is established and is translated to drumming music with potentially additional accompanying instruments. We foresee future potential applications in video creation and editing, which RhythmicNet can pave the way to unlock. As features for music generation we have chosen body keypoints, while is unclear which features are most informative for music generation. For human body movements, body keypoints strongly correlated with movements, and in addition, body keypoints are efficiently computed per each frame. In terms of limitations and future enhancements of the current setup of RhythmicNet, novel components will need to be considered to address the transition from rhythmic drum track to a full bodied music with symphony of instruments, since currently the addition of a single instrument (piano or guitar) to the drum track is implemented. Furthermore, enabling RhythmicNet to operate in real-time would allow the music to be interactive with people and their movements. However this may require a more computationally extensive generative approach. Due to the fact that the main cue for the generated music is human body movements, one possible concern could be that the soundtrack generated with RhythmicNet could be used to manipulate the original sounds of the video, and to create a fake impression of people activity. Also additional concern is that generated music could sound too familiar to the music on which the models in RhythmicNet have been trained. These are common concerns in the application of generative models. Failure in RhythmicNet may bring up an unsatisfying soundtrack but we do not expect serious consequences from this.
10

---

References
[1] Jeremy Montagu. How music and instruments began: a brief overview of the origin and entire development of music, from its earliest stages. Frontiers in Sociology, 2:8, 2017.
[2] Bjom H Merker, Guy Madison, and Patricia Eckerdal. On the role and origin isochrony in human rhythmic entrainment. Cortex, 45(1):4-17, 2009.
[3] James Blades. Percussion instruments and their history. Bold Strummer Limited, 1992.
[4] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. arXiv preprint arXiv:/801.0744 2018.
[5] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008, 2017.
[6] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assister intervention, pages 234-241. Springer, 2015.
[7] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoe Le, and Ruslan Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv: 02860, 2019.
[8] Relja Arandjelovic and Andrew Zisserman. Look, listen and learn In Proceedings of the IEEE International Conference on Computer Vision, pages 609-617, 2017.
[9] Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In Advances in neural information processing systems, pages 892-900, 2016.
[10] David Harwath, Antonio Torralba, and James Glass. Unsupervised learning of spoken language with visual context. In Advances in Neural Information Processing Systems, pages 1858-1866, 2016.
[11] Andrew Owens, Jiajun Wu, Josh H McDermott, William Freeman, and Antonio Torralba. Ambient sound provides supervision for visual learning, In European conference computer vision, pages 801-816. Springer, 2016.
[12] Ruohan Gao, Rogerio Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. In Proceedings of the European Conference on Computer Vision (ECCV), pages 35-53, 2018.
[13] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In Proceedings of the European conference on computer vision (ECCV), pages 570-586, 2018
[14] Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In Proceedings of the IEEE International Conference Computer Vision, pages 1735-1744, 2019.
[15] Chuang Gan. Deng Huang, Hang Zhao, Joshua B Tenenbaum, and Antonio Torralba. Music gesture for visual sound separation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10478-10487, 2020.
[16] Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, and Chenliang Xu. Audio-visual event localization in unconstrained videos. In Proceedings of the European Conference Computer Vision (ECCV), pages 247-263, 2018.
[17] Eli Shlizerman, Lucio Dery, Hayden Schoen, and Ira SenellaacherSShizeemnn Audio to body dynamics. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7574-7583, 2018.

---

[18] Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, and Jitendra Malik. Learning individual styles of conversational gesture. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3497-3506, 2019.
[19] Hsin-Ying Lee, Xiaodong Yang, Ming- Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, and Jan Kautz. Dancing music. In Advances in Neural Information Processing Systems, pages 3586-3596, 2019.
[20] Supasorn Suwajanakorn, Steven M Seitz, and Ira Kemelmacher-SShizzrman Synthesizing obama: learning lip sync from audio. ACM Transactions Graphics (TOG), 36(4):1-13, 2017.
[21] Amir Jamaludin, Joon Son Chung, and Andrew Zisserman. You said that?: Synthesising talking faces from audio. International Journal of Computer Vision, 127(1 (7711-12):1667-177) 2019.
[22] Tae-Hyun Oh, Tali Dekel, Changil Kim, Inbar Mosseri, William T Freeman, Michael Rubinstein, and Wojciech Matusik. Speech2face: Learning the face behind voice. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7539-7548, 2019.
[23] Yang Zhou, Xintong Han, Eli Shechtman, Jose Echevarria, Evangelos Kalogerakis, and Dingzeyu Li. Makelttalk: speaker-aware talking-head animation. ACM Transactions on Graphics (TOG), 39(6):1-15, 2020.
[24] Ruohan Gao, Tae- the-H Oh, Kristen Grauman, and Lorenzo Torresani. Listen to look: Action recognition by previewing audio. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10457-10467, 2020.
[25] Xiang Long, Chuang Gan, Gerard De Melo, Xiao Liu, Yandong Li, Fu Li, and Shilei Wen. Multimodal keyless attention fusion for video classification. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[26] Chenyu You, Nuo Chen, and Yuexian Zou. MRD-Net: Multi-Modal Residual Knowledge Distillation for Spoken Question Answering. IJCAI, 2021.
[27] Chenyu You, Nuo Chen, and Yuexian Zou. Self-supervised contrastive cross-modality represen- tation learning for spoken question answering. arXiv preprint arXiv:2109.0338// 2021.
[28] Nuo Chen, Fenglin Liu, Chenyu You, Peilin Zhou, and Yuexian Zou. Adaptive bi-directional attention: Exploring multi-granularity representations for machine reading comprehension. In ICASSP, 2020.
[29] Chenyu You, Nuo Chen, Fenglin Liu, Dongeaao Yang, and Yuexian Zou. Towards data distilla- tion for end-to-end spoken conversational question answering. arXiv preprint arXiv:2010.08923, 2020.
[30] Chenyu You, Nuo Chen, and Yuexian Zou. Knowledge distillation for improved accuracy in spoken question answering. In ICASSP, 2021.
[31] Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, et al. Threedworld: A platform for interactive multi-modal physical simulation. NeurIPS, 2021.
[32] Cheayu You, Qingsong Yang, Lars Gjesteby, Guang Li, Shenghong Ju, Zhuiyang Zhang, Zhen Zhao, Yi Zhang, Wenxiang Cong, Ge Wang, et al. Structurally-sensttiee multi-scale deep neural network for low-dose CT denoising. IEEE Access,
2018
[33] Chenyu You, Guang Li, Yi Zhang, Xiaoliu Zhang, Hongming Shan, Mengzhou Li, Shenghong Ju. Zhen Zhao, Zhuiyang Zhang. Wenxiang Cong, et al. CT super-reoolution GAN constrained by the identical, residual, and cycle learning ensemble (gan-circle). IEEE Transactions on Medical Imaging, 39(1): 111-1888--3 2019.
[34] Indranil Guha, Syed Ahmed Nadeem, Chenyu You, Xiaoliu Zhang, Steven M Levy, Ge Wang. James Torner, and Punam Saha. Deep learning based high-resolution reconstruction of trabecular bone microstructures from low-resolution et scans using gan-circle. In Medical Imaging 2020: Biomedical Applications in Molecular, Structural, and Functional Imaging, 2020.
12

---

[35] Chenyu You, Linfeng Yang. Yi Zhang, and Ge Wang. Low-Dose CT via Deep CNN with Skip Connection and Network in Network. Developments in X-Ray Tomography XII, volume 11113, page 111131 W. International Society for Optics and Photonics, 2019.
[36] Qing Lyu, Chenyu You, Hongming Shan, and Ge Wang- Super-resolution mri through deep learning. arXiv preprint arXiv: (rXX:v:810006776, 2018.
[37] Andrew Owens, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H Adelson, and William T Freeman. Visually indicated sounds. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2405-2413, 2016.
[38] Lele Chen, Sudhanshu Srivastava, Zhiyao Duan, and Chenliang Xu. Deep cross-modal audio- visual generation. In Proceedings of the on Thematic Workshops of ACM Multimedia 2017, pages 349-357, 2017.
[39] Yipin Zhou, Zhaowen Wang. Chen Fang, Trung Bui, and Tamara L Berg. Visual to sound: Generating natural sound for videos in the wild. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 3550-3558, 2018.
[40] Peihao Chen, Yang Zhang. Mingkui Tan, Hongdong Xiao, Deng Huang, and Chuang Gan. Generating visually aligned sound from videos. IEEE Transactions on Image Processing, 29:8292-8302, 2020.
[41] Tamara Berg, Debaleena Chattopadhyay, Margaret Schedel, and Timothy Vallier. Interactive music: Human motion initiated music generation using skeletal tracking by kinect. In Proc. Conf. Soc. Electro-Acoustic Music United States, 2012.
[42] Yujia Wang, Wei Liang, Wanwan Li, Dingzeyu Li, and Lap-Fai Yu. Scene-aware background music synthesis. In Proceedings of the 28th ACM International Conference Multimedia, pages 1162-1170, 2020.
[43] A Sophia Koepke, Olivia Wiles, Yael Moses, and Andrew Zisserman. Sight to sound: An end-to-end approach for visual piano transcription. ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1838-1842, IEEE, 2020.
[44] Kun Su, Xiulong Liu, and Eli Shlizerman. Audeo: Audio generation for silent performance video. Advances in Neural Information Processing Systems, 33, 2020.
[45] Chuang Gan, Deng Huang, Peihao Chen, Joshua Tenenbaum, and Antonio Torralba. Foley music: Learning to generate music from videos. In ECCV, 2020.
[46] Kun Su, Xiulong Liu, and Eli Shlizerman. Multi-instrumentaliss net: Unsupervised generation of music from body movements. arXiv preprint arXiv:20/2.034788 2020.
[47] Fabrizio Pedersoli and Masataka Goto. Dance beat tracking from visual information alone. ISMIR, 2020.
[48] Weo Dong, Wen-Yi Hsiao, Li-Chia Yang, and Yi-Hsuan Yang. Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompaniment. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
[49] Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, and Douglas Eck. A hierarchical latent vector model for learning long-term structure in music. In International Conference on Machine Learning, pages 4364-4373. PMLR, 2018.
[50] Jon Gillick, Adam Roberts, Jesse Engel, Douglas Eck, and David Bamman. Learning to groove with inverse sequence transformations. In International Conference on Machine Learning (ICML), 2019.
2019.
[51] Sageev Oore, Ian Simon, Sander Dieleman, Douglas Eck, and Karen Simonyan. This time with feeling: Learning expressive musical performance. Neural Computing and Applications, 32(4):955 2020.
13

---

[52] Cheng-Zhi Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer: Generating music with long-term structure. In International Conference on Learning Representations, 2018
[53] Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck. Enabling factorized piano music modeling and generation with the maestro dataset. arXiv preprint arXiv:
2018.
[54] Yu-Siang Huang and Yi-Hsuan Yang. Pop music transformer: Beat-based modeling and generation of expressive pop piano compasitions. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1180-1188, 2020.
[55] Jesse Engel, Matthew Hoffman, and Adam Roberts. Latent constraints: Learning to generate conditionally from unconditional generative models. arXiv preprint arXiv:1711.05772 2017.
[56] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint wrXiv:2005.00341 2020.
[57] Christine Payne. Musenet. OpenAl Blog. 2019.
[58] Kristy Choi, Curtis Hawthome, lan Simon, Monica Dinculeseu, and Jesse Engel. Encoding musical style with transformer autoencoders. arXiv preprint arXiv:1912.05537, 2019.
[59] Stefan Lattner and Maarten Grachten. High-level control of drum track generation using learned patterns of rhythmic interaction. In 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), pages 35-39. IEEE, 2019.
[60] Zhe Cao, Gines Hidalgo. Tomas Simon, Shih-En Wei, and Yaser Sheikh. Openpose: realtime multi-pen piriiipesonn 2d pose estimation using part affinity fields. arXiv preprint arXiv: 18/2.088//00000 2018.
[61] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position repre- sentations. arXiv preprint arXiv: 1803.0255 2018.
[62] Marco Körner and Joachim Denzler. Temporal self siff-iimi for appearance recognition in multi-view setups. In International Conference on Computer Analysis of Images and Patterns, pages 163-171. Springer, 2013.
sopearance-based based action
[63] Chuan Sun, Imran Nazir Junejo, Marshall Tappen, and Hassan Foroosh. Exploring sparseness and self-similarity for action recognition. IEEE Transactions on Image Processing, 24(8):2488- 2501, 2015.
[64] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Counting out time: Class agnostic video repetition counting in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10387-10396, 2020.
[65] Florian Krebs, Sebastian Bück, and Gerhard Widmer. An efficient state-space model for joint tempo and meter tracking. In ISMIR, pages 72-78, 2015.
[66] Peter Grosche, Meinard Müller, and Frank Kurth. Cyclic tempogram a mid-level tempo representation for musicsignals. In 2010 IEEE International Conference on Acoustics. Speech and Signal Processing, pages 5522-5525. 2010.
[67] Abe Davis and Maneesh Agrawala. Visual rhythm and beat. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pages 2532 2535, 2018.
[68] Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. Curl: A conditional transformer language model for controllable generation. arXiv preprint 1rXiv.1909.05858, 2019.
[69] Shuhei Tsuchida, Satoru Fukayama, Masahiro Hamasaki, and Masataka Goto. Aist dance video database: Multi-genre, multi-dancer, and multi-camera database for dance information processing. In ISMIR, pages 501-510, 2019.
14

---

[70] Colin Raffel. Learning
saaning-basedd based methods for comparing sequences, with applications to audio-to- midi alignment and matching. PhD thesis, Columbia University, 2016.
[71] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, sege-performacce deep learning library. arXiv preprint arXiv:1912.01 11111111177.,,. 2019.
[72] Matthew EP Davies, Norberto Degara, and Mark D Plumbley. Evaluation methods for musical audio beat tracking algorithms. Queen Mary University of London, Centre for Digital Music, Tech. Rep. C4DM-TR-09-JC 2009.
[73] Eitan Richardson and Yair Weiss. On gans and gmms. In Advances in Neural Information Processing Systems, pages 5847-5858, 2018
[74] Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Guirajani, Chris Donahue, and Adam Roberts. Gansynth: Adversarial neural audio synthesis. arXiv preprint arXiv:1902.08710, 2019.
[75] Li-Chia Yang and Alexander Lerch. On the evaluation of generative models in music. Neural Computing and Applications, 32(9))4773-4777 4784, 2020.
Funding Disclosure
Paper Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] [Yes], [No] or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] Yes. The sections of Methods and Experiments & Results clearly describe the claims we made.
(b) Did you describe the limitations of your work? [Yes] Yes. We describe the limitations of our work in the discussion section.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Yes. We discuss such impacts in the discussion section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] Yes.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A] N/A
(b) Did you include complete proofs of all theoretical results? [N/A] N/A
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [No] No. However, the code will available the Github after the review process as we mention in the main paper.
15

---

(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] Yes. We describe the training details in the implementation details section and more details can be found in the supplementary material.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] No. The error bars are not reported because would be too computationally expensive.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Yes. We describe the number and type of GPUs in the Implementation Details section.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Yes. We cite all the existing assets used in our work.
(b) Did you mention the license of the assets? [Yes] Yes. We mention the license of the assets in the supplementary material.
(c) Did you include any new assets either in the supplementary material or as URL? [No]
No. We haven't 't included any new assets.
(d) Did you discuss whether and how consent obtained from people whose data you's re using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable
information offensive content? [Yes] Yes. We discuss it the supplementary material.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text instructions given to participants and screenshots, if ap- plicable? [Yes] Yes. The Human Perceptual Test is fully described in the supplementary material.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [No] No. There is potential risk in our Human Perceptual Test.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes] Yes. We include these material in the supplementary material.
16


Reviews:
[{"review_id": "dhgttivU_3f", "reviewer": "Reviewer_vA5F", "summary": "In this paper, the authors propose to generate music based on human motion videos. The authors propose the Video2Rhythm to capture the  rhythmic nature of free body movements. Then they use Rhythm2Drum module to generate drum. Finally, piano and guitar tracks are added to enrich the music through Drum2Music. Experiments show that they can generate plausible music that aligns with the videos.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, I am fond of the topic and the results are indeed plausible. Though the whole paper can basically be decomposed into three independent parts with building blocks very similar with previously proposed methods such as [1,2,3]. The three modules are developed reasonably, and the design of the whole pipeline is of enough novelty for an application paper. The strengths are explicitly listed as follows:\n\n++ The topic is interesting and worth exploring.\n\n++ The supp video shows the effectiveness of the proposed method.\n\n++ The formulation and the pipeline of this paper are reasonable. \n\n++ The usage of three different datasets is clever.\n\nThe weaknesses are as follows:\n\n1. The authors could pay more attention to the writings. The descriptions in the method part are not clear enough. The captions are not detailed enough. The authors should try replacing the current png/jpg figure format with pdf.\n\na) L35 “but even beforehand”, the but is strange in this sentence. \n\nb) It would be better if the authors could re-split the “Music Beats Prediction” paragraph (starting from L130).\n\nc) The notion of “directogram” is proposed in [1], but is not mentioned.\n\nd) The representation of the “style” has never been clearly illustrated. The caption of figure 3 is too simple for a clear illustration.\n\nIn line 184: “We choose frames with the top 10% frequency magnitude as the style pattern to music beats for every second” what do you mean by “to music beats”? \n\nMy understanding is that for beats prediction, a binary prediction is given related to frames periodically. As for the style, according to the previous sentence, it seems also to be time steps associated with frames with sudden movements. I am not sure about “the output signal is re-sampled to have the same sampling rate as the music beats”. How to perform the re-sample? How are style and beat combined together?\n\ne) How is the training of the rhythm part supervised? Is there any supervision for the style pattern?\n\n2. The title of this paper is not informative enough. It somehow reminds me of “Objects that sound” or “Visually Indicated Sounds”, which correlate with general sounding objects. The authors are encouraged to change it to a one more related to motion/dance and music.\n\n3. All building blocks are designed with sophisticated models. I am not suggesting that the authors should do more meaningless ablations, but there are obviously easier solutions. The authors are encouraged to propose certain baselines for replacing, maybe the transformer structures.\n\nI will consider raising the score if the method part could be modified clearer with more details.\n\n[1] Visual rhythm and beat. TOG 2018.\n\n[2] Learning to groove with inverse sequence transformations. ICML, 2019.\n\n[3] Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. ACL 2019.\n\n---------------------------------------------------------------------------------------------------------------------------------------------------------------\n\nPlease make sure to include the clarifications and experiments in all responses to the final version.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed the negative impacts, however, the authors could include more limitations. For example, some failure cases can be shown for better illustration.\n\nThe authors claim that the code and models are proprietary, which would cast great difficult for follow-up works. As many parts are not clearly written, it might be difficult for others to reproduce the work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "24 hours"}, "extra_text": {}}, {"review_id": "d7sEzEWxy2r", "reviewer": "Reviewer_oiP8", "summary": "The authors propose a method to generate musical soundtracks for silent videos of human activity. The method is named RhythmNet and is composed of three parts: Video2Rhythm, Rhythm2Drum, and Drum2Music. These three are sequential modules which take an input silent video and generate a musical soundtrack to accompany the video.\n\nThe Video2Rhythm component predicts musical beats given the video. It uses keypoints extracted from the video and encodes the rhythm as the beat and style. The Rhythm2Drum component uses the predicted rhythm and generates a drum pattern following the setup of GrooveVAE using a transformer-based encoder-decoder architecture followed by U-net for velocity and offset prediction. Finally, the Drum2Music component adds either piano or guitar accompaniment to the drum track to make a full musical soundtrack. Each of these individual components are independently trained. ", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives:\n- The authors indeed develop a novel approach to generate music from a given video of human motion.\n- The proposed method is technically solid and the authors also present improvements over previous models for rhythm detection from video and for drum sequence generation.\n- The authors use objective evaluation metrics to evaluate the various components of their method\n\nNegatives:\n- The introduction consists of various claims about the composition process in music that need citations.\n- I understand the important role rhythm plays in music, but the paper makes it seem like drums are the only way to generate rhythm. This is not the case, one can simply play keys on a piano following a certain pattern and generate a musical rhythm. In fact, for videos in the wild, perhaps it is not a good idea to try to generate drums if the movements are not periodic since the drums track will simply sound out of rhythm.\n- While the numbers say that the proposed method is better than previous approaches, the videos shared in the supplementary material do not sound good. The rhythm is off in almost all cases and the audio is not in sync with the video.\n- While I mention earlier that the proposed method is technically solid, I’m not sure why a simple post processing of the rhythm outputs is not applied to fix some mistakes in the beat positions. Especially for the dance videos where there is a fixed tempo. Perhaps what is missing is a differentiation between beats and downbeats in the video to rhythm component.\n- The paper is not well organized especially the methods section. Please add subsections and fix the spacing between paragraphs.\n\nOverall Assessment: As mentioned, this paper demonstrates solid engineering by combining 3 independent components to approach a novel problem. While I cannot point out serious issues with the paper apart from the organization, I am not sure if the results are good enough for acceptance. Thus, the negatives slightly outweigh the positives in this paper and I rate this paper as a 4.\n\nComments:\n- I would consider adding a subtitle in the paper title. As it is, one cannot understand what the theme of the paper is.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, in the conclusion the authors briefly discuss this.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "VMPkPD9irVT", "reviewer": "Reviewer_587L", "summary": "The paper proposes a new method to produce a rhythmic sound consisting of basic instruments such as (piano roll, guitar chords and drum beat) given an input video containing at least one human activity. The authors leverage the input human action dynamics first using a keypoint estimation network and their velocity features. The latter features as used consequently to produce a rhythmic pattern which is used to generate a sequence of drum beats. The aforementioned drum beat sequence is then used as an output or as conditioning to a late-stage generation of a piano roll or a sequence of guitar chords. The intermediate representations and the architecture choices are discussed for each staged of the process. The authors provide both qualitative and quantitative results (opinion scores, video examples) for showing the effectiveness of their proposed architecture.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "First of all, I would like to mention that this paper discusses a really novel idea of generating musical patterns from human action videos and the architectural choices, as well as the experiments are extremely well described. The main limitation of the paper are the results of the proposed method as well as the evaluation metrics described in the main paper which do not capture the great potential of this paper. Please see below a more detailed review:\n\nTo the best of my understanding, the paper is very original since I am not aware of almost any work which tried to generate a musical pattern solely from human actions pertaining to dance or other random movements. I really like the idea and the possible real-world applications that this paper enables. The paper also details many experimental design choices that researchers would need to replicate the results which is a strong plus. Moreover, the many examples provided in the supplementary material is also really important for the readers to understand the performance of the proposed model. However, I have the following concerns / questions about this work:\n- I understand why the authors base their whole musical pattern generation architecture on top of the drum beat sequence generation because a fundamental aspect such as the base rhythm is a very effective way to build the generation upon (in parallel, multiple approaches in speech generation base their own approaches on first estimating pitch). However, the network could use this beat rhythm to do much more than using it to condition the generation of other basic instruments sequences. Specifically, the authors could use this rhythm as conditioning with other (probably pre-trained) networks such as Jukebox, etc. In this sense, the output could actually enjoy much richer musical textures and patterns. Have the authors tried something along those lines or they aim to do so in the future?\n- Although the keypoints and their velocity are convenient for estimating a base rhythm (e.g. drum beat) and might contain some information for the style, I would expect a simple video encoder to be able to produce representations containing much richer conditional style-information. For example, I have seen multiple videos in the supplementary material where the generated musical patterns for a hip-hop dancer and a ballerina would have somewhat similar styles (although the beat was somewhat aligned). I would expect the authors to at least discuss this future direction.\n- The authors can save space from Figures 3-5 since they are extremely sparse. Moreover, Figure 6 is nice but the authors can split the examples to 1-2 for the main text and put the rest of them in the supplementary material since they do not necessarily add up to the main point of the paper.\n- A very important omission of the main paper is the display of the qualitative assessment of the generated videos. To the best of my understanding, such an assessment is really important for generation tasks and especially for so subjective tasks as the one addressed in this paper. There is only a small evaluation like this in the supplementary material where the only question asked was about the synchronization. I do not criticize that those results are not perfect and the proposed model is compared to almost random baselines but they are very important to exist in the main paper. I would also suggest to the authors to include a subjective evaluation for the actual preference of humans of the produced musical patterns.\n- Another main problem is that there are many statements in the paper that build a really high excitement about the results which are far from perfect (e.g. Lines 20-21 “We show that the method can generate plausible music 21 that aligns well with different types of human movements.” Or Lines 66-67 “the output music generated by RhythmicNet will be consistent with the human body movements in 67 videos in large-scale dance dataset and ‘in the wild’ internet videos.”). I checked almost all of the examples provided in the supplementary material but I could easily imagine one piano roll being replaced to another video, probably showing that this first approach to the problem (still significant though) does not respect the style and of course leaving a large performance gap for future works to build on. I would strongly suggest to the authors to be more explicit with their wording if they decide to not include a subjective evaluation of the preference of the generated musical pattern over other baselines (e.g. Line 313 “soundtracks consistent with human body movements” → “drum beats and piano rolls synchronized with human body movements”).\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do vaguely discuss a few limitations of their work in the conclusion section but they can also include any of the points that I raised here (if they do not wish to address them in the rebuttal).\n\nIn my opinion, this work raises some potential negative societal impacts that the authors do not discuss. For instance, the generated music could be heavily biased against specific patterns, styles or artists as well as raising potential copyright issues (e.g. imagine that a specific generated pattern sounds extremely similar to a copyrighted song, who would get the responsibility?). ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "BCxZURjUWpW", "reviewer": "Reviewer_mrpe", "summary": "The paper proposes an approach to generate rhythmic music from a human movement video. In detail, the model is three step process: 1) Video2Rhythm: Uses the patterns in the motion of human skeleton to detect a rhythm. 2) Rhytm2Drum: Convert the rhythmic patterns to drum beats and 3) Drum2Music: Use the drum beats to update the remi representation of another instrument (or multi-instrument song). These three steps are trained independently in a supervised learning paradigm.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n- The paper proposes an interesting and potentially useful task of generating aligned music/beats for rhythmic human body motion\n- The approach involves training three modules that can be trained separately. An advantage here is that this approach does not require a singular dataset with all the output modalities. Instead multiple and independent datasets can be used to train these three modules which can be later connected in a sequential pipeline.\n- The encoder for body poses is a spatio-temporal graph convolutional network which can potentially model the hierarchy of the connections of the human body skeleton.\n\n## Weaknesses\n- A key contribution here is to use a self-similarity matrix as a prior to detect repetitions over time. A similar, and more general, approach was also proposed in [P1] which can detect repetitive patterns over time and it can do so in a self-supervised manner, unlike the proposed approach in this paper. \n- Another important modelling step seems to be style extraction (l170). While the motivation to extract style seems reasonable, the effect of obtaining kinematic offsets (as a proxy for style) could benefit from more rigorous experimentation.\n- The modelling choices for the Drum2Music module have not been compared with existing work such as [P2] in the paper\n\n## Originality\nOverall the methods for each separate module are well-known. The combination is somewhat unique for a task of generating music aligned with human body motion. Although the task itself is not new [P3], data-driven learning has not yet been done for this particular task.\n\n## Quality\nThe paper is generally well supported with experiments except for some missing relevant comparisons in literature. See weaknesses section for a more detailed review.\n\n## Clarity\nThe paper is clear and easy to understand. The only issue in terms of clarity was the section about style extraction. The definition and its importance in the pipeline weren't very clear. Maybe an example here could have made the definition more clear and an additional ablation study to study its impact would have been helpful\n\n## Significance\nWhile it is interesting to see a neat application of three well-studied tasks when applied together, the work presented in this paper is mostly incremental.  \n\n[P1] - Dwibedi, Debidatta, et al. \"Counting out time: Class agnostic video repetition counting in the wild.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n[P2] - Lattner, Stefan, and Maarten Grachten. \"High-level control of drum track generation using learned patterns of rhythmic interaction.\" 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2019.\n\n[P3] - Berg, Tamara, et al. \"Interactive music: Human motion initiated music generation using skeletal tracking by kinect.\" Proc. Conf. Soc. Electro-Acoustic Music United States. 2012.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the paper has a discussion section explaining the limitations and potential societal impact this work could have.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "1HltUuPqpXQ", "reviewer": "Reviewer_rRNM", "summary": "The authors propose a RhythmicNet that can generate a musical soundtrack associating with human movements in a video. Unlike most existing works that aim to generate sync sounds of audio sources in videos, this work focuses on generating musical rhythm and beat along with visual motions. It mainly consists of three parts: Video2Rhythm, Rhythm2Drum, and Drum2Music. Experiments can validate that the proposed RhythmicNet can generate plausible soundtracks for different body movements. \n\n***Post-Rebuttal***\n\nThe rebuttal has successfully addressed my major concerns. Thus, I would like to keep my positive rating. The authors should revise the main paper by improving writing and adding new results and discussions provided in the rebuttal.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n+ The rhythmic sound generation problem is interesting. Although several components, such as ST-GCN, Transformer, MIDI, and Key points have been used in previous visual-to-music sound generation methods, the proposed whole three-stage system for rhythmic sound generation is novel. \n\n+ Extensive ablation studies are conducted for each main module. The experimental results show that the proposed modules are better than baselines. \n\n+ The provided demo with video examples is a nice complement to demonstrate the effectiveness of the proposed method. \n\nCons: \n\n- The method section is relatively hard to follow since texts with symbols and equations are densely presented (see page 4).  I would like to suggest that the authors divide the section into subsections with subsubsections to better illustrate each component in the proposed modules. \n\n-  The proposed domain knowledge-inspired three-stage network can generate plausible sounds from body movements. However, it still requires some annotations to facilitate the learning process (e.g., Video2Rhyth), which makes it impossible to learn the model using large-scale unlabeled videos. I am wondering whether we can directly learn the video-to-drum or even video-to-music mapping without separated Video2Rhythm, Rhythm2Drum, and Drum2Music. We definitely can do it as previous methods. But, the question is whether they can achieve comparable results as the proposed method. Did the authors conduct any experiments to check results from a one-stage framework? \n\n- Considering this is a three-stage system, it would be good if the authors can provide some failure examples for each stage and demonstrate how each stage affects final generated music sounds. \n\nOverall, the investigated problem is interesting and the proposed method is novel and technically sound. I only have some minor questions and hope the authors can address them during rebuttal. \n\n  \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors discussed limitations and potential ethic concerns. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received 4 positive reviews and 1 negative review. In the rebuttal, the authors have addressed most of the concerns. AC feels this work is very interesting and deserves to be published on NeurIPS 2021. The reviewers did raise some valuable concerns (e.g., including more human studies) that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make other necessary changes. 

Decision: Accept (Poster)