Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: In this paper, the authors propose to generate music based on human motion videos. The authors propose the Video2Rhythm to capture the  rhythmic nature of free body movements. Then they use Rhythm2Drum module to generate drum. Finally, piano and guitar tracks are added to enrich the music through Drum2Music. Experiments show that they can generate plausible music that aligns with the videos.
main_review: Overall, I am fond of the topic and the results are indeed plausible. Though the whole paper can basically be decomposed into three independent parts with building blocks very similar with previously proposed methods such as [1,2,3]. The three modules are developed reasonably, and the design of the whole pipeline is of enough novelty for an application paper. The strengths are explicitly listed as follows:

++ The topic is interesting and worth exploring.

++ The supp video shows the effectiveness of the proposed method.

++ The formulation and the pipeline of this paper are reasonable. 

++ The usage of three different datasets is clever.

The weaknesses are as follows:

1. The authors could pay more attention to the writings. The descriptions in the method part are not clear enough. The captions are not detailed enough. The authors should try replacing the current png/jpg figure format with pdf.

a) L35 “but even beforehand”, the but is strange in this sentence. 

b) It would be better if the authors could re-split the “Music Beats Prediction” paragraph (starting from L130).

c) The notion of “directogram” is proposed in [1], but is not mentioned.

d) The representation of the “style” has never been clearly illustrated. The caption of figure 3 is too simple for a clear illustration.

In line 184: “We choose frames with the top 10% frequency magnitude as the style pattern to music beats for every second” what do you mean by “to music beats”? 

My understanding is that for beats prediction, a binary prediction is given related to frames periodically. As for the style, according to the previous sentence, it seems also to be time steps associated with frames with sudden movements. I am not sure about “the output signal is re-sampled to have the same sampling rate as the music beats”. How to perform the re-sample? How are style and beat combined together?

e) How is the training of the rhythm part supervised? Is there any supervision for the style pattern?

2. The title of this paper is not informative enough. It somehow reminds me of “Objects that sound” or “Visually Indicated Sounds”, which correlate with general sounding objects. The authors are encouraged to change it to a one more related to motion/dance and music.

3. All building blocks are designed with sophisticated models. I am not suggesting that the authors should do more meaningless ablations, but there are obviously easier solutions. The authors are encouraged to propose certain baselines for replacing, maybe the transformer structures.

I will consider raising the score if the method part could be modified clearer with more details.

[1] Visual rhythm and beat. TOG 2018.

[2] Learning to groove with inverse sequence transformations. ICML, 2019.

[3] Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. ACL 2019.

---------------------------------------------------------------------------------------------------------------------------------------------------------------

Please make sure to include the clarifications and experiments in all responses to the final version.


limitations_and_societal_impact: The authors have addressed the negative impacts, however, the authors could include more limitations. For example, some failure cases can be shown for better illustration.

The authors claim that the code and models are proprietary, which would cast great difficult for follow-up works. As many parts are not clearly written, it might be difficult for others to reproduce the work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes an approach to generate rhythmic music from a human movement video. In detail, the model is three step process: 1) Video2Rhythm: Uses the patterns in the motion of human skeleton to detect a rhythm. 2) Rhytm2Drum: Convert the rhythmic patterns to drum beats and 3) Drum2Music: Use the drum beats to update the remi representation of another instrument (or multi-instrument song). These three steps are trained independently in a supervised learning paradigm.

main_review: ## Strengths
- The paper proposes an interesting and potentially useful task of generating aligned music/beats for rhythmic human body motion
- The approach involves training three modules that can be trained separately. An advantage here is that this approach does not require a singular dataset with all the output modalities. Instead multiple and independent datasets can be used to train these three modules which can be later connected in a sequential pipeline.
- The encoder for body poses is a spatio-temporal graph convolutional network which can potentially model the hierarchy of the connections of the human body skeleton.

## Weaknesses
- A key contribution here is to use a self-similarity matrix as a prior to detect repetitions over time. A similar, and more general, approach was also proposed in [P1] which can detect repetitive patterns over time and it can do so in a self-supervised manner, unlike the proposed approach in this paper. 
- Another important modelling step seems to be style extraction (l170). While the motivation to extract style seems reasonable, the effect of obtaining kinematic offsets (as a proxy for style) could benefit from more rigorous experimentation.
- The modelling choices for the Drum2Music module have not been compared with existing work such as [P2] in the paper

## Originality
Overall the methods for each separate module are well-known. The combination is somewhat unique for a task of generating music aligned with human body motion. Although the task itself is not new [P3], data-driven learning has not yet been done for this particular task.

## Quality
The paper is generally well supported with experiments except for some missing relevant comparisons in literature. See weaknesses section for a more detailed review.

## Clarity
The paper is clear and easy to understand. The only issue in terms of clarity was the section about style extraction. The definition and its importance in the pipeline weren't very clear. Maybe an example here could have made the definition more clear and an additional ablation study to study its impact would have been helpful

## Significance
While it is interesting to see a neat application of three well-studied tasks when applied together, the work presented in this paper is mostly incremental.  

[P1] - Dwibedi, Debidatta, et al. "Counting out time: Class agnostic video repetition counting in the wild." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.

[P2] - Lattner, Stefan, and Maarten Grachten. "High-level control of drum track generation using learned patterns of rhythmic interaction." 2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2019.

[P3] - Berg, Tamara, et al. "Interactive music: Human motion initiated music generation using skeletal tracking by kinect." Proc. Conf. Soc. Electro-Acoustic Music United States. 2012.
limitations_and_societal_impact: Yes, the paper has a discussion section explaining the limitations and potential societal impact this work could have.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a RhythmicNet that can generate a musical soundtrack associating with human movements in a video. Unlike most existing works that aim to generate sync sounds of audio sources in videos, this work focuses on generating musical rhythm and beat along with visual motions. It mainly consists of three parts: Video2Rhythm, Rhythm2Drum, and Drum2Music. Experiments can validate that the proposed RhythmicNet can generate plausible soundtracks for different body movements. 

***Post-Rebuttal***

The rebuttal has successfully addressed my major concerns. Thus, I would like to keep my positive rating. The authors should revise the main paper by improving writing and adding new results and discussions provided in the rebuttal.
main_review: Pros:

+ The rhythmic sound generation problem is interesting. Although several components, such as ST-GCN, Transformer, MIDI, and Key points have been used in previous visual-to-music sound generation methods, the proposed whole three-stage system for rhythmic sound generation is novel. 

+ Extensive ablation studies are conducted for each main module. The experimental results show that the proposed modules are better than baselines. 

+ The provided demo with video examples is a nice complement to demonstrate the effectiveness of the proposed method. 

Cons: 

- The method section is relatively hard to follow since texts with symbols and equations are densely presented (see page 4).  I would like to suggest that the authors divide the section into subsections with subsubsections to better illustrate each component in the proposed modules. 

-  The proposed domain knowledge-inspired three-stage network can generate plausible sounds from body movements. However, it still requires some annotations to facilitate the learning process (e.g., Video2Rhyth), which makes it impossible to learn the model using large-scale unlabeled videos. I am wondering whether we can directly learn the video-to-drum or even video-to-music mapping without separated Video2Rhythm, Rhythm2Drum, and Drum2Music. We definitely can do it as previous methods. But, the question is whether they can achieve comparable results as the proposed method. Did the authors conduct any experiments to check results from a one-stage framework? 

- Considering this is a three-stage system, it would be good if the authors can provide some failure examples for each stage and demonstrate how each stage affects final generated music sounds. 

Overall, the investigated problem is interesting and the proposed method is novel and technically sound. I only have some minor questions and hope the authors can address them during rebuttal. 

  


limitations_and_societal_impact: Yes, the authors discussed limitations and potential ethic concerns.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors propose a method to generate musical soundtracks for silent videos of human activity. The method is named RhythmNet and is composed of three parts: Video2Rhythm, Rhythm2Drum, and Drum2Music. These three are sequential modules which take an input silent video and generate a musical soundtrack to accompany the video.

The Video2Rhythm component predicts musical beats given the video. It uses keypoints extracted from the video and encodes the rhythm as the beat and style. The Rhythm2Drum component uses the predicted rhythm and generates a drum pattern following the setup of GrooveVAE using a transformer-based encoder-decoder architecture followed by U-net for velocity and offset prediction. Finally, the Drum2Music component adds either piano or guitar accompaniment to the drum track to make a full musical soundtrack. Each of these individual components are independently trained. 
main_review: Positives:
- The authors indeed develop a novel approach to generate music from a given video of human motion.
- The proposed method is technically solid and the authors also present improvements over previous models for rhythm detection from video and for drum sequence generation.
- The authors use objective evaluation metrics to evaluate the various components of their method

Negatives:
- The introduction consists of various claims about the composition process in music that need citations.
- I understand the important role rhythm plays in music, but the paper makes it seem like drums are the only way to generate rhythm. This is not the case, one can simply play keys on a piano following a certain pattern and generate a musical rhythm. In fact, for videos in the wild, perhaps it is not a good idea to try to generate drums if the movements are not periodic since the drums track will simply sound out of rhythm.
- While the numbers say that the proposed method is better than previous approaches, the videos shared in the supplementary material do not sound good. The rhythm is off in almost all cases and the audio is not in sync with the video.
- While I mention earlier that the proposed method is technically solid, I’m not sure why a simple post processing of the rhythm outputs is not applied to fix some mistakes in the beat positions. Especially for the dance videos where there is a fixed tempo. Perhaps what is missing is a differentiation between beats and downbeats in the video to rhythm component.
- The paper is not well organized especially the methods section. Please add subsections and fix the spacing between paragraphs.

Overall Assessment: As mentioned, this paper demonstrates solid engineering by combining 3 independent components to approach a novel problem. While I cannot point out serious issues with the paper apart from the organization, I am not sure if the results are good enough for acceptance. Thus, the negatives slightly outweigh the positives in this paper and I rate this paper as a 4.

Comments:
- I would consider adding a subtitle in the paper title. As it is, one cannot understand what the theme of the paper is.

limitations_and_societal_impact: Yes, in the conclusion the authors briefly discuss this.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a new method to produce a rhythmic sound consisting of basic instruments such as (piano roll, guitar chords and drum beat) given an input video containing at least one human activity. The authors leverage the input human action dynamics first using a keypoint estimation network and their velocity features. The latter features as used consequently to produce a rhythmic pattern which is used to generate a sequence of drum beats. The aforementioned drum beat sequence is then used as an output or as conditioning to a late-stage generation of a piano roll or a sequence of guitar chords. The intermediate representations and the architecture choices are discussed for each staged of the process. The authors provide both qualitative and quantitative results (opinion scores, video examples) for showing the effectiveness of their proposed architecture.
main_review: First of all, I would like to mention that this paper discusses a really novel idea of generating musical patterns from human action videos and the architectural choices, as well as the experiments are extremely well described. The main limitation of the paper are the results of the proposed method as well as the evaluation metrics described in the main paper which do not capture the great potential of this paper. Please see below a more detailed review:

To the best of my understanding, the paper is very original since I am not aware of almost any work which tried to generate a musical pattern solely from human actions pertaining to dance or other random movements. I really like the idea and the possible real-world applications that this paper enables. The paper also details many experimental design choices that researchers would need to replicate the results which is a strong plus. Moreover, the many examples provided in the supplementary material is also really important for the readers to understand the performance of the proposed model. However, I have the following concerns / questions about this work:
- I understand why the authors base their whole musical pattern generation architecture on top of the drum beat sequence generation because a fundamental aspect such as the base rhythm is a very effective way to build the generation upon (in parallel, multiple approaches in speech generation base their own approaches on first estimating pitch). However, the network could use this beat rhythm to do much more than using it to condition the generation of other basic instruments sequences. Specifically, the authors could use this rhythm as conditioning with other (probably pre-trained) networks such as Jukebox, etc. In this sense, the output could actually enjoy much richer musical textures and patterns. Have the authors tried something along those lines or they aim to do so in the future?
- Although the keypoints and their velocity are convenient for estimating a base rhythm (e.g. drum beat) and might contain some information for the style, I would expect a simple video encoder to be able to produce representations containing much richer conditional style-information. For example, I have seen multiple videos in the supplementary material where the generated musical patterns for a hip-hop dancer and a ballerina would have somewhat similar styles (although the beat was somewhat aligned). I would expect the authors to at least discuss this future direction.
- The authors can save space from Figures 3-5 since they are extremely sparse. Moreover, Figure 6 is nice but the authors can split the examples to 1-2 for the main text and put the rest of them in the supplementary material since they do not necessarily add up to the main point of the paper.
- A very important omission of the main paper is the display of the qualitative assessment of the generated videos. To the best of my understanding, such an assessment is really important for generation tasks and especially for so subjective tasks as the one addressed in this paper. There is only a small evaluation like this in the supplementary material where the only question asked was about the synchronization. I do not criticize that those results are not perfect and the proposed model is compared to almost random baselines but they are very important to exist in the main paper. I would also suggest to the authors to include a subjective evaluation for the actual preference of humans of the produced musical patterns.
- Another main problem is that there are many statements in the paper that build a really high excitement about the results which are far from perfect (e.g. Lines 20-21 “We show that the method can generate plausible music 21 that aligns well with different types of human movements.” Or Lines 66-67 “the output music generated by RhythmicNet will be consistent with the human body movements in 67 videos in large-scale dance dataset and ‘in the wild’ internet videos.”). I checked almost all of the examples provided in the supplementary material but I could easily imagine one piano roll being replaced to another video, probably showing that this first approach to the problem (still significant though) does not respect the style and of course leaving a large performance gap for future works to build on. I would strongly suggest to the authors to be more explicit with their wording if they decide to not include a subjective evaluation of the preference of the generated musical pattern over other baselines (e.g. Line 313 “soundtracks consistent with human body movements” → “drum beats and piano rolls synchronized with human body movements”).

limitations_and_societal_impact: The authors do vaguely discuss a few limitations of their work in the conclusion section but they can also include any of the points that I raised here (if they do not wish to address them in the rebuttal).

In my opinion, this work raises some potential negative societal impacts that the authors do not discuss. For instance, the generated music could be heavily biased against specific patterns, styles or artists as well as raising potential copyright issues (e.g. imagine that a specific generated pattern sounds extremely similar to a copyrighted song, who would get the responsibility?).
review_title: null

Meta-review: This paper received 4 positive reviews and 1 negative review. In the rebuttal, the authors have addressed most of the concerns. AC feels this work is very interesting and deserves to be published on NeurIPS 2021. The reviewers did raise some valuable concerns (e.g., including more human studies) that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make other necessary changes. 

Decision: Accept (Poster)