Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper presents a transformer-based conditional model to generate multitrack symbolic music. The model utilizes a "description to sequence" learning scheme. The authors show that the model produces more desirable musical sequences when human-interpretable features are incorporated during training.


strength_and_weaknesses: Strengths:

- Novel training objective
- The model can handle multi-track, multi-signature symbolic music
- Comparative experiments between strong baselines in different settings with respect to numerous evaluation measures

There are a few cosmetic issues, which are explained below.


clarity,_quality,_novelty_and_reproducibility: - The paper is well-written and very easy to follow. The approach, experiments and results are clearly presented, and the appendices are very well prepared. 
- The literature review is comprehensive.
- The model is built on past work, yet it's sufficiently novel
- The code and model weights are shared. The implementation details (Appendix A) and extended data representing (REMI+) provide the readers with enough detail to reproduce the model.

Minor issues:
- At the start of page six: "macro overlapping area (MAO)" -> should have been MOA
- Hyperlinks to websites (e.g. SoundCloud, Google Colab) should be more explicit, e.g. as shortened URL
- Please cite a work describing "mode collapse"
summary_of_the_review: The work advances the state-of-the-art. The usage of learned and expert descriptions is sound. The experiments are comprehensive, and the dataset is large and representative enough (for music information research). The authors also extend the existing data representation in existing datasets. Given these strengths, I would like to recommend the paper for acceptance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper describes a music generation approach which aims to extend the type of conditioning variables typically used in this area of research. The authors describe this as so-called description-to-sequence learning. Inspired by recent progress in text-conditioned image generation, the authors propose to derive descriptions of music in terms of musical attributes which can be easily extracted from the data itself. These descriptions are termed as expert descriptions. In addition, to fill in the gaps in the expert descriptions per bar, they also propose to learn descriptions using a VQ-VAE. Combining the two: expert and learned descriptions, the authors learn a sequence to sequence model transforming said descriptions to musical sequences. The proposed method is compared against competing methods using both objective and subjective evaluations. In all cases, FIGARO outperforms the baseline approaches. The authors also carry out various ablation studies showing the benefit of different types of expert descriptions.
strength_and_weaknesses: Strengths:
- The proposed method is elegant in combining domain knowledge of music to extract attributes along with a VQ-VAE to learn latent descriptors of measures which are fed to a transformer encoder-decoder architecture to generate musical sequences.
- The approach extends the fine-grained control of previous approaches to more musical dimensions thus affording more control in the output.
- The results, both qualitative and quantitative look and sound good.

Weakness:
- The contributions seem a bit incremental. While it's true that this has not been done before in symbolic music, the setup is pretty similar to Jukebox with the addition of some extra musical attributes as conditioning minus the prior to generate conditioning latents/descriptions. In fact, the absence of the prior feels like a big weakness, but this is left as future work by the authors.

clarity,_quality,_novelty_and_reproducibility: The paper quality is good and writing is very clear. The work is also original, although I feel it lacks novelty a bit.
The authors have shared the code in the supplementary material and it seems like they plan to share it online as well.
summary_of_the_review: My overall impression of this paper is that it presents a simple yet effective method for controllable music generation. My only issue is that it feels a little incomplete and does not extend existing literature to the extent I would have hoped. Despite that, the results sound convincing and the user study shows that this method outperforms some competing approaches. Thus I am rating it as a weak accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper describes a system for training a multi-instrument symbolic music generation model conditioned on two types of features: human-interpretable attributes that can be extracted programmatically, and latent VQ-VAE encodings that are learned.  Both feature types can be used to exert fine-grained control over a piece of generated music, as demonstrated in several experiments.
strength_and_weaknesses: Strengths
-----------
1) The paper does exactly what it claims: proposes a system for controlling multi-instrument music generation with human-interpretable and learned latent features, and evaluates the technical aspects of the system.

2) The samples on SoundCloud sound quite reasonable.

Weaknesses
--------------
1) Far and away the biggest thing missing here, even though the authors may consider it out of scope for ICLR, is some kind of user study demonstrating that this method of controlling a music generation system is helpful to musicians or composers, expert or novice.  The experiments in the paper demonstrate that the system *obeys* the controls and produces plausible music, but not that it is usable.  And based on the baseline models, it isn't surprising that the model would obey the conditioning.  Much of the effectiveness of such a system seems like it would depend on exactly what attributes are used for expert conditioning, and this question isn't explored beyond measuring the impact of dropping out individual attributes, nor is there discussion of why these attributes were chosen.

2) Two major weaknesses of the system that I suspect limit its real-world utility are its lack of support for a) infilling and b) missing controls.  Even though the conditioning signal is temporally fine-grained, a change to the conditioning at time T1 may have effects at time T2 distant from T1, which is likely not what users want.  And having to provide two types of conditioning values (one with multiple attributes) at all positions in a piece of music is likely to be very cumbersome for users.

3) Minor issues:
    * The paper mentions that an online demo is available; however no link is provided.
    * "GitHub" is misspelled in the footnote on page 2.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, original if somewhat incremental, seemingly experimentally sound, and I believe I could reproduce it with some effort.
summary_of_the_review: The paper seems correct and is a valid research contribution to the area of music generation.  I sincerely hope the authors (or someone else) writes another paper that explores the HCI questions relevant to the system presented in this paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: FIGARO is a model trained on a seq2seq task, where the input is either a human-interpretable high-level description of the music or a learned sequence of VQ-VAE tokens. Output is a symbolic music composition in the proposed REMI+ format, an extension to REMI that adds multi-instrument and multi-time signature capabilities. The authors show via several experiments and metrics that the produced music reflects the inputs given to the model.
strength_and_weaknesses: **Strengths**

- The experiments and metrics are convincing that the model does utilize the inputs when constructing the output music.
- The listening study indicates that the output music is of an overall high quality (or at least as high as the training dataset).
- Code and weights are available for download.

**Weaknesses**

My main concern is that the motivation for the model is somewhat unclear. The authors say that providing the expert description is easier than actually writing the music, but it seems to be only marginally easier. You still have to annotate every bar and include attributes that aren't necessarily musically intuitive, like note density and mean pitch. I could perhaps see constructing an UI that makes these attributes more intuitive. The authors do mention some of these issues in the Discussion section, but it would have been nice to see a more clear path toward direct usability.

I also don't understand the motivation for the learned representation at all. It doesn't seem to be suitable for human input, so I guess another model would be necessary to translate between another representation and the learned tokens?
clarity,_quality,_novelty_and_reproducibility: **Novelty**

This work is most closely related to MMM (Ens & Pasquier, 2020) in that it allows for multi-track, multi-instrument symbolic generation. This paper expands on those ideas by adding more conditioning controls.

**Quality**

Extensive experiments are done to ensure model output is based on input and that overall quality is close to the training dataset.

**Reproducibility**

Code and weights are available for download. A colab demo provides the ability to interact with the model.

**Clarity**

Overall the paper is well written, but I had a few clarifying questions:

- Section 4, Description-to-Sequence Model: How is training done so that the model supports any combination of expert or learned annotations? Is a combination selected randomly for each input?

- Section 6: I thought calling unconditional generation "random guessing" was a little misleading. Perhaps you could say something about sampling a sequence based on the training distribution prior.

- Section 6.1: "both $F_{expert}$ and especially $F_{learned}$ require time and domain knowledge to create" How would someone go about creating $F_{learned}$, even with domain knowledge?

- Section 6.1: There's a mention of $F_{latent}$ that I think was supposed to be $F_{learned}$

- Section 6.2: This doesn't sound like style transfer to me, this is more of a reconstruction metric. Style transfer would usually imply that some semantic quality of the music remains constant while underlying details change, but here you're just looking for as close of a reconstruction as possible. The "Description mixing" test sounds closer to style transfer, but from listening to the outputs, it's not clear to me that it's successful as a style transfer process.

- Links throughout the paper are not clear. It was only by accident that I realized I could click on the Colab link. I would suggest highlighting the links or just including the URL to make it more obvious.
summary_of_the_review: FIGARO provides a way to condition symbolic music generation on several per-bar controls, and experiments show that these controls are successful. However, the use case is not completely clear.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a sequence-to-sequence model for music generation from discrete codes. The discrete codes can be human annotations or learned from vector quantization. Experiments show superior performance compared to the state of the art.

The paper has strong experimental results. The novelty is incremental, as 3 out of 4 reviewers commented on this. The value of the paper lies in the execution of the experiments.
justification_for_why_not_higher_score: The novelty is limited.
justification_for_why_not_lower_score: There is still merit in the execution of the experiments, and building of the system itself.
Decision: Accept: poster