Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper extends the DDSP generative modeling approach (Engel et al., 2020) to higher levels of abstraction for high-level MIDI -> audio synthesis with intermediate levels of semantic control over the generation process.
main_review: This paper develops a parametric model of musical audio that leverages insight into the music domain to construct meaningful featurizations of music, together with models trained to invert these featurizations, to construct a hierarchical autoencoder with meaningful semantic features that can be used to control the music generation process.

The technical contributions of this paper consist of a semantic "expression" featurization of DDSP parameters (Section 3.2) and two inverse models: (1) a synthesis generator for converting the expression features back to DDSP synthesis parameters and (2) an expression generator for converting MIDI data back to expression features. The features and models are well-described at a high level in the main paper, and the appendix provides a thorough & precise description of the remaining details. The empirical study seems convincing, with comparisons to both academic prior work (MIDIParams) and commercial software (Ableton, FluidSynth). And the demos sound qualitatively good to my ear.

The paper thoroughly relates this work to related work in the field, making many thoughtful connections to both the symbolic & audio generative modeling literature. These connections both help situate this work in the broader literature, and also provide some welcome scaffolding for thinking more broadly about the field of generative music modeling. I also appreciate the connections to the signal processing literature.


One possible criticism of this paper is that, because its focus is on applying machine learning rather than in the development of machine learning techniques, perhaps it would be better suited for a music domain conference (e.g., ISMIR). I strongly support its publication at ICLR: the paper does a thorough job situating itself in the machine learning literature, making it accessible and appealing to the broader machine learning audience.


One question I have is about the choice of 3 stage modeling (notes -> expression -> synthesis -> audio) as opposed to the 2 stage process used by MIDI2Params (notes -> synthesis -> audio). I appreciate that injecting semantically meaningful performance features provides an additional level of control over the generation process. Does this additional stage also account for the performance improvement of MIDI-DDSP over MIDI2Params? Or is this improvement more attributable to better/bigger models? 

Also, related to the previous point, I am a little confused by the MIDI2Params comparisons in Table 1. I am under the impression that the expression step is an innovation of this work (Section 3.2). How are you able to make these breakdown comparisons with MIDI2Params for these substages notes -> expression and expression -> synthesis. Have I misunderstood MIDI2Params?
summary_of_the_review: This is a well-written, creative paper. The models are interesting. The figures are excellent. The connections to related work are extensive and thoughtful.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a music performance modeling network using three sub-modules which are expression generator, synthesis generator, and DDSP inference. The idea of using these three sub-modules to create three-level performance modeling (the three-level control values are notes, performance features, and synthesis parameters) is interesting and the result shows that the proposed model can give more ability of control while not harming the overall audio generation performance. It seems the proposed method that having three trainable networks with two hand-crafted methods (actually note detection is not the part of the contribution of this paper, so I didn't count it) gives full pipeline of hierarchical modeling and this is a good research contribution.
main_review: The paper is well-written, so I only have few minor questions.

While reading the paper, I was wondering the three sub-modules are trained simultaneously. If not, can other dataset be used for training each sub-module to enhance robustness or the performance?

For synthesis generator module, I couldn't find some ablation test about the loss function. 

The authors argue that the method can be easily extended when multi-instrument transcription model is ready. However, I see there exist many challenges (e.g. the model relies on CREPE in DDSP synthesis part, also the type of an instrument set between transcription part and MIDI-DDSP should be matched.), the tone can be lowered.

In page 3, "Realistic Note Synthesis", "a" is missing in a word 'concatenative'.

summary_of_the_review: The paper opens a more user controllability in music performance modeling and this is a nice contribution. Also, the idea of having rule-based method with trainable networks in a series of a chain seems useful in many other applications.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents a controllable rendering engine for MIDI files, based on the DDSP framework. Given F0 and loudness contour, DDSP can estimate the parameters of a harmonic + noise synthesis model, to render a corresponding audio file. Similar to MIDI2Params, which predicts framewise FO and loudness contours from a MIDI file, MIDI DDSP introduces an intermediate hierarchical level, allowing to control some newly introduced "expression controls". A mapping from MIDI files to "expression controls" is learnt, so that MIDI files can be automatically rendered. However, because these note wise controls are explicit, they also allow human manipulation of the performance rendering.
Influence of the expression controls is assessed with a correlation study, showing that the human manipulation has the expected effect on the generated performance. This quantitative evaluation is further confirmed by convincing audio examples.
main_review: This paper introduces a layer of controllable expression parameters for the audio rendering of MIDI files. The contribution is threefold:
1. The authors introduce heuristics to extract note-wise expression parameters from low-level synthesis parameters. The choice of the controls are relevant to the dataset the authors use (acoustic instruments).
2. They introduce the synthesis generator, allowing to predict the frame-level synthesis parameters, hence bypassing the DDSP decoder that used to predict them from F0 and loudness contours. The introduction of this model allows a better reconstruction error than the original DDSP. Also, it transfers the manipulation of the sound from the mentioned contours (which are frame-level) to note-level expression controls, more expressive and controllable.
3. The smart design of the expression controls allows them to be controlled by humans, hence allowing further manipulation of the synthesis. Each control is well defined, normalized between 0 and 1, and coarse enough (note-level vs. frame-level) so that a user can easily set them to their preferred value, enhancing the customization possibilities. Human manipulation of the controls seem to have the desired effect, as shown by the correlation study in Table 2 and further supported by the audio material.
A few comments:
- In paragraph 3.3, the authors write that the note expression controls are pooled over the duration of the corresponding note. To my understanding, the controls would rather be unpooled, repeated or upsampled, as there are more frames than notes (very clear in B.4).
- The paragraph about the dataset doesn't mention that it also contains the MIDI ground truth. This could be made explicit.
- The paragraph entitled "Expression Generator" in 4.2 features what looks like a residue of an unwanted sentence
- In Appendix B.3, the definition for brightness seems to be lacking a sum over k, and I'm having a hard time trying to understand the multiplication by i rather than k.
- Some of these expression controls naturally lie in [0;1] (e.g. Volume Peak Position). However, it is not clear how others are normalized (e.g. Vibrato). If normalizing constants are precomputed using feature extraction on the whole dataset, it should be written.

summary_of_the_review: Overall, this works is very well written. Everything should appear pretty clear to anyone familiar with the DDSP framework. The principle is simple yet extremely effective, as it improves reconstruction quality and parameters estimation compared to the state of the art. This new control layer represents a lot of added value compared to MIDI2Params, which was already a great addition to the original DDSP framework. In addition to the paper, the website provides very convincing audio examples which further assess the quality of this work.
review_title: null

Meta-review: This paper proposed MIDI-DDSP, a structured hierarchical generative model which offers both detailed expressive controls (as in traditional synthesizers) as well as the realistic audio quality (as in black-box neural audio synthesis). Overall the reviews are very positive. All the reviewers unanimously agree that the paper is very well-written and presented a very convincing model and a meaningful step-up from the earlier work of DDSP. The authors also presented a well-documented website for the project and promised to release the source code. The reviewers raised some clarifying questions and minor corrections which the authors addressed during the response. Therefore, I vote for accept.
Decision: Accept (Oral)