Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper describes the creation of a corpus of freely-licensed classical music recordings along with corresponding MIDI-scores aligned to the audio.  It also describes experiments in polyphonic transcription using various deep learning approaches, which show promising results.

The paper is a little disorganised and somewhat contradictory in parts. For example, I find the first sentence in section 2 (MusicNet) would better be pushed one paragraph below so that the section be allowed to begin with a survey of the tools available to researchers in music. Also, the description for Table 3 should probably appear somewhere in the Methods section. Last example: the abstract/intro says the purpose is note prediction; later (4th paragraph of intro) there's a claim that the focus is "learning low-level features of music...." I find this slightly disorienting.

Although others (Uehara et al., 2016, for example) have discussed collection platforms and corpora, this work is interesting because of its size and the approach for generating features. I'm interested in what the authors will to do expand the offerings in the corpus, both in terms of volume and diversity.

review_title: Interesting corpus, what next?

final_score: rating: null
review_content: In the polyphonic transcription experiments, do you use linear-power spectrograms as input features or some sort of compressive non-linearity like cube root or log/dB?

Why vary both the classifier used and the features used at the same time in the experiments?  This makes it difficult to evaluate the contribution of either one separately to the results.  What happens if you vary just one at a time?

And finally, to echo anon reviewer 3, could you evaluate this using the metrics of the MIREX "Multiple Fundamental Frequency Estimation & Tracking" task?  What is the motivation for coming up with new metrics?
review_title: Clarifications on transcription experiments

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper introduces a new dataset called MusicNet (presumably analogous to ImageNet), featuring dense ground truth labels for 30+ hours of classical music, which is provided as raw audio. Such a dataset is extremely valuable for music information retrieval (MIR) research and a dataset of this size has never before been publicly available. It has the potential to dramatically increase the impact of modern machine learning techniques (e.g. deep learning) in this field, whose adoption has previously been hampered by a lack of available datasets that are large enough. The paper is clear and well-written.

The paper also features some "example" experiments using the dataset, which I am somewhat less excited about. The authors decided to focus on one single task that is not particularly challenging: identifying pitches in isolated segments of audio. Pitch information is a fairly low-level characteristic of music. Considering that isolated fragments are used as input, this is a relatively simple problem that probably doesn't even require machine learning to solve adequately, e.g. peak picking on a spectral representation could already get you pretty far. It's not clear what value the machine learning component in the proposed approach actually adds, if any. I could be wrong about this as I haven't done the comparison myself, but I think the burden is on the authors to demonstrate that using ML here is actually useful.

I would argue that one of the strenghts of the dataset is the variety of label information it provides, so a much more convincing setup would have been to demonstrate many different prediction tasks for both low-level (e.g. pitch, onsets) and high-level (e.g. composer) characteristics, perhaps with fewer and simpler models -- maybe even sticking to spectrogram input and forgoing raw audio input for the time being, as this comparison seems orthogonal to the introduction of the dataset itself. As it stands, I feel that the fact that the experiments are relatively uninteresting detracts from the main point of the paper, which is to introduce a new public dataset that is truly unique in terms of its scale and scope.

That said, the experiments seem to have been conducted in a rigorous fashion and the evaluation and analysis of the resulting models is properly executed.

Re: Section 4.5, it is rather unsurprising to me that a pitch detector would learn filters that resemble pitches (i.e. sinusoids), although the observation that this requires a relatively large amount of data is interesting. However, it would be more interesting to demonstrate that this is also the case for higher-level tasks. The authors favourably compare the features learnt by their model with prior work on end-to-end learning from raw audio, but neglect that the tasks considered in this work were much more high-level.

Some might also question whether ICLR is the appropriate venue to introduce a new dataset, but personally I think it's a great idea to submit it here, seeing as it will reach the right people. I suppose this is up to the organisers and the program committee, but I thought it important to mention this, because I don't think this paper merits acceptance based on its experimental results alone.
review_title: awesome new dataset, less interesting experiments

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper introduces MusicNet, a new dataset. Application of ML techniques to music have been limited due to scarcity of exactly the kind of data that is provided here: meticulously annotated, carefully verified and organized, containing enough "hours" of music, and where genre has been well constrained in order to allow for sufficient homogeneity in the data to help ensure usefulness. This is great for the community.

The description of the validation of the dataset is interesting, and indicates a careful process was followed.

The authors provide just enough basic experiments to show that this dataset is big enough that good low-level features (i.e. expected sinusoidal variations) can indeed be learned in an end-to-end context.

One might argue that in terms of learning representations, the work presented here contributes more in the dataset than in the experiments or techniques used. However, given the challenges of acquiring good datasets, and given the essential role such datasets play for the community in moving research forward and providing baseline reference points, I feel that this contribution carries substantial weight in terms of expected future rewards. (If research groups were making great new datasets available on a regular basis, that would place this in a different context. But so far, that is not the case.) In otherwords, while the experiments/techniques are not necessarily in the top 50% of accepted papers (per the review criteria), I am guessing that the dataset is in the top 15% or better.
review_title: Final Review: valuable new dataset

final_score: rating: null
review_content: I'm curious about the motivation for the task that was chosen for the second part of the paper: polyphonic pitch estimation on isolated fragments is a well-studied task for which many non-ML methods exist.  It also looks like the paper only considers learned baselines. Could the authors comment on why this particular task was chosen? I think using a more high-level task (where feature learning is much more relevant) could yield more impactful results.
review_title: Choice of task

final_score: rating: null
review_content: There is no question that such a dataset could be very useful indeed.

The validation description in Appendix A is thorough and interesting. However, I am curious what limitations it might have. For example, how well does it work for dense polyphonic music where there is already dissonance in the music itself, 
e.g. how noticeable in that case will the “extra” dissonance be created by the additional sinusoidal wave? 

How accurately will this wave help with detecting the correct release time of the note, i.e. are false negatives more likely if a sine wave is simply missing, since there is no dissonance to be noticed? Table 2 suggests that the release time of the notes is annotated very precisely, but I believe that this precision may belie the accuracy. 

Is there any relationship between the accuracy of the transcription and the dynamic level, e.g. are quiet passages more likely to have more errors? 

The more exactly the limitations of this dataset are outlined, the more useful it will ultimately be. For example, the very definition of “release time” might not be obvious (or important) for dense solo piano with a lot of pedal, yet is very informative in other contexts (e.g. analysis of certain performances, composers, expressive performance, various other cases). 

Similarly, there is often an unavoidable discrepancy between score and performance in the polyphonic moments (of which there are more than a few) in solo violin music; how does this affect accuracy of annotation, if at all?

Given that the precision on the best model was generally below 50%, it would be interesting eventually to see some samples of the kinds of errors that the system makes.
review_title: detailed accuracy of the data set

Meta-review: There was some question as to weather ICLR is the right venue for this sort of dataset paper, I tend to think it would be a good addition to ICLR as people from the ICLR community are likely to be among the most interested. The problem of note identification in music is indeed challenging and the authors revised the manuscript to provide more background on the problem. No major issues with the writing or clarity. Experiments and models explored are not extremely innovative, but help create a solid dataset introduction paper.
Decision: Accept (Poster)