Title: Learning Features of Music From Scratch
Authors: John Thickstun
Zaid Harchaoui
Sham Kakade
Venue: ICLR 2017
Keywords: Applications

Abstract: This paper introduces a new large-scale music dataset, MusicNet, to serve as a source 
of supervision and evaluation of machine learning methods for music research. 
MusicNet consists of hundreds of freely-licensed classical music recordings 
by 10 composers, written for 11 instruments, together with instrument/note 
annotations resulting in over 1 million temporal labels on 34 hours of chamber music
performances under various studio and microphone conditions. 

The paper defines a multi-label classification task to predict notes in musical recordings, 
along with an evaluation protocol, and benchmarks several machine learning architectures for this task: 
i) learning from spectrogram features; 
ii) end-to-end learning with a neural net; 
iii) end-to-end learning with a convolutional neural net. 
These experiments show that end-to-end models trained for note prediction learn frequency
selective filters as a low-level representation of audio. 

TL;DR: We introduce a new large-scale music dataset, define a multi-label classification task, and benchmark machine learning architectures on this task.

Review discussion:
I'm curious about the motivation for the task that was chosen for the second part of the paper: polyphonic pitch estimation on isolated fragments is a well-studied task for which many non-ML methods exist.  It also looks like the paper only considers learned baselines. Could the authors comment on why this particular task was chosen? I think using a more high-level task (where feature learning is much more relevant) could yield more impactful results.
The main goal of our paper is to introduce a new large music dataset with fine-grained labels to the research community. We agree that many there are many high-level tasks for which feature learning could be very impactful. We speculate about some of these tasks on the dataset website:
http://homes.cs.washington.edu/~thickstn/musicnet.html
We found that optimizing simple end-to-end models for the polyphonic pitch estimation task came with challenges that we needed to resolve before moving to more complex tasks. Similar issues were faced in http://www.mirlab.org/conference_papers/International_Conference/ICASSP%202014/papers/p7014-dieleman.pdf, which suggests that optimization instability is a general problem for raw-audio feature learning. We demonstrate in our paper that using a large dataset and careful optimization, clean stable features can be recovered.
We are willing to include the performance results of competing "non-ML methods" on the MusicNet dataset if code is publicly available for these methods and if comparison is meaningful. Precise pointers to relevant "non-ML methods" publicly available code are very welcome.
In the polyphonic transcription experiments, do you use linear-power spectrograms as input features or some sort of compressive non-linearity like cube root or log/dB?
Why vary both the classifier used and the features used at the same time in the experiments?  This makes it difficult to evaluate the contribution of either one separately to the results.  What happens if you vary just one at a time?
And finally, to echo anon reviewer 3, could you evaluate this using the metrics of the MIREX "Multiple Fundamental Frequency Estimation & Tracking" task?  What is the motivation for coming up with new metrics?
Our focus in Sec. 4-5 of the paper is to investigate the low-level features learned end-to-end models trained on raw audio waves, not to maximize performance.
However, the reviewer's suggestion to try a compressive non-linearity improves performance substantially for both the spectrograms and end-to-end models. We thank the reviewer for this fruitful suggestion, and we will report results obtained with a compressive non-linearity in the final version of the paper.
We will also update the paper to report classifier results for each of the learned representations.
Regarding metrics: average precision is a robust and standard multi-label classification metric in the machine learning community. We will additionally report accuracy and error scores a la MIREX Multi-F0 to clarify our results to the music community.
There is no question that such a dataset could be very useful indeed.
The validation description in Appendix A is thorough and interesting.
However, I am curious what limitations it might have.
For example, how well does it work for dense polyphonic music where there is already dissonance in the music itself,
e.g. how noticeable in that case will the “extra” dissonance be created by the additional sinusoidal wave?
How accurately will this wave help with detecting the correct release time of the note, i.e. are false negatives more likely if a sine wave is simply missing, since there is no dissonance to be noticed? Table 2 suggests that the release time of the notes is annotated very precisely, but I believe that this precision may belie the accuracy.
Is there any relationship between the accuracy of the transcription and the dynamic level, e.g. are quiet passages more likely to have more errors?
The more exactly the limitations of this dataset are outlined, the more useful it will ultimately be.
For example, the very definition of “release time” might not be obvious (or important) for dense solo piano with a lot of pedal, yet is very informative in other contexts (e.g. analysis of certain performances, composers, expressive performance, various other cases).
Similarly, there is often an unavoidable discrepancy between score and performance in the polyphonic moments (of which there are more than a few) in solo violin music; how does this affect accuracy of annotation, if at all?
Given that the precision on the best model was generally below 50%, it would be interesting eventually to see some samples of the kinds of errors that the system makes.
It is usually quite clear from listening to the aural representation described in the appendix if an alignment is incorrect.
When the authors independently check alignments for errors, our assessments are nearly identical. Our evaluation method indeed breaks down for music that is simultaneously rapid, highly polyphonic, and dissonant.
However, we observe no particular decline in alignment quality as performance speed and complexity increase (if anything, additional complexity seems to aid in alignment) so we don't have reason to believe that the alignments fail in these limiting cases.
Your intuition is correct that release times are harder to verify than onsets (and our listening test does not explicitly account for release times). Typically the release of one note coincides with the onset of a new note, which implicitly verifies the release.
However, we acknowledge that annotated release times at the end of phrases and movements may be less accurate. As you point out, the definition of release time may in some cases come into question. Does a note end when the bow leaves the string?
When the musicians put down their instruments?
When the string stops resonating audibly? These subtleties are not fully captured by our error analysis and we will clarify this.
Regarding solo violin, I would guess you are referring to performance necessities such as rolling triple and quadruple stops. Here is an example of a solo violin performance with mixed in sine tones that mark the aligned label onsets:
http://homes.cs.washington.edu/~thickstn/media/violin_markup.wav
Triple stops occur at times t=5.68s, 13.25s, 21.37s, 23.63s, and 26.59s. All three labels align to the entire roll, starting at the beginning.
Therefore, there is some time period at the beginning of the roll where the "top" note is labeled but has not yet occurred in the performance (and, depending on your definition of release time, there is a corresponding period at the end where the "bottom" note is labeled after it has ceased in the performance). There are reasonable interpretations of labeling under which these labels would be judged incorrect.
On the other hand, if the labels are used to supervise transcription then ours is likely the desired labeling. We will be sure to mention this subtlety as well!
We will add some discussion of common types of errors to the paper.
The paper introduces a new dataset called MusicNet (presumably analogous to ImageNet), featuring dense ground truth labels for 30+ hours of classical music, which is provided as raw audio. Such a dataset is extremely valuable for music information retrieval (MIR) research and a dataset of this size has never before been publicly available. It has the potential to dramatically increase the impact of modern machine learning techniques (e.g. deep learning) in this field, whose adoption has previously been hampered by a lack of available datasets that are large enough. The paper is clear and well-written.
The paper also features some "example" experiments using the dataset, which I am somewhat less excited about. The authors decided to focus on one single task that is not particularly challenging: identifying pitches in isolated segments of audio. Pitch information is a fairly low-level characteristic of music. Considering that isolated fragments are used as input, this is a relatively simple problem that probably doesn't even require machine learning to solve adequately, e.g. peak picking on a spectral representation could already get you pretty far. It's not clear what value the machine learning component in the proposed approach actually adds, if any. I could be wrong about this as I haven't done the comparison myself, but I think the burden is on the authors to demonstrate that using ML here is actually useful.
I would argue that one of the strenghts of the dataset is the variety of label information it provides, so a much more convincing setup would have been to demonstrate many different prediction tasks for both low-level (e.g. pitch, onsets) and high-level (e.g. composer) characteristics, perhaps with fewer and simpler models -- maybe even sticking to spectrogram input and forgoing raw audio input for the time being, as this comparison seems orthogonal to the introduction of the dataset itself. As it stands, I feel that the fact that the experiments are relatively uninteresting detracts from the main point of the paper, which is to introduce a new public dataset that is truly unique in terms of its scale and scope.
That said, the experiments seem to have been conducted in a rigorous fashion and the evaluation and analysis of the resulting models is properly executed.
Re: Section 4.5, it is rather unsurprising to me that a pitch detector would learn filters that resemble pitches (i.e. sinusoids), although the observation that this requires a relatively large amount of data is interesting.
However, it would be more interesting to demonstrate that this is also the case for higher-level tasks. The authors favourably compare the features learnt by their model with prior work on end-to-end learning from raw audio, but neglect that the tasks considered in this work were much more high-level.
Some might also question whether ICLR is the appropriate venue to introduce a new dataset, but personally I think it's a great idea to submit it here, seeing as it will reach the right people. I suppose this is up to the organisers and the program committee, but I thought it important to mention this, because I don't think this paper merits acceptance based on its experimental results alone.
Thank you for your positive comments regarding the dataset.
We disagree that note identification is “not particularly challenging.” This task is closely related to fundamental frequency estimation, which has been studied extensively in the music information retrieval community. For a survey, see
http://www.eecs.qmul.ac.uk/~emmanouilb/papers/JIIS-MIRrors-AMT-postprint.pdf
We have updated our related works section on page 5 to give more context for the experiments.
This paper describes the creation of a corpus of freely-licensed classical music recordings along with corresponding MIDI-scores aligned to the audio.  It also describes experiments in polyphonic transcription using various deep learning approaches, which show promising results.
The paper is a little disorganised and somewhat contradictory in parts.
For example, I find the first sentence in section 2 (MusicNet) would better be pushed one paragraph below so that the section be allowed to begin with a survey of the tools available to researchers in music.
Also, the description for Table 3 should probably appear somewhere in the Methods section.
Last example: the abstract/intro says the purpose is note prediction;
later (4th paragraph of intro) there's a claim that the focus is "learning low-level features of music...." I find this slightly disorienting.
Although others (Uehara et al., 2016, for example) have discussed collection platforms and corpora, this work is interesting because of its size and the approach for generating features. I'm interested in what the authors will to do expand the offerings in the corpus, both in terms of volume and diversity.
Thank you for your feedback on our writing. We agree with your observations and we have addressed some of these issues in the latest revision. We will be sure to also clarify that the purpose of our experiments is to understand low-level features of music.
This paper introduces MusicNet, a new dataset. Application of ML techniques to music have been limited due to scarcity of exactly the kind of data that is provided here: meticulously annotated, carefully verified and organized, containing enough "hours" of music, and where genre has been well constrained in order to allow for sufficient homogeneity in the data to help ensure usefulness. This is great for the community.
The description of the validation of the dataset is interesting, and indicates a careful process was followed.
The authors provide just enough basic experiments to show that this dataset is big enough that good low-level features (i.e. expected sinusoidal variations) can indeed be learned in an end-to-end context.
One might argue that in terms of learning representations, the work presented here contributes more in the dataset than in the experiments or techniques used.
However, given the challenges of acquiring good datasets, and given the essential role such datasets play for the community in moving research forward and providing baseline reference points, I feel that this contribution carries substantial weight in terms of expected future rewards. (If research groups were making great new datasets available on a regular basis, that would place this in a different context.
But so far, that is not the case.) In otherwords, while the experiments/techniques are not necessarily in the top 50% of accepted papers (per the review criteria), I am guessing that the dataset is in the top 15% or better.
Thank you for your positive comments.

Meta-review:
role: PC
data: comment: There was some question as to weather ICLR is the right venue for this sort of dataset paper, I tend to think it would be a good addition to ICLR as people from the ICLR community are likely to be among the most interested. The problem of note identification in music is indeed challenging and the authors revised the manuscript to provide more background on the problem. No major issues with the writing or clarity. Experiments and models explored are not extremely innovative, but help create a solid dataset introduction paper.

Decision: Accept (Poster)