Title: SYNTHNET: LEARNING SYNTHESIZERS END-TO-END

Abstract: Learning synthesizers and generating music in the raw audio domain is a challenging task. We investigate the learned representations of convolutional autoregressive generative models. Consequently, we show that mappings between musical notes and the harmonic style (instrument timbre) can be learned based on the raw audio music recording and the musical score (in binary piano roll format). Our proposed architecture, SynthNet uses minimal training data (9 minutes), is substantially better in quality and converges 6 times faster than the baselines. The quality of the generated waveforms (generation accuracy) is sufficiently high that they are almost identical to the ground truth. Therefore, we are able to directly measure generation error during training, based on the RMSE of the Constant-Q transform. Mean opinion scores are also provided. We validate our work using 7 distinct harmonic styles and also provide visualizations and links to all generated audio.

1. INTRODUCTION:

WaveNets (Van Den Oord et al., 2016) have revolutionized text to speech by producing realistic human voices. Even though the generated speech sounds natural, upon a closer inspection the waveforms are different to genuine recordings. As a natural progression, we propose a WaveNet derrivative called SynthNet which can learn and render (in a controlled way) the complex harmonics in the audio training data, to a high level of fidelity. While vision is well established, there is little understanding over what audio generative models are learning. Towards enabling similar progress, we give a few insights into the learned representations of WaveNets, upon which we build our model. WaveNets were trained using raw audio waveforms aligned with linguistic features. We take a similar approach to learning music synthesizers and train our model based on the raw audio waveforms of entire songs and their symbolic representation of the melody. This is more challenging than speech due to the following differences: 1) in musical compositions multiple notes can be played at the same time, while words are spoken one at a time; 2) the timbre of a musical instrument is arguably more complex than speech; 3) semantically, utterances in music can span over a longer time. Van Den Oord et al. (2016) showed that WaveNets can generate new piano compositions based on raw audio. Recently, this work was extended by Dieleman et al. (2018) , delivering a higher consistency in compositional styling. Closer to our work, Engel et al. (2017) describe a method for learning synthesizers based on individually labelled note-waveforms. This is a labourius task and is impractical for creating synthesizers from real instruments. Our method bypassses this problem since it can directly use audio recodings of an artist playing a given song, on the target instrument.

SynthNet can learn representations of the timbre of a musical instrument more accurately and efficiently via the dilated blocks through depthwise separable convolutions. We show that it is enough to condition only the first input layer, where a joint embedding between notes and the corresponding fundamental frequencies is learned. We remove the skip connections and instead add an additional loss for the conditioning signal. We also use an embedding layer for the audio input and use SeLU (Klambauer et al., 2017) activations in the final block.

The benchmarks against the WaveNet (Van Den Oord et al., 2016) and DeepVoice (Arik et al., 2017) architectures show that our method trains faster and produces high quality audio. After training, SynthNet can generate new audio waveforms in the target harmonic style, based on a given song which was not seen at training time. While we focus on music, SynthNet can be applied to other domains as well.

Our contributions are as follows: 1) We show that musical instrument synthesizers can be learned end-to-end based on raw audio and a binary note representation, with minimal training data. Multiple instruments can be learned by a single model. 2) We give insights into the representations learned by dilated causal convolutional blocks and consequently propose SynthNet, which provides substantial improvements in quality and training time compared to previous work. Indeed, we demonstrate (Figure 4 ) that the generated audio is practically identical to the ground truth. 3) The benchmarks against existing architectures contains an extensive set of experiments spanning over three sets of hyperparameters, where we control for receptive field size. We show that the RMSE of the Constant-Q Transform (RMSE-CQT) is highly correlated with the mean opinion score (MOS). 4) We find that reducing quantization error via dithering is a critical preprocessing step towards generating the correct melody and learning the correct pitch to fundamental frequency mapping.

2. RELATED WORK:

In music, style can be defined as the holistic combination of the melodic, rhythmic and harmonic components of a particular piece. The delay and sustain variation between notes determines the rhythmic style. The latter can vary over genres (e.g. Jazz vs Classical) or composers. Timbre or harmonic style can be defined as the short term (attack) and steady state (sustained frequency distribution) acoustical properties of a musical instrument (Sethares, 2005) . Our focus is on learning the harmonic style, while controlling the (given) melodic content and avoiding any rhythmic variations.

The research on content creation is plentiful. For an in depth survey of deep learning methods for music generation we point the reader to the work of Briot et al. (2017) . Generative autoregressive models were used in (Van Den Oord et al., 2016; Mehri et al., 2016) to generate new random content with similar harmonics and stylistic variations in melody and rhythm. Recently, the work of Van Den Oord et al. (2016) was extended by Dieleman et al. (2018) where the quality is improved and the artificial piano compositions are more realistic. We have found piano to be one of the easier instruments to learn. Donahue et al. (2018) introduce WaveGANs for generating music with rhythmic and melodic variations.

Closer to our work, Engel et al. (2017) propose WaveNet Autoencoders for learning and merging the harmonic properties of instrument synthesizers. The major difference with our work is that we are able to learn harmonic styles from entire songs (a mapped sequence of notes to the corresponding waveform), while their method requires individually labelled notes (NSynth dataset). With our method the overhead of learning a new instrument is greatly reduced. Moreover, SynthNet requires minimal data and does not use note velocity information.

Based on the architecture proposed by Engel et al. (2017) , and taking a domain adaptation approach, Mor et al. (2018) condition the generation process based on raw audio. An encoder is used to learn note mappings from a source audio timbre to a target audio timbre. The approach can be more error prone than ours, since it implies the intermediary step of correctly decoding the right notes from raw audio. This can significantly decrease the generation quality. Interestingly, Mor et al. (2018) play symphonic orchestras from a single instrument audio. However, there is no control over which instrument plays what. Conversely, we use individual scores for each instrument, which gives the user more control. This is how artists usually compose music.

3 END-TO-END SYNTHESIZER LEARNING Van Den Oord et al. (2016) and Arik et al. (2017) have shown that generative convolutional networks are effective at learning human voice from raw audio. This has advanced the state of the art in text to speech (TTS). Here, we further explore the possibilities of these architectures by benchmarking them in the creative domain -learning music synthesizers. There are considerable differences between the human voice and musical instruments. Firstly, the harmonic complexity of musical instruments is higher than the human voice. Second, even for single instrument music, multiple notes can be played at the same time. This is not true for speech, where only one sound utterance is produced at a time. Lastly, the melodic and rhythmic components in a musical piece span a larger temporal context than a series of phonemes as part of speech. Therefore, the music domain is much more challenging.

3.1. BASELINE ARCHITECTURES:

The starting point is the model proposed by Van Den Oord et al. (2016) with the subsequent refinements in (Arik et al., 2017) . We refer the reader to the these articles for further details. Our data consists of triplets {(x 1 , y 1 , z 1 ), . . . , (x N , y N , z S )} over N songs and S styles, where x i is the 256-valued encoded waveform, y i is the 128-valued binary encoded MIDI and z s ∈ {1, 2, . . . , S} is the one-hot encoded style label. Each audio sample x t is conditioned on the audio samples at all previous timesteps x <t = {x t-1 , x t-2 , . . . , x 1 }, all previous binary MIDI samples and the global conditioning vector. The joint probability of a waveform x = {x 1 , . . . , x T } is factorized as follows:

p(x|y, z) = T t=1 p(x|x <t , y <t , z).

(1)

The hidden state before the residual connection in dilation block is

EQUATION

while the output of every dilation block, after the residual connection is

EQUATION

where τ and σ are respectively the tanh and sigmoid activation functions, is the layer index, f indicates the filter weights, g the gate weights, r the residual weights and W , V and U are the learned parameters for the main, local conditioning and global conditioning signals respectively. The f and g convolutions are computed in parallel as a single operation (Arik et al., 2017) . All convolutions have a filter width of F . The convolutions with W and V are dilated.

To locally condition the audio signal, Van Den Oord et al. (2016) first upsample the y time series to the same resolution as the audio signal (obtaining y ) using a transposed convolutional network, while Arik et al. ( 2017) use a bidirectional RNN. In our case, the binary midi vector already has the same resolution.

We use an initial causal convolution layer (Equation 4) that only projects the dimensionality of the signal from 128 channels to the number of residual channels. The first input layers are causal convolutions with parameters W 0 and V 0 for the waveform and respectively the piano roll:

EQUATION

x 0 = W 0 * x.

(5) All other architecture details are kept identical to the ones presented in (Van Den Oord et al., 2016) and (Arik et al., 2017) as best as we could determine. The differences between the two architectures and SynthNet are summarized in Table 1 . We compare the performance and quality of these two baselines against SynthNet initially in Table 3 over three sets of hyperparmeters (Table 2 ). For the best resulting models we perform MOS listening tests, shown in Table 5 . Preliminary results for global conditioning experiments are also provided in Table 4 . 

3.2. GRAM MATRIX PROJECTIONS:

We perform a set of initial experiments to gain more insight towards the learned representations. We use Gram matrices to extract statistics since these have been previously used for artistic style transfer (Gatys et al., 2015) . After training, the validation data is fed through five locally conditioned networks, each trained with a distinct harmonic style. The data has identical melodic content but has different harmonic content (i.e. same song, different instruments). The Gram matrices are extracted from the outputs of each dilated block (Equation 3) for each network -timbre. These are flattened and projected onto 2D, simultaneously over all layers and styles via T-SNE (Maaten & Hinton, 2008) . The results presented in Figure 1 show that the extracted statistics separate further as the layer index increases. A broad interpretation is that the initial layers extract low-level generic audio features, these being common to all waveforms. However, since this is a controlled experiment, we can be more specific. The timbre of a musical instrument is characterized by a specific set of resonating frequencies on top of the fundamental frequency (pure sine wave). Typically one identifies individual notes based on their fundamental, or lowest prominent frequency. These depend on the physics of the musical instrument and effects generated, for example, by the environment. Since the sequence of notes is identical and the harmonic styles differ, we conjectured that Figure 1 could imply a frequency-layer correspondence. While the latter statement might be loose, the lower layers' statistics are nevertheless much closer due the increased similarity with the fundamental frequency. Figure 2 : SynthNet (also see Table 1 ) with a multi-label cross-entropy loss for binary midi.

L01 L02 L03 L04 L05 L06 L07 L08 L09 L10 L11 L12 L13 L14 L15 L16 L17 L18 L19 L20 L21 L22 L23 L24 L25 L26 ■ Guitar ◆ Trumpet +�Piano ✖ G l o c k e n s p i el • Cello

3.3. SYNTHNET ARCHITECTURE:

Figure 1 provides indicative results from many experiments. Additional Gram matrix projections are provided in Appendix B. We hypothesize that the skip connections are superfluous and the conditioning of the first input layer should suffice to drive the melodic component. We also hypothesize that the first audio input layer learns an embedding corresponding to the fundamental pitches. Then, we aim to learn mapping from the symbolic representation (binary midi code) to the pitch embeddings (Equation 8). Therefore, in SynthNet there are no parameters learned in each dilation block for local conditioning and the hidden activation with global conditioning (omitted in Figure 2 ) becomes

EQUATION

The input to the dilated blocks is the sum of the embedding codes and the autoencoder latent codes:

EQUATION

As it can be seen in Figure 2 there are no skip connections and Equation 4 no longer applies. We also found that using SeLU activations (Klambauer et al., 2017) in the last layers improves generation stability and quality. Other normalization strategies could have been used, we found SeLU to work well. In addition, we further increase sparsity by changing the dilated convolution in Equation 6 with a dilated depthwise separable convolution. Separable convolutions perform a channel-wise spatial convolution that is followed by a 1 × 1 convolution. In our case each input channel is convolved with its own set of filters. Depthwise separable convolutions have been successfully used in mobile and embedded applications (Howard et al., 2017) and in the Xception architecture (Chollet, 2017) .

As we show in Table 4 and Table 5 , the parsimonious approach works very well since it reduces the complexity of the architecture and speeds up training.

In training, SynthNet models the midi data y in an auto-regressive fashion which is similar to the way audio data is modeled, but with a simplified architecture (see Figure 2 ). In principle, this allows the model to jointly generate both audio and midi. In practice, the midi part of the model is too simple to generate interesting results in this way. Nonetheless, we found it beneficial to retain the midi loss term during training, which it turns out, tends to act as a useful regularizer -we conjecture by forcing basic midi features to be extracted. In summary, in contrast with Equation 1 we optimize the joint log p(x, y|z), so that

L = - 1 N N i=1   |x|=256 j=1 x i j log xi j + |y|=128 j=1 y i j log ŷi j + (1 -y i j ) log(1 -ŷi j )   .

4. EXPERIMENTS:

We compare exact replicas of the architectures described in Van Den Oord et al. (2016) ; Arik et al.

(2017) with our proposed architecture SynthNet. We train the networks to learn the harmonic audio style (here instrument timbre) using raw audio waveforms. The network is conditioned locally with a 128 binary vector indicating note on-off, extracted from the midi files. The latter describes the melodic content. For the purpose of validating our hypothesis, we decided to eliminate extra possible sources of error and manually upsampled the midi files using linear interpolation. For the results in Table 4 the network is also conditioned globally with a one-hot vector which designates the style (instrument) identity. Hence, multiple instrument synthesizers are learned in a single model. For the hyperparameter search experiments (Table 3 ) and the final MOS results (Table 5 ) we train one network for each style, since it is faster.

We use the Adam Kingma & Ba (2014) optimization algorithm with a batch size of 1, a learning rate of 10 -3 , β 1 = 0.9, β 2 = 0.999 and ε = 10 -8 with a weight decay of 10 -5 . We find that for most instruments 100-150 epochs is enough for generating high quality audio, however we keep training up to 200 epochs to observe any unexpected behaviour or overfitting. All networks are trained on Tesla P100-SXM2 GPUs with 16GB of memory.

4.1. SYNTHETIC REGISTERED AUDIO:

We generate the dataset using the freely available Timidity++1 software synthesizer. For training we selected parts 2 to 6 from Bach's Cello Suite No. 1 in G major (BWV 1007). We found that this was enough to learn the mapping from midi to audio and to capture the harmonic properties of the musical instruments. From this suite, the Prelude (since it is most commonly known) is not seen during training and is used for measuring the validation loss and for conditioning the generated audio.

After synthesizing the audio, we have approximately 12 minutes of audio for each harmonic style, out of which 9 minutes (75%) training data and 3 minutes (25%) of validation data. We experiment with S = 7 harmonic styles which were selected to be as different as possible. Each style corresponds to a specific preset from the 'Fluid-R3-GM' sound font. These are (preset number -instrument): S01 -Bright Yamaha Grand, S09 -Glockenspiel, S24 -Nylon String Guitar, S42 -Cello, S56 -Trumpet, S75 -Pan Flute and S80 -Square Lead.

For training, the single channel waveforms are sampled at 16kHz and the bit-depth is reduced to 8 bit via mu-law encoding. Before reducing the audio bit depth, the waveforms are dithered using a triangular noise distribution with limits (-0.009, 0.009) and mode 0, which reduces perceptual noise but more importantly keeps the quantization noise out of the signal frequencies. We have found this critical for the learning process.

Without dithering there are melodic discontinuities and clipping errors in the generated waveforms. The latter errors are most likely due to notes getting mapped to the wrong set of frequencies (artifacts appear due to the quantization error). From all harmonic styles, the added white noise due to dithering is most noticeable for Glockenspiel, Cello and Pan Flute. The midi is upsampled to 16kHz to match the audio sampling rate via linear interpolation. Each frame contains a 128 valued vector which designates note on-off times for each note (piano roll).

4.2. MEASURING AUDIO GENERATION QUALITY:

Quantifying the performance of generative models is not a trivial task. Similarly to Van Den Oord et al. (2016); Arik et al. (2017) we have found that once the training and validation losses go beyond a certain lower threshold, the quality improves. However, the losses are only informative towards convergence and overfitting (Figure 3 ) -they are not sensitive enough to accurately quantify the quality of the generated audio. This is critical for ablation studies where precision is important. Theis et al. (2015) argue that generative models should be evaluated directly. Then, the first option is the mean opinion score (MOS) via direct listening tests. This can be impractical, slowing down the hyperparameter selection procedure. MOS ratings for the best found models are given in Table 5 . Similarly to the Fourier transform, the CQT (Brown, 1991) is built on a bank of filters, however unlike the former it has geometrically spaced center frequencies that correspond to musical notes. Other metrics were evaluated as well however only the RMSE-CQT was correlated with the quality of the generated audio. This (subjective) observation was initially made by listening to the audio samples and by comparing the plots of the audio waveforms (Figure 4 ). Roughly speaking, as we also show in Figure 4 (top captions) and Figure 3 (lower plots), we find that a RMSE-CQT value below 10 corresponds to a generated sample of reasonable quality. The RMSE-CQT also penalizes temporal delays (Figure 4 -right) and is also correlated with the MOS (Table 3 and Table 5 ).

We generate every 20 epochs during training and compute the RMSE-CQT to check generation quality. Indeed, Figure 3 shows that the generated signals match the target audio better as the training progresses, while the losses flatten. However, occasionally the generated signals are shifted or the melody is slightly inaccurate -the wrong note is played (Figure 4 -right). This is not necessarily only a function of the network weight state the generation process is stochastic. We set a fixed random seed at generation time, thus we only observe changes in the generated signal due to weight changes. To quantify error for one model, the RMSE-CQT is averaged over all epochs.

4.3. HYPERPARAMETER SELECTION:

There are many possible configurations when it comes to the filter width F , the number of blocks B, and the maximum dilation rate R. The dilation rates per each block are: {2 0 , 2 1 , . . . , 2 R-1 }. In addition there is the choice of the number of residual and skip channels. Since the latter two works are also focused on music, we use 512 channels for both the residual and skip convolutions and set the final two convolutions to 512 and 384 channels respectively.

We hypothesize that it is better to maximize the receptive field ∆ while minimizing the number of layers. Therefore, in our first experiments (Table 3 ) we limit the receptive field to 1 second and vary the other parameters according to Table 2 . We have observed that the networks train faster and the quality is better when the length of the audio slice is maximized within GPU memory constraints. It can be seen in Table 3 that SynthNet outperforms both baselines. Some instruments are more difficult to learn than others (also see Figure 3 ). This is also observable from listening to and visualizing the generated data (available here http://bit.ly/synthnet_table3).

The lowest errors for the first four instruments are observed for SynthNet L48 while the last three are lowest for SynthNet L24 (Table 3 slanted). This could be due to either an increased granularity over the frequency spectrum, provided by the extra layers of the L48 model or a better overlap. The

:

http://timidity.sourceforge.net/

Figures and Tables:

Figure 1: Gram matrix projection from Eq. 3 Layers in color, shapes are styles (timbre).

Figure 3: networks are trained, each with a different harmonic style. Top, losses: training (left) validation (right). Bottom, RMSE-CQT: DeepVoice (left [Tbl. 3, col. 6]) and SynthNet (right [Tbl. 3, col. 8]). DeepVoice overfits for Glockenspiel (top right, dotted line). Convergence rate is measured via the RMSE-CQT, not the losses. The capacity of DeepVoice is larger, so the losses are steeper.Instead, we propose to measure the root mean squared error (RMSE) of the Constant-Q Transform (RMSE-CQT) between the generated audio and the ground truth waveform (Figure3, lower plots). Similarly to the Fourier transform, the CQT(Brown, 1991) is built on a bank of filters, however unlike the former it has geometrically spaced center frequencies that correspond to musical notes.

Figure 4: Left: 1 second of ground truth audio of Bach's BWV1007 Prelude, played with FluidSynth preset 56 Trumpet. Center: SynthNet high quality generated. Right: DeepVoice low quality generated showing delay. Further comparisons over other instrument presets are provided in Appendix A. We encourage the readers to listen to the samples here: http://bit.ly/synthnet_appendix_a

Differences between the two baseline architectures and SynthNet.

Table content: <table><tr><td/><td/><td>Input</td><td/><td>Dilated conv</td><td>Skip</td><td/><td>Final block</td></tr><tr><td/><td>Channels</td><td>Type</td><td colspan="5">Activation Separable Connection 1x1 Conv Activation</td></tr><tr><td>WaveNet</td><td>1 Scalar</td><td>Conv</td><td>None</td><td>No</td><td>Yes</td><td>No</td><td>ReLU</td></tr><tr><td colspan="2">DeepVoice 256 1-hot</td><td>Conv</td><td>Tanh</td><td>No</td><td>Yes</td><td>Yes</td><td>ReLU</td></tr><tr><td colspan="2">SynthNet 1 Scalar</td><td>Embed</td><td>Tanh</td><td>Yes</td><td>No</td><td>No</td><td>SeLU</td></tr></table>

For speech Arik et al. (2017) use 64 residual channels and 256 skip channels, Engel et al. (2017) use 512 residual channels and 256 skip channels, while Mor et al. (2018) use 512 for both. These methods have receptive field ∆ < 1.

Table content: <table/>

Three setups for filter, dilation and number of blocks resulting in a similar receptive field. Filter width F Num blocks B Max dilation R Receptive field ∆

Table content: <table><tr><td>L24</td><td>3</td><td>2</td><td>12</td><td>1.0239 sec</td></tr><tr><td>L26</td><td>2</td><td>2</td><td>13</td><td>1.0240 sec</td></tr><tr><td>L48</td><td>2</td><td>4</td><td>12</td><td>1.0238 sec</td></tr></table>

. Two baselines are benchmarked for three sets of model hyperparameter settings (Table2), all other parameters identical. One second of audio is generated every 20 epochs (over 200 epochs) and the error versus the target audio is measured and averaged over the epochs, per instrument. Total number of parameters and training time are also given. All waveforms and plots available here: http://bit.ly/synthnet_table3 56±4.67 14.83±5.39 18.15±2.88 10.80±1.66 9.32±1.97 17.28±2.19 6.30±1.01 5.96±1.10 5.51±0.85 S09 24.01±5.38 22.20±4.56 25.47±3.96 22.65±5.25 17.54±3.86 27.48±2.55 11.58±1.50 12.53±2.37 10.91±1.40 S24 17.68±3.05 18.95±4.13 19.30±1.26 18.03±1.58 16.33±1.79 19.19±1.25 8.00±1.71 8.53±1.51 7.82±1.32 S42 15.83±3.91 17.20±3.53 16.29±3.38 11.92±0.94 13.77±2.06 13.89±1.73 8.61±1.12 8.84±0.96 8.33±0.74 S56 18.50±2.23 17.25±2.98 22.89±1.73 17.04±0.34 17.16±1.05 21.45±1.37 8.90±0.95 10.37±1.41 8.97±1.42 S75 20.89±6.90 20.03±6.73 19.78±5.15 11.93±1.30 12.75±1.93 11.30±0.64 9.68±1.22 9.83±1.10 10.20±1.60 S80 27.73±2.29 26.74±3.92 26.96±4.71 20.41±1.80 20.09±2.91 20.95±2.77 5.14±1.46 7.66±2.31 7.91±2.41 All 20.02±1.79 19.60±1.73 21.35±1.48 16.18±1.30 15.31±1.10 18.57±1.36 8.32±0.63 9.10±0.70 8.52±0.62

Table content: <table><tr><td/><td/><td>WaveNet</td><td/><td/><td>DeepVoice</td><td/><td/><td>SynthNet</td><td/></tr><tr><td/><td>L24</td><td>L26</td><td>L48</td><td>L24</td><td>L26</td><td>L48</td><td>L24</td><td>L26</td><td>L48</td></tr><tr><td colspan="2">S01 16.Params 8.23e+7</td><td>6.18e+7</td><td>1.14e+8</td><td>8.90e+7</td><td>6.89e+7</td><td>1.27e+8</td><td>7.35e+6</td><td>7.80e+6</td><td>1.36e+7</td></tr><tr><td>Time</td><td>4d2h</td><td>4d2h</td><td>8+ days</td><td>4d10h</td><td>3d9h</td><td>8+ days</td><td>16 hours</td><td>16 hours</td><td>1d5h</td></tr></table>

Peer review (iclr2019):
 This paper proposes a neural model for synthesizing instrument sounds, using an architecture based on the WaveNet and DeepVoice models. The model generates raw waveforms conditioned on a piano roll representation of aligned MIDI input. My biggest gripe with this work is that the model is trained entirely on a synthetic dataset generated from a sample-based synthesizer using a sound font. I feel that this defeats the purpose, as it will never work better than just sampling the original sound library. One potential argument in favour would be to save storage space, but the sound font used for the work is only ~140 MB, which is not prohibitive these days (indeed, many neural models require a comparable amount of storage). It would be much more interesting to train the model on real instrument recordings, because then it could capture all the nuances of the instruments that sample-based synthesizers cannot replicate. As it stands, all the model has to do is reproduce a fixed (and fairly small) set of audio samples. This is arguably a much simpler task, which could also explain why reducing the model size (SynthNet*s depthwise convolutions have many fewer parameters than the regular convolutions used in WaveNet and DeepVoice) works so well here. That said, I think the proposed architectural modifications for raw audio models could be interesting and should be tested for other, more challenging tasks. The proposed RMSE-CQT error measure is potentially quite valuable for music generation research, and its correlation with MOS scores is promising (but this should also be tested on more realistic audio). The fact that the models were trained to convergence on only 9 minutes of data per instrument is also impressive, despite the limitations of the dataset. The use of dithering to reduce perceptual noise is also interesting and some comparison experiments there would have been interesting, especially to corroborate the claim that it is critical for the learning process. I think the paper slightly overstates its contributions in terms of providing insight into the representations that are learned in generative convolutional models. The Gram matrix projections showing that the activations of different layers diverge for different input types as we advance through the model is not particularly surprising, and similar plots could probably be made for almost any residual model. Overall, I feel the work has some fundamental flaws, mostly stemming from the dataset that was used. Miscellany: - In the abstract: "is substantially better in quality", compared to what? - In the introduction, it is mentioned that words in a speech signal cannot overlap, but notes in a musical signal can. I would argue that these are not comparable abstractions though, words themselves are composed of a sequence of phonemes, which are probably a better point of comparison (and phonemes, while they don*t tend to overlap, can affect neighbouring phonemes in various ways). That said, I appreciate that this is probably quite subjective. - Overall, the formulation of paragraph 2 of the introduction is a bit unusual, I think the same things are said in a much better way in Section 3. - "Conditioning Deep Generative Raw Audio Models for Structured Automatic Music" by Manzelli et al. (2018) also proposes a MIDI-conditional neural audio generation model, trained on real instrument recordings from the MusicNet dataset. I think this is a very relevant reference. - In the contributions of the paper, it is stated that "the generated audio is practically identical to ground truth as can be seen in Figure 4" but the CQTs in this figure are visibly different. - I don*t think it is fair to directly compare this setup to Engel et al. (2017) and Mor et al. (2018) as is done in the last paragraph of Section 2, as these are simply different tasks (mapping from audio to audio as opposed to generating audio). - At the start of Section 3.1 it would be good to explicitly mention whether 8-bit mu-law audio is used, to explain why the waveform is 256-valued. - Why is the conditioning causal? It does not need to be, as the piano roll is fully available in advance of the audio generation. I guess one argument in favour would be to enable real-time generation, but it would still be good to compare causal and non-causal conditioning. - Since the piano roll representation is binary, does that mean MIDI velocity is not captured in the conditioning signal? It would probably be useful for the model to provide this information, so it can capture the differences in timbre between different velocities. - The use of a MIDI prediction loss to regularise the conditioning part of the model is interesting, but I would have liked to see a comparison experiment (with/without). - In Section 4.3, specify the unit, i.e. "Delta < 1 second". - For the task of recreating synthetic audio samples, the WaveNet models seem to be quite large. As far as I can tell the size hyperparameters were chosen based on the literature, but the inherited parameters were originally optimised for different tasks. - In Section 4.3 under "global conditioning", the benchmark is said to be between DeepVoice L26 and SynthNet L24, but Table 4 lists DeepVoice L26 and SynthNet L26, which version was actually used?