Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes an end-to-end method for audio super resolution. The proposed approach treats this problem as a regression task: predict the high resolution audio signal from the lower resolution one operating directly with raw audio samples.

Pros:

+ The method is very simple and straight forward (if it works well, it would be a big plus)
+ The paper is well written.

Cons:

+ The method optimizes a measure that is not correlated with perceptual quality.
+ Current results seem to be competitive with previous approaches in terms of PSNR, but it is not clear the perceptual relevance

It would be very surprising to me that using an L2 loss in the time domain would lead to good perceptual results, particularly when having large upscaling factors, as there is a lot of uncertainty in the desired reconstruction. While some high frequencies can be predicted based on the statistics of the training data, some aspects of the signal are essentially unpredictable. For instance, predicting in time domain the samples of an unvoiced sound (which is essentially colored noise) would be impossible, thus, the best strategy (in terms of L2) would be to predict the mean. Because of this, I would expect a good reconstruction in terms of MSE would lead to smooth signals. The results shown in Figure 2, show that this is certainly not very dramatic, but ultimately qualitative evaluation is key.

A similar issue is encountered when up-sampling images, where people have looked for better losses, see for instance (many more references look at the same issues):

Bruna, et al. "Super-resolution with deep convolutional sufficient statistics." ICLR 2016.
Johnson, et al. "Perceptual losses for real-time style transfer and super-resolution." ECCV, 2016.
Ledig, Christian, et al. "Photo-realistic single image super-resolution using a generative adversarial network." arXiv preprint arXiv:1609.04802 (2016).

I am not surprised that the model gives good results in terms of SNR, as this is the loss being optimized for. However, it is well known that SNR is not linked to good perceptual quality. It would be interesting to provide PESQ scores for the speech samples, as well as audio examples of the enhanced samples (if not a perceptual evaluation). 

How does the proposed model compare with Li et al in terms of computational complexity? Meaning, operating in raw audio should be much more demanding than in higher level features.

Please provide more details regarding the architecture. I understand that the up-sampled signal could be a good input to the network, but it is a bit counter intuitive to me to up-sample the audio signal to then use downsampling modules. Why not to just start from the original input?

The authors mention that this technique would help speed up methods that work directly in the audio domain (I assume that by producing compressed samples and then expanding them with the proposed method). For example in WaveNet type of models, it could be a good idea to replace the L2 norm with the likelihood function given by the learned model. That would be in my a opinion a more sensible loss function. 

I am open to reconsider my review based on the the authors response and, in particular, if they provide audio samples.

SCORE UPDATED FROM 4 to 6.
review_title: Review

final_score: rating: 3: Clear rejection
review_content: The paper proposes a neural network approach to audio upsampling, or equivalently, reconstructing high-frequency signal components from low-frequency components. Most of the paper is pretty clearly written, but it has some significant flaws in model design and evaluation.

- The introduction refers to Zhang et al. (2017) for speech recognition on raw audio, instead of much earlier work by e.g. Sainath et al., Jaitly et al. and Hoshen et al.

- The statement in section 2 that "very little work has been done on audio signals" is unnecessarily broad and ignores a large body of prior work on using neural networks for e.g. source separation.

- All citations seem to be in-text (using \citet), but this is not appropriate in most cases.

- The definition of R in section 2 is wrong. It is supposed to represent the sample rate, but following this definition it actually represents the total number of samples (which is only equal to the sample rate in Hz if the audio signal is exactly one second long). This is confusing.

- The model described in section 3 / figure 1 has some unusual architectural properties that aren't really justified anywhere. The residual blocks only seem to contain a single ReLU activation layer, so that means the network has multiple linear layers directly following each other in many places, which is redundant. If this is not the case, the model should be described more clearly.

- Why use filter size 9? The motivation for using large filters is not explained. It would also be useful to discuss the use of dilated convolutions instead (and why the authors chose not to use them).

- The downsampling procedure ("... produce corresponding low-resolution patches by recording every r-th position in j...") is improper: an anti-aliasing low-pass filter needs to be applied before decimation. (Unless all input signals are expected to be obtained by downsampling high-resolution signals this way, but that would defeat the point.)

- The description of magnatagatune in section 4 is wrong: the 188 tags do not correspond to genres, and they are not mutually exclusive. The validation split used is also non-standard and random, making reproduction more difficult.

- Figure 2 is difficult to interpret. The spectrogram for the low-resolution signal seems to have been produced with an FFT that assumes the full samplerate, as it stands this shows the spectrogram of a signal that was sped up 4x, so it's unclear how this should actually be interpreted. Comparing the first and 3rd spectrogram, it looks like it features some spurious harmonics (e.g. around timestep 300). It would be useful to provide the corresponding audio samples somewhere.

- PSNR in the time domain is a poor error measure for this task, because it does not take into account human perception: logarithmic loudness perception, frequency masking, frequency sensitivity (Fletcher-Munson), phase insensitivity etc. It is difficult to find the right way to measure this quantitatively, but providing more than one measure would be much more informative.

- The paper is missing baselines. Even measurements for a few simple upsampling strategies (nearest-neighbour, bilinear, ...) would be very informative. As it stands the numbers are difficult to interpret.

review_title: significant flaws in model design and evaluation

Decision: Accept