Title: ADVERSARIAL AUDIO SUPER-RESOLUTION WITH UNSUPERVISED FEATURE LOSSES

Abstract: Neural network-based methods have recently demonstrated state-of-the-art results on image synthesis and super-resolution tasks, in particular by using variants of generative adversarial networks (GANs) with supervised feature losses. Nevertheless, previous feature loss formulations rely on the availability of large auxiliary classifier networks, and labeled datasets that enable such classifiers to be trained. Furthermore, there has been comparatively little work to explore the applicability of GAN-based methods to domains other than images and video. In this work we explore a GAN-based method for audio processing, and develop a convolutional neural network architecture to perform audio super-resolution. In addition to several new architectural building blocks for audio processing, a key component of our approach is the use of an autoencoder-based loss that enables training in the GAN framework, with feature losses derived from unlabeled data. We explore the impact of our architectural choices, and demonstrate significant improvements over previous works in terms of both objective and perceptual quality.

1. INTRODUCTION:

Deep convolutional neural networks (CNNs) have become a cornerstone in modern solutions for image and audio analysis. Such networks have excelled at supervised discrimination tasks, for instance on ImageNet (Deng et al., 2009; Simonyan & Zisserman, 2014) , where image classifier networks are trained on a large corpus of labeled data. More recently, CNNs have successfully been applied to data synthesis problems in the context of generative adversarial networks (GANs) (Goodfellow et al., 2014) . In the GAN framework, a neural network is used to synthesize new instances from a modeled distribution, or resolve missing details given lossy observations. In the latter case, the GANs have been shown to greatly improve reconstruction of fine texture details for images, compared to standalone sample-space losses that result in overly smoothed outputs (Dosovitskiy & Brox, 2016; Isola et al., 2017; Ledig et al., 2017) . However, GANs are notoriously hard to train, and the use of conventional sample-space objectives in conjunction with an adversarial loss either de-stabilizes training, or results in outputs with significant artifacts (Figure 1 ).

To address the smoothness problem described above, previous works typically augment or replace conventional sample-space losses with a feature loss (also called a perceptual loss) (Dosovitskiy & Brox, 2016; Ledig et al., 2017; Johnson et al., 2016) . Instead of distance in raw sample-space, such feature losses reflect distance in terms of the feature maps of an auxiliary neural network. While classifier-based feature losses are effective, they require either a pre-trained neural network that is applicable to the problem domain (e.g., synthesizing images of cats), or a labeled dataset that is amenable to training a relevant classifier. Training new classifiers for use in a feature loss can be non-trivial for numerous reasons. Besides the difficulty of training large classifiers that are commonly used for feature losses, such as VGG (Simonyan & Zisserman, 2014) , creating a labeled dataset that is sufficiently large and diverse is often infeasible.

In this work, we sidestep the difficulty of training auxiliary classifiers by developing a feature loss that is unsupervised. In particular, we focus on an audio modeling task called super-resolution, where the goal is to generate high-quality audio given down-sampled, low-resolution input. Inspired by previous work on audio and image super-resolution, we develop a neural network architecture for end-to-end super-resolution that operates on raw audio. In addition to providing new algorithms to model audio, our work suggests new techniques to improve GAN-based methods in other domains such as images and video. Specifically, our contributions are as follows:

1. We formulate a new, general-purpose feature loss that is fully unsupervised and circumvents the need for problematic classifier-based models.

2. We successfully adapt the adversarial framework for audio processing, and provide solutions to previously unsolved problems associated with the application of GANs to audio.

3. We demonstrate our methods in an end-to-end architecture for audio super-resolution, with stateof-the-art results on both speech and music tasks.

4.:

We analyze important architectural parameters of our model, and in particular discover previously-unobserved behavior with effective receptive field sizes.

2. BACKGROUND & RELATED WORK:

Audio super-resolution Audio super-resolution is the task of constructing a high-resolution audio signal from a low-resolution signal that contains a fraction of the original samples. Concretely, given a low-resolution sequence of audio samples x l = (x 1/R l , . . . , x R l T /R l ), we wish to synthesize a high-resolution audio signal

x h = (x 1/R h , . . . , x R h T /R h ),

where R l and R h are the sampling rates of the low and high-resolution signals, respectively. We denote R = R h /R l as the upsampling ratio, which ranges from 2 to 6 in this work. Thus, the audio super-resolution problem is equivalent to reconstructing the missing frequency content between frequencies R l /2 and R h /2.

There is a vast body of prior work on audio super-resolution in the signal and audio processing communities under the term artificial bandwidth extension (Larsen & Aarts, 2004) . Neural networkbased methods in this domain generally apply a DNN on top of hand-crafted features as part of complex bandwidth extension systems (Liu et al., 2015; Abel & Fingscheidt, 2018) . Gaussian mixture and hidden Markov models have also been used (Bachhav et al., 2017; Tokuda et al., 2013) , but these methods generally perform worse compared to neural networks (Abel & Fingscheidt, 2018) .

In contrast with the works above, our method does not rely on hand-crafted features (e.g., transformations or cepstrum coefficients), and is not specific to problems in speech modeling.

Audio modeling with neural networks Learning-based approaches for audio have also been explored in the largely in the context of representation learning, generative modeling, and text-tospeech (TTS) systems. Unsupervised methods such as convolutional deep belief networks (Lee et al., 2009) and bottleneck CNNs (Aytar et al., 2016) have been shown to learn useful representations from audio, such as phonemes and sound textures. Stacked autoencoders (Vincent et al., 2010) and variational autoencoders (Kingma & Welling, 2014; Sonderby et al., 2016) have been used for denoising, image generation, and music synthesis (Sarroff & Casey, 2014) . Bottleneck-like CNNs have also demonstrated significant improvements for audio super-resolution in supervised settings compared to previous DNN and spline-based methods (Kuleshov et al., 2017) . Donahue et al. (2018) is among the first to develop methods for raw audio synthesis with GANs. Notably, Donahue et al. (2018) show that non-trivial modifications of GAN architectures are required to generative diverse and plausible audio outputs. We build on the works above by developing a GAN framework for audio super-resolution with an improved bottleneck-style generator, and show that leveraging representations learned from unsupervised training greatly aid the super-resolution task. Autoregressive probabilistic models have recently demonstrated state-of-the-art results for generation of music (Engel et al., 2017 ), general audio (van den Oord et al., 2016; Mehri et al., 2017) , and for parametric TTS systems (Sotelo et al., 2017) . Several works have leveraged model distillation (van den Oord et al., 2018; ClariNet, 2019) 1 to mitigate the overhead of autoregressive methods, making them feasible for real-time audio generation. In general, our work can be used to augment existing speech synthesis systems, including those that employ autoregressive methods. For instance, the unsupervised feature loss proposed in our work could be used as a drop-in replacement for the classifier-based feature loss used by van den Oord et al. ( 2018). While we are not aware of any efforts that explore autoregressive modeling for audio super-resolution, we believe it may be a promising future direction.

Generative adversarial networks for images Generative methods have been extensively explored for image generation and super-resolution. Building upon the original formulation of Goodfellow et al. ( 2014), GANs have been continuously improved to generate plausible, high-fidelity images (Radford et al., 2015; Denton et al., 2015; Berthelot et al., 2017; Karras et al., 2018) . GAN variants conditioned on class labels or object sketches have also demonstrated promising results on tasks such as in-painting and style transfer (Mirza & Osindero, 2014; Isola et al., 2017) .

3. METHOD:

GANs for Super-Resolution GANs developed for super-resolution tasks have several important differences compared to the original formulation of Goodfellow et al. (2014) . When used to generate new instances from a data distribution p data , the generator (G) parameterized by θ G learns the mapping to data space as G(z; θ G ), where z is a latent noise prior. The discriminator (D) parameterized by θ D then estimates the probability that G(z; θ G ) was drawn from p data rather than the generator distribution p g . In contrast, for super-resolution, G is no longer conditioned on noise and learns the mapping to high-resolution data space p h as G(x l ; θ G ), where x l is drawn from the low-resolution data distribution p l . The task of D is to discriminate between samples from the high-resolution and super-resolution (generator) distributions p h and p g , respectively. Since low-resolution data x l corresponds directly to a downsampled version of x h during training, we expect G(x l ; θ G ) ≈ x h . G and D are optimized according to the two-player minimax problem:

EQUATION

This framework enables the joint optimization of two neural networks -G generates super-resolution data with the goal of fooling D, and D is trained to distinguish between real and super-resolved data. Thus, the GAN approach encourages G to learn solutions that are hard to distinguish from real, highresolution datum.

Architecture overview MU-GAN (Multiscale U-net GAN) is composed of three models that all operate on raw audio2 -a generator (G), discriminator (D), and convolutional autoencoder (A) (Figure 2 ). The generator's task is to learn the mapping between the low and high-resolution data spaces, corresponding to signals x l and x h , respectively. The discriminator's task is then to classify whether presented data instances are real, or produced by the generator. In addition to G and D, the autoencoder extracts perceptually-relevant features from both real and super-resolved data for use in feature-space loss functions. The use of A is crucial in the GAN framework, as generators trained solely on L2 or other sample-space losses suffer from training instability or output artifacts (Ledig et al., 2017) . receptive fields are required to create high quality raw audio (Donahue et al., 2018; van den Oord et al., 2016) . Previous work with classifier models also suggests that varying the filter size within a network helps capture information at multiple scales (Szegedy et al., 2015) . Leveraging these observations, we use a multiscale convolutional building block composed of concatenated 3x1, 9x1, 27x1, and 81x1 filters. In practice, and with a fixed number of parameters for a given layer, we found that filters larger than 81x1 provided no additional benefit, while omitting large filter sizes resulted in significantly degraded audio quality. We interpret the poor performance of small filters as being a byproduct of their frequency selectivity; it is well known from signal processing theory that the resolution of an FIR filter's frequency response is proportional to the length of the filter.

Multiscale convolutional layers:

Superpixel layers Recently, it has been shown that pooling and strided convolutions tend to induce periodic "checkerboard" artifacts (Odena et al., 2016; Donahue et al., 2018) . Shi et al. (2016) developed a subpixel layer to increase spatial resolution, and showed that it is less prone to checkerboard artifacts. While the subpixel layer was subsequently adopted by several works (Kuleshov et al., 2017) , no efforts have evaluated the performance of the inverse operation for decreasing spatial resolution. Concretely, the inverse subpixel operator interleaves samples from the time dimension into the channel dimension, and thus reduces the spatial resolution by an integer factor. We refer to this simple inverse operation as a superpixel layer (Figure 3 ), and use it as a drop-in replacement for strided convolution and pooling layers.

Generator network:

The high-level architecture for the generator network (Figure 4 , top) is inspired by autoencoder-like U-net models (Ronneberger et al., 2015; Isola et al., 2017; Kuleshov et al., 2017) . In a U-net-style model, the first half of the network consists of B downsampling blocks (D-blocks) that perform feature extraction at multiple scales and resolutions3 . The second half the model consists of B upsampling blocks (U-blocks), which successively increase the spatial resolution of the signal. We use multi-scale convolutional layers throughput the generator network, and replace all strided convolutions with superpixel layers.

Discriminator network The discriminator (Figure 4 , bottom) is used during training to differentiate between real, high-resolution audio and super-resolved signals produced by the generator. Our design is loosely based on the recommendations of Radford et al. (2015) , and the image discriminator from Ledig et al. (2017) . All discriminator activations are LeakyReLU (Maas et al., 2013) with α = 0.2. As with the generator, we use multi-scale convolutions, and the superpixel layer described above instead of strided convolutions to minimize artifacts in the loss gradients (Odena et al., 2016) .

Autoencoder network:

The autoencoder A is used to extract perceptually relevant features from the low and high-resolution signals. The features extracted by A are incorporated in the generator's feature loss L f , which is described in more detail in following sections. For the specific implementation of A, we use a modified version of the generator model that excludes all additive and stacking skip connections. Hence, the model for A is a convolutional autoencoder, augmented with multiscale convolutional layers, and super/subpixel layers for down/up-sampling.

Loss functions MU-GAN incorporates several loss terms for training the generator and discriminator. The first term in the generator loss is the sample-space L2 loss, given by4 

L L2 = 1 W W i=1 x h,i -G(x l ) i 2 2 .

(2)

We found that using only the sample-space and adversarial losses either resulted in little to no improvement over the baseline non-GAN model, or introduced persistent audible artifacts (e.g., high-frequency tones, Figure 1 ). These findings are in line with those of Ledig et al. (2017) , who experience similar issues with images5 . As described in Section 2, the use of a feature loss with GAN training encourages the generator to learn solutions that incorporate perceptually relevant texture details. Given the autoencoder A, we denote the output feature tensor at the bottleneck of the autoencoder as φ. The feature loss L f is then given by

EQUATION

where W f and C f denote the width and channel dimensions for the feature maps of autoencoder bottleneck. The adversarial loss L adv is determined by discriminator's ability to discern whether data produced by the generator is real or fake. We use the gradient-friendly formulation originally posed in Goodfellow et al. (2014) , given by

EQUATION

The composite loss L G for the generator is then given by the sum of the losses above, and the discriminator loss L D derives directly from the GAN optimization objective in Equation 1, i.e.,

EQUATION

EQUATION

where λ f and λ adv are constant scaling factors.

4. EXPERIMENTS:

Datasets We evaluate our methods on three super-resolution tasks derived from the VCTK Corpus (Yamagishi) , and the non-vocal music dataset from Mehri et al. (2017) . For speech from VCTK, we compose a dataset with recordings from a single speaker (the Speaker1 task), and a dataset with recordings from multiple speakers (the Speaker99 task). Speaker1 consists of the first 223 recordings from VCTK speaker 225 for training, and the final 8 recordings for testing. Speaker99 uses all recordings from the first 99 VCTK speakers for training, and recordings from the last 10 speakers for testing. Piano uses the standard 88%-6%-6% train/validation/test split. For all tasks, the dataset is created by first applying an anti-aliasing lowpass filter, and then sampling random patches of fixed length from the resulting audio. Note that for the sake of direct comparison, the datasets above are the same as those used in Kuleshov et al. (2017) .

Training methodology For Speaker1, we instantiate variants of MU-GAN and train for 400 epochs. For the larger datasets Speaker99 and Piano, models are trained for 150 epochs. The epoch number is empirically selected based on observed convergence, and performance saturation on the validation set. For all models, we use the ADAM optimizer with learning rate 1e-4, β 1 = 0.9, β 2 = 0.999, and a batch size of 32. For the autoencoder feature losses, we instantiate a model with L = 4, and train for 400 epochs on the same dataset as its associated GAN model. The loss scaling factors λ f and λ adv are fixed at 1.0 and 0.001, respectively. Additional details on model hyperparameters can be found in Appendix A.1.

Performance metrics We use three metrics to assess the quality of super-resolved audio:

(1) signal-to-noise ratio (SNR), ( 2) log-spectral distance (LSD), and (3) mean opinion score (MOS).

The SNR is a standard metric in signal processing communities, defined as

EQUATION

)

where x is an approximation of reference signal x ref .

LSD (Gray & Markel, 1976) measures differences between signal frequencies, and has better correlation with perceptual quality compared to SNR (Jie et al., 2014; Kuleshov et al., 2017) . Given short-time discrete Fourier transforms X and X ref , the LSD is given by LSD (X,

EQUATION

)

where w and k are the window and frequency bin indices, respectively6 . Perceptual evaluation of speech quality (PESQ) (ITU-T, 2001) is an industry-standard methodology for the assessment of speech communication systems. Given reference and degraded audio signals, PESQ models the mean opinion score (MOS) of a group of listeners. Specifically, we use PESQ to produce MOS-LQO (listening quality objective) scores (ITU-T, 2003), which range from 1 to 5.

Impact of superpixel layers:

We find that the use of superpixel layers results in ∼14% improvement in training time across model sizes, with insignificant differences in terms of objective quality metrics. Differences in audio produced by the two methods were also imperceptible in informal selfblinded listening tests. This indicates that superpixel layers may be a suitable replacement for conventional strided convolutions, while offering improvements in training time without performance loss. Additional details are provided in Appendix A.2.

Objective performance evaluation Table 1 shows the quantitative performance of MU-GAN against other recent works. We denote MU-GAN8 as an instance of MU-GAN with a depth parameter of L = 8, i.e., with 8 downsampling and 8 upsampling blocks. U-net4 is the model with L = 4 from Kuleshov et al. (2017) . To eliminate depth as a factor in the performance comparison, we reimplement the architecture from Kuleshov et al. ( 2017) with L = 8, denoted as U-net8.

Table 1 shows that MU-GAN8 often performs worse in terms of SNR compared to the baseline models, but has lower LSD and higher MOS-LQO. This indicates that while MU-GAN8 produces reconstructions with lower SNR, deviating in terms of sample-wise distance results in synthesis of more perceptually-relevant frequency content. The exception is with the Piano task, where MU-GAN8 performs orders of magnitude better than the U-net baseline in terms of SNR. In general, we 

. Up. Ratio R = 2 Up. Ratio R = 4 Up. Ratio R = 6 U-net4 U-net8 MU-GAN8 U-net4 U-net8 MU-GAN8 U-net4 U-net8 MU-GAN8 Speaker1 SNR

R = 4: Piano #1, Speaker99 #1, #3 R = 6: Piano #2, Speaker99 #2, #4

Table 3 : Speaker1 objective metrics for MU-GAN8 trained with the speech classifierbased loss, and proposed loss (L f,SV , L f ). also find that performance on the speech tasks generally saturates at R = 2 for both U-net8 and MU-GAN8. Informal listening tests confirm that there are minimal differences at R = 2, indicating that more difficult up-sampling ratios (i.e., R = 4, 6) are better suited for grounds of further comparison.

R = 2 R = 4 R = 6 L f,SV L f L f,SV L f L f,

Subjective quality analysis To evaluate the performance of MU-GAN with real listeners, we perform a randomized, single-blinded user study with 22 participants (Table 2 ). The study presents pairs of audio clips produced by MU-GAN8 and the best baseline model U-net8, and asks participants to select a preferred clip, or "No preference." We present two clips from Piano, and four sonically diverse clips from Speaker99. Table 2 shows that in all cases, listeners prefer audio produced by MU-GAN8 over the baseline method.

In general, we observe that audio produced by MU-GAN has greater clarity compared to audio produced by the baseline networks. The quality difference is most apparent during consonant sounds, which have more high-frequency content compared to typical vowel sounds. For instance, in the phrase "Ask her to bring these things from the store," (Figure 5 , bottom row) the consonant sounds in 'Ask,' 'things,' and 'store' have noticeably better articulation. In contrast, audio from the best baseline, U-net8, sounds relatively dull and "muffled" in comparison. Note that on some audio clips super-resolved at R = 4, 6, we observe intermittent, audible noise that is not present in the baseline reconstructions. To ensure a fair analysis, we include audio clips in the user study where this noise is apparent. Additional comments on spurious noise can be found in Appendix A.3. (Mesaros et al., 2018) . Table 3 shows objective results obtained using the proposed loss L f and the classifier-based loss L f,SV on the Speaker1 task. Across all up-sampling ratios, the proposed unsupervised method performs on-par (and slightly better in some cases) compared to the classifier-based loss. Thus, our results suggest that using a domain-specific classifier-based loss may not provide any advantage in terms of performance. Given the issues related to training classifier models on general audio (Section 1), our method may be an attractive solution that does not compromise audio quality. Ablation Analysis Table 4 shows the MOS-LQO metrics for the MU-GAN architecture with ablated model parameters. While almost all variations perform similarly well at R = 2, adding depth (i.e., from L = 4 to L = 8) and additional loss terms improves performance on harder up-sampling ratios. Furthermore, adding the adversarial loss and unsupervised feature loss terms improve MOS-LQO monotonically. For MU-GAN8, we see diminishing returns from adding the additional loss terms; much of the improvement over MU-GAN4 appears to come from the additional depth. On the other hand, adding the L f and L adv losses to the MU-GAN4 variant yields significant benefits, such that its performance is comparable to that of MU-GAN8. This indicates that the feature and adversarial losses may be particularly useful to mitigate underfitting, or to decrease model size iso-performance.

-GAN4 -L f -L adv MU-GAN4 -L adv (+L f ) MU-GAN4 (+L adv + L f ) MU-GAN8 -L f -L adv MU-GAN8 -L adv (+L f ) MU-GAN8 (+L adv + L f ) R = 2 3.

Receptive Field Analysis While previous works have stressed the importance of large receptive fields (van den Oord et al., 2016; Luo et al., 2016; Yu & Koltun, 2016) , little work exists to quantify receptive field sizes on practical problems. We use the methods of Luo et al. (2016) to measure the effective receptive field (ERF)8 of our model on different tasks with varying architectural hyperparameters. Notably, we find that while the theoretical receptive field of our network is on the order of hundreds of milliseconds (thousands of samples), measured ERF's are generally no wider than 100 samples. Furthermore, we find that ERF size is strongly correlated with problem difficulty, rather than architectural hyperparameters such as depth and specific loss terms. While Luo et al. (2016) found that ERF size always increased compared to the ERF at initialization, we find that ERF decreases in many cases (Figure 6 ). Our findings imply that there may be subtle but important tradeoffs involved with receptive field size (for instance, ERF size versus noise rejection), and suggest promising avenues for deeper investigation of architectures that rely on large receptive fields. 

5. CONCLUSION:

In this work we develop methods to enable the application of GANs to audio processing, in particular with classifier-free feature losses. In addition to several new model building blocks, we show that a convolutional autoencoder can be used to implement a high-performance feature loss in the context of audio super-resolution. Demonstrated on several speech and music super-resolution tasks, we show that our architecture achieves state-of-the-art performance in both objective and subjective metrics. We perform a detailed analysis of our model, and also show that effective receptive field size may be an important property that is not well-explored. Finally, our work raises new possibilities for the design and analysis of neural network-based synthesis methods in important problem domains beyond audio processing.

:

https://openreview.net/forum?id=HklY120cYm

Data is encoded in

32-bit floating point format.

Note that to have matching resolutions at the input and output of G, the LR signal is first upsampled with a cubic spline.

We write losses with respect to a single sample, with an implicit mean over the minibatch dimension.

Ledig et al. (2017) posit that this poor performance is due to the competing nature of the adversarial and sample-space losses.

We use non-overlapping Fourier transform windows of length 2048.

The authors' pre-trained classifier models are obtained from https://github.com/francoisgermain/SpeechDenoisingWithDeepFeatureLosses

As inLuo et al. (2016), we define samples that lie within 2σ of the gradient-magnitude sample mean as "within-ERF."

Figures and Tables:

Figure 1: (a) High-resolution, (b) low-resolution, (c) and super-resolution spectrograms. The superresolution spectrogram corresponds to audio generated by the GAN described in Section 3, trained only with an adversarial loss and conventional L2 loss. Training tends to either diverge, or results in audio with persistent high-frequency tones and alias-like artifacts.

Figure 4: Generator and discriminator models.

Figure 5: Spectrograms from the Speaker1 task at R = 2 (top row, Speaker 225), and Speaker99 task at R = 4 (bottom row, Speaker 360). (a-d) high-resolution, (b-e) super-resolved with U-net8, and (c-f) super-resolved with MU-GAN8. Increased synthesis of high-frequency content by MU-GAN8 becomes more pronounced at difficult up-sampling ratios.

Figure 6: Measured gradient magnitude at the model input and associated Gaussian fit with MU-GAN4 for (a-c) R 2, 4, 6, respectively. (d) 2σ ERF trend, compared to ERF at initialization.

Objective comparison with baseline super-resolution networks †

Table content: <table/>

Metrics for U-net4 are taken directly from Kuleshov et al. (2017); those for U-net8 are from our reimplementation.

Table content: <table><tr><td/><td/><td>21.1</td><td>21.94</td><td>21.40</td><td>17.1</td><td>18.68</td><td>17.72</td><td>14.4</td><td>14.85</td><td>13.98</td></tr><tr><td/><td>LSD</td><td>3.2</td><td>2.24</td><td>1.63</td><td>3.6</td><td>2.34</td><td>1.92</td><td>3.4</td><td>2.92</td><td>1.95</td></tr><tr><td/><td>MOS-LQO</td><td>-</td><td>4.54</td><td>4.54</td><td>-</td><td>3.81</td><td>3.79</td><td>-</td><td>2.97</td><td>3.21</td></tr><tr><td/><td>SNR</td><td>20.7</td><td>20.05</td><td>20.01</td><td>16.1</td><td>14.30</td><td>14.03</td><td>10.0</td><td>11.11</td><td>10.92</td></tr><tr><td>Speaker99</td><td>LSD</td><td>3.1</td><td>2.22</td><td>2.14</td><td>3.5</td><td>2.92</td><td>2.72</td><td>3.7</td><td>3.23</td><td>2.97</td></tr><tr><td/><td>MOS-LQO</td><td>-</td><td>3.68</td><td>3.75</td><td>-</td><td>2.68</td><td>2.93</td><td>-</td><td>2.44</td><td>2.69</td></tr><tr><td>Piano</td><td>SNR LSD</td><td>30.1 3.4</td><td>44.98 1.12</td><td>52.03 0.90</td><td>23.5 3.6</td><td>31.71 1.35</td><td>32.28 1.30</td><td>16.1 4.4</td><td>22.53 1.53</td><td>24.71 1.41</td></tr><tr><td>†</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

A/B test user study scores.

Table content: <table><tr><td/><td colspan="2">Piano</td><td/><td colspan="2">Speaker99</td><td/></tr><tr><td/><td colspan="2">#1 #2</td><td colspan="4">#1 #2 #3 #4</td></tr><tr><td>MU-GAN8</td><td colspan="2">9 15</td><td colspan="4">14 11 15 10</td></tr><tr><td>U-net8 (baseline)</td><td>5</td><td>3</td><td>4</td><td>6</td><td>4</td><td>8</td></tr><tr><td>No preference</td><td>8</td><td>4</td><td>4</td><td>5</td><td>3</td><td>4</td></tr></table>

MOS-LQO for ablated models on the Speaker1 task.

Table content: <table><tr><td>Configuration</td></tr></table>

Peer review (iclr2019):
 PRO’s: +well-written +nice overall system: GAN framework for super-sampling audio incorporating features from an autoencoder +some good-sounding examples CON’s: -some confusing/weakly-presented parts (admittedly covering lots of material in short space) -I am confused about the evaluation; would like additional qualitative/observational understanding of what works, including more on how the results differ from baseline SUMMARY: The task addressed in this work is: given a low-resolution audio signal, generate corresponding high-quality audio. The approach is a generative neural network that operates on raw audio and train within a GAN framework. Working in raw sample-space (e.g. pixels) is known to be challenging, so a stabilizing solution is to incorporate a feature loss. Feature loss, however, usually requires a network trained on a related task, and if such a net one does not already exist, then building one can have its own (possibly significant) challenges. In this work, the authors avoid this auxiliary challenge by using unsupervised feature losses, taking advantage of the fact that any audio signal can be downsampled, and therefore one has the corresponding upsampled signal as well. The training framework is basically that of a GAN, but where, rather than providing the generator with a low-dimensional noise signal input, they provide the generator with the subsampled audio signal. The architecture includes a generator ( G(lo-fidelity)=high-fidelity ), a discriminator ( D(high-fidelity) = real or by super-sampled ? ), and an autoencoder ( .phi( signal x) = features of signal x at AE’s bottleneck). COMMENTS: The generator network appears to be nearly identical to that of Kuleshov et al (2017)-- which becomes the baseline-- and so the primary contribution differentiating this work is the insertion of that network into a GAN framework along with the additional feature-based loss term. This is overall a nice problem and a nice approach! In that light, I believe that there is a new focus in this work on the perceptual quality of the outputs, as compared to (Kuleshov et al 2017). I would therefore ideally like to see (a) some attempts at perceptually evaluating the resulting output (beyond PESQ, e.g. with human subjects and with the understanding that, e.g. not all AMT workers have the same aural discriminative abilities themselves), and/or (b) more detailed associated qualitative descriptions/visualization of the super-sampled signal, perhaps with a few more samples if that would help. That said, I understand that there are page/space limitations. (more on this next) Given the similarity of the U-net architectures to (Kuleshov et al 2017), why not move some of those descriptions to the appendix? For example, I found the description and figure illustrating the “superpixel layers” to be fairly uninformative: I see that the figure shows interleaving and de-interleaving, resulting in trading-off dimensionalities/ranks/etc, and we are told that this helps with well-known checkerboard artifacts, but I was confused about what the white elements represent, and the caption just reiterated that resolution was being increased and decreased. Overall, I didn’t really understand exactly the role that this plays in the system; I wondered if it either needed a lot more clarification (in an appendix?), or just less space spent on it, but keeping the pointers to the relevant references. It seems that the subpixel layer was already implemented in Kuleshov 2017, with some explanation, yet in the present work a large table (Table 1(b)) is presented showing that there is no difference in quality metrics, and the text also mentions that there is no significant perceptual difference in audio. If the subpixel layer were explained in detail, and with justification, then I would potentially be OK with the negative results, but in this case it’s not clear why spend this time on it here. It’s possible that there is something simple about it that I am not understanding. I’m open to being convinced. Otherwise, why not just write: “Following (Kuleshov et al 2017), we use subpixel layers (Shi et al) [instead of ...] to speed up training, although we found that they make no significant perceptual effects.” or something along those lines, and leave it at that? I did appreciate the descriptions of models’ sensitivity to size/structure of the conv filters, importance of the res connections, etc. My biggest confusion was with the evaluation & results: Since the most directly related work was (Kuleshov 2017), I compared the super resolution (U-net) samples on that website (https://kuleshov.github.io/audio-super-res/ ) to the samples provided for the present work ( https://sites.google.com/view/unsupervised-audiosr/home ) and I was a bit confused, because the quality of the U-net samples in (Kuleshov 2017) seemed to be perceptually significantly better than the quality of the Deep CNN (U-net) baseline in the present work. Perhaps I am in error about this, but as far as I can tell, the superresolution in (Kuleshov et al 2017) is significantly better than the Deep CNN examples here. Is this a result of careful selection of examples? I do believe what I hear, e.g. that the MU-GAN8 is clearly better on some examples than the U-net8. But then for non-identical samples, how come U-net4 actually generally sounds better than U-net8? That doesn’t make immediate sense either (assuming no overfitting etc). Is the benefit in moving from U-net4 to U-net8 within a GAN context but then stabilizing it with the feature-based loss? If so, then how does MU-GAN8 compare to U-net4? Would there be any info for the reader by doing an ablation removing the feature loss from the GAN framework? etc. I guess I would like to get a better understanding of what is actually going on, even if qualitative. Is there any qualitative or anecdotal observation about which “types” of samples one system works better on than another? For example, in the provided examples for the present paper, it seemed to be the case that perhaps the MU-GAN8 was more helpful for supersampling female voices, which might have more high-frequency components that seem to get lost when downsampling, but maybe I’m overgeneralizing from the few examples I heard. Some spectrograms might be helpful, since they do after all convey some useful information despite not telling much of the perceptual story. For example, are there visible but inaudible artifacts? Are such artifacts systematic? Were individual audio samples represented as a one-hot encoding, or as floats? (I assume floats since there was no mention of sampling from a distribution to select the value). A couple of typos: descriminator → discriminator pg 6 “Impact of superpixel layers” -- last sentence of 2nd par is actually not a sentence. “the reduction in convolutional kernels prior to the superpixel operation.” Overall, interesting work, and I enjoyed reading it. If some of my questions around evaluation could be addressed-- either in a revision, or in a rebuttal (e.g. if I completely misunderstood something)-- I would gladly consider revising my rating (which is currently somewhere between 6 and 7).