Deep Audio Priors Emerge From Harmonic Convolutional Networks
Authors: Zhoutong Zhang
Yunyun Wang
Chuang Gan
Jiajun Wu
Joshua B. Tenenbaum
Antonio Torralba
William T. Freeman
Venue:  (2020)
Keywords: Audio
Deep Prior

Abstract: Convolutional neural networks (CNNs) excel in image recognition and generation. Among many efforts to explain their effectiveness, experiments show that CNNs carry strong inductive biases that capture natural image priors. Do deep networks also have inductive biases for audio signals? In this paper, we empirically show that current network architectures for audio processing do not show strong evidence in capturing such priors. We propose Harmonic Convolution, an operation that helps deep networks distill priors in audio signals by explicitly utilizing the harmonic structure within. This is done by engineering the kernel to be supported by sets of harmonic series, instead of local neighborhoods for convolutional kernels. We show that networks using Harmonic Convolution can reliably model audio priors and achieve high performance in unsupervised audio restoration tasks. With Harmonic Convolution, they also achieve better generalization performance for sound source separation.

Published as a conference paper at ICLR 2020
DEEP AUDIO PRIORS EMERGE FROM HARMONIC CONVOLUTIONAL NETWORKS
Zhoutong Zhang' Yunyun Wang Chuang Gan
Jiajun Wu
Joshua B. Tenenbaum'
Antonio Torralba'
William T. Freeman 1.5
Massachusetts Institute of Technology IIS, Tsinghua University MITTIBM Watson Lab
*Stanford University
Google Research
ABSTRACT
Convolutional neural networks (CNNs) excel in image recognition and generation. Among many efforts to explain their effectiveness, experiments show that CNNs earry strong inductive biases that capture natural image priors. Do deep networks also have inductive biases for audio signals? In this paper, empirically show that current network architectures for audio processing do not show strong evi- operation dence in capturing such priors. We propose Harmonic Convolution, that helps deep networks model priors in audio signals by explicitly utilizing the harmonic structure. This is done by engineering the kernels be supported by sets of harmonic series, instead of by local neighborhoods as convolutional ker- nels. We show that networks using Harmonic Convolution can reliably model au- dio priors and achieve high performance on unsupervised audio restoration. With Harmonic Convolution, they also achieve better generalization performance for supervised musical source separation. Code and examples are available at our project page: http: / /ap.. csail .mit.c.. edu.
INTRODUCTION
Deep neural networks, in various forms and designs, have been proved extremely successful in both discriminative tasks such as image classification (He al.. 2016), machine translation (Sutskever et al., 2014), speech recognition (Hinton et al., 2012), and in generative tasks such as image and audio generation (Goodfellow et al., 2014; Oord et al., 2017). Recently, Lempitsky et al. (2018) showed that convolutional neural networks (CNNs) come with strong inductive biases suitable for capturing natural image priors. Specifically, a randomly initialized neural network is optimized by fitting a degenerate signal, such noisy image, with random Gaussian noise vector input. During the fitting process, the network first outputs a clean image, even though it has only seen its noisy version. Lempitsky al. (2018) argued that this phenomenon shows that CNNs capture image priors by its deep convolutional structure; this is an intriguing perspective on the effectiveness of CNNs in generative modeling of images.
Due to the success of CNNs, audio processing networks usually adopt similar designs from their image processing counterparts, using spatial or temporal convolutions as building blocks. Though such operations could capture image priors, are they capable of capturing audio priors? Is it possible that the task of audio signal modeling needs unique components that cannot be found in image processing networks?
As an attempt to answer the questions above, we investigate whether various audio processing net- works can capture audio priors, similar to what CNNs do for images (Lempitsky et al., 2018). Re- cently, Michelashvili & Wolf (2019) reported that deep priors do exist for Wave-U-Net (Stoller al., 2018), which can be exploited to perform unsupervised audio denoising. However, the definition of deep priors slightly different between works by Michelashvili & Wolf (2019) and Lempitsky et al. (2018). Michelashvili & Wolf (2019) reported that the noisy signal causes more violent fluctuations on the spectrogram, which is then utilized as a prior for estimating the noise signal. In contrast, focus on the deep priors defined by Lempitsky al. (2018), and investigate whether there is evidence that current deep networks for audio related tasks carry inductive biases for audio priors; if not, what might be alternative?

---

Published as a conference paper at ICLR 2020
In this work, we first empirically show that current architectures for audio signal modeling do not show strong evidence in capturing audio priors. Specifically, we look at two general types of de- sign: temporal CNNs (Michelashvili & Wolf, 2019; Stoller et al., 2018; Aytar al., 2016) and spectrogram- based CNNs (Shen et al., 2018; 8: Zhao et al., 2018). To test their prior modeling ability, al. (2018): the networks are initialized randomly and
the setup identical to Lempitsky tasked to fit a single degenerated audio signal using random Gaussian noises as input. If the network capable of modeling the signal priors by its structure, it would fit the signal faster than noise. We show an illustrative example in Figure 1, where no substantial evidence is found for temporal or ppectral-teppore CNNs, even with this simplest case.
What might be missing? As shown in psychoacoustics experiments, the structure of the harmonic series closely related to human perception (Moore et al., 1986; Popham et al., 2018). We therefore propose the Harmonic Convolution, operation that explicitly utilizes harmonic structures in audio signals. Then, with multiple experiments, we show that Harmonic Convolution does enable neural networks to model audio signal priors better.
Finally, we show that Harmonic Convolution is useful downstream applications, and prove its performance by comparing against various baselines. The most natural application is unsupervised audio restoration, where we aim to recover clean signal from its corrupted version, either by high power Gaussian noise aggressive quantizations. In addition, also demonstrate that networks with Harmonic Convolution achieve better generalization performances for supervised sources separation tasks.
In summary, our contributions are threefold. First, we show that current audio processing architec- tures do not model audio signal priors naturally. Second, propose an operation called Harmonic Convolution, serving as an effective inductive bias for neural networks to model audio signal priors. Finally, we demonstrate that networks with Harmonic Convolution achieve state-of-the-arr perfor- mances in unsupervised audio restoration tasks and improve the generalization ability on supervised musical source separation tasks.
2 MOTIVATION
In this section, we first give a brief review of deep image priors. We also provide short survey on current popular network architecture designs for processing audio signals. Then, we show a motivating toy example, where current architectures fail model the signal priors, even when the signal is stationary. Finally, we provide a heuristic analysis of why convolution- based approaches may not capture the audio priors, using local signal statistics.
2.1
DEEP PRIORS
Lempitsky et al. (2018) first proposed the notion of deep priors on images. Specifically, they show that given a corrupted image To, a deep neural network fe. parameterized by 0, can serve as a natural regularization without any pretaining. Formally, the deep prior method optimizes
min E (fe(2); xo)
where E(::-) is task-specific data term, is the corrupted image, and is a fixed random noise vector. Lempitsky et al. (2018) showed that optimizing in the parameter space of the neural network So is surprisingly effective: after several iterations, the optimized network parameters 0* gives the restored image x by forwarding the noise vector, i.c. x = fe- (2). Note that the network is randomly initialized and only trained to fit the corrupted signal zo using random vector. The fact that it fits the clean signal first suggests that CNNs might be well suited for modeling images, where its structure and operation provide strong inductive biases.
2.2 CURRENT NETWORK DESIGNS FOR AUDIO PROCESSING
The network architectures for audio signal processing fall into two broad categories. The first one to directly apply ID convolutions on the raw audio signals (Michelashvili & Wolf, 2019; Stoller et al., 2018; Aytar et al., 2016). For instance, Wave-U-Net (Stoller et al., 2018) is ID adaptation of the U-Net architecture, which utilizes skip connections and multiscale processings to model signal properties different time scales. The other category is characterized by performing 2D convolu- tions on spectrograms (Shen et al., 2018; Zhao et al., 2018). A common practice is first to extract a spectral-temporal representation from the audio signal, then apply 2D convolutions.
2

---

Published as a conference paper at ICLR 2020
Firting: steps
Fitring: steps
Pitting: steps
Firrin:: steps
Finiing: steps
Fining: 1000 steps
Input Signal
(b) Convolution
(c) Dilated Convolution
Fitting: 50 steps
Fittin:: 200 steps
Fiting: 1000 steps
Fitting: 50 sleps
Fitting: 200 stops
Fitting: 1000 steps
(d) Clean Signal
(e) Wave-U-Net
(1) Harmonic Convolution
Figure A simple illustrative case: fitting a harmonic sound corrupted by Gaussian noises. The detailed setup of all networks can be found in the appendix. (a) The input to networks is harmonic sound composed by sinusoids of 1,000, 2,000, and 3,000Hz, plus random Gaussian noises. (d) The clean, target signals. They are shown here references, not accessible any models. (b)(c)(e) Both temporal and spectral-temporal convolution networks start with very noisy output at 50 iterations; they then fit the signal as well the background noises at 200 iterations. At 1,000 iterations, the output is similar to the one at 200 iterations. This suggest that these networks fit noises and the target signal simultaneously. (f) Harmonic Convolution starts with noisy output well, but 200 steps, the network selectively fits the harmonic series instead. Note that the 1,000-step result even cleaner than the 200-step one.
Image
Spectrogram
Raw Waveform
(a) clean
(b) noisy
(c) clean
(d) noisy
(e) clean
(I) noisy
Figure 2: The frequency statistics for images, spectrograms, and raw waveforms. All magnitudes in this figure are in log scale. The DC component is not shown in this visualization. (a) Clean image patch statistics. The 1/ f2 law can be observed as there an approximately linear energy falloff from low frequencies to high frequencies in the log scale. (b) Noisy image patch statistics. (c) Clean speech spectrogram statistics. Note that the energy fall-off is different from (a). (d) Noisy speech spectrogram statistics. Note that the energy fall-off along the frequency dimension is different from the fall-off along the temporal dimension, a unique phenomenon to this representation. (e) Clean speech statistics. The power distribution spreads to higher frequencies. (f) Noisy speech statisties.
2.3 A MOTIVATING EXAMPLE
Inspired by the deep image prior experiments, we would like to see if the deep architectures above possess the proper inductive biases model audio signals. To this end, first test the above architectures to reconstruct a simple signal: a stationary signal composed of 1,000Hz, 2,000Hz, and 3,000Hz sinusoidal waves. The corrupted version of this signal is generated by adding stationary Gaussian noise with a standard deviation of 0.1. As can be seen from Figure 1, all the methods start with very noisy fittings at very early iterations (50 in the figure). Then, they start to fit the signal and the noise at similar speed, rendering a noisy signal at 200 steps. At 1,000 steps, the network would fit noisy signal, with slightly less noise than the input signal.
2.4 HEURISTIC ANALYSIS
Here we provide heuristic analysis on why plain convolution networks would fail for mod- eling audio signal priors by structure. We show that the natural statistics (Torralba & Oliva, 2003) of images and audio signals are different. This suggests that different designs might be needed to capture their priors. In Figure 2, we show, side by side, the frequency statistics of clean natural im- ages, clean speech spectrograms, clean speech signals, and their noisy versions by adding Gaussian noise. All modalities share the same signal-to-noiss noise ratio. The natural images share 1/f² energy distribution in the frequency domain. This can be seen in Figure 2(a), where the energy falloff is ap-
3

---

Published as a conference paper at ICLR 2020
Regular Convolution
Harmonic Convolution
Harmonic Convolution anchoring =2 Zoa
Harmonic Convolution anchoring
anchoring
3t 4o 5t 600
300 4ta So 660
Zoo 3to So 60
Y
X
X
Sea 6w
20 30
40x
2w 30 4e So Y[w] = Σ, X(kw//2
4G0 S0 60
Y[m] Es X(ta @)] Y[w] = Eg X[ko]
Y[w) = En X|koo /3]
Figure 3: Illustrations for regular convolution and Harmonic Convolution. (a) Regular convolution
kernels are supported by a local region. The shape the support is translational invariant. (b) Hanmonic Convolution with anchoring of 1. This interprets the output frequency the funda- mental frequency for a harmonic series. (c) With an anchoring of 2, the output frequency is the second lower frequency of harmonic series. (d) Similarly, with anchoring of the output location is seen as the third harmonics.
proximately linear in the log scale. However, this does not hold for spectrograms or raw waveforms. Details for generating the statistics in Figure 2 can be found in Appendix Section B.
APPROACH
As shown in Figure 1, the architectures above, despite their success in discriminative tasks, do not show strong evidence for encoding deep audio priors. In light of such facts, we aim to introduce new components for audio signal modeling. We start with harmonic structures, the most common patterns for audio signals, which are also shown to be closely related to human perception (Popham et al., 2018; McPherson & McDermott, 2018). Subsequently, we propose Harmonic Convolution, an operation that treats harmonic structures explicitly.
3.1 HARMONIC CONVOLUTION
Harmonic Convolution is designed to explicitly exploit harmonic structures as an inductive bias for auditory signal modeling. Specifically, Harmonic Convolution operates the discrete Short-Tine Fourier Transform (STFT) of given audio signal, utilizing the representation. For audio signal æft), its discrete STFT spectrogram X [w, r] is given by
structure this
X|w -]
x[[]w[[ - TJe-surr jui
where w(-) a locally supported windowing function and j denotes the imaginary unit. Regular 2D convolutions on the spectrogram X X(w, T] with kernel function K is defined as:
(X K)
Σ
Σ
x - w T]K[w, 7],
where the kernel K is supported [- [-T.T]. Note that regular convolution aggregates information 2Ω x 27 window on local regions of X. To utilize harmonic structures, we modify this information aggregation scheme to align with harmonies. Specifically, Harmonic Convolution is defined as an operation mapping X ((w,T) to Y(,, F), where
K
T
(a,,)) 7) = Σ
Σ
Xlkū, F T].
(3)
k=1T=-1
Note Harmonic Convolution interprets the frequency dimension of the kernel as weights for K har- monic series each target frequency location W, where regular convolutions interpret the kernel as weights for local neighborhood at target spectral temporal locations. Figure shows an illustration for both plain convolutions and Harmonic Convolution.
3.2 ANCHORS AND MIXING
As indicated by Equation 3, the output target frequency @ is weight of its K harmonic series, starting from @ to K&. Note that there also exist other possible harmonic series that include w. For example, (0.250, 0.5á, 0.75%. .....} is also valid harmonic series, but Equation never
4

---

Published as a conference paper at ICLR 2020
aggregates the information from frequencies lower than W. To over come this problem, add an extra parameter to Equation called anchor, which indicates the order of harmonics at target frequency location W. Specifically,, given the anchoring parameter n, we modify Equation 3 as:
K
kw
Y,(,, [| Σ
X
(4)
k=1==-1
An illustration for the effect of different anchoring parameters are shown in Figure 3(b)c)(d), In addition, we can make the output at the frequency location w depend on multiple anchoring pa- rameters. To this end, we mix different Y, using a weighted sum: V|ū; f] = En=1 Wn Yn(4, (2, f]. where N is the largest anchoring parameter, and w,, be convolution kernels K. Therefore, the final Harmonic Convolution is defined as:
learnable parameters, similar to
K
kB n
Y f[
Σ
Wr
K
(5)
where the learnable parameters are the convolution kernel K and the weights Wn.
Implementation details. We implement Equation 5 using the Deformable Convolution operation introduced by Dai (2017). For better efficiency, we factorize the 2D kernel K[k; r] as the product of two ID kernels, i.e. [[k; r] K [[]]]. Anchoring is implemented using grouped Deformable Convolution (Dai et al., 2017), and the weighted sum mixing is implemented as an extra 1x1 convolution. Note that the notation is defined spectrograms, all operations implemented by real-valued operations. In experiments, we treat complex STFT coefficients as two separate real-valued channels.
4 EXPERIMENTS
In experiments, we test Harmonic Convolution under the deep prior modeling setup introduced by Lempitsky et al. (2018), where the networks are asked fit corrupted signal. We define network's 's ability to model audio priors as the quality of the audio produced during the fitting process. Under this definition, we show that networks equipped with Harmonic Convolution can model audio priors better than various baselines. As by-product, Harmonic Convolution performs comparably well with several state-of-art methods for unsupervised audio restoration. Finally, we demonstrate that Harmonic Convolution improves the generalization performance for supervised sound separation.
4.1 EXPERIMENT SETUPS
We the LJ-Speech (Ito, 2017) dataset and the MUSIC (Zhao al., 2018) dataset. LJ-Speech is a speech dataset consisting of short audio clips of single speaker reading passages. MUSIC is video dataset of musical instrument solos crawled from Youtube. We only use their audio tracks for all the experiments.
For fair comparisons, we use the same U-Net (Ronneberger al., 2015) architecture with different operations, i..,, regular convolutions, dilated convolutions, and the Harmonic Convolution. The details of our network architecture can be found the appendix. We train all the networks using the Adam optimizer (Kingma & Ba, 2015) with learning rate of 0.001 for all the experiments.
4.2 EXPERIMENTS ON DEEP AUDIO PRIORS
Following Lempitsky et al. (2018), test networks' ability model audio priors by fitting corrupted audio signal using a fixed random input and randomly initialized weights. If the network can produce restored signal faster and with better quality, then we call this network having stronger ability to model audio priors.
Setup. The random input drawn from a standard Gaussian distribution, and the weights are ini- tialized by drawing from zero-mean Gaussian distribution with standard deviation of 0.02. We test networks that rely on 2D spectral-temporal operations, i.c. regular convolutions, dilated convo- lutions, and Harmonic Convolution, to fit the complex STFT coefficients of the corrupted signal. We also test networks that operate on the raw waveform, i.e., Wave-U-Net. Again, all operations done by the network on real numbers; we treat the real and imaginary part of the coefficients as two separate channels. The corrupted signal is generated by adding clean signal with a zero-mean Gaussian noise with standard deviation of 0.1. The complex coefficients are generated by taking
5

---

Published as a conference paper at ICLR 2020
Wave-U-Net
Convolution
Dilated
25
15.0
15.0
12.5
12.5
PSNR between firting result and clean signal
10.0
10.0
PSNR.00)
15-
PSNRLA
7.5
75-
40-
2.5
2.5-
PSNR between fitting result and comppte signal (urget of fitting)
0.2-
-25
-2.-
-5.0
-5.0
0
5000
10000
15000
20000
0
5000
10000
15000
20003
0
5000
10000 15030 20000
iteration
iteration
iteration
(a)
(b)
(c)
Harmonic convolution Multiple Anchors+ Mixing
Harmonic convolution
Harmonic convolution Single Anchor
Multiple Anchors, No Mixing
15.0
15.0
15.0
12.5
12.5
10.0
10.0
10.0
PSNWOIB
5.0
5.0
5.0
0.0
0.0
0.0
-5.0
-5.0
-5.0
13000
13000
13000
ieeration
iteration
iseration (f)
(d)
(e)
Figure 4: Experiments on deep audio priors using Harmonic Convolution, regular convolution, di- lated convolution, and Wave-U-Net. Two PSNR scores are calculated at each iteration: comparing the output against the input noisy signal (green), and against the ground truth clean signal (red). (a) Wave-U-Net fits the input noisy signal fast, but with strong evidence in producing the clean signal in the process, as the fitting result at each step has a low PSNR against the clean signal. (b)(c) Networks with convolution operations and dilated convolution operations fit the input signal fast, but with only moderate restoration capabilities as measured by PSNR against clean signals. (d) Under the same setup, Harmonic Convolution produces samples of significantly higher quality, compared with (a), (b), and (c). (f) Ablation studies for anchoring and mixing. Without anchoring, the fitting speed measured by restoration quality is slower. With anchoring but without mixing, the final fitting quality is lower.
the Short- Fome-Fourier-- Transform (STFT) of the corrupted signal, with a box filter of length 1,022. We use hop length of 64 to provide enough overlapping, reducing the noise introduced by taking the inverse STFT transformation of the fitted result. When comparing against Wave-U-Net, we use the publicly available implementations provided by the authors of Michelashvili & Wolf (2019).
Results. We compare the fitting progress of the networks with Harmonic Convolution against various baselines, where the quality of the output signal is measured by Peak Signal-to-Noiee Ratio the temporal domain at each step. As demonstrated in Figure 4, Harmonic Convolution produces cleaner signal than other methods (-3.5dB boost compared with networks using convolution and dilated convolution). The example of the fitting is randomly sampled from the IJ-Speech dataset. More examples can be found at https: // //appaapp.///p/ csail mit edu.
In addition, we also conduct an ablation study using this fitting process, showing that the design of anchoring and mixing helps with modeling audio priors. As shown in Figure 4 (d) and (f), without the anchoring operation, the fitting speed is much slower. Using anchors without the mixing operation would lead to sub-optimal fitting results.
4.3 AUDIO RESTORATION
Similar to the experiments in Lempitsky et al. (2018), networks that model signal priors can be used to perform unsupervised restoration tasks. Here we examine the performance of the network with the Harmonic Convolution on restoring speech and music audios cornupted by random Gaussian noises and aggressive quantizations.
Setup. We conducted experiments both LJ-Speech and MUSIC datasets. For each dataset, we randomly sample 200 audio clips and clip them to seconds long for restoration tasks. For restoring from Gaussian noise, set the noise be zero-mean, with standard deviation of 0.1.

---

Published as a conference paper at ICLR 2020
Speech
Music
Methods
CSIG CBAK COVL PESQ SSNR SSNR
Wiener
1.00
1.37
1.00
1.08
0.01
0.68
Speech
Wavelet
1.00
1.92
1.01
1.16
2.06
6.50
Methods
CSIG CBAK COVL PESQ SSNR Human
DFL
1.00
1.51
1.00
1.03
-0.42
DNP+LSA
1.00
1.42
1.00
1.02
-3.73
4.26
Conv.
1.00
1.03
1.00
1.09
2.78
0.095
DNP+Wiener
1.00
1.41
1.00
1.02
-3.33
4.74
Dilated
1.00
1.02
1.00
1.10
2.38
0.115
Wave-U-Net
1.00
1.36
1.00
1.02
-4.62
3.99
Harmonic 1.00
1.01
1.00
1.09
2.05
0.79
Regular
1.17
2.15
1.10
1.09
4.13
5.07
Dilated
1.29
2.22
1.17
1.13
4.85
5.38
Table 2: Quantitative results on the quantization
Harmonic
1.76
2.36
1.43
1.20
7.12
9.85
audio restoration task.
Table 1: Quantitative results on the speech and music restoration tasks.
For recovering from quantization noise, and quantize them into 16 bins, which uniformly covers the range from -1 to 1. The input and output of the network share the same details described in Section 4.2.
Is clips of the 200 randomly sampled speech signals
Baselines. We compare with the following baselines:
Wiener: Wiener filtering (Scalart & Filho, 1996) is optimization- based methods utilizing Signal-to-Noise Ratio (SNR) prior. We adopt the implementation where the SNR is estimated from the first 1,024 samples of the signal.
Wavelet: We use the MATLAB implementation for wavelet denoising. Wavelet denoising is based on the sparse prior of audio signals, which assumes the wavelet coefficients should be sparse for clean signals. We use the 8-tap symlet wavelets for this task.
DFL: Deep feature losses (Germain et al., 2018) is state-of-the-art supervised speech denois- ing approach using perceptual loss tailored for speech signals. When testing LJ-Speech with Gaussian noise, we are testing its generalization performances under unseen settings.
Deep network priors (Michelashvili & Wolf, 2019) is unsupervised method for audio denoising using deep priors. Contrary to our method, the authors observed that during the fitting process, the injected noise varies more violently than the signal itself. Therefore, this property can be used to identify noise regions and provide an SNR estimate for traditional filtering methods such as LSA (Ephraim & Malah, 1985) or Wiener filtering (Scalart & Filho, 1996).
DNP:
Metrics. For the speech restoration task, adopt multiple quality metrics measure the audio restoration results, including the mean opinion score (MOS) predictor of signal distortion (CSIG), intrusiveness (CBAK), the MOS predictor of overall signal
the MOS predictor of background- quality (COVL). the perceptual evaluation of speech quality (PESQ), and the segmental Signal-to- Noise Ratio (SSNR). For the music restoration task, we only report results measured by the SSNR, since other metrics do not apply to the music signals. CSIG, CBAK, COVL, and PESQ are scale of I to 5, where is the best quality.
Results. Table summarizes our results against the mentioned baselines over these metrics. Our method consistently outperforms all the baselines according to all measures considerable mar- gin. This directly demonstrates that Harmonic Convolution can make neural networks more suited for modeling audio signals.
We also conduct an experiment to verify that Harmonic Convolution is not limited to the additive Gaussian noise case. In this experiment, we quantize 200 randomly sampled one-second speech signals into 16 bins, uniformly covering the range of [-1, 1]. The results are reported in Table 2. Since the scores for each metric are rather close, we conduct a carefully designed perceptual ex- periment with human listeners. For each model, we take 200 audio restorations from the same set for evaluation and each audio is evaluated by three independent Amazon Mechanical Turk (AMT) workers. We present the model results in random order and ask the annotator to select the audio clip that has the best restoration results. The results show nearly 80% of the subjects vote for the results produced by the network using Harmonic Convolution.

---

Published as a conference paper at ICLR 2020
Guitar
Xylophone
Congas
Methods
SAR
SIR
SDR
SAR
SIR
SDR
SAR
SIR
SDR
Conv.
13.3
6.1
4.6
14,0
10.3
7.8
12.7
6.6
4.9
Dilated
14.6
7.0
5.7
13.8
12.6
9.2
12.8
7.8
6.0
Harmonic
15.1
7.9
6.7
13.7
14.0
9,9
13.2
7.6
6.1
Table 3: Quantitative results on the generalized sound separation task. The units are in dB.
4.4
GENERALIZED SOUND SEPARATION
Here we examine whether the Harmonic Convolution can improve generalizations of the supervised sound separation task, compared with regular and dilated convolutions.
Setup. To evaluate the generalization ability of sound separation networks, we select five musical instruments from the MUSIC dataset (Zhao et al., 2018): violin, cello, congas, erhu, and xylophone. Each category consists of 50 6-seconds solo audio clips. We aim to test the model's ability to generalize to unseen musical instrument mixtures. Specifically, the model is tasked with separating the sound of a target instrument from clip that also contains sounds of another instrument. During training. we avoid using clips that contain sounds of a selected holdout instrument class, so that the model has never 'heard' of the sound of that type of instrument before. We then test the separation performance of this model on mixtures made from the sounds of the model's 's target instrument and the holdout instrument. In particular, we train models separate the sound of violins and use three different holdout instruments to test its generalizability (congas, violin, and xylophone). The input the network is the complex STFT coefficients of the mixed audio, using box filter of size 1022 and a hop length of 64. The output of the network is a ratio mask, calculated as the ratio of the spectrogram between the sound of the target instrument and the input mixed sound. To produce separated audios, we apply an inverse STFT transformation to the input complex STFT coefficients multiplied by the predicted ratio mask.
Implementations. We adopt the Mix-and- Sixpaad-Separate framework (Zhao et al., 2018) for this task. We first generate a synthetic separation training set by mixing the audio signals from two different audio clips and then trains neural network to separate the sound of the target instrument, e.g., violin.
During training, we take 1- 1-second mixed audio clip as input and transform it into complex coeffi- cients using the Short-Time-Fi Sanee-Foorrrrr--nsttiee whose architecture is described in Section 4.1. The U-Net outputs a ratio mask, and the network is trained by minimizing the LI loss between the predicted ratio masks and the ground- truth masks. We use 90: 10 train-val split, and test the performance on the mixture between sounds of the target instrument and the sounds of the holdout instrument.
(STFT). The spectrogram then fed into a U-Net,
Results. We compare the performance of the proposed Harmonic Convolution against the regular and dilated convolutions used in previous papers. We use the Signal-to-Distortion Ratio (SDR), Signal-to- -nttrffe-neeeeeeeecee Ratio (SIR), and Sigrfl-to-Artffatt Ratio (SAR). metrics from the open- source mir l library (Raffel et al., 2014) to quantify performances, Quantitative results are reported in Table 3. We observe that while all networks suffer when tested mixtures under novel recording conditions, the Harmonic Convolution exhibits better generalization performances. This suggests that Harmonic Convolution not only can be used as a prior for unsupervised tasks but also has the potential to be helpful for supervised tasks.
RELATED WORK
Deep priors. Our work is inspired by the recent paper on deep image priors (Lempitsky et al., 2018), which shows that the structure of CNNs imposes a strong prior to restore a single original image from the degraded image. The idea of deep priors has also shown to be useful in many appli- cations, including semantic photo manipulation (Bau et al., 2019), image super-resolution (Shocher et al., 2018), and image decomposition (Gandelsman et al., 2018). While most prior papers focused on images, little work has explored deep priors on audio signals.
Deep learning for auditory signal modeling. Deep networks have gained remarkable success on the audio signal modeling, such as speech recognition (Hinton et al., 2012; Amodei et al., 2016), sound separation (Stoller et al.. 2018; Zhao et al., 2018; 2019), audio denoising (Rethage et al., 2018; Germain et al., 2018), audio generation (Oord et al., 2017; Mchri et al., 2017), text to speech synthesis (Wang al., 2017; Shen et al., 201 8), and voice conversion (Hsu et al., 2017). A detailed
8

---

Published as a conference paper at ICLR 2020
survey can be found at (Purwins et al., 2019; Qian al., 2019). However, it remains unclear if these architectures themselves capture the audio signal priors. Most related to our work is Michelashvili & Wolf (2019), where they used deep networks trogram and then used classical post-processing algorithms to perform the speech denoising. Note that the deep prior mentioned in works of Mehri al. (2017) is different from the ones mentioned in Lempitsky et al. (2018). The former uses fitting time variances as noise indicating priors. We aim to find designs that bias towards clean audio signal.
prior to estimate the SNR prior on the spec-
Structured operators for deep models. Various algorithms have been proposed to include more dynamic structures beyond translational invariant kernels. Dai et al. (2017) proposed a convolution operator with dynamic offsets at each spatial location, which improves the object detection accu- racy by distinguishing foreground and background features. Shelhamer et al. (2019) introduced a spatially adaptive Gaussian kernel to help scale and steer convolution kernels, without introducing a significant amount of free parameters. Ravanelli & Bengio (2018); Sainath et al. (2013) proposed to learn parametrized filter banks that adapt to the convolution structure. While most of these al- gorithms focus on the discriminative tasks, our research topic is more related to generative tasks. Moreover, the learned filter banks in Ravanelli & Bengio (2018); Sainath et al. (2013) are used only in the first layer and not invertible in general, making them less suitable for the generative tasks where the output is an audio signal.
Psychoacoustics. Harmonic structures are closely related to the human perception of audio sig- nals. The famous missing fundamental auditory illusion suggests that human can infer the missing fundamental frequency by only hearing its overtones (Todd & Loy, 1991). Moore et al. (1986) showed that shifts in harmonic components would be perceived as separate tones. More recently, Popham et al. (2018) showed that the harmonic structure plays an important role for human to solve the cock-tail problem, where inharmonicity would cause difficulties for human to track speakers for the cocktail party problem. McPherson & McDermott (2018) showed that pitch perception is closely related to the harmonicity of the sound.
CONCLUSION
In this paper, examined various architectures deep audio prior modeling. We then proposed a novel operation called Harmonic Convolution, which can help networks better capture priors in audio signals. We showed that fitting randomly-initaaized network equipped with Harmonic Con- volution can achieve high performance for unsupervised audio restoration tasks. We also showed that Harmonic Convolution improves the generalization ability in sound separation.
Acknowledgment. We would like to thank Josh H. McDermott for helpful discussions. This work supported by the NSF award # 447476 and IBM Research.
REFERENCES
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng. Guoliang Chen, et al. Deep speech 2: End- to-end speech recognition in english and mandarin. In ICML, 2016. 8
Yusuf Aytar, Carl Vondrick, and Antonio Torralba. Soundnet: Learning sound representations from unlabeled video. In NeurIPS, 2016. 2
David Bau, Hendrik Strobelt, William Peebles, Jonas Wulff, Bolei Zhou, Jun-Yan Zhu, and Antonio Torralba. Semantic photo manipulation with a generative image prior. ACM Transactions Graphics (TOG), 38(4):59, 2019.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In ICCV, 2017. 5,
Yariv Ephraim and David Malah. Speech enhancement using minimum mean-square error log- spectral amplitude estimator. IEEE Transactions on Acoustics, Speech, and Signal Processing, 33 (2):443-445 1985.
Yossi Gandelsman, Assaf Shocher, and Michal Irani. "double-dip": Unsupervised image decompo- sition via coupled deep-image-priors. In CVPR, 2018. 8

---

Published as a conference paper at ICLR 2020
Francois G. Germain, Qifeng Chen, and Vladlen Koltun. Speech Denoising with Deep Feature Losses. In ICASSP, 2018. 7.8
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NeurlPS, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, An- drew Senior, Vincent Vanhoucke, Patrick Nguyen, Brian Kingsbury, al. Deep neural networks for acoustic modeling in speech recognition. IEEE Signal Processing Magazine, 29, 2012. 1, 8
Chin-Cheng Hsu, Hsin-Te Hwang, Yi-Chiao Wu, Yu Tsao, and Hsin-Min Wang. Voice conver- sion from unaligned corpora using variational autoencoding wasserstein generative adversarial networks. arXiv preprint 1rXb::174.00044/ 2017. 8
Keith Ito. The LJ Speech Dataset.
com http://hhhhttcaccmpe-eeeeeaaaaaa 2017.
5, 12
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurlPS, 2012. 12
Victor Lempitsky, Andrea Vedaldi, and Dmitry Ulyanov. Deep image prior. In CVPR, 2018. 1, 2, 5,
6, 8, 9
Malinda McPherson and Josh H McDermott, Diversity in pitch perception revealed by task depen- dence. Nature Human Behaviour, 2(1):52, 2018. 4, 9
Soroush Mchri, Kundan Kumar, Ishaan Guirajani. Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. Samplernn: An unconditional end-to-end neural audio generation model. In ICLR, 2017. 8,
Michael Michelashvili and Lior Wolf. Audio denoising with deep network priors. In ICASSP, 2019.
1, 2, 6, 7,9
Brian CJ Moore, Brian R Glasberg, and Robert W Peters. Thresholds for hearing mistuned partials as separate tones in harmonic complexes. The Journal of the Acoustical Society of America, 80 (2):479-483, 1986. 2, 9
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In ICLR, 2017. 1,
Sara Popham, Dana Boebinger, Dan PW Ellis, Hideki Kawahara, and Josh H McDermott. Inhar- monic speech reveals the role of harmonicity in the cocktail party problem. Nature Communica- tions, (1):2122, 2018. 2, 4,9
Hendrik Purwins, Bo Li, Tuomas Virtanen, Jan Schlüter, Shuo- Yiin Chang, and Tara Sainath. Deep learning for audio signal processing. IEEE Journal of Selected Topics Signal Processing, 13 (2):206-219, 2019.
Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang. and Mark Hasegawa-Johnson. Zero-shot voice style transfer with only autoencoder loss. In ICML, 2019.
Colin Raffel, Brian McFee, Eric Humphrey, Justin Salamon, Oriol Nieto, Dawen Liang, Daniel PW Ellis, and C Colin Raffel. mir.eval: A transparent implementation of common mir metrics. In ISMIR, 2014. 8
Mirco Ravanelli and Yoshua Bengio. Speaker recognition from raw waveform with sincnet. In SLT, 2018. 9
Dario Rethage, Jordi Pons, and Xavier Serra. A wavenet for speech denoising. In ICASSP, 2018.
10

---

Published as a conference paper at ICLR 2020
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In MICCAI, 2015. 5
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3)2211-252 2015. 12
Tara N Sainath, Brian Kingsbury, Abdel-rahman Mohamed, and Bhuvana Ramabhadran. Learning filter banks within deep neural network framework. ASRU, 2013. 9
Pascal Scalart and Jozue Vieira Filho. Speech enhancement based on a priori signal to noise estima- tion. In ICASSP, 1996.
Evan Shelhamer, Dequan Wang, and Trevor Darrell. Blurring the line between structure and learning to optimize and adapt receptive fields. arXiv preprint arXiv:1904.1 114877 2019. 9
Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et Natural its synthesis by con- ditioning wavenet on mel spectrogram predictions. In ICASSP, 2018. 2,8
Assaf Shocher, Nadav Cohen, and Michal Irani. Zero-Shot super-resolution using deep internal learning. CVPR, 2018.
Daniel Stoller, Sebastian Ewert, and Simon Dixon. Wave-u-net: A multi-scale neural network for end-to-end audio source separation. In ISMIR, 2018. 2.8
Ilya Sutskever, Oriol Vinyals, and Quoe Le. Sequence to sequence learning with neural networks. In NeurlPS, 2014.
Peter M Todd and D Gareth Loy. Music and Connectionism. MIT Press, 1991. 9
Antonio Torralba and Aude Oliva. Statistics of natural image categories. Network: Computation in Neural Systems, ((3):391-412, 2003. 3
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint 1607 2016. 12
Yuxuan Wang, RJ Skerry-Raan, Daisy Stanton, Yonghui Wu, Ron Weiss, Navdeep Jaitly, Zongheng Yang. Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end-to-end speech synthesis. In INTERSPEECH, 2017. 8
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba. The sound of pixels. In ECCV, 2018. 2,5,
Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba. The sound of motions. In ICCV, 2019. 8

---

Published as a conference paper at ICLR 2020
A NETWORK ARCHITECTURE
The U-Net used by all methods consists of 5 blocks, each block contains two operation layers, which can be instantiated by regular convolutions, dilated convolutions or the Harmonic Convolution. The feature map is downsampled by average pooling with stride of and window size of 2. Down- sampling performed after the first two operation layers. As in common designs of U-Net, the upsampling performed before the last two layers through bilinear upsampling with a scale of 2. Finally, we attach x convolution layer after the U-Net's last layer give the final output. Fea- ture map sizes for all the layers are [input-rinput, input-+35], [35-+35, 35-+70J, [70~70, 70-+70), [140-+140, 140-+35, [70-70] 70-+35,, then followed by 1 x regular convolution layer mapping the final 35 channels to the desired number of output channels.
In addition, we also keep the kernel sizes fixed for all different operations. For regular convolutions, we use 7x7 kernels for all the layers. For dilated convolutions, we use the same kernel size (7x7) with dilation of 3 for all the layers. For the Harmonic Convolution, we use frequency kernel K of length 7 and a temporal kernel K2 also of length 7. We use 7 anchors (N=7 in Equation 5) for all the Harmonic Convolution operations. We use instance normalization (Ulyanov et al., 2016) and ReLu activations (Krizhevsky et al., 2012) for all the experiments.
B DETAILS FOR NATURAL STATISTICS ANALYSIS
The image statistics computed on 1,000 images randomly sampled from ImageNet (Russakovsky et al., 2015). The audio spectrogram statistics are computed on 1,000 speech signals randomly sampled from the IJ-Speech dataset (Ito, 2017). We compute their spectrograms using the Short- Fon--oorrrr--Trnffrnnn Transform and calculate the spatial frequency distribution of the spectrograms as they are images. The ID frequency distribution of the audio signals is calculated on the same speech signals, where we interpret audio clips like ID image patches.
Time-Fourier-
C RELATING HARMONIC CONVOLUTION WITH DILATED CONVOLUTION
The simplest form of Harmonic Convolution is defined in Equation 3:
Y(,, f)
X & - r [k, r).
(6)
1= T=-T
Contrasting dilated convolution, one can write its operation as
f
T
Y(,
Σ
X|a
7] [kk r],
(7)
w=
R==-T
where n the dilation parameter.
12


Reviews:
[{"review_id": "rJeNYCa6KS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers the effectiveness of standard convolutional blocks for modelling learning tasks with audio signals. The effectiveness of a neural network architecture is assessed by evaluating its ability to map a random vector to a signal corrupted with an additive noise. Figure 1 illustrates this process with a network taking a single standard normal vector as input and having a single target output consisting of some signal corrupted with additive noise.\n\nThe paper is not well written and it is rather difficult to follow. It is also not well structured with a number of relevant concepts properly described only sections after they appear for the first time.\n\nThe first issue I had with the paper was the notion of audio prior. It was only after reading the whole paper that I have realized what this means. Having said this, it is unclear why the employed notion would work in general. I see why it could work when the distribution of the input vector and additive noise are correlated. This has not been clarified nor discussed and I believe it merits a couple of sentences.\n\nIn the introduction, the paper states \"... unlike CNNs for image modelling, the design of deep neural networks for auditory signals has not yer converged\". First, it is not clear what it means for the architecture to converge. If we assume that it refers to standard convolutions with a couple of widely accepted filter size and max pooling, the I would say that in speech recognition the structures that work are quite similar for mel-frequency coefficients or fbank features as inputs (which are again convolutional feature extraction layers).\nShortly after this, there is a question on justification of network designs. I disagree with a potential implication that this is well understood for image processing. For some insights relevant to speech, the work by Mallat (\"Group invariant scattering\", 2012) might be useful.\n\nFigure 1 and the paragraph just below its caption are not clear. It is not explained what is the input/output of the network and this is of great importance for the understanding of the illustration in Figure 1.\n\nThe introduction does not explicitly define the notion of audio prior and the whole paper is about this. In my opinion, it is wrong to assume that a reader has seen the paper by Lempitsky et al (2018).\n\nSection 2.1, the optimization objective as formulated implies that z and x_0 are completely independent. I do not see how any meaningful conclusion can be derived by fitting a map between independent input and output vectors. Some assumption is required for the proper notion of \"audio prior\" (if not, then a discussion arguing for the opposite).\n\nSection 3, opening paragraph concludes that standard CNNs are not the best blocks to model learning tasks with audio signals. For this implication, one needs the exact structure of CNN network and more details with regard to the experiment itself. In particular, there are deep CNNs (with mel-frequency coefficients as inputs) that work very well in speech recognition (e.g., on noisy datasets such as aurora4). This illustration does not say anything about the influence of the depth and number of convolutional blocks on a learning task. The language should be more moderate here and, in general, some additional work is required on the motivation of harmonic convolutions.\n\nIn my understanding, harmonic convolutions are a special case of deformable convolutions (Dai et al., 2017). In essence, standard convolution is applied over time and deformable over the frequency axis of a spectrogram. The main contribution seems to be in that the work provides a structure to the offsets in Dai et al. (Section 2.1, 2017). If I am correct, then this should be discussed in details and the harmonic convolution needs to be placed in the context of prior work. It might help by starting with a review of that work and then introducing the imposed structure on the offset vectors. I am having problems understanding the illustration in Figure 3.\n\nIn the experiments, the work is evaluated on signal de-noising (audio restoration) and sound separation. \n\nThe first task is carried out under the assumption that the signal has been corrupted with Gaussian noise and shows advantages of the approach over baselines which include standard convolutional networks. It would be interesting here to see how the depth of a convolutional network affects the performance. Also, as the approach is (in my understanding) a special case of deformable convolutions it would be insightful to show an experiment with that baseline. While additive noise is difficult on its own, many signals are corrupted by channel noise. It would be interesting to add an experiment with different types of channel noise and which network design is more likely to de-convolve the noise from the signal.\n\nThe second experiment deals with separation of sounds of different musical instruments and the results again show advantages of harmonic convolutions over the baselines.  \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HJeBEdKvtr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors introduce a new convolution-like operation, called a Harmonic Convolution, which operates on the STFT of an audio signal.  This Harmonic convolution are like a weighted combination of dilated convolutions with different dilation factors/anchors.  The authors show that for noisy audio signals, randomly initialized/untrained U-Nets with harmonic convolutions can yield cleaner recovered audio signals than U-Nets with plain convolutions or dilated convolutions.  The authors beat a variety of audio denoising tasks on a variety of metrics for speech and music signals.  The authors also show that harmonic convolutions in U-Nets are better than plain and dilated convolutions in U-Nets for a particular sound separation task.\n\nI recommend a weak accept for this paper because a new architecture for audio priors was presented, with reasonable empirical data supporting that this architectural choice an improvement over other more immediate alternatives.  It is important to extend the work on deep nets for imaging to other domains, such as audio. My recommendation is not stronger because of the following concerns. \n\nI think the paper could be strengthened by\n(a) a comparison to other methods (outside the current framework) for sound separation\n(b) a significant clarification of Figure 4.  The authors claim that this data shows that Harmonic Convolutions produce a \"cleaner signal faster\" than other methods.  When I look at Figure 4abcd, it appears that the Convolution and Dilated Convolutions fit a clean signal faster (it is just not as clean.  Additionally, the Wave-U-Net appears to reach the same accuracy as the Harmonic Convolution with many fewer iterations (while also continuing to get much higher PSNRs).  Perhaps I am misreading this plot, but it is not obvious to me that this plot supports the claims the authors are making.\n(c) The authors should present what they mean by a dilated convolution using the notation of the paper.  \n(d) In Figure 2, it is unclear to me how the 1/f^2 law is observed in (a) but not in (c) or (e).", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A"}, "extra_text": {}}, {"review_id": "BJl815FntH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the problem of how to design generative networks for auditory signals in order to capture natural signal priors. Compared to state-of-art methods in images [Lempitsky et al., 2018], this problem is not so easy on audio signals. Existing work [Michelashvili &Wolf] trains generative networks to model signal-to-noise ratio rather than the signal itself. This paper proposes a new convolutional operator called Harmonic Convolution to improve these generative networks to model both signals or signal-to-noise ratio. Applications on audio restoration and source separation are given. \n\nThe paper starts to show that an existing generative network Wave-U-Net does not capture audio signal priors. The explanation in Fig 2 on why this is the case seem to me not so clear. Are you trying to show that the Wave-U-Net does not work since there is no 1/f^2 law for clean audio signals? \n\nThe Harmonic Convolution is similar to deformable convolutions, but specifically designed to capture audio harmonics. It is further combined with the idea of anchors and mixing to capture fractional frequencies. The explanation of this section is slightly unclear. There is a little typo in Formula 1 for the STFT spectrogram, I would use the modulus |.| rather than || . ||. Is Harmonic Convolution applicable to complex STFT coefficients as well? It seems to be yes based on Section 4.2. If so it would be better to define the operator in a more general notation.\n\nNumerical experiments show that the Harmonic Convolution improves over existing regular and dilated convolutions in various settings. Section 4.2 aims to fit the complex STFT coefficients of corrupted signals. However, the setting is less clear to me for both the unsupervised speech/music restoration and supervised source separation problems. In Section 4.3 and 4.4, is the x_0 (defined in Section 2.1) complex-valued STFT coefficients or something else? It seems to me x_0 = ratio mask in Section 4.4. What is the L1 loss defined in Section 4.4? To obtain the final separated audio waveform, an inverse STFT is applied on what? These details can be written in supplementary material if more space is needed. After all, the numerical results seem to me encouraging.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new convolution-like operation, called a Harmonic Convolution (weighted combination of dilated convolutions with different dilation factors/anchors), which operates on the STFT of an audio signal. Experiments are carried on audio denoising tasks and sound separation and seems convincing, but could have been more convincing: (i) with different types of noises for the denoising task (ii) comparison with more methods for sound separation. Apart those two concerns, the authors seem to have addressed most of reviewers' complaints.

Decision: Accept (Poster)