Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper considers the effectiveness of standard convolutional blocks for modelling learning tasks with audio signals. The effectiveness of a neural network architecture is assessed by evaluating its ability to map a random vector to a signal corrupted with an additive noise. Figure 1 illustrates this process with a network taking a single standard normal vector as input and having a single target output consisting of some signal corrupted with additive noise.

The paper is not well written and it is rather difficult to follow. It is also not well structured with a number of relevant concepts properly described only sections after they appear for the first time.

The first issue I had with the paper was the notion of audio prior. It was only after reading the whole paper that I have realized what this means. Having said this, it is unclear why the employed notion would work in general. I see why it could work when the distribution of the input vector and additive noise are correlated. This has not been clarified nor discussed and I believe it merits a couple of sentences.

In the introduction, the paper states "... unlike CNNs for image modelling, the design of deep neural networks for auditory signals has not yer converged". First, it is not clear what it means for the architecture to converge. If we assume that it refers to standard convolutions with a couple of widely accepted filter size and max pooling, the I would say that in speech recognition the structures that work are quite similar for mel-frequency coefficients or fbank features as inputs (which are again convolutional feature extraction layers).
Shortly after this, there is a question on justification of network designs. I disagree with a potential implication that this is well understood for image processing. For some insights relevant to speech, the work by Mallat ("Group invariant scattering", 2012) might be useful.

Figure 1 and the paragraph just below its caption are not clear. It is not explained what is the input/output of the network and this is of great importance for the understanding of the illustration in Figure 1.

The introduction does not explicitly define the notion of audio prior and the whole paper is about this. In my opinion, it is wrong to assume that a reader has seen the paper by Lempitsky et al (2018).

Section 2.1, the optimization objective as formulated implies that z and x_0 are completely independent. I do not see how any meaningful conclusion can be derived by fitting a map between independent input and output vectors. Some assumption is required for the proper notion of "audio prior" (if not, then a discussion arguing for the opposite).

Section 3, opening paragraph concludes that standard CNNs are not the best blocks to model learning tasks with audio signals. For this implication, one needs the exact structure of CNN network and more details with regard to the experiment itself. In particular, there are deep CNNs (with mel-frequency coefficients as inputs) that work very well in speech recognition (e.g., on noisy datasets such as aurora4). This illustration does not say anything about the influence of the depth and number of convolutional blocks on a learning task. The language should be more moderate here and, in general, some additional work is required on the motivation of harmonic convolutions.

In my understanding, harmonic convolutions are a special case of deformable convolutions (Dai et al., 2017). In essence, standard convolution is applied over time and deformable over the frequency axis of a spectrogram. The main contribution seems to be in that the work provides a structure to the offsets in Dai et al. (Section 2.1, 2017). If I am correct, then this should be discussed in details and the harmonic convolution needs to be placed in the context of prior work. It might help by starting with a review of that work and then introducing the imposed structure on the offset vectors. I am having problems understanding the illustration in Figure 3.

In the experiments, the work is evaluated on signal de-noising (audio restoration) and sound separation. 

The first task is carried out under the assumption that the signal has been corrupted with Gaussian noise and shows advantages of the approach over baselines which include standard convolutional networks. It would be interesting here to see how the depth of a convolutional network affects the performance. Also, as the approach is (in my understanding) a special case of deformable convolutions it would be insightful to show an experiment with that baseline. While additive noise is difficult on its own, many signals are corrupted by channel noise. It would be interesting to add an experiment with different types of channel noise and which network design is more likely to de-convolve the noise from the signal.

The second experiment deals with separation of sounds of different musical instruments and the results again show advantages of harmonic convolutions over the baselines.  

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper studies the problem of how to design generative networks for auditory signals in order to capture natural signal priors. Compared to state-of-art methods in images [Lempitsky et al., 2018], this problem is not so easy on audio signals. Existing work [Michelashvili &Wolf] trains generative networks to model signal-to-noise ratio rather than the signal itself. This paper proposes a new convolutional operator called Harmonic Convolution to improve these generative networks to model both signals or signal-to-noise ratio. Applications on audio restoration and source separation are given. 

The paper starts to show that an existing generative network Wave-U-Net does not capture audio signal priors. The explanation in Fig 2 on why this is the case seem to me not so clear. Are you trying to show that the Wave-U-Net does not work since there is no 1/f^2 law for clean audio signals? 

The Harmonic Convolution is similar to deformable convolutions, but specifically designed to capture audio harmonics. It is further combined with the idea of anchors and mixing to capture fractional frequencies. The explanation of this section is slightly unclear. There is a little typo in Formula 1 for the STFT spectrogram, I would use the modulus |.| rather than || . ||. Is Harmonic Convolution applicable to complex STFT coefficients as well? It seems to be yes based on Section 4.2. If so it would be better to define the operator in a more general notation.

Numerical experiments show that the Harmonic Convolution improves over existing regular and dilated convolutions in various settings. Section 4.2 aims to fit the complex STFT coefficients of corrupted signals. However, the setting is less clear to me for both the unsupervised speech/music restoration and supervised source separation problems. In Section 4.3 and 4.4, is the x_0 (defined in Section 2.1) complex-valued STFT coefficients or something else? It seems to me x_0 = ratio mask in Section 4.4. What is the L1 loss defined in Section 4.4? To obtain the final separated audio waveform, an inverse STFT is applied on what? These details can be written in supplementary material if more space is needed. After all, the numerical results seem to me encouraging.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: In this paper, the authors introduce a new convolution-like operation, called a Harmonic Convolution, which operates on the STFT of an audio signal.  This Harmonic convolution are like a weighted combination of dilated convolutions with different dilation factors/anchors.  The authors show that for noisy audio signals, randomly initialized/untrained U-Nets with harmonic convolutions can yield cleaner recovered audio signals than U-Nets with plain convolutions or dilated convolutions.  The authors beat a variety of audio denoising tasks on a variety of metrics for speech and music signals.  The authors also show that harmonic convolutions in U-Nets are better than plain and dilated convolutions in U-Nets for a particular sound separation task.

I recommend a weak accept for this paper because a new architecture for audio priors was presented, with reasonable empirical data supporting that this architectural choice an improvement over other more immediate alternatives.  It is important to extend the work on deep nets for imaging to other domains, such as audio. My recommendation is not stronger because of the following concerns. 

I think the paper could be strengthened by
(a) a comparison to other methods (outside the current framework) for sound separation
(b) a significant clarification of Figure 4.  The authors claim that this data shows that Harmonic Convolutions produce a "cleaner signal faster" than other methods.  When I look at Figure 4abcd, it appears that the Convolution and Dilated Convolutions fit a clean signal faster (it is just not as clean.  Additionally, the Wave-U-Net appears to reach the same accuracy as the Harmonic Convolution with many fewer iterations (while also continuing to get much higher PSNRs).  Perhaps I am misreading this plot, but it is not obvious to me that this plot supports the claims the authors are making.
(c) The authors should present what they mean by a dilated convolution using the notation of the paper.  
(d) In Figure 2, it is unclear to me how the 1/f^2 law is observed in (a) but not in (c) or (e).
review_title: Official Blind Review #2

Meta-review: This paper introduces a new convolution-like operation, called a Harmonic Convolution (weighted combination of dilated convolutions with different dilation factors/anchors), which operates on the STFT of an audio signal. Experiments are carried on audio denoising tasks and sound separation and seems convincing, but could have been more convincing: (i) with different types of noises for the denoising task (ii) comparison with more methods for sound separation. Apart those two concerns, the authors seem to have addressed most of reviewers' complaints.

Decision: Accept (Poster)