Retrieving Signals in the Frequency Domain with Deep Complex Extractors
Authors: Chiheb Trabelsi
Olexa Bilaniuk
Ousmane Dia
Ying Zhang
Mirco Ravanelli
Jonathan Binas
Negar Rostamzadeh
Christopher  J Pal
Venue:  (2020)
Keywords: Deep Complex Networks
Signal Extraction

Abstract: Recent advances have made it possible to create deep complex-valued neural networks. Despite this progress, the potential power of fully complex intermediate computations and representations has not yet been explored for many challenging learning problems. Building on recent advances, we propose a novel mechanism for extracting signals in the frequency domain. As a case study, we perform audio source separation in the Fourier domain. Our extraction mechanism could be regarded as a local ensembling method that combines a complex-valued convolutional version of Feature-Wise Linear Modulation (FiLM) and a signal averaging operation. We also introduce a new explicit amplitude and phase-aware loss, which is scale and time invariant, taking into account the complex-valued components of the spectrogram. Using the Wall Street Journal Dataset, we compare our phase-aware loss to several others that operate both in the time and frequency domains and demonstrate the effectiveness of our proposed signal extraction method and proposed loss. When operating in the complex-valued frequency domain, our deep complex-valued network substantially outperforms its real-valued counterparts even with half the depth and a third of the parameters. Our proposed mechanism improves significantly deep complex-valued networks' performance and we demonstrate the usefulness of its regularizing effect.

Under review as a conference paper at ICLR 2020
RETRIEVING SIGNALS IN THE FREQUENCY DOMAIN WITH DEEP COMPLEX EXTRACTORS
Anonymous authors Paper under double-blind review
ABSTRACT
Recent advances have made it possible to create deep complex-valued neural net- works. Despite this progress, the potential power of fully complex intermediate computations and representations has not yet been explored for many challenging learning problems. Building on recent advances, we propose novel mechanism for extracting signals in the frequency domain. As a case study, we perform audio source separation in the Fourier domain. Our extraction mechanism could be regarded local ensembling method that combines a complex-valuee tional version of Feature- Wise Linear Modulation (FiLM) and a signal averaging operation. We also introduce new explicit amplitude and phase-aware loss, which is scale and time invariant, taking into account the complex-var complex-valued components of the spectrogram. Using the Wall Street Journal Dataset, aware loss to several others that operate both in the time and frequency domains and demonstrate the effectiveness of our proposed signal extraction method and proposed loss. When operating in the complex-valued frequency domain, our deep complex-vall -valued network substantially outperforms its real-valued counterparts even with half the depth and third of the parameters. Our proposed mecha- nism improves significantly deep complex-valude -aluud networks' performance and we demonstrate the usefulness of its regularizing effect.
convolu-
compare our phase-
INTRODUCTION
Complex-valued neural networks have been studied since long before the emergence of modern deep learning techniques (Georgiou Koutsougeras) 1992, Zemel et 1995] Kim Adali [0003]
Hirose 2003 Nitta 2004).. Nevertheless, deep complex- complex-valued models have only started to gain
momentum (Reichert & Serre 2014 Arjovsky et al. 2015 Danihelka et al. 2016 Trabelsi et al. & Yao 2018b Choi et al. 2019), with the great majority of models in deep learning still relying on vellev representations. The motivation for using complex-valus representations for deep learning is twofold: On the one hand, biological nervous systems actively make use of synchronization effects to gate signals between neurons a mechanism that can be recreated in artificial systems by taking into account phase differences (Reichert & Serre 2014). On the other hand, complex-valuee representations are better suited to certain types of data, particularly those that are naturally expressed in the frequency domain.
2017 Jose et al.
Other benefits provided by working with complex-valued inputs in the spectral or frequency domain computational. In particular, short-time Fourier transforms (STFTs) can be used to considerably reduce the temporal dimension of the representation for an underlying signal. This is critical advantage, as training recurrent neural networks (RNNs) or convolutional neural networks (CNNs) on long sequences remains challenging due to unstable gradients and the computational requirements of backpropagation through time (BPTT) (Hochreiter) 199T Bengio et al.) 1994). Applying the STFT on the raw signal, the other hand, is computationally efficient, in practice it is implemented with the fast Fourier transform (FFT) whose computational complexity is On (og(n)).
The aforementioned biological, representational and computational considerations provide compelling motivations for designing learning models for tasks where the complex-valuee representation of the input and output data is more desirable than their real-counterpart. Recent work has provided building blocks for deep complex-valued valued neural networks (Trabelsi et al. (017). These building blocks have been shown, in many cases, to avoid numerical problems during training and, thereby, enable the

---

Under review as a conference paper at ICLR 2020
use of complex-valueds representations. These representations are well-suited for frequency domain signals, as they have the ability to explicitly encode frequency magnitude and phase components. This motivates us to design new signal extraction mechanism operating in the frequency domain. In this work, our contributions are summarized as follows:
1. We present new signal separation mechanism implementing local ensembling procedure. More precisely, a complex-valued convolutional version of Feature-wise Linear Modulation (FiLM) (Perez et al. 2018) is used to create multiple separated candidates for each of the signals we aim to retrieve from a mixture of inputs. A signal averaging operation on the candidates is then performed in order to increase the robustness of the signal to noise and interference. Before the averaging procedure, form of dropout is implemented on the signal candidates order to reduce the amount of interference and noise correlation existing between the different candidates.
2. We propose and explore a new magnitude and phase-aware loss taking explicitly into account the magnitude and phase of signals. A key characteristic of our loss that it is scale- and time- invariant...
We test our proposed signal extraction mechanism in the audio source separation setting where we aim to retrieve distinct audio signals associated with each speaker in the input mix. Our experiments demonstrate the usefulness of our extraction method, and show its regularizing effect.
2 RELATED WORK
2.1 RELATED WORK ON LEARNING REPRESENTATIONS IN THE FOURIER DOMAIN
I Leveraging the Convolution Theorem to retrieve information has been done decades ago in the machine learning community using holographic reduced representations (HRRs) in the context of associative memories (Plate 1991 1995). HRRs enable one to store key-value data. Retrieval of value in the data associated with a given key can be performed by convolving the whole data with the key or by applying an inner product between these two. By applying fast Fourier transform (FFT) on the keys and the data, one can perform elementwise multiplication between the Fourier transforms and apply inverse FFT to convert the result the time domain. This would be equivalent to performing circular convolution between the key and the data in the time domain and has the advantage of being less expensive. Recently, Danihelka et al. (2016) have used associative memories to augment the capacity of LSTMs and to increase their robustness to noise and interference. For that, they applied independent permutations on the memory to create multiple copies of it. This enables one to obtain decorrelated noise in each of the permuted copies. A complex multiplication is then performed between the key and each of the copies. A signal averaging on the resulted multiplications eliminates the decorrelated noise in them and strengthens the signal-to-noise ratio (SNR) of the retrieved signal. Danihelka et al. 2016)), however, have not relied on FFTs in order to convert the temporal signals to the frequency domain. In fact, they assumed that complex- valued multiplication between the key and the data is itself enough to perform retrieval, and they have assumed that for each input representation the first half is real and the second one is imaginary.
During this decade, interest in Fourier domain representations has started to grow in the machine learning community. [Bruna et al.) (2013) introduced generalization of convolutions to graphs using the Graph Fourier Transform, which is defined as the multiplication of a graph signal by the eigenvector matrix of the graph Laplacian. However, the computation of the eigenvector matrix is expensive. Recently, methods that are computationally more efficient have been introduced in Defferrard al. (2016) and Kipf & Welling (2016) avoid an explicit use of the Graph Fourier basis, In the context of Convolutional Neural Networks (CNNs), Rippel et al. (2015) introduced spectral pooling, which allows one to perform pooling in the frequency domain. This allows one to maintain the output spatial dimensionality. and thus the technique can retain significantly more information than other pooling approaches. Rippel et al. (2015) have also observed that the parametrization of the convolution filters in the Fourier domain induces faster convergence during training. AArjossky et al. (2016) designed a recurrent neural network (RNN) where the transition hidden matrix is unitary. More precisely, the hidden transition matrix is constructed using the product of specific unitary transformations such as diagonal matrices, permutations, rotations, the Discrete Fourier Transform and its inverse. This allows one preserve the norm of the hidden state, and consequence,
2

---

Under review as a conference paper at ICLR 2020
mitigates the problem of vanishing and exploding gradients. Wolter & Yao (201 designed an RNN where the imput is converted to the frequency domain using a Short Time Fourier Transform (STFT). The output is converted back to the time domain by applying an inverse STFT. Zhang et al.) (2018) proposed a Fourier Recurrent Unit (FRU) where they showed that FRU has gradient lower and upper bounds independent of the temporal dimension. They have also demonstrated the great expressivity of the sparse Fourier basis from which the FRU draws its power. As we consider the task of speech separation as case study, we provide a related work section on both time domain and frequency domain speech separation methods in section 2.2
2.2 RELATED WORK ON TIME DOMAIN AND FREQUENCY DOMAIN SPEECH SEPARATION
Speech separation has been the subject of extensive study within the audio processing literature for a considerable amount of time. Recently, there has been growing interest in leveraging deep learning techniques (Du et al., 2014; Huang et al.) 2014; Hershey et al., 2015 Gao et al. 2018 Ephrat et al. 2018) to tackle the speech separation problem. Hershey et al. (2015) proposed a deep clustering approach to speech separation. The basic idea is to learn high-dimensional embeddings of the mixture signals, that is later exploited to separate the speech targets with standard clustering techniques. A recent attempt to extend deep clustering led to the deep attractor network proposed by learned, but the
Chen et al. (2 Similarly to deep clustering, high dimensional embeddings network also creates the so-called "attractors" to better cluster time time-frequecyy points dominated by different speakers. The aforementioned approaches estimate only the magnitude of the STFTs and reconstruct the time-domain signal with the Griffin-Lim algorithm (Griffin & Lim 1984) or other similar procedures (Sturmel & Daudet 2006). Other papers have recently proposed integrate the thase-informationn within a speech separation system without necessarily working in the complex- valued frequency domain. The work by Erdogan al. (2015), for instance, proposes train a deep neural network with phase sensitvee loss. Another noteworthy attempt has been described in Wang jt al. (201)), where the neural network still estimates the magnitude of the spectrum, but the time- domain speech signals are retrieved by directly integrating the Griffin-Lim reconstruction into the neural layers. Furthermore, methods reported in Wang et al. (0018) integrate the phase-information within a speech separation system by reconstructing the clean phase of cach source starting from the estimated magnitude of each source and the mixture phase. This is fundamentally different from our proposed framework, complex-ve valudd frequency domain, and process both spectrogram magnitude and phase information rather than working only on magnitude representation with heuristic reconstruction of phase. Another attempt to estimate the clean phase is reported in Le Roux et al. (2019) where the clean phase of each speaker estimated using discrete representation. This is also fundamentally different from our work as it considers discrete representation of the phase for source separation and, in our case, we consider continuous representation of the complex-domani signal.
provide an end-to-end solution to perform signal retrieval in the
Instead of explicitly integrating phase- chasseinformmtton, other recent work perform speech separation in the time domain directly, as described in Venkataramani & Smaragdis (2018). Likewise, the TasNet architecture proposed in Luo & Mesgarani (2017) and Conv' TasNet (Luo & Mesgarani (2018) perform speech separation using the mixed time signal as input. Operating directly on the time-domain signal using the ConyTas.Ne architecture, which implements temporal convolutional networks (TCN) (Bai jet al. (2018), has led to state-of-the-art results audio speech separation (Luo & Mesgarani 2018) Shi et | 2019). The studies by (Lee et al., 2017; Hu & Wang 2004) Huang et al.) (2144) are more related to our work as they address the speech separation problem by processing the complex-vaiie valued spectral input of the mixed speech. However, this was done without leveraging the recent advances in complex-valued deep learning.
CONNECTION TO SIGNAL PROCESSING: MOTIVATION FOR USING FILM AND SIGNAL AVERAGING
Our signal extraction method takes advantage of the convolution theorem which states that the Fourier transform of two circularly convolved signals is the elementwise product of their Fourier transforms. It also implements signal averaging procedure that allows reduce the energy of the noise existing in the estimates of a clean signal, and so, to increase their respective signal to noise ratio (SNR). We detail here the motivation for using FiLM (Perez et al. 2018) and how extraction method
3

---

Under review as a conference paper at ICLR 2020
increases the signal to noise ratio. Let's consider a clean signal corrupted by the environment impulse response r and an additive noise e. The corrupted signal can expressed as y = @ r + where @ denotes the circular convolution operator. By leveraging the convolution theorem and the linearity of the Fourier transform we get:
Fly) = F(s) o F(r) F(c),
where F denotes the Fourier transform and @ the complex element- multiplication. If want to retrieve the spectral information of the clean signal s, we can express it as:
F(()
F(s) = Fly) F(r)
F(r)
where Pie and T(r) are respectively scaling and shifting representations. These representations could easily be inferred using FiLM (Perez et al. 2018) it conditionally learns scaling I and shifting B representations. To be more rigorous, we can assume in the case of speech separation that, for each speaker, there exists an impulse response such that when it is convolved with the clean speech of the speaker, allows to reconstruct the mix. We would then have:
Flo
mix = 8; @ r; + 63 Vi E (1,.., Nb speakerss)
F(ei)
F(s;) = F(mix) ④ F(r,) F(r,)
F(s,) F(mix) O B,.
Now, let's assume that is stochastic process such that y x where is the noise component which mean E[e] = 0. x is the clean signal that we want to estimate such that x is constant for all observations and that an 7t observation of y is given by y, x + c,. The signal-to to-nise ratio (SNR), which a measure of the signal quality, is defined as the ratio of the power of a clean signal to the power of noise, i.e, SNR to discard the noise component as E = x. In that case &
|
Estimating the clean speech by approximating E allows
c.
(x + G)) = =x+ + E G-- NZ F|| >Σ ना. If E, are uncorrelated, B - -== ΣE | = N E = SNR = N x
B #1
x
The SNR would then be: SNR
E| + MM1 GP)
This shows that the signal averaging operation and the uncorrelated noises allow to increase the SNR by factor of N. If we want to approximate F(s:) by performing signal averaging, we would then have:
El
B[F(s:)] F F(mix)
ET E[,,]
B(F(s)) F (mix)
E
E[B
N Σ ri j=1
N Σ
=F(mix)
: N
+
Bij.
N
j=1
where F(mix) is constant. In equation (4) N is equal to the number of scaling and shifting representations generated to approximate respectively each of E[T, and E[B.].
4 AMPLITUDE AND PHASE-AWARE Loss
In Choi et al. (2019) a weighted version of the cosine similarity is proposed in order to maximize the signal-to-distortion ratio (SDR) proposed in Vincent al. (2006). Recall that cosine similarity loss is defined in the real-valued domain and it is given by the following equation:
- -, x, 0::
cOstime (y,
(5)
| |xyl
where denotes the element-wise real-valued multiplication operation. Both y and x are real-valued in the above equation is the target signal in the temporal domain and x is the estimated signal after performing an inverse STFT on the spectrogram. The phase is then taken implicitly into account as the real-valued target signal encodes inherently the phase of the spectrogram. As the task in Choi et al. (2019) is speech enhancement (which is different from ours as we are performing speech separation), the authors used weighted version of the co8sime loss to weight the part of the loss
4

---

Under review as a conference paper at ICLR 2020
corresponding the speech signal and also the complementary part corresponding to the noise signal. This weighting is performed according to their respective target energies. In our case we are interested in extracting the clean speech signals of all the involved speakers whether each speaker signal has either high or low energy in the mixture. This is why we are not interested in penalizing the retrieved speech of each speaker by its corresponding energy.
Here, we suggest the use of a loss function which explicitly takes into account both magnitude and phase. This accomplished by computing the inner product, between the reference signal and its estimate, in the complex plane. In fact computing the inner product in the frequency domain is equivalent to computing the cross correlation in the time domain followed by weighted average. The inner product in the frequncy domain is then invtrinnvaiaant (time-invariant). The complex inner product between signals is given by the following equation:
(xly) = Σ (R(z)))R()) + 3(x))S(()) + (R(x)) - ((xx))((1)))
(6)
If x and y are identical, which is equivalent of having |x||| = ||||| and Lx = Ly, then, (xly) (xly) ||yl12 + 0i. If x and y are parallel, then ||x||||| = 1 + Oi = 1. The inner product between the signals normalized by the product of their amplitudes, is then scale and time invariant. We chose a loss that maximizes the real part of that normalized inner product and minimizes the square of its imaginary part. Note that each of the real and imaginary parts of the normalized inner product lies between [-1, 1]. We refer the reader to section A. in the appendix for more information on how the complex inner product is both amplitude and phase aware, how the real part of equation (6) is responsible of the amplitude similarity between the reference and estimate signals and how the imaginary part of the same equation is responsible for the phase matching between them. We define the following similarity loss denoted by CSimLoss as:
( (xly) [x||| [lyl] + Ximag 322
(xly) |yll
CSimLoss(x,,)) = Areat R
|
where Areat and Xemag are penalty constants. Areat is penalizing amplitude mismatch and Xsmag is penalizing phase mismatch. We fixed Areat to in all our experiments. We tried different values of Asmag {10?, 103, 10*). we found that the only value of Ximag that allows the phase matching part of the train loss have same range of values than the amplitude matching part is Ximag 10 All the results are reported in Table 2 and Table 3 for CSimLoss correpond to Ximag 10
5 DETAILS OF THE U-NET ARCHITECTURE USED FOR SPEECH SEPARATION
We detail here the architecture we used to perform speech separation | For this, we rely the U-Net architecture proposed by RRonneberger et al. (2015) and the complex-valued building blocks proposed by Trabelsi et al. (2017). This is similar to the complex-valued U-Net architecture used in Dedmari cct al. (2018) who reported state-of- Mithhhhan art results in MRI reconstruction using complex-van -valud raw input. Our primary goal is to demonstrate that our proposed signal extraction mechanism can improve upon the performance of baseline models.
For our task, required the addition of residual connections inside the U-Net blocks and replaced complex batch normalization with complex layer normalization, as the model was otherwise unable to learn, yielding instabilities during training. The reasons why complex LayerNorm outperformed complex BatchNorm are discussed in the appendix in section A.2 We describe, in section 6 how our extraction mechanism implemented in the context of audio source separation.
5.1 COMPLEX RESIDUAL U-NET
Residual networks (He et al. 201 and identity connections (He et al. 2016b) have had a significant impact on image segmentation. These architectural elements have also been combined with U-Nets (Drozdzal et al. (2066) for image segmentation. In our case, we use simple basic complex residual blocks (Figure in appendix) inside each of the U-Net encoding and decoding paths (Figure [ Figure 2 (Left) and (Middle) illustrate the basic structure of our Residual U-Net upsampling and
'The source code located at http://gmmtioooooooeeegglletttta
5

---

Under review as a conference paper at ICLR 2020
downsampling blocks (U; and D2) used in Figure 1 while Figure (Right) illustrates the structure of the complex residual blocks used in Figure 2 (Left) and Figure (Middle).
Each U-Net block begins with downsampling block (in the encoding U-Net path) upsampling block (in the decoding U-Net path). It also contains a block that doubles the number of feature maps (in the encoding path), or halves them (in the decoding path). The upsampling. downsampling. doubling and halving blocks each applies successively a complex layer normalization, a CReLU and complex convolution to their inputs. All complex convolutions have kernel size of 3 3 except for the case of downsampling block, where the convolution layer has kernel size of 1 x 1 and a stride of In the case of upsampling, we use bilinear interpolation instead of transposed convolution because we found empirically that it yielded better results. Immediately before and immediately after the doubling halving blocks, we use k = or k residual blocks. We have opted for this residual U-Net block architecture because of memory constraints and because residual connections are believed to perform inference through iterative refinement of representations (Greff al. 2016) Jastrzebski et al. 2017).
Figure I: The architecture of our Deep Complex Extractor. It consists of pipeline containing a U-Net and a Complex Extractor Masking operator (see Algorithm [T. The Deep Extractor takes input the mixed speech signal which is fed to the U-Net. The downsampling blocks of the U- Net are denoted by D; where E {1, 2, 3, 4} and the upsampling blocks are denoted by U; where i E [1, 2, 3, 4). The output of the U-Net along with the input mix are then fed to the Complex Extractor Masking operator in order to estimate the clean speech for each of the speakers.
6 COMPLEX MASK GENERATION
Featurewise Linear Modulation (FiLM) (Pree et af. 2018) techniques have yielded impressive results in visual question answering (VQA). The FiLM approach applies an affine transformation to convolutional feature maps, given the embedding of the question. In our approach, we create multiple transformations of the complex input spectrogram using FiLM. The FiLM parameters are determined from the output of our U-Net (See Figure 1). We then generate a complex mask for the original input spectrogram as well as for each the tiin--rranfoomee spectrograms. This is accomplished by using a ResNet conditioned on the U-Net output, the spectrogram and its FiL. transformations. Each spectrogram is multiplied by its corresponding complex mask. This leads to multiple candidates for the separated speech of each speaker. The resulting outputs are averaged to produce the final estimated clean speech. This could be interpreted as a local ensembling procedure to estimate the clean speech of the different speakers. More precisely, given the output of the last upsampling block of the U-Net, we generate scaling matrices I and shift matrices By. j E [1, C| of the same size as the input mix spectrogram. These parameters operate on the input mix as described by the following equation:
imput_ transoormatoonnn H inputmix Bj.
where F and B, are functions of the output of the last upsampling block in the U-Net, and @ is the clementwise complex product. In our case, we used a simple complex convolution layer with kernel of size 3 x 3 to generate r and By. The original input mix and its C scaled and shifted

---

Under review as a conference paper at ICLR 2020
transformations together form C + representations of the input mix. Given these C + complex representations, we generate C. + corresponding complex masks, with which the representations then multiplied. These masks are generated by sequence of complex convolution layer which kernel size is 3 x followed by two residual blocks. Once we have performed the complex multiplication of the masks with their respective inputs, C + 1 separated speech candidates obtained for given speaker. This procedure is repeated for the maximum number of speakers that could exist an input mix. The main motivation for this process is increase the separation capability and reduce interference between the separated speakers. Each transformation can focus on specific pattern in the representation. Each mask corresponding to specific input transformation can be seen as a feature of the speaker embedding. Grouped together, the masks generated to retrieve the speech of given speaker could be interpreted as an embedding identifying the speaker. The proposed complex masking procedure is summarized in Algorithm]
Algorithm 1 Complex Extractor Masking
Input: U-Net output:
Input: Nb transformations (XFs):
Input:
Nb speakers:
Input:
Input Mix:
mix
Output: Speakers separated speeches: Si,...,
I: function C-FILMED MASKING(O, C, N, mix)
2: for i t to N do
3:
Tic Ba Bic <- FilMFunction()))
4: end for
XFS +
6: for
< to do for j <- 1 to Co do XF,, To mix Bg
7:
8:
append))))))
10: end for
11:
end for
12: XFS concatenate (XFI),, XFS
13: masks GenerateMasks((O, XFFS
candidates masks @ XFS cleanspeeches <
15:
16:
for i 1 N do
17:
cleanspeech, average sleampecches.appandcceeme end for
1):: 1:++1:(C+]) (C+))(())
18:
19:
20:
return cleanspeeches
21:
end function
7
SYNOPSIS OF THE EXPERIMENTS
We present in Table I the most important results obtained when conducting our experiments. The complete results and the extended empirical analysis can be found in the appendix in section A.5 The data pre- processsingg and training protocol can be found in the appendix, in section A.4
We explore several variants of our architecture and report the test SDR. They are parametrized by:
k. the number of residual blocks used inside the residual U-Net block (See Figure
START FMAPS, the number of feature maps in the first layer of the encoding path in the
U-Net. START FMAPS defines the depth of each of the successive layers in the model.
PARAMS, the number of parameters, in millions.
TRANSFORMS, the number of input mixture transformations.
DROPOUT, the mask dropout rate.
"The effective number of feature maps for complex feature map equal the number of reported feature maps This is due the fact that it has real and imaginary part.
'rropping out mask is equivalent to a dropout of input mixture transformations or clean speech candidates. Performing dropout the masks reduces the correlation of the different noise components existing in the different candidates of clean speech. Along with signal averaging, dropout regularizes the retrieval mechanism.
7

---

Under review as a conference paper at ICLR 2020
Table I: The most important results obtained for the task of speech separation conducted on mixtures of speakers using the standard setup with the Wall Street Journal corpus.
MODEL
START FMAPS
PARAMS
TRANSFORMS
DROPOUT
Loss
TEST SDR
REAL U-NET
8.45
12
REAL UI-NRT
14.76
12
7.92
COMPLEX U-NET
32
4.29
12
9.61
COMPLEX
7.4
12
COMPLEX U-NET
40
11.67
12
10.93
CUMPLEX U-NET
11.67
CSimLoss
10.91
COMPLEX UI-NET
40
11.61
10
122mme
10.86
COMPLEX
coffine
CUMPLEX U-NEST
44
13.97
L2nea
9.88
COMPLEX
13.97
CSimLoss
COMPLEX U-NET
44
14.09
15
0.1
10.91
COMPLEX U-NETT
14.03
CSimLoas
From the first four rows of the results contained in Table we will highlight that the complex-valued baseline models vastly outperform their real-valued counterparts. These baselines (both real and complex) are architecturally the same as the U-Net of Figure but do not include our extraction mechanism (the FiLM, GenerateMask and signal-averaging operations). The real and complex U-Nets' outputs are masks that are complex-multiplaas with the mix to infer the clean speech of the speakers. All complex models, whether they have approximately the same number of parameters (R:8.455 & C:7,4M), half 45M: C::.39M) or a third, with half the depth (I::14.66M; C:4.39M) outperformed by convincing margin their real counterparts. Thus, natively-comple input, inference and output give complex networks such an overwhelming advantage that almost no handicap of size or depth can mask it. We will therefore not consider real-valued models, transformations and losses any further.
second highlight from Table that, while our signal extraction mechanism inexpensive in terms of parameter count, the extraction mechanism substantially improves the quality of the retrieved signal. For instance, when 10 mixture transformations are use, the number of parameters is marginally increased by less than 1% (13.97M to 14.03M) while substantial jump in terms of SDR observed (from 9.87 to 11.34). This can be also observed in Figure 33 in the appendix. Dropping out the speech candidates with low probability has further regularization effect on the wider models that have more feature maps, as shown in appendix Figure 5
The third highlight is that spectral-domain losses, i.e CSimLoss and L2treq outperform their time- domain counterparts. Our proposed CSimLoss posts the best reported result, 1 1..44 SDR, and Tables 2 and 3 and Figure demonstrate that extraction mechanism is ideally paired with the CSimLoss objective.
Finally, and although this is of scope for this paper, compare ourselves to ConvTasNet (Luo & Mesgarani 2018), which operates on a time-domain input mixture (as mentioned in 2.2b. Cony'l Tas- Net claims state-of-the-ard results in speech separation, and has led to even further improvements (Shi et al. 2019). Its headline achievement of 15.6 SDR must, however, be understood in light of significant difference in their preparation of the dataset. Whereas follow the standard setup described in Hershey et al. (2015). with input mixtures generated using an SNR between 0 and dB, [uo Mesgarani (2018) use an SNR between -5 and dB. Keeping this in mind, we retrain pptimally-oontigured Conv TasNer but using the standard setup, and obtain 12.1 SDR, compared to our own model's 11.3 SDR.
8 CONCLUSION
In this work, we introduced new complex-valued extraction mechanism for signal retrieval in the Fourier domain. As case study, we considered audio source separation. We also proposed a new phase- loss taking, explicitly, into account the magnitude and phase of the reference and estimated signals. The amplitude and phase-aware loss improves over other frequency and time-domain losses. We believe that our proposed method could lead to new research directions where signal retrieval is needed.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. ICML, 2015.
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks, In International Conference Machine Learning, pp. 1120-1128, 2016.
Shaojie Bai, Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.0122) 2018.
Yoshua Bengio, Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 1994.
Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally 2013.
connected networks on graphs. arXiv preprint
Zhuo Chen, Yi Luo, and Nima Mesgarani. Deep attractor network for single-microphone speaker separation. CoRR, abs/161 1. 1.08930, 2016.
Hycong-Seok Choi, Janghyun Kim, Jaesung Huh, Adrian Kim, Jung-Woo Woo Ha, and Kyogu Lec. Phase-aware speech enhancement with deep complex u-net. arXiv preprint arXiv: arXiv:193.008 1903.03107, 2019.
Ivo Danihelka, Greg Wayne, Benigno Uria, Nal Kalchbrenner, and Alex Graves. Associative long short-term memory. 20M.,.2016
Muneer Ahmad Dedmari, Sailesh Conjeti, Santiago Estrada, Phillip Ehses, Tony Stöcker, and Martin Reuter. Complex fully convolutional neural networks for mr image reconstruction. In International Workshop on Machine Learning for Medical Image Reconstruction, pp. 30-38. Springer, 2018.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in neural information processing systems, pp. 3844-3852, 2016.
Michal Drozdzal, Eugene Vorontsov, Gabriel Chartrand, Samuel Kadoury, and Chris Pal. The importance of skip connections biomedical image segmentation. Deep Learning and Data Labeling for Medical Applications, pp. 179-187. Springer, 2016.
J. Du, Y. Tu, Y. Xu, L. Dai, and C. Lee. Speech separation of target speaker based on deep neural networks. In Proc. of ICSP, pp. 473-477, 2014.
Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William T. Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A speaker- independent audio-visuse model for speech separation. CoRR, 2018.
H. Erdogan, J. Hershey, S. Watanabe, and Le Roux. Phase-sensitive and recognition- boossee speech separation using deep recurrent neural networks. In Proc. of ICASSP, pp. 708 712, 2015.
Ruohan Gao, Rogério Schmidt Feris, and Kristen Grauman. Learning to separate object sounds by watching unlabeled video. CoRR, 2018
George M Georgiou and Cris Koutsougeras. Complex domain backpropagation. IEEE transactions on Circuits and systems analog and digital signal processing, 1992.
Klaus Greff, Rupesh K Srivastava, and Jürgen Schmidhuber. Highway and residual networks learn unrolled iterative estimation. arXiv preprint arXiv:/612.0777/1 2016.
D. Griffin and Jae Lim. Signal estimation from modified short-time fourier transform. IEEE Transactions Acoustics, Speech, and Signal Processing, 32(2):236-243, 1984.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034 2015.

---

Under review as a conference paper at ICLR 2020
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference computer vision, pp. 630-645. Springer. 2016b.
John R. Hershey, Zhuo Chen, Jonathan Roux, and Shinji Watanabe. Deep clustering: Discrimina- tive embeddings for segmentation and separation. CoRR, 2015.
Akira Hirose. Complex-valued neural networks: theories and applications. World Scientific, 2003,
Sepp Hochreiter. Untersuchungen zu dynamischen neuronalen Netzen. PhD thesis, 1991.
Guoning Hu and DeLiang Wang. Monaural speech segregation based on pitch tracking and amplitude modulation. Trans. Neur. Netw., 2004.
Po-Sen Huang, Kim Minje, Mark Hasegawa-Johnson, and Paris Smaragdis. Deep learning for monaural speech separation. ICASSP, 2014.
Stanislaw Jastrzebski, Devansh Arpit, Nicolas Ballas, Vikas Verma, Tong Che, and Yoshua Bengio. Residual connections encourage iterative inference. arXiv preprint arXiv:1710.047775 2017.
Cijo Jose, Moustpaha Cisse, and Francois Fleuret. 100.10420 2017.
Kronecker recurrent units. arXiv preprint
Tachwan Kim and Tülay Adal. Approximation by fully complex multilayer perceptrons. Neural computation, 2003.
Thomas N Kipf and Max Welling, Semi-supervised classification with graph convolutional networks. arXiv preprint 1rXiv:1609. 1609009000 2016.
Jonathan Le Roux, Gordon Wichern, Shinji Watanabe, Andy Sarroff, and John R Hershey. Phasebook Leveraging discrete representations for source separation. IEEE Journal of Selected Topics in Signal Processing, 13(2)3370-322, 2019.
and friends:
Yuan-Shan Lee, Chien-Yar Wang, Shu-Fan Wang, Jia-Ching Wang, and Chung-Hsien Wu. Fully complex deep neural network for chase-incorporating monaural source separation. In ICASP, 2017.
Yi Luo and Nima Mesgarani. Tasnet: time-domain audio separation network for real-time, single- channel speech separation. CoRR, abs/1711..05444 2017.
Yi Luo and Nima Mesgarani. Tasnet: Surpassing ideal time-frequency masking for speech separation. arXiv preprint arXiv: /0t.::/80000744. 2018.
Yurii Nesterov. A method of solving convex programming problem with convergence rate (1/k2). 1983.
Tohru Nitta. Orthogonality of decision boundaries in complex-valued neural networks. Neural Computation, 2004.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with general conditioning layer. 2018.
Tony Plate. Holographic reduced representations: Convolution algebra for compositional distributed representations. IJCAL, pp. 30-35, 1991.
Tony A Plate. Holographic reduced representations. IEEE Transactions on Neural nefworks, 6(3): 623-641, 1995.
David P Reichert and Thomas Serre. Neuronal synchrony in complex- valued deep networks. ICLR, 2014.
10

---

Under review as a conference paper at ICLR 2020
Oren Rippel, Jasper Snoek, and Ryan P Adams. Spectral representations for convolutional neural networks. In Advances in neural information processing systems, pp. 1449-2457, 2015.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. pp. 234-244-2 241,221,220 2015.
Ziqiang Shi, Huibin Lin, Liu Liu, Rujie Liu, and Jiqing Han. Furcanext: End-to-end monaural speech separation with dynamic gated dilated temporal convolutional networks. arXiv preprint arXiv:1902.04891, 2019.
N. Sturmel and L. Daudet. Signal reconstruction from stft magnitude: A state of the art. In In Proc. of the International conference on digital audio effects, 2006.
Chiheb Trabelsi, Olexa Bilaniuk, Ying Zhang, Dmitriy Serdyuk, Sandeep Subramanian, João Felipe Santos, Soroush Mehri, Negar Rostamzadeh, Yoshua Bengio, and Christopher J Pal. Deep complex networks. ICLR, 2017.
Shrikant Venkataramani and Paris Smaragdis. End-to-end networks for supervised single-channel speech separation. CoRR, abs/1810.02568, 2018.
Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement in blind audio 2006.
source separation. IEEE Trans. Audio, Speech & Language Processing,
Zhong-Qiv Wang, Jonathan Le Roux, DeLiang Wang, and John R. Hershey. End-to-end speech separation with unfolded iterative phase reconstruction. CoRR, abs/1 11111804.100000 2018.
Chao Weng, Dong Yu, Michael L Seltzer, and Jasha Droppo. Deep neural networks for single- channel multi-talker speech recognition. IEEE/ACM Transactions on Audio, Speech and Language Processing ( (TASLP), 23(10):1600-1679 2015.
Moritz Wolter and Angela Yao. Fourier mns for sequence analysis and prediction. arXiv preprint 1rXiv:1812.006645, 2018a.
Moritz Wolter and Angela Yao. Gated complex recurrent neural networks. arXiv preprint 2018b.
Dong Yu, Morten Kolbæk. Zheng-Hus Tan, and Jesper Jensen. Permutation invariant training of deep models for ipeaker-indepennent multi-talker speech separation. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 241-244 IEEE, 2017.
Richard $ Zemel, Christopher KI Williams, and Michael C Mozer. Lending direction to neural
networks. NIPS, 1995.
Jiong Zhang, Yibo Lin, Zhao Song, and Inderjit Dhillon. Learning long term dependencies via fourier recurrent units. arXiv preprint 1-Xiv:1803.060 2018.
A APPENDIX
A.1 DETAILS ABOUT THE AMPLITUDE AND PHASE-AWARE LOss
We show here that solving the two-equation system, assimilating the real part of the inner product, between the two signals x and y, to the square of the amplitude y, and canceling its imaginary part, amounts to canceling the differences in amplitude and phase between x and y, respectively (See equation For this we will use the following trigonometric properties:
cos(O2) cos(Op)
ccos(0, 0y) + coos (0, + 0g)
sin(O.) sin(
coss(2 0,) coss 0.. + 0w)
(9)
cos(Oz) sin(O,,)
=
=Osin0d= + .,y) fsin(0. 0yy)
sin(O.)) cos
=ssin(0 + -,,) + ssin(O. - -8y):
11

---

Under review as a conference paper at ICLR 2020
where og. 8y E R. For simplicity of notation, we will denote complex valuee target scalar y and a its estimate as x instead ÿ;
y lule = lol Jcos (.)) + (f)) E C
(10)
x lyle t = |a|| [cos( 0.) + (i)n E C.
Oy and O2 are the corresponding phases of the reference y and its complex estimate ær respec- Resolvng the system of equations below is equivalent of having both magnitude and phase of the reference and estimation identical OR when y O. Recall that R((xl))) Σ, ((,) (wz))(wwi) and (((x)>) = 2 (R(,)
R(,, (x,)]-
R()))(()) + (z)) - 1u/2 = 0 R())3(y) (x))((r) = 0
M()))(()) + (v)) = RV))² 31y)² R)x)3(y) Rx))3(x)
| (0.) |y| co,) + |o (0)) |ua sin = lu|
[z c0x((2)) |// sin (0,, = y c0, |ao| sfn(
|ar lwv (oos(2)) cos( (0)) + sin( sin (0w)] |M/ [ [3/ (o0((22) sin((,,) cos(O,) sin (02)) 0
[x:| cos(O,)) + sin(O=) sin (i,)) |M| [0]] OR y/ 0 OR (00) s0, - cos(8,) sin(O=)) =
|u|
OR |=| (co cos(6,) sin sin(
lul
x
OR [y| OR cos s0,) cos(O,)) sin)
|u|
OR
|=| (0 0,) + cosss(0 0v)) + (00 (0.
+ +00)) ly9
x
OR [y| = OR siin (0.. 0,) = 02) =0i 0,)) =isin(0= 0g)
|ul
OR
|x| cos(0. 0,)
OR [y| OR sin(0,- 0g) sin 0,)
|
0
OR
|=|| cos((,, 0,)
OR
[4| = 0 OR 0g 0, 0 ()
|
0
OR
|=|| cos(O, 0,)
0
OR 1p] = 0 OR sin(0=- 0y) =sin(0, - 0₂)
|v|
OR
|=| (k=) 1y/. E Z
OR
11 0 0, = 0g ko, k E
|v|
OR
|=|| cos (2k
OR tr/oos((2k
+ 1)T) lel (because cos( cos((2 1)x) -1)
OR
16] = OR 0, fg , EZ
|vl
OR
|=| lvl OR |_| M =
OR
lv| OR 0, 0g 2K k EZ
|ul 0 OR |x| lyl
8. 0, + 2 T, k' E Z
(1I)
We have just shown that (x(x))))
= 1y AND (((ly)), = equivalent of having ((y) = OR 1,5| |ry)) AND Lx; Zyzl. This means that the real and imaginary parts of the inner product between the estimate and target are respectively responsible the amplitude and phase matching between the estimate and the target. Now, solution corresponding to null reference vector y could be problematic as it leads to an infinite number of choices for the estimated signal In fact, Choi et al. (2019) mentioned this issue and chose to work with cosine similarity based function in order to learn from noisy-only data. This is why it is more convenient to work with the normalized inner product loss.
12

---

Under review as a conference paper at ICLR 2020
A.2 COMPLEX LAYER NORMALIZATION
Just as in complex batch normalization, complex layer normalization consists in whitening 2D vectors by left-multilying the O-centered data (x - E[æ]] by the inverse square root of the 2 x 2 covariance matrix V. = (V) (a -[(z]) where the covariance matrix V
Vyr V
v
Vir
Vi
(ov()(2),3(z)) Cov(S(()) Rxx)) (ov(3(z),,(zz))
Cov(Bt(a:) (())))
Complex layer normalization is distinguished from complex batch normalization by its computation of the mean and covariance statistics over the layer features instead of the batch instances. This allows us, as in the real-valued version, to avoid estimating batch statistics during training. An intuition for batch normalization's inappropriateness is related to the sparsity, in both time and frequency domains, of speech. This is reflected in the spectrograms. Speech is temporally halting and restarting, and spectrally consists of most few simultaneously-hlld fundamentals and their discrete overtones. Mixing few speakers does not significantly change this property.
In the light of this observation, it stands to reason that statistics computed across a batch's 's multiple utterance mixtures are almost meaningless. Speakers within and across utterance mixtures are not controlled for volume, nor can their pauses be meaningfully aligned. Batch statistics will therefore be inappropriately driven by the mixture with the most simultaneous speakers, the loudest speaker(s), or the speaker(s) with the "dirtiest" spectrogram. Finally, the absence of any speech, batch statisties will inappropriately boost background noise to standardized magnitude.
The above motivates the use of exclusively satrae normalization techniques like Layer Normal- ization for speech data. Batch normalization is more appropriate for natural images, which are dense, both in space and frequency.
In addition to the fact that ensures a more robust normalization data when the number of feature maps is sufficiently large. In fact, according to the weak law of large numbers, the sample size increases, the sample statistics approximate their expected values. Therefore, when the number of feature maps far exceeds the number of batch instances, we obtain more robust estimates because they converge, probability, to the corresponding expected values.
normalization is more appropriate for speech signals, CLN
A.3 FIGURES
Complex Laver Norm
Dowssssmpiigg Block
Residiaa Books
Complex Activation
è Residual Faods
Halving Block
Complex Consolitiin
Doubling Bock
Residial
Compler Lager Norm
Revidual Biods
Upsempling Block
Complex Activation
Complec Covollition
Figure 2: The basic structures of our U-Net downsampling block D; (Left) and our U-Net upsampling block U, (Middle) used respectively the encoding and the decoding paths Figure [ The structure of basic complex residual block (Right) in each of D, and U,.
13

---

Under review as a conference paper at ICLR 2020
A.4 DATA PRE- PRE-PROCSSIING AND TRAINING DETAILS
The speech mixtures are generated using the procedure adopted in Erdogan et al. (2015); Wang et al. 2018). More precisely, the training set consists of 30 hours of two-speaker mixtures that were generated by randomly selecting sentences (uttered by different speakers) from the Wall Street Journal WSJO training set called si tr__s__ s. The signals are then mixed with different amplitude factors, leading signal-to-noise ratios (SNR) ranging between 0 dB and 5 dB. Using the same method, we also generated 10 hours of validation set. The test set is composed of 5 hours that were generated similarly using utterances from the different speakers belonging to the WSJO development set s dt 05. The data sampling rate is 8KHz. Regarding the STFT parameters, Hann window of size 256 and a hop length equal to 128 are used.
Table 2 (see section A.5) and Table contain the results for the experiments conducted using the Wall Street Journal dataset. All models in Tables (see section A.5) and 3 were trained using the backpropagation algorithm with Stochastic Gradient Descent with Nesterov momentum (Nesterov) 1983) set at 0.9, The gradient norm was clipped to 1. We used the learning rate schedule described in Trabelsi al. (2017). In order to warm up the model during training, constant learning rate of 0.01 was fixed for the first 10 epochs. From epoch 10 to 100, the learning rate was increased to 0.1. Later, annealing of the learning rates by factor of 10, epochs, 120 and 150 was performed. We ended up the training at epoch 200. Models in Table [see section AA.5) have been trained using batch size of 40. Models in Table 3 have been trained using batch size of 24 to fit in the GPU memory. All the models have been trained in parallel using 8 V100 GPUs. For all the tested losses, we used the Permutation Invariant Training criterion knows as PIT (Yu et al.) 2017). The PIT criterion allows to take into account all possible assignments between the target signals and the estimated clean speeches. This is done by computing all possible permutations between the targets and the estimated clean speeches. During training, the assignment with the minimal loss is considered for backpropagation. This is due to the fact that for the synthetically mixed input, the order of the target speakers is randomly chosen and it doesn't 't satisfy specific criterion. This random order in the target speakers causes the well-know label permutation problem (Hershey et al. (2015) Weng et al. 2015). The PIT criterion allows then to reduce significantly this problem by considering the output-target assignment yielding the minimal training loss. During inference, we assume that the model has learned to produce output that does not permute speeches. (Yu et al.) (2017) mention that output-to-speaker assignment may change across time frames. This would have the effect of decreasing the Signal to Noise Ratio (SNR) and the Signal to Distortion Ratio (SDR) as it causes interference of speakers speeches.
A.5 EXPERIMENTS
We tried different configurations combining unitary and standard complex initializations. All of these initializations have been proposed by Trabelsi et al. (2017). It turned out that the best configuration is obtained when using a complex standard initialization for all layers, except for the convolutional layer, generating the FiLM parameters. and the first convolutional layer in the generating mask function which precedes the 2 residual blocks. For the Above-mentioned convolutional layers unitary initialization respecting the He criterion (He et al.) 2015) was applied. This is not surprising as unitary weight matrix E dd constitutes a basis of Cd Therefore any complex-valus vector in Cd, such as those representing the FiLM parameters or the masks, could be generated using linear combination of the row vectors of that unitary matrix.
In Tables 2 and 3 we experiment with architectures that use different number of mixture transfor- mations. Adding mixture transformations does not significantly increase the number of parameters compared to the size of the whole model. In the case where 15 transformations are adopted, the number of parameters increased by less than 1% of the total number.
Since Table 2 first row contains baselines, they exclude our proposed masking method and loss. These baselines (both real and complex) are architecturally the same as the U-Net of Figure without the FiLM, the GenerateMask and the averaging operation. A real counterpart of complex model is one where the convolution and the normalization layers are real, the nonlinearity is plain ReLU and He init is used for the weights. The real and complex U-Nets output the masks which are complex multiplied with the mix in order to infer the clean speech of the speakers. All the complex models, whether they have approximately the same number of parameters (R:8.44M C.7.4M),
14

---

Under review as a conference paper at ICLR 2020
Table 2: Speech separation experiments on two speakers using the standard setup with the Wall Street Journal corpus.
We explore different real and complex-valued model variants and report the test SDR. à is the the number of residual blocks used inside the residual U-Net block (See Figure 1). Start Fmaps the number of feature maps in the first layer of the encoding path in the U-Net. The Start Fmaps defines width of each of the successive layers in the model. We respectively double and half the size of the layers in each of the successive downsampling and upsampling stages. The effective number of feature maps for complex feature map equal to number of reported feature maps x This is due to the fact that it has real and an imaginary part. The number of parameters is expressed in millions. The number of imput mixture transformations also reported. Test SDR scores for different time and spectral domain losses are inserted in the last column.
MODEL
k:
START FMAPS
PARAMS
TRANSFORMS
Loss FUNCTION
TEST SDR
REAL U-NET
1
8.45
L2ireg
4.59
REAL U-NET 2
14.76
L2ireq
7.92
COMPLEX U-NET
1
32
4.29
L2ireq
9.61
COMPLEX U-NET
2
32
7.4
0
L.2rreq
9.70
COMPLEX U-NET
2
40
11.55
T.2ireq
10.30
COMPLEX U-NET
2
40
11.55
0
CSimLoss
10.21
COMPLEX U-NET
2
40
11.55
T.2mme
9.31
COMPLEX U-NET
2
11.55
C0Ssime
9.34
COMPLEX U-NET
2
40
11.57
L2ireq
10.58
COMPLEX U-NET
2
11.57
CSimLoss
10.87
COMPLEX U-NET
2
40
11.57
5
L2
10.31
COMPLEX U-NET
2
40
11.57
5
CU8sime
10.14
COMPLEX U-NET
2
40
11.61
10
L2iveg
10.59
COMPLEX U-NET
2
40
11.61
10
CSimLoss
10.90
COMPLEX U-NET
2
40
11.61
10
L2tine
10.86
COMPLEX U-NET
2
11.61
008cime
10.47
COMPLEX U-NET
2
40
11.67
15
1.2ipeq
10.93
COMPLEX U-NET
2
11.67
CSim Loss
10.91
COMPLEX U-NET
2
40
11.67
15
T.2mmm
10.66
COMPLEX U-NET
2
40
11.67
15
C08sime
10.74
half (R:8.45MM C:4.39M) third, with half the depth (R:14.76M)) C:4.39M) outperformed by large margin their real counterparts. This shows that whether the comparison is fair, or even where advantages in terms of capacity and depth given to the real network, it doesn't perform well as complex models when comes to process complex input and infer complex output. Thus, we will no longer focus real-valued models, but, instead, will concentrate transformations and losses that are appropriate for complex-valuee models.
Three major observations can be inferred from the numbers displayed in Table 1- Wider and deeper models improve the quality of separation in terms of SDRs; 2- The increase in the number of input transformations has a positive impact the task of separating audio sources, as additional input transformations achieve higher SDR scores; 3- For given number of input transformations, the best results are obtained with losses computed in the spectral domain. For all the experiments reported in Table 2 either the CSimLoss or the 1.2 achieve the highest SDR.
The scores reported in Table show that the local ensembling procedure is beneficial to the task of speech separation. This rewarding impact is confirmed in all experiments of Table (See also Figure 3). As mentioned section each mask could seen as a feature the speaker embedding and the generated masks together constitute the whole embedding. Performing dropout on the masks might then allow to perform regularization for the retrieval and separation mechanism. Dropping out a mask is equivalent dropout of input mixture transformations clean speech candidates. Since spectral loss functions yielded higher SDRs than their time-domain counterparts, we adopted them to evaluate the effect of applying different dropout rates the input transformations, Wider and deeper models
15

---

Under review as a conference paper at ICLR 2020
Table 3: Experiments on two speaker speech separation using the standard setup with the Wall Street Journal corpus. We explore different numbers of imput mixture transformations and different dropout rates on the latter using the training losses defined in the spectral domain. The losses in questions are 1.2rreg and CossL The number of parameters expressed in millions. All tested models contain 44 feature maps in the first downsampling layer of the U-Net instead of 40 in Table The same number = residual blocks used inside the basic structure of residual U-Net block. SDR scores are shown in the last column.
PARAMS
TRANSFORMS
DROPOUT
Loss FUNCTION
TEST SDR
13.97
0
12
9.88
13.99
0
L2tte
10.11
14.03
10
0
L2treq
10.91
14.09
L2treq
9,92
13.97
0
CSimLoss
9.87
13.99
0
CSimLoss
10.64
14.03
10
0
CSimLoss
11.05
14.09
15
0
CSimLoss
10.82
13.99
0.1
L2treq
10.54
14.03
10
0.1
L.2traq
10.72
14.09
0.1
T.2ntqq
10.91
13.99
0.1
CSimLoss
10.96
14.03
10
0.1
CSimLoss
11.34
14.09
15
0.1
CSimLoss
11.22
13.99
0.2
L2ne
10.67
14.03
10
0.2
L2nea
10,90
14.09
0.2
L2treq
10,90
13.99
0.2
CSimLoss
11.23
14.03
10
0.2
CSimLoss
11.29
14.09
15
0.2
CSimLoss
11.23
13.99
5
0.3
L2
10.71
14.03
10
0.3
L2tta
10,06
14.09
0.3
T.2treq
10.91
13.99
0.3
CSimLoss
11.21
14.03
10
0.3
CSimLoss
11.12
14.09
15
0.3
CSimLoss
11.06
13.99
0.4
L2trea
10.72
14.03
10
0.4
L2treq
10.74
14.09
15
0.4
L2treq
10.83
13.99
0.4
CSimLoss
11.09
14.03
10
0.4
CSimLoss
11.08
14.09
15
0.4
CSimLoss
11.12
with Start Fmaps = 44 and k=2 residual blocks are tested in the conducted experiments. Results are reported in Table
In the absence of dropout and multiple transformations. we observe from the results displayed in Table that wider models are not necessarily more beneficial the separation task. The SDRs reported the case of no mixture transformations are 9.88 and 9.87 for the wider model. These SDR scores correspond to the L22reg and CSimLoss losses respectively. However, for the narrower models, SDRs of 10.30 and 10.21 were respectively reported for the same losses in Table 2 This means that wider models have the potential to overfit. On the other hand, if input transformations are taken into account, jump in the SDR is observed. When 10 input transformations are introduced, SDR scores of 11.05 and 10.90 are recorded with the CSimL and the L2treq losses, respectively. Lower SDR performances are recorded when ensembling is implemented with mixtures of 5 and 15 transformations, respectively. This means that the local ensembling procedure is acting as a regularizer. However, tradeoff in terms of the number of input transformations (and in terms of clean speech candidates) has to be made as increasing the number input transformations might worsen the performance of the model and lead to overfitting.
16

---

Under review as a conference paper at ICLR 2020
Validation Curves for the best models willvwthoutt transformations
12
10
SCR (18)
modals obtainod multipl otons
Moan for the best models ubtainod where no tran malkon was used
Standdarid devistion for the best modele oblteieed where mutiple Iranaformatione
75
125
150
175
Epochs
Figure Validation curves of models with and without performing multiple input transfor- mations. The plotted curves relate to models reported in Table 3 Models with multiple input transformations outperform those without transformations. The former achieved higher SDR scores, on average.
Dropping out the speech candidates using small probability rate has further regularization effect on the wider model. This could be inferred from the results reported in 3 also Figure 5) 0.4. rate of yielded the best result as caused jump of SDR score from 11.05 to 11 11.34. It is important to emphasize again the importance of having a compromise terms of the number of transformations. For instance, for most of the dropout rates experimented, number of 10 mixture transformations yielded the highest SDRs. In all the experiments reported in Table the CSimLoss clearly outperformed the L.2treg (See Figure In fact, regardless of the dropout rate and the number of input transformations employed, for wider models using the L2iren training loss function, the SDR score did not cross the threshold of 10.91 dB. The highest SDR score obtained, when using the 1.2 loss function, is 10.93. This value corresponds to a narrower model with 15 input transformations (see Table 2).
We employed different dropout rates varying from
17

---

Under review as a conference paper at ICLR 2020
Validation Curves for the best models using the L2 spectral Loss and the CSimLoss
8
SDR (d8)
Mean for the best models obtained with the L2 spectral loss
Mean for the best models obtained with the CSimLoss
standartd
standartd deviation the best models obtained with CSimLoss
125
175
Epochs
Figure 4: Validation curves of the models that yielded the highest SDRs using either the L2 spectral loss or our CSimLoss, The Drawn curves are related models reported Table 3
models
SDR (08)
dropout was
Moan the medels abtained where dropout was used
Standarid deviation models drapout
Standarid deviation for the models where dropout was used
Epochs
Figure 5: Validation curves of the models that yielded the highest SDRs for both cases where dropout on the input mixture transformations was used and where it was not. The Drawn curves are related to models reported in Table
18

---

Under review as a conference paper at ICLR 2020
A.6 OFATE-OF-THE-ART TABLE
Table 4: Listing of some existing state-of-the-an methods for two speaker speech separation
Model
SNR
Window Size
Hop Length
Temporal Model Used
Inpui
SDR
Hershey es 2015
Clustering
Uniform (0, 3]
ms 256 sumples
ms samples
BLSTMs
log [STF)(())
6.5
Uniform
256 samplex
samples
RR.STMs
log [STF)(())
[1.10 al, 2017)
Ancher 10ig iitractoo
Unifoem [0, 4B
32 ms 256 samples
8 ms 64 samples
BLSIMs
log (sTF
10.8
Lue Mesgarani
2017
[0,5] dB
segment size ms = 40 samples
domain signal
Raw
TusNet Mengarani
Uniform
Time dome-damiinnnee filter samples
50% overlap samples
Temporal Convolutio Networks
Raw time
signal
15,6 (cloitnd)
(opprdacod
wilh [0.5]
Deep Comples U-Net
(Ours, w/o extraction mechanism)
Uniferm [0,
32 256 samples
16 128 samples
Nose (No temperal
model used)
STFT())
9.70
Deep Complex U.Nex
(Ours, w/ extrsction mechonism)
Uniform (0, 5)
32 ms 256 sumples
mss
128 samples
recireet model used)
(No tensporal
STFT())
11.34
As can be seen from Table state-of-the-art results speech separation depend largely on the following:
1. The use of model that takes into account short and long term temporal dependencies such as BLSTMs or Temporal Convolutional Networks (Bai et al. 2018). Almost all the methods since (Hershey et al, 2015) that have led to improvements in state-of-the-art speech separation have used either BLSTMs or TCN;
2. The STFT window size and hop length the time-domain input segment size when using the raw signal. Yu et al. (2017) demonstrated that the smaller the window size, hop length are, the better the quality of separation. This probably explains Luo & Mesgarani] (2017) and & Mesgarani (2018) selection of very short time-domain segment sizes of and 2 ms for the TasNet and ConyTasNi archtiectures respectively.
L
19


Reviews:
[{"review_id": "rygaxccJcB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose complex valued neural networks to perform audio source separation in the Fourier domain. The adapt a well known U-Net architecture to the task by introducing a complex-valued FiLM layer and a new complex similarity loss that explicitly takes magnitude and phase into account. They motivate the use of complex values well and demonstrate performance and parameter efficiency improvements over real-valued baselines. Importantly, they do not need to perform spetrogram inversion because their network works natively in the complex domain. Despite the quantitative improvements over spectral models, they still slightly underperform the ConvTasNet baseline that operates directly in the waveform domain (which was slightly misleading to not include in the table). The authors perform an extensive hyperparameter search to tune the model and provide sufficient detail to reproduce their experiments. While the results did not improve upon the best baseline, they do provide further evidence to the value of using complex-valued neural networks to handle complex-valued data (where phase and synchronicity matter), which I believe will be of value to the ICLR community, and thus I lean slightly in favor of acceptance. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HygJE9ZXYr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new method for source separation, by using deep learning UNets, complex-valued representations and the Fourier domain. Concretely, their contribution is : i) a complex-valued convolutional version of the Feature-Wise Linear Modulation, able to optimise the parameters needed to create multiple separated candidates for each of signal sources that are then combined using signal averaging; ii) the design of a loss that takes into account magnitude and phase while being scale and time invariant. It was then tested and compared with real-valued versions, and also some state-of-the-art methods.\n\nOverall, I think this paper is of good quality and proposes an interesting method for this crucial task of source separation. However, I found the paper too dense and difficult to read (even if well written), and it looks like a re-submission from a journal paper of more than 8 pages. I would suggest the authors to shrink the paper so it *really* fits into the 8-pages (without important figures or important implementation details in the appendices), maybe at the cost of leaving some parts (such as old related works) out of the paper. The experiments are important here, and it is too bad that the comparison with state-of-the-art is just in the last paragraph, while the results do not seem to show any improvements compared to the other methods. The computational time might be very important here, as the claim is that FFT reduces time computation, but I did not had time to go through all the appendices.\n\nPositive aspects:\n- The work is well documented and motivated, and I found that the reflexion leading to the method is of good quality. \n- Concretely, I found interesting the use of the FiLM, originally designed for another application, for minimizing the SNR of the signal sources. The motivation/proof is quite clear too.\n- Equally, the motivation for the design of the new loss is clear and interesting.\n- I also found important the experiments, that shows in the same table the difference between the method without the complex-valued part and with different parameter values.\n\nQuestions and remarks:\n- I have to recall that I am not an expert on source separation and complex-valued deep learning. Yet, I have had difficulties in understanding the structure of the method, even if the different parts were clearly explained. The figure 1 is very useful, but I found it not clear enough and too small. I went to find some informations in the appendices, but there are too much crucial information there and I did not have time to go through all of it.\n- The use of the U-net architecture is not explained (just some citations are given). What is supposed to be the output of it? \n- When you say 'to be more rigorous, we can assume in the cse of speech separation that, for each speaker, there exists an impulse response such that when it is convolved with the clean speech of the speaker, it allows to reconstruct the mix' : why can we be sure that it is always possible, and why is it more rigourous?\n- Why are the additive noises epsilon_i supposed to have the same E(|epsilon_i|^2|) ? Even if they are uncorrelated, what is the hypothesis behind that?\n- In the CSimLoss, why (i) is the real part negative and the imaginary part positive; (ii) is the imaginary part squared?\n- have you tested with a higher lambda_imag (as the larger is now the best)?\n- It looks from the end of the paper that the method is still not achieving better results than the state-of-the-art. I agree with the authors as it might not be the scope of the paper, but then what is it? If it's time computation, it is not shown in the paper. If it is just a methodology, what would be required in the future to beat the best method?\n- In the results, table 1, in the last 4 lines: it looks from 1st and 2nd line that the new loss CSimLoss is not very different from the L2 (9.88 compared to 9.87). The best result, in the 4th line, cannot be compared to the 3rd line as both the loss and the number of transforms are different. I then found those values in the appendices, but it would be best to show fewer parameters varying in the main paper, but show some results that can be easily compared. \n- What is the importance of the first paragraph in 2.1? I was not aware of the holographic reduced representations, but I don't understand it more now, and I don't see why explaining that for 15 lines.\n\nSmall remarks:\n- 'deep complex valued models have *just* started to gain momentum'... with citations beginning in 2014, I would not say 'just'.\n- 'in the frequncy domain is then, ...' --> frequency + no coma\n- Figure 2 is in the appendix, while in the text it is not said so. I was lost. This figure should not be in the appendix as the appendix should not have key elements, but just details that are not important for the understanding of the paper.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "B1gkXWp6tr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work researches the deep complex-valued neural networks. Specifically, it proposes a new signal extraction mechanism that operates in frequency domain and applies to address the speech separation issue. Also, a function is proposed to explicitly consider both the magnitude and phase information of a signal. Related work on learning representation in frequency domain and speech separation is well introduced. Theoretical analysis is conducted to show the motivation and connection to signal processing. The architecture of the deep neural networks is presented in details, with the elaboration of the complex mask generation. Experimental study is conducted on a benchmark dataset to compare the proposed complex networks with those using real-part values only to demonstrate the improvement. \n\nThe rating is 3: Weak Reject considering that the novelty is limited and the experimental study is weak. \n\n1. The significance of the theoretical analysis in Eq.(1) to Eq.(4) needs to be better explained. Currently, they seem to be some straightforward results in the field of signal processing;\n2. The proposed CSimLoss is interesting. However, its effectiveness seems to be limited as demonstrated in Table 1. It can be found that the CSimLoss in some cases is only comparable (or even inferior) to the L2freq loss;\n3. The mask generation proposed in Section 6 conceptually is largely an attention mechanism that has been widely applied in deep networks; \n4. The experimental comparison in Table 1 is limited, although some improvements have been demonstrated. This work shall also make a comparison with some of the existing methods on speech separation as described in Section 2.2.\n5. It was mentioned in the last paragraph of Section 7 that (Shi et al. 2019) uses different data preparation than this paper. Can this paper use the same data preparation as (Shi et al. 2019) and perform some comparisons?\n6. Why did (Shi et al. 2019) achieve better SDR (12.1 vs. 11.3) than the proposed method using standard setup?\n7. What if the mechanism of mask generation is also applied to Real U-Net? How much improvement can this bring? \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper discusses audio source separation with complex NNs.  The approach is good and may increase an area of research.  But the experimental section is very weak and needs to be improved to merit publication.
Decision: Reject