Retrieving Signals in the Frequency Domain with Deep Complex Extractors

1 INTRODUCTION . Complex-valued neural networks have been studied since long before the emergence of modern deep learning techniques ( Georgiou & Koutsougeras , 1992 ; Zemel et al. , 1995 ; Kim & Adalı , 2003 ; Hirose , 2003 ; Nitta , 2004 ) . Nevertheless , deep complex-valued models have only started to gain momentum ( Reichert & Serre , 2014 ; Arjovsky et al. , 2015 ; Danihelka et al. , 2016 ; Trabelsi et al. , 2017 ; Jose et al. , 2017 ; Wolter & Yao , 2018b ; Choi et al. , 2019 ) , with the great majority of models in deep learning still relying on real-valued representations . The motivation for using complex-valued representations for deep learning is twofold : On the one hand , biological nervous systems actively make use of synchronization effects to gate signals between neurons – a mechanism that can be recreated in artificial systems by taking into account phase differences ( Reichert & Serre , 2014 ) . On the other hand , complex-valued representations are better suited to certain types of data , particularly those that are naturally expressed in the frequency domain . Other benefits provided by working with complex-valued inputs in the spectral or frequency domain are computational . In particular , short-time Fourier transforms ( STFTs ) can be used to considerably reduce the temporal dimension of the representation for an underlying signal . This is a critical advantage , as training recurrent neural networks ( RNNs ) or convolutional neural networks ( CNNs ) on long sequences remains challenging due to unstable gradients and the computational requirements of backpropagation through time ( BPTT ) ( Hochreiter , 1991 ; Bengio et al. , 1994 ) . Applying the STFT on the raw signal , on the other hand , is computationally efficient , as in practice it is implemented with the fast Fourier transform ( FFT ) whose computational complexity is O ( n log ( n ) ) . The aforementioned biological , representational and computational considerations provide compelling motivations for designing learning models for tasks where the complex-valued representation of the input and output data is more desirable than their real-counterpart . Recent work has provided building blocks for deep complex-valued neural networks ( Trabelsi et al. , 2017 ) . These building blocks have been shown , in many cases , to avoid numerical problems during training and , thereby , enable the use of complex-valued representations . These representations are well-suited for frequency domain signals , as they have the ability to explicitly encode frequency magnitude and phase components . This motivates us to design a new signal extraction mechanism operating in the frequency domain . In this work , our contributions are summarized as follows : 1 . We present a new signal separation mechanism implementing a local ensembling procedure . More precisely , a complex-valued convolutional version of Feature-wise Linear Modulation ( FiLM ) ( Perez et al. , 2018 ) is used to create multiple separated candidates for each of the signals we aim to retrieve from a mixture of inputs . A signal averaging operation on the candidates is then performed in order to increase the robustness of the signal to noise and interference . Before the averaging procedure , a form of dropout is implemented on the signal candidates in order to reduce the amount of interference and noise correlation existing between the different candidates . 2 . We propose and explore a new magnitude and phase-aware loss taking explicitly into account the magnitude and phase of signals . A key characteristic of our loss is that it is scale- and time-invariant . We test our proposed signal extraction mechanism in the audio source separation setting where we aim to retrieve distinct audio signals associated with each speaker in the input mix . Our experiments demonstrate the usefulness of our extraction method , and show its regularizing effect . 2 RELATED WORK . 2.1 RELATED WORK ON LEARNING REPRESENTATIONS IN THE FOURIER DOMAIN . Leveraging the Convolution Theorem to retrieve information has been done decades ago in the machine learning community using holographic reduced representations ( HRRs ) in the context of associative memories ( Plate , 1991 ; 1995 ) . HRRs enable one to store key-value data . Retrieval of a value in the data associated with a given key can be performed by convolving the whole data with the key or by applying an inner product between these two . By applying a fast Fourier transform ( FFT ) on the keys and the data , one can perform elementwise multiplication between the Fourier transforms and apply an inverse FFT to convert the result to the time domain . This would be equivalent to performing circular convolution between the key and the data in the time domain and has the advantage of being less expensive . Recently , Danihelka et al . ( 2016 ) have used associative memories to augment the capacity of LSTMs and to increase their robustness to noise and interference . For that , they applied independent permutations on the memory to create multiple copies of it . This enables one to obtain decorrelated noise in each of the permuted copies . A complex multiplication is then performed between the key and each of the copies . A signal averaging on the resulted multiplications eliminates the decorrelated noise in them and strengthens the signal-to-noise ratio ( SNR ) of the retrieved signal . Danihelka et al . ( 2016 ) , however , have not relied on FFTs in order to convert the temporal signals to the frequency domain . In fact , they assumed that complex-valued multiplication between the key and the data is itself enough to perform retrieval , and they have assumed that for each input representation the first half is real and the second one is imaginary . During this decade , interest in Fourier domain representations has started to grow in the machine learning community . Bruna et al . ( 2013 ) introduced a generalization of convolutions to graphs using the Graph Fourier Transform , which is defined as the multiplication of a graph signal by the eigenvector matrix of the graph Laplacian . However , the computation of the eigenvector matrix is expensive . Recently , methods that are computationally more efficient have been introduced in Defferrard et al . ( 2016 ) and Kipf & Welling ( 2016 ) to avoid an explicit use of the Graph Fourier basis . In the context of Convolutional Neural Networks ( CNNs ) , Rippel et al . ( 2015 ) introduced spectral pooling , which allows one to perform pooling in the frequency domain . This allows one to maintain the output spatial dimensionality , and thus the technique can retain significantly more information than other pooling approaches . Rippel et al . ( 2015 ) have also observed that the parametrization of the convolution filters in the Fourier domain induces faster convergence during training . Arjovsky et al . ( 2016 ) designed a recurrent neural network ( RNN ) where the transition hidden matrix is unitary . More precisely , the hidden transition matrix is constructed using the product of specific unitary transformations such as diagonal matrices , permutations , rotations , the Discrete Fourier Transform and its inverse . This allows one to preserve the norm of the hidden state , and as a consequence , mitigates the problem of vanishing and exploding gradients . Wolter & Yao ( 2018a ) designed an RNN where the input is converted to the frequency domain using a Short Time Fourier Transform ( STFT ) . The output is converted back to the time domain by applying an inverse STFT . Zhang et al . ( 2018 ) proposed a Fourier Recurrent Unit ( FRU ) where they showed that FRU has gradient lower and upper bounds independent of the temporal dimension . They have also demonstrated the great expressivity of the sparse Fourier basis from which the FRU draws its power . As we consider the task of speech separation as case study , we provide a related work section on both time domain and frequency domain speech separation methods in section 2.2 . 2.2 RELATED WORK ON TIME DOMAIN AND FREQUENCY DOMAIN SPEECH SEPARATION . Speech separation has been the subject of extensive study within the audio processing literature for a considerable amount of time . Recently , there has been growing interest in leveraging deep learning techniques ( Du et al. , 2014 ; Huang et al. , 2014 ; Hershey et al. , 2015 ; Gao et al. , 2018 ; Ephrat et al. , 2018 ) to tackle the speech separation problem . Hershey et al . ( 2015 ) proposed a deep clustering approach to speech separation . The basic idea is to learn high-dimensional embeddings of the mixture signals , that is later exploited to separate the speech targets with standard clustering techniques . A recent attempt to extend deep clustering led to the deep attractor network proposed by Chen et al . ( 2016 ) . Similarly to deep clustering , high dimensional embeddings are learned , but the network also creates the so-called “ attractors '' to better cluster time-frequency points dominated by different speakers . The aforementioned approaches estimate only the magnitude of the STFTs and reconstruct the time-domain signal with the Griffin-Lim algorithm ( Griffin & Lim , 1984 ) or other similar procedures ( Sturmel & Daudet , 2006 ) . Other papers have recently proposed to integrate the phase-information within a speech separation system without necessarily working in the complexvalued frequency domain . The work by Erdogan et al . ( 2015 ) , for instance , proposes to train a deep neural network with a phase-sensitive loss . Another noteworthy attempt has been described in Wang et al . ( 2018 ) , where the neural network still estimates the magnitude of the spectrum , but the timedomain speech signals are retrieved by directly integrating the Griffin-Lim reconstruction into the neural layers . Furthermore , methods reported in Wang et al . ( 2018 ) integrate the phase-information within a speech separation system by reconstructing the clean phase of each source starting from the estimated magnitude of each source and the mixture phase . This is fundamentally different from our proposed framework , as we provide an end-to-end solution to perform signal retrieval in the complex-valued frequency domain , and process both spectrogram magnitude and phase information rather than working only on magnitude representation with heuristic reconstruction of phase . Another attempt to estimate the clean phase is reported in Le Roux et al . ( 2019 ) where the clean phase of each speaker is estimated using discrete representation . This is also fundamentally different from our work as it considers a discrete representation of the phase for source separation and , in our case , we consider continuous representation of the complex-domain signal . Instead of explicitly integrating phase-information , other recent work perform speech separation in the time domain directly , as described in Venkataramani & Smaragdis ( 2018 ) . Likewise , the TasNet architecture proposed in Luo & Mesgarani ( 2017 ) and ConvTasNet ( Luo & Mesgarani , 2018 ) perform speech separation using the mixed time signal as input . Operating directly on the time-domain signal using the ConvTasNet architecture , which implements temporal convolutional networks ( TCN ) ( Bai et al. , 2018 ) , has led to state-of-the-art results in audio speech separation ( Luo & Mesgarani , 2018 ; Shi et al. , 2019 ) . The studies by ( Lee et al. , 2017 ; Hu & Wang , 2004 ; Huang et al. , 2014 ) are more related to our work as they address the speech separation problem by processing the complex-valued spectral input of the mixed speech . However , this was done without leveraging the recent advances in complex-valued deep learning .

Summary: This paper proposes a new method for source separation, by using deep learning UNets, complex-valued representations and the Fourier domain. Concretely, their contribution is : i) a complex-valued convolutional version of the Feature-Wise Linear Modulation, able to optimise the parameters needed to create multiple separated candidates for each of signal sources that are then combined using signal averaging; ii) the design of a loss that takes into account magnitude and phase while being scale and time invariant. It was then tested and compared with real-valued versions, and also some state-of-the-art methods.