Retrieving Signals in the Frequency Domain with Deep Complex Extractors

1 INTRODUCTION . Complex-valued neural networks have been studied since long before the emergence of modern deep learning techniques ( Georgiou & Koutsougeras , 1992 ; Zemel et al. , 1995 ; Kim & Adalı , 2003 ; Hirose , 2003 ; Nitta , 2004 ) . Nevertheless , deep complex-valued models have only started to gain momentum ( Reichert & Serre , 2014 ; Arjovsky et al. , 2015 ; Danihelka et al. , 2016 ; Trabelsi et al. , 2017 ; Jose et al. , 2017 ; Wolter & Yao , 2018b ; Choi et al. , 2019 ) , with the great majority of models in deep learning still relying on real-valued representations . The motivation for using complex-valued representations for deep learning is twofold : On the one hand , biological nervous systems actively make use of synchronization effects to gate signals between neurons – a mechanism that can be recreated in artificial systems by taking into account phase differences ( Reichert & Serre , 2014 ) . On the other hand , complex-valued representations are better suited to certain types of data , particularly those that are naturally expressed in the frequency domain . Other benefits provided by working with complex-valued inputs in the spectral or frequency domain are computational . In particular , short-time Fourier transforms ( STFTs ) can be used to considerably reduce the temporal dimension of the representation for an underlying signal . This is a critical advantage , as training recurrent neural networks ( RNNs ) or convolutional neural networks ( CNNs ) on long sequences remains challenging due to unstable gradients and the computational requirements of backpropagation through time ( BPTT ) ( Hochreiter , 1991 ; Bengio et al. , 1994 ) . Applying the STFT on the raw signal , on the other hand , is computationally efficient , as in practice it is implemented with the fast Fourier transform ( FFT ) whose computational complexity is O ( n log ( n ) ) . The aforementioned biological , representational and computational considerations provide compelling motivations for designing learning models for tasks where the complex-valued representation of the input and output data is more desirable than their real-counterpart . Recent work has provided building blocks for deep complex-valued neural networks ( Trabelsi et al. , 2017 ) . These building blocks have been shown , in many cases , to avoid numerical problems during training and , thereby , enable the use of complex-valued representations . These representations are well-suited for frequency domain signals , as they have the ability to explicitly encode frequency magnitude and phase components . This motivates us to design a new signal extraction mechanism operating in the frequency domain . In this work , our contributions are summarized as follows : 1 . We present a new signal separation mechanism implementing a local ensembling procedure . More precisely , a complex-valued convolutional version of Feature-wise Linear Modulation ( FiLM ) ( Perez et al. , 2018 ) is used to create multiple separated candidates for each of the signals we aim to retrieve from a mixture of inputs . A signal averaging operation on the candidates is then performed in order to increase the robustness of the signal to noise and interference . Before the averaging procedure , a form of dropout is implemented on the signal candidates in order to reduce the amount of interference and noise correlation existing between the different candidates . 2 . We propose and explore a new magnitude and phase-aware loss taking explicitly into account the magnitude and phase of signals . A key characteristic of our loss is that it is scale- and time-invariant . We test our proposed signal extraction mechanism in the audio source separation setting where we aim to retrieve distinct audio signals associated with each speaker in the input mix . Our experiments demonstrate the usefulness of our extraction method , and show its regularizing effect . 2 RELATED WORK . 2.1 RELATED WORK ON LEARNING REPRESENTATIONS IN THE FOURIER DOMAIN . Leveraging the Convolution Theorem to retrieve information has been done decades ago in the machine learning community using holographic reduced representations ( HRRs ) in the context of associative memories ( Plate , 1991 ; 1995 ) . HRRs enable one to store key-value data . Retrieval of a value in the data associated with a given key can be performed by convolving the whole data with the key or by applying an inner product between these two . By applying a fast Fourier transform ( FFT ) on the keys and the data , one can perform elementwise multiplication between the Fourier transforms and apply an inverse FFT to convert the result to the time domain . This would be equivalent to performing circular convolution between the key and the data in the time domain and has the advantage of being less expensive . Recently , Danihelka et al . ( 2016 ) have used associative memories to augment the capacity of LSTMs and to increase their robustness to noise and interference . For that , they applied independent permutations on the memory to create multiple copies of it . This enables one to obtain decorrelated noise in each of the permuted copies . A complex multiplication is then performed between the key and each of the copies . A signal averaging on the resulted multiplications eliminates the decorrelated noise in them and strengthens the signal-to-noise ratio ( SNR ) of the retrieved signal . Danihelka et al . ( 2016 ) , however , have not relied on FFTs in order to convert the temporal signals to the frequency domain . In fact , they assumed that complex-valued multiplication between the key and the data is itself enough to perform retrieval , and they have assumed that for each input representation the first half is real and the second one is imaginary . During this decade , interest in Fourier domain representations has started to grow in the machine learning community . Bruna et al . ( 2013 ) introduced a generalization of convolutions to graphs using the Graph Fourier Transform , which is defined as the multiplication of a graph signal by the eigenvector matrix of the graph Laplacian . However , the computation of the eigenvector matrix is expensive . Recently , methods that are computationally more efficient have been introduced in Defferrard et al . ( 2016 ) and Kipf & Welling ( 2016 ) to avoid an explicit use of the Graph Fourier basis . In the context of Convolutional Neural Networks ( CNNs ) , Rippel et al . ( 2015 ) introduced spectral pooling , which allows one to perform pooling in the frequency domain . This allows one to maintain the output spatial dimensionality , and thus the technique can retain significantly more information than other pooling approaches . Rippel et al . ( 2015 ) have also observed that the parametrization of the convolution filters in the Fourier domain induces faster convergence during training . Arjovsky et al . ( 2016 ) designed a recurrent neural network ( RNN ) where the transition hidden matrix is unitary . More precisely , the hidden transition matrix is constructed using the product of specific unitary transformations such as diagonal matrices , permutations , rotations , the Discrete Fourier Transform and its inverse . This allows one to preserve the norm of the hidden state , and as a consequence , mitigates the problem of vanishing and exploding gradients . Wolter & Yao ( 2018a ) designed an RNN where the input is converted to the frequency domain using a Short Time Fourier Transform ( STFT ) . The output is converted back to the time domain by applying an inverse STFT . Zhang et al . ( 2018 ) proposed a Fourier Recurrent Unit ( FRU ) where they showed that FRU has gradient lower and upper bounds independent of the temporal dimension . They have also demonstrated the great expressivity of the sparse Fourier basis from which the FRU draws its power . As we consider the task of speech separation as case study , we provide a related work section on both time domain and frequency domain speech separation methods in section 2.2 . 2.2 RELATED WORK ON TIME DOMAIN AND FREQUENCY DOMAIN SPEECH SEPARATION . Speech separation has been the subject of extensive study within the audio processing literature for a considerable amount of time . Recently , there has been growing interest in leveraging deep learning techniques ( Du et al. , 2014 ; Huang et al. , 2014 ; Hershey et al. , 2015 ; Gao et al. , 2018 ; Ephrat et al. , 2018 ) to tackle the speech separation problem . Hershey et al . ( 2015 ) proposed a deep clustering approach to speech separation . The basic idea is to learn high-dimensional embeddings of the mixture signals , that is later exploited to separate the speech targets with standard clustering techniques . A recent attempt to extend deep clustering led to the deep attractor network proposed by Chen et al . ( 2016 ) . Similarly to deep clustering , high dimensional embeddings are learned , but the network also creates the so-called “ attractors '' to better cluster time-frequency points dominated by different speakers . The aforementioned approaches estimate only the magnitude of the STFTs and reconstruct the time-domain signal with the Griffin-Lim algorithm ( Griffin & Lim , 1984 ) or other similar procedures ( Sturmel & Daudet , 2006 ) . Other papers have recently proposed to integrate the phase-information within a speech separation system without necessarily working in the complexvalued frequency domain . The work by Erdogan et al . ( 2015 ) , for instance , proposes to train a deep neural network with a phase-sensitive loss . Another noteworthy attempt has been described in Wang et al . ( 2018 ) , where the neural network still estimates the magnitude of the spectrum , but the timedomain speech signals are retrieved by directly integrating the Griffin-Lim reconstruction into the neural layers . Furthermore , methods reported in Wang et al . ( 2018 ) integrate the phase-information within a speech separation system by reconstructing the clean phase of each source starting from the estimated magnitude of each source and the mixture phase . This is fundamentally different from our proposed framework , as we provide an end-to-end solution to perform signal retrieval in the complex-valued frequency domain , and process both spectrogram magnitude and phase information rather than working only on magnitude representation with heuristic reconstruction of phase . Another attempt to estimate the clean phase is reported in Le Roux et al . ( 2019 ) where the clean phase of each speaker is estimated using discrete representation . This is also fundamentally different from our work as it considers a discrete representation of the phase for source separation and , in our case , we consider continuous representation of the complex-domain signal . Instead of explicitly integrating phase-information , other recent work perform speech separation in the time domain directly , as described in Venkataramani & Smaragdis ( 2018 ) . Likewise , the TasNet architecture proposed in Luo & Mesgarani ( 2017 ) and ConvTasNet ( Luo & Mesgarani , 2018 ) perform speech separation using the mixed time signal as input . Operating directly on the time-domain signal using the ConvTasNet architecture , which implements temporal convolutional networks ( TCN ) ( Bai et al. , 2018 ) , has led to state-of-the-art results in audio speech separation ( Luo & Mesgarani , 2018 ; Shi et al. , 2019 ) . The studies by ( Lee et al. , 2017 ; Hu & Wang , 2004 ; Huang et al. , 2014 ) are more related to our work as they address the speech separation problem by processing the complex-valued spectral input of the mixed speech . However , this was done without leveraging the recent advances in complex-valued deep learning .

Summary: This work researches the deep complex-valued neural networks. Specifically, it proposes a new signal extraction mechanism that operates in frequency domain and applies to address the speech separation issue. Also, a function is proposed to explicitly consider both the magnitude and phase information of a signal. Related work on learning representation in frequency domain and speech separation is well introduced. Theoretical analysis is conducted to show the motivation and connection to signal processing. The architecture of the deep neural networks is presented in details, with the elaboration of the complex mask generation. Experimental study is conducted on a benchmark dataset to compare the proposed complex networks with those using real-part values only to demonstrate the improvement.