Speech Separation Using an Asynchronous Fully Recurrent Convolutional Neural Network

1 Introduction . Speech separation aims to extract individual speeches from a mixture of speeches of multiple speakers . It is an important preprocessing step for speech recognition in noisy environment . Recent development of speech separation methods at the waveform level has aroused researchers ’ interest [ 21 , 22 , 20 ] , avoiding the traditional representation of STFT amplitude and phase used in so-called time-frequency ( TF ) domain methods [ 7 , 11 ] . Among these so-called time-domain methods , some presented mechanisms fuse information processed at various time scales , called multi-scale fusion ( MSF ) methods , such as in FurcaNeXt [ 40 ] or SuDoRM-RF [ 33 ] , and yield impressive results on the speech separation task . In this work we aim to explore if there exist even better MSF methods . Evidence from observations of sensory systems of mammals show them to utilize MSF in their processing . For instance , the visual system includes multiple processing stages ( from lower functional areas such as the lateral geniculate nucleus to higher functional areas such as the inferior temporal cortex ) , which process different scales of information [ 1 ] : the higher the stage , the coarser the scale . See Figure 1a for illustration . Similar mechanisms and areas have also been identified and located in the auditory system [ 1 ] . More importantly , physiological and anatomical studies have revealed abundant recurrent synaptic connections within the same stage ( also called lateral ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . connections ) and bottom-up/top-down synaptic connections between stages [ 3 ] . The intra-stage and inter-stage connections bring different scales of sensory information together and each stage performs information fusion . These connections fuse different scales of information more completely , and may lead to better results than existing MSF methods . However , Figure 1a merely reflects a purely static structure in the brain and does not show the dynamics of the sensory system . In biological systems , given a stimulus , the neurons along a sensory hierarchy do not fire simultaneously like shown in Figure 1b . For example , it was reported that the neural response initialized at a retinotopic position in anesthetized rat V1 propagated uniformly in all directions with a velocity of 50–70 mm/s , slowed down at the V1/V2 area border , after a short interval , spread in V2 , then reflected back in V1 [ 38 ] . In general , “ the speed of an action potential varies among neurons in a range from about 2 to 200 miles per hour ” [ 24 ] . The time at which a neuron starts to fire depends on a variety of factors including the neuron type , the stage at the sensory pathway , the number of the dendrites connected to it and the morphology of the neural fibers . This precludes the possibility of faithfully replicating the sensory system to obtain an excellent artificial neural network ( ANN ) . Nevertheless , the history of ANN development indicates that getting inspiration from the brain is enough to make great progress if task-specific techniques are combined . Inspired by the discovery of simple cells and complex cells in cat visual cortex [ 9 , 10 ] , a hierarchical model Neocognitron [ 6 ] was proposed and later developed into convolutional neural networks [ 15 ] by applying the backpropagation algorithm . We investigate empirically if there exists an asynchronous updating scheme for the structure shown in Figure 1a that provides improvement for speech separation performance . As the model has bottom-up , top-down and lateral connections as shown in Figure 1a , we call the model a fully recurrent convolutional neural network ( FRCNN ) . This name emphasizes the presence of both lateral and top-down recurrent connections in the model , distinguishing the model from an existing model [ 17 ] named recurrent convolutional neural network ( RCNN ) that has lateral recurrent connections only . The model with the synchronous updating scheme ( Figure 1b ) is called the synchronous FRCNN or S-FRCNN , which was studied for visual recognition [ 19 ] . We aim to propose an asynchronous FRCNN or A-FRCNN for speech separation . We notice that SuDoRM-RF [ 33 ] also has the three types of connections and we start from its framework to study different updating schemes of FRCNN . The architecture of our proposed A-FRCNN is illustrated in Figure 1c . The information first passes through stages one by one in the bottom-up direction , then fuses between adjacent stages in parallel , and finally fuses together with skip connections to the bottom stage . In the S-FRCNN , the information transmission from the bottom stage to any upper stage then back to the bottom stage is too fast : one step upward and one step downward ( Figure 1b ) . In contrast , in the A-FRCNN , the information starting from the bottom stage goes through more processing steps before it goes back to the bottom stage , which is advantageous for comprehensive MSF . Increasing the depth of a model is one of the keys for the success of deep learning . We will show the merit of A-FRCNN compared to S-FRCNN in experiments . 2 Related Work . 2.1 Speech Separation Methods . A typical speech separation method is to model different sources in the temporal-frequency ( TF ) domain . First , the short-time Fourier transform ( STFT ) calculates the TF representation of the mixed sound . Second , the subsequent process approximates the clean spectrogram of each source from the mixed spectrogram and uses the inverse STFT ( iSTFT ) to synthesize the source waveform , as in DPCL [ 7 ] , uPIT [ 12 ] etc . So-called time-domain methods were also proposed for speech separation , making use of non-STFT encoders for extracting meaningful representations out of the waveform ( in a bio-inspired fashion [ 4 ] , or fully learned over training [ 22 ] ) . DualPathRNN [ 20 ] extracts overlapped short sequences ( called chunks ) from the mixed speech signal and applies intra- and inter-chunk operations iteratively by using recurrent neural networks ( RNNs ) . It has achieved very good results at the cost of high computational complexity . The idea of intra-chunk and inter-chunk was adopted in recently proposed models such as Sepformer [ 31 ] , Sandglasset [ 13 ] and Gated DualPathRNN [ 23 ] . These models achieved even better speech separation results but the computational complexity is also higher . While RNNs were naturally used to perform speech separation given the sequential aspect of the input representation , they often require long training and inference time . Conv-TasNet [ 22 ] has been proposed to solve this problem , replacing RNN with Temporal Convolutional Networks , much faster to train . However , this model has limited MSF capability . Recently , several models with multiple branches have been proposed for speech separation , where different branches adopt different time resolutions for the processing of their respective inputs , then the outputs are fused with some rule or dedicated module . For instance , SuDoRF-RF [ 33 ] uses repetitive U-Nets [ 28 ] , obtaining good results with high efficiency . FurcaNeXt [ 40 ] , a variant of Conv-TasNet [ 22 ] , uses multiple branch learning methods to improve the performance of speech separation . MSGT-TasNet [ 41 ] uses Transformer [ 34 ] to capture features of different scales for speech separation . 2.2 Lateral and Top-Down Recurrent Connections . The lateral and top-down recurrent connections have been modeled by ANN researchers for a long time . In 1990s recurrent connections were introduced into the multi-layer Perceptrons [ 5 ] [ 27 ] , and in 2015 they were introduced into the CNN , resulting in the RCNN [ 17 ] [ 18 ] . In 2000s top-down connections were introduced into unsupervised deep learning models [ 8 ] [ 16 ] . A general framework with both lateral and top-down recurrent connections was proposed in 2016 [ 19 ] . If a hierarchical model has recurrent connections , the neurons can be updated in different orders . In [ 19 ] , only the conventional synchronous updating scheme was presented . However , no evidence has shown that this is how the neural system works , or that it outperforms asynchronous updating schemes on engineering tasks . In fact , in [ 18 ] , it was shown that an asynchronous updating scheme for RCNN outperformed the synchronous updating scheme on an image segmentation task . We here propose a novel asynchronous scheme for the FRCNN that achieved better results with fewer parameters than the synchronous scheme on speech separation . 3 Methods . 3.1 Overall pipeline . An algorithm dedicated to speech separation aims to extract individual speech signals of different speakers from a mixture . We denote the waveform of the mixture as x ∈ R1×T : x = C∑ i=1 si + σ ( 1 ) where si ∈ R1×T denotes the waveform of speaker i , σ ∈ R1×T denotes the noise signal , T denotes the number of samples of the signal , and C denotes the number of speakers . The task is to estimate si from x for all i . We use the same pipeline as ConvTasNet [ 22 ] , as shown in Figure 2 . It consists of an encoder , a separation network and a decoder . The encoder divides x into K overlapping segments xk ∈ R1×L and transforms each segment into a feature vector rk ∈ R1×N : rk = xkUe ( 2 ) where Ue ∈ RL×N is a weight matrix . The two steps can be realized by a trainable 1D convolution with kernel Ue and an appropriate stride . The separation network receives rk to estimate a mask Mi ∈ R1×N for speaker i . We apply a fully-connected layer with ReLU activation to the output of the separation network to produce Mi . The detailed structure of the separation network is introduced in Section 3.2 . The decoder reconstructs the waveform segment si , k = ( rk Mi ) UTd ( 3 ) where Ud ∈ RL×N is a weight matrix and UTd is the transpose of Ud , and stands for element-wise multiplication . The estimated waveform ŝi is obtained by summing K overlapping segments si , k . The two steps can be realized by a 1-D transposed convolution operation . 3.2 Separation Network . 3.2.1 Structure of FRCNN . We use the FRCNN as the separation network . It can be represented by a graph with nodes denoting stages and edges denoting connections . Figure 1a shows an example with S = 4 stages . In biological terms , every node corresponds to a set of neurons in a certain stage in the sensory pathway , e.g. , the inferior colliculus in the auditory pathway . In our model , every node corresponds to a convolutional layer . Different nodes process different scales of the input information . The higher the node , the coarser the information . There are three types of connections : bottom-up , top-down and lateral connections . Note that both bottom-up and top-down connections can be between adjacent stages and non-adjacent stages . In the latter case , the connections are called skip-connections . 3.2.2 Updating Schemes in the Micro-level . To run a recurrent neural network ( RNN ) with intricate connections , one needs to first determine the updating order of the neurons . This order determines the RNN unfolding or unrolling scheme . A commonly used approach is to update all neurons simultaneously . In the case of FRCNN as shown in Figure 1a , it corresponds to updating all stages synchronously . This scheme is depicted in Figure 1b [ 19 ] , and denoted by S-FRCNN . However , if the stages are allowed to be updated asynchronously , there will be a large number of possible unfolding schemes . For example , without considering the skip connections , we can update the stages one by one in the upward direction then update them one by one in the downward direction . In the present work , we propose an efficient updating scheme A-FRCNN , as shown in Figure 1c . In the proposed A-FRCNN , we first sequentially update the stages in the bottom-up direction , then update them simultaneously by fusing information from adjacent stages , and finally , fuse information from all stages to the bottom stage . This entire process is repeated several times ( Sec . 3.2.4 ) . The two types of fusions can be viewed as local and global fusions , respectively . As a stage represents a unique set of neurons as its biological counterpart , the connections between two stages ( e.g. , the vertical upward connection and the oblique upward connection between stage 3 and stage 4 ) should use the same operation and parameters . The A-FRCNN is adapted from the S-FRCNN in a step-by-step manner . 1 . Inspired by the structure of the U-Net [ 28 ] , we design a bottleneck structure at the bottom stage as shown in Figure 3a . All upper stages exchange information between different blocks through the bottom stage . This design increases the number of steps including down-sampling and up-sampling operations for processing the input coming at the highest resolution . This block is denoted by Control 1 . 2 . The block Control 1 has too many connections which make the model inefficient in both parameters and computation . The bottom-up skip-connections and top-down skipconnections are symmetric , and may be redundant . Therefore , we remove the bottom-up skip-connections , which results in the block shown in Figure 3b , denoted by Control 2 . 3 . It is too fast to fuse the information across non-adjacent stages through top-down skipconnections in the block Control 2 . One possible way to represent an increasing firing delay from widely separated units would be to fuse the information across adjacent stages first , then across non-adjacent stages . This change increases the shortest path from higher stages to the bottom stage . In addition , to save parameters and computation , we only keep the topdown skip-connections to the bottom stage and removed other top-down skip-connections . We also remove the vertical downward connections because the top-down stage-by-stage fusion has already been performed through the oblique downward connections . This is made possible by the delayed global fusion ; otherwise , the stages would become disconnected after removing the vertical downward connections . We then obtain the A-FRCNN ( Figure 1c ) . Note that the sequential fusion method in the third step is more biologically plausible than the synchronous fusion method since biological connections between non-adjacent stages are longer than those between adjacent stages , while signal transmission through connections is not instantaneous . The proposed A-FRCNN block is closely related to the U-Net [ 28 ] ( Figure 3c ) . To investigate the potential advantage of delayed global fusion , we add top-down skip-connections to the U-Net block and obtain a new block , denoted by U-Net-Delay ( Figure 3d ) .

Summary: The paper introduce a novel method to separate mixture of  sounds. The paper introduce a new neural architecture called Fully Recurrent Convolutional Neural Network (FRCNN) which use neural network with lateral connections. The main variant of the proposed method is the A-FRCNN which use asynchronous updating. The main novel parts are: (1) novel bottleneck structure (2) removing some connection that seems to be redundant.