Speech Separation Using an Asynchronous Fully Recurrent Convolutional Neural Network

1 Introduction . Speech separation aims to extract individual speeches from a mixture of speeches of multiple speakers . It is an important preprocessing step for speech recognition in noisy environment . Recent development of speech separation methods at the waveform level has aroused researchers ’ interest [ 21 , 22 , 20 ] , avoiding the traditional representation of STFT amplitude and phase used in so-called time-frequency ( TF ) domain methods [ 7 , 11 ] . Among these so-called time-domain methods , some presented mechanisms fuse information processed at various time scales , called multi-scale fusion ( MSF ) methods , such as in FurcaNeXt [ 40 ] or SuDoRM-RF [ 33 ] , and yield impressive results on the speech separation task . In this work we aim to explore if there exist even better MSF methods . Evidence from observations of sensory systems of mammals show them to utilize MSF in their processing . For instance , the visual system includes multiple processing stages ( from lower functional areas such as the lateral geniculate nucleus to higher functional areas such as the inferior temporal cortex ) , which process different scales of information [ 1 ] : the higher the stage , the coarser the scale . See Figure 1a for illustration . Similar mechanisms and areas have also been identified and located in the auditory system [ 1 ] . More importantly , physiological and anatomical studies have revealed abundant recurrent synaptic connections within the same stage ( also called lateral ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . connections ) and bottom-up/top-down synaptic connections between stages [ 3 ] . The intra-stage and inter-stage connections bring different scales of sensory information together and each stage performs information fusion . These connections fuse different scales of information more completely , and may lead to better results than existing MSF methods . However , Figure 1a merely reflects a purely static structure in the brain and does not show the dynamics of the sensory system . In biological systems , given a stimulus , the neurons along a sensory hierarchy do not fire simultaneously like shown in Figure 1b . For example , it was reported that the neural response initialized at a retinotopic position in anesthetized rat V1 propagated uniformly in all directions with a velocity of 50–70 mm/s , slowed down at the V1/V2 area border , after a short interval , spread in V2 , then reflected back in V1 [ 38 ] . In general , “ the speed of an action potential varies among neurons in a range from about 2 to 200 miles per hour ” [ 24 ] . The time at which a neuron starts to fire depends on a variety of factors including the neuron type , the stage at the sensory pathway , the number of the dendrites connected to it and the morphology of the neural fibers . This precludes the possibility of faithfully replicating the sensory system to obtain an excellent artificial neural network ( ANN ) . Nevertheless , the history of ANN development indicates that getting inspiration from the brain is enough to make great progress if task-specific techniques are combined . Inspired by the discovery of simple cells and complex cells in cat visual cortex [ 9 , 10 ] , a hierarchical model Neocognitron [ 6 ] was proposed and later developed into convolutional neural networks [ 15 ] by applying the backpropagation algorithm . We investigate empirically if there exists an asynchronous updating scheme for the structure shown in Figure 1a that provides improvement for speech separation performance . As the model has bottom-up , top-down and lateral connections as shown in Figure 1a , we call the model a fully recurrent convolutional neural network ( FRCNN ) . This name emphasizes the presence of both lateral and top-down recurrent connections in the model , distinguishing the model from an existing model [ 17 ] named recurrent convolutional neural network ( RCNN ) that has lateral recurrent connections only . The model with the synchronous updating scheme ( Figure 1b ) is called the synchronous FRCNN or S-FRCNN , which was studied for visual recognition [ 19 ] . We aim to propose an asynchronous FRCNN or A-FRCNN for speech separation . We notice that SuDoRM-RF [ 33 ] also has the three types of connections and we start from its framework to study different updating schemes of FRCNN . The architecture of our proposed A-FRCNN is illustrated in Figure 1c . The information first passes through stages one by one in the bottom-up direction , then fuses between adjacent stages in parallel , and finally fuses together with skip connections to the bottom stage . In the S-FRCNN , the information transmission from the bottom stage to any upper stage then back to the bottom stage is too fast : one step upward and one step downward ( Figure 1b ) . In contrast , in the A-FRCNN , the information starting from the bottom stage goes through more processing steps before it goes back to the bottom stage , which is advantageous for comprehensive MSF . Increasing the depth of a model is one of the keys for the success of deep learning . We will show the merit of A-FRCNN compared to S-FRCNN in experiments . 2 Related Work . 2.1 Speech Separation Methods . A typical speech separation method is to model different sources in the temporal-frequency ( TF ) domain . First , the short-time Fourier transform ( STFT ) calculates the TF representation of the mixed sound . Second , the subsequent process approximates the clean spectrogram of each source from the mixed spectrogram and uses the inverse STFT ( iSTFT ) to synthesize the source waveform , as in DPCL [ 7 ] , uPIT [ 12 ] etc . So-called time-domain methods were also proposed for speech separation , making use of non-STFT encoders for extracting meaningful representations out of the waveform ( in a bio-inspired fashion [ 4 ] , or fully learned over training [ 22 ] ) . DualPathRNN [ 20 ] extracts overlapped short sequences ( called chunks ) from the mixed speech signal and applies intra- and inter-chunk operations iteratively by using recurrent neural networks ( RNNs ) . It has achieved very good results at the cost of high computational complexity . The idea of intra-chunk and inter-chunk was adopted in recently proposed models such as Sepformer [ 31 ] , Sandglasset [ 13 ] and Gated DualPathRNN [ 23 ] . These models achieved even better speech separation results but the computational complexity is also higher . While RNNs were naturally used to perform speech separation given the sequential aspect of the input representation , they often require long training and inference time . Conv-TasNet [ 22 ] has been proposed to solve this problem , replacing RNN with Temporal Convolutional Networks , much faster to train . However , this model has limited MSF capability . Recently , several models with multiple branches have been proposed for speech separation , where different branches adopt different time resolutions for the processing of their respective inputs , then the outputs are fused with some rule or dedicated module . For instance , SuDoRF-RF [ 33 ] uses repetitive U-Nets [ 28 ] , obtaining good results with high efficiency . FurcaNeXt [ 40 ] , a variant of Conv-TasNet [ 22 ] , uses multiple branch learning methods to improve the performance of speech separation . MSGT-TasNet [ 41 ] uses Transformer [ 34 ] to capture features of different scales for speech separation . 2.2 Lateral and Top-Down Recurrent Connections . The lateral and top-down recurrent connections have been modeled by ANN researchers for a long time . In 1990s recurrent connections were introduced into the multi-layer Perceptrons [ 5 ] [ 27 ] , and in 2015 they were introduced into the CNN , resulting in the RCNN [ 17 ] [ 18 ] . In 2000s top-down connections were introduced into unsupervised deep learning models [ 8 ] [ 16 ] . A general framework with both lateral and top-down recurrent connections was proposed in 2016 [ 19 ] . If a hierarchical model has recurrent connections , the neurons can be updated in different orders . In [ 19 ] , only the conventional synchronous updating scheme was presented . However , no evidence has shown that this is how the neural system works , or that it outperforms asynchronous updating schemes on engineering tasks . In fact , in [ 18 ] , it was shown that an asynchronous updating scheme for RCNN outperformed the synchronous updating scheme on an image segmentation task . We here propose a novel asynchronous scheme for the FRCNN that achieved better results with fewer parameters than the synchronous scheme on speech separation . 3 Methods . 3.1 Overall pipeline . An algorithm dedicated to speech separation aims to extract individual speech signals of different speakers from a mixture . We denote the waveform of the mixture as x ∈ R1×T : x = C∑ i=1 si + σ ( 1 ) where si ∈ R1×T denotes the waveform of speaker i , σ ∈ R1×T denotes the noise signal , T denotes the number of samples of the signal , and C denotes the number of speakers . The task is to estimate si from x for all i . We use the same pipeline as ConvTasNet [ 22 ] , as shown in Figure 2 . It consists of an encoder , a separation network and a decoder . The encoder divides x into K overlapping segments xk ∈ R1×L and transforms each segment into a feature vector rk ∈ R1×N : rk = xkUe ( 2 ) where Ue ∈ RL×N is a weight matrix . The two steps can be realized by a trainable 1D convolution with kernel Ue and an appropriate stride . The separation network receives rk to estimate a mask Mi ∈ R1×N for speaker i . We apply a fully-connected layer with ReLU activation to the output of the separation network to produce Mi . The detailed structure of the separation network is introduced in Section 3.2 . The decoder reconstructs the waveform segment si , k = ( rk Mi ) UTd ( 3 ) where Ud ∈ RL×N is a weight matrix and UTd is the transpose of Ud , and stands for element-wise multiplication . The estimated waveform ŝi is obtained by summing K overlapping segments si , k . The two steps can be realized by a 1-D transposed convolution operation . 3.2 Separation Network . 3.2.1 Structure of FRCNN . We use the FRCNN as the separation network . It can be represented by a graph with nodes denoting stages and edges denoting connections . Figure 1a shows an example with S = 4 stages . In biological terms , every node corresponds to a set of neurons in a certain stage in the sensory pathway , e.g. , the inferior colliculus in the auditory pathway . In our model , every node corresponds to a convolutional layer . Different nodes process different scales of the input information . The higher the node , the coarser the information . There are three types of connections : bottom-up , top-down and lateral connections . Note that both bottom-up and top-down connections can be between adjacent stages and non-adjacent stages . In the latter case , the connections are called skip-connections . 3.2.2 Updating Schemes in the Micro-level . To run a recurrent neural network ( RNN ) with intricate connections , one needs to first determine the updating order of the neurons . This order determines the RNN unfolding or unrolling scheme . A commonly used approach is to update all neurons simultaneously . In the case of FRCNN as shown in Figure 1a , it corresponds to updating all stages synchronously . This scheme is depicted in Figure 1b [ 19 ] , and denoted by S-FRCNN . However , if the stages are allowed to be updated asynchronously , there will be a large number of possible unfolding schemes . For example , without considering the skip connections , we can update the stages one by one in the upward direction then update them one by one in the downward direction . In the present work , we propose an efficient updating scheme A-FRCNN , as shown in Figure 1c . In the proposed A-FRCNN , we first sequentially update the stages in the bottom-up direction , then update them simultaneously by fusing information from adjacent stages , and finally , fuse information from all stages to the bottom stage . This entire process is repeated several times ( Sec . 3.2.4 ) . The two types of fusions can be viewed as local and global fusions , respectively . As a stage represents a unique set of neurons as its biological counterpart , the connections between two stages ( e.g. , the vertical upward connection and the oblique upward connection between stage 3 and stage 4 ) should use the same operation and parameters . The A-FRCNN is adapted from the S-FRCNN in a step-by-step manner . 1 . Inspired by the structure of the U-Net [ 28 ] , we design a bottleneck structure at the bottom stage as shown in Figure 3a . All upper stages exchange information between different blocks through the bottom stage . This design increases the number of steps including down-sampling and up-sampling operations for processing the input coming at the highest resolution . This block is denoted by Control 1 . 2 . The block Control 1 has too many connections which make the model inefficient in both parameters and computation . The bottom-up skip-connections and top-down skipconnections are symmetric , and may be redundant . Therefore , we remove the bottom-up skip-connections , which results in the block shown in Figure 3b , denoted by Control 2 . 3 . It is too fast to fuse the information across non-adjacent stages through top-down skipconnections in the block Control 2 . One possible way to represent an increasing firing delay from widely separated units would be to fuse the information across adjacent stages first , then across non-adjacent stages . This change increases the shortest path from higher stages to the bottom stage . In addition , to save parameters and computation , we only keep the topdown skip-connections to the bottom stage and removed other top-down skip-connections . We also remove the vertical downward connections because the top-down stage-by-stage fusion has already been performed through the oblique downward connections . This is made possible by the delayed global fusion ; otherwise , the stages would become disconnected after removing the vertical downward connections . We then obtain the A-FRCNN ( Figure 1c ) . Note that the sequential fusion method in the third step is more biologically plausible than the synchronous fusion method since biological connections between non-adjacent stages are longer than those between adjacent stages , while signal transmission through connections is not instantaneous . The proposed A-FRCNN block is closely related to the U-Net [ 28 ] ( Figure 3c ) . To investigate the potential advantage of delayed global fusion , we add top-down skip-connections to the U-Net block and obtain a new block , denoted by U-Net-Delay ( Figure 3d ) .

Summary: This paper proposes a new speech separation architecture based on asynchrony fully recurrent CNN (A-FRCNN). The paper first describes FRCNN variants inspired by biological systems and discusses implementing such techniques for speech separation. Based on this discussion, the paper proposes a novel A-FRCNN speech separation technique. The experimental results show the effectiveness of the proposed method in two speech separation benchmarks (WHAM! and Librimix), achieving state-of-the-art performance compared with DPRNN. This new architecture can be applied to other similar problems, e.g., speech enhancement, and the contribution of this paper in terms of the technical novelty, experimental effectiveness, and potential wide applications is significant. My major concern about this paper is the clarity of the presentation. I could not fully understand the definition of the stage, bottom-up, and top-down operations until the actual speech separation discussions in Section 3.2. These concepts are not familiar, especially for speech separation researchers, and I recommend that the authors provide concrete examples of these concepts and elaborate the discussions.  Other comments: 1. Is it possible to use a transformer as a node instead of CNN? Recently, transformer-based processing becomes popular in speech separation. 2. Please discuss the effect of the reverberation. The real condition always has to deal with the reverberation. 3. Section 3.3 is a bit trivial, and it can be removed (or reduced). There are several important discussions in the appendix, and they can be put in the main document instead.