Speech Separation Using an Asynchronous Fully Recurrent Convolutional Neural Network
Authors: Xiaolin Hu
Kai Li
Weiyi Zhang
Yi Luo
Jean-Marie Lemercier
Timo Gerkmann
Venue: NeurIPS 2021 Poster (2021)
Keywords: speech separation
single-channel
time-domain
recurrent convolutional neural network
deep learning

Abstract: Recent advances in the design of neural network architectures, in particular those specialized in modeling sequences, have provided significant improvements in speech separation performance. In this work, we propose to use a bio-inspired architecture called Fully Recurrent Convolutional Neural Network (FRCNN) to solve the separation task. This model contains bottom-up, top-down and lateral connections to fuse information processed at various time-scales represented by stages. In contrast to the traditional approach updating stages in parallel, we propose to first update the stages one by one in the bottom-up direction, then fuse information from adjacent stages simultaneously and finally fuse information from all stages to the bottom stage together. Experiments showed that this  asynchronous updating scheme achieved significantly better results with much fewer parameters than the traditional synchronous updating scheme on speech separation.  In addition, the proposed model achieved competitive or better results with high efficiency as compared to other state-of-the-art approaches on two benchmark datasets. 

Speech Separation Using an Asynchronous Fully Recurrent Convolutional Neural Network
Xiaolin Hu' Kai Li', Weiyi Zhang', Yi Luo Jean-Marie Lemercier Timo Gerkmann'
DDpartmeet of Computer Science and Technology.
Tsinghua Laboratory of Brain and Intelligence (THBI). IDG/McGovern Institute of Brain Research Tsinghua University, Beijing, China "Department of Electrical Engineering, Columbia University, NY, USA
3 'Department of Informaties, University of Hamburg. Hamburg, Germany x1 xlhu@ts inghuu. edu. cn, (1k21, 4k22,wy-zhang199 @mails tsinghua. edu. en, y Iuo@columbia edu, lemercier@informatti uni wwwuuniiiiiitee haanhhhambr.coo de, timo gerkmann@uni aattaannuuui-hamburre .e
Abstract
Recent advances in the design of neural network architectures, in particular those specialized in modeling sequences, have provided significant improvements in speech separation performance. In this work, we propose to use a bio-inspired architecture called Fully Recurrent Convolutional Neural Network (FRCNN) solve the separation task. This model contains bottom-up. top-down and lateral connections to fuse information processed various time-scales represented by stages. In contrast to the traditional approach updating stages in parallel, we propose to first update the stages one by one in the bottom-up direction, then fuse information from adjacent stages simultaneously and finally fuse information from all stages to the bottom stage together. Experiments showed that this asynchronous updating scheme achieved significantly better results with much fewer parameters than the traditional synchronous updating scheme. In addition, the proposed model achieved good balance between speech separation accuracy and computational efficiency as compared to other state-of-the-ant models on three benchmark datasets.
Introduction
Speech separation aims to extract individual speeches from a mixture of speeches of multiple speakers. It is important preprocessing step for speech recognition in noisy environment, Recent development of speech separation methods at the waveform level has aroused researchers' interest 21 (2 20). avoiding the traditional representation of STFT amplitude and phase used in so-called time-rreueenyy (TF) domain methods [7 [1] Among these so-called time-domain methods, some presented mechanisms fuse information processed at various time scales, called multi-scale fusion (MSF) methods, such as in FurcaNeXt [40 or SuDoRM-RF 33 and yield impressive results on the speech separation task. In this work we aim to explore if there exist even better MSF methods.
Evidence from observations of sensory systems mammals show them to utilize MSF in their processing, For instance, the visual system includes multiple processing stages (from lower functional areas such as the lateral geniculate nucleus to higher functional areas such as the inferior temporal cortex), which process different scales of information [I] the higher the stage, the coarser the scale. See Figure T for illustration. Similar mechanisms and areas have also been identified and located in the auditory system [1]. More importantly, physiological and anatomical studies have revealed abundant recurrent synaptic connections within the same stage (also called lateral
"Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Bottom-up
Tup-down
Lateral
ISI Conv
FRONN
(b) S-FRCNN
A-FRONN
(d) Concatenation operation
Figure 1: The structure of FRCNN and typical updating schemes. The number of stages S 4. (a) The structure of the FRCNN. Every node denotes stage, corresponding group of neurons in functional area in the sensory pathway (c.g., the inferior colliculus the auditory pathway). Red, bluc and orange arrows denote bottom-up. top-down and lateral connections, respectively. Both bottom-up and top-down connections can be made between adjacent stages and non con-adjacent stages. (b) Synchronous updating scheme in one block [19). (c) The proposed asynchronous updating scheme in one block. The dashed box in each subfigure indicates the basic building block for constructing complete RNN (see Figure 4. (d) Multi-scale information fusion for an example stage receiving three types inputs.
connections) and bottom--pltop-down synaptic connections between stages 3 inter-stage connections bring different scales of sensory information together and each stage performs information fusion. These connections fuse different scales of information more completely, and may lead to better results than existing MSF methods.
The intra-stage and
However, Figure D merely reflects a purely static structure in the brain and does not show the dynamics of the sensory system. In biological systems, given stimulus, the neurons along sensory hierarchy do not fire simultaneously like shown in Figure Dp. For example, was reported that the neural response initialized at a retinotopic position anesthetized rat V1 propagated uniformly in all directions with a velocity of 50-70 mm/s, slowed down at the V1/V2 area border, after a short interval, spread in V2, then reflected back in VI [38. In general, "the speed of action potential varies among neurons in a range from about 2 to 200 miles per hour" [24]. The time at which a neuron starts to fire depends on variety of factors including the neuron type, the stage at the sensory pathway, the number of the dendrites connected to it and the morphology of the neural fibers. This precludes the possibility faithfully replicating the sensory system obtain an excellent artificial neural network (ANN). Nevertheless, the history of ANN development indicates that getting inspiration from the brain enough to make great progress if task-specific techniques are combined. Inspired by the discovery of simple cells and complex cells in cat visual cortex (9 [IO, a hierarchical model Neocognitron (6] was proposed and later developed into convolutional neural networks [1] by applying the backpropagation algorithm. We investigate empirically if there exists an asynchronous updating scheme for the structure shown Figure] Ta that provides improvement for speech separation performance.
As the model has bottom-up, top-down and lateral connections as shown in Figure Da we call the model a fully recurrent convolutional neural network (FRCNN). This name emphasizes the presence of both lateral and top-down recurrent connections in the model, distinguishing the model from an existing model [17] named recurrent convolutional neural network (RCNN) that has lateral recurrent connections only. The model with the synchronous updating scheme (Figure TT) is called the synchronous FRCNN or S-FRCNN, which was studied for visual recognition [9 We aim to propose an asynchronous FRCNN or A-FRCNN for speech separation. We notice that M-R 33 also has the three types of connections and we start from its framework to study different updating schemes of FRCNN.
The architecture of our proposed A-FRCNN is illustrated in Figure .e. The information first passes through stages one by one in the bottom-up direction, then fuses between adjacent stages in parallel, and finally fuses together with skip connections to the bottom stage. In the S-FRCNN, the information transmission from the bottom stage to any upper stage then back to the bottom stage too fast: one step upward and one step downward (Figure [p). In contrast, in the A-FRCNN, the information starting from the bottom stage goes through more processing steps before it goes back to the bottom stage, which is advantageous for comprehensive MSF Increasing the depth of model is one of the keys for the success of deep learning. We will show the merit of A-FRCNN compared to S-FRCNN in experiments.

---

2 Related Work
2.1 Speech Separation Methods
typical speech separation method is to model different sources in the domain. First, the short-time Fourier transform (STFT) calculates the TF representation of the mixed sound. Second, the subsequent process approximates the clean spectrogram of each source from the mixed spectrogram and uses the inverse STFT (iSTFT) to synthesize the source waveform, in DPCL 77. uPIT 12 etc.
(TF)
So-called time-domain methods were also proposed for speech separation, making use of non-STFT encoders for extracting meaningful representations out of the waveform (in bio-inspired fashion 4]. fully learned
training (22)). DualPathRNN (20] extracts overlapped short sequences (called chunks) from the mixed speech signal and applies intra- and inter-chunk operations iteratively by using recurrent neural networks (RNNs). It has achieved very good results at the cost of high computational complexity. The idea of intra-chunk and inter-chunk was adopted in recently proposed models such Sepformer [31], Sandglasset [13 and Gated DualPathRNN [23] These models achieved even better speech separation results but the computational complexity is also higher.
While RNNs input representation, they often require long training and inference time. Conv-TasNet [22] has been proposed to solve this problem, replacing RNN with Temporal Convolutional Networks, much faster to train. However, this model has limited MSF capability. Recently, several models with multiple branches have been proposed for speech separation, where different branches adopt different time resolutions for the processing of their respective inputs, then the outputs are fused with some rule or dedicated module. For instance, SuDoRF-RE RF [33] uses repetitive U-Nets (28), obtaining good results with high efficiency. FurcaNeXt [40] a variant of Cony-TasNet [22], uses multiple branch learning methods to improve the performance of speech separation. MSGT-TasNet 41 uses Transformer [34
naturally used to perform speech separation given the sequential aspect of the
to capture features of different scales for speech separation.
2.2 Lateral and Top-Down Recurrent Connections
The lateral and top-down recurrent connections have been modeled by ANN researchers for a long time. In 1990s recurrent connections were introduced into the multi-layer Perceptrons [5] [27]. and in 2015 they were introduced into the CNN, resulting in the RCNN [17 [8]. In 2000s top-down
connections were introduced into unsupervised deep learning models 8 [66. A general framework with both lateral and top-down recurrent connections was proposed in 2016 [9] If hierarchical model has recurrent connections, the neurons can be updated in different orders. In [19] only the conventional synchronous updating scheme was presented. However, no evidence has shown that this is how the neural system works, or that it outperforms asynchronous updating schemes on engineering tasks. fact, in [88, was shown that an asynchronous updating scheme for RCNN outperformed the synchronous updating scheme novel asynchronous scheme for the FRCNN that achieved better results with fewer parameters than the synchronous scheme speech separation.
image segmentation task. We here propose
3 Methods
3.1
Overall pipeline
An algorithm dedicated to speech separation aims to extract individual speech signals of different speakers from a mixture. We denote the waveform of the mixture as x E RIXT,
S +o
where S; RT denotes the waveform of speaker a E RIXT denotes the noise signal, denotes the number of samples of the signal, and C denotes the number of speakers. The task is to estimate
8; from x for all i.
3

---

We use the same pipeline as Conv- TasNet [22], as shown in Figure 2 It consists of an encoder, separa- tion network and decoder. The en- coder divides into K overlapping segments X5 E R and transforms each segment into feature vector FR RIXN;
Separation Network
Encoder
Decoder
Figure 2: The overall pipeline for speech separation.
Xx U
(2)
where U. E RLXN is weight matrix. The two steps can be realized by trainable ID convolution with kernel Ug and an appropriate stride.
The separation network receives F₃ to estimate a mask M, E R x for speaker i. We apply fully-connected layer with ReLU activation to the output the separation network to produce M,. The detailed structure of the separation network is introduced in Section 3.2
The decoder reconstructs the waveform segment
5i,k (Tx O M.)U
where Ua E RI is a weight matrix and T is the transpose of Ua. and @ stands for element-wise multiplication. The estimated waveform &; is obtained by summing K overlapping segments Si,A- The two steps can be realized by a 1-D transposed convolution operation.
3.2
Separation Network
3.2.1 Structure of FRCNN
We the FRCNN the separation network. It can be represented by graph with nodes denoting stages and edges denoting connections. Figure shows an example with S 4 stages. In biological terms, every node corresponds inferior colliculus in the auditory pathway. In our model, every node corresponds to convolutional layer. Different nodes process different scales of the input information. The higher the node, the coarser the information. There are three types of connections: bottom-up. top-down and lateral connections. Note that both bottom-up and top-down connections can be between adjacent stages and non odj-anjjaeent stages. In the latter case, the connections are called skip-commections.
set of neurons in a certain stage in the sensory pathway, e.g., the
3.2.2 Updating Schemes in the Micro-level
To run a recurrent neural network (RNN) with intricate connections, one needs to first determine the updating order of the neurons. This order determines the RNN unfolding or unrolling scheme. A commonly used approach is to update all neurons simultaneously. In the case of FRCNN as shown in Figure p. it corresponds to updating all stages synchronously. This scheme is depicted in Figure Ip [1), and denoted by S-FRCNN. However, if the stages are allowed to be updated asynchronously. there will be large number of possible unfolding schemes. For example, without considering the skip connections, we can update the stages one by one in the upward direction then update them one by one in the downward direction. In the present work, propose efficient updating scheme A-FRCNN, as shown in Figure [Tc.
In the proposed A-FRCNN, we first sequentially update the stages in the bottom-up direction, then update them simultaneously by fusing information from adjacent stages, and finally, fuse information from all stages to the bottom stage. This entire process is repeated several times (Sec. 3.2.4). The two types of fusions can be viewed as local and global fusions, respectively. As a stage represents unique set of neurons as its biological counterpart, the connections between two stages (e.g., the vertical upward connection and the oblique upward connection between stage 3 and stage 4) should use the same operation and parameters.
The A- A-FRCNN is adapted from the S-FRCNN in a step-by-step manner.
1. Inspired by the structure of the U-Net 28 we design bottleneck structure at the bottom stage as shown Figure 3h. All upper stages exchange information between different blocks through the bottom stage. This design increases the number of steps including

---

Bottom
Top-down
Lateral
Coutrol
(b) Control
(c) U-Net
(d) U-Net-Delay
Figure 3: Various asynchronous updating schemes the micro-level.
down-sampling and up-sampling operations for processing the input coming at the highest resolution. This block is denoted by Control 1.
2. The block Control has too many connections which make the model inefficient in both parameters and computation. The bottom-up skip-connections and top-down skip- connections are symmetric, and may redundant. Therefore, we remove the bottom-up skip-connections, which results in the block shown in Figure 3p, denoted by Control 2.
3. It is fast to fuse the information across non-adjacent stages through top-down skip- connections in the block Control 2. One possible way to represent an increasing firing delay from widely separated units would be to fuse the information across adjacent stages first, then across non-adjacent stages. This change increases the shortest path from higher stages the bottom stage. In addition, to save parameters and computation, down skip-connections to the bottom stage and removed other top-down coiecconneciios. We also remove the vertical downward connections because the top-down stage-by-stagee fusion has already been performed through the oblique downward connections. This is made possible by the delayed global fusion; otherwise, the stages would become disconnected after removing the vertical downward connections. We then obtain the A-FRCNN (Figure mT))
only keep the top-
Note that the sequential fusion method in the third step is more biologically plausible than the synchronous fusion method since biological connections between non-adjacent stages are longer than those between adjacent stages, while signal transmission through connections is not instantancous.
The proposed A-FRCNN block closely related the U-Net 28 (Figure P.) To investigate the potential advantage of delayed global fusion, we add top-down skip-connections to the U-Net block and obtain new block, denoted by U-Net-Delay (Figure 3d).
3.2.3 Multi-scale Information Fusion inside Blocks
The blocks depicted in Figures and are RNN blocks, and the nodes in the same horizontal row represent the same stage (or in biological terms, the same set of neurons in a sensory area) but at different time. this study we use C feature maps for every stage. Multi-scale information fusion is performed at the input of every stage. First the C' feature maps from each of the K inputs are concatenated the channel dimension, resulting KC feature maps. is then used reduce the number of feature maps to C'. Figure D illustrates this process. This concatenation method was used by default in our experiments. One can also sum up the K inputs to obtain C' feature maps.
x convolutional layer
3.2.4 Unfolding Methods in the Macro-level
Figures and show single blocks of the entire unfolding schemes. An entire unfolding scheme usually consists of multiple such blocks with tied weights. there are blocks total, we say "FRCNN is unfolded for B time steps". At the macro-level, the FRCNN be unfolded by simply repeating these blocks along time such that the output of one block the input of the next block.
To further fuse the multi-scale information, we add 1 convolution between two consecutive blocks (Figure 4h). This method is formulated as follows:
R(I 1) f(p R()))),
where f(.) denotes a block shown in Figures and 3 R(I) denotes the output of block at time step and to denotes x convolution. This is called the direct connection (DC) method.
5

---

Another idea is to integrate the input of the model with the output of every block via fea- ture map concatenation
R((
Conv
R(H1)
Conv
Block
Block
summation before sending to the next block. This rule was used in constructing the recurrent CNN in previous study [17]. Again, we add 1 x convolution to further fuse information (Figure4p) Formally,
(a) Direct connection
Cony
R((+1)
Conv
Block
Block
Concalenation sammation connection
R(t + = (()()(())))))))))) @ r))
(5)
Figure 4: Macro-level unfolding schemes of the FR-
CNN. Every blue box corresponds dashed box Figures The pink boxes in a model denote 1 convolutions with shared weights.
where r denotes the input feature maps and + denotes concatenation or summation of two sets of feature maps. This is called the concatenation connection (CC) or summation connection (SC) depending on which feature map integration method used.
For single-input inn-iiiiiuu--------eeeee- single-output output blocks, i.e, A-FRCNN and the blocks shown in Figure 3 we directly use the unfolding methods described above. For the multi-input-mutti-uunge block, i.e., S-FRCNN, we apply these unfolding methods for each input-output pair corresponding to the same stage. should be noted that Figure[Tp only illustrate the intermediate blocks of S-FRCNN unfolding scheme. In the beginning of unfolding we use downsampling to obtain different scales feature maps, and in the end of unfolding use up-sampling to fuse different scales of feature maps.
3.3 Training Method
We use the standard BP algorithm to train the model. The object is to maximize the scale-invariant signal-to-noise ratio (SI-SNR) 14. See Supplementary Materials for details. SI-SNR is also metric to evaluate the performance of speech separation methods.
4
Experimental Settings
4.1 Dataset
Libri2Mix (2). This dataset was constructed using train-100, train-360, dev, and test set in the
LibriSpeech dataset (25). Random extracts were selected for different speakers and mixed with uniformly sampled Loudness Units relative Full Scale (LUFS) [29] between -25 and -33 dB. Random noise samples were added, with loudness uniformly sampled between -38 and -30 LUFS. We used train-100 as the training which has 58 hours. We used the same test set to compare different methods.
WSJ0-2Mix (7). This dataset contains 30-hour training set, 10-hour validation and 5-hour test set. was generated by combining the speech signals of different speakers in the Wall Street Journal corpus. The speech signals were randomly selected and mixed with a Signal-to-Noise Ratio uniformy sampled between -5dB and 5dB.
WHAM! (37. WHAM! added noise to WSJO-2Mix, which was recorded in scenes such as cafes, restaurants and bars. The speeches were mixed with noise with a SNR uniformly sampled between -6dB and 3dB. Due the existence of noise, WHAM! is more challenging for speech separation than WSJ0-2Mix.
4.2
Implementation Details
The encoder and decoder in the speech separation pipeline were respectively a 1-D convolutional layer and a 1-D transposed convolutional layer. We set their kernel size to 21, stride to 10, and channel number to 512, i.e., L = 21 and N 512. Unless otherwise specified, the number of stages was set to 5, the number of channels C' in every stage was set to 512 and the SC method was used for unfolding in the macro-level. Whenever C N, 1 x convolutional layer was added between the encoder and the separation network.
We designed two methods to realize the connections shown in Figures I and

---

Method A: The bottom-u up and top-down connections were realized by convolution (kernel size 5 and stride and the PixelShuffle technique 30] (kernel size 5), respectively. The PixelShuffle technique was shown to be better than other upsampling techniques for image super-resolution reconstruction. The lateral connections were realized by xx convolution.
Method B: The bottom-up connections were realized by the convolution operation with kernel size 5 and appropriate strides. For example, one operation was used for 2x down- sampling and two consecutive operations were used for 4x down-sampling, and so on. The top-down connections were realized by interpolation. The lateral connections were realized by simply copying the feature maps.
All convolutions were depthwise separable convolutions. Method A all connections had trainable parameters, resembling plastic synapses in biological systems. In Method B, only the bottom-up connections had parameters, and it is therefore less biologically plausible. However, Method B is more parameter efficient and computing efficient.
We trained models for 200 epochs on 3-second utterances for Libri2Mix and 4-second utterances for WHAM! and WSJO-2Mix with 8K Hz sampling rate. Batch size was set to 8. The initial learning rate of Adam optimizer was 10-3, and decayed to 1/3 of the previous rate every 40 epochs. During training, gradient clipping with maximum (2-norm of 5 was used. For each architecture, we picked the best model based on its results on the validation set, then performed testing.
All experiments were conducted on server with Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz and GeForce RTX 1080 Ti 11G x8. The Pytorch implementation of the models is publicly availabld is based on the code of SuDoRM-RF This project is MIT Licensed.
Results
We used the scale-invariant signal-to-noise ratio improvement (SI-SNRi) [1] and signal-to-distortion ratio improvement (SDRi) (55 as the evaluation metrics to measure the speech separation accuracies of models. Their definitions are found in Supplementary Materials. In Secs. 3.T 552) we report the
msa of these metrics over 5 models trained from different random seeds. We report inference time on CPU, indicated by "Time" in tables throughout the paper. It was calculated by processing a four-second audio on CPU then averaged over 1000 trials.
5.1 Comparison of Micro-level Updating Schemes
We compared the performances of the micro-leved updating schemes on Libri2Mix. The results are presented in Table [ By using either Method or Method for implementing the connections (Sec. (.2). the proposed A-FRCNN scheme performed better than the S-FRCNN scheme and the two control schemes. By comparing the results of S-FRCNN and Control 1, we see that the bottleneck design brought large improvement on the SI-SNRi and SDRi metrics. Control and Control achieved similar SI-SNRi and SDRi values indicating that the bottom-up skip-connections are indeed redundant. The delayed global fusion further improved the results of Control 2. From S-FRCNN to Controls and and A-FRCNN, the number of connections decreased; as a result, the amount of parameters and inference time also decreased.
For implementing the connections in the four models in Table I Method obtained similar or even better results with fewer parameters and less inference time than Method A. Therefore, we adopted Method B in later experiments. All results reported what follows were obtained with Method
The S-FRCNN had much more parameters than the A-FRCNN. We then reduced C' to 412 and obtained a new model, S-FRCNN (light), which had similar number of parameters the A-FRCNN. Its results were even worse than the original S-FRCNN (T.ble(T). suggesting that the poor performance of the S-FRCNN was not due overfitting caused by large amount of parameters. We removed all Skip-connection from S-FRCNN and obtained model called S-FRCNN (no-skip). A CNN model with similar architecture has been used in face parsing 42 39]. but different blocks do share
http:://eellkaaa.ccmrrr/fe/////
https://github. http:///gichbbbooooo////ss/////s http:///aaaaaaaaooooooooooaaa/aa http:////ataateeeuooo.mm///////a
improved_sudormrff.p

---

Table 1: Comparison of different FRCNN updating schemes the Libri2Mix test set. The values are presented as "Method Method B". Each model was unfolded for time steps.
Model
S1-SNRi
SDRi
Params (M)
Time (s)
S-FRCNN
12.6 / 12. 10.05
13.0 / 12.5
9.8/ 9.6
1.35 / 1.12
Control
15.020.06 / 14.640.04
15.5=2.06 / 15.00+..03
8.3 / 8.0
1.41 / 0.96
Control
15.1 / 14.640.04
15.4_0.02
/ 15.0
6.6/6.4
1.16/ 0.81
A-FRCNN (ours)
10.2 15.0
15.7 10.04
6.2/6.1
1.07 0.72
S-FRCNN (light)
-/ 1/118_
-/ -2.2.200.2
-/6.4
-/9.97
S-FRCNN (no-skip)
/ 11.44
-/ +0.02
6.4
-/0.51
Table Comparison with U-Net and U-Net-Delay on the Libri2Mix test set. Each model was unfolded for time steps.
Model
S1-SNRi
SDRi
Params (M)
Time (s)
U-Net
11.422.01
11.810.01
4.0
0.39
U-Net-Delay
13.2+0.05
13.640.06
5.1
0.60
U-Net (large)
11.5
11.940.01
6.1
0.69
U-Net-Dede Deely (large)
12.1
12.5
6.4
0.88
A-FRCNN (ours)
15.5
15.9
6.1
0.72
weights. From Table]] 1 S-FRCNN.
it is seen that S-FRCNN (no-skip) achieved worse results than the original
The U-Net structure did not yield good results (Table 2.
With delayed global fusion, the U-Net-Delay block yielded better results. For fair comparison with the A-FRCNN, we changed the number of channels C from 512 to 684 and 580 in U-Net and U-Net-Delay blocks, respectively, and obtained the U-Net (large) and U-Net-Delay (large) blocks. The two blocks had similar amount of parameters to the A-FRCNN. With more parameters, the U-Net-Delay (large) tended to overfit the training data and yielded worse test results than the original U-Net-Delay, Even in this case, U-Net-Delay (large) yielded better results than the U-Net (large), verifying the effectiveness of the delayed global fusion. Nevertheless, they both yielded much lower SI-SNRi and SDRi values than the A-FRCNN.
5.2 Comparison with Existing Models
We compared A-FRCNN with some popular models for speech separation. Some models work in the freeuree domain: DPCL++ (7]. uPIT-BLSTM-ST [12] and Chimera++ [36]. Some
models work the time-domain: BLSTM- TasNet (21), Conv-TasNet (22], Two-Step TDCN [32],
MSGT-TasNet [41) SuDoRMM-RF 33, DualPathRNN 20) Sepformer 31 and Gated DualPathRNN
233. SuDoRM-RF has four variants which are labeled by appending 0.25x, 0.5x, 1.0x and 2.5x to
the end of the name, indicating the variants consist of 4, 8, 16 and 40 blocks, respectively. None of these models have reported results on all of the three datasets and we run some of them to obtain missing results by using the Asteroid toolkit (26). See Table Sepformer and Gated DualPathRNN have achieved the best results on WSJO-2Mis but we could not afford the computational resource to obtain the missing results other datasets the two models require single GPU memory larger than 20 GB.
We tested three variants of A-FRCNN by unfolding for 4, 8, 16 times in the macro-level. We also tested variants which the concatenation in Figure was replaced with summation, and their names have "sum" attached to the end in Table 3
Separation accuracy. On Libri2Mix we had the following observations. First, among existing models DualPathRNN achieved the best results. Second, with more unfolding times in the macro- level, A-FRCNN achieved better results the cost of increasing inference time. Third, A-FRCNN-16 achieved better results than DualPathRNN.
Some audio examples from Libri2Mix separated by different models including DualPathRNN and A-FRCNN-16 16 are provided in Supplementary Materials. In most examples we found that separated speeches by A-FRCNN-16 sounded clearer than those by other methods.
On WHAM! and WSJ0-2Mix, DualPathRNN, Gated DualPathRNN and Sepformer obtained higher SI-SNRi and SDRi values than A-FRCNN-16, but will show below that this is at the cost of
8

---

Table 3: Performance of different models. The SI-SNRi and SDRi values marked with *** indicate that they
were not reported original papers but obtained by using Asteroid toolkit (26] The toolkit is licensed under the MIT licence.
Libri2Mix
WHAM!
WSJO-2MiX
Model
Params (M)
Time (s)
SI-SNRi
SDRi
SI-SNRi
SDRi
S1-SNRi
SDRi
DPCL+
5.9
6.6*
10.8
11.2
13.6
0.37
uPIT-BLSTM-ST
7.6"
8.2"
9.8
10.0
92.7
0.77
Chimera++
6.3"
7.0"
10.0
11.5
11.8
32.9
0.66
BLSTM-TasNet
7.9"
8.7"
13.2
13.6
23.6
2.90
Conv-TasNet
12.2
12.7
12.7
15.3
15.6
5.6
0.39
Two-Step TDCN
12.0
12.5
16.1
8.6
1.85
MSGT-TasNNtt (light)
12.3
16.8
17.1
37,4
3.37
MSGT-TasNer (dense)
13.1
17.0
66.8
5.56
SuDoRM-RE 0.25x
10.8*
11.3*
10.4
10.8
13.4
13.6
0.8
0.24
SuDoRM-RF 0.5x
12.2
12.6
11.8*
12.2
15.4
15.6
1.4
0.33
SuDoRM-RF 1.0x
13.5"
14.0"
12.9
13.3"
17.1
17.3
2.7
0.53
SuDoRM-RF 2.5x
14.0"
14.4"
13.7"
14.1"
17.4
17.6"
6.4
1.15
DualPathRNN
14.1"
14.6
13.7"
14.1"
18.8
19.0
4.66
Gated DualPathRNN Sepformer
15.2
20.1
7.5
7.35
20.4
20.5
26.0
5.22
A-FRCNN-4
13.1
12.3
15.6
15.8
0.33
10.01
10:02
20.02
10.02
20.01
A-FRCNN-8
15.5
15.9
13.4
13.8
17.1
17.3
6.1
0.72
10.04
10.04
20.02
20.02
10.03
10.02
A-FRCNN-16
16.7
17.2
14.5
14.8
18.3
18.6
6.1
1.51
1:0.03
#0.02
20003
+0.02
A-FRCNN-4 (sum)
13.5
12.6
14.9
1.7
0.29
10.01
1:0.02
:10.002
2:0.00
1.0.02
1:000
A-FRCNN-8 8 (sum)
15.0
15,4
13.0
13.4
16.7
17.1
1.7
0.52
#0.01
+.002
=00002
=0.02
20000
#0.02
A-FRCNN-16 (sum)
16.2
16.7
14.0
14.6
17.9
18.3
1.7
0.98
significantly more computational expense. In addition, Sepformer had significantly more parameters than A-FRCNN-16.
Efficiency. We have seen that the three models based on intra- and inter-chunk framework, Du- alPathRNN, Gated DualPathRNN and Sepformer obtained SOTA SI-SNRi and SDRi values WHAM! and WSJ0-2Mix datasets. We then investigated the computing efficiency of these models.
that DualPathRNN was 3x slower than A-FRCNN-LLL 16 during inference. Second, training DualPathRNN was also very slow. Training DualPathRNN on Libri2Mix took 142 hours on 8 GPUs while training A-FRCNN-16 took 39 hours (see Supplementary Materials for the training time of typical models). Third, Gated DualPathRNN and Sepformer were even slower than DualPathRNN.
First, from Table 3
We list more computational complexity metrics of Gated DualPathRNN, Sepformer and A-FRCNN- 16 in Table including the forward and backward GPU memory/time and GFlops of the forward pass by processing four-second audio. We see that Gated DualPathRNN took about and 2x more GPU memory than A-FRCNN-16 for forward and backward passes, respectively; Gated DualPathRNN took about 7x and 3x more GPU time than A-FRCNN- 16 for forward and backward passes, respectively. Sepformer also took significantly more GPU memory and GPU time than A-FRCNN-16 during training. addition, we found that the GFlops of A-FRCNN- 16 could be significantly reduced when the concatenation
replaced with summation for information fusion, while SI-SNRi value dropped only slightly. In fact, A-FRCNN-16 (sum) took about 6x less GFlops than Gated DualPathRNN and Sepformer. In terms of these efficiency metries, SuDoRM-RF 2.5x performed as well as A-FRCNN-16 (sum).
Taken together, compared with other SOTA models, A-FRCNN achieved good balance between speech separation aceuracy and computational efficiency, and can obtain good separation results with limited computing resource.

---

Table 4: Computational complexity metrics of different SOTA models.
Model
F/B GPU Memory (GB)
F/B GPU Time (ms)
GFlops
Gated DualPathRNN
0.43/7.78
503. 81888/
125.28
Sepformer
0.62/11.3
311.8/702.1
145.58
SuDoRM-RF 2.5x
0.10/4.85
77.2/210.3
19.8
A-FRCNN-16 A-FRCNN-16 (sum)
0.14/4.07
74.4/263.5 70.2/223.3
123.3
0.13/3.54
22.8
5.3 Ablation Study
The experiments
the Libri2Mix dataset. We studied the influence of the number of stages S (Figure [] by fixing the unfolding method to SC. The results was the best with S 5 (TableB). We then compared the results with different unfolding methods (Figure 44 by fixing the SC method was the best (Table 66. In the two experiments the A-FRCNN was unfolded for 8 time and the SC method in all other experiments. Only the results with and
and found that
steps. We therefore used this setting are average results over 5 different runs in Tables 5 and 66 and we did not train models
with other settings for multiple times considering the small standard deviations in previous tables.
Table 5: Test results of the A-FRCNN with differ- ent number of stages S.
Table Test results of A-FRCNN with different unfold-
ing methods the macro-level.
S
S1-SNRi
SDRi
Params
Method
SI-SNRi
SDRi
Params
Time (s)
13.8
14.5
4.0M
DC
12.8
11.4
6.1M
0.70
4
14.4
14.9
5.1M
CC
14.3
14.8
6.7M
0.96
5
15.5
15,9
6.1M
SC
15.5
15.9
0.72
6
14.4
14.9
7.2M
5.4 Transfer to DualPathRNN and Sandglasset
Both DualPathRNN 20 and Sandglasset 13 use intra-chunk and inter-chunk operations repeatedly to fuse information. We used proposed MSF method in A-FRCNN fuse 5 multi-scale intra- chunks and sent the result to the inter-chunk in one block, and repeated multiple blocks with tied weights. The new models, called A A-DualPahhRNN and A-Sandglasset, obtained better results than the original models, further demonstrating the merit of the proposed MSF method. Details can be found in Supplementary Materials.
6 Conclusion
We investigated an asynchronous updating scheme for a bio-inspired FRCNN architecture for solv- ing the speech separation task. The resulted model A-FRCNN achieved better results than the conventional synchronous updating scheme in experiments. Compared with the models based DualPathRNN, the proposed A-FRCNN achieved inferior results but was significantly more efficient in both training and inference. therefore achieved good trade-off between accuracy and efficiency.
This work has some limitations. First, for fair comparison with many existing models, we did not consider reverberation, which we have to deal with in the real world. It is unknown how reverberation will influence the performance of our proposed model. Second, we only considered two speakers in the mixed speech and did not consider the more natural setting where the number of speakers is unknown. Third, the proposed unfolding scheme was obtained by hand. would be interesting to use data-driven learning such as the neural architecture search techniques to determine the best design.
Acknowledgments and Disclosure of Funding
We thank the authors of the Sandglasset model (Max W. Y. Lam and Jun Wang) for providing us the model code and some constructive comments about training strategies. This work was supported in part by the National Key Research and Development Program of China (No. 2017YFA0700904).
10

---

the National Natural Science Foundation of China (Nos, 62061 136001, 61836014, U19B2034 and 61620106010) and the Tsinghua-Top Toyota Joint Research Fund.
References
[1] Mark Bear, Barry Connors, and Michael A Paradiso. Neuroscience: Exploring the Brain. Lippincott Williams & Wilkins, 2007.
[2] Joris Cosentino, Manuel Pariente, Samuele Cornell, Antoine Deleforge, and Emmanuel Vin- cent. Librimix: An open-source dataset for generalizable speech separation. arXiv preprint 1rXiv:2005.11262, 2020.
[3] Peter Dayan and Laurence F Abbott. Theoretical Neuroscience: Computational and Mathemati- cal Modeling of Neural Systems. The MIT Press, 2001.
[4] David Ditter and Timo Gerkmann. A multi-phase gammatone filterbank for speech separation via tasnet. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 36-40, 2020.
[5] Benito Fernandez, Alexander G Parlos, and Wei Kang Tsai. Nonlinear dynamic system identification using artificial neural networks (ANNs). In Proceedings of the International Joint Conference on Neural Networks (LJCNN), pages 133-141 vol.2, 1990.
[6] Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36:193-220 202, 1980.
[7] John R Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. Deep clustering: Discriminative embeddings for segmentation and separation. In Proceedings of the International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 31-35, 2016.
[8] Geoffrey E Hinton, Simon Osindero, and Yee-Why Teh. A fast learning algorithm for deep belief nets. Neural Computation, 1877):12271154, 2006.
[9] David H Hubel. Single unit activity in striate cortex of unrestrained cats. The Journal of Physiology, 147(2):226-255 238, 1959.
[10] David Hubel and Torsten Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of Physiology, 166-141 1962.
[11] Yusuf Isik, Jonathan Le Roux, Zhuo Chen, Shinji Watanabe, and John R. Hershey. Single- channel multi speakrr separation using deep clustering. In Proceedings of the International Speech Communication Association (Interspeech), pages 545-549, 2016.
[12] Morten Kolbæk, Dong Yu, theng--Hus Tan, and Jesper Jensen. Multitalker speech separation with utterance-level permutation invariant training of deep recurrent neural networks. JEEE/ACM Transuctions on Audio, Speech, and Language Processing, 25(10):900111113, 2017.
[13] Max WY Lam, Jun Wang, Dan Su, and Dong Yu. Sandglasset: A light multi-granularig self-attentive network for time-domain speech separation. In Proceedings of the International Conference on Acoustics, Speech and Signal (ICASSP), pages 5759-5763. IEEE, 2021.
[14] Jonathan Le Roux, Scott Wisdom, Hakan Erdogan, and John R Hershey. SDR-half-baked well done? In Proceedings of the International Conference Acoustics, Speech and SignalProcessing (ICASSP), pages 626-630. IEEE, 2019.
[15] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence Jackel. Backpropagation applied handwritten zip code recognition. Neural Computation, 1(4):541-551, 1989.
[16] Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y Ng. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In Proceedings of the 26th Annual International Conference on Machine Learning (ICML), pages 609-616, 2009,
[17] Ming Liang and Xiaolin Hu. Recurrett convolutional neural network for object recognition. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2015.
11

---

[18] Ming Liang, Xiaolin Hu, and Bo Zhang. Convolutional neural networks with intra-layer recurrent connections for scene labeling. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), volume 28, 2015.
[19] Qianli Liao and Tomaso Poggio. Bridging the gaps between residual learning, recurrent neural networks and visual cortex. arXiv preprint
2016.
[20] Yi Luo, Zhuo Chen, and Takuya Yoshioka. Dual-path rnn: efficient long sequence modeling for time-domain single-channel speech separation. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 46-50, 2020.
[21] Yi Luo and Nima Mesgarani. Tasnet: time-domain audio separation network for real-time, single-channel speech separation. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 696-700, 2018.
[22] Yi Luo and Nima Mesgarani. Conv-tasnet: Surpassing ideal time-frequency magnitude masking for speech separation. IEEE/ACM Trunsactions on Audio, Speech, and Language Processing, 27(6) 2019.
[23] Eliya Nachmani, Yossi Adi, and Lior Wolf. Voice separation with an unknown number of multiple speakers. Proceedings the International Conference on Machine Learning (ICML), pages 7164-7175 PMLR, 2020.
[24] James S. Nairne. Psychology. Jon-David Hague, 2014.
[25]
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206-5210, 2015.
[26] Manuel Pariente, Samuele Comnlll, Joris Cosentino, Sunit Sivasankaran, Efthymios Tzinis, Jens Heitkaemper, Michel Olvera, Rabian-Robert Stöter, Mathieu Hu, Juan M. Martín-Doñar David Ditter, Ariel Frank, Antoine Deleforge, and Emmanuel Vincent. Asteroid: the PyTorch-based audio source separation toolkit for researchers. In Proceedings of the International Speech Communication Association (Interspeech), 2020.
[27] Gintaras V Puskorius and Lee A Feldkamp. Neurocontrol of nonlinear dynamical systems with kalman filter trained recurrent networks. IEEE Transactions on Neural Networks, 5(2):279-297, 1994.
[28] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pages 234-241, 2015.
[29] BS Series. Algorithms measure audio programme loudness and true- sue-peak audio level. 2011.
[30] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolutie using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1874-1883, 2016
[31] Cem Subakan, Mirco Ravanelli, Samuele Cornell, Mirko Bronzi, and Jianyuan Zhong, Attention is all you need in speech separation. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 21-25. IEEE, 2021.
[32] Efthymios Tzinis, Shrikant Venkataramani, Zhepei Wang, Cem Subakan, and Paris Smaragdis. Two-step sound source separation: Training on learned latent targets. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 31-35, 2020.
[33] Efthymios Tzinis, Zhepei Wang, and Paris Smaragdis. SuDo RM-RF:
Efficient networks for universal audio source separation. In JEEE 30th International Workshop on Machine Learning for Signal Processing (MLSP), pages 1-6, 2020.
[34] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the Interna- tional Conference on Neural Information Processing Systems (NeurIPS), pages 5998-6008 2017.
[35] Emmanuel Vincent, Rémi Gribonval, and Cédric Févotte. Performance measurement in blind audio source separation. IEEF/ACM Transactions on Audio, Speech, and Language Processing, (44)):462-1460) 2006.
12

---

[36] Zhong-Qio Wang, Jonathan Le Roux, and John R Hershey, Alternative objective functions for deep clustering. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 686-690, 2018.
[37] Gordon Wichern, Joe Antognini, Michael Flynn, Licheng Richard Zhu, Emmett McQuinn, Dwight Crow, Ethan Manilow, and Jonathan Le Roux. Wham!: Extending speech separation noisy environments. In Proceedings of the International Speech Communication Association (Interspeech), 2019.
[38] Weifeng Xu, Xiaoying Huang. Kentaroh Takagaki, and Jian-young Wu. Compression and reflection of visually evoked cortical waves. Neuron, 19-1): 2007.
[39] Zi Yin, Valentin Yiu, Xiaolin Hu, and Liang Tang. End-to-end face parsing via interlinked convolutional neural networks. Cognitive Neurodynamics, pages 169-179, 2021.
[40] Liwen Zhang, Ziqiang Shi, Jiqing Han, Anyan Shi, and Ding Ma. Furcanext: End-to-end monaural speech separation with dynamic gated dilated temporal convolutional networks. In Proceedings of the International Conference on Multimedia Modeling (MMM), pages 653-665,
Springer, 2020.
[41] Yucheng Zhao, Chong Luo, Zheng-Jun Zha, and Wenjun Zeng. Multi-scale group transformer for long sequence modeling in speech separation. In Proceedings of the International Joint Conference Artificial Intelligence (IJCAI), pages 3251-3257, 2020.
[42] Yisu Zhou, Xiaolin Hu, and Bo Zhang. Interlinked convolutional neural networks for face parsing. In Proceedings of the International Symposium on Neural Networks (ISNN), pages 222 231, 2015.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec. 6
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] We provide the code for our main experiments, the data generation code, and instructions for running in the Supplemental Materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] The objective function for training was described Supplementary Materials; the data split is specified in Sec. 4.1 the hyperparameters are described in
how they were chosen specified in Sec. [5.3
(c) Did you report error bars (e.g., with respect
random seed after running experi- ments multiple times)? [Yes] We report the mean and standard deviation of the results
of our proposed model and its variants over 5 training runs.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] The specific hardware configurations used in this work were described in Sec. 4.2
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] MIT License
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're re
using/curating? [No] The data we used purposely designed for public usage.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "yoidmaV4WK9", "reviewer": "Reviewer_EiDq", "summary": "The paper introduce a novel method to separate mixture of  sounds. The paper introduce a new neural architecture called Fully Recurrent Convolutional Neural Network (FRCNN) which use neural network with lateral connections. The main variant of the proposed method is the A-FRCNN which use asynchronous updating. The main novel parts are: (1) novel bottleneck structure (2) removing some connection that seems to be redundant. ", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper has interesting ideas however if found that it has marginal contribution to the community due to the following:\n\n1. The proposed architecture is very similar to the previous one in the literature ״Bridging the Gaps Between Residual Learning, Recurrent Neural Networks and Visual Cortex״ [https://arxiv.org/pdf/1604.03640.pdf]\n\n2. The improvement is marginal for WHAM! dataset - less then 1dB. For Libri2Mix the results in the paper is weird: in all of previous papers the DPRNN method get better results then Conv-TasNet. Here, the author provide 14.1dB for DPRNN and 14.7dB for Conv-TasNet. I'm sure that with the right training procedure the DPRNN will get better results. Therefore, i don't think that the proposed method improve the results by more then 2dB. \n\n3. The comparison is updated for 2 years ago, the DPRNN paper published at 2019. Furthermore, the paper luck of comparison with state-of-the-art results from previous publications such as Wavsplit [https://arxiv.org/pdf/2002.08933.pdf], Gated DPRNN [https://arxiv.org/pdf/2003.01531.pdf], SepFormer [https://arxiv.org/pdf/2010.13154v2.pdf], Sandglasset [https://arxiv.org/pdf/2010.13154v2.pdf]\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discuss the limitation of the proposed method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "t9WVw1-SAYb", "reviewer": "Reviewer_SsVq", "summary": "This paper proposes a new speech separation architecture based on asynchrony fully recurrent CNN (A-FRCNN). The paper first describes FRCNN variants inspired by biological systems and discusses implementing such techniques for speech separation. Based on this discussion, the paper proposes a novel A-FRCNN speech separation technique. The experimental results show the effectiveness of the proposed method in two speech separation benchmarks (WHAM! and Librimix), achieving state-of-the-art performance compared with DPRNN. This new architecture can be applied to other similar problems, e.g., speech enhancement, and the contribution of this paper in terms of the technical novelty, experimental effectiveness, and potential wide applications is significant. My major concern about this paper is the clarity of the presentation. I could not fully understand the definition of the stage, bottom-up, and top-down operations until the actual speech separation discussions in Section 3.2. These concepts are not familiar, especially for speech separation researchers, and I recommend that the authors provide concrete examples of these concepts and elaborate the discussions.\n\nOther comments:\n1. Is it possible to use a transformer as a node instead of CNN? Recently, transformer-based processing becomes popular in speech separation.\n2. Please discuss the effect of the reverberation. The real condition always has to deal with the reverberation.\n3. Section 3.3 is a bit trivial, and it can be removed (or reduced). There are several important discussions in the appendix, and they can be put in the main document instead.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "originality:\nThe paper's concept of formulating speech separation problems with FRCNN is very novel for me. \n\nquality:\nProposed A-FRCNN with several considerations of the implementation options and experimental validations make this paper very strong. Also, the paper has various experiments, including the ablation study, the careful comparisons with u-net, and the comparisons of the other methods with public benchmarks, which makes the experimental conclusions very valid.\n\nclarity:\nAs I mentioned before, it is difficult to read the first part of the paper as it does not fully define/introduce the node, stage, bottom-up, top-down, lateral connections. I recommend the authors provide more concrete examples of them.\n\nsignificance:\nThe paper's architecture and discussions can be applied to other machine learning problems (at least other speech processing problems, including speech enhancement). Thus, the impact of the paper would not be limited to speech separation and can have more general impacts in the machine learning community.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper explicitly discusses the deficiency of the current techniques for more realistic scenarios.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "IezYisrA2Ac", "reviewer": "Reviewer_UprN", "summary": "The paper presents biologically inspired design of recurrent convolutional neural networks with bottom-up, lateral, and top-down connections. In contrast to a ‘synchronous’ update of all nodes according to connectivity, the biologically motivated unrolling of recurrent connections and information flow in time results in an ‘asynchronous’ update where different nodes get updated at different time steps with bottom-up, lateral, and top-down information flow.  These architectures are termed A-FRCNN (asynchronous fully recurrent convolutional neural networks).  A-FRCNN architectures can equivalently be viewed as feed-forward networks with specific weight tying.  These models are applied to the task of speech separation and state of the art results are obtained on two data sets.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well written with clear descriptions and explanations and presents strong results.  A few clarifying questions / comments:\n\n* At inference time I’m assuming (but couldn’t confirm in the paper) the A-FRCNN architecture is applied as well, or do the A-FRCNN architectures only guide how the FRCNN model is trained and then at inference time the fully connected FRCNN model is used in synchronous mode?  If it is the former, that is A-FRCNN architecture is used for both training and inference, then the model architecture (and not just parameter updates at training time) is different from FRCNN, in which case is it fair to call the proposed approach asynchronous updating scheme of FRCNN?\n\n* Given the biological inspiration for A-FRCNN, a control (aside from the two controls presented in the paper) that seems missing is a model that is similar to S-FRCNN but is without all the skip connections (thus restricting information flow to strictly one layer in one time step).  Did the authors try this model, or any specific reason for not trying it?  Similarly could there be variants where skip connections (both bottom-up and top-down) will skip in the time dimension (a connection between node N and N+2 will result in an unfolded model that connects node N at time T to N+2 at time T+2)?\n\n* For a given FRCNN network, the proposed approach of asynchronous updates and corresponding network design essentially adds another dimension of possibilities to explore.  Can there be a data driven learning based approach to determine optimal design, or at least guide towards that?  This seems important, because as the results with various asynchronous configurations show, the gains are quite sensitive to the model architecture (and hence it is quite possible that substantially better configurations remain undiscovered).\n\n* In Section 3.2.4 please clarify how is the mask M_i generated for i^th speaker.\n\n* In terms of speech separation background, there have been strong graphical model based approaches such as\nSteven Rennie, John R. Hershey and Peder A. Olsen\n\"Single Channel Multi-talker Speech Recognition: Graphical Modeling Approaches,\"\nIEEE Signal Processing Magazine, Special issue on Graphical Models, November 2010.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations adequately addressed, and no negative societal impact I can think of.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns I can think of.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "15-20"}, "extra_text": {}}, {"review_id": "-syvyQjUjl_", "reviewer": "Reviewer_sPgG", "summary": "The paper proposes a new separation module for extracting multi-scale features and aggregating them for improving the performance of source separation models. The proposed architecture consists of an asynchronous update mechanism which operates at different time-scales and fuses the features in parallel at the latter stage. The authors claim that the inspiration behind their proposed model is the non-simultaneous stimulus of human neurons when they get triggered from some sensory activity. The authors provide quantitative results on widely used benchmarks datasets and show improvement over strong baselines. Finally, the authors conduct ablations studies to show the importance of each update scheme inside the proposed separation module block.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Pros\nThis is a well written paper, the authors have put significant effort into presenting in detail their proposed architecture both in text and with explanatory figures showing the differences between different update mechanisms. Moreover, the parallel with the biological aspect of how the neurons work is quite interesting and is something that is missing from many current models in the audio source separation community. The experiments seems quite extensive and solid by using multiple baselines and evaluating different variations of the proposed architecture. This is definitely a quality paper in terms of the experimental framework, the evaluation and the derivations. Overall, I believe that the paper has great potential if some concerns are addressed and could contribute significantly to the field of source separation.\n\n### Limitations / Criticism\nHowever, the paper has several limitations which downplay the true potential of the paper. Please see below my concerns and questions to the authors in descending order of importance (0 is the most important):\n\n0. The authors claim that the inspiration of the proposed architecture comes from the morphology of the human neural activations. However, the submitted code is weirdly similar to the original implementation of the SuDo RM-RF [24] model which seems to be the backbone structure of the proposed models (in fact after checking with an online tool, the code for control 1 and 2 has > 80% **identical** lines with this file https://github.com/etzinis/sudo_rm_rf/blob/master/sudo_rm_rf/dnn/models/improved_sudormrf.py but also the proposed A-FRCNN has the same level of similarity). Although, using open source code is encouraged (**if it’s accredited appropriately**), in this specific case, it downplays the inspiration from the biological aspect and one could misinterpret the proposed architecture as an ad-hoc variation of a SuDo RM-RF [24] model.\n1. The authors could build the inspiration of their module architecture by first identifying the limitations of the SuDo-RM-RF [24] and then identifying biologically plausible solutions. Moreover, I believe that such story would make more sense and it would definitely avoid the negative affect on the originality of the paper, especially to a reader who is knowledgeable wrt the SuDo-RM-RF [24] architecture and can immediately see the similarities of the two works. Also, not that important, but why did the authors prefer to use the Asteroid [18] implementations for training all the models when they had access to the original repository of SuDo-RM-RF [24] and other models?\n2. The authors need to delve deeper into the details of their biologically inspire model. Specifically, I can find only two references wrt the biological inspiration [1, 3], whereas I can see whole paragraphs (Lines 34-45) where multiple things stated should be backed up by the corresponding citations. By doing so, and in combination with the limitations of used separation models, the authors can pass a clear message to the reader on why such biological mechanisms should be employed towards building better architectures.\n3. Since the authors refer to the efficiency of their model (e.g. Line 13) and they also use the term “light” for their proposed architecture, I have the following suggestions. It is quite convenient to put one number (e.g. the number of parameters or the inference time) and compare the efficiency. However, the authors propose a complex architecture that might severely enlarge the actual memory footprint of a forward pass of the network (or backward) mainly because of the intermediate activations tensors. Moreover, the same could also apply for the FLOPs or any other computational aspect. The authors should also report several other factors like actual memory requirements, time consuming on training on GPU / CPU and floating point operations and compare with the SuDo-RM-RF [24]. Based on that, the group communication mechanism [1*] has been shown to be able to significantly reduce the number of trainable parameters by dividing the channels into groups and processing them independently (with shared parameters across the groups). The authors could also integrate it to their proposed method and compare their numbers since it has also been shown beneficial for the SuDo-RM-RF [24] and DPRNN [14] models in [1*, 2*].\n4. The authors should also compare against stronger baselines with respect to the upper bound of the separation performance. A stronger SOTA architecture could be the transformer architecture proposed in [3*], which also has an open source repository.\n5. Lines 221-222: “Scale invariance is ensured by normalizing and zero-mean before the calculation.” There is no need to normalize the targets and the estimates since scale-invariant signal-to-noise ratio (SI-SNR) [4*] is by itself scale-invariant. The weirdly noted term  $ A_{\\operatorname{target}} $ in equation 7 ensures that the magnitude of the projection of the estimated source $ \\hat{s} $ onto the target $ s $ would be irrelevant to SI-SDR (since $ A_{\\operatorname{target}} $  is a vector it should probably be lowercase).\n6. The citation for the SI-SDR in Line 217 is incorrect, it should be [4*] instead of [26] which is the SDR paper. \n7. Differences smaller than <0.1 dB are not either significant nor hearable, thus I would suggest to round up all those performance numbers to a one decimal precision (it would also make the Tables less cluttered).\n8. Lines 82-83. \"For instance, Sudo-RM-RF [24] uses a U-Net structure [20], obtaining good results on smaller-sized models\". This is not true, SuDo-RM-RF uses a repetitive U-ConvBlock structure. Now, each U-ConvBlock uses a U-Net architecture. \"Smaller sized models\" is also loosely defined considering the efficiency analysis I also referred above (See point 3.) where the number of parameters is not the major concern.  \n\nTypos that I found:\nLine <>: \n159: Therefore → Therefore,\n164: increase → increases\n264: wham! → WHAM!\n\nI am willing to increase my score if most of the most important above concerns are addressed by the authors since I truly believe that the paper has a great potential to change the way how we design neural network architectures for source separation. \n\n[1] Mark Bear, Barry Connors, and Michael A Paradiso. Neuroscience: Exploring the Brain. Jones & Bartlett Learning, LLC, 2020.\n\n[3] Peter Dayan and Laurence F Abbott. Theoretical Neuroscience: Computational and Mathemati362 cal Modeling of Neural Systems. The MIT Press, 2001.\n\n[14] Yi Luo, Zhuo Chen, and Takuya Yoshioka. Dual-path rnn: efficient long sequence modeling for time-domain single-channel speech separation. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 46–50, 2020.\n\n[18] Manuel Pariente, et. al.. Asteroid: the PyTorch-based audio source separation toolkit for researchers. In Proceedings of the International Speech Communication Association (Interspeech), 2020.\n\n[24] Efthymios Tzinis, Zhepei Wang, and Paris Smaragdis. Sudo rm-rf: Efficient networks for universal audio source separation. In Proceedings of the Machine Learning for Signal Processing (MLSP), pages 1–6, 2020.\n\n[26] Emmanuel Vincent, Rémi Gribonval, and Cédric Fevotte. Performance measurement in blind audio source separation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 14(4):1462–1469, 2006.\n\n[1*] Luo Y, Han C, Mesgarani N. Ultra-lightweight speech separation via group communication. InICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2021 Jun 6 (pp. 16-20). IEEE.\n\n[2*] Tzinis E, Wang Z, Jiang X, Smaragdis P. Compute and memory efficient universal sound source separation. arXiv preprint arXiv:2103.02644. 2021.\n\n[3*] Subakan C, Ravanelli M, Cornell S, Bronzi M, Zhong J. Attention is all you need in speech separation. InICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2021 Jun 6 (pp. 21-25). IEEE.\n\n[4*] Le Roux J, Wisdom S, Erdogan H, Hershey JR. SDR–half-baked or well done?. InICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) 2019 May 12 (pp. 626-630). IEEE.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do vaguely discuss a few limitations of their work in the conclusion section but they can also include any of the points that I raised here (if they do not wish to address them in the rebuttal).\n\nIn my opinion, this work raises some potential negative societal impacts, especially environmental ones, that the authors do not discuss. For instance, the hours required to train all those models could be at least be stated in the appendix and how the authors improve upon the training time of the baseline models.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to use a bio-inspired asynchronous fully recurrent convolutional neural network (A-FRCNN) for speech separation. In contrast to the conventional synchronous update, the authors argue that with asynchronous updates via the bottom-up, top-down and lateral connections in the network, the model can fuse information flows at various time scales for improved learning.  Experiments on WHAM! and Librimix demonstrate the effectiveness of the proposed approach.  The authors show that A-FRCNN can deliver high-performance results on these two datasets compared to some of the existing best performing models under the synchronous update mechanism.  While some of the results obtained by A-FRCNN are state of the art and some are slightly underperform,  A-FRCNN also offers high efficiency with fewer parameters.  The work is well motivated.  In the rebuttal, the authors also cleared numerous concerns raised by the reviewers.   Overall, this bio-inspired approach is novel and interesting.   I would recommend accept.  The authors should revise the paper accordingly based on suggestions in the reviews and discussion. 
Decision: Accept (Poster)