Exploiting Spatial Separability for Deep Learning Multichannel Speech Enhancement with an Align-and-Filter Network
Authors: Ching-Hua Lee
Chouchang Yang
Yilin Shen
Hongxia Jin
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multichannel speech enhancement
microphone array beamforming
spatial filtering
signal alignment
relative transfer functions

Abstract: Multichannel speech enhancement (SE) systems separate the target speech from background noise by performing spatial and spectral filtering. The development of multichannel SE has a long history in the signal processing field, where one crucial step is to exploit spatial separability of sound sources by aligning the microphone signals in response to the target speech source prior to further filtering processes. However, most existing deep learning based multichannel SE works have yet to effectively incorporate or emphasize this spatial alignment aspect in the network design – we postulate that it is owing to the lack of suitable datasets with sufficient spatial diversity of the speech sources. In this paper, we highlight this important but often overlooked step in deep learning based multichannel SE, i.e., signal alignment, by introducing an Align-and-Filter network (AFnet) featuring a two-stage sequential masking design. The AFnet estimates two sets of masks, the alignment masks and filtering masks, and multiplies the estimated masks with the respective input signals to each stage sequentially, while leveraging the relative transfer functions (RTFs) for guiding the model to align signals with various speech source locations during training. For exploration purposes, we argue that the popular CHiME-3 multichannel dataset has its own limitation in representing spatially diverse speech data as the speakers were mostly located at the front side, and thereby adopt simulated and real-world measured room impulse responses to generate multichannel recordings where the target speech sources might come from arbitrary directions. Our findings suggest that for spatially diverse speaker scenarios, careful consideration of exploiting spatial characteristics is of great importance for deep learning based multichannel SE especially when the number of microphone gets increased. We show that utilizing the RTFs for signal alignment purposes in the two-stage, sequential masking framework consistently improves the capability of the network to separate the target speech from the noise signals, supporting that spatial separability is being effectively exploited by the proposed model. Our studies advocate for the advantages and significance of considering the signal alignment aspect, a wisdom coming from conventional signal processing, for developing future deep based multichannel SE algorithms to improve enhancement outcomes with positional diverse target speech scenarios.

Under review as a conference paper at ICLR 2023
EXPLOITING SPATIAL SEPARABILITY FOR DEEP LEARNING MULTICHANNEL SPEECH ENHANCEMENT NETWORK
WITH AN
Anonymous authors Paper under double-bling review
ABSTRACT
Multichannel speech enhancement (SE) systems separate the target speech from background noise by performing spatial and spectral filtering. The development of multichannel SE has a long history in the signal processing field, where one crucial step is to exploit spatial separability of sound sources by aligning the mi- crophone signals in response to the target speech source prior to further filtering processes. This is similar the human listening behavior of facing toward the speaker for better perception of the speech. However, most existing deep learning- based multichannel SE works have yet to effectively incorporate or emphasize this spatial alignment aspect in the network design; some of them rely on integrating conventional madell-ased beamformer units extract useful spatial features im- plicitly while others just let the network figure everything out by itself. However, the beamformer operation could be computationally expensive and numerically unstable when trained with the network while without it the model lacks guid- ance on learning meaningful spatial features. In this paper, we highlight this im- portant but often overlooked step in deep learning-basse multichannel SE, i.e., signal alignment, by introducing an Align-and-Filte network (AFnet) featuring two-stage sequential masking design. The AFnet aims at estimating of masks, the alignment masks and filtering masks, to carry out temporal align- ment and spectral filtering processes. During training, we propose to supervise the learning of alignment masks by predicting the relative transfer functions (RTFs) of various speech source locations followed by learning the filtering masks for signal reconstruction. During inference, the AFnet sequentially multiplies the es- timated alignment and filtering masks with the microphone signals, performing the "align- "align-then-fe then-filter" process similar to the human listening behavior. Due to the incorporation RTF supervision, the AFnet explicitly learns interpretable spatial features without integrating traditional beamformer operations.
INTRODUCTION
Speech enhancement (SE) systems can be categorized into single-channel (single microphone) and multichannel (multiple microphones) schemes. An important aspect of multichannel SE against single-channel SE is the exploitation of spatial separability, as known as spatial filtering or beam- forming, enabled by the difference between the amplitudes and times of arrival of the received mi- crophone signals due to the different acoustic paths the sound waveform travels to the microphones. In many signal processing beamforming methods (Gannot al.) 2001 Cohen 2004 Krueger al.) 2010 Koldovsky et al. 2015). a key step is to align the microphone signals in response to the target signal source before any further filtering processes. This step, by steering the array toward the lo- cation of the target signal, aims to compensate for the difference of the amplitudes and time delays (or correspondingly the magnitudes and phases in the frequency domain) of the microphone signals with respect to the target source. Ideally, after the alignment step, each microphone should contain the same target speech component with no difference in amplitude and time delay (or magnitude and phase). For a linear array in the far-filed, anechoic setting, perfectly steering the microphone array makes as if the target signal comes from the broadside, which renders the speech extraction task easier in the later filtering stage. Such process is similar to the human listening behavior of facing toward the speaker for better perception of the speech. Thus, an efficient SE system can first align its microphone signals in response to the target speech, followed by spectral processing for fine-tuning

---

Under review as a conference paper at ICLR 2023
and enhancement. Surprisingly, though well-known in signal processing approaches, there are only few deep learning SE systems designed mainly based on this observation. Instead, several recent works (Wang et al. 2020: 2021) have been utilizing conventional beamformer units such the the www.raannnnndtttttooneeeeee
(MVDR) beamformer (Capon 1969) to extract spatial characteristics implicitly in deep learning methods. However, matrix inversion or eigendecomposi- tion are often required which in turn increase the computation burden and numerical instability.
In this paper, we revisit this important but often overlooked alignment aspect from conventional sig- nal processing algorithms and recognize its importance for efficient deep learning multichannel SE network design that requires no intermediate beamformer units such as MVDR extract meaning- ful spatial features. Specifically, we propose the Align-and-Fille network (A (AFnet) shown in Figure for exploiting spatial separability within multichannel data with the following main contributions:
The AFnet features a two-stage sequential masking design, i.c., Align Net and Filter Net, where two sets of masks, alignment and filtering masks, are estimated and multiplied with the micro- phone signals to perform the "align-then- -hent-ihe-fit"" process mimicking the human listening behavior.
During the training stage, we propose to supervise the learning of the alignment masks by estimat- ing the relative transfer functions (RTFss (Gannot et al.] 2001; Cohen. 2004) for speech sources coming from various locations, prior learning the filtering masks for final enhancement.
During inference, the AFnet is able to first align the microphone signals with respect to a speech source coming from an unknown direction due to supervised learning of alignment masks. Sub- sequently, the model performs filtering on the roughly aligned signals to achieve denoising.
It is demonstrated that the RTF supervision incorporated with the sequential masking mechanism is the key effectively learn useful, interpretable spatial characteristics. On situations where the target speech may come from arbitrary positions, the "align-then-filee" mechanism consistently improves the SE performance by more efficiently exploiting spatial separability of sound sources.
2 RELATED WORK
Multichannel SE has been well known topic in signal processing for decades. Aside from lever- aging spectral characteristics, multichannel SE can exploit positional information to perform spatial filtering, beanforming, that allows extracting the target speech from noise based on spatial separa- bility. Conventional beamforming approaches rely on the so-called "steering vector" which carries positional information about the target speech (Doclo et al. 2015 Trees 2004), e.g., the MVDR beamformer (Capon) 1969) and its variants (Frost. 1972] Griffiths & Jim 1982). It is an important step in beamforming that the microphone array is steered toward the target signal location prior to further filtering processes. To this end, certain knowledge about the acoustic paths between the tar- get signal and the microphones have to be known. Many signal processing- baased multichannel SE systems utilize the ratio of the acoustic transfer functions, i.c., the relative transfer function (RTF), that represents the coupling between sensors in response to a desired source (Gannot et al. 2001
Cohen 2004 Krueger et al. 2010 Koldovský et al. 2015) for improving the denoising process.
In the past decade, deep learning approaches have remarkably changed the way of developing SE systems. Along with the success of deep neural networks (DNNs) on single-channel SE (Lu et al.)
2013 Williamson et al. 20155] Pascual et al. 2017: Luo & Mesgarani 2019 Kim et al. 2020 Zheng
et al. 2021| et al. 2020). several multichannel SE systems have also been proposed, e.g., Erdogan
ct al. (2016); Variani et al. (2016): Sainath et al. (2017): Wang & Wang (2018); Bu et al. (2019);
Koyama & Raj (2019); Luo et al. (2020); Tolooshams et al. (2020); Koyama & Raj (020)) ; Wang ct al. 2021): Kim et al. (2022); Li et al. (2022). However, the utilization of such important RTF information is often overlooked in the model design of DNN-based multichannel SE.
ct al. (2021):
Although several SE works have incorporated RTFs (Wang& & Wangl 20188) ZZhang et al., (021) the
RTF estimation is mostly used as an intermediate step to assist the MVDR beamformer only. We postulate that the overlook may be due to the lack of sufficient spatial variety of the speech sources in popular datasets such as CHiME-3 Barker et al. (2015), where the benefit of utilizing RTFs could be only marginal. However, many practical situations can have the target speech coming from arbitrary directions and thus a deep dive into the RTF spatial alignment aspect is still of great importance.
3 PROPOSED METHOD
We consider an acoustic scenario with one desired speech source and several interfering noise signals in reverberant environment. The SE system will be developed in the fime-frequency domain using
2

---

Under review as a conference paper at ICLR 2023
RTF Alignment Loss
Speech
Clean speech STFT
A
Signal
Reconstruction
Loss
$
Av
Fx
dConv (2C,
dCanv
(M,
(M, C.1
(2C,
Noley signal STFTs
Alignment
Filtering masks
dCorr (C,
dCane
dCanv (2Cg.
Enhanced speech
(2C, Ca)
(G, Gil
dCane (2C, C))
dCanv (2Cg
(C,, C.)
[C, Call
max pool 2x2
cConv (2C, Ci)
dConv (2Cu
conv 2x2
(G, Fal
(G, Cal
slip connection
conv 1xl
dConm [Ca, C3
dCom
wlement-wise (complex) multplication
over tharnels
Align Net
Filter Not
Figure 1: The proposed AFnet for exploiting spatial separability for multichannel SE. During train- ing, the Align Net estimates the alignment masks by predicting the speech RTFs. The Filter Net es- timates the filtering masks to reconstruct the clean speech by linearly combining the aligned signals after the Align Net. The RTF supervision and the two-stage sequential masking design contribute to the improved multichannel SE performance.
the short-time Fourier transform (STFT) (Parchami et al., 2016) where the time-domain waveforms are transformed complex-valued STFT representations. Let f, stand for the frequency index and time frame index (with total of F frequency bins and T time frames), we consider additive noise model where the i-th microphone noisy signal STFT X; CFXT of an N-element local microphone array can be written (Doclo al. 2015):
x,(f,t)
S.(F.)
where Ss(f.t) f) is the speech component received by mierophone i, H.(f,t) t) is the (potentially time- varying) acoustic transfer function between microphone and the location of the far-field speech source signal So(S, (), and Vi(J, is the noise component captured by microphone i. Typically, goal of multichannel SE is recover the speech component S S, E CF of selected refer- ence microphone r E {l...., N] given the noisy microphone signals XI.... .N. Multichannel SE systems typically perform the "filter-and-sum" operation, more generally known "beam- forming" - linearly combining the signals of different microphones to extract the target signal from background noise. In the STFT domain, the estimation process be expressed
$
W. W.O.X
where W, E FXT is the corresponding set of filter weights for microphone i, S is the enhanced signal, and denotes element-wise complex multiplication.
3.1 TWO-STAGE ALIGN-AND -ILTEE FRAMEWORK WITH SEQUENTIAL MASKING
Inspired by the alignment concept in signal processing based algorithms described in Section and Appendix A introduce two-stage Align-and-Filke framework for deep learning multichannel
3

---

Under review as a conference paper at ICLR 2023
SE in Figure I where two modules are utilized to carry out alignment and filtering purposes respec- tively. The first module (Align Net) aims at estimating a set alignment masks for steering the input microphone signals toward the target speech location The second module (Filter Net) focuses on cleaning up the interference in the aligned signals by estimating another set of filtering masks and linearly combines all channel outputs. Overall, the enhancement process can be expressed as:
$ Σ (F, 2m
(3)
where A, E CF is the alignment mask and F, E CF is the filtering mask of the i-th micro- phone, This corresponds to having W; = F, o A; for the equivalent beamforming filter weights applied on X; in (2) Such sequential masking scheme also introduces signal level skip connec- tions on top of the two modules for improved optimization.
3.2 RWARE TRAINING
The spatial relationship between the far-field target source and the microphones along with the acoustic environment characteristics determine the inter-channel level difference (ILD) and inter- channel time difference (ITD) of the microphone signals, which are important cues for sound source localization. In the STFT domain, the ILD and ITD correspond the magnitude and phase of the RTF respectively, which is the ratio between the acoustic transfer functions of two microphones with regards to the sound source (Li et al. 2016). Formally, we can express the RTFs H, E XT as:
t. & Hf(f,t H (()(f))) (f
(4)
(f (.(f.))
S,(f.t)
Vf,t, where (f. denotes the acoustic transfer function of the selected reference microphone NJ. From (4) we can see that the RTFs can be inferred from the ratios of the clean microphone signals which are often available in supervised learning SE schemes, and thus can be collected without knowing simulating the noise. Note that the RTF information is only needed for training and is not required during inference.
Element- wise multiplying the RTFs H. with input X; we have Z. = = Ø X, CFXT where:
t(f,t) = -H.U. t.(f,t) (H,(S, + v.(f,)] (f. +,(,t)) (5) H,(f,t
Hf,t, where Sy(f.t is the speech component of the reference microphone and (((ftt) (H, (f. is the noise component. It can be seen that multiplying the RTFFs with the respective input STFT leads to aligning all the microphones with respect the clean speech component at the reference channel as each (z(f, t) contains S, f). In the case of far-field, ane- choic setting, this means the microphone array has steered to the direction of the signal source. For the case of linear arrays, it can be viewed as the signal is coming from the broadside after steered.
We incorporate RTFs during the model learning phase via an RTF-aware training protocol consist- ing of two phases: First adjusting only the Align Net parameters so that the alignment mask A; approximates the RTFs H; by minimizing the following RTF alignment loss:
Lus
Σ (A, H.,,
(6)
NFT
where we have chosen to use the mean squared error (MSE) loss. Note that truncation of the magni- tude of the ground-truth RTFs H; (f.t) to maximum value of, e.g., 10, can be applied for avoiding training instability, and the alignment mask corresponding to the reference microphone is always all ones and is possible not to be estimated. Optimizing for this loss function guides the network to learn to steer the microphone signals toward the target speech, making the model aware of the spatial information of the target source. The second phase performs parameter learning of the whole model (i.e.. jointly adjusting both Align Net and Filter Net parameters) to reconstruct the clean speech. To this end, we minimize the signal reconstruction loss:
Croe = Σ ( ( + ())().)))e
|S( ()
f.t
where we have chosen to use the combined power-law compressed MSE loss first appeared (Wil- son et al., 2018). In this work we use B = 0.3 and = 0.3 as suggested by consolidated study in (Braun & Tashev 202I).
4

---

Under review as a conference paper at ICLR 2023
3.3
AFNET BASED ON COMPLEX W-NET ARCHITECTURE
We introduce the AFnet (Figure D for multichannel SE based on implementing a W-Net (Xa & Kulis 2017) architecture suitable for realizing the two-stage framework. For the SE application here, extension to complex operations can be more suitable when working in the STFT domain. Choi et al. (2018) have proposed to utilize complex-valued U-Net for the single-channel SE task. We follow this idea and extend the W-Net by incorporating complex operations for multichannel SE. In the current work, main modifications to the W-Net include: Convolutional layers of W-Net are all replaced to complex convolutional layers. For the activation function, complex leaky ReLU, i.e., an activation function which applies leaky ReLU on both real and imaginary values, are utilized. The number of feature maps in each layer are also modified. Note that in Figure [ "dConv (Cin- Coua" stands for "double Convolutions" with C.n input channels and Caut output channels. Each normalization ReLU." The convolution layer of the first stack takes Cin channels and outputs Cout channels; the convolution layer of the second stack takes Cour channels and outputs Cour channels.
dConv unit consists of two stacks of "(complex) 3x3 convolution--batee
3.4 DIFFERENTIATION FROM EXISTING METHODS
Existing works such Wang al. 2021 Zhang et al. 2021 leverage the conventional beamformer modules for incorporating spatial features implicitly. However, matrix inversion or eigendecompo- sition are often required and additional network modules might otherwise be needed replace such operations. Other works (Koyama & Raj. 2019 2020 Li et al. 2022) concatenate two DNN mod- ules and jointly train both toward the enhanced speech. However, without utilizing any beamforming units within the model it is hard to interpret if the network actually learns to exploit spatial features. Differently, our method introduces a simple skip connection of alignment masks trained via super- vised learning of the RTFs to explicitly incorporate meaningful spatial characteristics.
4 EXPERIMENTS
4.1 DATASETS AND EXPERIMENTAL SETUP
4.1.1 PRIMARY DATASET
Speech corpus: We leverage the AVSpeech dataset which is a large collection of video clips of
single speakers talking with no audio background interference introduced in Ephrat et al.] (2018). The dataset is based public instructional YouTube videos, from which short, 3-10 second clips were automatically extracted, where the only audible sound in the soundtrack belongs to a single speaking person. We downloaded 8308 clips from the training set and another 1199 clips from the test set for our experiments, using only the audio portion (converted to 16 kHz. wwav files).
Room impulse response (RIR) data: We mix the speech files with noise files of several types also downloaded from YouTube and consider different reverberation conditions for generating the noisy data using simulated impulse responses via Pyroomacoustics (Scheibler et al. 2018), which is Python-based acoustic simulator for generating different acoustic scenes. We create a room of size 8 x 8 x 3 (length x width x height in meters). Reverberation time is randomly chosen from {0. 16, 0.32, 0.48. 0.64) second. The simulating approach for estimating RIRs is based on the image source method. Speech and noise signals are randomly positioned at a distance of {1, 2, 3, 4] meters from the microphone array with arbitrary directions of arrival. The microphones form a 2-by-4 planar array placed vertically at the center of the room, and the spacing between two adjacent microphones is cm. 4 types of commonly seen interference (blender, vacuum, washer, baby cry) are used for training and another 4 types (dog barking, kids playing sound, hair dryer, food sizzling) are used for testing. Each type of noise is multiplied with a scale randomly chosen from {0, 0.5, 1, 1.5, 2} before added up, and the combined noise is then added to the clean speech according to specific SNR level randomly selected from {-10, -6, -3, 0, 3, 6, 10) dB.
4.1.2 AUXILIARY DATASET:
The CHiME-3 dataset (Barker et al. 2015). made available as part of a speech separation and recog- nition challenge, is used for training and evaluating SE performance. The dataset is a 6-channel microphone recording of talkers speaking in a noisy environment, sampled at 16 kHz. It consists of 7,138 and 1,320 simulated utterances with an average length of 3 seconds for training and testing, re- spectively. We take this dataset only auxiliary due to limited positional variety of the target speech
5

---

Under review as a conference paper at ICLR 2023
most of the speakers have been speaking from the front side with respect to the microphones. As such, the dataset is less ideal for exploration of spatial characteristics and signal alignment purposes.
4.1.3 MODEL SETTINGS AND EVALUATION METRICS
In this paper, we set the number of channels of the AFnet (Figure 1 as: C2,C2, C3, C4 32, 64, 64, 64, resulting
model size of approximately 2.6M parameters. The slope of leaky ReLU is set to 0.01. Regarding the STFT processing of the input signal, for the primary dataset we use the Hann window with 512-point fast Fourier transform (FFT) and for CHiME-3 with 1024-point FFT, both using a hop size of 256. During training, 4-second long segments are randomly cropped from the training data samples while during testing the whole utterances are used. The RTFs are ob- tained by computing the ratios between the clean speech signals at all microphones according to 4 We use the following three commonly seen SE metrics for evaluation purposes. PESQ: Perceptual Evaluation of Speech Quality (ITU-T Recommendation P.862.2, 2005). STOI: Short-Time Objec- tive Intelligibility (Taal et al., 2011). SSNR: Segmental Signal-to-Nor Ratio (Hansen & Pellom 1998) (segment length 30 msec, 75% overlap. SNR 10 dB, SNRma 335 dB). For all the metrics, the higher the score, the better the performance.
More details of experiential settings can be found int Appendix B
4.2 RESULTS
4.2.1
ALIGNED VS. UNALIGNED SIGNAL ENHANCEMENT
To get an understanding of the importance of alignment, we first conduct an experiment to com- pare the difference between enhancement outcomes of aligned and unaligned input signals with the primary dataset. To this end, two schemes are considered, where in the first scheme we manually align the noisy microphone signals by multiplying them with the corresponding ground truth speech RTFs, while in the other scheme we directly use the original unaligned signals. In each scheme, single U-Net model, or equivalently, the Filter Net (the second half of AFnet), is trained for the signal reconstruction loss free and tested on the corresponding aligned or unaligned data. The per- formance numbers of the cases are presented in Table It can be seen that when the signals are aligned, the Filter Net model can do a much better job of separating the target signal from back- ground noise and achieves significantly better scores than unaligned results in all three metrics. This observation is consistent across all three number of microphone settings. The results suggest that for a spatially diverse dataset where the target speech may come from various locations, having a signal alignment module in front of the signal filtering module can be beneficial.
We also present the results for the CHiME-3 dataset in Table We can see that the margin between the results of using the aligned and unalinged signals is much smaller here. It could be attributed to that most of the CHiME-3 recording data samples have the speaker almost speaking from the front side with respect to the microphones which results in less variety in directional angles (the common situation when the speaker was asked for holding a tablet to speak the sentence listed in the tablet, and the speech data were recorded through 6 embedded microphones in the same tablet. See the recording setup and recording demonstration picture in CHiME-3 official website) and thus the less spatial diversity. Moreover, the speech sources mostly come from the front side, they in sense already roughly aligned. Thus, there is the less benefit brought by performing the alignment, meaning that the requirement of aligning signals is bypassed some extend.
Table 1: SE performance comparison between aligned (A) and unaligned (U) signals.
PESQ
STOI
SSNR
Mic
U
A
A
U
2
2.68
1.68
0.842
0.697
7.86
4.66
4
2.89
1.67
0.862
0.696
8.28
4.56
8
3.04
1.76
0.879
0.719
8.66
4.68
CHiME-3
2.62
2.42
0.973
0.963
10.32
9.13
4.2.2 EXPLOITING SPATIAL SEPARABILITY WITH AFNET
Having the above insight, move to studying the proposed AFnet for exploitation of spatial separability on the primary dataset with different model/training settings and comparing with typical direct filter weight estimation approaches. For detailed analysis, consider four cases below:

---

Under review as a conference paper at ICLR 2023
U-Net for filter weights: using only a single U-Net (same model the Filter Net of AFnet) to directly estimate the sets of filter weights W, in typical multichannel SE
ii) W-Net for filter weights: using W-Net (same model the AFnet, but the output of the first U-Net block goes directly to the next U-Net block without multiplying the input signals) to directly estimate the sets of filter weights W, in typical multichannel SE
11) AFnet w/o RTF loss: the two-stage sequential masking AFnet model trained by normal training scheme without incorporating RTFs
iv) AFnet: the two-stage sequential masking AFnet model trained with RTF-aware scheme
Cases and ii) represent the typical approach which uses DNN directly estimate the beamformer weights. Cases iii) and iv) are considered to observe the role of RTF alignment in the AFnet.
Figure presents the enhancement results. One can see that the typical approaches of U-Net and W-Net for direct filter weights estimation do not seem to actually benefit from the multichannel in- put data, as the performance is not necessarily improving with increased number of microphones. In fact, the W-Net even performs worse than the single U-Net model for the cases with more micro- phones (4-mic, 8-mic) though doubling the model size, with its best result surprisingly reached for the 2-mic case. This observation implies that to further take advantage of audio data with more mi- crophone channels, simply stacking U-Net models may not straightforwardly work well. This might be due to the potential optimization difficulty of having deeper model and more complicated mul- tichannel input data to extract useful information. On the other hand, the AFnet seems to mitigate these issues by introducing the sequential masking mechanism which also provides certain input signal level skip connections, as we can see that even without RTF supervision it is able to perform much better than the W-Net. Nevertheless, the performance of AFnet without RTF supervision does not necessarily improve with increased number of microphones. This may be a result of lacking guidance on learning meaningful spatial features. Finally we see that with the RTF loss incorpo- rated for explicitly leveraging spatial separability, the AFnet is able to obtain better performance as the number of microphones increases, and achieves the best performance overall.
2.0-
Arred, HIT Alhee
6.76
Afreet 10 los
5,4
Uil for weighte
Afhet
522
Afret
2.8
3.7
4.5
0.72
4.4
3.5
2.Mcc
4MCG
5.9%
2.9%
4:94:
SMc
4.0
2-Mc
4Mc
5.kk
Figure 2: SE performance of different networks and training settings.
4.2.3 VISUALIZING THE ALIGNMENT OUTCOMES
To further observe the effect of RTF supervision, we visualize the distributions of the alignment masks in Figure 3 from AFnet trained with RTF loss and that without RTF loss. To obtain these plots, we take noisy signal samples from the test set in which the speech source in each sample is located at of three pre-defined positions in the trained AFnet models (both with and without RTF supervision) and obtain the alignment masks for each sample. The distribution of the principal components (PCs) of the obtained masks are plotted. It can be seen that with RTF-aware training, the distributions show clearer clusters of speech sources corresponding to the three different positions than with normal training. In addition, with increased number of microphones the clusters also become more separate. The results together with improved enhanced speech quality provide evidence of the fact that the system actually exploiting spatial separability of the audio sources for achieving better enhancement performance.
We feed the taken noisy samples to the
4.2.4 COMPARISON TO SOTA METHODS
To demonstrate the superiority our method, we compare the AFnet with several SOTA architec- tures, including: Tas-TasNett (Luo & Mesgarani] 2019): a single-channel SE system performing masking- based separation method in the learned transform domain via learnable encoder and de- coder. DCUnet (Choi al. 2018): phase-aware single-channel SE approach utilizing complex- valued operations with the U-Net architecture. FaSNet (Luo et al., 2020): a time-domain, multi- channel SE system exploiting cross-channel correlation based ann method.

---

Under review as a conference paper at ICLR 2023
With RIF Superck on Aen
Wh Sparaiio A-rric
With RIF Sups vioee S-mtic
10
at
-73
-70
PG-1
Norma Traln ng 2-mec
Nermal Trainng 4-mcc
Norma Trainin, Birsc
sourc wibe anlec
souch
100
sounch wt an
1.5
64
00
-29
-50
-2.5
15.0
-50
75
FC11
Figure 3: Visualization of the distributions of alignment masks. The RTFF-aware training cases (Upper Row) show obvious clusters of signals coming from the three different locations (loc loc 2, loc 3) as the number of microphone increases, while the normal training cases (Lower Row) do not present clear boundaries.
We train the SOTA models by ourselves. For faimess, all models are trained by minimizing the reconstruction loss Love of the combined power-law compressed MSE of (7), using 512-point FFT with a hop size of 256 via the Hann window. The results are presented in Table 2 can be seen that the single-channel methods do not perform well even with larger model size. The FaSNet, with a lighter model. performs slightly better than the two single-channel approaches when the number of microphones is larger. Finally, our proposed AFnet outperforms all the SOTA approaches by significant amount in all the metrics used, while being of the smallest model size. Note that the performance of AFnet lies in between the aligned and unaligned signal cases in Table [ which is reasonable as the alignment is never perfect but only to certain degree.
Although a less ideal dataset for our study on spatial characteristics, we still benchmark the AFnet with several SOTAs on the CHiME-3 dataset, including: Neural BF (Erdogan et al. 2016). MVDRGc (Bu et al. 2019). rSDFCN (Liu al., 2020 CA Dense U-Net (Tolooshams et al.) 2020). and IC Conv- Tas (Lee et al. 2021). In Table 3 we report the PESQ and STOI taken from the corresponding papers and the missing entries in the table indicate that the metric is not reported in the reference paper (we do not report SSNR as most papers do not report it). We present both results of the AFnet trained with and without RTF supervision (with the 5-th channel selected as the reference microphone). One can see that the two-stage, sequential masking design of the AFnet is efficient in that achieves comparable or better performance to other SOTA approaches while maintaining small model size. Notably, the RTF supervision helps improve the A.net per- formance, while the gains are minor in this case due to the less spatial diversity of the speech data. Nevertheless, the results still show the benefit of incorporating signal alignment into the network design, even though on a dataset with limited positional variety of the speech target.
4.2.5 IMPORTANCE OF PHASE ALIGNMENT IN RTF ESTIMATION
The Align Net trained with the RTF loss performs both temporal alignment as well as level adjust- ment. It would be interesting to look at how the magnitude and phase components of the RTFs, which correspond to ILD and ITD in time domain, affect the enhancement performance of the AFnet. In Figure we compare four different RTF alignment schemes no alignment, aligning RTF magni- tude only, aligning RTF phase only, and aligning everything for the RTF loss Lnt. From the results can be seen that phase alignment is the key to obtaining improved performance over the alignment- agnostic system. This is reasonable as aligning phase in the frequency domain corresponds to com- pensating the delay in the time domain, which is crucial for any beamforming algorithms that rely on spatial separability. The results suggest that the AFnet indeed exploits spatial information for improved enhancement performance.
8

---

Under review as a conference paper at ICLR 2023
Table SE comparison with SOTA methods on the primary dataset.
Table 3: SE comparison with SOTA methods on the CHiME-3 dataset.
Methods
Params
PESQ
STOI
SSNR
Methods
# Params
PESQ
STOI
Noisy
1.21
0.577
-1.70
Conv-TasNet
8.7M
1.47
0.636
3.00
Noisy
1.27
0.870
Neural BF
2.29
DCUnet
7.6M
1.49
0.660
2.91
MVDRGC
0.5M
0.952
FaSNet (2-mic)
1.55
0.663
3,70
ISDFCN
2.1M
2.15
0.937
FaSNet (4-mic)
2.8M
1.57
0.667
3,64
CA Dense U-Net
>20
2.44
FaSNet (8-mic)
1.64
0.683
3,66
IC Conv-TasNet
1.7M
2.67
0.973
AFnet (2-mic)
1.84
0.728
5.06
AFnet w/o RTF Loss (ours)
2.68
0.972
AFnet (4-mic)
2.6M
1.92
0.739
5.26
AFnet (ours)
2.6M
2.72
0.972
AFnet (8-mic)
1.99
0.753
5.35
Notal lane
gred
Mug elgres)
0.72
Mug elgned
2.00
Prans linad
Mag shane al gred
Mag & shase gred
Prane a lgned
aligedd
5.2-
5
0.74
1.65
5.0
1.75
Figure 4: SE comparison of different RTF alignment schemes for AFnet training.
4.2.6 ARE INTERMEDIATE ALIGNMENT MASKS NECESSARY?
Given the results in Figure where the phase importance recognized, one would ask if the inter- mediate alignment mask design of Annet is necessary, and if we can directly apply phase alignment at the output filtering masks? To answer this question, we train the W-Net for predicting the filter weights (i.e., the filtering masks) to minimize Lres + XL phose, where Lree the same signal construction loss as (7) and Cat phase is the RTF alignment loss similar to (6) but imposed on the filtering masks given by the W-Net for the phase portion only and regularized by A O. Several values of X are considered and the results are compared to AFnet in Table 4 We can see that over wide range of À the W-Net is still not able to achieve as good performance as AFnet which utilizes the alignment masks with RTF supervision. This implies that the intermediate masking mechanism for performing alignment is critical for achieving improved SE with two-stage network design.
Table 4: SE comparison of AFnet and W-Net with RTF phase regularization on the filtering masks.
PESQ 4-mic
STOI
SSNR 4-mic
Method
2-mic
S-mic
2-mic
4-mic
S-mic
2-mic
8-mic
W-Net filter weights, nо phase reg.
1.74
1.61
0.703
0.679
4.80
4.07
W-Net for filter weights. phase reg. 0.0001
1.76
1.61
1.65
0.713
0.686
0.692
4.95
4.21
4.42
W-Net for filter weights, phase reg. À 0.001
1.63
1.60
1.68
0.688
0.686
0.702
4.36
4.14
4,60
W-Net for filter weights. phase reg. À 0.01
1.71
1.59
1.61
0.705
0.683
0.684
4.55
4.14
4.30
W-Net for filter weights, phase reg. À 0.1
1.73
1.66
1.74
0,708
0.701
0.710
4.09
3.54
3.87
AFnet
1.84
1.92
1.99
0.728
0.739
0.753
5.06
5.26
5.35
Additional results variety of testing scenarios, e.g., SNR conditions, room configurations, train-
ing protocols, time-varying RIR and realistic RIR data are provided in Appendix C Comparison of Align Net with signal processing-basee algorithms for RTF estimation is presented in Appendix D
CONCLUSION
In this paper, we presented the AFnet, deep learning- multichannel SE approach that performs signal alignment followed by filtering for extracting the target speech from noisy observations. We showed that leveraging spatial information inherent in the RTFs for signal alignment purposes of the two-stage, sequential masking network during training the key to remarkable improvements. Our findings suggest that RTF alignment, especially the phase estimation, plays crucial role in deep learning multichannel SE for target speech that may come from arbitrary locations.

---

Under review as a conference paper at ICLR 2023
REFERENCES
J. Barker, R. Marxer, E. Vincent, and S. Watanabe. The third 'CHiME' speech separation and recog- nition challenge: Dataset, task and baselines. In Proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding (ASRU), pp. 504-511, 2015.
A. Benichoux, L. S. Simon, E. Vincent, and R. Gribonval. Convex regularizations for the simul- taneous recording of room impulse responses. IEEE Transactions on Signal Processing, 62(8): 1976-1986, 2014.
S. Braun and Tashev. A consolidated view of loss functions for supervised deep learning-bast speech enhancement. In Proceedings of International Conference Telecommunications and Signal Processing (TSP), pp. 72-76, 2021.
S. Bu. Y. Zhao, and M.-Y. Y. Hwang. A novel method to correct steering vectors in MVDR beamformer for noise robust ASR. In Proceedings of the Anmual Conference of the International Speech Communication Association (Interspeech), pp. 4284, 2019.
Capon. High-resolution freuuencywwavenunen spectrum analysis. Proceedings of the IEEE, 57 (8):1408-1418, 1969.
H.-S. Choi, J. Kim, Huh, A. Kim, J.-W. Ha, and K. Lee. Phase-aware speech enhancement with deep complex U-Net. In Proceedings of International Conference Learning Representations (ICLR), 2018.
I. Cohen. Relative transfer function identification using speech signals. IEEE Transactions on Speech and Audio Processing, 12(5):451-45, 2004.
Doclo, W. Kellermann, Makino, and S. E. Nordholm. Multichannel signal enhancement al- gorithms for assisted listening devices: Exploiting spatial diversity using multiple microphones. IEEE Signal Processing Magazine, 32(2):18-30, 2015.
A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, W. T. Freeman, and M. Ru- binstein. Looking to listen at the cocktail party: A speaker-indepenneet ausia-visual model for speech separation. ACM Transactions Graphics, 37(4):109:1-109:1 2018.
H. Erdogan, R. Hershey, S. Watanabe, M. Mandel, and Le Roux. Improved MVDR beamform- ing using single-channel mask prediction networks. In Proceedings of the Anmual Conference of the International Speech Communication Association (Interspeech), pp. 1981-1985, 2016.
O. L. Frost. An algorithm for linearly constrained adaptive array processing. Proceedings of the IEEE, 60(8):926-99) 935, 1972.
S. Gannot and I. Cohen. Speech enhancement based on the general transfer function GSC and postfiltering. IEEE Transactions on Speech and Audio Processing, 1266)5561,77 2004.
Gannot, D. Burshtein, and E. Weinstein. Signal enhancement using beamforming and nonsta- tionarity with applications to speech. IEEE Transactions Signal Processing, 1916))1614-1626, 2001.
R. Giri, B. D. Rao, F. Mustiere, and T. Zhang. Dynamic relative impulse response estimation using structured sparse Bayesian learning. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 514-518, 2016.
R. Giri, T. A. Srikrishnan, D. Rao, and Zhang. Empirical bayes based relative impulse response estimation. The Journal of the Acoustical Society of America, (4366))3922-3933 2018.
L. Griffiths and C. Jim. An alternative approach to linearly constrained adaptive beamforming. IEEE Transactions on Antennas and Propagation, 30(1):27-34, 1982.
E. Hadad, F. Heese, P. Vary, and S. Gannot. Multichannel audio database in various acoustic envi- ronments. In International Workshop on Acoustic Signal Enhancement (IWAENC), pp. 313-317, 2014.
10

---

Under review as a conference paper at ICLR 2023
J. H. L. Hansen and B. L. Pellom. An effective quality evaluation protocol for speech enhancement algorithms. In Proceedings of International Conference on Speech and Language Processing (ICSLP), volume 7, pp. 2819-2822 1998.
Y. Hu, Y. Liu, S. Lv, M. Xing, S. Zhang, Y. Fu, J. Wu, B. Zhang, and L. Xie. DCCRN: Deep complex convolution recurrent network for phase-aware speech enhancement. In Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech), pp- 2472-2476, 2020.
ITU-T Recommendation P.862.2. Wideband extension to recommendation P.862 for the assessment of wideband telephone networks and speech codecs. International Telecommunication Union, 2005.
H. Kim, K. Kang, and J. W. Shin. Factorized MVDR deep beamforming for multi-channel speech enhancement. IEEE Signal Processing Letters, 29:1898 100:1888-1999 -9088899992111001 2022.
J. Kim, M. El-Khamy, and J. Lee. T-GSA: Transformer with Gaussian-weighte self-attention for speech enhancement. In Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp.
2020.
Z. Koldovsky, P. Tichavský, and D. Botka. Noise reduction in dual-microphone mobile phones using bank of pre-measured cagget-cancellation filters. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 679-683,2011
7. Koldovsky, J. Málek, and S. Gannot. Spatial source subtraction based on incomplete measure- ments of relative transfer function. IEEE/ACM Transactions Audio, Speech, and Language Processing, 1335):1335-138 1347, 2015.
Y. Koyama and B. Raj. W-Net BF: DNN-based beamformer using joint training approach. arXiv preprint arXiv: 1.X.:429... 2019.
Y. Koyama and Raj. Exploring optimal DNN architecture for end-to-end beamformers based on time-frequency references. arXiv preprint 22X.::2005512683, 2020.
A. Krueger, E. Warsitz, and R. Haeb-Umbach Speech enhancement with a GSC-like structure employing eigenvector-based transfer function ratios estimation. IEEE Transactions on Audio, Speech, and Language Processing, 19(1):00-211, 2010.
B. Laufer, R. Talmon, and S. Gannot. Relative transfer function modeling for supervised source localization. In IEEE Workshop on Applicutions of Signal Processing to Audio and Acoustics (WASPAA), pp. 1-4, 2013.
D. Lee, S. Kim, and J.-W. Choi. Inter-channel Conv-TasNet for multichannel speech enhancement. arXiv preprint arXiv:21 1204312, 2021.
J. Li, Zhu, D. Luo, Y. Liu, G. Cui, and Z. Li. The PCG-AIID system for L3DAS22 Challenge: MIMO and MISO convolutional recurrent network for multi channel speech enhancement and speech recognition. In IEEE International Conference on Acoustics, Speech, and Signal Process- ing (ICASSP), pp. 9211 191519 2022.
X. Li, L. Girin, R. Horaud, and S. Gannot. Estimation of the direct-path relative transfer function for supervised sound-source localization. IEEE/ACM Transactions Audio, Speech, and Language Processing, 24(11):2171-2)) 2016.
Y. Lin, Chen, Y. Kim, and D. D. Lee. Blind channel identification for speech dereverberation using Iı-norm sparse learning. Advances in Neural Information Processing Systems (NIPS), 20, 2007.
C.-L. Liu, S.-W. Fu, Y.J. Li, J.-W. Huang, H.-M. Wang, and Y. Tsao. Multichannel speech enhance- ment by raw waveform waveform-mmpping using fully convolutional networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28: 18118881190 1900, 2020.

---

Under review as a conference paper at ICLR 2023
X. Lu, Y. Tsao, S. Matsuda, and C. Hori. Speech enhancement based on deep denoising autoencoder. In Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech), volume 2013, pp. 436-440, 2013.
Y. Luo and N. Mesgarani. Conv-TasNet. Surpassing ideal time-frequency magnitude masking for speech separation. JEEE/ACM Transactions on Audio, Speech, and Language Processing, (7): 1256-1266, 2019.
Y. Luo, Z. Chen, N. Mesgarani, and T. Yoshioka. End-to-end microphone permutation and number invariant multi-channel speech separation. In Proceedings of IEEE International Conference Acoustics, Speech, and Signal Processing (ICASSP), pp. 6394-6398, 2020.
M. Parchami, W.-P. Zhu, B. Champagne, and E. Plourde. Recent developments in speech enhance- ment in the short-time Fourier transform domain. IEEE Circuits and Systems Magazine, 16(3): 45-77, 2016.
S. Pascual, A. Bonafonte, and J. Serra. SEGAN: Speech enhancement generative adversarial net- work. In Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech), pp. 3642-3646, 2017.
N. Sainath, Weiss, K. Wilson, B. Li, Narayanan, E. Variani, M. Bacchiani, I. Shafran, A. Senior, K. Chin, A. Misra, and C. Kim. Multichannel signal processing with deep neural networks for automatic speech recognition. IEEE/ACM Transactions on Audio, Speech, and Lan-
guage Processing, 25(5):965-979, 2017.
R. Scheibler, E. Bezzam, and I. Dokmanić. Pyroomacoustics: A python package for audio room simulation and array processing algorithms. In Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 351-355, 2018
T. A. Srikrishnan, B. D. Rao, R. Giri, and T. Zhang. Improved noise characterization for relative impulse response estimation. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 41144111444 4111-415,22 2018.
C. H. Taal, R. C. Hendriks, R. Heusdens, and J. Jensen. An algorithm for intelligibility prediction of fimeenryy weighted noisy speech. IEEE Transactions on Audio, Speech, and Language Processing, 19(7):2125- 2212221366,2222 2011.
B. Tolooshams, R. Giri, A. H. Song, U. Isik, and A. Krishnaswamy. Channel-attention dense U- Net for multichannel speech enhancement. Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 836 846-800,5 2020.
H. L. Van Trees. Optimum Array Processing: Part IV of Detection, Estimation, and Modulation Theory. Honoken, NJ, USA: John Wiley & Sons, 2004.
E. Variani, T.N Sainath, I. Shafran, and M. Bacchiani. Complex linear projection (CLP): A discrim- inative approach to joint feature extraction and acoustic modeling. In Proceedings of the Annual Conference of the International Speech Communication Association (Interspeech), pp. 808-812, 2016.
Z.-Q. Wang and D. Wang. Mask weighted STFT ratios for relative transfer function estimation and its application to robust ASR. In Proceedings of IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 5619-5623. IEEE, 2018.
Z.-Q. Wang, P. Wang. and D. Wang. Complex spectral mapping for single-and multi-channel speech enhancement and robust ASR. IEEE/CM Transactions on Audio, Speech, and Language Pro- cessing, 28: 17771-1877 2020.
Z.-Q. Wang, H. Erdogan, S. Wisdom, K. Wilson, D. Raj. S. Watanabe, Z. Chen, and R. Hershey. Sequential multi-frame neural beamforming for speech separation and enhancement. In IEEE Spoken Language Technology Workshop (SI.T), pp. 905-911, 2021.
D. S. Williamson, Y. Wang, and D. Wang. Complex ratio masking for monaural speech separation. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 24(3):483 -422, 2015.
12

---

Under review as a conference paper at ICLR 2023
K. Wilson, M. Chinen, J. Thorpe, B. Patton, J. Hershey, R. A. Saurous, J. Skoglund, and R. F. Lyon. Exploring tradeoffs in models for low-latency speech enhancement. In Proceedings of International Workshop Acoustic Signal Enhancement (IWAENC), pp. 366-370, 2018.
X. Xia and B. Kulis. W-Net: A deep model for fully unsupervised image segmentation. arXiv preprint arXiv:1711.08506, 2017.
Z. Zhang, Y. Xu, M. Yu, S.-X. Zhang, L. Chen, and D. Yu. ADL-MVDR: All deep learning MVDR beamformer for target speech separation. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), pp. 6089-6093, 2021.
C. Zheng, X. Peng, Y. Zhang. S. Srinivasan, and Y. Lu. Interactive speech and noise modeling for speech enhancement. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), pp. 14549-14557 2021.
13

---

Under review as a conference paper at ICLR 2023
APPENDIX A CONVENTIONAL SIGNAL PROCESSING BEAMFORMING ALGORITHMS BASED ON ALIGN- THEN-FILTER PROCESS
To support the motivation behind the proposed AFnet which is designed based the align-then- filter concept well-known in signal processing, we review several spatial filtering algorithms based on using this design principle.
The intuition for performing alignment followed by filtering is relatively simple. Consider the RTF aligned signal expression (5). We can see that in the aligned signals each (f. contains tf. Since all the (,,f,t t] among the N channels are in in-phase (and actually are identical if magnitudes are also aligned in this case), summing up all the Ze(,,) 1) over the N channels will not be destructive to the speech component. However, the noise components Vi(f.t) t) are usually not in-phase (unless coming from the same direction as the speech source), destructive combination can happen by summing them up. Consequently, combining all the tf) across the N channels will boost the SNR the speech component is preserved while the noise components are suppressed.
The above process is actually the classic "delay-and- aeday--add-umm (DS)" beamformer Trees (2004); Doclo et al. (2015). In DS beamformer, steering vector containing the time delay information between the microphone sensors with respect to the target speech is first applied to the input multichannel sig- nals compensate the time delay (or equivalently the phase difference) of the speech components in all channels. This step is called "alignment." " Then summation operation over the aligned channels performed to suppress noise for boosting SNR. This step is called "filtering." " Mathematically, the enhancement process of the DS beamformer can be expressed as:
$(f,t) = h² (f,
Vf,t, where h(f.t) = (f. (f Hs (f.)) is the steering vector consisting of RTFs (poten- (x, is the vector of micro-
tially only the phase components) and x(f.t) phone signals.
Another famous beamformer utilizing the align-then-filter concept is the "generalized sidelobe can- celler (GSC)" (Griffiths & Jim 1982 Frost 1972). In typical GSC, the input noisy signals are first aligned by the steering vector. Then, two branches of processing follow, where one branch sums up all the aligned microphone signals to perform a DS beamforming, while the other branch introduces a "blocking matrix" to extract noise references while blocking out the target speech component. Then, the noise-only output of the blocking matrix will go through an adaptive filtering algorithm to estimate the residual noise component in the DS beamforming output signal in an adaptive man- ner. Note that the alignment step plays an important role here. First, it leads to the DS beamformer for boosting SNR. In addition, it is beneficial for the blocking matrix stage to extract noise-only references. For example, simple blocking mechanism can be just subtracting all other (, t), i * r from the reference channel Z, (f. so that the S,(ftt will be cancelled out and only the noise remains. The GSC is obviously another "align-then-filter" example with more sophisticated filtering stage.
Finally, the popular "minimum- wwtee-vaaviiiiiiiie www.toottionessssesponno (MVDR)" beamformer (Capon 1969 Doclo et al. 2015)) can also belong to the align-then-filter framework. The derivation of the MVDR filter weights w [Wi(J,)) (), Ww (.() D) CV which applies to the noisy signals
to obtain the enhanced speech as ((f,t) = (f((tt)x(ff. starts from considering the following optimization problem:
min w H4 (f,t), subject to w w(((ftt =
(9)
where (f,t) ((y(,, E CNN is the power spectral density of the vector of noise signals v(ftt- = [Vi(f,t).... (f(f t)T²T and h(f,t) = [H,
(f. t))² is the (re-
ciprocal) vector of RTFs. Solving the above we obtain the MVDR filter as:
Φ (f'(f.t)h(ff)
w(, h (H(,0) (1. (F,O)(
(10)
Vf. t. We see that the RTFs which are related to signal alignment are involved in the computation of the MVDR filter weights.
14

---

Under review as a conference paper at ICLR 2023
From the above, beamforming algorithms. This has motivated us to devise the AFnet following the alignment prin- ciple. Note that the signal processing beoeessing-based beamforming algorithms typically rely modeling assumptions of signal or noise statistics to perform denoising, and therefore the performance is lim- ited due to mismatch of real-world observations and modeling assumptions. Our approach is based on data-driven, deep learning techniques to overcome such limitations while taking advantage of the RTFs.
that the RTF information plays essential role in many conventional
APPENDIX B IMPLEMENTATION DETAILS
B.I RTF-EWARE TRAINING ALGORITHMS
The main innoooooeeedated training scheme is summarized in Algorithm [ which performs two- step training consisting of i) training the Align Net for the RTF loss followed by ii) training both Align and Filter Nets jointly for the signal reconstruction loss.
Algorithm RTF-Aware Training (Two-Step) inputs
Model(X; E CPXT, Vi = I,................ I,..., N; Align Net, Filter Net)
Initialize Align Net. Filter Net for number training iterations do
Sample minibatch of new input spectrograms X,, Vi 1...;
Update Align Net by minimizing Lint end for for number of training iterations do
Sample a minibatch of new input spectrograms X,, Vi N
Update Align Net and Filter Net by minimizing free end for
Another way of RTF-aware training for the AFnet by combining the RTF loss and signal recon- struction loss together and jointly train the entire AFnet from scratch. In this sense, the RTF loss serves as a regularization term weighted by a scaler > 0. The training scheme depicted in Algorithm 2 Later in Section C.I we will show that this is less effective than Algorithm
Algorithm 2 RTF-Aware Training (via Regularization)
inputs
Model(X; E CPXT Vi L, N; Align Net, Filter Net)
Initialize Align Net, Filter Net for number of training iterations do
Sample a minibatch of new input spectrograms X;, Vi = I...,N Update Align Net and Filter Net by minimizing free + XLnt
end for
B.2 PRIMARY DATASET
Corpus: We use Audio- Visual Speech Dataset (AVSpeech) (Ephrat et al. 2018) available online. AVSpeech is large collection of video clips of single speakers talking with audio background interference. The audio files of the AVSpeech dataset serve as proper target speech files due to their variety (diversity of language, gender, age, etc.) The dataset can be found at: https://
We collected 8 types noise profiles from YouTube which are commonly seen sound sources in daily life. The noise types include blender, vacuum, washer, baby cry (for training) and dog barking, kids playing sound, hair dryer, food sizzling (for testing). Each type of noise profile contains recordings from different sound sources of the same type.
Noise data:
15

---

Under review as a conference paper at ICLR 2023
Data pre-processing: All the raw audio data are first converted to 16 kHz mono files before feeding to the room acoustic simulator to generate the multichannel data samples for experimenta- tion (the AVSpeech audio originally may contains or 2 channels of audio). For spectral processing, we use the Hann window with 512-point FFT and a hop size of 256 for the STFT. By conjugate sym- metry of the FFT, only half of the frequency bins (i.e., 257) are actually needed for the network to process the spectrograms.
Room acoustic simulator: Pyroomacoustics is utilized to generate multichannel audio which can be found at hbtpps: //gittub.
ics / https//eeepppp// /pypi -re lease in meters). Reverberation time is randomly
We create room of size 8 8 x chosen from {0.16, 0.32, 0.48, 3.64) second. The RIRs are estimated based on the image source method. 8 microphones are placed form planar array (4 columns, 2 rows) vertically, where the distance between adjacent microphones is set to 5 cm. The microphone array center is placed at the center of the room. Speech and noise signals are randomly positioned at a distance of {1. 2, 3, 4} meters from the microphone array center. In the experiments, the upper leftmost and rightmost microphones are used for the 2-mic scheme, the microphones at the four corners are used for the 4-mic scheme, and all microphones are used for the 8-mic scheme.
To further validate the proposed approach's generalization capabilities to more realistic scenarios, we also conduct experiments on multichannel au- dio data generated using real-world measured acoustic RIRs. A popular multichannel impulse response dataset is from the work by Hadad et al. (2014), where the impulse responses of several 8-channel microphone arrays with respect to various target source directions (from -90 90 degrees) are provided. We utilize the 3-3-3-8-3-3-3 array im- pulse responses of the dataset including reverberation time of 0.16, 0.36, and 0.61 second cases (https: : / //www www.:i.....ttttt......m.... wwwp:/ttaaaahheeeeeeeeeee http://uwtiittiiaaaaaaaaeeeeeeee http://////eeeeerccttccoolooo/// low.ooowssooo oads/] caabases/multtee /t///muliiii wwwllii-chaannee channel iwwuliiplieiieae response- watpooonsee-daaaaaeeeeaeee
Real-world measured RIRs:
together with the AVSpeech utterances and the YouTube noise profiles to generate the multichannel noisy data for experiments. The experimental results and related discussion are presented in Appendix C.7
databases/mma
Mixing speech with noise: We mix the multichannel speech files with noise profiles to generate noisy-clean data pairs. Each type of noise is multiplied with a scale randomly chosen from {0, 0.5, the clean speech according to a specific SNR level randomly selected from {-10, -6, -3, 0, 3, 6, 10} dB. In this way, we are adopting the supervised learning SE scheme of mix-and-separate, where the clean and noise signals separately collected and subsequently mixed together to become the noisy signals for training purposes. This mix-and-separate scheme by assuming additive noise model is still being used in the majority of SE works (not only signal processing- but also deep learning-basse To apply it to real data, one may collect the clean and noise recordings using the target device in real-world environments and mix them up generate noisy mixtures for training.
1, 1.5, 2) before added up, and the combined noise then added
approaches).
Network architectures:
AFnet: the implementation of AFnet (Figure 1 is based on the deep complex U-Net model at: phttps::/// / httpbb.ttbb..g//.//// com/ sweet cocoa//ccc///aaoooo DewpCoepeeeeeeeeeeeeeeeeeeeee www.et PyTrchy for both the Align Net and Filter Net parts. The number of layers and the number of features in each layer are modified to the one shown in Figure [ where C1, C2, Ca, C4 = 32, 64, 64, 64.
Cony-TasNet the experimental results of Conv-TasNet is based on the implementation from htttps://gg thub. cww. ka tooxu TaNNe using their own model settings instead of the original Cony-TasNe: Tas paper. Note that although the model hyper- parameters were designed based on kHz sampling rate, directly use them for 16 kHz sampling rate of our data without further modifications. Hyperparameters: N = 256, L 20, B 256, H 512, P = 3, X 8, R = 4, Norm=gINN, Noncausal.
DCUnet: the experimental results of DCUnet is based the implementation of the code at We choose the Large- DCUnet-20 20 model which realizes the 20-layer model in the original DCUnet paper. Note that although in the original system a 1024-point FFT was used, here we use 512 512-point FFT instead.
https: /
TorchSSooorrr
16

---

Under review as a conference paper at ICLR 2023
FaSNet:
the experimental results of FaSNet is based the implementation of the code wttp://w..w..o.//////e/./// We choose to use the FaSNet_TAC model Hyperparameters: enc dim=64,
at: where TAC stands for muunform-average-commeeee feature dim=64, hidden dim=128, layer=4, segment size=50, nspk=1, win len=4, con- text_len=16, sr=16000.
Model optimization: All experiments CUDA memory.
NVIDIA Tesla V100 GPU of 32 GB
the AFnet is trained via the RTF-aware training scheme. The Adam optimizer is used for minimizing Lut with a total of 80 epochs, where for the first 50 epochs learning rate of 0.001 is used and for the For minimizing Lree we train the network for another 55 epochs with Adam, where for the first 50 epochs learning rate of 0.001 used and for the rest 5 epochs is decreased to 0.0001. The total number of epochs is 80+55=135. A batch size of is used.
AFnet training:
30 epochs it is decreased 0.0001.
AFnet training (w/o RTF supervision): the AFnet trained without RTF loss is just mini- mizing Lies. In this case, the Adam optimizer with learning rate of 0.001 is used, which is decreased to 0.0001 at the 50-th epoch, and with total of 80 epochs. A batch size of is used.
Cony-TasNet, DCUnet, and FaSNet training: for fair comparison purposes, the training of the SOTA methods is same by minimizing the reconstruction loss f using 51 512-point FFT with hop size of 256 via the Hann window (same STFT setting AFnet). For the time-domain approaches (Conv-TasNet and FaSnet), the waveform of the model output is converted STFTs for computing the loss. In all cases, the Adam optimizer with learning rate of 0.001 is used, which is decreased to 0.0001 at the 50-th epoch, and with total of 80 epochs. A batch size of used.
B.3 CHIME-3 DATASET
The publicly available CHiME-3 dataset (Barker et al. 2015). made available as part of speech separation and recognition challenge, is used for training and evaluating SE performance. The dataset is a 6-channel microphone recording of talkers speaking in a noisy environment, sampled 16 kHz. It consists of 7,138 and 1,320 simulated utterances with average length of seconds for training and testing, respectively.
Dataset:
SOTAs to compare:
Neural BF (Erdogan et al. 2016): An MVDR beamforming with mask estimation through AndirectionalLLTTe
MVDRGC (Bu et al.) 2019): MVDR beamforming using a neural network-based method to identify and correct phase emror in the steering vector.
rSDFCN (Luu et al. 2020): A time-domain, fully convolutional network (FCN) with sinc and dilated convolutional layers for multichannel SE.
CA Dense U-Net (Tolooshams et al. 2020): A time-frequnecy domain multichannel SE model that combines the merits of DenseNet, U-Net, and channel attention (CA) mecha- nism.
IC Conv-TasNet (Lee al. 2021 The Inter-Channel (IC) Cony-TasNet is the extension of the time-domain Conv-TasNer for single-channel SE to the multichannel SE case.
Spectral processing: We use the Hann window with 1024-point FFT and a hop size 256 for the STFT. By conjugate symmetry of the FFT, only half of the frequency bins (i.e., 513) actually needed for the network to process the spectrograms.
Model optimization:
The AFnet training was run on one NVIDIA Tesla V100 GPU of 32 GB CUDA memory using the same model optimization scheme discussed in Section B.2
17

---

Under review as a conference paper at ICLR 2023
B.4 EVALUATION METRICS
PESQ:
a speech quality measure using the wide-band version recommended in ITU-T P.862.2.
basically models the mean opinion scores (MOS) that cover scale from 1 (bad) to 5 (excellent). We use the Python-based PESQ implementation from: https:///ithub: com/ cwtpt//ooooooooloeeoooe ludlows/
python pesq
STOI:
a function that well represents the average intelligibility of the degraded speech. provides value from 0 to 1, which be interpreted the percentage of correctly recog- nized words by normal-hearing people. We use the Python-based STOI implementation from: https: htaps:///ithub.... wwwcc///////aae hotp://////mmmaieee/////sssse
SSNR: an SNR measure, instead working on the whole signal, that calculates the average of the SNR values of short segments (segment length 30 msec, 75% overlap, SNR 10 dB, SNR = 35 dB). We use the Python-based SSNR implementation from: https: /ttps://github.
com/ haan/scchmiph//pyyeeee
APPENDIX C ADDITIONAL EXPERIMENTAL RESULTS
C.I COMPARISON OF RTF-AWARE TRAINING SCHEMES (TWO-STEP VS. REGULARIZED)
In this section we try to answer whether the two-step RTF-aware training scheme (Algorithm) I or the regularized training scheme (Algorithm 2 is better. To this end, train the AFnet model with several values of Ả using Algorithm 2 and the results are compared to AFnet trained with Algorithm iin Table 5 It can be seen that both training schemes help improve over the RTF-agnostic results (l.e., À = 0) as they both incorporate RTF information for explicitly learning spatial characteristics. However, the two-step approach performs consistently better than the regularized approach. The results suggest that learning to align prior to learning denoise during the training stage is more beneficial.
Table 5: SE performance comparison of RTF-aware training schemes for AFnet.
PESQ 4mm
STOI
SSNR
Method
2-mic
8-mic
2-mic
4-mic
8-mic
2-mic
4-mic
8-mic
Algorithm
1.84
1.92
1.99
0.728
0.739
0.753
5.06
5.26
5.35
Algo Algorithm
= 0
1.80
1.72
1.88
0.720
0.711
0.734
4.99
4.66
5.05
Al Agorith
À 0.0001
1.77
1.77
1.88
0.718
0.714
0.735
4.80
4.76
5.10
Algorithm
À 0.001
1.80
1.73
1.88
0.721
0.714
0.735
4.98
4.64
5.10
Algorithm
0.01
1.78
1.82
1.84
0.718
0.729
0.726
4.91
4.94
4.98
Algorthh
0.1
1.82
1.82
1.95
0.723
0.726
0.744
5.09
4.94
5.28
Algorithm
1.76
1.79
1.88
0.713
0.720
0.731
4.81
4.89
5.13
C.2 FIXING VS. UNFIXING ALIGN NET WHILE MINIMIZING RECONSTRUCTION LOSS
We compare two additional training schemes of the AFnet fixing or unfixing the Align Net parame- ters after has been trained via the RTF loss Lut, when we train the full AFnet for the reconstruction loss L Table with the Filter Net parameters for optimizing the reconstruction loss is crucial for achieving better performance compared to fixing the Align Net after the RTF loss minimization phase. This indicates that both stages may share the task performing spatial and spectral denoising jointly, where the first stage shares more loading the spatial denoising part while the second stage is more the spectral filtering portion. Dividing the SE task into the two subtasks makes the network learn to more efficiently denoise the speech.
the results. It can be seen that adjusting the Align Net parameters together
18

---

Under review as a conference paper at ICLR 2023
Table 6: SE performance comparison between using fixed (F) and unfixed (U) Align Net for training the AFnet after the Align Net has been trained with the RTF loss.
PESQ
STOI
SSNR
# Mic
F
F
2
1.77
1.84
0.712
0.728
4.93
5,06
1.81
1.92
0.721
0.739
5.01
5.26
8
1.84
1.99
0.728
0.753
4.99
5.35
C.3 VISUALIZING THE DATASET SPATIAL DIVERSITY
Given the ability of AFnet to exploit spatial separability, is also interesting utilize the AFnet for inspecting the spatial diversity the dataset. Such information can provided by observing alignment masks of the data points. In Figure we compare the distributions of the intermediate alignment masks Iearned by the AFnet models trained on the primary dataset (high spatial diversity) and the CHiME-3 dataset (low spatial variety). Here, we plot the distributions of the alignment masks estimated by the trained models for all the test data samples of the two datasets. It can be seen that for the primary dataset, the RTF--warr training leads to more widespread distribution than that of the normal training case, corresponding to the greater spatial diversity of the target speech source positions. In contrast, in the CHiME-3 dataset the RTF-aware training actually results in more aggregated cluster than the normal training scheme, indicating that the target speech sources are coming from similar directions. This visualization of the alignment masks provides another angle to inspect the data properties from the spatial diversity aspect for exploring SE performance.
Primary Datasel
CHIM-33 Datasel
RTF-aware training
KTF-aware traning
10
Normal training
30
Normal training
20
10
PC3
-10
-20
-30
-5
10
-10
10
20
30
PC-2
Figure 5: Visualization of the distributions of alignment masks. (Left): the estimated masks for the primary dataset using AVSpeech audio with Pyroomacoustics. (Right): the estimated masks for the CHiME-3 dataset.
C.4 ENIIANCEMENT PERFORMANCE VS. INPUT SNR CONDITION COMPARISON
To look more into the testing results, we present the PESQ, STOI, and SSNR performance numbers vs. input SNR conditions in Figure for for the three schemes:
AFnet: the two-stage AFnet model trained with the RTF-aware training scheme
ii) AFnet w/o RTF loss: the two-stage AFnet model trained with normal training scheme without incorporating RTFs
iii W-Net for filter weights: using W-Net (same model the AFnet, but the output of the first U-Net block goes directly to the next U-Net block without multiplying the input signals) to directly estimate the sets of filter weights W. in typical multichannel SE
We take 160 utterances for each specified input SNR value from the test set and obtain the cor- responding performance number of each input SNR value. The results for the 8-mic systems presented. From the results we can see that the proposed AFnet consistently improves the baseline method over wide range of input SNR conditions.
19

---

Under review as a conference paper at ICLR 2023
Afrst
sinrtwjj
W.NE worlee
16-
WeNett fr wenggrts
1630
0.3
1ro44.5311
1100. 3615
Figure 6: SE performance vs. input SNR conditions.
C.5 GENERALIZATION OTHER ROOM CONFIGURATIONS
To further test the generalization of the proposed AFnet unseen room configurations, utilize the Pyroomacoustic simulator to generate test data randomly sampled from three different room sizes: (8 x 5 3), (7 x 6 4), and (8 6 3) in terms of (length x width widtthxheigh)) in meters, which are different from the training room (i.e., (8 x x 3)). Figure 7 presents the SE results under the unseen room configurations. We schemes as still improves over the typical direct filter weight estimation approaches in most cases. Moreover, the performance of AFnet consistently improves as the number of microphones increases.
the the proposed AFnet generalizes well to unseen room
5.35-
2.0
Afreb Afret
Af-iet solo KIP
Afrett RIF
Afrizt
Africt
1.0
X
8
5.00
0.72
1.7
3.5
2.9h
4PM
6.9%
2.9%
4.5r
4.50
2.5g
6.Vi
Figure 7: SE performance generalization unseen room configurations.
C.6
SE UNDER TIME-VARYING RIR SCENARIOS
The above results have been tested on a relatively static environments where the target speech posi- tion of an utterance, once given, is changing. However, in most real-world scenarios the speaker may naturally move around and therefore the RIRs will not always be time-invariant, To test the proposed AFnet under such circumstances, we leverage the Pyroomacoustic simulator to generate another set of test data (same amount of data as the static RIR case in the previous sections) in which during an utterance the RIRs are changed from one set to another emulate speaker movements. We evaluate the algorithms
such time-varying RIR generated multichannel dataset and the results are presented in Figure g Note that the testing is performed on the same models that have been trained only with the data generated with static RIRs before. From the results
that the proposed AFnet has better tracking ability to tackle time-varying target speech locations, even though the model has not observed any changing RIR data during the training stage. This indicates that the RTF supervision of the alignment masks is also crucial for SE under time-varying RIR environments by incorporating phase alignment into the model learning, the positional information of the target speech can be better captured for improved SE even under the moving target scenario.
C.7 RESULTS ON REAL-WORLD MEASURED RIR GENERATED DATA
The results presented in previous sections on the primary dataset are based on the multichannel audio data synthesized by using the image source method via the Pyroomacoustics library, which is useful for methodological study but also has its limitation to represent real data. To further validate the proposed approach's generalization capabilities
realistic scenarios, also conduct
20

---

Under review as a conference paper at ICLR 2023
3.9
Afrel wis HIT hoa
Uivee the weiteee Whhsttfor four weight
0.71
fiter gfte
Hilt me
5.-
Uive then Pe Whhssffort tou weighta Afral ars IiT hoa. Afreet
Alleet
liher
1.5
0.72
3.74
0.10
1.6
0.48
4Hc
B.Mcc
Figure SE performance under time-varying RIR scenarios.
experiments multichannel audio data generated using real-world measured acoustic RIRs taken from the work of Hadad et al. (2014), where the impulse responses of several 8-channel linear microphone arrays with respect to various target source directions of arrival (DoAs) (from -90 to 90 degrees) are provided. We utilize the 3-3-3-8-3-3-3 array impulse responses of the dataset including reverheration time of 0.16, 0.36, and 0.61 second cases, together with the AVSpeech utterances and the YuuTube noise profiles to generate the multichannel noisy data for experiments. To observe the relation the AFnet performance with the speech source spatial diversity, we generate a spatially diverse dataset where the speech DoA ranges from -90 to 90 degrees, and another spatially constant dataset where the speech comes from only one direction.
Table 7 and Table 8 present the results for the two datasets respectively. One the spatially diverse dataset of Table 7 the AFnet clearly benefits from the sequentially masking mechanism and the RTF loss supervision, while for the spatially constant dataset of Table the three models perform comparably with the RTF loss marginally improves AFnet. These results show that model (such as the W-Net) that performs well on dataset lacking spatial variety of the target speech may not generalize well to a spatially diverse dataset, unless further attention is paid to exploiting spatial characteristics of the speech sources. The RTF information leveraged by the two- stage sequential masking design of the AFnet is shown to take advantage of the spatial separability for enhanced SE performance on spatially diverse datasets.
that for for
Table 7: Results on spatially diverse multi- channel data generated using measured acous- tic RIRs where the target speech DoA range is [-90, 90] degrees.
Table 8: Results on spatially constant multi- channel data generated using measured acoustic RIRs where the target speech DoA is fixed one direction.
Methods
PESQ STOI
SSNR
Methods
PESQ
STOI
SSNR
Noisy
1,40
0.598
-0.91
Noisy
1.42
0.609
-0,51
W-Net for filter weights
1.89
0.693
3.20
W-Net for filter weights
1.93
0.708
3.81
AFnet w/o RTF Loss
0.728
3.83
AFnet w/o RTF Loss
1.93
0.702
3.95
AFnet
2.22
0.759
4.16
AFnet
1.99
0.717
3.40
Similar to what have been done Figure 3 for the simulated RIR case, we also visualize the es- timated alignment masks for the spatially diverse dataset of the realistic RIR case. The results in Figure P once again show that the RTF-aware training results in clear clusters of the speech sources coming from difference directions compared to the normal training case, confirming that the network is more aware of the spatial separability of the sound sources.
APPENDIX D ALIGN NET VS. CONVENTIONAL SIGNAL PROCESSING APPROACHES FOR RTF ESTIMATION
In this section, we study the effectiveness of the deep leaning-based Align Net for estimating the RTFs compared to conventional model-base signal processing methods in the presence of noise. We show that the Align Net performs reasonably well in the low SNR regime where signal processing approaches fail to, justifying its usage of alignment purposes for later enhancement processing in Annet.
21

---

Under review as a conference paper at ICLR 2023
With RTF Supervision
Normal Training
speech at
speach at loc
speech al
-1
-2
4
-2
0 PC-2
2
-2 -1 0
2
PC-2
Figure 9: Visualization of the distributions of alignment masks for the spatially diverse data gen- erated using weal-world measured acoustic RIRs. Again, the RTF-aware training case (Leftt) shows obvious clusters of signals coming from the three different locations (loc 1, loc 2, loc 3), while the normal training case (Right) does not present clear boundaries.
D.1 RTF ESTIMATION PROBLEM
RTFs or their time-domain counterparts, relative impulse responses (RelRs) are important tools in several multichannel audio processing tasks (Gannot & Cohen 2004 Laufer et al. 2013). The estimation of RTFs has been studied for a while in the audio signal processing field (Gannot et al.) 2001 Koldovský et al. 2015 Giri et al., 2016 2018 Srikrishnan al. 2018), where the task is to estimate the correlation between the target source components received by the two microphones, either in terms of RTFs or RelRs, given the noisy recordings. Following the previous works, will discuss the scenario with two microphones (i.e., left and right microphones) while the idea may be generalized more microphones.
Consider a two-channel noisy recording of a target speech in a noisy environment, whose position fixed for a certain time interval. This situation can be represented as:
(n (n) s(n) (n),
(11)
(n) = hR(n)) s(n) +UR(n),
sR(n)
where is the time sample index taking values I,...,N N: are, respectively, the signals from the left (L) and right (R) microphones; hy and hR are the impulse responses between the target and the two microphones; is the far-field target speech; and ug and un are the noise components.
denotes the convolution; æy and zg
Let hyet represent the ReIR between the speech signal components arriving at the two microphones, we have following relation:
x (n) = (nee(n) * IR()) + (w)(n) - (nee(1)) * (R(n))
(12)
The main goal is to estimate hyef given IL and JR- The issue is in the presence of noise the estimation becomes challenging. The oracle ReIR is given as hyet = hr. * h where htz denotes the filter inverse to h. To ensure that the solution is causal, a fixed delay of a few milliseconds can
be introduced (Lin et al. 2007 Koldovský et al.) 2013),, i.c., hre! = hy hp * 8(n D). where 8(-) is the unit impulse function and D is the delay in samples. In the STFT domain, cquivalent to:
12 can be
Xz(f.t) Hra
((f(),
(13)
where the oracle RTF is given by:
Hre((f = HL(f)
(14)
HR(s)
22

---

Under review as a conference paper at ICLR 2023
The goal again is to estimate Hee given the microphone signals Xz and Xx. Note that under the assumption that the source position is fixed, we have the transfer functions independent of the time (thus omitting the index t).
D.2 METHODS
Signal processing-basee methods
Frequency-domain approaches: The RTF estimation problem be addressed in the frequency domain by utilizing signal and noise statistics. i.c., using the power spectral density (PSD) entities. deal with noise, Gannot al. (200T) propose the non-stationarity based frequency domain (NSFD) method relying on the assumption that noise signals are stationary, less dynamic, when compared to the target speech signal. To be more exact, let & (f((tt denote the (cross)-PSD between A and B during the (-th frame, we have:
xx, Xx, = (r,(())(xx (J: (v
(15)
Assuming over / = I..... P frames the the noise is stationary, we can write V (J1)) (f) and solve the overdetermined set of equations:
$ xx.xx
S (xxxxx(f1)
-
1
(16)
2 Xx ().
(f,) 1
In practice the PSDs the above set equations are replaced by their sample estimates.
Time-domain approaches: The ReIR estimation problem can be formulated in the time domain as: hyel = arg mR L(x1, X,,h,, where X [xz(1 - ,)....,I IN D)1"' and D is an integer delay for causality, and XR is the convolution matrix of dimen- sions N x M constructed from XR. When cost function C(::.) the squared Fuclidean distance, the problem can be solved by least squares (noise-free) or regularized least squares (noisy) methods. To better handle the noise, prior knowledge about the ReIR can be lever- aged to improve the estimation, c.g., utilizing the structural sparsity of ReIRs. One popular approach is the weighted & approach (Benichoux et al. 2014 Koldovský et al. 2015
Giri
et al. 2018):
h = arg min - x 2 + A|/w @ h||,, hER
(17)
where [w0,- .,,, War] is vector of non-negative weights and denotes the Hadamard product. To mimic the expected structure of ReIR, the weights are chosen as follows:
w = D 1,..., where ky, kg, and lig are positive constants and D is the integer delay.
(18)
Deep learning-basse approach (proposed Align Net): In the proposed AFnet, the first half Align Net is utilized to perform signal aligmment by estimating RTFs, which can be viewed as a deep learning-base RTF estimation method. By exploiting the power of complex-valued deep networks for predicting the RTFFs the data-driven Align Net approach can be shown to outperform conven- tional model-based signal processing methods under noisy environments.
D.3 EXPERIMENTS
Settings: We generate 2-mic noisy data using Pyroomacoustics library for conducting the RTF estimation experiments. We create room of size 8 x 3 (length x width in meters). Re- verberation time is randomly chosen from (0.16, 0.32, 0.48, 0.64) second. The simulating approach for estimating RIRs based on the image source method. The two microphones are placed with distance of 15 cm in between, where the center is placed at distance of I meter to one of the walls, with a height of 1.525 meters, and the array is positioned horizontally to the wall with equal distance to the two sides. Speech and noise signals are randomly positioned at distance of 21, 2, 3, 4} meters from the microphone array. 649 files from the training set of AVSpeech dataset are used for training the Align Net and 160 files from the testing are used for evaluating the competing
23

---

Under review as a conference paper at ICLR 2023
algorithms (i.e., Align Net, time-domain weighted l approach, and NSFD approach). 4 types of commonly seen interference (blender, vacuum, washer, baby cry) are used for training and another 4 types (dog barking, kids playing sound, hair dryer, food sizzling) of noise is multiplied with a scale randomly chosen from {0. 0.5, 1, 1.5, 2 before added up, and the combined noise is then added the clean speech according
used for testing. Each type
specific SNR level randomly selected from {-10, -6, -3, O, 3, 6, 10] dB for training and {0, 5, 10, 15} dB for testing.
For the fregency-domall point FFT and a hop size of 256 for the STFT. For Align Net we use Ci, C2 C3, C4 = 32, 64, 64, 64 and train for the RTF loss Lut. For NSDF we use 250 frames as a processing block and smoothing over 5 frames for computing the PSD entities. For the time-domaij weighted & we use L1 = 0.1, li2 = 0.11 and h3 0.3 with À= 0.0001; and a ReIR length of M 1024 for each processing block of 2048 samples.
approaches (Align Net and NSFD) we use the Hann window with 512-
Evaluation metric: To quantitatively evaluate the competing algorithms,, we use a well- cell-known and widely used performance metric called the Attenuation Rate (ATR) (Koldovský et al. 2015))
which can be evaluated as the ratio between SN Rout and SN Rin in dB scale, where:
Eie(L. (n) s(n)))
SN
(19)
EiELL, [n(o((n)
and
Rout Emlhrel(() # sR(n) s1(21
(20)
Enthrol UR(N) ur (n))
The numerator of SN Rout measures the leakage of the target signal whereas the denominator mea- sures the attenuation of the noise signal. The more negative the value (in dB) of ATR is, the better the evaluated algorithm performs.
Results: The ATR results are presented in Table can be seen that for all the methods the performance degrades SNR decreases, since it becomes more challenging estimate the RTF when noise is stronger. However, the proposed Align Net significantly outperforms the conventional model-based approaches over all SNR settings, indicating the effectiveness of the data-driven ap- proach for the RTF estimation task against existing methods. In future study, improving the design of Align Net for better RTF estimation as well its applications to other audio processing tasks (e.g., sound localization, speech separation, etc.) can be interesting topics to explore.
Comparison of ATR (in dB) of proposed Align Net vs. conventional signal processing- Table 9:
based methods. The Align Net gives much better estimation outcomes for all the SNR settings.
Input SNR
Methods
d
dB
10 dB
15 dB
Unprocessed
-1.33
-1.34
-1.20
-1.64
NSFD
-2.79
-3.79
-5.14
-5.78
Weighted &
-2.15
-3.99
-5.86
-6.64
Align Net (proposed)
-5.00
-6.27
-7.79
-8.63
24


Reviews:
[{"review_id": "dj91AnhLCAJ", "reviewer": "Reviewer_6rDM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new deep-learning-based method for multichannel speech enhancement.\nThe proposed method so called AFnet uses a W-net 2D architecture which considers as the input a multi-channel time-frequency representation and  which predict both the alignment mask and the filtering mask which minimize the reconstruction error.", "summary_of_the_review": "This is a well-written paper with reproducible results and a suitable evaluation methodology with freely available codes and dataset.\n\nHowever, the proposed work is only an incremental contribution of a previously published work from Yoyama et al, 2020.\n\nThe authors could provide more arguments to explain in what their contribution is significant in comparison to the state of the art from a  theoretical point of view.", "strength_and_weaknesses": "Strength:\n-well written paper\n-reproducible results (code and dataset)\n-technically correct\n-Convincing results\n\n\nWeaknesses\n-Not very original method", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SLwFfsw4-4", "reviewer": "Reviewer_4Sgp", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a two-stage model for multi-channel speech enhancement. The first stage predicts STFT domain filters for each microphone that will take them close to a reference microphone's clean signal, by estimating time-varying RTFs. This is called an alignment estimator. The second stage predicts further filters for each channel and filtered channels are summed up to obtain the estimated signal which needs to be close the target signal for the reference microphone. This second step can be seen as a typical deep beamforming network.\n\nThe model for each stage is a W-net model which is like a U-net model but uses complex matrix multiplications and convolutions.", "summary_of_the_review": "The paper is nice and innovative but it has some flaws in experimentation as listed in the weaknesses above.\n\nSome specific issues are highlighted below:\n\n1. In Figure 1, one of the align filters are always all ones (the one corresponding to the reference mic) and does not need to be estimated. It should be noted.\n2. In Section 3, the discussion assumes a time-varying single frame STFT domain filtering to describe room impulse response filtering. However, typically RIRs are much longer than a single frame and we may assume they are time-invariant if there is no motion.\n3.  Instead of SSNR, maybe report SI-SNR which is more commonly used. If using SSNR, maybe mention the segment length.\n4. In Section 4.2.3, it seems obvious that alignment masks would give more spatially diverse coefficients when there is an RTF loss on them. When there is no such loss, it is not clear what those initial filters mean and how they contribute to overall system.", "strength_and_weaknesses": "Strengths:\n1. The idea seems novel.\n2. It seems the two-stage model could work well by dividing the beamformer calculation into two separate stages and having individual targets (RTFs) for filters of the first stage.\n3. It is nice that the method seems to be able to handle time-varying environments, but it has not been tested on such environments.\n\nWeaknesses:\n1. The experimentation is only on simulated data. Would the method work well for real data?\n2. Comparison with another sequential multichannel method [Wang 2021] would have been nice. In this method, a beamformed signal based on an initial neural estimate is fed into a second stage (and even more stages) which refines the neural network output, hence the model makes use of spatial information implicitly.\n3. The burden on the second stage is still high since it still needs to do some spectral denoising as well as spatial denoising.\n4. The training uses a single room configuration which is limited. Does it generalize to unseen rooms?\n5. U-net seems better than W-net itself (for mics > 2) in Figure 2. What if we used U-nets in AFNet instead of W-nets? It has been observed that using complex weights in a network is usually not beneficial, and it is better to use real valued weights within the neural network.\n\n[Wang 2021] Wang, Zhong-Qiu, et al. \"Sequential multi-frame neural beamforming for speech separation and enhancement.\" 2021 IEEE Spoken Language Technology Workshop (SLT). IEEE, 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GnKthloFkj", "reviewer": "Reviewer_jGjP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an align-and-filter structure for neural network-based multi-channel speech enhancement.\nThe paper claims that the proposed decomposition of a filter into an element-wise product of alignment gain and a filtering gain is critical for an improved performance. Furthermore, it is claimed that training with an RTF loss is critical for the success of the proposed method.", "summary_of_the_review": "The paper claims that the proposed two-stage network with an RTF-based loss is a better alternative to single-stage multi-channel processing systems. Experimental results are promising.\nHowever, there are several open questions which need to be addressed before publishing the paper.\n\n---\n\nThe authors addresses most of my comments in their responses, and I've updated the score to 6.", "strength_and_weaknesses": "I think the basic idea is sensible and relevant: adding the RTF-based loss to improve construction of a multi-channel filter. The presented results indicate that this has a considerable effect on the proposed method.\n\nSome weaknesses which should be addressed\n- \"To the best of our knowledge, the only work that incorporates RTFs for SE is by Wang & Wang (2018), in which, however, the RTF estimation is used as an intermediate step to assist the conventional MVDR beamformer only.\" -- Many DNN-based speech enhancement papers estimate RTFs for speech enhancement. For example, \"ADL-MVDR: All deep learning MVDR beamformer for target speech separation\" is a relatively recent example which estimates a steering vector, which is an RTF. It is true that, in this instance, the vector is used to estimate the filter.\n- Furthermore, very relevant work is the following paper: The PCG-AIID System for L3DAS22 Challenge (https://arxiv.org/abs/2202.10017). The system used there is using a structure very similar to the one used here (W-like, multichannel intermediate output and single-channel final output).\n- I think it should be clarified that the AlignNet is not doing only temporal alignment, but also level adjustment. Even more critical, the results from B.1 should be included in the main text. One of the main questions for me was if the magnitude equalization is really important for this work, and I expected the phase to be the important part. B.1 clearly gives the answer: phase adjustment is sufficient. It's therefore not clear why is this not included in the main text.\n- The above brings another questions. If phase alignment is enough, would it be suffice to have a phase alignment loss only for the final output mask, without using the intermediate mask? I would assume that this would be sufficient, and that having alignment in the middle of the W structure is not necessarily critical. However, it would be good to get clarification from the authors, and to understand if the two-stage structure is critical (or it's suffice to have a skip connection and a combined loss at the output).\n- Leaky ReLU is used for real and imaginary components independently. However, the slope is not defined anywhere in the paper. This is interesting, since it may mean the complex gain values are relatively strongly pushed to be in the first quadrant, which is not very intuitive.\n- Table 1 -- These results indicate that the proposed FilterNet fails on the original unaligned signals. Operating on aligned signals is expected to perform better, but FilterNet on unaligned signals fails completely. This is quite surprising, since multi-channel systems proposed in other papers work well without the two-stage process (align+filter). I'm curious if this failure could be related to the choice of nonlinearity (above).\n- Section 4.2.2 / Table 2 -- It would be interesting to connect these results to Table 1, e.g., what's the gap of the AFnet compared to perfectly adjusted signals from Table 1.\n- Some typos, e.g., \"Nerual BF\"\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "G97zEffnY1", "reviewer": "Reviewer_aM96", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes an autoencoder based multi-channel speech enhancement framework. Overall the work is applied and mostly empirical in nature. The proposed framework is based on existing well-established ideas from empirical deep learning. There are no theoretical insights and the contribution in terms of the combination of existing ideas is relatively marginal.", "summary_of_the_review": "While the problem addressed is interesting, the proposed ideas are just incremental and not novel enough for the ICLR main conference.\nAuthors are recommended to resubmit to a suitable workshop, but the question about the theoretical contribution is the main weakness of the paper.", "strength_and_weaknesses": "The Paper is very hard to read, where I found lots of details, equations, symbols being intermingled within inline text.\n\n- I couldn't find the motivation behind the proposed framework. Why the choices are made, how they impact the learning process, and what extra they provide over existing studies.\n- This is another classical example of an empirical deep-learning paper where authors have tried to combine various existing ideas without any theoretical backing.\n- The link between speech production/perception as well as multichannel setting with the choices made is clearly missing.\n\n- Authors mentioned that their work is [Inspired by the alignment concept in signal processing]. What is this alignment process? no reference is provided, neither any discussion on why this is important.\n- The alignment and filter network are very similar. Why not just train a single big network with the same capacity?\n  In experiments, authors have shown results with individual networks empirically, but it is unclear if the model capacity is similar or not.\n-  ILD and ITD correspondence to the magnitude and phase spectra is exploited using a complex neural network. \nWhile working in complex STFT domain using conventional methods is well understood, the same is not the case with DNNs. In-fact their are a many works which have shown that the same task can be performed entirely in time domain by using learned filterbanks modelling raw waveform directly in an end-to-end fashion.\nComplex networks are very difficult to optimize in general, especially for speech/audio tasks.\n- Authors are suggested to avoid MSE loss as existing works have highlighted the drawbacks and shown better performance by using STOI or its variants for enhancement tasks. \n- I fail to understand the rationale behind split training. Why not joint training right from scratch? May be some visualization based on the loss landscape of the overall model (in complex and real settings) and what the network is learning in terms of filter responses/saliency maps or geometric properties of network weights is highly encouraged for explainability.\n\n\nExperiments: \n- lots of details are missing, which makes it very hard to reproduce the experiments\n- given the empirical nature of the paper, Authors should submit the code and pre-trained models.\n- details about existing works with which comparison is made are missing. Why these were chosen? I only see FaSNet closely related to this work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject