# Multichannel Speech Enhancement in Vehicle Environment Based on Interchannel Attention Mechanism

Authors: Xueli Shen; Zhenxing Liang; Shiyin Li; Yanji Jiang
Journal: Journal of Advanced Transportation
 (2021)

---

## Abstract

Speech enhancement in a vehicle environment remains a challenging task for the complex noise. The paper presents a feature extraction method that we use interchannel attention mechanism frame by frame for learning spatial features directly from the multichannel speech waveforms. The spatial features of the individual signals learned through the proposed method are provided as an input so that the two-stage BiLSTM network is trained to perform adaptive spatial filtering as time-domain filters spanning signal channels. The two-stage BiLSTM network is capable of local and global features extracting and reaches competitive results. Using scenarios and data based on car cockpit simulations, in contrast to other methods that extract the feature from multichannel data, the results show the proposed method has a significant performance in terms of all SDR, SI-SNR, PESQ, and STOI.


---

## Body

## 1. Introduction

In the process of driving, the speech signals recorded by a microphone are often corrupted by reverberation and background noise, such as wind noise, engine noise, and tire noise, leading to considerable degradation in speech quality, particularly at low signal-to-noise ratios (SNRs) [1]. Speech enhancement technology can improve the speech quality of the interphone system and the ability of the speech recognition system. Multichannel enhancement in vehicle scenarios uses microphone arrays that are convenient and flexible for speech-enabled applications [2]. The multichannel structure could provide more spatial information from the interchannel data and better results than the signal channel.Although the technology of microphone array has been developed for a long time, multichannel speech enhancement is still a great challenge in the field of speech recognition. The methods can be divided into two categories: one is based on the frequency domain, and the other is based on the time domain. Researchers mostly use the frequency-domain methods, which are based on the short-time spectrum estimation. Chakrabarty and Habets [3] proposed a multichannel online speech enhancement method based on time-frequency masking. Convolutional recurrent neural network (CRNN) is used to estimate the mask, and the effects of the ideal ratio mask (IRM) and ideal binary mask (IBM) on the results are discussed. The results show that the method is robust to different angles of sound sources. In [4], a multichannel speech enhancement system based on a deep neural network is proposed. Firstly, the audio signal is transformed into the frequency domain by STFT, the time-frequency mask is estimated by DNN, and the multichannel Wiener filtering is performed by using the power spectral density of speech and noise. The experimental results show that the method is effective. A beamforming method different from the traditional DNN is proposed in [5]. The spectrum of each channel is mapped to the non-Euclidean space, usually using the phase information to improve real-time performance, and the graph neural network is used for end-to-end training. Compared with the existing methods, the experiment result is better. A time-domain beamforming method named FaSNet (Filter and Sum Network) suitable for the low delay is proposed in [6]. The author selects the reference channel for filtering, calculates the filter of other channels by the reference channel, and then adds the filtered speech of each channel as the denoised speech. The model size of the algorithm is small, and the performance is better than that of traditional beamforming methods. In [7], a streaming speech enhancement system is proposed, which adopts the Wave-U-Net framework, adds temporal convolution and attention mechanism into the encoding and decoding structure, and explores the history caching mechanism. This method achieves almost the same noise reduction effect as the nonstreaming model. The time-domain convolutional denoising autoencoders (TCDAEs) method is proposed in [8]. It is used to learn the mapping structure between noisy speech waveform and clean speech waveform and solve the problem of speech signal delay between different channels effectively. Compared with the traditional denoising autoencoder, the effect has been significantly improved.The multichannel speech enhancement model has the most significant advantage of obtaining abundant information between channels compared with the single channel. Therefore, for the multichannel model, the way that extracts the spatial features between channels more effectively becomes the key to achieving better performance. In [9], a multichannel convolution sum (MCS) is used to extract features between channels. On the contrary, in [9], inspired by the IPD [10] feature, the interchannel convolution feature (ICD) is proposed. The method is to perform one-dimensional convolution subtraction on a pair of microphones. Based on GCC-PHAT, [6, 11] considered the normalized cross-correlation (NCC) method, which uses cosine similarity to calculate the information between channels. All the above methods achieve better performance improvement for multichannel speech enhancement. To address the problem of speech enhancement in the car cockpit, this paper proposes a novel method based on interchannel attention mechanism frame by frame (IAF), which helps analyse the influence of each channel on speech signal by using the characteristic information of the channel. Moreover, the proposed method also explores interchannel relationships and achieves more information representation on channel structure. It provides a new idea for multichannel speech enhancement based on vehicle environment and can also be applied to smart homes, teleconference, and other scenes.The main contents of this paper are as follows: Section1 introduces the related research work in this field. The structure of the multichannel speech enhancement model based on IAF is proposed in Section 2. The algorithm performance in the vehicle environment is evaluated in Section 3. The experimental results of the algorithm on several microphone arrays are analysed and discussed in Section 4, and Section 5 draws the conclusion and points out the future of the research work.


## 2. Problem Formulation

The proposed method aims to obtain an accurate estimate of the features for all the channels of a single time frame, given the input feature representation of the corresponding frame. The multichannel speech enhancement process of vehicle data is divided into four successive steps. First, spatial features from multichannel data added context information is extracted by IAF. Then, the frame-level beamforming filters are estimated by a well-trained two-stage BiLSTM model using spatial features, and the original waveforms computed by 1-dimensional convolution, forNN>=2 microphones, N beamforming filters are estimated. Next, the filters are adopted to filter the noisy speech in every channel, thereby obtaining the beamformed speech. Finally, add the beamformed speech as the denoised speech. The detail is presented in the following sections. A block diagram of the proposed multichannel enhancement framework is shown in Figure 1.Figure 1
The overall process of multichannel speech enhancement model based on IAF. (a) The speech signal with context. (b) The original speech signal. “IAF” and “TsBiLSTM” denote the feature extraction and feature filter using the TsBiLSTM model, respectively.


### 2.1. Data Preprocessing

It is assumed that the input signal corresponding to each microphone is represented as (2). Here, the frame length is M, the frameshift is K∈0,M−1, the total length of the speech signal is l, and the total number of frames is Z:(1)Z=lK+2,(2)xti=xitK:tK+M−1,t∈0,Z,i=1,…,N,where t is the frame index value and i is the channel index. xti∈ℝ1×M indicates that the signal vector of frame t is collected by microphone i.Due to the different distance between each microphone and the sound source, there is a time delay between the signals received by each microphone. Add a context window to make sure the model can capture interchannel delays of signal samples [12]. We add a group of contextual speech information in xti and define it as x¯ti:(3)x¯ti=xitK−W:tK+W+M−1,where W is the size of the context window and x¯ti∈ℝ2W+M is the signal vector of the microphone i containing the context information at frame t. The input sequence to these networks consists of W past and W future time frame.


### 2.2. Interchannel Attention Mechanism Frame by Frame

We calculate the corresponding weights of different parts of the channel by constructing the score function to describe the transmission characteristics of the signal in the channel. The principle of interchannel attention mechanism frame by frame is shown in Figure2.Figure 2
The module of the frame-level interchannel attention mechanism: different colors represent different weight values, and multiply the weight values with the original data.In order to extend the context informationx¯ti, firstly, average pooling is performed in the frame length dimension:(4)Fa=zti=12W+M∑j=tK−WtK+W+M−1x¯ij,i=1,…,N.zti∈ℝ1 is the average value of microphone i at the number of frames t. Then, the results are input into multiple fully connected layers:(5)Fb=Gt=SPzt1,zt2,…,ztN,Gt∈0,1.Gt∈ℝ1×N is the microphone array feature at the frame t. P∗ is a set of fully connected layers with parameter modified linear unit (PReLU) activation function, S∗ is a set of fully connected layers with sigmoid activation function, and the output of P∗ and S∗ are [128, 64, 128] and [N].\, respectively. Then, input Gt into the softmax activation function:(6)Fs=F¯t=SoftmaxGt,where F¯t is a vector whose sum is 1. The final output out is obtained by multiplying with F¯t and xt:(7)outti=F¯ti×xti,i=1,…,N.outti∈ℝ1×2W+M presents the speech feature sequence of the t -th frame data in the i -th channel.By using the attention mechanism frame by frame of the speech signal in multiple channels, the model could learn the characteristics of each channel and capture spatial features between channels more accurately.


### 2.3. Two-Stage BiLSTM Network

The two-stage bidirectional LSTM (TsBiLSTM) is used to derive a beamformer as BiLSTM is adopted to estimate the global feature. For the beamformer, the approach aims to improve the SNR without destroying the target speech.Figure3 shows the TsBiLSTM architecture employed in this work. We divide the data into blocks, consider using the BiLSTM network model to obtain local and global features of the blocks and establish the timing relationship of the signal, and use the residual connections to alleviate the gradient dispersion problem.Figure 3
The structure of the TsBiLSTM module and illustration of the proposed module architecture. (a) The input of the TsBiLSTM includes interchannel features and original waveform. (b) The processing chain shows the two-stage BiLSTM with residual connections. (c) Carry out 1 × 1 convolution on the output after overlap-add method operation, then operate conv1d layer with sigmoid and tanh activation function, respectively, and obtain beamforming filter.In this work, we combine the speech signal with context information in the first stage. The observed signal can be expressed as follows:(8)yt=GroupNormConv1dxt,(9)xbt=concatoutt,yt.xbt∈ℝN×2M+W represents the speech features of frame t and xb∈ℝZ×N×2M+W represents all the speech features; then, we do the one-dimensional convolution on xb:(10)c=Conv1dxb,where c∈ℝZ×N×M, then divide c into S blocks of the same size. Each block presents Bs∈ℝU×N×M,s∈1,S, and all the blocks will be connected to form a four-dimensional vector O∈ℝN×M×S×U.We transform the shapeO∈ℝN×M×S×U to O∈ℝS×N×U×M and input the first BiLSTM:(11)out=reshapeO,out1=GroupNormLinearBiLSTMout,out=reshapeout1+O.The output of BiLSTM passes through the linear layer and GroupNorm operation and then outputout1∈ℝS×N×U×M. Reshape out1∈ℝS×N×U×M to out1∈ℝN×M×S×U, add the vector using the residual connection to reduce the problem of gradient disappearance or gradient explosion, and finally obtain out∈ℝN×M×S×U.In the next stage, changeout∈ℝN×M×S×U into out∈ℝU×N×S×M, then input the next BiLSTM, as the first BiLSTM block, and finally, obtain out∈ℝN×M×S×U. Because the signals are transmitted to the BiLSTM model in different block forms, we can obtain the local and global features of the signals, respectively:(12)out1=reshapeout,out2=GroupNormLinearBiLSTMout1,out=reshapeout2+out.Then, use the overlap-add operation to convert the segmented block back to the original sequence:(13)out3=ODout,where out3∈ℝZ×N×M, OD⋅ is the overlap-add method, which means to restore the partitioned data. Then, convolve out3∈ℝZ×N×M in two dimensions with the convolution kernel of size set one:(14)out4=Conv2dout3,where out4∈ℝZ×N×M. We perform twice one-dimensional convolution operations on out4, then use the activation function of Tanh and Sigmoid, respectively, and multiply the results to get the filters for each channel:(15)h=TanhConv1dout4⊙SigmoidConv1dout4,where h∈ℝN×Z×2W+1, ⊙ is the Hadamard product symbol, Tanh⋅⊙Sigmoid⋅ is the gating mechanism of filter that controls the output data.Figure4 shows the structure of the BiLSTM block. The input layer is the feature vector of noisy speech with dimension 64, which is input into the BiLSTM layer with dimension 128. The output dimension is 256 since bidirectional LSTM is used. Then, input the linear hidden layer of 64, and get the output after the GroupNorm operation.Figure 4
The structure of the BiLSTM block. (a) The module contains BiLSTM, linear, and GroupNorm layers, where the numbers in parentheses indicate the size of the output dimension. (b) The internal structure diagram of the BiLSTM layer, where A refers to the LSTM module,x is the input data, s is the output of the hidden layer, and y is the output result BiLSTM.


### 2.4. Summation

Integrating the signals of multiple channels into one signal output is an important step in the multichannel speech enhancement problem. After passing the signals of each channel through the channel filter, the results obtained are summed and averaged, that is, the final enhanced speech signal:(16)y¯=1N∑i=1Nhi⊛x¯i,i=1,…,N,where y¯∈ℝN×Z×M and ⊛ is convolution operation. Ultimately, y¯ is inversed from segmentation into an enhanced speech waveform by overlapping.


### 2.5. Loss Function

In training and evaluation, the scale-invariant source-to-noise ratio (SI-SNR) is used as the loss function.(17)starget=x¯,xxx2,enoise=x¯−starget,SI−SNR=10log10starget2enoise2,where x¯ is the denoised speech and x is pure speech signal.


## 2.1. Data Preprocessing

It is assumed that the input signal corresponding to each microphone is represented as (2). Here, the frame length is M, the frameshift is K∈0,M−1, the total length of the speech signal is l, and the total number of frames is Z:(1)Z=lK+2,(2)xti=xitK:tK+M−1,t∈0,Z,i=1,…,N,where t is the frame index value and i is the channel index. xti∈ℝ1×M indicates that the signal vector of frame t is collected by microphone i.Due to the different distance between each microphone and the sound source, there is a time delay between the signals received by each microphone. Add a context window to make sure the model can capture interchannel delays of signal samples [12]. We add a group of contextual speech information in xti and define it as x¯ti:(3)x¯ti=xitK−W:tK+W+M−1,where W is the size of the context window and x¯ti∈ℝ2W+M is the signal vector of the microphone i containing the context information at frame t. The input sequence to these networks consists of W past and W future time frame.


## 2.2. Interchannel Attention Mechanism Frame by Frame

We calculate the corresponding weights of different parts of the channel by constructing the score function to describe the transmission characteristics of the signal in the channel. The principle of interchannel attention mechanism frame by frame is shown in Figure2.Figure 2
The module of the frame-level interchannel attention mechanism: different colors represent different weight values, and multiply the weight values with the original data.In order to extend the context informationx¯ti, firstly, average pooling is performed in the frame length dimension:(4)Fa=zti=12W+M∑j=tK−WtK+W+M−1x¯ij,i=1,…,N.zti∈ℝ1 is the average value of microphone i at the number of frames t. Then, the results are input into multiple fully connected layers:(5)Fb=Gt=SPzt1,zt2,…,ztN,Gt∈0,1.Gt∈ℝ1×N is the microphone array feature at the frame t. P∗ is a set of fully connected layers with parameter modified linear unit (PReLU) activation function, S∗ is a set of fully connected layers with sigmoid activation function, and the output of P∗ and S∗ are [128, 64, 128] and [N].\, respectively. Then, input Gt into the softmax activation function:(6)Fs=F¯t=SoftmaxGt,where F¯t is a vector whose sum is 1. The final output out is obtained by multiplying with F¯t and xt:(7)outti=F¯ti×xti,i=1,…,N.outti∈ℝ1×2W+M presents the speech feature sequence of the t -th frame data in the i -th channel.By using the attention mechanism frame by frame of the speech signal in multiple channels, the model could learn the characteristics of each channel and capture spatial features between channels more accurately.


## 2.3. Two-Stage BiLSTM Network

The two-stage bidirectional LSTM (TsBiLSTM) is used to derive a beamformer as BiLSTM is adopted to estimate the global feature. For the beamformer, the approach aims to improve the SNR without destroying the target speech.Figure3 shows the TsBiLSTM architecture employed in this work. We divide the data into blocks, consider using the BiLSTM network model to obtain local and global features of the blocks and establish the timing relationship of the signal, and use the residual connections to alleviate the gradient dispersion problem.Figure 3
The structure of the TsBiLSTM module and illustration of the proposed module architecture. (a) The input of the TsBiLSTM includes interchannel features and original waveform. (b) The processing chain shows the two-stage BiLSTM with residual connections. (c) Carry out 1 × 1 convolution on the output after overlap-add method operation, then operate conv1d layer with sigmoid and tanh activation function, respectively, and obtain beamforming filter.In this work, we combine the speech signal with context information in the first stage. The observed signal can be expressed as follows:(8)yt=GroupNormConv1dxt,(9)xbt=concatoutt,yt.xbt∈ℝN×2M+W represents the speech features of frame t and xb∈ℝZ×N×2M+W represents all the speech features; then, we do the one-dimensional convolution on xb:(10)c=Conv1dxb,where c∈ℝZ×N×M, then divide c into S blocks of the same size. Each block presents Bs∈ℝU×N×M,s∈1,S, and all the blocks will be connected to form a four-dimensional vector O∈ℝN×M×S×U.We transform the shapeO∈ℝN×M×S×U to O∈ℝS×N×U×M and input the first BiLSTM:(11)out=reshapeO,out1=GroupNormLinearBiLSTMout,out=reshapeout1+O.The output of BiLSTM passes through the linear layer and GroupNorm operation and then outputout1∈ℝS×N×U×M. Reshape out1∈ℝS×N×U×M to out1∈ℝN×M×S×U, add the vector using the residual connection to reduce the problem of gradient disappearance or gradient explosion, and finally obtain out∈ℝN×M×S×U.In the next stage, changeout∈ℝN×M×S×U into out∈ℝU×N×S×M, then input the next BiLSTM, as the first BiLSTM block, and finally, obtain out∈ℝN×M×S×U. Because the signals are transmitted to the BiLSTM model in different block forms, we can obtain the local and global features of the signals, respectively:(12)out1=reshapeout,out2=GroupNormLinearBiLSTMout1,out=reshapeout2+out.Then, use the overlap-add operation to convert the segmented block back to the original sequence:(13)out3=ODout,where out3∈ℝZ×N×M, OD⋅ is the overlap-add method, which means to restore the partitioned data. Then, convolve out3∈ℝZ×N×M in two dimensions with the convolution kernel of size set one:(14)out4=Conv2dout3,where out4∈ℝZ×N×M. We perform twice one-dimensional convolution operations on out4, then use the activation function of Tanh and Sigmoid, respectively, and multiply the results to get the filters for each channel:(15)h=TanhConv1dout4⊙SigmoidConv1dout4,where h∈ℝN×Z×2W+1, ⊙ is the Hadamard product symbol, Tanh⋅⊙Sigmoid⋅ is the gating mechanism of filter that controls the output data.Figure4 shows the structure of the BiLSTM block. The input layer is the feature vector of noisy speech with dimension 64, which is input into the BiLSTM layer with dimension 128. The output dimension is 256 since bidirectional LSTM is used. Then, input the linear hidden layer of 64, and get the output after the GroupNorm operation.Figure 4
The structure of the BiLSTM block. (a) The module contains BiLSTM, linear, and GroupNorm layers, where the numbers in parentheses indicate the size of the output dimension. (b) The internal structure diagram of the BiLSTM layer, where A refers to the LSTM module,x is the input data, s is the output of the hidden layer, and y is the output result BiLSTM.


## 2.4. Summation

Integrating the signals of multiple channels into one signal output is an important step in the multichannel speech enhancement problem. After passing the signals of each channel through the channel filter, the results obtained are summed and averaged, that is, the final enhanced speech signal:(16)y¯=1N∑i=1Nhi⊛x¯i,i=1,…,N,where y¯∈ℝN×Z×M and ⊛ is convolution operation. Ultimately, y¯ is inversed from segmentation into an enhanced speech waveform by overlapping.


## 2.5. Loss Function

In training and evaluation, the scale-invariant source-to-noise ratio (SI-SNR) is used as the loss function.(17)starget=x¯,xxx2,enoise=x¯−starget,SI−SNR=10log10starget2enoise2,where x¯ is the denoised speech and x is pure speech signal.


## 3. Experiment Section

The speech enhancement tasks are evaluated in four kinds of microphone array structures to simulate the location of the microphone in the car. The speech source and locations of the noise source are shown in Figure5, where the black circle represents the microphones, the green square represents the speech source, and the red five-pointed star represents the noise source. The design of the microphone array is as follows:(i) C
onsider a uniform linear array with 2 microphones with intermicrophone distance of 3 cm, and the microphone array is located in the front of the car cockpit, as Figure5(a)(ii)
Consider a uniform linear array with 2-uniform linear distributed 2-channel microphone array with intermicrophone distance of 3 cm, and the microphone array is located in the front and middle of the car cockpit, respectively, as shown in Figure5(b)(iii)
Consider a uniform linear array with 4 microphones with intermicrophone distance of 3 cm, and the microphone array is located in the front of the car cockpit, as shown in Figure5(c)(iv)
Consider a distributed array with 4 microphones with intermicrophone distance of 80 cm, and the microphone is located around the car cockpit, as shown in Figure5(d)Figure 5
The distribution map of four microphone arrays. (a) Uniform linear 2-channel microphone array. (b) Two-uniform linear distributed 2-channel microphone array. (c) Uniform linear 4-channel microphone array. (d) Distributed 4-channel microphone array.
(a)(b)(c)(d)Different microphone array structures can reflect different spatial characteristics. In order to make the method independent of the spatial position of the required speech source, each microphone array position and source-array distance are considered under the training condition.


### 3.1. Datasets Building

For training, we used 3000 randomly chosen speech utterances from the LibriSpeech [13] dataset which are open and well-studied dataset used for speech enhancement, each 4 s long, with sampling frequency of 16 k Hz, and 500 were used as a validation set. Volvo car noise [14] was added to the training data as noisy speech in the car cockpit with randomly chosen SNRs between −10 dB and −5 dB. Additionally, since the number of noise is small, spsquare noise [15] as a noise source, with randomly chosen SNRs between −10 dB and −5 dB, was also added. All dataset are divided into frames with 64 sampling points length, 50% overlapping, and the context window is 256.To simulate a car cockpit, we designed the space size to be 3.4 meters long, 1.8 meters wide, and 1.4 meters high. The vehicle cockpits impulse responses required to simulate real acoustic conditions are generated by gpuRIR toolbox [16], with the reverberation time (T60) selected from 0.1 seconds to 0.3 seconds randomly.


### 3.2. Experiment Settings

The experiment aims to verify the generalization capability of the proposed method over different microphone arrays and compare the performance to that of traditional beamformers. For a fair comparison, we make the comparison as all models, including NCC, MCS, and ICD, are based on the same two-stage BiLSTM modules presented in Section2.3 for each microphone channel. The architecture of each BiLSTM network consists of 128 hidden layers. Set layer number 4. For MCS and ICD, the size of the convolution kernel is 64, the number of convolution kernels is 16, the step size is 2, and the expansion number is 2, leading to the output where the filter estimate for each microphone is obtained.The BiLSTM network was trained using the Adam-based optimizer, with minibatches of 128 input signals and a learning rate of 0.001. Meanwhile, the L2 norm of 5 is used for gradient pruning to prevent gradient explosion. During training, if the loss value of the latest 10 epoch model does not decrease on the validation sets, the training will stop automatically. Dynamic strategy warmup [17] is used to adjust the learning rate during the training. This operation can warm up the model at a small learning rate in the initial stage to increase the stability of the model and then gradually reduce it with a decay rate of 0.98 every 2 epochs. The specific approach is similar to [18]. All the implementations were done in PyTorch:(18)lr=a1⋅n⋅dmodel−0.5⋅n_warmups−1.5,n≤n_warmups,lr=a2⋅0.98epoch/2,n>n_warmups,where n is the number of training steps anda1, a2, warmups, and dmodel present the hyperparameter. In the experiment, we set a1=0.2, a2=1e−3, n_warmups=4000 , and dmodel=64.


## 3.1. Datasets Building

For training, we used 3000 randomly chosen speech utterances from the LibriSpeech [13] dataset which are open and well-studied dataset used for speech enhancement, each 4 s long, with sampling frequency of 16 k Hz, and 500 were used as a validation set. Volvo car noise [14] was added to the training data as noisy speech in the car cockpit with randomly chosen SNRs between −10 dB and −5 dB. Additionally, since the number of noise is small, spsquare noise [15] as a noise source, with randomly chosen SNRs between −10 dB and −5 dB, was also added. All dataset are divided into frames with 64 sampling points length, 50% overlapping, and the context window is 256.To simulate a car cockpit, we designed the space size to be 3.4 meters long, 1.8 meters wide, and 1.4 meters high. The vehicle cockpits impulse responses required to simulate real acoustic conditions are generated by gpuRIR toolbox [16], with the reverberation time (T60) selected from 0.1 seconds to 0.3 seconds randomly.


## 3.2. Experiment Settings

The experiment aims to verify the generalization capability of the proposed method over different microphone arrays and compare the performance to that of traditional beamformers. For a fair comparison, we make the comparison as all models, including NCC, MCS, and ICD, are based on the same two-stage BiLSTM modules presented in Section2.3 for each microphone channel. The architecture of each BiLSTM network consists of 128 hidden layers. Set layer number 4. For MCS and ICD, the size of the convolution kernel is 64, the number of convolution kernels is 16, the step size is 2, and the expansion number is 2, leading to the output where the filter estimate for each microphone is obtained.The BiLSTM network was trained using the Adam-based optimizer, with minibatches of 128 input signals and a learning rate of 0.001. Meanwhile, the L2 norm of 5 is used for gradient pruning to prevent gradient explosion. During training, if the loss value of the latest 10 epoch model does not decrease on the validation sets, the training will stop automatically. Dynamic strategy warmup [17] is used to adjust the learning rate during the training. This operation can warm up the model at a small learning rate in the initial stage to increase the stability of the model and then gradually reduce it with a decay rate of 0.98 every 2 epochs. The specific approach is similar to [18]. All the implementations were done in PyTorch:(18)lr=a1⋅n⋅dmodel−0.5⋅n_warmups−1.5,n≤n_warmups,lr=a2⋅0.98epoch/2,n>n_warmups,where n is the number of training steps anda1, a2, warmups, and dmodel present the hyperparameter. In the experiment, we set a1=0.2, a2=1e−3, n_warmups=4000 , and dmodel=64.


## 4. Results and Discussion

Following the common speech enhancement metrics, we adopt average SI-SNR, SDR, PESQ, and STOI improvement to evaluate the performance of multichannel speech enhancement. For a more comprehensive evaluation of the speech quality, we also report the performances under different SNRs of speech and noise to give a more comprehensive model assessment. The experimental results are summarized in Table1, where the highlighted numbers with black are the best scores for each model. The results indicate that the performance of proposed method is better than other methods when tested at different SNRs, which verifies the effectiveness of the model. By assigning weight values to each channel frame by frame, using attention mechanism to learn the feature expression between channels, the proposed method leads to the best improvement in terms of four metrics. It learns from the magnitude spectrum and phase spectrum of the individual microphone signals and exploits the difference in the spatial characteristics of the speech and noise sources.Table 1
Evaluation results of our proposed model compared with other methods on the same dataset. Four metrics and two SNRs are considered.
Method# MicsSDRSI-SNRPESQSTOI−10 dB−5 dB−10 dB−5 dB−10 dB−5 dB−10 dB−5 dBNoise2 linear−6.720.34−7.190.241.071.130.400.644 linear−5.930.11−6.400.021.061.110.400.642 × 2 dB−6.720.32−6.180.221.071.130.400.644 dB−6.890.68−7.170.601.071.130.460.70+NCC2 linear7.4511.986.4211.301.201.760.730.874 linear8.8912.447.8611.751.281.830.750.882 × 2 dB7.6313.147.5012.371.231.850.710.894 dB8.3814.587.6013.961.302.020.760.91+ICD2 linear7.4211.956.3911.271.191.750.720.864 linear8.8012.387.7911.681.271.820.740.872 × 2 dB7.5513.067.4412.301.221.830.710.894 dB8.3314.517.5613.931.272.010.760.91+MCS2 linear7.4011.926.3611.251.191.750.720.864 linear8.8312.407.8311.721.271.820.740.872 × 2 dB7.5213.047.4212.281.211.820.700.884 dB8.2814.467.5213.791.252.000.750.90Proposed2 linear7.5212.066.4811.361.221.770.740.884 linear8.9412.497.9111.831.291.830.750.882 × 2 dB7.7213.237.5812.451.241.860.720.904 dB8.4114.667.6614.071.332.040.760.91The best evaluation results are shown in bold, comparing the results of the four speech enhancement methods used in the four microphone arrays.In the four kinds of microphone array structures designed in the experiment, we obtain 13.60 dB improvement in SI-SNR in the structure of 2 microphones with −10 dB SNR and 14.76 dB improvement in SI-SNR in the distributed 4-microphone array structure.Another conclusion from the experimental results is that the array structure with four microphones is better than that with the two microphones, indicating that the more the channels are, the more the feature information can be provided to the speech enhancement model. In addition, compared with the other structures, the 4-channel distributed microphone array has the optimum performance. The SDR increase [15.24, 13.87], respectively, in SNR = −10 dB and −5 dB, and the performance improvement of the other structures are [14.16, 11.63], [14.79, 12.31], [14.32, 12.79], respectively. The distributed microphone array structure has advantages in obtaining the spatial characteristics of the entire cockpit due to the difference in the location of the speech source and the noise source, which is helpful to train a better beamforming filter.Figure6 is the speech spectrogram, including the pure speech spectrogram, the noisy speech spectrogram with SNR = −10 dB, and the speech spectrogram enhanced by four methods. The four methods have good noise reduction effects. Compared with the enhanced noise energy spectrum in the box, the method proposed in this paper has significant advantages. At the same time, compared with the enhanced speech spectrogram and pure speech spectrogram, the method did not cause speech damage and ensured the integrity of the speech signal.Figure 6
The spectrum of four methods under distributed 4-channel microphone array. (a) NCC method. (b) ICD method. (c) MCS method. (d) IAF method, where (i) represents clear speech, (ii) represents noisy speech with SNR = −10 dB, and (iii) represents denoised speech.
(a)(b)(c)(d)


## 5. Conclusions

This work proposed an interchannel attention mechanism frame by frame (IAF) method and jointed with the two-stage BiLSTM network to learn the spatial features directly from multichannel waveforms to solve the problem of multichannel speech enhancement in the car cockpit. Experimental results show the IAF method is more effective than the traditional NCC, MCS, and ICD method in learning spatial features directly from the multichannel speech waveforms. The proposed model based on four distributed microphone arrays obtains the optimal enhancement performance in terms of SDR, SI-SNR, STOI, and PESQ. The results indicated that the method is suitable for different structures of the microphone array and has good robustness. This work provided valuable conclusions for improving the performance of multichannel speech enhancement in the vehicle cockpit. In future work, we will explore the effect of the position of the voice source on the performance using the proposed method.



---

*Source: 9453911-2021-11-15.xml*