2020 RISP International Workshop on Nonlinear Circuits,
Communications and Signal Processing (NCSP 2020)
Honolulu, Hawaii, USA, February 28 - March 2, 2020

Underdetermined Multichannel Speech Enhancement Using
Time-Frequency-Bin-Wise Switching Beamformer and Gated CNN-Based Time-Frequency Mask
for Reverberant Environments
Riki Takahashi1 , Kouei Yamaoka2 , Li Li1 , Shoji Makino1 , Takeshi Yamada1 , Mitsuo Matsumoto1
1

University of Tsukuba, Japan
E-mail: {r.takahashi@mmlab.cs, lili@mmlab.cs, maki@tara, takeshi@cs, makilab-research@tara}.tsukuba.ac.jp
2
Tokyo Metropolitan University, Japan
E-mail: yamaoka-kouei@ed.tmu.ac.jp
bin-wise switching (TFS) beamformer, which applies the optimal filter selected from a set of predefined beamformers according to the minimum absolute value of outputs to each TF
bin, has been proposed for underdetermined speech enhancement. All of these methods can be interpreted as adopting
beamformers to multiple masked spectrograms summing up
to the observed one, which exploits the benefits of both the
beamforming technique and TF masking.
Taking account of its low computational cost and highnoise suppression performance, in this paper, we consider the
TFS beamformer. With the assumption that only the target
speech and one interferer exist at each TF bin, all N − 1interferers can be suppressed by constructing N − 1 beamformers and switching them among the TF bins. Note that
N can be larger than the number of microphones M . However, there are still interferer components remaining when theassumption does not hold, namely, multiple interferers exist
simultaneously at one TF bin. To further suppress these residuals, a TF mask constructed on the basis of the directionsof arrival (DOA) of the sources is employed for postprocessing [10]. It has been shown to be effective in improving theperformance of speech enhancement. However, the precision
1. Introductionof the constructed TF mask tends to decrease in highly reverIn recent years, with the development of automatic speech berant environments, where accurately estimating the DOA
recognition (ASR) and robot hearing, the importance of of sources becomes an extremely difficult task [12–14]. To
speech enhancement has considerably increased. Owing to address this drawback and increase the accuracy of mask estithe wide usage of stereo microphone built-in small devices mation, in this paper, we propose the use of a mask estimationsuch as smartphones and voice recorders, speech enhance- method based on deep neural networks for the postprocessing
ment that serves dual-channel signals is a particular requisite. of the TFS beamformer, where a TF mask is estimated usBeamforming and blind source separation (BSS) [1] are the ing monaural signals instead of multichannel signals. DNNtwo main methods to deal with this problem. Beamforming based TF masking [15,16] has recently gained much attention
and BSS methods using spatial filtering [2–4] are noteworthy because of its impressive performance in speech separation.
in the low distortion of the enhanced target speech. How- Specifically, we adopt deep clustering (DC) [17] with gatedever, their performance degrades when there are fewer micro- convolutional neural networks (CNNs) to take the advantage
phones than sources, i.e., underdetermined conditions.
that high mask estimation accuracy can be achieved with aOn the other hand, methods applying time-frequency (TF) remarkably small amount of training data [18].
masking or multichannel Wiener filters [5–8] can sufficiently
suppress noise and interferers under underdetermined condi- 2. Time-frequency-bin-wise switching beamformer
tions, while they are tend to distort the target speech. To
Let xi (ω, t) be the ith microphone signal at the angular
accomplish high noise suppression performance with low frequency ω in the tth time frame in the short-time Fourier
distortion under underdetermined conditions, some attempts transform (STFT) domain. With two microphones, a beamhave recently been made to combine beamforming with former is generally given by the following equations:masking techniques on the basis of the sparseness assumptiony(ω, t) = wH (ω)x(ω, t),(1)of speeches [9–11]. In [9], a limited amount of pretrained dicT
tionaries of mechanical noise covariance to construct the mostx(ω, t) = [x1 (ω, t), x2 (ω, t)] ,(2)
suitable filter in each TF bin so that the output includes fewerTnoise residuals. Similar to [9], in [10,11], the time-frequencyw(ω) = [w1 (ω), w2 (ω)] ,(3)
Abstract
To achieve high performance of speech enhancement withlow signal distortion under underdetermined conditions, thetime-frequency-bin-wise switching (TFS) beamformer has
recently been proposed. Its extension that applies a timefrequency (TF) mask constructed on the basis of direction ofarrival (DOA) estimation for postprocessing, has been shown
to further improve the performance. However, the precisionof the TF mask estimated using spatial information strongly
depends on acoustic factors such as reverberation. To address
this issue and improve the accuracy of the mask estimation,in this paper, we propose a novel method that combines the
TFS beamformer with a deep clustering (DC)-based TF mask,
which is estimated using monaural signals. We investigate theperformance of the TFS beamformer under various reverberation conditions. The experiments revealed that the proposed
method further improved the performance of the TFS beamformer in speech enhancement even under heavy reverberation, whereas the conventional method using DOA estimation
failed.

645

Table 1: Dominant sound patterns at each TF bin of different tion, the TFS beamformer selects the output only including
signals. tgt, i1 , and i2 stand for target speech, interferer 1, and the target speech. In the case in the 7th or 8th row of Table 1,
the TFS beamformer cannot suppress both interferers becauseinterferer 2, respectively.
there are M or more interferers in a TF bin, whereas the TFS
row
x
y1
y2
yTFS
yDC
beamformer suppresses only the most dominant interferer.
1
2
3
4
5
6
7

tgt
i1
i2
tgt, i1
tgt, i2
i1 , i2

tgt
0
i2
tgt
tgt, i2
i2

tgt
i1
0
tgt, i1tgt
i1tgt, i1 , i2tgt, i2tgt, i1

tgt
0
0
tgt
tgt
i1 or i2tgt, i1
or tgt, i2

tgt
0
0
tgt
tgt
0
tgt, i1
or tgt, i2

3. Postprocessing with deep clustering
3.1 Proposed method
Speech enhancement by the TFS beamformer resulted ina performance of high noise reduction with less distortion.
However, as shown in the 7th and 8th rows of Table 1, either interferer 1 or 2 still remains. To reduce these residualcomponents without distorting the target speech, we consider
using the following TF mask

where y(ω, t) is the output of the beamformer, w(ω) denotes
the spatial filter vector, (·)T denotes the transpose, and (·)Hdenotes the Hermitian transpose. For the design of the spatial
filter w(ω), adaptive beamformers, e.g., the minimum variance distortionless response (MVDR) beamformer [19, 20],
are widely used. Theoretically, a beamformer with M microphones can suppress up to M − 1 interferers, which meansthat speech enhancement with a beamformer cannot achievesatisfactory performance under underdetermined conditions(M < N ).
To achieve sufficient speech enhancement performance under such conditions, the TFS beamformer [10] has been proposed, where the optimal filter selected from multiple beamformers is used at each TF bin. For simplicity, we consider acase where a three-source mixture x(ω, t) consisting of a target speech and interferers 1 and 2 is observed by two microphones. In this case, we cannot construct a null beamformer
that suppresses both interferers simultaneously. Supposingthat only the target speech and interferer k are observed (k =1, 2), we can construct the beamformer k with a spatial filter wk (ω) that suppresses the interferer k by a conventional
linear beamforming method. By applying the K constructedbeamformers, we obtain K outputs from the observed signalx(ω, t):yk (ω, t) =wHk (ω)x(ω, t), k = 1, . . . , K.

(4)Here, wk (ω) is the spatial filter of the beamformer k. By
considering statistically independent target speech and interferers, we can assume that the magnitude of the target speech
is smaller than that of the sum of the target speech and any
interferer. The TFS beamformer is then performed according
to the following criterion:{y1 (ω, t) if |y1 (ω, t)| ≤ |y2 (ω, t)|,yTFS (ω, t) =(5)y2 (ω, t) otherwise.
Equation (5) implies that the TFS beamformer selects the output of the beamformer with the smaller magnitude, which isequivalent to perform optimal filter selection in terms of minimizing the output. Table 1 summarizes the seven patterns of
the dominant sound source at each TF bin in the three-source
mixture case. In the case in the 5th or 6th row of Table 1,
one beamformer outputs the target speech, and the other one
outputs the target speech and a slightly altered version of aninterferer. According to the previously mentioned assump-ŷ( ω, t) = m(ω, t)yTFS (ω, t),(6)where m(ω, t) is a soft or binary mask, the element of whichequals to 1 at those TF bins including the target speech components regardless of the existence of interferers. ŷ( ω, t) represents yTFS (ω, t) for the conventional method and yDC (ω, t)
for the proposed method. In [10], the mask is constructedon the basis of DOA estimation, where the accuracy of themask directly depends on the DOA estimates. In long reverberant environments, however, it is difficult to estimate
DOA accurately [12–14] since the correct spatial information,
namely, the phase difference, is difficult to obtain because ofreflections. Moreover, since DOA estimation is performed
for the observed multichannel signals, which is independentof the TFS beamformer, the benefits gained from the higher
signal-to-noise ratio (SNR) thus cannot be exploited for the
DOA estimation. To deal with this issue, motivated by theremarkable results recently achieved using a DNN in monaural source separation, we propose using a DNN to estimate
the TF mask. Using training data and appropriately designed
labels, namely, the speech enhanced by the TFS beamformer
and DNN-based mask defined as{
1 |s(ω, t)| > threshold(7)m(ω, t) =0 otherwise,
we can train a DNN to estimate the TF mask for the output of the TFS beamformer. Here, s(ω, t) denotes the target
speech. Note that the W-DO assumption [21] is not required.
The enhanced speech y(ω, t) is finally obtained using for (6).
The process flow of the proposed method is shown in Fig. 1.
Specifically, we employ DC [17] with a gated CNN [18] for
DNN-based mask estimation. The main difference between
gated CNN and typical CNN is that a gated linear unit (GLU),
the second term of (9), is used as nonlinear activation function instead of, e.g., regular rectified linear units. Since a
GLU controls information passed on in the hierarchy, thegated CNN can capture long-term dependences without thevanishing gradient problem even with many layers.
3.2 Deep clustering with gated CNN
On the basis of the assumption that each TF bin of a mixture signal is dominated by a single source, i.e., W-DO, DC[17] separates sources by clustering feature vectors embedded from each TF bin. Let X = {Xc } ∈ RC×1 , where c646

at the 8th row in Table 1 having small energy can also be suppressed.
4. Experiments
4.1 Experimental conditions
Figure 1: Process flow of TFS+DC.

denotes a pair of frequency and time indices (ω, t) and C isequal to the number of TF bins Ω × T . A set of feature vectors Vc = (Vc,1 , . . . , Vc,D )T , which are called embedding
vectors, are obtained by a DNN V = gΘ (X), where g(·) denotes a nonlinear function, Θ denotes the parameters of the
network, and V = {Vc } ∈ RC×D is a matrix consisting ofembedding vectors in a row. The network is trained by minimizing the following objective function:
J (V) = ||VVT − YYT ||2F
= ||VT V||2F − 2||VT Y||2F + ||YT Y||2F ,(8)

where || · || is the squared Frobenius norm and Y = {Yc,n } ∈RC×N is a matrix composed of N -dimensional one-hot vectors in a row indicating to which source each TF bin belongs.
During the test, a clustering algorithm (e.g., K-means) is applied to the assigned embedding vectors to construct binarymasks for each source. Since the aim of using DC is to further enhance the speech outputted by the TFS beamformer,
in the proposed method, the input is yTFS (ω, t) and only themask for the target speech is estimated. For the network architecture, we use a gated CNN [18].
By using Hl−1 to denote the output of the (l − 1)th layer,
we obtain the output Hl of the lth layer as
Hl = (Hl−1 ∗ Wlf + bfl ) ⊗ σ(Hl−1 ∗ Wlg + bgl ),(9)where Wlf and Wlg are the weight parameters, bfl and bgl are
the bias parameters of the lth layer, ⊗ denotes element-wise
multiplication, and σ is the sigmoid function.
3.3 Advantages of combining TFS beamformer with
DNN-based TF masking
Comparing with the method combining the TFS beamformer with DOA-based TF masking, the proposed method
has the following merits. First, since the TFS beamformer
suppresses one of the interferers in each TF bin, the input signal of the DNN, i.e., the output of the TFS beamformer, ismore sparse than the observed signals, which facilitates themask estimation. Second, the performance of noise reduction
can be controlled by adjusting the threshold in (7). For example, by setting the threshold at a sufficiently small value (e.g.,10−5 ), we can mainly suppress interferers described at the
7th row in Table 1 while holding the distortion of the target
speech minimum. Since the TFS beamformer can guarantee
a distortionless response to the target speech, forcing the TF
mask to keep this property is important to realize distortionless enhancement. If we set the threshold at a relatively large
value, the DNN is trained to suppress all the components having less energy than the threshold so that interferers described

To evaluate the effect of the proposed method, we conducted experiments using the ATR503 database [22]. The
database consists of four male and six female speakers, andthe number of audio files uttered by each speaker is 503. We
divided the entire database into training and test sets. The
training set consisted of six speakers and 450 utterances ofeach speaker. The remaining 53 utterances spoken by theother four speakers were used as the test set so that the experiments was conducted in a speaker- and utterance-independent
manner.
We generated a total of 720 mixture signals (about 1 hour)for training the network by summing up three convolved
speech signals randomly selected from the training set. Room
impulse responses (RIRs) with reverberation times of {300,400, 500, 600, 700, 800}ms were generated using the image
method with the codes available in [23]. The DOA of the target speech was fixed at 90◦ and those of interferers were set
at 70◦ and 130◦ . All the audio files were downsampled to 8
kHz. We computed complex spectrograms using a Hann window with a length of 512 ms and shift of 256 ms. To obtainthe speeches enhanced by the TFS beamformers yTFS andthe corresponding labels, we applied a TFS beamformer withan MVDR beamformer [10] to the generated mixture signals
and computed the label using (7) with threshold set at 10−5 .
We trained two networks using data under each reverberation
condition.
Mixture signals for the test with reverberation times at{300, 400, 500, 600, 700, 800}ms were generated in thesame way, whereas DOAs of interferers were randomly selected from {10◦ , 30◦ , 50◦ , 70◦ , 110◦ , 130◦ , 150◦ , 170◦ }.
For each reverberation condition, we generated 10 mixture
signals, which were about 5 seconds long. Other experimental conditions are shown in Table 2.
We calculated signal-to-distortion ratio (SDR), signal-tointerference ratio (SIR), and signal-to-artifacts ratio (SAR)[24] and compared the performance of the proposed method
TFS+DC with the degenerate unmixing estimation technique(DUET) [6], which is BSS via TF masking with a stereo
microphone, the TFS beamformer (TFS) [10], and the TFS
beamformer with the DOA-based mask (TFS+DOA) [10].
4.2 Results and discussion
Figure 2 shows results achieved by each method underclosed reverberation time conditions. It is noteworthy that
TFS+DC led to few artifacts, which are detrimental to many
speech processing applications such as ASR. The methods using the TFS beamformer significantly outperformed DUET.
The SDRs obtained by TFS+DOA and TFS+DC were higher
than or equivalent to that achieved by the TFS beamformer,
which confirmed the effectiveness of applying TF masking
for postprocessing in improving the speech enhancement performance. Moreover, the proposed method achieved the bestscores, especially in the relatively highly reverberant casewhere TFS+DOA failed to achieve further improvement from647

Table 2: Experimental conditions.
Number of microphones
Distance between microphones
Reverberation time
(training/test)
DOA of the target (training/test)
DOA of interferers(training/test)

2
4 cm
300, 500 ms /300, 400, 500,
600, 700, 800 ms
90◦ / 90◦70◦ , 130◦ /10◦ , 30◦ , 50◦ , 70◦ ,110◦ , 130◦ , 150◦ , 170◦

Distance between
sources and microphones
Embedding dimension
Number of hidden layers
Number of channels
Learning rate
Optimizer
Minibatch size
Number of epochs

Figure 2: Average SDR, SIR, and SAR over 10 test signals
under each reverberation condition.

2m
20
5
128
0.001
Adam
4
50

the TFS beamformer. These results confirmed the observationthat the performance of TFS+DOA tends to decrease underhighly reverberant conditions. They also confirmed the effectiveness of the proposed method.
To confirm the generalization ability of the trained
network, we conducted an experiment employing themodel trained under the 500 ms reverberant condition(TFS+DC(500)) to enhance signals under various reverberation conditions. The results are shown in Fig. 3. Although,as the reverberation time increased, the performances of the
TFS beamformer and TFS+DC decreased in terms of all thecriteria, TFS+DC always exceeded the TFS beamformer at
a definite degree. The performance of using the model of
TFS+DC(500) was very close to the results obtained using
the model of TFS+DC(matched) which is trained on the datawith the same reverberation time as the test data. The results indicates that the proposed method can accomplish further improvement regardless of the reverberation conditions.
Furthermore, these results confirmed that the trained network
generalized well to cope with a wide range of reverberation
times.

Figure 3: Results of speech enhancement by TFS and
TFS+DC for reverberation times of 300–800 ms.

Acknowledgments
This work was supported by Japan Society for the Promotionof Science (JSPS) KAKENHI Grant Numbers JPH04131 and
JP19J20420, and the SECOM Science and Technology Foundation.
References[1] S. Makino et al., Blind speech separation, Springer, 2007.
[2] P. Smaragdis, Neurocomputing, vol. 22, no. 1-3, pp. 21–34, 1998.
[3] D. Kitamura et al., IEEE/ACM TASLP, vol. 24, no. 9, pp. 1622–1637,
2016.
[4] T. Higuchi et al., IEEE/ACM TASLP., vol. 25, no. 4, pp. 780–793,
2017.
[5] O. Yilmaz et al., IEEE TSP, vol. 52, no. 7, pp. 1830–1847, 2004.
[6] S. Rickard, in Blind Speech Separation, pp. 217–241, Springer,
2007.
[7] H. Sawada et al., IEEE TASLP, vol. 19, no. 3, pp. 516–527, 2011.
[8] H. Sawada et al., IEEE TASLP., vol. 21, no. 5, pp. 971–982, 2013.
[9] M. Togami et al., Proc. EUSIPCO, pp. 741–745, 2010.
[10] K. Yamaoka et al., Proc. EUSIPCO, pp. 1582–1586, 2018.
5. Conclusion[11] K. Yamaoka et al., Proc. ICASSP, pp. 7908–7912, 2019.
In this paper, we proposed a combination of the TFS beam- [12] O. Schwartz et al., IEEE/ACM TASLP, vol. 22, no. 2, pp. 392–402,
2013.
former and TF masking based on DC with a gated CNN as
an underdetermined speech enhancement method having high [13] O. Schwartz et al., Proc. IWAENC, pp. 1–5, 2016.
noise reduction performance with low signal distortion. The [14] J. R. Jensen et al., Proc. ICASSP, pp. 176–180, 2016.
TFS beamformer efficiently suppresses interferers even un- [15] P.-S. Huang et al., Proc. ICASSP, pp. 1562–1566, 2014.
Wang et al., IEEE/ACM TASLP, vol. 26, no. 10, pp. 1702–1726,
der underdetermined conditions. Furthermore, by applying [16] D.
2018.
DNN-based TF masking for postprocessing, it is possible to [17] J. R. Hershey et al., Proc. ICASSP, pp. 31–35, 2016.
suppress interferers that cannot be suppressed by the TFS [18] L. Li et al., Proc. ICASSP, pp. 16–20, 2018.
beamformer. To confirm the effectiveness of the proposed [19] H. L. Van Trees, Optimum array processing, 2002.
method, we conducted simulation experiments in reverberant [20] O. L. Frost, Proceedings of the IEEE, vol. 60, no. 8, pp. 926–935,
1972.
environments. As the results, we confirmed that the proposed
method achieved superior speech enhancement performance [21] O. Yılmaz et al., IEEE TSP, vol. 52, no. 7, pp. 1830–1847, 2004.
to the conventional method with the DOA-based TF masking, [22] A. Kurematsu et al., Speech communication, vol. 9, no. 4, pp. 357–
363, 1990.
especially in long reverberant environments. Additionally, we [23] E.
A. P. Habets, Available at: https://www.audiolabs-erlangen.de/also confirmed that the proposed method can enhance the tarfau/professor/habets/software/rir-generator, 2008.
get speech while keeping the distortion of the target speech to [24] E. Vincent et al., IEEE TASLP, vol. 14, no. 4, pp. 1462–1469, 2006.

the minimum with appropriately designed labels.

648