PROCEEDINGS of the
23rd International Congress on Acoustics
9 to 13 September 2019 in Aachen, Germany

Deep Learning Applied to Dereverberation and Sound Event Classification
in Reverberant Environments
Mingsian R. Bai1; Wenchuan Chen2
1

Department of Power Mechanical Engineering, National Tsing Hua University, Taiwan

ABSTRACTThis paper investigates dereverberation and sound event classification techniques, with the aid of deeplearning. The proposed system consists of two units: a microphone array front-end and a deep learning
back-end. The system is examined in the context of two important acoustic signal processing:
dereverberation and sound event classification (SEC). For the dereverberation problem, a neuralnetwork-based approach is compared with other state-of-the-art methods such as beamforming, themulti-channel inverse filtering (MINT), the multi-channel Wiener filter (MWF), and the variance-normalized
delayed linear prediction (NDLP). For the SEC problem, two approaches are also compared. Approach 1 isbased on the signals enhanced by the microphone array front-end, which serves as the input to the back-end
deep neural network (DNN) classifier. The DNN is implemented by using the VGGNet®. Approach 2 is adirect approach that uses reverberant data directly to train the classifier without the front-end enhancement.
The audio features are extracted as MFSC from the AudioSet from Google®. A room response simulator
based on the image source method is employed to create reverberant signals for numerous RT60 conditions in
the training phase. Perceptual evaluation of speech quality (PESQ) and F1-score are adopted to assess theaudio quality and classification performance.
Keywords: Deep Learning, Reverberation, Sound Event Classification

1. INTRODUCTION
In Automatic Speech Recognition (ASR), the quality of speech signals can be degraded due toacoustically adverse factors such as background noise and reverberation. Reverberations are more
intractable than background noise because the direct sound and reflections are mutually correlated. In
the past, dereverberation algorithms (1-3) were proposed to enhance the speech quality.
A straightforward way of eliminate reverberation is to deconvolve the reverberant signals using
the room impulse responses (RIRs). The Multiple Input /Output Inverse Theorem (MINT) (4, 5) was
suggested to deal with dereverberation problem with great success. However, this approach requires
the prior knowledge of RIRs with fixed source-receiver positions. Alternatively, the Multichannel
Wiener Filter (MWF) (6) which consists of a Minimum Variance Distortionless Response (MVDR)
cascaded with a single channel post filter can be used. On the other hand, linear prediction(LP)-based methods exploit the fact that the late reverberation is correlated with the direct signals.
Based on this notion, the Variance-Normalized Delayed Linear Prediction (NDLP) (3, 8) was
proposed to eliminate late reverberation signals.
Recently, Deep Neural Networks (DNN) have emerged as powerful tools for regression andclassification problems. For dereverberation tasks, some methods (8, 9) attempt to map reverberant
features into anechoic features, while others (10, 11) map input features into a time-frequency mask.
In this paper, DNNs are employed to handle dereverberation and sound event classification (SEC)
tasks. For dereverberation problems, fully connected neural networks are compared with thetraditional methods. For classification problems, two approaches are compared. In approach 1,
dereverberation algorithms were applied to enhance the sound event data convolved with room
impulse responses (RIRs). Approach 2 can be regarded as an end-to-end approach, where thereverberant features and clean features are jointly fed into the neural network. Finally, theperformance of algorithms is discussed and concluded.

12

msbai@pme.nthu.edu.tw
s106033548@m106.nthu.edu.tw4917

2. MULTICHANNEL DEREVERBERATION METHODS
2.1

Multiple-input/output Inverse Theorem (MINT)
Consider the scenario with one source and a 2-microphone array. The system convolution matrix
can be written as0⎡ g n (0)⎢ g (1)g n (0)n⎢⎢!
g n (1)⎢!
G n = ⎢ g n ( Lg − 1)⎢0
g n ( Lg − 1)
⎢
!
"
⎢
⎢
0
#
⎣

⎤
⎥
⎥
⎥
!
⎥
g n (0) ⎥ , n = 1, 2,
g n (1) ⎥⎥!
⎥g n ( Lg − 1) ⎥⎦

0

0

0

0

"
"
"
"
0

(1)

where g n stands for room impulse response from the source to the n-th microphones. The
dereverberation problem can be written as a model matching problem

m = G1 h1 + G 2h 2 = ⎡⎣G1

(2)

m stands for matching model. The model matchingwhere h stands for filter coefficients andproblem can be rewritten as0⎡0 ⎤ ⎡ g1 (0)⎢ ! ⎥ ⎢ g (1)g1 (0)1⎢ ⎥ ⎢!
g1 (1)⎢1 ⎥ ⎢
⎢ ⎥ ⎢
g
(
L
−
1)
!
=
!
⎢ ⎥ ⎢ 1 g⎢⎢! ⎥0
g1 ( Lg − 1)
⎢ ⎥ ⎢
!
"
⎢! ⎥ ⎢⎢0 ⎥ ⎢
0
#
⎣ ⎦ ⎣

⎡h ⎤
G 2 ⎤⎦ ⎢ 1 ⎥ = Gh ,⎣h 2 ⎦00

g 2 (0)

0

0

0

0

g 2 (1)g 2 (0)0

g 2 (1)

"

"

!

!

"
"

g1 (0)
g1 (1)

g 2 ( Lg − 1)
0

"

!

!

"

"

0

g1 ( Lg − 1)

0

#

0

!
"
g 2 ( Lg − 1) "⎤ ⎡ h1 (0) ⎤⎥⎢⎥⎥ ⎢ h1 (1) ⎥
⎥⎢
!
⎥
!
⎥⎢
⎥
g 2 (0) ⎥ ⎢ h1 ( Lh1 − 1) ⎥g 2 (1) ⎥ ⎢ h2 (0) ⎥
⎥⎢
⎥
!
!
⎥⎢
⎥
g 2 ( Lg − 1) ⎥⎦ ⎢⎣ h2 ( Lh 2 − 1) ⎥⎦
0

0

(3)

A regularized least-square solution can be written as

h = (G H G + β 2I)−1 G H m ,(4)where β stands for regularization parameter.
2.2

Multichannel Wiener Filter (MWF)
Consider a scenario with a single speech source and M ≥ 2 microphones. The m-th microphonesignal, m = 1, ..., M , of frequency index k and frame index l is given by
X m (k , l ) = X d ,m (k , l ) + X r ,m (k , l ) , where X d , m (k , l ) denotes the direct and early reverberant signal
and X r ,m ( k , l ) denotes the late reverberant signal. Thus, the reverberant signal can be written invector formx(k , l ) = xd (k , l ) + xr (k , l ) ,(5)where x(k , l ) = [ X1 (m, l ) X 2 (m, l ) ! X M (m, l )] . Modeling the source signal as single point-source, thevector xd (k , l ) can be written asT

x d ( k , l ) = d( k ) S ( k , l ) ,(6)where S (k , l ) is the target signal and d(k ) = [ D1 (k ) D2 (k ) ! DM (k )] is the steering vector of theTmicrophone array. The Power Spectral Density (PSD) matrix, R x (k , l ) = E{x( k , l )x H ( k , l )} , can be
written as
R x (k , l ) = R xd (k , l ) + R xr (k , l )(7)Using Equation 6 in the matrix R x (k , l ) leads tod4918

R xd (k , l ) = Φs (k , l )d(k )d H (k ) ,(8)Γ(k ) denotes the time-invariant
normalized spatial coherence matrix of the late reverberation. The output signal Z ( k , l ) is given as
where Φ r (k , l ) is the PSD of the late reverberant signal andZ (k , l ) = w H (k , l )x d (k , l ) + w H (k , l ) x r (k , l )(9)From (12), the optimal weight of the MWF can be derived as an MVDR beamformer w MVDR cascadedwith a single-channel Wiener postfilter

w MWF =

Φ so (l )Γ−1d
H −1d Γ d Φ so (l ) + Φ ro (l )(10)

In (6), the Eigenvalue Decomposition (EVD) estimator was proposed to estimate reverberation levelˆ EVD =
Φr

1(tr{R x (l )Γ−1} − λ1{R x (l )Γ−1}) ,
M −1

(11)

where tr{}⋅ denotes matrix trace and λ1{}⋅ denotes the largest eigenvalue.
2.3

Normalized Delayed Linear Prediction
Consider an acoustic scenario in which a single source signal is captured by M microphones in areverberant room. The signal xt( m ) picked up at the m-th microphone and time index t can be modeledasxt( m ) = dt( m ) + rt( m ) ,
(m)
t

(12)

where dis the direct signal and the early reflections and rtin (13), the observe signal in Equation 12 can be rewritten as(m)is the late reverberation. As shown

Lm Lc −1xt( m) = ∑ ∑ xt(−mD′)−k ck( m′,m) + dt( m) ,(13)

m′=1 k =0

Where ck( m ) is the filter coefficient of m-th microphones,and

Lm is the number of the microphones

Lw is the length of the filter coefficient. Equation 13 can be rewritten in vector form asxt( m ) = (c( m ) )T xt − D + dt( m ) ,

(14)

where

xt = [(xt(1) )T(xt(2) )T ! (xt( Lm ) )T ](15)

xt( m) = [ xt( m)xt(−m1) ! xt(−mL)c +1 ]T(16)c( m ) = [(ct(1,m ) )T

(ct(2,m ) )T ! (ct( Lm ,m ) )T ]T(17)c( m′,m) = [(c1( m′,m) )T(c2( m′,m) )T ! (cL( mc ′,m) )T ]T(18)

In Equations 12 and 13, the late reverberation in the observed signal is predicted using a D-step(m)linear predictor by convolving past observed signals with the filter coefficients cof the order of

Lc . By doing so properly, the prediction residual signal is the desired signal d t( m ) . So the desired
signal can be estimated from Equation 14. By maximizing the log-likelihood function in (7), theleast-square solution of the regression coefficients is

4919

cˆ

( m)

xt − D xTt − D + T xt − D xt( m )
= {∑
} {∑
},
σˆ t2
σˆ t2
t =1
t =1
T

(19)

where the superscript “ +” stands for pseudo inverse.
2.4

DNNs-based Dereverberation
A deep neural network (DNN) is a machine learning method based on artificial neural networks
with multiple hidden layers inserted between the input layer and the output layer. Similar to the
network architecture in (9), the proposed DNN is depicted in Figure 1

Figure 1 – Fully connected neural network with 5 hidden layers of 1024, 1024, 1024, 512, 128 nodes
Let the signals received by M microphones be ri (t ), i = 1, 2,! M . In order to incorporate temporal
dynamics, we include the spectral features of neighboring times into a feature vector ri (t ) :ri (t ) = [ri (t − di ) ! ri (t ) ! ri (t + di )],(20)

where d i denotes the frame size in the i-th channel. As a result, the input vector is a concatenation of
selective signal frames comprised spatially from each microphone and temporally from adjacent timeindexes:
r!i (t ) = [ r1 (t ) r2 (t ) " rM (t ) ](21)
In the training phase, the output signals of the DNN x (t ) are calculated by forward propagation,
while the network parameters are updated through backpropagation by minimizing the mean squareerror. In the testing phase, the forward propagation will be performed to reconstruct the desired
signals from the input features.

3. SOUND EVENT CLASSIFICATION (SEC)
3.1

Sound Event Dataset and Feature Extraction
The dataset is a subset of AudioSet from Google®. It consists of 4 classes of domestic soundevents and 4400 sound clips in total. The data are firstly clipped into 10-second segments and thendivided into training dataset (3400 audio clips) and evaluation dataset (1000 audio clips),
respectively. Features extraction is performed on the data input to models. Mel-frequency cepstralcoefficients (MFCCs) have been widely used as features for conventional methods. However, log
mel-frequency spectral coefficients (MFSCs) are increasingly popular because the cosine transform4920

performed at the last stage in calculating MFCCs may not preserve locality. In this paper, frames areprocessed with hamming windows and 1024-point FFT, and 256 hop size.
3.2

Network Architecture
Convolutional neural networks that have been widely used in image recognition tasks are utilized
here in sound event classification tasks. Visual Geometry Group Network (VGGNet) (14) from
ImageNet Large Scale Visual Recognition Competition (ILSVRC) 2014 is one of many welldeveloped CNN-based architectures. By using 2 concatenated 3 x 3 filters, it actually covers a 5 x 5region. Thus, large-size filters are not required and the number of parameters can be reduced. In thispaper, VGG-like Network was used. Since the size of dataset is smaller than ImageNet (15), theproposed VGG-like network only adopted 6 convolutional layers to prevent overfitting. The overallarchitecture is depicted in Figure 2.

Figure 2 – The proposed VGG-like network. Hyper-parameters are indicated. For example, (3 x 3, 40)
symbolizes that the filter size is 3 x 3 and the number of filters is 40. 1280 means the number of node.

4. SIMULATION AND DISCUSSION
4.1

Image Source Method (ISM) Simulation
In this section, the proposed DNN is examined in a reverberant environment with reverberation
time T60 ranging from 0.1 to 1 sec with an increment of 0.1 sec. The RIRs were simulated using ISM(16). Eight microphones are placed along a straight line at (4, 1, 2), (4, 1.1, 2), (4, 1.2, 2), (4, 1.3, 2),(4, 1.4, 2), (4, 1.5, 2), (4, 1.6, 2), and (4, 1.7, 2) meters in a rectangular room with dimensions 6 x 4 x
2 meters. The sound source is positioned at (2, 3, 1.5) meters.
4.2

PESQ Evaluation
To begin with, PESQ is used to evaluate the dereverberation performance when applying to a
10-second speech clip. The speech signal is filtered by RIRs with different T60’s. Figure 3 comparesthe dereverberation performance of all algorithms investigated.

Figure 3 – Comparison of dereverberation algorithms for different reverberation times (T60).

4921

It can be seen from the results that the MINT performs the best in dereverberation, while it requires theprior knowledge of the room response. On the other hand, the DNN approach outperforms the NDLP and
MWF methods in a more practical setting where the information of room responses is not available.
4.3

F1-score Evaluation
In this section, F1-score is used as the performance metric of sound event classification withoutor with the reverberation, as shown in Table 1.
Table 1 – F1-score Evaluation
Case

Training

Testing

F1-score

1

Clean

Clean

0.84

2

Clean

Reverb

0.76

3

Clean

MINT

0.84

4

Clean

MWF

0.78

5

Clean

NDLP

0.70

6

Clean

DNN

0.82

7

Mixed

Reverb0.85

In the table, “Clean” represents the unprocessed anechoic data. “Reverb” means the reverberant
signal (T60=0.5sec). “Mixed” means the signals combining the “Clean” and “Reverb” signals.
“MINT”, “MWF”, “NDLP”, and “DNN” represent the signals dereverberated by using MWF, NDLP,
and DNN.

5. CONCLUSIONS
Dereverberation approaches using an end-to-end DNN-based approach has been compared withexisting two-stage methods, MINT, MWF, and NDLP. The results have demonstrated that the
DNN-based approach is more advantageous than the two-stage methods in terms of its
dereverberation performance and its room acoustics-free implementation. However, as a limitation,
the proposed DNNs-based method requires a large quantity and high quality dataset.
Dereverberation algorithms are also compared for SEC tasks. Although the two-stage methods are
effective in dereverberation, they suffer signal distortions and hence degraded classification rate inF1-score. In contrast, the end-to-end DNN-based approach using mixed data (case 7 in Table 1) has
demonstrated superior classification performance over two-stage methods.

ACKNOWLEDGEMENTS
The work was supported by the Ministry of Science and Technology (MOST) in Taiwan, Republicof China, under the project number 107-2221-E-007 -039 -MY3.

REFERENCES
1.
2.
3.
4.

4922

S. Mosayyebpour, M. Esmaeili, and T. A. Gulliver, Single-microphone early and late reverberation
suppression in noisy speech. IEEE Trans. Audio Speech, Lang. Process. 21(2), 322–335, 2013.
E. Habets and J. Benesty, A two-stage beamforming approach for noise reduction and dereverberation.
IEEE Trans. Audio, Speech Lang. Process. 21(5), 945–958, 2013.
K. Kinoshita, M. Delcroix, T. Nakatani, and M. Miyoshi, Suppression of late reverberation effect on
speech signal using long-term multiple-step linear prediction. IEEE/ACM Transactions on Audio,
Speech, and Language Processing. 17(4), 534–545, 2009.
M. Miyoshi and Y. Kaneda, Inverse filtering of room acoustics, IEEE Trans. Acoustics, Speech, andSignal Processing, 36(2), 145-152, 1988.

5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.

I. Kodrasi, S. Goetze, and S. Docolo, Regularization for Partial Multichannel Equalization for Speech
Dereverberation, IEEE Trans. Audio, Speech, and Language Processing, 21(9), 1879-1890, 2013.
I. Kodrasi and S. Doclo, Late Reverberation Power Spectral Density Estimation Based on An
Eigenvalue Decomposition, Int. Conf. Acoustics, Speech and Signal Processing (ICASSP), New
Orleans, 5 pages, 2017.
T. Nakatani, T. Yoshiok, K. Kinoshita, M. Miyoshi, and B. Juang, Speech Dereverberation Based on
Variance-Normalized Delayed Linear Prediction, IEEE/ACM Transactions on Audio, Speech, andLanguage Processing, vol. 18, no. 7, 2010.
K. Han, Y. Wang, and D. Wang, Learning spectral mapping for speech dereverberation and denoising,IEEE/ACM Trans. Audio, Speech, Lang, Process. 23(6), 982–992, 2015.
B. Wu, M. Yang, K. Li, Z. Huang, M. Siniscalchi, T. Wang, and C. Lee, A reverberation-time-aware
DNN approach leveraging spatial information for microphone array dereverberation, EURASIP
Journal on Advances in Signal Processing, 2017.
D. S. Williamson and D. Wang, Time-Frequency Masking in the Complex Domain for Speech
Dereverberation and Denoising, in IEEE/ACM Transactions on Audio, Speech, and Language
Processing, vol. 25, no. 7, pp. 1492-1501, 2017.
O. Hazrati, J. Lee, Jaewook, and P. C. Loizou, Blind binary masking for reverberation suppression incochlear implants, The Journal of the Acoustical Society of America. 133. 1607-14.
10.1121/1.4789891, 2013.
S. Braun and E. A. P. Habets, Dereverberation in noisy environments using reference signals and amaximum likelihood estimator, in Proc. European Signal Processing Conference, Marrakech,
Morocco, 2013.
D. Gesbert and P. Duhamel, Robust blind channel identification and equalization based on multi-step
predictors, in Proc. ICASSP’97, 1997.
S. Liu and W. Deng, Very deep convolutional neural network based image classification using small
training sample size, 2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), Kuala
Lumpur, 2015, p. 730-734.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, ImageNet Classification with Deep Convolutional
Neural Networks, Neural Information Processing Systems. 25, 10.1145/3065386, 2012.
J. Allen and D. Berkley, Image Method for efficiently simulating small- room acoustics, The Journalof the Acoustical Society of America, Vol 65, No.4, p. 943- 950, 1978.

4923