Monitoring of Domestic Activities Using Multiple Beamformers and Attention Mechanism
Yuki Kaneko, Takeshi Yamada, and Shoji Makino
University of Tsukuba
1-1-1 Tennodai, Tsukuba, Ibaraki, 305-8573 Japan
E-mail: y.kaneko@mmlab.cs.tsukuba.ac.jp

To verify the effectiveness of the proposed method, we generate acoustic data by mixing the sounds of the target scene andAcoustic scene classification is one of the important techthe interference scene using the dataset of DCASE (Detection
nologies for classifying domestic activities. When considand Classification of Acoustic Scene and Event) 2018 Task 5ering domestic activities as acoustic scenes, unlike the gen[4] and conduct a classification experiment.
eral task of acoustic scene classification, there is the problem that the sounds of the target scene and interference scene 2. Proposed method
can become mixed. To deal with this problem, we propose a 2.1 Overview of the proposed methodclassification method using multiple beamformers and an atFig. 1 shows the process flow of the proposed method.
tention mechanism. In the proposed method, multiple beamFirst,
the emphasized sounds are obtained by using M
formers for different target directions are prepared and their
MVDR（minimum variance distortionless response）beamoutputs are input to a classifier. The proposed method thenformers[5].
estimates the importance of each beamformer output by usTheemphasizedsounds are then input to the CNNs (convoing an attention mechanism. To verify the effectiveness oflutional
neuralnetworks)
after converting to 128-dimensional
the proposed method, we generated acoustic data by mixing
log
Mel-filterbank
energyfeatures. As shown in Fig. 1, thethe sounds of the target scene and the interference scene, andnetwork
weights
are
tied
among
the CNNs. In the proposed
conducted a classification experiment. The experimental remethod,
we
use
CNNs
consistingof eight convolution layers
sults confirmed that the F-score could be greatly improved by
as
in
[6].
Fig.
2
shows
the
detailsof the CNNs. The number
the proposed method.
of filters, pooling size, padding size, and filter size are shown
1. Introduction
in parentheses, respectively. In the conv. block, zero padding,
In recent years, interest in smart homes to realize safe, se- convolution, batch normalization, and application of ReLU
cure, and comfortable living has been increasing. Typical function are repeated.
After that, the importance（weight）of each output of CNNs
functions of smart homes include security, monitoring, andhome automation [1]. To put these functions into practical is estimated using the attention mechanism, and the weighted
use, technology for classifying domestic activities is indis- sum of each output is obtained. Finally the classification repensable, and there are high expectations for acoustic scene sult is obtained using the weighted sum. Here, A in Fig. 1 is
the weight matrix of M × N , and N is the number of timeclassification.
When considering domestic activities as acoustic scenes, frames. We compare two types of weight matrices: timeunlike the general task of acoustic scene classification, there variant (different weights can be set for each time frame)is the problem that the sounds of the target scene and inter- and time-invariant (same weights are set for each time frame)
ference scene can become mixed. One way to deal with this types. The details of the beamformer and attention mechaproblem is to emphasize the desired sound by using a beam- nism are described below.
former [2]. However, which direction should be emphasized 2.2 MVDR beamformer
is unclear in many situations, so it is difficult to apply the
In many speech enhancement techniques, the microphonebeamformer simply.
observed signal is represented in the time–frequency domain
In this paper, we propose a classification method using by a short-time Fourier transform. Here, x (ω, t) representsi
multiple beamformers and an attention mechanism [3]. In the the ith observed signal at frequency ω and time frame t. Conproposed method, multiple beamformers for different target sidering the case of two microphones for the sake of simplicdirections are prepared and their outputs are input to a clas- ity, a linear beamformer is generally given bysifier after converting to log Mel-filterbank energy features.
The proposed method then estimates the importance of eachy(ω, t) = wH (ω)x(ω, t),(1)
beamformer output using an attention mechanism. This corx(ω, t) = [x1 (ω, t), x2 (ω, t)]T .
(2)
responds to automatically finding the activity to be classified.
Abstract

145

Figure 1: Process flow of the proposed method.

Figure 2: Configuration of the convolution model.
y(ω, t), w(ω), ()T , and ()H represent the output of the beamformer, the spatial filter, transpose, and complex conjugatetranspose, respectively. In the proposed method, the MVDR
beamformer, which is one of the typical linear beamformers,
is used. The formula for calculating the spatial filter of the
MVDR beamformer is
R−1 a(ω, θ(ω))w(ω) =.
a(ω, θ(ω))H R−1 a(ω, θ(ω))(3)

R and a(ω, θ(ω)) represent the spatial correlation matrix andthe steering vector for frequency ω and the direction θ(ω) of
the target, respectively.
Since it is not obvious from which direction the sound of
the target scene comes, the proposed method uses M MVDR
beamformers with different target directions to obtain the emphasized sound from each target direction. In calculating the
spatial filter of each MVDR beamformer, the steering vector
for the corresponding target direction is given by considering
only the time delay. The deviation between the target direction and the actual direction can be reduced by increasing the

Figure 3: Details of the attention mechanism.
number of MVDR beamformers.
2.3 Classifier with attention mechanism
Fig. 3 shows the details of the attention mechanism inthe proposed method. First, the nth column vector en =(e1n , · · · , eM n )T of the M × N matrix, which is obtained byconcatenating the outputs of the CNNs, is input to the FCN
to obtain the output un = (u1n , · · · , uM n )T .
un = f (en ), n = 1, 2, · · · , N(4)

f is the nonlinear function corresponding to the FCN. N andM represent the number of time frames and the number of
MVDR beamformers, respectively. In this paper, the FCN
has one layer with 32 units. Next, as shown by (5), the weight
amn for each direction and each time frame is obtained using

146

this experiment, the five types of target scenes were cooking,dishwashing, eating, working, and conversation, and the two
types of interference scenes were watching TV and vacuum
cleaning. The data of the target scene and the interference
scene recorded by the same microphone array were randomly
selected and mixed.
The total number of acoustic data after mixing is 32,444.
Table 1 shows the number of acoustic data of each acoustic scene before mixing. “# sessions” indicates the numberof recordings and each recording is divided into segments of
10 s, the number of which is given under “# 10 s segments.”
The microphone interval in the microphone array is 5 cm.
The sampling frequency is 16 kHz and the quantization bits
is 12. The features are 128th-order log Mel-filterbank energy.
The frame length and frame shift length in the frame analyFigure 4: Layout of the microphone arrays and the soundsis are 40 and 20 ms, respectively. The Adam algorithm [8]scenes (quoted from [7]).
is used as the optimization method for training the classifier;the number of epochs during training is set up to be 50 and
Table 1: Number of acoustic data of each class.
the epoch that gives the best F-score is adopted. The Chainer
activity# sessions # 10 s segments
framework [9] is used for the implementation of the proposedcooking135124
method. In addition, M in Fig. 1 was 3 and the target direction was set to 30◦ , 90◦（direct front), and 150◦ . Only mixed
dishwashing
10
1424
acoustic data were used for learning the classifier.
eating13
2308
working
33
18644
3.2 Results and discussionsocial activity
214944
To verify the effectiveness of the proposed method, we
compare the performance in the following three cases.
the softmax layer.
• single channelexp(umn )amn = ∑M
(5)
, n = 1, 2, · · · , N• proposed (correct weight)
m=1 exp(umn )• proposed
We compare two types of weight matrices: time-variant andtime-invariant types. Finally, the weighted sum of the CNN ‘Single channel’ is the case where only a single channel of
outputs is obtained for each time frame using (6) and then the 4-channel microphone array is used for classification, andused to obtain the classification result.
corresponds to classifying mixed data as is. In addition, ‘proposed (correct weight)’ is the case in which the correct weight
M
∑
e′n =
amn emn , n = 1, 2, · · · , N(6) matrix was given in the proposed method and refers to theupper limit performance of the proposed method. In the prom=1
posed method, the cases of time-variant and time-invariant
weight matrices are also compared.
3. Experiment
Table 2 summarizes the experimental results. First, the F3.1 Experimental conditions
score of ‘single channel’ was 64.61%. Since the F-score was
To verify the effectiveness of the proposed method, we 89.27% when we conducted a similar experiment without thegenerated acoustic data by mixing the sounds of the target interference scenes, we can see that the F-score was signifiscene and the interference scene using the dataset of DCASE cantly decreased by the presence of interference scenes. Next,
2018 Task 5 [4] and conducted a classification experiment. the F-score of ‘proposed (correct weight)’ was 83.46%. This
This dataset comprises the sounds of a person living in a villa result means that the F-score was greatly improved by emphafor one week, which were recorded using seven 4-channel sizing the sound of the target scene. Similarly, the F-score of
microphone arrays.
‘proposed’ was 76.18 %. While it was less than that of ‘proFig. 4 shows the layout of the microphone arrays and the posed (correct weight)’, about a 12 % improvement was obacoustic scenes. The red arrows, the green circles, the blue tained compared with that of ‘single channel’. It is necessary
circles, and the orange circles indicate the direction of the to optimize the network structure of the attention mechanism
microphone array, the four microphone arrays used in this ex- in order to approach the classification accuracy of ‘proposed
periment, the positions where the target scene occurs and the (correct weight)’. Finally, the F-score of the proposed method
positions where the interference scene occurs, respectively. In with the time-variant weight matrix was almost the same as147

Table 2: Experimental results (F-score [%]).
time-invariant time-variant
single channel
64.61—
proposed (correct weight)
83.46—
proposed
76.18
75.68
Table 3: F-score [%] of each microphone array.
array 1 array 2 array 3 array 4
proposed 68.20
76.63
79.37
73.46
that of the time-invariant case. This is because there is no
major movement of the sounds in this experiment.
Next, Table 3 shows the F-score of the proposed method
for each microphone array. The microphone array indexes
correspond to Fig. 4. The F-score of the microphone array 1
was 68.20%, which is lower than those of other microphonearrays. It is considered that this is because the distance to theinterference scene is small, and the directions of the target
scene and the interference scene are often close to each other,such as for the pair of vacuum cleaning and eating. In addition, the F-score of microphone array 3 is the highest among
the four microphone arrays and was 79.37%. It is consideredthat this is because the directions of the target scene and theinterference scene are sufficiently separated. The fact that thethree directions set as the target directions matched the actual direction of the target scene the interference scene would
also be a factor in increasing the F-score of microphone array
3. From this result, it is considered that the F-score can befurther improved by increasing the number of beamformers.

Figure 5: Weights in each direction estimated by the attention
mechanism.
Acknowledgment
This work was supported by the Japan Society for the Promotion of Science (JSPS) KAKENHI Grant No. 20K11880
and 19H04131.
References[1] M. Alaa et al., “A review of smart home applications
based on Internet of Things,” Journal of Network andComputer Applications, Vol. 97, pp. 48–65, 2017.

In addition, Fig. 5 shows an example of the weight matrix estimated by the attention mechanism, when eating andwatching TV were recorded by microphone array 3. The horizontal axis represents the time frame, and the vertical axis
represents the MVDR beamformer index and target direction.
We can see that the weight in the 30◦ direction, where the target scene is located, is large. However, the weight of 90◦ or150◦ is locally large, and this is considered to be the difference in the performance between ‘proposed (correct weight)’
and ‘proposed’.
4. Conclusions
In this paper, we proposed a classification method of using multiple beamformers and the attention mechanism forthe situation in which the sounds of the target scene and interference scene are mixed. To evaluate the effectiveness ofthe proposed method, wee generated acoustic data by mixing the sounds of the target scene and the interference scenetaken from the dataset of DCASE2018 Task 5 and conducteda classification experiment. As a result, it was confirmed thatthe proposed method can automatically find and emphasize
the sound to be classified, and the classification accuracy was
greatly improved.

148[2] R. Tanabe et al., “Multichannel acoustic scene classification by blind dereverberation, blind source separation, data augmentation, and model ensembling,”
DCASE2018 Challenge Technical Report, 2018.
[3] D. Bahdanau et al., “Neural machine translation by jointly learning to align and translate,”
arXiv:1409.0473, 2014.
[4] G. Dekkers et al., “DCASE 2018 Challenge Task
5: Monitoring of domestic activities based on multichannel acoustics,” arXiv:1807.11246v2, 2018.
[5] H. L. Van Trees, Optimum Array Processing, John Wiley & Sons, 2002.
[6] Y. Han et al., “Convolutional neural networks with binaural representations and background subtraction for
acoustic scene classification,” DCASE2017 Challenge
Technical Report, 2017.
[7] G. Dekkers et al., “The sins database for detection ofdaily activities in a home environment using an acoustic
sensor network,” Proc. DCASE 2017 Workshop, pp. 3256, Nov. 2017.
[8] D. P. Kingma et al., “Adam: a method for stochastic
optimization,” ArXiv:1412.6980v1, Dec. 2014.
[9] Chainer website, https://chainer.org/.