2020 RISP International Workshop on Nonlinear Circuits,Communications and Signal Processing (NCSP 2020)
Honolulu, Hawaii, USA, February 28 - March 2, 2020

Spatial Feature Extraction Based on Convolutional Neural Networkwith Multiple Microphone Inputs for Monitoring of Domestic Activities
Yuki Kaneko, Rika Kurosawa, Takeshi Yamada, and Shoji Makino
University of Tsukuba
1-1-1 Tennodai, Tsukuba, Ibaraki, 305-8573 Japan
E-mail: y.kaneko@mmlab.cs.tsukuba.ac.jp

and television. Since the dataset in this task was recorded using four channels (microphones), the performance of acousticIn acoustic scene classification, the use of multiple micro- scene classification can be expected to be improved by utilizphone inputs enables to improve classification performance ing spatial information. However, many methods submitted
by utilizing spatial information. Recently, a method of spa- for this task utilized each individual signal recorded using
tial feature extraction using CNNs (convolutional neural net- multiple channels, so did not make full use of spatial inforworks), 2D-CNN, has been proposed. In this method, CNNs mation. The method with the best performance among theare separately applied to the time-frequency domain, time- methods submitted for this task focused on data augmentaspace domain, and frequency-space domain, respectively. tion and did not use spatial features explicitly[2]. It is thereFurthermore, a method using CNN with a three-dimensional fore necessary to consider how to utilize spatial information
filter, 3D-CNN, has been proposed. It extracts spatial fea- contained in multichannel signals.
tures across the time-frequency-space domain. Both methods
Recently, a method of spatial feature extraction usintend to extract spatial features suitable for classification. In ing CNNs (convolutional neural networks) with a twothis paper, to examine an appropriate structure of CNN for dimensional filter, 2D-CNN, has been proposed[3]. This
extracting spatial features, we compared 2D-CNN and 3D- method extracts spatial features suitable for classification by
CNN by applying them to the dataset with four microphone separately applying CNNs to the time–frequency domain,
inputs prepared for DCASE2018 Task 5. The experimental time–space domain and frequency–space domain. The efresults confirmed that the F-score of 3D-CNN is better than fectiveness of this method was shown in experiments usingthat of 2D-CNN for different number of microphone inputs. two channel signals. Furthermore, a method using CNN
The best F-score of 87.7 was given by 3D-CNN with three with a three-dimensional filter, 3D-CNN, has been proposed
microphone inputs.
[4]. This method extracts features across the time–frequency–
space domain by applying CNN with a three-dimensional filter. In this paper, to examine a suitable structure of CNN for
1. Introduction
extracting spatial features, we apply 2D-CNN and 3D-CNNto DCASE2018 Task 5 and compare their performance. We
In recent years, interest in improvements to the quality of also investigate the effect of increasing the input channels.
life such as smart homes has been increasing. Typical functions of a smart home include a system for monitoring children and elderly people and smart speakers that turn on the 2. Spatial feature extraction using CNNs
light and play music in response to the vocal instructions ofpeople. In a smart home, the information acquired using varFig. 1 shows the process flow of the method of spatial
ious sensors is utilized, and sound information also plays an feature extraction using CNNs[3]. In this method, the inimportant role.
put signal is assumed to be a 10 s stereo signal, and a
Extracting sound information from the sound generated in 128-dimensional log Mel filter bank output is calculated fora certain environment is called environmental sound recogni- both the left and right channels. CNN is then applied totion. The DCASE (Detection and Classification of Acoustic each domain. The blue squares in Fig. 1 represent the twoScenes and Events) Challenge [1] has been held continuously dimensional filter for each domain, and the upper, middle,since 2013 for the purpose of improving the technical level of and bottom squares represent the CNNs being applied to theenvironmental sound recognition. As DCASE2018 Task 5, an time–frequency domain, time–space domain and frequency–acoustic scene classification task was provided to classify the space domain, respectively. For example, in the CNN for thesounds recorded at home into nine classes, such as cooking time–frequency domain, the time–frequency spectrogram isAbstract

175Figure 1: Process flow of spatial feature extraction method using CNNs
in the pooling layer is 2 × 1. This is because it maintains thedimension of the space. The total dimension of the feature
map after global average pooling is 256 × n, where n is thenumber of channels inputted to each CNN.
Finally, we explain the conv. block. In the conv. block, zero
padding with a size of 1×1 is first performed for the input feature map. Next, a convolution with filters of kernel size 3 × 3
is performed. Then, batch normalization [5] is conducted. Finally, the ReLU function is applied as an activation function.
These processes are performed twice in the conv. block.
Figure 2: Conv. model and conv. block in the 2D-CNN
The effectiveness of this method was confirmed by an experiment using stereo signals. In this paper, we apply thismethod to the dataset with four channel signals prepared forregarded as a feature map and space (stereo) is regarded as DCASE2018 Task 5. The part shown in red in Fig. 2 is thethe channels of the CNN. Therefore, the number of chan- part whose size has been changed. The kernel size of thenels of the CNN is two. On the other hand, in the CNN for convolution is the same as that in the above method in thethe time–space domain, since frequency is regarded as the time–frequency domain. On the other hand, the kernel size ofchannels of the CNN, the number of channels of the CNN is the convolution is changed to 3 × 5 in the time–space domain
128, the number of Mel frequency bins. Convolutions for the and frequency–space domain. This means that the dimensiontime–space domain and frequency–space domain correspond of the space is 5. This change enables to extract spatial feato extracting spatial information between stereo channels in a tures spanning two or more microphones. The convolution isfeature map.
performed while maintaining the number of channels in theIn this method, the CNN consists of eight convolution lay- same way as in the above method. The zero padding in theers. Fig. 2 shows the conv. model and conv. block when the time–frequency domain is 1 × 1. On the other hand, the zero
number of channels of each CNN is one. In the conv. model, padding in the time–space domain and the frequency–space
the conv. block that performs convolutions is repeatedly per- domain is 1 × 2. This is to prevent the reduction of the diformed with max-pooling. The two numbers separated by a mension when performing convolution.
comma in the parentheses of the conv. model are the number of input channels in the conv. model on the left and thenumber of output channels on the right. The numbers in the 3. Spatial feature extraction with a three-dimensional filparentheses of the pooling layer are the window sizes used terfor pooling. The window size in the pooling layer is 2 × 2 for
The process flow of 3D-CNN supposing two channels as
the time–frequency domain. On the other hand, for the time–
space domain and frequency–space domain, the window size the input is shown in Fig. 3. The blue box represents a three-176

Figure 3: Process flow of the 3D-CNN

Figure 4: Conv. model and conv. block in the 3D-CNN
Table 1: Overview of the dataset# of scenes# of signals in development set# of signals in evaluation set# of channels
microphone interval
sampling frequency
quantization bits

97298472972
45 cm
16 kHz12Figure 5: Layout of the microphone arrays (quoted from [1])the experiment. This dataset is part of the SINS dataset [6].
This dataset comprises the sounds of a person living in a villarecorded using four-channel microphone arrays for a week.
The arrangement of the microphone arrays in the room isshown in Fig. 5. Table 1 shows an overview of the dataset.
There are nine types of sound scene, and the total number
dimensional filter when applying the CNN. This method can of signals in the development dataset is 72984. The number
extract features across the time–frequency–space domain and of signals of each sound scene is shown in Table 2. # sesthe number of parameters in the CNN is significantly re- sions” indicates the number of recordings of the sound signals
duced as compared with the 2D-CNN. Fig. 4 represents the of each class, and each sound signal is divided into segments
conv. model and conv. block when the number of channels of 10 s, the number of which is given by “# 10 s segments.”of the CNN is one. These are basically the same as those in The number of sound signals for dishwashing and vacuum
Fig. 2. The numbers of input and output channels, the kernel cleaning is small while the number of sound signals for absize of the convolution, and the window size of pooling are sence, working and watching TV is large. The number ofdesigned in accordance with the same objective as in Fig. 2. datasets for evaluation is 72972, the microphone interval in
When this method is applied with four channels as the in- the microphone array is 5 cm, and the sampling frequency isput, the kernel size of the convolution is changed. The ker- 16 kHz. The quantization bits is 12. We train the classifier
nel size of the convolution is 3 × 3 × 5 and the size of zero using the development dataset and evaluate it using the evalpadding is 1 × 1 × 2. The dimensions of the space are 5 and uation dataset. To evaluate only the effect of spatial feature
2, respectively. The reason for this size is the same as that for extraction, we do not use external data or data augmentation.
the two-dimensional filter.
Table 3 shows the conditions of acoustic features andCNNs[3]. The features are 128th-order log mel filter bankoutputs. The frame length and frame shift length in the frame
4. Experiment
analysis are 40 and 20 ms, respectively. When we apply
We use the dataset provided in DCASE2018 Task 5 for CNNs to features, we divide the time series of the features

177

Table 2: Number of acoustic signals of each class in the development set
activityabsencecooking
dishwashingeatingworkingsocial activity
vacuum cleaning
watching TV
othertotal# sessions
4213101333
21
99118
268

# 10 s segments
18860
5124
1424
2308
18644
4944
972
18648
2060
72984

tures. The experimental results confirmed that the classification performance is improved by expanding the filter to threedimensions and further increasing the number of input channels. The best F-score of 87.7% was given by 3D-CNN withthree channels.
Acknowledgment
This work was supported by the Japan Society for the Promotion of Science (JSPS) KAKENHI Grant No. 19H04131.

Table 3: Conditions of the acoustic features and CNNs
acoustic featureframe length
frame shift length
audio block size
# of conv. layersoptimization method
epochTable 4: F-score of each method for different number of input
channels
2ch
3ch
4ch
2D-CNN 84.6 86.5 85.2
3D-CNN 85.4 87.7 86.8

References

128th-orderlog mel filter bank outputs
40 ms
20 ms
10 frame (without overlap)
8Adam
50

[1] DCASE 2018 Challenge Task 5 website,
http://dcase.community/challenge2018/taskmonitoring-domestic-activities.
[2] T. Inoue, P. Vinayavekkhin, S. Wang, D. Wood, N. Greco, R. Tachibana,Domestic activities classification
based on CNN using shuffling and mixing data augmentation, DCASE2018 Challenge Technical Report,2018.

into blocks with ten frames. Adam [7] is used as the optimization method for training the classifier, and the numberof epochs during training is set up to 50 and the epoch thatgives the best F-score is adopted. Chainer [8] is used for theimplementation of the 2D-CNN and 3D-CNN.
Table 4 shows the F-scores of 2D-CNN and 3D-CNN for
different number of input channels. The F-scores of 3D-CNN
are better than that of 2D-CNN in any channels. It shows thatfeatures across the time–frequency–space domain give better results. Also, we examine the effectiveness of increasing
the number of input channels. In both methods, increasing
the number of input channels improves the F-score. This isbecause more rich spatial information becomes available byusing three or more microphone inputs. The best F-score wasgiven by 3D-CNN with three channels, which is 87.7%. Thisis equivalent to the 5th place in DCASE2018 Task 5. The useof data augmentation technique, like the top ranking methods,
would give better results.

[3] G. Takahashi, T. Yamada, S. Makino, Acoustic sceneclassification based on spatial feature extraction using
convolutional neural networks, Proc. NCSP2018, pp.
351-354, Mar. 2018.

5. Conclusion[7] D. P. Kingma, J. L. Ba, Adam: A method for stochasticoptimization,” ArXiv:1412.6980v1, Dec. 2014.

In this paper, we compared 2D-CNN and 3D-CNN to examine a suitable structure of CNN for extracting spatial fea-

[8] Chainer website, https://chainer.org/.

178[4] S. Adavanne, A. Politis, T. Virtanen, ”Multichannel sound event detection using 3D convolutional neural networks for learning inter-channel features,”Proc. IJCNN2018, pp. 1–7, July 2018.
[5] S. Ioffe, C. Szegedy, “Batch normalization: accelerating deep network training by reducing internal covariate
shift,” ArXiv:1502.03167v1, Feb. 2015.
[6] G. Dekkers, S. Lauwereins, B. Thoen, M. W. Adhana,
H. Brouckxon, B. V. D. Bergh, T. V. Waterschoot,B. Vanrumste, M. Verhelst, P. Karsmakers,
Thesins database for detection of daily activities in ahome environment using an acoustic sensor network, ”
Proc. DCASE2017, pp. 32-56, Nov. 2017.