JAIST Repository
https://dspace.jaist.ac.jp/

Title

Speech Emotion Recognition Using 3D Convolutions
and Attention‑Based Sliding Recurrent Networks
With Auditory Front‑Ends

Author(s)Peng, Zhichao; Li, Xingfeng; Zhu, Zhi; Unoki,
Masashi; Dang, Jianwu; Akagi, Masato

Citation

IEEE Access, 8: 16560‑16572

Issue Date

2020‑01‑20

Type

Journal Article

Text version

publisher

URL

http://hdl.handle.net/10119/16212

RightsZhichao Peng, Xingfeng Li, Zhi Zhu, Masashi
Unoki, Jianwu Dang, and Masato Akagi, IEEE
Access, 8, 2020, pp.16560‑16572.
DOI:10.1109/ACCESS.2020.2967791. This work islicensed under a Creative Commons Attribution 4.0
License. For more information, see
http://creativecommons.org/licenses/by/4.0/

Description

Japan Advanced Institute of Science and Technology

Received December 16, 2019, accepted January 11, 2020, date of publication January 20, 2020, date of current version January 28, 2020.
Digital Object Identifier 10.1109/ACCESS.2020.2967791

Speech Emotion Recognition Using 3D
Convolutions and Attention-Based Sliding
Recurrent Networks With Auditory Front-Ends
ZHICHAO PENG 1,2 , XINGFENG LI
AND MASATO AKAGI 1

1 , ZHI ZHU

1 , MASASHI UNOKI

1 , JIANWU DANG

1,2,3 ,

1 Graduate School of Advanced Science and Technology, Japan Advanced Institute of Science and Technology, Ishikawa 9231292, Japan
2 Tianjin Key Laboratory of Cognitive Computing and Application, College of Intelligence and Computing, Tianjin University, Tianjin 300350, China
3 Peng Cheng Laboratory, Shenzhen 518055, China

Corresponding author: Zhichao Peng (zcpeng@jaist.ac.jp)
This work was supported in part by the JSPS KAKENHI Grant under Grant 16K00297, and in part by the Research Foundation of
Education Bureau of Hunan Province, China, under Grant 18A414.

ABSTRACT Emotion information from speech can effectively help robots understand speaker’s intentions innatural human-robot interaction. The human auditory system can easily track temporal dynamics of emotion
by perceiving the intensity and fundamental frequency of speech, and focus on the salient emotion regions.
Therefore, speech emotion recognition combined with the auditory mechanism and attention mechanism
may be an effective way. Some previous studies used auditory-based static features to identify emotion while
ignoring the emotion dynamics. Some other studies used attention models to capture the salient regions ofemotion while ignoring cognitive continuity. To fully utilize the auditory and attention mechanism, we firstinvestigate temporal modulation cues from auditory front-ends and then propose a joint deep learning modelthat combines 3D convolutions and attention-based sliding recurrent neural networks (ASRNNs) for emotion
recognition. Our experiments on the IEMOCAP and MSP-IMPROV datasets indicate that the proposed
method can be effectively used to recognize the emotions of speech from temporal modulation cues. The
subjective evaluation shows that the attention patterns of the attention model are basically consistent withhuman behaviors in recognizing the emotions.
INDEX TERMS Auditory front-ends, 3D convolutions, joint spectral-temporal representations, attentionbased sliding recurrent networks, speech emotion recognition.

I. INTRODUCTION

Speech is the most natural way for communication between
humans and robots. The key point of effective communication is to make robots or virtual agents understand speakers’
true intentions. However, only using the linguistic information is by no means sufficient enough for understanding ofintentions. The vocal emotion information as a kind of nonlinguistic information can significantly help robots or virtual
agents to understand speakers’ true intentions. Therefore,
speech emotion recognition (SER) is the research hotspot innatural human-robot interaction (HRI). Nevertheless, effective SER is still a very challenging problem, partly due to
The associate editor coordinating the review of this manuscript andapproving it for publication was Shiqing Zhang
16560

.

the cultural differences, various expression types, context,
ambient noise, etc.
In most of the past SER, low-level descriptors (LLDs)
were extracted from speech and were used to classify different emotion states by means of the conventional machinelearning methods such as hidden Markov models (HMM),
Gaussian mixture model (GMM), and support vector machine(SVM) [1]. However, it is still difficult to find the salientfeature set from LLDs to recognize distinct emotions, becauseof the aforementioned challenging factors. The human auditory system can easily perceive the intensity and fundamental
frequency of speech, and can track temporal dynamics ofemotion from the perceived information and focus on thesalient emotion regions. Therefore, speech emotion recognition combined with the auditory mechanism of auditory

This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see http://creativecommons.org/licenses/by/4.0/

VOLUME 8, 2020

Z. Peng et al.: Speech Emotion Recognition Using 3D Convolutions and Attention-Based Sliding Recurrent Networks

front-ends and attention mechanism of auditory back-ends
may be an effective way.
In auditory front-ends, temporal modulation cues are
obtained using auditory filtering of speech signal and modulation filtering of temporal amplitude envelope. These cues
contain rich spectral-temporal information to perceive thevariations of intensity, duration and pitch of speech [2]and have been widely used in sound-texture perception [3],speaker-individuality perception [4], speech recognition [5],and emotion recognition [6], [7]. Most studies extracted themodulation spectral features (MSFs) from temporal modulation cues by calculating the spectral centroid, flatness,skewness, kurtosis, and other statistical features. Wu et al. [6]showed that the MSFs perform better than the traditional
acoustic features such as Mel frequency cepstral coefficient(MFCC) and perceptual linear predictive (PLP) coefficientfor SER. Zhu et al. [7], [8] further confirmed that the MSFs
contribute to the perception of vocal emotion. However,the MSFs are only calculated in each modulation channel andproduce time-averaged static features in those studies. Since
emotion in speech is often communicated by varying temporal dynamics in the signal, the temporal dynamics are very
important factors in emotion recognition. The MSFs cannot
reflect the real emotion in speech since it lost the important
temporal cues. For these reasons, we should extract the joint
spectral-temporal features from temporal modulation cues toaccurately describe emotion dynamics.
Recent convolutional neural networks (CNNs) show powerful abilities of feature learning and have been used foracoustic modeling and feature extraction for SER. As human
auditory system responds to joint spectral-temporal patterns
in the speech signal rather than temporal-only or spectralonly patterns [9]. Inspired by auditory signal processing,in our previous study [10], we proposed an end-to-end SER
system using 3D CNNs to learn a joint spectral-temporalfeature from temporal modulation cues containing acoustic
frequency components, modulation frequency components,
and temporal features. The modulation frequency components consist of six filters spaced on a logarithm scale from
2 to 64 Hz. Such modulation frequency components include
the local information about variations of intensity and duration. However, it did not take into account of obtaining theperiodicity information about F0 from the modulation frequency band. The frequency band between about 50 and
500Hz is related to the periodicity information about F0,
which has been shown to be important for speech perception [11]. To obtain both the local features and periodicity
information, in this study, we improve the 3D convolution
model by increasing the modulation filters and reducing theconvolutional kernel size.
To capture the variations of local features and periodicity information from the feature sequence, we need to
extract utterance-level features for classifying emotional
speeches through time series modeling. Long short-term
memory recurrent neural networks (LSTM-RNNs) have powerful abilities of time series modeling to handle temporal
VOLUME 8, 2020

dynamic information. LSTM can effectively capture the longrange time dependencies for sequence classification. However, it cannot avoid the slow training speed caused bybackpropagation-through-time (BPTT) in long sequences.
To reduce the training cost, in [10], the time sequence isdivided into non-overlapping subsequences in extraction ofsegment-level features. These discontinuous segment-levelfeatures cannot fully reflect the dynamic changes of realemotions. From a cognitive point of view, people can obtain
important information by scanning the temporal sequence
continuously and transmit it for higher-level processing.
In addition, people have superior abilities in paying attention
to the emotional regions meanwhile ignoring the emotionless regions. Most of studies did not take into account of
the human mechanism how to focus on the emotional segments while ignoring the emotionless segments. An utterance consists of a number of voiced and unvoiced segments.
The voiced segments can express emotion more than theunvoiced ones. It is unknown what kind of acoustic features
attract human to pay more attention on the salient emotional regions. Therefore, we will investigate the relation of
the acoustic features and human attention mechanism, andpropose a sliding recurrent method to realize the attention
mechanism. In the temporal attention method, the continuous segment-level internal representations are extracted by asliding window, and are used to capture the salient emotional
regions.
To fully utilize the human auditory mechanism and attention mechanism, in this study, we begin with the investigation of temporal modulation cues from auditory front-ends
and then find out a method to capture the salient emotional
regions. Based on the achievements, we propose a joint deeplearning model that combines 3D convolutions and attentionbased sliding recurrent neural networks (ASRNNs) as the
back-ends of the SER system. To show the benefit of the proposed model, we evaluate it on the IEMOCAP [12] and MSPIMPROV [13] datasets by comparing various models with theproposed model. Our results show that the proposed model
can achieve better results compared with traditional model
on both datasets. We also conduct the subjective evaluation toinvestigate the relevance between the attention patterns of thetemporal attention model and human attention in perceiving
emotional speech.
The main contributions of this work are summarized asfollows:
1) Inspired by the auditory signal processing andtemporal attention mechanism, we propose a speech emotion recognition system that combines auditory perceptionbased front-end and attention-based back-end. In this system,
the front-end is used to generate temporal modulation cues,and the attention-based back-end is used to identify the emotional states in natural speech.
2) We propose a 3D convolution model to obtain boththe local features and periodicity information of emotional
speech by a joint spectral-temporal feature learning from thetemporal modulation cues.
16561

Z. Peng et al.: Speech Emotion Recognition Using 3D Convolutions and Attention-Based Sliding Recurrent Networks

FIGURE 1. Speech emotion recognition system with auditory front-ends.

3) We propose an ASRNN to continuously scan the temporal sequence and focus on the emotional region. In thisneural network, the continuous segment-level internal representations are extracted by a sliding window and focus on thesalient emotion regions using a temporal attention model.
The rest of the paper is organized as follows. In Section 2,
we introduce the auditory front-ends to produce temporalmodulation cues. Section 3 details the 3D convolutions tolearn a joint spectral-temporal feature representation fromthose cues and ASRNNs to focus on the salient emotion
regions. In Section 4, we also investigate the impacts of experiments on different situations. We discuss the implicationsof this study in Section 5. Finally, we draw conclusions in
Section 6.
II. PROPOSED AUDITORY FRONT-ENDS OF EMOTIONRECOGNITION SYSTEM
A. OVERVIEW OF EMOTION RECOGNITION SYSTEM

An overview of the proposed SER system is illustrated
in Fig. 1. The auditory front-ends of this system are used
to functionally simulate the signal processing in the auditory
system from the cochlea through the thalamus, as depicted in
the left part of Fig. 1.
The auditory front-ends are composed of three parts: auditory filterbank, temporal envelope extraction and modulation
filterbank. The auditory filterbank is responsible to decompose speech signals into acoustic frequency components as afunction of the acoustic frequency analyzer in the cochlea.
In this study, we use Gammachirp filterbank [14] as theauditory filterbank because this filter is adequate for reproducing psychophysically estimated human auditory filters
over a wide range of center frequencies and levels [15], [16].
Furthermore, temporal envelope extraction from the acoustic
frequency components is used to effectively simulate themechanical-to-neural signal transduction in the inner haircells (IHCs).
Modern psychophysical models of temporal modulation
processing suggest that the temporal envelope is processedby joint spectral-temporal modulations [17]. The spectraltemporal modulation contains the 3D modulated spectrum
16562

with dynamic peaks, which relates directly to speech perception [9]. Hence, the modulation filterbank is introduced
to generate 3D spectral-temporal representations from thetemporal envelope.
The back-ends of this system are depicted in the right partof Fig. 1. 3D convolutions are firstly used to extract joint
frame-level features including not only variations informationof intensity and duration but also the periodicity information.
Further, ASRNNs are used to focus on the salient emotional
regions by extracting segment-level features in a sliding window manner and utterance-level features with a temporalattention model.
B. FRONT-END SIGNAL PROCESSING

In the auditory front-end, the emotional speech signal y (t) is
first filtered by a bank of Gammachirp auditory filters. The
output of the nth channel signal is given bysg (n, t) = gc (n, t) ∗ y (t) ,1 ≤ n ≤ N,

(1)

where gc (n, t) is the impulse response of the nth channel, t
is the sample number in the time domain, N is the number ofchannels in the auditory filterbank, and ∗ denotes the convolution. The center frequencies of these filters are proportional
to their bandwidths, which in turn are characterized by theequivalent rectangular bandwidth (ERBN ) [18]:ERBN (fn ) =fn+ Bmin ,Qear(2)

where fn is the center frequency of the nth filter, Qear isan asymptotic filter quality at large frequencies, Bmin is
minimum bandwidth at low frequencies. Filter quality is ameasure of its center frequency divided by the bandwidth.
The most widely accepted is provided by [19] in which Qear
and Bmin are 9.26449 and 24.7, respectively. This impulse
response of Gammachirp filter is the product of the Gamma
distribution and sinusoidal tone.
gc (n, t) = At a1 −1 exp −2π wf ERBN (fn ) t
× cos (2πfn t + c1 ln(t) + ϕ),(3)

VOLUME 8, 2020

Z. Peng et al.: Speech Emotion Recognition Using 3D Convolutions and Attention-Based Sliding Recurrent Networks

where At a1 −1 exp(−2π wf ERBN (fn )t) is the amplitude term
represented by the Gamma distribution, A, a1 and wf are theamplitude, filter order, and bandwidth of the filter, respectively. The c1 ln (t) term is the monotonic frequency modulation term, ϕ is the original phase, and ERBN (fn ) is abandwidth of the auditory filter in fn . The chirping propertiesof the Gammachirp filter are largely determined by those ofits ‘‘passive’’ asymmetric filter at all levels and have beenshown to fit those of auditory nerve fibers well [14].
The envelope is extracted using the Hilbert transform tocalculate the instantaneous amplitude se (n, t) of the nthchannel signal. The se (n, t) is computed from sg (n, t) as
the magnitude of the complex analytic signal sbg (n, t) =sg (n, t)+jH{sg (n, t)}, where H{·} denotes the Hilbert transform. Hence,qse (n, t) = sbg (n, t) = s2g (n, t) + H2 sg (n, t) . (4)Furthermore, the mth modulation filter in the nth channel
signal is used to obtain the spectral-temporal modulation
signal sm (n, m, t) .
sm (n, m, t) = mf (m, t) ∗ se (n, t),

1 ≤ m ≤ M,

(5)

where mf (m, t) is the impulse response of the modulation
filterbank and M is the number of channels in the modulation
filterbank.
This type of signal generates a frequency-domain-specifictime-domain signal for each sub-channel and many subchannels comprise the 3D spectral-temporal representation.
Due to the high time-resolution of the spectral-temporal representations, a reduction in the number of samples for thetime domain has to be carried out. The reduction in the timeresolution is simply carried out by downsampling spectraltemporal representations with an 800-Hz rate. This operation
reduces the sequence length by a factor of 20.
C. MODULATION SPECTRAL REPRESENTATIONS

Figure 2 shows the different emotion examples of themodulation spectral representation with 32 acoustic channels and nine modulation channels from the IEMOCAP
dataset. Each utterance comes from the same speaker, named
Ses01F_impro05_F009 (Angry), Ses01F_impro03_F001(Happiness), Ses01F_impro04_F000 (Neutral emotion), andSes01F_impro02_F005 (Sadness), respectively. The y-axis
and x-axis of these representations are acoustic and modulation channels, respectively. Both channels are spaced on alogarithm-scale frequency. Modulated signals with standard
deviation are projected into the modulation and acoustic frequency space. Panels (a) to (d) in Fig. 2 show the modulation
spectral representations of anger, happiness, neutral emotion
and sadness, respectively. As slow modulation frequency,
particularly below 16 Hz (modulation channel equals to 4),
can extract local information about variations of intensity,
duration, attack, decay, and segmental cues of speech [20].
From these panels, we can find that the different emotion
has different low frequency modulation information, suggesting they could be discriminated from each other. In [10],VOLUME 8, 2020

FIGURE 2. Different emotion examples of the modulation spectral
representation with 32 acoustic channels and nine modulation channels
from the IEMOCAP dataset.

we therefore used six modulation filters to extract low frequency information (below 64 Hz) for emotion recognition.
Although fast modulation frequency is less important thanslow modulation frequency, it still contains the periodicity
information to reflect emotional changes. Figure 2 also showsthat the periodicity information is retained between the seventh and ninth modulation channels. In addition, for the same
fast modulation frequency, it shows that the acoustic frequency of anger and happiness is higher than that of sadness
and neutral emotion. For this reason, we use nine modulation
filters with upper limit of modulation frequency (512 Hz)
instead of six filters to obtain periodicity information for
emotion recognition.
III. METHODSAs illustrated in the right a of Fig. 1, the proposed backends of the SER system are composed of two components:
3D convolutional model and attention-based sliding recurrent
networks.
A. 3D CONVOLUTIONAL MODEL

Since deep convolutional model keeps the spectral-temporaltranslation invariance for speech signal processing, it isoften used to extract high-level features for speech emotion
recognition. Most studies used CNNs to extract 2D feature representations from speech spectrograms [21], [22] or
Mel-scaled filterbank representation [23], [24]. Recently
some studies proposed 3D convolution models to better capture the spectral-temporal relationship of the feature representations for emotion recognition. Chen et al. [25] proposed
attention-based CRNN from a 3D feature representation bycomputing the log Mel-spectrogram with deltas and deltadeltas for emotion recognition. Kim et al. proposed deep
16563

Z. Peng et al.: Speech Emotion Recognition Using 3D Convolutions and Attention-Based Sliding Recurrent Networks

TABLE 1. 3D convolutional neural networks architecture.

3D CNNs for spectral-temporal feature learning by dividing the speech signal into several sub-segments and thesesub-segments contain 2D feature maps with 256 points logspectrogram for every 20 ms [26]. In this study, the temporalmodulation cues from the auditory front-ends contain 3D
spectral-temporal representation. The back-ends of the SER
system are responsible for extracting high-level features fromthe 3D representation. CNNs have superior feature extraction power inspired from biological neural networks andcan extract high-level local feature representations using thespectral-temporal receptive field of the neuron. Therefore,
we use 3D CNNs to learn a joint spectral-temporal featurefrom the 3D representation for obtaining local features andperiodicity information.
The architecture of 3D CNNs is described in Table 1.
The first convolutional layer (Conv1) is used to extract 3Dfeatures that are composed of acoustic frequency, modulation
frequency, and time sequences. These features are fed into thenext two convolutional layers (Conv2 and Conv3) to model
high-level feature representations for time series. The data
format of the input and output data is designed as ‘‘D × H ×
W’’, where D, H, and W are the data in the acoustic channels(depth), modulation channels (height), and time sequence(width), respectively. In this study, the input size is set as
32 × 9 × 6000 and the size of the kernels is 2 × 2 × 4.
To reduce computational complexity, the stride for Conv1 is
set to 1 × 1 × 2, and that for the other convolutional layers
is set to 1 × 1 × 1. Each convolutional layer includes batchnormalization and rectified linear unit (ReLU) operations.
Batch normalization is used to accelerate training of deep
network [27]. The first pooling layer (Pool1) before conv2 hasa kernel size of 2 × 2 × 1 and stride of 2 × 2 × 1 with
max-pooling operation. The second pooling layer (Pool2)
has a kernel size of 2 × 2 × 2 and stride of 2 × 2 × 2.
This means that spectral-temporal pooling is executed on
Pool2. The third pooling layer (Pool3) has a kernel sizeof 2 × 1 × 2 and stride 2 × 1 × 2. This means that the
acoustic frequency channel and temporal pooling is executed
while the modulation frequency channel remains on Pool3.
The max-pooling operations in each pooling layer is used to
extract robust features against background noise, especiallyfor the waveform signals. These three pooling layers reduce
the output size of the time sequence by a factor of 20 on thetemporal length. This means that the 3D convolution only
16564

FIGURE 3. Attention-based sliding recurrent networks.

learns the frame-level features in 22.5ms for each point. The
feature maps of the three convolution layers are 20, 32, and64, respectively. Finally, we obtain the output of Pool3 withthe shape of 750 × 4 × 2 × 64 after transposing the axis of
the tensor then reshape it to 2D shapes of 750 × 512.
B. ATTENTION-BASED SLIDING RECURRENT NEURALNETWORKSPart of the attention system of the brain is involved in thecontrol of thoughts, emotions, and behavior. In human auditory system, selective auditory attention tracks the temporal
dynamics of emotion by continuous scanning and encodingof the speech signals [28]. Inspired by the selective auditory
attention in auditory system, we propose an ASRNN model
to seize the emotional parts from temporal dynamics information in speech. Among them, a sliding window is used
to extract the continuous segment-level emotional featurescontaining temporal dynamics information. Then, a temporalattention model is used to capture the important information
related to emotion in each utterance.
1) SLIDING RECURRENT NEURAL NETWORKS

The sliding recurrent neural networks (SRNNs) are used tocontinuously extract the intermediate segment-level representations for short-term sequence depicted in Fig. 3. The inputof the SRNNs is T × D, where T represents the total length of
the time sequence and D represents the feature vector size. xk
is the input to the LSTM block of kth sliding input sequencewith Z time frames.
xk = {x(k,1) , . . . , x(k,Z ) },x(k,t) ∈ RD , 1 ≤ t ≤ Z(6)

VOLUME 8, 2020

Z. Peng et al.: Speech Emotion Recognition Using 3D Convolutions and Attention-Based Sliding Recurrent Networks

Each xk is fed frame-by-frame into the LSTM units. The
formulation of LSTM with peephole connections can be
described by the following equations:i(k,t) = σ (Wix x(k,t) + Wih h(k,t−1) + Wic c(k,t−1) + bi )(7)

f(k,t) = σ (Wfx x(k,t) + Wfh h(k,t−1) + Wfc c(k,t−1) + bf )(8)cg(k,t) = tanh(Wcx x(k,t) + Wch h(k,t−1) + bc )(9)c(k,t) = f(k,t)(10)

c(k,t−1) + i(k,t)

cg
(k,t)

o(k,t) = σ (Wox x(k,t) + Woh h(k,t−1) + Woc c(k,t) + bo )h(k,t) = o(k,t) tanh c(k,t) ,

(11)
(12)

where i(k,t) , f(k,t) , o(k,t) , c(k,t) , and h(k,t) are the input gate,
forget gate, output gate, cell state, and output of the LSTM
block, respectively, at the current time step t. The weightmatrices Wi∗ , Wf ∗ , and Wo∗ transform xk and hidden stateh(k,t−1) , respectively, to cell update cg(k,t) and three gatesi(k,t) , f(k,t) , and o(k,t) . Finally, bi , bf , bo are the additive biasesof the input gate, forget gate, and output gate, respectively.
The set of activation functions consists of the logistic sigmoid
function σ (·), element-wise multiplication , and hyperbolic
tangent function tanh(·).
Specifically, we use a bidirectional LSTM (BLSTM) network in this study, where the sequence of received signals
is once fed in the forward direction into one LSTM cell,and once fed in backwards into another LSTM cell. The
forward LSTM reads the time sequence in its original orderand generates a hidden state fh(k,t) = {fh(k,1) , . . . , fh(k,Z ) }at each time step. Similarly, the backward LSTM reads thetime sequence in its reverse order and generates a sequence ofhidden states bh(k,t) = {bh(k,Z ) , . . . , bh(k,1) }. The last stateof the forward and backward LSTM cells carry informationof the entire source sequence. We concatenate the last state of
the forward and backward LSTM cells to produce the hk of ksequence.
hk = [fh(k,Z ) , bh(k,1) ](13)

Each hidden state hk contains information of each sliding
window sequence. The hidden states of the recurrent layer
along the different frames of the window are used to computethe extracted features. The output of this layer for each sliding
window is the cell state vector of the last time frame in eachsliding window. After processing in each sliding window,
we shift S time frames to compute the next sliding window
with the valid padding. The number of sliding window L is
calculated as
L = d(T − Z)/Se.

(14)

The BLSTM has 512 hidden units for both directions ineach sliding window. Finally, we create a new sequence withthe shape of L ×1024 to put into the attention model. The
same parameters of the LSTM cell are used in each sliding
sequence, then a new context sequence h is produced.
h = {h1 , . . . , hL },VOLUME 8, 2020hk ∈ R2D , 1 ≤ k ≤ L(15)

FIGURE 4. Attention weights.

2) TEMPORAL ATTENTION MODEL

Because there are many speech frames that are unrelated tothe expressed emotion, such as silence, the attention mechanism is mainly used to focus only on the significant emotionalpart of the speech signal. Recently, some studies proposed
attention models to adjust weights for each of the speechframes depending on their importance based on LLDs using
a RNN [29], [30]. The silence regions can be addressedusing a voice activity detection (VAD) [31] or by null labelalignment [32]. Wang et al. [31] proposed an attention modelof learning utterance-level representations to improve classification after using a VAD to filter out silence frames andmini-batch training in each utterance. Lee and Tashev [32]extracted high-level representation of emotional states withregard to its temporal dynamics using the BLSTM approach,in which they assume that different frames should have different labels and the label sequence should be alternating
between the utterance-level label and a newly introduced
NULL state. Neumann and Vu [33] proposed an attentive convolutional neural network (ACNN) to test the emotional discrimination of different feature set. In addition, self-attention
based deep model [34], [35] demonstrated the effectivenessto improve the performances for SER. Unlike these studies,
we apply a temporal attention model to the sliding window
sequence instead of applying one based on LLDs.
Sequence h is fed into feedforward neural networks
then concatenated with sinit , as depicted in Fig.4. Subsequently, a ReLU is used to produce non-linear transformations R(sinit , hk ).
R (sinit , hk ) = Uk ReLU (sinit + Wk hk + bk ),(16)

where Wk , Uk are the trainable parameter matrices, bk is thebias vector, and sinit is the initial hidden state of the sliding
recurrent sequence. We use the non-linear function of the
ReLU due to its good convergence performance. For each hk ,16565

Z. Peng et al.: Speech Emotion Recognition Using 3D Convolutions and Attention-Based Sliding Recurrent Networks

the αk can be computed as follows:

TABLE 2. Accuracy comparison of static features on IEMOCAP and
MSP-IMPROV dataset (%).

exp R(sinit, hk )αk = PL.
l=1 exp R(sinit, hl )
(17)

We then obtain the attention weights αk of each sliding
sequence from the attention model. The output of the attention layer, attention_sum, is the weighted sum of h.
XLattention_sum =αk hk .
(18)
k=1

The weighted sum of sequence h is fed into a unidirectional LSTM cell to obtain a hidden vector hs . The featuresconcatenated by h and hs are fed into feedforward neuralnetworks. Subsequently, we use a ReLU as the activation
function, which brings the non-linearity into the networks.
Finally, we use the softmax to produce the emotion state
distribution. To avoid overfitting when training our networks,
we use a dropout rate of 0.5 before feed forward layers during
training.
IV. EXPERIMENT RESULTS
A. EXPERIMENTAL DATASET AND EVALUATION METRICS

We conduct speaker-independent experiments using the
IEMOCAP and MSP-IMPROV datasets. Both datasets arecomposed of multimodal interactions of dyadic sessions andlabeled by three annotators for emotions such as happy, sad,angry, excited, and neutral, along with dimensional labels
such as valence and arousal. In this study, we only use four
emotional categories for both datasets: Happy, Sad, Angry,
and Neutral.
The IEMOCAP dataset consists of five sessions, where
each session contains scripted and improvised utterancesfrom two speakers (one male and one female). For thisstudy, we include excitement utterances with happiness ones.
We take 5,531 utterances (1636 happy, 1084 sad, 1103 angry,
1708 neutral) for all sessions. The mean length of all the turns
is 4.55 s (max.: 34.14 s, min.: 0.58 s).
The MSP-IMPROV dataset consists of six sessions in thesame manner (12 unique speakers). Each session includes
all the speaking turns of the improvisation and the naturalinteraction based on the 20 target sentences in the improvised
scene. The final dataset contains a total of 7798 utterances(2644 happy, 885 sad, 792 angry, 3477 neutral). The mean
length of all the turns is 4.09 s (max.: 31.09 s, min.: 0.41 s).
Since the input length for a CNN has to be equal for all
samples, we set the maximal length to 7.5 s (mean duration
plus standard deviation). Longer turns are cut at 7.5 s, andshorter ones are padded with zeros. The class distribution is
unbalanced in both datasets, especially for MSP-IMPROV
dataset, the number of utterances belonging to happy/neutralclass more than three times that of angry/sad. Unweighted
accuracy (UA) is the average classification accuracy for eachemotion. It is a better measurement if the class distribution is
not balanced. Hence, we use UA as the performance metricof the proposed framework to avoid being biased to the larger
classes.
16566

B. EMOTION RECOGNITION SYSTEM WITH STATICFEATURESFirstly, we investigate the conventional emotion recognition
system with static features which are computed using fixedstatistical functions to the hand-crafted LLDs. We extract
MFCC, emobase2010, IS09 [36], and IS13 ComParE [37]features using the Munich open Speech and Music Interpretation by Large Space Extraction (openSMILE) toolkit [38].
All features are first normalized by specific z-normalization.
Secondly, to investigate the effectiveness of static modulation
features on emotion recognition, we also extract the MSFs
by calculating the spectral centroid, spread, skewness, andkurtosis from the modulation spectral representation. For
each feature set, we train a linear SVM model to recognize
the speech emotion using LibSVM [39] and Weka toolkits [40]. All results are presented by leave-one-session-out
cross-validation. Table 2 shows the accuracy comparison of
static features on IEMOCAP and MSP-IMPROV datasets.
The best result is 54.9 percent for IEMOCAP using theoriginal static features with 1,582 dimensions whereas thebest result is 43.2 percent for MSP-IMPROV using the static
modulation features with 160 dimensions. The results also
show that MFCC features achieve the worst results, which
may be due to the minimum number of MFCC features(only 39 dimensions features). Similar to the results from [6],the MSFs perform better than MFCC for emotion recognition
on both datasets. Emotion information from speech changes
dynamically over time, but the static features do not containtemporal dynamics information which plays a key role in theemotion recognition process.
C. SETUP OF AUDITORY-BASED DEEP LEARNING MODELSIn the front-end signal processing, we first resample thespeech signal with a sampling frequency of 16000 Hz andapply a pre-emphasis filter to compensate for the effect ofsound source. We subsequently use normalization to remove
the difference of the speakers by mapping the signal values
to mean 0 and the standard derivation to 1 in each utterance.
The sound-pressure level is set to 60 dB, which approximatesto a normal voice. Furthermore, we introduce the compressive Gammachirp filterbank with 32 filters to provide thecompressive characteristics. The frequency of Gammachirp
filterbank distributed on the ERBN scales is between 0.1 and
8 kHz. The modulation filterbank is also used to control theenvelopes of octave bands from 2 to 512 Hz, consisting of
VOLUME 8, 2020

Z. Peng et al.: Speech Emotion Recognition Using 3D Convolutions and Attention-Based Sliding Recurrent Networks

nine filters (one low-pass filter and eight band-pass filters).
The low-pass filter is a 2nd order Butterworth infinite impulse
response (IIR) filter with a cut-off frequency of 2 Hz. The cutoff frequencies of the band-pass filters are equally spaced ona logarithm scale from 2 to 512 Hz.
In the back-ends of the SER system, a joint deep learning model combined 3D convolution and ASRNN is used.
To train the model with a speaker-independent property,
we use leave-one-session-out cross-validation. In each experiment, four sessions are used for training the deep model andone session is divided into two sub-sessions depending on thegender in both datasets. For all random weight initializations,
we choose L2 regularization. The parameters are learned in an
end-to-end manner, meaning that all parameters of the modelare optimized simultaneously using the Adam optimization
method with a learning rate of 1e-4 to minimize cross-entropy
loss. The batch size is 10, and maximum epoch is 30 withearly-stopping. The process stops if the UA does not improvefor 8 consecutive epochs.

TABLE 3. Accuracy comparison with different sliding-widows and shift
lengths in ASRNN architecture on IEMOCAP and MSP-IMPROV dataset(%).

TABLE 4. Confusion matrix (%) of ASRNN with an average accuracyof 62.6% on the IEMOCAP dataset.

TABLE 5. Confusion matrix (%) of ASRNN with an average accuracyof 55.7% on the MSP-IMPROV dataset.

D. IMPACTS OF SLIDING WINDOW AND SHIFT LENGTHSSRNNs are used to obtain continuous internal representations while maintaining good computational efficiency. The
continuous internal representations can be extracted using asliding window. At the same time, computational efficiency
can be improved by segmenting a feature sequence into
multi sub-sequence. However, choosing different lengths ofwindow and shift will affect the recognition accuracy andcomputational efficiency of emotional recognition system.
To reach higher recognition accuracy and computational
efficiency, we investigate the effect of the sliding window andshift lengths using IEMOCAP dataset. First, the entire feature sequence is divided into multi-subsequences in a sliding
manner. The length of each subsequence is much shorter thanthe original sequence, and the model can be trained rapidly
using BPTT. Then, we run the proposed system five times andobtain the average accuracy in the case of different sliding
window and shift lengths. We consider the different sliding
window lengths of 10, 20, 30, 40, 50, and 100, which mean
the duration of the sequence from 200 to 2000 ms. We also
consider the shift lengths of 5, 10, and 20, which mean that
it will produce 150, 75, and 38 sliding subsequences in thesame padding manner for the duration of the convolutionalsequence with 750 × 512. When the sliding window lengthis 100 with a shift length of 10, the training time of the
ASRNN architecture is close to that of the entire sequence
fed into the recurrent networks. Hence, we do not consider alonger sliding window that will take longer time in training
the model. One session in the dataset is chosen for testing andothers for training. We find that the computational efficiency
will be improved with the shortening of window length andthe lengthening of shift. But in this case, the recognition
accuracy will decrease due to the inability to extract more
emotional features. In addition, because only the feature of
the last time frame in each sliding window is retained, when
the window length is too long, not only the computational
VOLUME 8, 2020

efficiency will be reduced, but also the recognition accuracy
will be reduced. The results obtained for each method areshown in Fig.5. Recognition accuracy is closer when the shift
length is 5 or 10, but it became worse when the shift lengthis 20. This figure also shows that the ASRNN architecture
resulted in better accuracy when the sliding window length is
20 or 40. Therefore, we only consider sliding window lengthsof 20 and 40 and shift lengths of 5 and 10.
E. RESULTS WITH ASRNNS ARCHITECTURE

Table 3 shows the recognition results using different lengths
and shift of the sliding window with the ASRNNs architecture for both datasets. One can see that the ASRNNsarchitecture with the sliding window length of 20 and shift
length of 10 performed better than the others, whose recognition accuracy is 62.6% for IEMOCAP and 55.7% for MSPIMPROV. The results are much better than those obtainedusing the traditional parameters shown in Table 2. According
to the results, the window length of 20 frames (about 400ms)
is suitable for expressing segment-level emotions, while theshift length of 10 is better for classification than that withthe shift length of 5. Comparing with the best results oftraditional recognition system in Table 2, the proposed systemachieved +7.7 and +12.5% absolute accuracy improvements
on IEMOCAP and MSP-IMPROV, respectively. These results
indicate that the proposed system with temporal dynamics
information is better to recognize emotional states than theconventional system with static features.
16567

Z. Peng et al.: Speech Emotion Recognition Using 3D Convolutions and Attention-Based Sliding Recurrent Networks

TABLE 6. Accuracy comparison (%) between RNN architectures on the
IEMOCAP and MSP-IMPROV dataset.

FIGURE 5. The impact of sliding window and shift length on recognition
accuracy.

F. IMPACTS OF MODULATION CHANNEL, SLIDINGWINDOW AND ATTENTION MODEL

to 0, it becomes an ARNN. Hence, the attention model is usedon the entire time sequence.
Third, SRNNs with max and mean pooling are designed toevaluate whether the attention model can seize the emotional
regions. A SRNN has the same sliding window and shift
lengths as the ASRNN. There are two types of pooling used
in a SRNN: maximum and average, denoted as SRNN-Maxpooling and SRNN-Mean-pooling, respectively. These models mentioned above use the same convolutional networks
with the input shape of 32 × 9 × 6000.
Table 6 shows the comparison of results on different
types of SRNNs with attention and non-attention models andone ARNN. Compared with ASRNN-6MFB, the ASRNN
achieves the same improvements of +0.9% on both datasets.
This means that the proposed system with nine channels
may extract more information from speech than ASRNN6MFB. Compared with the ARNN, the ASRNN achieves+1.3% and +0.5% absolute improvements on the IEMOCAP and MSP-IMPROV datasets, respectively. This meansthat the segment-based attention model is better than framebased attention model. Compared with SRNN-Max-poolingand SRNN-Mean-pooling, the ASRNN achieves +0.9% and+1.5% absolute improvements on the IEMOCAP and MSPIMPROV datasets, respectively. This means that the attention
model is better than max- and mean-pooling.

In order to evaluate the effects of modulation channel number,sliding window and attention model on the SER system,
we design a number of comparative experiments in different
situations.
First, we evaluate the effects of the nine modulation filterbank in obtaining local features and periodicity information by comparing it to the one with six modulation filters(ASRNN-6MFB). ASRNN-6MFB is set as the same layers
as the ASRNN, but different inputs shape of 32 × 6 × 6000
result in different kernel and stride. Compare to ASRNN,
the difference is that the kernel and stride are 2×1×2 insteadof 2 × 2 × 2 in Pool2. In addition, the convolutional maps
are 40 instead of 64 to keep similar features in each frame.
Finally, the output shape is 4 × 3 × 750 in pool3. Then thislayer is reshaped to 2D shapes of 750 × 480.
Second, an attention-based recurrent neural network(ARNN) is designed to evaluate whether the sliding window can obtain more temporal dynamics information or not.
ARNN is a special case of an ASRNN. That is, when thesliding window length of an ASRNN is equal to the length ofthe entire convolution sequence and the shift length is equal

Recently, Kell et al. [42] demonstrated that a deep neuralnetwork made human-like error patterns. If our attention
model reflects human mechanism, its result should be similar
to human behaviors when they recognize speech emotion.
For this reason, a listening testing is designed to evaluate thesimilarity of the behaviors between the proposed attention
model and human. Thirty sentences from IEMOCAP dataset
are used for the listening tests. Each sentence with a duration
between 4.5 to 7.5 s is presented to at least 25 listeners(14 female and 11 male with ages ranging from 20 to 28)
in random orders. The listeners are asked to concentrate on
listening to each utterance and choose the two locations thatbest show the emotions of the utterance.
Figure 6 illustrates an example of comparisons between
the attention model and human temporal attention. The top
panel shows the waveform of an emotional sentence, and theupper middle panel shows the spectrogram of the sentence.
The lower middle panel shows the attention weights (αi )that are calculated based on auditory front-ends and deep

Table 4 and 5 show the confusion matrix of the best results
for the IEMOCAP and MSP-IMPROV datasets, respectively.
In general, the class distributions of the confusion matrix
for different session are basically similar. One can see thathappiness is easily confused with neutral emotion and viceversa. Anger is more easily misclassified as happiness thanhappiness being misclassified as anger. Unlike the study [41],the proposed system reduces the confusion between anger
and happiness categories to a major extent, especially in
MSP-IMPROV. Sadness is easily confused with neutral emotion in IEMOCAP, while it is easily confused with happiness
in MSP-IMPROV. The confusion in the proposed method
mainly happen between the neutral one and the others. Thisimplies that emotion recognition based on auditory front-ends
is basically consistent with people’s recognition of emotion.
In terms of the databases, the overall performance on IEMOCAP is better than MSP-IMPROV. The reason for this seems
to be that the MSP-IMPROV dataset is highly imbalanced.

16568

G. LISTENING TEST FOR TEMPORAL ATTENTIONVOLUME 8, 2020

Z. Peng et al.: Speech Emotion Recognition Using 3D Convolutions and Attention-Based Sliding Recurrent Networks

TABLE 7. Accuracy comparison of proposed system and other systems on
IEMOCAP and MSP-IMPROV dataset (%).

FIGURE 6. Analysis and comparison of attention model and humanselective attention for test example. Top panel: raw waveform(Ses01F_impro04_F033.wav from IEMOCAP dataset); upper middle panel:
spectrogram; lower middle panel: attention weight (α i ) over sliding
window time sequence; bottom panel: histogram shows attention
numbers for subjective judgments, and dashed line shows
moving-average with 2 data points.

frameworks. The bottom panel shows a histogram that isthe point numbers of attention position given by subjective
judgements, and a dashed line that is the moving-averageon two neighbor data points. One can see that the curveof the attention weights is similar to that of the subjective judgment. Pearson’s correlation coefficient is used toquantitatively measure the similarity between the attention
model and human temporal attention. The correlation coefficient is P = 0.552 (ρ< 0.001) between the attention
weights and histogram in this particular utterance. If wecalculate the correlation between the moving average values
and the attention weights, the correlation coefficient becomes
P = 0.715 (ρ < 0.001). This indicates that there is astrong correlation between human temporal attention andthe attention model. This implies that the proposed attention
model can reflect human selective attention to a large extent.
V. DISCUSSION

Taking into account, that the human auditory system has avery strong ability to perceive the intensity and fundamental frequency of speech, furthermore, it can track temporal
dynamics of emotion from the perceived information andfocus on the salient emotion regions, therefore, we propose a
SER system by combining auditory mechanism and attention
mechanism of human auditory system.
The auditory front-ends of the SER system are used to
produce temporal modulation cues, which contain local features and periodicity information of emotional speech. During the process of temporal modulation cues extraction,
VOLUME 8, 2020an additional correlation in neighboring channels will be
introduced because of the partially overlapped frequency.
Traditional methods use discrete cosine transform to decorrelate the temporal modulation features in the acoustic andmodulation frequency domains. Since CNNs can successfully
de-correlate the features in neighboring channels, we directly
use 3D CNNs to learn a joint spectral-temporal feature from
temporal modulation cues. Furthermore, temporal dynamic
information is obtained by continuously scanning the temporal sequence and then is transmitted to higher-level processing center. To focus on the emotional regions while ignore
the emotionless regions, an attention model is used to extractutterance-level features.
To show the benefit of the proposed model, we compare
our results with the studies [41], [43], [44] on both datasets aspresented in Table 7. In [43], the authors used Mel filterbank
features as the input to CNNs and showed that CNNs withthese features can produce competitive results to the popularfeature sets. In [41], the authors used Log-Mel filterbank
features as the input to autoencoder and used attentive CNN
for representation learning. In [44], the authors used raw
speech as input to parallel convolution layer and showedthat CNN-LSTM can capture multi-temporal dependencies.
Compared to these studies, we are achieving the better resultof 62.6% and 55.7% respectively on both datasets using
3D convolutions and ASRNNs from temporal modulation
cues. This indicates that the auditory front-ends can provide
spectral-temporal representations, and deep frameworks caneffectively extract emotional information from such representation for emotion recognition.
In addition, four representative studies with reported
results on IEMOCAP are selected as comparisons. In [45],the authors used static features of LLDs for representation
learning, and deep belief network for emotion recognition.
In [46], the authors used FFT bins with autoencoder for representation learning, and used RNN to identify the emotion
states. In [47], the authors used attention-based BLSTM models on LLDs for emotion recognition. Additionally, comparedwith our previous study [10], we are able to obtain faster
16569

Z. Peng et al.: Speech Emotion Recognition Using 3D Convolutions and Attention-Based Sliding Recurrent Networks

training speed with SRNNs, and this system can better identify happiness and anger. This may be benefited by the
9-channel modulation filterbanks that contain fundamental
frequency information, which is important for emotions.
In contrast, our study exceeded the accuracy compared to theleading studies.
Other studies used attention models to identify emotions
on IEMOCAP databases, but the experimental conditions aredifferent. For example, [25], [29], [30] did not merge happyand excited into one class, while [33] just reported weighted
accuracy. Unlike these frame-based attention models, we use
a sliding window based attention model to focus on the salient
emotion regions. The results of experiments showed that
this model can effectively obtain the emotional information.
The subjective evaluation shows that the attention patternsof the attention model are basically consistent with humanbehaviors in recognizing emotions.

VI. CONCLUSION

We proposed a SER system using 3D convolutions andattention-based sliding recurrent neural network based on
auditory front-ends. As the human auditory system is powerful in spectral-temporal signal analysis and processing,an auditory model, which mimics the function of the human
auditory system, is used as a front-end to extract spectraltemporal features in the SER system. Additionally, comparedwith modulation spectral features, these 3D features containtemporal dynamics characteristics and can avoid the modulation correlation problem.
Considering that local features and periodicity information can better express emotions, we used 3D convolutions
to extract frame-level features from nine modulation filters.
We then used recurrent networks to obtain temporal dynamics
information in each utterance. We also used an attention
model to focus on the emotionally salient parts of a speech
signal. Therefore, we propose a joint deep learning modelthat combines 3D convolutions and attention-based sliding
recurrent neural networks. To the best of our knowledge, thisis the first study on speech emotion recognition combining
auditory and cognitive mechanisms. Our experiments demonstrated that the proposed system can obtain spectral-temporal
representations and exhibit better recognition accuracy compared to that of state-of-the-art SER systems on both datasets.
In summary, an auditory model as a front-end can extract
rich spectral-temporal information, and the proposed system can effectively extract high-level features for emotion
recognition. This system is possibly applied to other audioevent perception and recognition. For future work, we plan
to conduct an experiment using categorical and dimensional
speech emotional datasets to analyze noise-robust emotion
recognition. In addition, inspired from the study [48] using
a filterbank layer in DNN to learning the filterbank features,
we plan to design the auditory and modulation filterbank
layers to produce 3D spectral-temporal representations foremotion recognition.
16570

REFERENCES[1] M. El Ayadi, M. S. Kamel, and F. Karray, ‘‘Survey on speech emotion recognition: Features, classification schemes, and databases,’’ Pattern
Recognit., vol. 44, no. 3, pp. 572–587, Mar. 2011.
[2] L. Atlas and S. A. Shamma, ‘‘Joint acoustic and modulation frequency,’’
EURASIP J. Appl. Signal Process., vol. 2003, no. 7, pp. 668–675, 2003.
[3] J. Mcdermott and E. Simoncelli, ‘‘Sound texture perception via statistics ofthe auditory periphery: Evidence from sound synthesis,’’ Neuron, vol. 71,
no. 5, pp. 926–940, Sep. 2011.
[4] Z. Zhu, R. Miyauchi, Y. Araki, and M. Unoki, ‘‘Contribution of modulation
spectral features on the perception of vocal-emotion using noise-vocoded
speech,’’ Acoust. Sci. Technol., vol. 39, no. 6, pp. 379–386, Nov. 2018.
[5] N. Moritz, J. Anemueller, and B. Kollmeier, ‘‘An auditory inspired amplitude modulation filter bank for robust feature extraction in automatic
speech recognition,’’ IEEE/ACM Trans. Audio Speech Lang. Process.,
vol. 23, no. 11, pp. 1926–1937, Nov. 2015.
[6] S. Wu, T. H. Falk, and W.-Y. Chan, ‘‘Automatic speech emotion recognition
using modulation spectral features,’’ Speech Commun., vol. 53, no. 5,
pp. 768–785, May 2011.
[7] Z. Zhu, R. Miyauchi, Y. Araki, and M. Unoki, ‘‘Modulation spectralfeatures for predicting vocal emotion recognition by simulated cochlear
implants,’’ in Proc. Interspeech, 2016, pp. 262–266.
[8] Z. Zhu, R. Miyauchi, Y. Araki, and M. Unoki, ‘‘Contributions of temporalcue on the perception of speaker individuality and vocal emotion fornoise-vocoded speech,’’ Acoust. Sci. Technol., vol. 39, no. 3, pp. 234–242,
May 2018.
[9] T. Chi, Y. Gao, M. C. Guyton, P. Ru, and S. Shamma, ‘‘Spectro-temporalmodulation transfer functions and speech intelligibility,’’ J. Acoust. Soc.
Amer., vol. 106, no. 5, pp. 2719–2732, Nov. 1999.
[10] Z. Peng, Z. Zhu, M. Unoki, J. Dang, and M. Akagi, ‘‘Auditory-inspired
end-to-end speech emotion recognition using 3D convolutional recurrent
neural networks based on spectral-temporal representation,’’ in Proc. IEEE
Int. Conf. Multimedia Expo, Jul. 2018, pp. 1–6.
[11] S. Rosen, ‘‘Temporal information in speech: Acoustic, auditory and linguistic aspects,’’ Philos. Trans. Roy. Soc. London B, Biol. Sci., vol. 336,
no. 1278, pp. 367–373, 1992.
[12] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim,
J. N. Chang, S. Lee, and S. S. Narayanan, ‘‘IEMOCAP: Interactive emotional dyadic motion capture database,’’ Lang Resour. Eval., vol. 42, no. 4,
pp. 335–359, Dec. 2008.
[13] C. Busso, S. Parthasarathy, A. Burmania, M. Abdelwahab, N. Sadoughi,and E. M. Provost, ‘‘MSP-IMPROV: An acted corpus of dyadic interactions
to study emotion perception,’’ IEEE Trans. Affect. Comput., vol. 8, no. 1,
pp. 67–80, Jan. 2017.
[14] T. Irino and R. Patterson, ‘‘A dynamic compressive gammachirp auditory
filterbank,’’ IEEE Trans. Audio Speech Lang. Process., vol. 14, no. 6,
pp. 2222–2232, Nov. 2006.
[15] R. D. Patterson, M. Unoki, and T. Irino, ‘‘Extending the domain of center
frequencies for the compressive gammachirp auditory filter,’’ J. Acoust.
Soc. Amer., vol. 114, no. 3, pp. 1529–1542, Sep. 2003.
[16] M. Unoki, T. Irino, B. Glasberg, B. C. J. Moore, and R. D. Patterson,‘‘Comparison of the roex and gammachirp filters as representations of theauditory filter,’’ J. Acoust. Soc. Amer., vol. 120, no. 3, pp. 1474–1492,
Sep. 2006.
[17] T. Dau, B. Kollmeier, and A. Kohlrausch, ‘‘Modeling auditory processingof amplitude modulation. II. Spectral and temporal integration,’’ J. Acoust.
Soc. Amer., vol. 102, no. 5, pp. 2906–2919, Nov. 1997.
[18] B. C. J. Moore and B. R. Glasberg, ‘‘Suggested formulae for calculating
auditory-filter bandwidths and excitation patterns,’’ J. Acoust. Soc. Amer.,vol. 74, no. 3, pp. 750–753, Sep. 1983.
[19] B. R. Glasberg and B. C. Moore, ‘‘Derivation of auditory filter shapes
from notched-noise data,’’ Hearing Res., vol. 47, nos. 1–2, pp. 103–138,
Aug. 1990.
[20] R. Drullman, ‘‘Temporal envelope and fine structure cues for speech
intelligibility,’’ J. Acoust. Soc. Amer., vol. 95, no. 5, p. 3009, May 1994.
[21] Q. Mao, M. Dong, Z. Huang, and Y. Zhan, ‘‘Learning salient features forspeech emotion recognition using convolutional neural networks,’’ IEEE
Trans. Multimedia, vol. 16, no. 8, pp. 2203–2213, Dec. 2014.
[22] S. Zhang, S. Zhang, T. Huang, and W. Gao, ‘‘Speech emotion recognition
using deep convolutional neural network and discriminant temporal pyramid matching,’’ IEEE Trans. Multimedia, vol. 20, no. 6, pp. 1576–1590,
Jun. 2018.
VOLUME 8, 2020

Z. Peng et al.: Speech Emotion Recognition Using 3D Convolutions and Attention-Based Sliding Recurrent Networks[23] W. Lim, D. Jang, and T. Lee, ‘‘Speech emotion recognition using convolutional and recurrent neural networks,’’ in Proc. Asia–Pacific Signal Inf.
Process. Assoc. Annu. Summit Conf. (APSIPA), Dec. 2016, pp. 1–4.
[24] G. Keren and B. Schuller, ‘‘Convolutional RNN: An enhanced model for
extracting features from sequential data,’’ in Proc. Int. Joint Conf. NeuralNetw. (IJCNN), Jul. 2016, pp. 3412–3419.
[25] M. Chen, X. He, J. Yang, and H. Zhang, ‘‘3-D convolutional recurrent
neural networks with attention model for speech emotion recognition,’’IEEE Signal Process. Lett., vol. 25, no. 10, pp. 1440–1444, Oct. 2018.
[26] J. Kim, K. P. Truong, G. Englebienne, and V. Evers, ‘‘Learning spectrotemporal features with 3D CNNs for speech emotion recognition,’’ inProc. 7th Int. Conf. Affect. Comput. Intell. Interact. (ACII), Oct. 2017,
pp. 383–388.
[27] G. Zhu, L. Zhang, P. Shen, and J. Song, ‘‘Multimodal gesture recognition
using 3-D convolution and convolutional LSTM,’’ IEEE Access, vol. 5,
pp. 4517–4524, 2017.
[28] C. Stevens and D. Bavelier, ‘‘The role of selective attention on academic
foundations: A cognitive neuroscience perspective,’’ Develop. Cognit.
Neurosci., vol. 2, pp. S30–S48, Feb. 2012.
[29] S. Mirsamadi, E. Barsoum, and C. Zhang, ‘‘Automatic speech emotion
recognition using recurrent neural networks with local attention,’’ in Proc.
IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), Mar. 2017,
pp. 2227–2231.
[30] C.-W. Huang and S. S. Narayanan, ‘‘Attention assisted discovery of subutterance structure in speech emotion recognition,’’ in Proc. Interspeech,
Aug. 2016, pp. 1387–1391.
[31] Z.-Q. Wang and I. Tashev, ‘‘Learning utterance-level representations forspeech emotion and age/gender recognition using deep neural networks,’’in Proc. IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP),
Mar. 2017, pp. 5150–5154.
[32] J. Lee and I. Tashev, ‘‘High-level feature representation using recurrent
neural network for speech emotion recognition,’’ in Proc. 16th Annu. Conf.
Int. Speech Commun. Assoc. (INTERSPEECH), 2015, pp. 1537–1540.
[33] M. Neumann and N. T. Vu, ‘‘Attentive convolutional neural network based
speech emotion recognition: A study on the impact of input features,
signal length, and acted speech,’’ in Proc. Interspeech, no. 3, Aug. 2017,
pp. 1263–1267.
[34] Y. Li, T. Zhao, and T. Kawahara, ‘‘Improved end-to-end speech emotion
recognition using self attention mechanism and multitask learning,’’ inProc. Interspeech, Sep. 2019, pp. 2803–2807.
[35] L. Tarantino, P. N. Garner, and A. Lazaridis, ‘‘Self-attention for speechemotion recognition,’’ in Proc. Interspeech, Sep. 2019, pp. 2578–2582.
[36] B. Schuller, S. Steidl, and A. Batliner, ‘‘The interspeech 2009 emotion challenge,’’ in Proc. 10th Annu. Conf. Int. Speech Commun. Assoc.
(INTERSPEECH), 2009.
[37] B. Schuller, ‘‘The Interspeech 2013 computational paralinguistics challenge: Social signals, conflict, emotion, autism,’’ in Proc. 14th Annu. Conf.
Int. Speech Commun. Assoc. (Interspeech), Aug. 2013, pp. 1–5.
[38] F. Eyben, M. Wöllmer, and B. Schuller, ‘‘Opensmile: The munich versatileand fast open-source audio feature extractor,’’ in Proc. 18th ACM Int. Conf.
Multimedia, 2010, pp. 1459–1462.
[39] C.-C. Chang and C.-J. Lin, ‘‘LIBSVM: A library for support vector
machines,’’ ACM Trans. Intell. Syst. Technol., vol. 2, no. 3, p. 27, 2011.
[40] M. Hall, E. Frank, G. Holmes, B. Pfahringer, P. Reutemann, andI. H. Witten, ‘‘The WEKA data mining software: An update,’’ ACM
SIGKDD Explor. Newslett., vol. 11, no. 1, pp. 10–18, 2009.
[41] M. Neumann and N. T. Vu, ‘‘Improving speech emotion recognition
with unsupervised representation learning on unlabeled speech,’’ in Proc.
IEEE Int. Conf. Acoust., Speech Signal Process. (ICASSP), May 2019,
pp. 7390–7394.
[42] A. J. Kell, D. L. Yamins, E. N. Shook, S. V. Norman-Haignere, andJ. H. Mcdermott, ‘‘A task-optimized neural network replicates human
auditory behavior, predicts brain responses, and reveals a cortical processing hierarchy,’’ Neuron, vol. 98, no. 3, pp. 630.e16–644.e16, May 2018.
[43] S. Latif, R. Rana, S. Khalifa, R. Jurdak, and J. Epps, ‘‘Direct modellingof speech emotion from raw speech,’’ 2019, arXiv:1904.03833. [Online].
Available: https://arxiv.org/abs/1904.03833[44] Z. Aldeneh and E. M. Provost, ‘‘Using regional saliency for speech emotion
recognition,’’ in Proc. IEEE Int. Conf. Acoust., Speech Signal Process.
(ICASSP), no. 3, Mar. 2017, pp. 2741–2745.
[45] R. Xia and Y. Liu, ‘‘A multi-task learning framework for emotion recognition using 2D continuous space,’’ IEEE Trans. Affect. Comput., vol. 8,
no. 1, pp. 3–14, Jan. 2017.
VOLUME 8, 2020[46] S. Ghosh, E. Laksana, L.-P. Morency, and S. Scherer, ‘‘Representation
learning for speech emotion recognition,’’ in Proc. Interspeech, 2016,
pp. 3603–3607.
[47] Z. Zhao, Y. Zheng, Z. Zhang, H. Wang, Y. Zhao, and C. Li,‘‘Exploring spatio-temporal representations by integrating attention-based
bidirectional-LSTM-RNNs and FCNs for speech emotion recognition,’’ inProc. Interspeech, Aug. 2018, pp. 272–276.
[48] T. N. Sainath, B. Kingsbury, A.-R. Mohamed, and B. Ramabhadran,‘‘Learning filter banks within a deep neural network framework,’’ inProc. IEEE Workshop Autom. Speech Recognit. Understand., Dec. 2013,
pp. 297–302.

ZHICHAO PENG received the B.S. degree incomputer science from Hunan Normal University,
China, in 2000, and the M.S. degree in signal andinformation processing from Central South University, China, in 2007. He is currently pursuing
the joint Ph.D. degree with the Japan Advanced
Institute of Science and Technology, Japan, andTianjin University, China. His research interests
include emotion recognition, deep learning, auditory signal processing, and speech and language
processing.
XINGFENG LI received the B.E. degree in software engineering from the Changchun University of Science and Technology, China, in 2013,
and the M.S. degrees in software engineeringand information science from Tianjin University,
China, and Japan Advanced Institute of Science and Technology (JAIST), Japan, in 2016.
He started his research as a member at the Acoustic Information Science (AIS) Laboratory, JAIST,in 2014. His research interests are in affective
computing, speech processing, and speech perception with an emphasis
on how para/non-linguistic information (speech emotion) impacts spoken
communication.
ZHI ZHU received the B.E. degree in communication engineering from the Nanjing Universityof Posts and Telecommunications, in 2012, andthe M.S. and Ph.D. degrees in information sciencefrom the Japan Advanced Institute of Science andTechnology, in 2015 and 2018, respectively. He iscurrently a Scientist with Fairy Devices Inc. He isinterested in hearing and speech science.

MASASHI UNOKI received the M.S. and Ph.D.
degrees in information science from the Japan
Advanced Institute of Science and Technology(JAIST), in 1996 and 1999, respectively. His main
research interests are in auditory motivated signal
processing and the modeling of auditory systems.
He was a Japan Society for the Promotion of
Science (JSPS) Research Fellow, from 1998 to
2001. He was a Visiting Researcher with the
ATR Human Information Processing Laboratories,
from 1999 to 2000 and the Centre for the Neural Basis of Hearing (CNBH),
Department of Physiology, University of Cambridge, from 2000 to 2001.
He has been on the faculty of the School of Information Science, JAIST,
since 2001, where he is currently a Full Professor. He is a member of the
Research Institute of Signal Processing (RISP), the Institute of Electronics,
Information and Communication Engineers (IEICE) of Japan, the AcousticalSociety of America (ASA), the Acoustical Society of Japan (ASJ), and theInternational Speech Communication Association (ISCA). He received theSato Prize from ASJ, in 1999, 2010, and 2013 for an Outstanding Paperand the Yamashita Taro ‘‘Young Researcher’’ Prize from the Yamashita Taro
Research Foundation, in 2005.
16571

Z. Peng et al.: Speech Emotion Recognition Using 3D Convolutions and Attention-Based Sliding Recurrent Networks

JIANWU DANG received the B.E. and M.E.
degrees from Tsinghua University, China,
in 1982 and 1984, respectively, and the Ph.D.
degree from Shizuoka University, Japan, in 1992.
He was a Lecturer with Tianjin University, Tianjin,
China, from 1984 to 1988. From 1992 to 2001,
he worked at ATR Human Information Processing
Laboratories, Japan. Since 2001, he has been on
the faculty of the School of Information Scienceof JAIST as a Professor. He joined the Centerof National Research Scientific, Institute of Communication Parlee (ICP),
France, as a Research Scientist the first class, from 2002 to 2003. Since 2009,he has been with Tianjin University. He built a 3D physiological model for
speech and swallowing, and endeavors to apply the model on clinics. His
research interests include speech production, speech synthesis, and speechcognition.

16572

MASATO AKAGI received the B.E. degree fromthe Nagoya Institute of Technology, in 1979, andthe M.E. and Ph.D. (Eng.) degrees from the Tokyo
Institute of Technology, in 1981 and 1984, respectively. He joined the Electrical Communication
Laboratories, Nippon Telegraph and Telephone
Corporation (NTT), in 1984. From 1986 to 1990,
he worked at the ATR Auditory and Visual Perception Research Laboratories. Since 1992, he has
been on the faculty of the School of Information
Science, JAIST, where he is currently a Full Professor. His research interests
include speech perception, modeling of speech perception mechanisms inhumans, and the signal processing of speech.

VOLUME 8, 2020