## 4 Speech Emotion Recognition
### Emotion Recognition Model  
This study used LSTM model to recognize emotional state from speech. LSTM is a modified RNN structure proposed by [PERSON] et al. (2016), where hidden layer nodes are replaced by blocks. The LSTM block comprises memory cells with cyclic structure; and input, forget, and output gates. They differ from conventional RNN models in that LSTM blocks control flow by sending or not sending information appropriately using their gates  
Figure 4: An example of determining threshold. depending on the specific circumstances. LSTM calculates the \(i\)-th value for each node at time \(t\) as  
\[c_{i}(t)=tanh(r_{i}^{c}(t)),\qquad r_{i}^{c}(t)=\sum_{j}U_{ij}^{c}x_{j}(t)+\sum_ {j}W_{ij}^{c}h_{j}(t-1)+b_{i}^{c}, \tag{3}\]  
where \(x_{j}(t)\) is the j-th input variable, \(c_{i}(t)\) is the \(i\)-th input state, calculated by applying weights \(U\), \(W\), and \(b\) to hidden layer \(h(t\)\(-\)1) at input times \(t\) and \(t\)\(-\)1. Equation (3) is applied to input, forget, and output gates \(g,f\), and \(q\), respectively, and the weighted sum of the constant term is converted into the range of 0 to 1 using a non-linear function. Gate output = 1 means that all information is maintained, whereas gate output = 0 means that all information is deleted. Cell state \(s_{i}(t)\) at \(t\) is calculated from \(c_{i}(t)\) and gates \(g\) and \(f\),  
\[s_{i}(t)=f_{i}(t)s_{i}(t-1)+q_{i}(t)c_{i}(t), \tag{4}\]  
where the \(i\)-th hidden node is  
\[h_{i}(t)=q_{i}\tan\text{h}(s_{i}(t)), \tag{5}\]  
Thus, the first input value cannot influence the next value when gates \(f\) and \(g\) = 1 and 0, respectively, hence the cell state never changes, and short-term memory can be removed; whereas long-term memory can be maintained in the opposite case. Data from previous time(s) influence current data for most time-series models. Therefore, RNN models predict status for current time \(t\) using data from previous times. However, current status can often be predicted more accurately using future data. For example, prediction accuracy may improve for Korean language objects if the verb, which appears after the object, is utilized rather than the subject. Thus, inferring in forward as well as backward directions may produce more meaningful results. Equation (4) shows that conventional LSTM uses data for past time \(t\)-1 time to predict the data for current time \(t\). Backward connection does not exist so the current time cannot be predicted using future time data. Therefore, bi-directional LTSM models are commonly employed to resolve this drawback, as shown in Figure 5.  
This study used the feature vectors discussed in Section 4.1 and a bi-directional LSTM to recognize emotions from speech. Figure 6 shows the bi-directional LTSM model employed. Figure 5: Example of Bi-directional LSTM network nodes. Generally, emotion can be recognized when we hear the entire spoken sentence or phrase. However, it is very difficult to recognize emotion when we hear the speech into short segments (e.g., one second). Thus, emotion recognition accuracy generally improves for longer rather than shorter intervals. However, LSTM has a problem that a gradient is not delivered as the layer is deeper (meaning that speech is recognized at a longer interval). This study set LSTM interval as short as possible, and then calculated emotion probability over each section to decide the final emotion label, hence avoiding the gradient loss problem while still using longer speech intervals to recognize emotion. Input speech data was divided into speech sections through the pre-processing procedure that searches the speech section as mentioned in Section 4.2.1. Combined feature vectors, discussed in Section 4.1, were extracted from the divided speech sections in chronological order, and overlapped as much as 0.3 for approximately 15 ms time units. Extracted feature vectors were re-configured into 78 vectors, emotion for each part was calculated approximately every second chronologically, and each frame was classified using the trained model.