## 5 Surrounding-Aware Individual Emotion Prediction Model
### T-LSTM Network  
LSTM [42] was designed to capture long- and short-term dependencies while overcoming the vanishing gradient problem. However, a standard LSTM network implicitly assumes that the time intervals between the elements of an input sequence are the same, which means that LSTM has difficulty in handling irregular time series data with missing values.  
In recent years, to use LSTM network to capture information in irregular series data, more innovative solutions have been proposed [43]. One method involves imputing data to make the time intervals between the elements of the input sequence to be regular. Based on this idea, many studies have estimated missing data by treating imputed values as trainable variables. However, the limitation of this method lies in the fact that the estimated missing data are evidently different from reality.  
Another method to solve the time irregularity is to adjust the memory unit of LSTM by using the short-term memory discount. This method of using time gaps to adjust memory is called T-LSTM [44]. The T-LSTM architecture is presented in Figure 6. Blue boxes denote networks, and green circles indicate point-wise operators. T-LSTM takes time series and time intervals as input. The major component of the T-LSTM architecture is the subspace decomposition applied to the memory of the previous time stamp, which decomposes the previous memory into long-term and short-term components, and uses the time interval (\(\Delta_{t}\)) to discount the short-term influences.  
Figure 5: Basic process of SAEP model.  
First, using the memory of the previous moment, the short-term memory is obtained through the network. Note that this decomposition operation is data-driven, and its parameters are learned at the same time as the rest of the network parameters by back-propagation. After the short-term memory \(c_{t}^{s}\) is computed, considering that the effect of short-term memory is also related to the time gap, a time decay function \(g(\Delta t)\) is used to calculate the weight and adjust short-term memory to obtain the discounted short-term memory \(\bar{c}_{t}^{s}\). Finally, to compose the adjusted previous memory back \(c_{t-1}^{*}\), the complement subspace of the long-term memory \(c_{t-1}^{L}\) is combined with the discounted short-term memory. The detailed mathematical expressions of the T-LSTM network in Figure 6 are given by Equations (3)-(8) [44].  
\[c_{t-1}^{s}=tanh(W_{d}c_{t-1}+b_{d}) \tag{3}\]  
\[\bar{c}_{t}^{s}=c_{t-1}^{s}\cdot g(\Delta t) \tag{4}\]  
\[c_{t-1}^{L}=c_{t-1}+c_{t-1}^{s} \tag{5}\]  
\[c_{t-1}^{*}=c_{t-1}^{L}+\bar{c}_{t-1}^{s} \tag{6}\]  
\[\bar{c}=tanh(W_{c}x_{t}+U_{c}h_{t-1}+b_{c}) \tag{7}\]  
\[c_{t}=f_{t}\cdot c_{t-1}^{*}+i_{t}\cdot\bar{c} \tag{8}\]  
where \(h_{t-1}\) is the previous hidden state. \(c_{t}\) and \(c_{t-1}\) represent the current and previous cell states, respectively. \(\Delta t\) is the time interval, and \(g(\cdot)\) is a heuristic decay function, so the larger the value of \(\Delta t\), the less the effect of short-term memory. Different types of \(g(\cdot)\) can be chosen according to the application requirements. For example, \(g(\Delta t)=1/\Delta t\) is preferred for less time-consuming datasets, and \(g(\Delta t)=1/log(\Delta t+e)\) can be selected for more time-consuming datasets. After the experimental analysis, a better performance of \(g(\Delta t)=1/\Delta t\) is used in the proposed SAEP model.