# Time-Series Classification Based on Fusion Features of Sequence and Visualization
## 3 Model Design
### Lstm-A  
An RNN is a type of neural network used to process sequence data. There are many variants of RNNs, some representative ones including LSTM, Gated Recurrent Unit (GRU) [44], Bi-directional RNN [45], and so on. [PERSON] et al. [46] compared popular RNN variants, showing that the RNN variants have almost the same performance and that LSTM is superior to other simplified LSTMs (such as GRU). In [47], the authors tested more than 10,000 RNN structures and found that, in certain tasks or situations, some RNN variants work better than LSTM; but only in special cases. Comparing GRU and LSTM, on the one hand, GRU has fewer parameters, so its training is faster and requires less data to generalize. On the other hand, if given enough training data, the great expressive power of LSTM may produce better results than GRU. In addition, in the field of TSC research, many studies have chosen LSTM [39; 40; 41] to do TSC and achieved promising results. So, we chose LSTM to learn the temporal dependencies of TS data. However, the temporal dependencies of long input sequences cannot be reasonably learned and, so, we added attention mechanisms to learn these long-term dependencies [48]. Finally, we used the LSTM-A model (Figure 2) to learn sequence features.  
Figure 1: Overview of the framework. \(AUX_{1}\) and \(AUX_{2}\) are loss functions in the pre-training stage.  
Given a time-series data set \(D\), the LSTM-A model processes the records as follows:  
\[z = tanh(\overbrace{x^{t},h^{t-1}}^{t}), \tag{2a}\] \[z^{f} = sigmoid(\overbrace{x^{t},h^{t-1}}^{t}),\] (2b) \[z^{i} = sigmoid(\overbrace{x^{t},h^{t-1}}^{t}),\] (2c) \[z^{o} = sigmoid(\overbrace{x^{t},h^{t-1}}^{t}),\] (2d) \[c^{t} = c^{t-1}\odot z^{f}\odot z^{i}\odot z,\] (2e) \[h^{t} = z^{o}\odot tanh\left(c^{t}\right). \tag{2f}\]  
where \(h^{t-1}\) and \(c^{t-1}\) are the outputs of the last LSTM cell; \(x^{t}\) is the current input; stands for the splicing operation; \(z^{i}\), \(z^{f}\), and \(z^{o}\) are the input, forget, and output gates obtained by different parameters, respectively; \(\odot\) and \(\oplus\) represent matrix multiplication and matrix addition, respectively; and \(h^{t}\) and \(c^{t}\) are the outputs of the current LSTM cell.  
The T-Attention block is used to enhance the performance of sequence feature learning for very long input records. Its calculation formula is as follows:  
\[M^{t} = tanh\left(w^{h}h^{t}\right), \tag{3a}\] \[a^{t} = softmax\left(w^{t}M^{t}\right),\] (3b) \[F^{s} = FC\left(a^{t}\otimes h^{t}\right),\] (3c) \[P^{s} = softmax\left(F^{s}\right),\] (3d) \[AUX_{1} = H\left(P^{s},Y\right)=-\sum_{i}y_{i}log\left(p_{i}^{s}\right). \tag{3e}\]  
where \(h^{t}\) is the output of the \(t^{\text{th}}\) hidden unit of the last LSTM layer, \(a^{t}\) is the \(t^{\text{th}}\) attention weight, \(w^{h}\) and \(w^{t}\) are weighted matrices, \(a^{t}\) and \(h^{t}\) are merged, and \(\otimes\) represents the matrix merge operation. Then, we use a Fully-connected layer to transform it to obtain sequence features \(F^{s}\). To prevent overfitting and gradients from disappearing, we use dropout and Batch Normalization continuously after the Fully-connected layer. Finally, a softmax layer is used to obtain the classification results, where \(P^{s}\) is the output prediction probability sequence of LSTM-A. In the pre-training phase of LSTM-A, we use cross-entropy as the loss function \(AUX_{1}\), \(Y\) is the true probability sequence, and \(p_{i}^{s}\) and \(y_{i}\) are the predicted probability and true probability of the record belonging to category \(i\), respectively.