# Time-Series Classification Based on Fusion Features of Sequence and Visualization
## 4 Experiment
### Integrated Model Evaluation  
Among the existing four types of methods, we used 15 methods for comparison, among which we reproduced 1 NN_DTW, ST and FCN, supplementing the results on the 27 data sets of the additional data sets in UCR, and the results of the remaining methods come from [15]. The chosen methods were as follows:  
* Distance-based methods: 1 NN_DTW [2] and PF [17];
* Feature-based methods: BOSS [20], ST [18; 19], STC [24], WEASEL [23], TSF [21] and RISE [25];
* Ensemble methods: HIVE-COTE [4], TS-CHIEF [34], ROCKET [35] and Catch22 [36];
* Deep learning methods: ResNet [38], FCN [38] and InceptionTime [42]. For the detailed structure and parameters of each method, please refer to their respective references. All results were obtained by taking the average value from multiple experiments. All models were evaluated using classification accuracy and mean-per-class-error (MPCE), which is defined as the average error of each class for all data sets and mathematically represented in Equation (5). The Average Arithmetic Rank (AVG Rank) is the mean of the classification accuracy ranking over the 112 data sets. Table 2 lists the comparison results. In order to facilitate the display, the names of some methods have been abbreviated again, including 1 NN_DTW (labelled DTW), WEASEL (WS), HIVE-COTE (HCT), TS-CHIEF (CHI), ROCKET (RK), Catch22 (C2), ResNet (RN), InceptionTime (IcT). \[PCE_{k} =\frac{1-accuracy}{number\ of\ classes}, \tag{5a}\] \[MPCE =\frac{1}{N}\sum_{k=1}^{N}PCE_{k}. \tag{5b}\]  
Compared with the baseline methods on 112 data sets in UCR, the proposed mthod TSC-FF won or tied on 20 data sets. Judging from the AVG Rank and MPSE, its overall ranking was fifth, after HIVE-COTE, TS-CHIEF, ROCKET and InceptionTime. By comparison, we found that TSC-FF was far better than six feature-based methods. Therefore, even though the deep learning-based method TSC-FF proposed in this paper did not achieve the best performance, it showed the greatest potential of the deep learning methods considered for feature learning, proving that DNN can automatically complete feature extraction, not requiring complicated feature engineering. With the development of DNN and the accumulation of data, deep learning methods can even achieve better performance. Although TSC-FF could not defeat the deep learning method InceptionTime, which achieved the lowest MPSE as a whole, by comparing the results of FordA, FordB, and Wafer, we found that our framework achieved higher accuracy on these data sets. This shows that, when a DNN is given enough data to learn features, using pre-trained sub-models to learn features from different aspects and then fusing features can improve DNN performance. Of course, the amount of data and the number of categories also has an effect on DNN performance, which will be explored in later experiments. On the data set ElectricDevices, the featured-based method BOSS had the highest accuracy. The reason for this is the artificially constructed features used by BOSS are better than those learned by the DNN. After all, there still exists a certain gap between DNNs and the human brain. On some data sets, the features learned by DNN are not as effective as those designed by humans. However, the results of BOSS on other data sets proved the weak generality of artificially constructed features. Among all methods, HIVE-COTE achieved the best performance. Comparing the two distance-based methods, the accuracy of PF is much improved compared to DTW. The difference between the two is that PF uses multiple distance functions. Among the feature-based methods, STC achieved the best performance. What makes STC special is that it uses some innovative methods to find more effective shapelet features. In the deep learning methods, the best performance is achieved by InceptionTime. InceptionTime combines several DNNs with the same structure but different parameters, and it can be considered that multiple DNNs learn different features, thus InceptionTime improved the performance compared to other deep learning methods.