# Time Series Classification with Shapelet and Canonical Features
## 5 Experimental Analysis
### Accuracy Comparison  
The UCR time series dataset [7] is the standard dataset in the area of time series classification study [30], containing a total of 128 datasets. [PERSON] et al. [18] recently conducted a new evaluation of time series classification algorithms using 112 of them (15 of the 16 excluded datasets are not typically used to evaluate algorithm performance due to inconsistent time series lengths or defect values [7,18], and the other is a \"Fungi\" dataset with only one training instance per category), and gave the accuracy of time series classification algorithms that represent the most advanced level of the current state of affairs.  
In order to fully verify the performance of the RSFCF algorithm proposed in this paper, we used the same 112 datasets and compared them with the seven algorithms that performed the best and are most relevant: gRSF [10], STC [17], CIF [3], WESAEI [25], Proximity Forest [5], HIVE-COTE [21], ResNet [19] (the above algorithms are abbreviated as: gRSF, STC, CIF, WS, PF, HCT, RN). These algorithms belong to the six different types of time series classification methods introduced in Section 2, representing the advanced level of the corresponding types. The experiment follows the original training and test set split of the dataset [7], and the results of the comparison algorithms are taken from the results proposed by [PERSON] et al. [18]. Table 2 shows the accuracy rate of algorithm RSFCF on 112 datasets.  
We use the critical difference plots described in [32] to compare the accuracy of multiple classifiers across multiple datasets. We set the significance level \(\alpha\) to 0.05, and then use the [PERSON] test to determine whether the hypothesis \"there is no difference in the average ranking of accuracy between multiple classifiers on multiple datasets\" is true, and if the hypothesis is not true, we then use the Nemenyi test to group the classifiers (the classifiers connected to the same line in Figure 7 form a group), and there is a significant difference in the average accuracy ranking of any two classifiers in different groups.  
Figure 7: Average ranks and critical differences on accuracy of RSFCF and 7 comparison algorithms over 112 UCR datasets.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & **RSFCF (8 Features)** & **Catch22 (22 Features)** & **Meansdslope (3 Features)** \\ \hline
**Accuracy** & 84.52 & 84.50 & 82.52 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Ablation experiment result.  
The critical difference plot shown in Figure 7 shows that the average ranking of RSFCF's accuracy rates outperforms gRSF, STC, CIF, and WEASEL, and significantly exceeds Proximity Forest. It is worth mentioning that according to the recent evaluation of [PERSON] et al. [18], the above four algorithms represent the current most advanced level of time series classification methods based on Shapelet, interval, dictionary, and distance. Compared with deep learning methods, RSFCF surpasses the powerful baseline algorithm ResNet. The meta-ensemble method HIVE-COTE obtained the highest average accuracy ranking (2.9063), but there was no significant difference from the algorithm RSFCF in this paper, and the efficiency comparison in the next section verified that the efficiency of RSFCF was much higher than that of HIVE-COTE.  
Table 4 shows the results of the two-by-two comparison of RSFCF and seven algorithms. Although HIVE-COTE ranks higher on average accuracy than RSFCF, it can be seen from Table 4 that RSFCF still surpassed HIVE-COTE on 36 datasets. Compared to the remaining six algorithms, RSFCF wins on near to or even more than half of the datasets, while the other algorithms only win on about one-third of the datasets, which highlights the accuracy advantage of RSFCF.