# Interval Feature Transformation for Time Series Classification Using Perceptually Important Points
## 4 Experimental Studies
### Evaluating Indicator  
To the classification problem, classification accuracy is the most important criterion to evaluate algorithm performance. In addition to accuracy, the Friedman test and Nemenyi test are widely used in machine learning to evaluate the performance of algorithms over multiple data sets. In order to consider the efficiency of the IFT method, we establish the multi-classifier comparison procedure over multiple data sets suggested by [PERSON] [36]. The Friedman test [37] is followed by the Nemenyi test [38] if the Friedman test shows a significant difference between the classifiers. After getting the performance of the \(K\) algorithms on the \(N\) data set, the Friedman test ranks algorithms for each data set separately. When multiple algorithms have the same result for a data set, \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \# & **Dataset** & **Type** & **Train** & **Test** & **Length** & **Class** & \(\mathbf{\omega}\) & \(\mathbf{\tau}\) & \(\mathbf{k}\) \\ \hline
1 & BirdChicken & Image & 20 & 20 & 512 & 2 & 25 & 5 & 256 \\
2 & FreezerRegularTrain & Sensor & 150 & 2850 & 301 & 2 & 10 & 2 & 150 \\
3 & ShapeletSim & Simulated & 20 & 180 & 500 & 2 & 25 & 5 & 250 \\
4 & ToeSegmentation1 & Motion & 40 & 228 & 277 & 2 & 16 & 5 & 138 \\
5 & Worms & Motion & 181 & 77 & 900 & 5 & 10 & 1 & 450 \\
6 & Rock & Spectrum & 20 & 50 & 2844 & 4 & 5 & 1 & 100 \\
7 & Meat & Spectro & 60 & 60 & 448 & 3 & 20 & 2 & 224 \\
8 & Beef & Spectro & 30 & 30 & 470 & 5 & 40 & 5 & 235 \\
9 & InlineSkate & Motion & 100 & 550 & 1882 & 7 & 8 & 1 & 400 \\
10 & Coffee & Spectro & 28 & 28 & 286 & 2 & 15 & 5 & 143 \\
11 & DodgerLoopGame & Sensor & 20 & 138 & 288 & 2 & 20 & 2 & 144 \\
12 & DodgerLoopWeekend & Sensor & 20 & 138 & 288 & 2 & 20 & 2 & 144 \\
13 & ECGFiveDays & ECG & 23 & 861 & 136 & 2 & 14 & 5 & 68 \\
14 & Ham & Spectro & 109 & 105 & 431 & 2 & 20 & 10 & 215 \\
15 & Herring & Image & 64 & 64 & 512 & 2 & 20 & 10 & 256 \\
16 & PowerCons & Power & 180 & 180 & 144 & 2 & 24 & 2 & 72 \\
17 & Wine & Spectro & 57 & 54 & 234 & 2 & 12 & 5 & 117 \\
18 & Yoga & Image & 300 & 3000 & 426 & 2 & 5 & 2 & 213 \\
19 & FaceFour & Image & 24 & 88 & 350 & 4 & 20 & 10 & 175 \\
20 & OliveOil & Spectro & 30 & 30 & 570 & 4 & 7 & 50 & 200 \\
21 & Fish & Image & 175 & 175 & 463 & 7 & 6 & 1 & 231 \\
22 & Plane & Sensor & 105 & 105 & 144 & 7 & 10 & 1 & 72 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summary of data sets and the combination of parameters in the experiment. the average rank is used. In this way, we can get a rank matrix of \(N\times K\). \(r_{ij}\) is the rank mark of the \(i\)th data set on the \(j\)th algorithm, and the average ranges \(R_{j}\) are calculated as shown in Equation (13). \[R_{j}=\frac{1}{N}\sum_{j=1}^{N}r_{ij} \tag{13}\]  
Under the null hypothesis, all algorithms are equivalent, so their \(R_{j}\) should be equal. The [PERSON] statistics are defined as Equation (14),  
\[\chi_{F}^{2}=\frac{12N}{K(K+1)}\left[\sum_{j=1}^{K}R_{j}^{2}-\frac{K(K+1)^{2}} {4}\right] \tag{14}\]  
which is according to \(\chi_{F}^{2}\) with \(K-1\) degrees of freedom. The research of [PERSON] et al. [36] shows that [PERSON]'s statistics are too conservative, and proposed a better statistical formula such as Equation (15),  
\[F_{F}=\frac{(N-1)\chi_{F}^{2}}{N(K-1)-\chi_{F}^{2}} \tag{15}\]  
which is according to the F-distribution with \(K-1\) and \((K-1)(N-1)\) degrees of freedom. If the null hypothesis is rejected, indicating significant differences between these algorithms, the difference between the algorithms can be tested by the Nemenyi test to compare all the algorithms to each other. At a significance level of \(\alpha\), the critical difference (\(CD\)) value is defined as Equation (16). \[CD=q_{\alpha}\sqrt{\frac{K(K+1)}{6N}} \tag{16}\]  
All algorithms were divided into different groups by \(CD\) value, so that there was no significant difference in the performance of the algorithms in the group.