## 4 Experimental Results and Analysis
### Base Classifiers and MFELCM
#### 4.2.4 XGBoost Experimental Results  
The maximum depth of trees in XGBoost is 20 and the learning rate is 0.03. The python toolkit named tsfresh can automatically generate a large number of features from time series, but it requires a lot of computing resources. In addition, many features are useless for classification, and thus it is necessary to filter the features outputted by tsfresh, which is done as follows. 1. Step 1. Extract features without filtering using tsfresh on a small dataset M and create the time-series dataset of M (denoted as \(TS(M)\)), and then use \(TS(M)\) to train an XGBoost (denoted as XGBoost1). 2. Step 2. Use XGBoost1 to output the top n important features on dataset M.
3. Step 3. Use tsfresh to extract the n features obtained in Step 2 on the dynamic dataset (Table 5), and create the time-series dataset (denoted as \(TS\)), then use \(TS\) to train another XGBoost (denoted as XGBoost2). 4. Step 4. If \(n\) is not zero, return to Step 2 and decrease \(n\) with a certain interval. 5. Step 5. When \(n\) can no longer decrease, choose the XGBoost2 with the best performance obtained in Step 3. The features used by this model are the most important feature that is generated by tsfresh. We select 50,000 dynamic messages from the four types of ships, and each ship should have more than 500 pieces of messages. After removing those sub-trajectories which contain less than 10 pieces of messages, there are 106,169 pieces of dynamic messages left, which is the dataset \(M\). Table 11 illustrates the amount of \(M\), whose distribution is shown in Figure 25. The XGBoost1 is trained on \(TS(M)\) with 3765 features extracted from M by tsfresh. Figure 26 shows the importance of the features outputted by XGBoost1. Table 12 shows the performance of XGBoost2 which trains on \(TS\) with a different number of features. The XGBoost2 performs best when the top 40 important features are considered. The features' names and weights are shown in Table A2, and the detailed definition of these features can be obtained from [33]. Figure 23: Learning curve (Bi-GRU). Figure 24: (**a**) Confusion matrix of sub-trajectories (Bi-GRU); (**b**) Confusion matrix of ships (Bi-GRU). When using 40 features to complete the classification of ships, the learning curve of XGBoost2 and the confusion matrix on the testing set are as shown in Figure 27. In Figure 27b, some sub-trajectories of passenger ships are misclassified into the other three categories. The classifier tends to confuse tankers with cargo ships, and tankers are more likely to be mislabeled as cargo ships. In Figure 27c, the confusion of tankers and cargo ships remains significant. Figure 28 shows the importance score of these 40 features, whose names and values of importance score are shown in Table A2. The features are mostly \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Type** & **Number of Messages** & **Number of Ships** & **Number of Sub-Trajectories** \\ \hline Passenger Ships & 28,513 & 135 & 1197 \\ Cargo Ships & 27,984 & 249 & 1494 \\ Tankers & 28,818 & 289 & 1493 \\ Fishing boats & 20,854 & 104 & 1084 \\ Total & 106,169 & 777 & 5268 \\ \hline \hline \end{tabular}
\end{table} 
Table 11: The amount of data in dataset \(M\). Figure 26: Features and their importance score on dataset \(M\). Figure 25: Ships distribution of dataset \(M\). In addition, we carry out an experiment to illustrate the necessity of splitting the dataset by MMSL. The XGBoost trained on the time-series feature dataset \(TSF\) (\(TSF\) is split by \(TSF_{n}^{i_{m}}\) instead of \(TSF^{i=}\)) is denoted as XGBoost3. Under the same parameters with XGBoost2, the learning curve and the confusion matrixes XGBoost3 are shown in Figure 29. The F1 score and total accuracy of XGBoost3 evaluated on \(TSF^{i=}\) are 0.8180 and 0.8186, respectively, and those evaluated on ships are 0.7996 and 0.7994.