# A Framework for the Detection of Search and Rescue Patterns Using Shapelet Classification
## 4 Experimental Evaluation & Results
### Classification on Real Data  
After the models have been trained on the synthetic data they can be used for identifying SAR patterns in real vessel trajectories. In our experiments we employed the trajectory data from 5 vessels, for the period between 1 July 2015 and 29 July 2015 and focused on the special case of the vessel SIEM PILOT (this vessel is a cargo vessel that has been hired by European NGOs to patrol in the Mediterranean sea and rescue immigrants who are in danger), which in our previous study, was found to exhibit a clear creeping line pattern at its trajectory. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Scenario** & **Class A** & **Class B** & **Average Accuracy** \\ \hline
9 & Spiral & Random & 99.38 \(\pm\) 0.25 \\
10 & Expanding square & Random & 97.38 \(\pm\) 0.62 \\
11 & Creeping line & Random & 91.62 \(\pm\) 2.08 \\
12 & Sector & Random & 92.25 \(\pm\) 1.2 \\
13 & Spiral & Steps & 99.63 \(\pm\) 0.2 \\
14 & Expanding square & Steps & 99.38 \(\pm\) 0.26 \\
15 & Creeping line & Steps & 96.5 \(\pm\) 0.69 \\
16 & Sector & Steps & 95.5 \(\pm\) 0.67 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Results for the binary classification scenarios for complex patterns. The average accuracy scores are reported in the 95% confidence interval. \begin{table}
\begin{tabular}{c c c} \hline \hline  & **Scenarios 9–12** \\ \hline
**Accuracy Level** & **Number of Experiments** & **Percentage of Experiments** \\ \hline Bad & 15 & 3.75\% \\ Acceptable & 34 & 8.5\% \\ Excellent & 220 & 55\% \\ Perfect & 131 & 32.75\% \\ \hline \hline \multicolumn{3}{c}{**Scenarios 13–16**} \\ \hline
**Accuracy Level** & **Number of Experiments** & **Percentage of Experiments** \\ \hline Bad & 1 & 0.25\% \\ Acceptable & 6 & 1.5\% \\ Excellent & 197 & 49.25\% \\ Perfect & 196 & 49\% \\ \hline \hline \end{tabular}
\end{table} 
Table 3: A summary of the classifier accuracy performance for binary classification scenarios on synthetic data. To test our classification approach in real data, we selected 5 cargo vessels (BOURBON ARGOS-BA, DIGNITY I-DGN, PHOENIX-PHO, SIEM PILOT-SP, VOS HESTIA-VH) that have been reported in the news as vessels that took part in SAR missions. The trajectories of these vessels had a size between 10,000 to 22,000 samples, which is too large even for the Scale-down algorithm to fit to the required time series length (of 25). So we decided to segment the trajectories to chunks of 500 points, which are considered enough for a SAR maneuver to occur (since for every vessel we had one point every 3 min, 500 points correspond to an approximate duration of the trajectory of one day). After the creation of each chunk, we evaluated the different pre-processing strategies to scale them down to time series of length 25 and classified them using all the binary classifiers defined in the previous section. As depicted in Figure 5 when more than one pattern have been matched with probability higher than 0.75 the pattern with the higher probability has been selected. Figure 6 depicts an example from a trajectory of the SIEM PILOT vessel, which has been pre-processed with the four different methods. The plot on the left (Figure 5(a)) shows how the Scale-down and RDP methods compare to the original trajectory (of 500 points length), whereas the plot on the right (Figure 5(b)) shows the result from the application of the trajectory segmentation method to small chunks (of 180 points length). The RDP & scale down method had the same result as the Scale down method and thus has been omitted from the plot. In the first case the classification method detected the creeping line pattern, whereas in the second case a steps pattern has been detected in the smaller chunk. It is worth mentioning that all other chunks have not been classified as any SAR pattern. The classifier we applied in all experiments was the logistic regression classifier trained on the synthetic data.