# A Machine Learning Approach to Determine Oyster Vessel Behavior
## 5 Experimental Results  
In this section, we present the experimental results obtained through validation and independent test of the proposed method. For the validation of the model, a widely used 10-fold CV technique is used. Following validation, the model is tested using an independent test dataset. The accuracy measures used to evaluate the performance of the proposed classifier are accuracy (_ACC_), precision, recall, \(F1\)-score and area under the receiver operating characteristic curve (_AUC_). The following equations define the _ACC_, precision, recall and \(F1\)-score, respectively:  
\[ACC=\frac{TP+TN}{TP+TN+FP+FN} \tag{3}\]  
\[Precision=\frac{TP}{TP+FP} \tag{4}\]  
\[Recall=\frac{TP}{TP+FN} \tag{5}\]  
\[F1-score=2.\frac{Precision\cdot Recall}{Precision+Recall} \tag{6}\]  
where, _TP_, _TN_, _FP_ and _FN_ are the numbers of true positive, true negative, false positive and false negative, respectively. Here, precision is a measure to identify a predictor's exactness and recall measures a predictor's completeness. Therefore, the harmonic mean of precision and recall, called \(F1\)-score, measures a classifier's overall correctness. These are the main performance measures used in the literature as they capture the two error types (_FP_ and _FN_), the number of correctly classified instances (_TP_ and _TN_) and the relationship between them. The objective of this work is to correctly categorize the oyster vessel trajectory data points into one of the four behavior types: fishing, traveling, poling and docked. At first, we optimized the parameters C and \(\gamma\) of RBF kernel SVM using grid search. As grid search is a time consuming procedure, we collected a subset of the original dataset. The subset is created by considering paths with only over 50 points from the original dataset. These paths in a subset resulted in total of 157,088 points (3528 docked points, 14,638 fishing points, 136,295 traveling points and 2627 poling points). To perform validation and independent test of the proposed method, the subset is further divided in the validation set of 80% and the independent test set of 20%. The validation set consist of 125,670 points (2822 docked points, 11,710 fishing points, 109,036 traveling points and 2102 poling points) and the independent test set consist of 31,418 points (706 docked points, 2928 fishing points, 27,259 traveling points and 525 poling points). Initially, the validation set is utilized to perform a grid search with 3-fold CV using SVM with RBF kernel as a classifier to find the optimal value of parameters C and \(\gamma\). From the grid search, we found \(C=10\) K and \(\gamma=1000\) to be the optimal value of \(C\) and \(\gamma\) parameters. Next, using optimal value of \(C\) and \(\gamma\) in the proposed SVM with RBF kernel based multi-class behavior classification method, we performed 10-fold CV on the validation set (consisting of 125,670 points) as well as on the original dataset (consisting of 612,700 points). We found that both the simulations yielded similar accuracy of 99.99%. The 10-fold CV accuracy is obtained by taking the average of ACC's obtained while testing on individual folds. In addition, we computed the AUC values for each behavior category by performing 10-fold CV on the validation set using SVM with RBF kernel and optimal value of \(C\) and \(\gamma\) found earlier through grid search. The value of AUC close/equal to 1 indicates high test accuracy and a robust classifier, whereas the value close/equal to 0 indicates low test accuracy and a very poor performing classifier. In our implementation, the AUC values for docked, fishing, travel and poling categories are found to be 1.00, 0.999999, 1.00 and 0.999990, respectively. The larger values of AUC's for all the behavior categories clearly indicate that the proposed method achieves high 10-fold CV accuracy and ensures the robustness of the classifier.