## 2 Materials and Methods
### Algorithm Performance Evaluation  
Some of the metrics used in this study are the ones commonly reported in the literature and applied investigations when evaluating the performance of computer vision systems (8). These are the following:  
_Accuracy:_ Commonly defined as the ratio of true positives and true negatives to all positive and negative observations. That is, how often we can expect the computer vision system to correctly predict an outcome out of the total number of times it made predictions. \begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Image Ratio**} & **Number of Images** & **Number of Images** & **Number of Images** \\  & **of Class 0** & **of Class 1** & **of Class 2** \\ \hline
1/1 & 330 & 330 & 330 \\
1/2 & 660 & 330 & 330 \\
2/5 & 660 & 264 & 264 \\
3/1 & 660 & 196 & 196 \\
1/4 & 660 & 165 & 165 \\
1/8 & 656 & 82 & 82 \\
1/16 & 656 & 41 & 41 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Distribution of images considered in Experiment 2. Mathematically, it is formulated as the ratio of the sum of true positives and true negatives out of all the predictions, namely:  
\[Accuracy=\frac{TP+TN}{(TP+FN+TN+FP)} \tag{1}\]  
where _TP_ = true positives; _TN_ = true negatives; _FN_ = false negatives; and _FP_ = false positives. _Precision_: It represents the proportion of labels that were correctly predicted to be positive. That is, it is a performance metric that is most useful when trying to control false positives. As well as for _Accuracy_, _Precision_ is also affected by class distribution; if there are more images for a class that does not happen frequently, precision becomes lower. Mathematically, it is formulated as the ratio of true positive to the sum of true positives and false positives, namely:  
\[Precision=\frac{TP}{(FP+TP)} \tag{2}\]  
_Recall_: It represents the system's capacity to correctly predict the positives from the set of actual positives. _Recall_ is most useful when identifying positives as critical. Mathematically, it represents the ratio of true positive to the sum of true positives and false negatives. \[Recall=\frac{TP}{(FN+TP)} \tag{3}\]  
_F1 score_: It is obtained as a harmonic mean of the _Precision_ and _Recall_ scores, giving each of them an equal weight. It is often used as a single value that provides high-level information about the model's output quality and _Precision_/_Recall_ balance. Mathematically, it is formulated as a harmonic mean of the _Precision_ and _Recall_ scores. \[F1\ Score=\frac{2*\ Precision*\ Recall}{(Precision+Recall)} \tag{4}\]  
In the case of experiment 1, where the objective is to validate the algorithm generated for its application in port environmental management, the prior metrics are relevant and sufficient. However, in experiment 2, as well as in realistic system application, we would need an additional index that evaluates the performance of the system as an alternative to the common _Accuracy_ metric. This is due to the fact that the _Accuracy_ metric is not the most reliable in computer vision models trained on datasets where one event (in this case clean water) is much more frequent than the rest of the events (in this case spill or waste). In this case, _Accuracy_ will mostly determine that clean water is detected correctly most of the time but will not provide decisive information on the spill and waste detection performance. As the latter are the actual events (alarms) to be detected by a computer vision system applied in a port setting, _Accuracy_ is not a parameter that becomes useful in the present study or in real-life applications of the system. _Precision_, _Recall_ and _F1-Score_ indexes are also not suitable for experiment 2 because they are class specific and for comparison purposes an all-class synthetic index is needed. Consequently, a novel index has been defined for the purpose of this application (as well as others that might face similar issues as the one presented): the _Error index_.