# From Ethnographic Research to Big Data Analytics--A Case of Maritime Energy-Efficiency Optimization
## 4 Applying Big Data Analytics for Maritime Energy Efficiency
### Evaluation
#### 4.3.1 Comparing MLP versus SOM Predictions  
We use two evaluation metrics to compare the two different models (SOM, MLP) with each other regarding accuracy and generalization and regarding accuracy to existing tools. All evaluations are performed 5-fold: models were trained on 80% of the data and evaluated on the remaining 20% (the validation fold). This process was repeated five times, such that every evaluation was included in the training data set four times and used for evaluation once each. Described evaluation metrics in this section are based on the average over these five runs. Models are evaluated on the following metrics.  
1. Accuracy: relative difference between outputs and expected outputs.
2. Generalization: like accuracy, but for models trained on a subset of ships and validated on another, disjunct set of ships.  
We evaluated the following model variants regarding accuracy and generalization for each of the model types (SOM and MLP) (Table 5). All models are the same regarding hyper parameter, except the dimension of the input layer for experiment A.  
Ships 1 and 5 operate on the same route, whereas ship 2 and 4 operate on two further routes. Ship 3 was not used for evaluation because the data quality was not sufficient (too many missing values, glitches, etc.). The output of the model (prediction) is the estimated fuel consumption at the time of the given inputs included parameter groups D or E (see Table 4). The signed relative error between model output and measured fuel consumption is computed. We used a signed metric for these experiments to see if models were biased to compute results that are too large or too small.  
Figure 10 shows the distribution over all evaluated samples. Boxes mark second quartile, median and third quartile. Whiskers indicate first and fourth quartile. The line inside the box marks the median, the upper and lower edge of the box the second and third quartile. The whiskers around the box depict the range of the first and fourth quantile. The MLP model performs best in experiment C, when looking at accuracy. This could be explained through the increase of available training data compared to experiments A and B. Models of experiment A and B predict too low fuel consumptions on the generalization data set. Ship 5 might have different engines or is built in a way that increases the fuel consumption compared to Ship 1. Experiment C shows that a model that was trained for a specific route cannot be used to predict fuel consumption for other routes. Prediction errors range from \(-55\%\) to \(+170\%\). We can see that models that have been trained on a limited set of routes do not generalize well to predict fuel consumption for other routes. It is not acceptable to build reliable prediction tools on a mode with a relative error from up to \(50\%\). More data have to be acquired to make these models more robust. SOMs do perform worse than the MLP models throughout all experiments. Relative errors of SOM experiments have a median close to \(0\%\) but spread to up to over \(100\%\). Although SOMs are, like MLPs, not robust enough to be used as solid predictors, considering the data set we obtained (just over a one-year period), the chosen hyper parameters enable the model to be generalizable, when trained on different ships, meaning that the model's performance is not dependent on a single ship.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Experiment** & **Training Data Set** & **Generalization Test on** \\ \hline A & Ship 1, with weather & Ship 5, with weather \\ B & Ship 1 & Ship 5 \\ C & Ship 1, Ship 5 & Ship 2 and 4 \\ D & All, except Ship 3 & - \\ \hline \hline \end{tabular}
\end{table}
Table 5: Evaluation configurations.  
This is also manifested by the fact that the model that was trained on multiple routes and multiple vessel voyages perform as well as the models that were trained on only one or two vessels on the same route (experiment D).