# A Machine Learning Predictive Model for Ship Fuel Consumption
## 2 Materials and Methods
### Model Building and Training
#### 2.5.2 Assembly Method  
The basic concept in using the Ensemble Method is the development of a predictive model formed from the integration of simple models to obtain a better-performing model [25]. Two techniques of the Ensemble Method were used in the proposed project: Boosting [23] and Bagging [61]. The basic idea behind the ensemble method is the derivation of a predictive model by combining several simpler models. The Bagging method uses samples with replacement from the original data set to train reduced variance models [61; 63]. In contrast, each partition of the Augmentation Method uses the entire dataset, assigning higher weights to observations where previous models have underperformed [63]. Figure 4: Data distribution for training, testing, and hold-out validation. The random forest algorithm proposed by [PERSON] [60] relied on the bagging technique to reduce variance by averaging many imprecise but approximately unbiased decision trees [64]. To build a Random Forest model, a series of independent Regressor Decision Trees (RDTs) were generated using the training database laid out. The model response was determined by averaging the individual results of the decision trees, as laid out by Equation (6):  
\[\hat{y}_{i}(x)=\frac{1}{M}\sum_{m,i=1}^{M}f_{m}(x_{i}) \tag{6}\]  
where M is the number of decision trees in the model (n_estimators). Similar to the Random Forest (RFs) algorithm, the Extra Tree (ET) method is based on the principle of developing a set of regression-type decision trees from a standard structure containing sequentially root, child and leaf nodes, built from the top to the bottom of the tree. However, for an Extra Tree model, the cut-off points selected for splitting the tree nodes are extremely random, and it does not apply to the subsampling of the training database--all training samples are used [65]. In a scikit-learn implementation, the hyperparameters were similar to those used in a Decision Tree, including the parameter for the number of trees in the forest. Usually, a larger number of trees trains the data better. However, adding too many trees can slow down the training process considerably, so a parametric search to find the optimal configuration was necessary. The Gradient Boosting (GB) model is based on the boosting method, which produces an effective predictive model from a set of weak regression models, usually using decision trees [66]. A new regressor is built in addition to the model at different stages, allowing for optimization of the loss functions. In developing a Gradient Boosting model, a set of trees uses M additive functions to estimate the output, according to Equation (7):  
\[\hat{y}_{i}(x)=\sum_{m=1}^{M}f_{m}(x_{i}),f_{m}\in\zeta \tag{7}\]  
As presented in Equation (8), \(\zeta\) denotes the domain of the function that includes all regression trees:  
\[\zeta=f(x)=w_{q(x)},w\in\mathfrak{N}^{T},q:\mathfrak{N}^{d}\to T; \tag{8}\]  
where q denotes the structure of each tree that maps to the corresponding leaf index; T denotes the number of leaves in a tree. Each \(fm\) corresponds to an independent tree structure (q) and a leaf weight (w). Different from DTs, each regression tree contains a continuous score on each leaf, wj being the score of the respective leaf jth. Extreme Gradient Boosting (XGBoost)  
During the prediction process, the results of each tree were summed to obtain the final model results. The parameters of each tree (f\({}_{t}\)), which included the structure of the tree and the scores obtained by each leaf node, were determined. Using the additive training method, the results of a tree are added to the model at a given time. The predicted value (\(\hat{y}_{i}^{(t)}\)) obtained at step \(t\) could be used in the algorithm development process, according to Equation (9) [25]:  
\[\hat{y}_{i}^{(t)}=\sum_{m=1}^{M}f_{m}(x_{i})=\hat{y}_{i}^{(t-1)}+f_{t}(x_{i}) \tag{9}\]When setting the initial parameters of the model, each observation was predicted as the average of all observed response variables, determining an equal weight for all.