Automation for Interpretable Machine Learning Through a Comparison of Loss Functions to Regularisers
Year: 2021

Abstract:
To increase the ubiquity of machine learning it needs to be automated. Automation 1 is cost-effective as it allows experts to spend less time tuning the approach, which 2 leads to shorter development times. However, while this automation produces 3 highly accurate architectures, they can be uninterpretable, acting as ‘black-boxes’ 4 which produce low conventional errors but fail to model the underlying input-output 5 relationships—the ground truth. This paper explores the use of the Fit to Median 6 Error measure in machine learning regression automation, using evolutionary 7 computation in order to improve the approximation of the ground truth. When used 8 alongside conventional error measures it improves interpretability by regularising 9 learnt input-output relationships to the conditional median. It is compared to 10 traditional regularisers to illustrate that the use of the Fit to Median Error produces 11 regression neural networks which model more consistent input-output relationships. 12 The problem considered is ship power prediction using a fuel-saving air lubrication 13 system, which is highly stochastic in nature. The networks optimised for their 14 Fit to Median Error are shown to approximate the ground truth more consistently, 15 without sacrificing conventional Minkowski-r error values. 16 1 Development of Interpretable Machine Learning 17 Machine learning regression models are increasingly being used in industrial and engineering con18 texts for high-stakes decision making, automation and control. These methods often produce low 19 conventional error values, yet are known to produce physically inconsistent results which cannot 20 generalise off test set and so cannot be relied upon to model the ground truth of the system. The 21 models are designed to be accurate, not interpretable, and so a human cannot understand how changes 22 in the inputs change the prediction. In real-world applications, where the output can have a direct 23 effect on human life or the environment, model accuracy alone is not sufficient. 24 Trust is increased if a trained model approximates the true input-output relationships, performing 25 accurately within the bounds of the training data set and beyond it. It has been demonstrated that for 26 many applications minimising traditional error measures cannot guarantee an accurate approximation 27 of the ground truth (Willard et al. 2020). This is due to a poor inductive bias, the inherent prioritisation 28 of one solution over another (Battaglia et al. 2018), produced by conventional error measures which 29 are based on Minkowski-r metrics (Hanson & Burr 1987). 30 This trust can be increased by manually tuning to remove overfitting or to provide a solution that 31 makes more sense to the user. However, the expert knowledge and domain experience required to 32 properly tune a machine learning method manually are not always available in industry. Genetic 33 algorithms are therefore increasingly used to search a method’s hyperparameter space more efficiently 34 Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute. (Yang et al. 2021) (Kumar et al. 2021); which minimise conventional error measures on a test set, 35 often combined with lowering the complexity of the network. This automation exacerbates the 36 lack of interpretability, as models have a large flexibility, and prediction accuracy is prioritised, 37 a low conventional error is achieved without certainty that the method has modelled the correct 38 internal functions. Regularisation hyperparameters can be optimised alongside other neural network 39 parameters (Tani et al. 2021) (Luketina et al. 2016), which increases the search space and creates 40 more flexibility for methods to produce ‘accurate’ predictions and avoid overfitting. 41 Common regression regularisation methods are l1 and l2 regularisation and dropout. For l1 and l2 42 regularisation, large network weights are penalised in the loss function (Nowlan & Hinton 1992). The 43 absolute value of the weights is penalised in l1 regularisation and the squared value in l2, meaning 44 l1 encourages weights towards zero and l2 encourages weights to be small but non-zero. The l1, l2 45 and elastic net (l1+l2) regularisers improve a networks generality, increasing the applications where 46 the trained methods can be applied, by penalising complexity. Dropout, where a randomly selected 47 subset of weights are optimised at each epoch rather than the full set, improve the generality of the 48 trained models by preventing co-adaption of weight values (Srivastava et al. 2014). Dropout has been 49 shown to be equivalent to l2 regularisation after scaling by Fisher information (Wager et al. 2013), 50 suggesting that the two should not be used in unison. The neural network regularisation methods 51 discussed above aim to improve generality, reducing overfitting by simplifying the relationships 52 modelled by the networks. 53 Regularisers improve the modelling of the ground truth in scenarios adhering to the assumptions in the 54 proof in Bishop (1995), under which minimum Minkowski-r error values approximate the conditional 55 average of the dataset. This is because the inductive bias from the loss function guides the input56 output relationships towards the conditional average, while the regularisation stops overfitting by 57 simplfying the input-output relationships being modelled. However, these assumptions are restrictive 58 and it is noted that few regression applications adhere to them. For example, one assumption is that 59 the dataset is homoscedastic. In scenarios not adhering to these assumptions, network regularisation 60 simplifies the relationships being modelled but this does not necessarily improve the generality, or 61 model the ground truth. 62 The Fit to Median Error measure (Parkes et al. 2021) produces more interpretable regression, when 63 used in conjunction with conventional error measures. This is achieved by regularising the learnt 64 input-output relationships to the conditional median of the training dataset: the median output value, 65 conditioned on each isolated input variable in turn (Bishop 1995). For many regression applications 66 the conditional medians are a good approximation of the ground truth input-output relationships but 67 as yet it has not been explored as part of an automated approach. 68 A challenging regression problem is ship power prediction for a vessel using air lubrication to reduce 69 fuel consumption. It is chosen to be used in this study as it violates the assumptions in Bishop (1995), 70 where the noise in the output space is non-Gaussian and heteroscedastic. In this situation, correctly 71 modelling the ground truth and accurate prediction is required but there is limited understanding 72 of that ground truth (Parkes et al. 2018). The literature shows that shaft powering of a vessel can 73 be predicted with average accuracies of between 1.5-5% error with the use of a regression neural 74 network trained with high frequency data from the vessel (Pedersen & Larsen 2009), (Petersen et al. 75 2012), (Le et al. 2020), (Jeon et al. 2018), (Liang et al. 2019). All neural network applications to 76 ship power prediction in the literature use a combination of local searches and domain knowledge to 77 identify hyperparameter values. The addition of an air lubrication device increases the complexity of 78 the regression problem, as the system interacts with a number of interrelated input variables. 79 This paper explores the automation of neural network training to a new problem, with a focus 80 on producing a network which accurately models the ground truth. It compares the ground truth 81 representation of a neural network when a genetic algorithm optimises the network’s hyperparameters 82 to reduce the Mean Fit to Median Error measure and compares it to standard regularization using l1, 83 l2 and dropout, and to a network optimised to minimise the Maximum Absolute Error. It is illustrated 84 that neural network regularisation methods (l1, l2 and dropout) can be replaced by the use of the 85 Mean Fit to Median performance measure as an objective in the genetic algorithm, reducing the 86 complexity of the search space and producing networks which more consistently model the ground 87 truth. 88

To increase the ubiquity of machine learning it needs to be automated. Automation1 is cost-effective as it allows experts to spend less time tuning the approach, which2 leads to shorter development times. However, while this automation produces3 highly accurate architectures, they can be uninterpretable, acting as ‘black-boxes’4 which produce low conventional errors but fail to model the underlying input-output5 relationships—the ground truth. This paper explores the use of the Fit to Median6 Error measure in machine learning regression automation, using evolutionary7 computation in order to improve the approximation of the ground truth. When used8 alongside conventional error measures it improves interpretability by regularising9 learnt input-output relationships to the conditional median. It is compared to10 traditional regularisers to illustrate that the use of the Fit to Median Error produces11 regression neural networks which model more consistent input-output relationships.12 The problem considered is ship power prediction using a fuel-saving air lubrication13 system, which is highly stochastic in nature. The networks optimised for their14 Fit to Median Error are shown to approximate the ground truth more consistently,15 without sacrificing conventional Minkowski-r error values.16
1 Development of Interpretable Machine Learning17
Machine learning regression models are increasingly being used in industrial and engineering con-18 texts for high-stakes decision making, automation and control. These methods often produce low19 conventional error values, yet are known to produce physically inconsistent results which cannot20 generalise off test set and so cannot be relied upon to model the ground truth of the system. The21 models are designed to be accurate, not interpretable, and so a human cannot understand how changes22 in the inputs change the prediction. In real-world applications, where the output can have a direct23 effect on human life or the environment, model accuracy alone is not sufficient.24
Trust is increased if a trained model approximates the true input-output relationships, performing25 accurately within the bounds of the training data set and beyond it. It has been demonstrated that for26 many applications minimising traditional error measures cannot guarantee an accurate approximation27 of the ground truth (Willard et al. 2020). This is due to a poor inductive bias, the inherent prioritisation28 of one solution over another (Battaglia et al. 2018), produced by conventional error measures which29 are based on Minkowski-r metrics (Hanson & Burr 1987).30
This trust can be increased by manually tuning to remove overfitting or to provide a solution that31 makes more sense to the user. However, the expert knowledge and domain experience required to32 properly tune a machine learning method manually are not always available in industry. Genetic33 algorithms are therefore increasingly used to search a method’s hyperparameter space more efficiently34
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
(Yang et al. 2021) (Kumar et al. 2021); which minimise conventional error measures on a test set,35 often combined with lowering the complexity of the network. This automation exacerbates the36 lack of interpretability, as models have a large flexibility, and prediction accuracy is prioritised,37 a low conventional error is achieved without certainty that the method has modelled the correct38 internal functions. Regularisation hyperparameters can be optimised alongside other neural network39 parameters (Tani et al. 2021) (Luketina et al. 2016), which increases the search space and creates40 more flexibility for methods to produce ‘accurate’ predictions and avoid overfitting.41
Common regression regularisation methods are l1 and l2 regularisation and dropout. For l1 and l242 regularisation, large network weights are penalised in the loss function (Nowlan & Hinton 1992). The43 absolute value of the weights is penalised in l1 regularisation and the squared value in l2, meaning44 l1 encourages weights towards zero and l2 encourages weights to be small but non-zero. The l1, l245 and elastic net (l1+l2) regularisers improve a networks generality, increasing the applications where46 the trained methods can be applied, by penalising complexity. Dropout, where a randomly selected47 subset of weights are optimised at each epoch rather than the full set, improve the generality of the48 trained models by preventing co-adaption of weight values (Srivastava et al. 2014). Dropout has been49 shown to be equivalent to l2 regularisation after scaling by Fisher information (Wager et al. 2013),50 suggesting that the two should not be used in unison. The neural network regularisation methods51 discussed above aim to improve generality, reducing overfitting by simplifying the relationships52 modelled by the networks.53
Regularisers improve the modelling of the ground truth in scenarios adhering to the assumptions in the54 proof in Bishop (1995), under which minimum Minkowski-r error values approximate the conditional55 average of the dataset. This is because the inductive bias from the loss function guides the input-56 output relationships towards the conditional average, while the regularisation stops overfitting by57 simplfying the input-output relationships being modelled. However, these assumptions are restrictive58 and it is noted that few regression applications adhere to them. For example, one assumption is that59 the dataset is homoscedastic. In scenarios not adhering to these assumptions, network regularisation60 simplifies the relationships being modelled but this does not necessarily improve the generality, or61 model the ground truth.62
The Fit to Median Error measure (Parkes et al. 2021) produces more interpretable regression, when63 used in conjunction with conventional error measures. This is achieved by regularising the learnt64 input-output relationships to the conditional median of the training dataset: the median output value,65 conditioned on each isolated input variable in turn (Bishop 1995). For many regression applications66 the conditional medians are a good approximation of the ground truth input-output relationships but67 as yet it has not been explored as part of an automated approach.68
A challenging regression problem is ship power prediction for a vessel using air lubrication to reduce69 fuel consumption. It is chosen to be used in this study as it violates the assumptions in Bishop (1995),70 where the noise in the output space is non-Gaussian and heteroscedastic. In this situation, correctly71 modelling the ground truth and accurate prediction is required but there is limited understanding72 of that ground truth (Parkes et al. 2018). The literature shows that shaft powering of a vessel can73 be predicted with average accuracies of between 1.5-5% error with the use of a regression neural74 network trained with high frequency data from the vessel (Pedersen & Larsen 2009), (Petersen et al.75 2012), (Le et al. 2020), (Jeon et al. 2018), (Liang et al. 2019). All neural network applications to76 ship power prediction in the literature use a combination of local searches and domain knowledge to77 identify hyperparameter values. The addition of an air lubrication device increases the complexity of78 the regression problem, as the system interacts with a number of interrelated input variables.79
This paper explores the automation of neural network training to a new problem, with a focus80 on producing a network which accurately models the ground truth. It compares the ground truth81 representation of a neural network when a genetic algorithm optimises the network’s hyperparameters82 to reduce the Mean Fit to Median Error measure and compares it to standard regularization using l1,83 l2 and dropout, and to a network optimised to minimise the Maximum Absolute Error. It is illustrated84 that neural network regularisation methods (l1, l2 and dropout) can be replaced by the use of the85 Mean Fit to Median performance measure as an objective in the genetic algorithm, reducing the86 complexity of the search space and producing networks which more consistently model the ground87 truth.88
2 Neural Networks Parameters89
Previous applications of neural networks to ship power prediction use between 1 and 3 hidden layers90 (Leifsson et al. 2008) (Parkes et al. 2019), and between 5 and 300 neurons in each hidden layer (Jeon91 et al. 2018). To provide a sufficiently large search space to allow verification, or otherwise, of these92 parameters a maximum of 4 hidden layers and 1000 neurons in each layer are used. The majority93 of the literature treats the problem as time-invariant and use feed-forward networks, so no recurrent94 parameters are optimised. As the optimiser or activation functions are rarely documented in the95 literature, the state-of-the-art optimisers and activation functions available in the Keras framework96 (Chollet et al. 2015) are used in the optimisation, Table 1.97
The number of epochs and early stopping procedure are not optimised, as there was a need for98 predictable compute requirements and allowing the optimisation of these parameters leads to unpre-99 dictable run times. The number of epochs to train each network increases for increasing generation100 number in the genetic algorithm, from 1 epoch in the first 15 generations to 20 in the final 15. This101 was also implemented to reduce compute and it was validated that when more than 20 epochs were102 allowed, that the early stopping, with a patience of 5, stopped the training within 20 epochs for the103 majority of networks. The loss function is similarly not optimised, the Mean Absolute Error is used,104 as the conditional medians are closer to the ground truth input-output relationships in these datasets105 than the conditional means.106
The performance measures, or the genetic algorithm’s fitness functions, are the Mean Absolute107 Relative Error, the Maximum Absolute Relative Error and the Mean Fit to Median Error. Different108 combinations of these, alongside the use of regularisation parameters in the search space are compared109 to illustrate the effect of different types of regularisation.110
3 cMLSGA Parameters111
In this study cMLSGA1 is selected as it shows the top performance on a range of evolutionary112 benchmarking problems (Grudniewski & Sobey 2021) and practical problems (Grudniewski & Sobey113 2019). Genetic algorithms are increasing used to tune neural network hyperparameters including114 regularisation parameters for use on new problems (Jin et al. 2004). Many approaches have multiple115 genetic algorithm objectives, although these all minimise an error measure and a measure of network116 complexity (Wang et al. 2019) and (Smith & Jin 2014). The use of multiple different performance117 measures as objectives is yet to be explored in the literature.118
Four approaches are investigated in this study, summarised in Table 3, for approach (GAi) and119 (GAii) the genetic algorithm cMLSGA optimises all variables in Table 2, including the l1 and l2120 regularisation rate and the dropout rate of the networks. Although it is advised that l2 regularisation121 and dropout are not used in the same network the genetic algorithms are provided with zero options122 for all regularisation parameters, to identify if one is preferable in this scenario.123
Approach (GAi) is a single objective genetic algorithm optimising the Mean Absolute Error which is124 compared to a multi-objective formulation where the (GAii) approach optimises both Mean Absolute125 Error and Maximum Absolute Error. For approaches (GAiii) and (GAiv) no network regularisation126 parameters are optimised: l1, l2 and dropout rates are all set permanently to zero. They avoid127 producing networks that have overfitted by the use of two performance metrics as multi-objectives,128 (GAiii) uses the Mean Fit to Median and Mean Absolute Errors to be minimised and (GAiv) uses the129 Maximum Absolute and Mean Absolute. All approaches use 40 CPUs with 2.0 GHz Intel Skylake130 processors and 192 GB of DDR4 memory, and take less than 3 days, this setup may not be feasible131 for widespread industrial application, although it is suggested it is within reach of some industries.132
4 Data133
The data used in this study are from a large vessel equipped with the Silverstream® Air Lubrication134 System. The air lubrication system works through use of fluid sheering to create an air microbubble135 carpet directly captured within the boundary layer on the ship hull bottom. The bubble carpet reduces136 the frictional resistance thereby increasing the speed and reducing the shaft power. Compressors137 provide a constant supply of air to the hull bottom to maintain a uniform bubble carpet operated at138 the optimal compressor power that maximises the energy balance. The study is performed on both139 system on and system off datasets, however for brevity only results for system off are presented as140 they show similar performance.This prediction is required for a baseline determination of how the141 system is working, but the relationships between the power, weather, ocean and operating conditions142 are complex and difficult to model.143
The variables considered in this study are the shaft power, speed through water, relative wind speed144 and direction, draught and trim, with shaft power the target variable. These are selected based on145 a detailed study into variable selection for shaft power prediction (Parkes et al. 2019). The speed146 through water is selected over the speed over ground, for use as an input variable, as it is more147 hydrodymanically relevant and its accuracy is validated by comparison to the speed over ground.148 The dataset is cleaned by removing rows with missing or non-physical values and all datapoints149 below 0.05 normalised shaft power are removed. The dataset is split into two using the air lubrication150 system status: system on and system off, where system on is defined as air lubrication system power151 greater than zero. The system on dataset contains 352,690 datapoints and system off contains 237,962.152 The data is split into training, testing and validation sets of 70%, 15% and 15% respectively. Each153
1The code for cMLSGA is available at https://www.bitbucket.org/*******.
network in the genetic algorithm trains on a randomly sampled 35,000 datapoints from the training154 set and uses randomly sampled sets of size 7,500 from validation and testing sets for validation during155 training, and testing to produce the fitness of the network for the genetic algorithm. The errors stated156 in the paper are from networks on the Pareto fronts of each approach, which are validated on the full157 testing set.158
The datasets contain large regions of sparse data in all input variable domains, this is exemplified159 by the ship speed domain where each half-knot interval below 16 knots contains less than 0.8% of160 the data, which accounts for more than half the speed domain, Figure 1. In addition, the boxplot161 ranges and outliers show high heteroscedicity with idiosyncratic noise caused by situations where the162 angle of the propeller blades is varied to achieve the required speed. This highlights the complexity163 in developing models of the powering of this vessel, as the dataset also contains the effects from other164 latent variables, such as piloting behaviour and route taken.165
5 Optimisation including regularisation parameters: (GAi) and (GAii)166
Previous studies predicting ship powering using neural networks report that l1, l2 and elastic net167 increase both test set and off-test set errors and that optimal values for both l1 and l2 are zero.168 Therefore the genetic algorithm setup is biased towards low and zero values of regularisation rates by169 using a set of exponentially decreasing values and an explicit zero option.170
The single objective (GAi) fails to identify that zero regularisation rates produce the lowest errors,171 favouring networks with the highest possible rate of l2 (0.01), Figure 2b. (GAi) produces networks172 with the highest Mean Absolute Relative Errors of all the approaches, (5.19± 0.00)% from Figure173 4a. In contrast, (GAii) favours lower l1 and l2 rates of 0 or 0.00001, Figures 2c and 2d, which results174 in networks with the lowest Mean Absolute Relative Errors of all four approaches, on average, with a175 value of (2.87±0.45)%, shown in Figure 4a. This is around 0.5% higher than the lowest documented176 error for ship power prediction.177
It is posited that the high error for the single objective problem is directly related to the use of a178 large l2 regularisation rate, as noted in previous studies for ship power prediction. It is possible179 that the use of a multi-objective search algorithm for a single-objective problem means that the180 optimal hyperparameters can’t be found, resulting in large errors. The implementation also requires181 restrictions in the number of epochs used for training in the initial generations, it is possible this182 biases (GAi) towards certain size networks, where higher l2 rates are preferable. This hypothesis is183
supported by the fact that 74.4% of networks in the first 15 generations of (GAi) have 1 hidden layer,184 and that over 99.8% of the networks in the final 15 generations have 1 hidden layer, with 880± 17185 neurons in this layer, Figure 3b. This is significantly more neurons than those in the hidden layer of186 networks in the final 15 generations of (GAii) which range from 3-952 with a median value of 709,187 Figure 3b. The added objective of minimising Maximum Absolute Error in (GAii) may cause these188 slightly smaller networks to be more attractive as they are in a sense regularised by their size, as they189 have reduced modelling flexibility therefore are less likely to overfit and produce high Maximum190 Absolute Errors.191
Another explanation for the difference in l2 rates chosen by (GAi) and (GAii) is the equivalence192 of l2 and dropout. Since l2 and dropout are equivalent up to a Fisher transformation, their use in193 conjunction is not recommended. The evidence for this is that (GAi) favours the highest l2 rate and194 has a median dropout rate in the final 15 generations of 0.116, whereas (GAii) favours the zero l2 rate195 and has a median dropout rate of 0.624, Figure 3a. This illustrates that the genetic algorithms will196 chose either l2 or dropout to minimise the Mean Absolute Relative Error. The l1 rates also support197 this hypothesis, as chosen rates for l1 regularisation in the final 15 generations are more comparable198 for (GAi) and (GAii).199
6 Optimisation using multiple performance measures: (GAiii) and (GAiv)200
For approaches (GAiii) and (GAiv) all neural network regularisation parameters are set to zero.201 The regularisation is performed by minimising different network performance measures, the Mean202 Absolute and Mean Fit to Median for (GAiii), and the Mean Absolute and Maximum Absolute for203 (GAiv). The trade-off between the two objectives produces regularised neural networks, without204 explicitly changing the architecture or loss function. The Mean Fit to Median is chosen as it indicates205 how close the relationships modelled by a network are to the conditional averages of the dataset, in206 many regression examples this is akin to the ground truth input-output relationships (Parkes et al.207 2021). The Maximum Absolute is chosen as for many industrial applications of machine learning the208 maximum prediction error is more pertinent than the mean error. The Mean Absolute Error is used209 instead of the Mean Squared Error in both approaches, as the conditional medians are closer to the210 ground truth input-output relationships in these datasets than the conditional means.211
Differently shaped networks are favoured by (GAiii) and (GAiv), compared to (GAi) and (GAii),212 focusing on networks with 3 hidden layers and on average less than 400 neurons in each layer, Figure213 3b. These networks have 51 times the number of connections than the networks chosen in (GAi) and214 (GAii). Apart from (GAi), (GAiii) has the most consistently sized networks in the final 15 generations,215 with an interquartile range of 46 neurons, compared to (GAiv) which have an interquartile range of216 131 neurons. It is suggested that as the Mean Fit to Median Error biases networks towards specific217 input-output relationships, there is a smaller range of potential network architectures which habitually218 model these relationships. Whereas networks which minimise the Maximum Absolute Error are less219 restricted and can model a wider range of input and output relationships.220
The Mean Absolute Relative Errors from networks in the Pareto fronts are (2.97±0.25)% for (GAiii)221 and (3.10 ± 0.28)% for (GAiv). It is expected that (GAiv) would produce higher Mean Absolute222 Relative Errors as discussed above, minimising the Maximum Absolute Error should bias predictions223 towards the midpoint of the conditional output distributions, whereas minimising the Mean Absolute224 Error should bias predictions towards the median of these distributions. As it is established that noise225 in the output distribution is non-Gaussian, Figure 1, these values will not align so some sacrifice226 in Mean Absolute Error is expected from (GAiv). Both (GAiii) and (GAiv) produce comparable227 Maximum Absolute Errors, of (49.5± 1.1)% and (51.9± 4.5)%. It is suggested that this is because,228 although the conditional median output value and conditional midpoint output value do not align229 for the majority of the input domain, they are sufficiently close to produce comparable Maximum230 Absolute Errors.231
Across all four approaches, the genetic algorithm producing networks with the highest Mean Absolute232 Error is the approach which does not provide extra weighting to sparse areas of data. The approaches233 minimising Maximum Absolute Error are implicitly biased away from networks which predict the234 majority of the testing datapoints correctly, but predict one datapoint poorly, favouring networks235 which predict all testing datapoints to a moderate degree of error. Approach (GAiii) more explicitly236 weights prediction in sparse areas of data by favouring networks which model the conditional median237 of the dataset across all input domains, irrespective of the quantity of data across each input domain.238 The regression problem of ship power prediction is chosen in part because of it’s irregular data239 distribution; more than 9% of the dataset lies in less than a 0.5 knot interval of ship speed, Figure 1.240 This provides an explanation for the high testing errors from (GAi), where only the Mean Absolute241 Error is minimised, there is little incentive for the genetic algorithm to produce networks which242 generalise across the full range of the input domain well.243
7 Comparison of the interpretability244
To asses the interpretability of the networks selected by the four different approaches the learnt245 relationship between an input, the ship speed, and the output, shaft power, for the networks in246 the Pareto front of each approach are visualised, Figure 5. These are extracted with the following247 procedure: set all but one input variable to be constant at the mode; cycle the remaining variable from248 its minimum to its maximum recorded values with 150 points evenly spaced along the domain and249 run the new dataset through the trained network.250
The approach which produces the most consistent speed-power relationships is (GAi), with an average251 variation of 1.8%2, Figure 5a. However, the relationship modelled by the 5 networks with the lowest252 Mean Absolute Relative Error in (GAi) all approximate a piece-wise linear relationship which clearly253 underfits the dataset in Figure 1. The expected trend between ship speed through the water and shaft254
2Average variation in just the speed-power curves are discussed in this section, but it is verified that all input-power curves follow the same trends with variation around 0.5% across input variables.
power is a cubic polynomial, therefore as well as producing the highest Mean Absolute Relative255 Errors, networks chosen by (GAi) model the ground truth input-output relationships the worst out of256 the four approaches. Both (GAii) and (GAiv) produce 5 fairly consistent speed-power curves, with257 average variations of 5.9% and 10% respectively, Figures 5b and 5d. Both approaches approximate258 smooth polynomial curves, although the degrees of the polynomials might differ, as multiple curves259 intersect at various points along the speed axis. The spread of learnt relationships is greater at the260 highest and lowest speeds for (GAii), with a decrease in spread for speeds of around 15 knots, where261 many of the curves intersect. The curves from (GAiv) show equal spread across the speed domain.262
The approach with both accurate and consistent learnt speed-power curves is (GAiii), with limited263 intersections of curves and an average spread of 3.0%. It is suggested that the reason using the264 Mean Fit to Median Error as an objective in a multi-objective genetic algorithm produces more265 interpretable results, or more consistent learnt relationships, is because instead of encouraging the266 networks to model more simple relationships. It encourages the networks to model the conditional267 median functions of the dataset, supported by the increase in network connections. Whereas the other268 approaches leave room for networks to fail to model the conditional averages, especially in irregularly269 distributed and non-normally distributed datasets. The Mean Absolute Error values from networks270 selected by (GAiii) are on average 0.1% higher than those from (GAii), and the Maximum Absolute271 Error values are 1.6% higher.272
A limitation of the approach is that the Fit to Median Error measure will likely perform best at273 improving interpretability on datasets which violate the assumptions in Bishop (1995); the ship274 powering example is chosen to illustrate this as it provides a clearly heteroscedastic dataset. For275 applications where noise profiles are Gaussian, and there is no effect from latent or interrelated input276 variables, the Fit to Median Error will not improve interpretability, but will perform the same as277 conventional Minkowski-r metrics, either Mean Squared or Mean Absolute Error depending on the278 convexity of input-output relationships.279
Interestingly, the approach producing the lowest Mean Absolute and Maximum Absolute Errors280 does not model the ground truth the most accurately. This creates a potential for negative societal281 impacts, as the standard performance metrics for regression neural networks do not provide a full282 picture of performance or expected behaviour. Interpretability of trained methods is essential for283 safe application of machine learning in the real world, especially when automated methods are284 used to replace experienced professionals. (GAii) demonstrates the same accuracy of approach as285 those with standard network regularisation, but with a better fit to the ground truth. This approach286 bypasses the need to use, and therefore to optimise the parameters of the regularisation methods.287 If evolutionary computation is already being used to optimise network parameters, then compute288 is saved by removing the network regularisation parameters l1, l2 and dropout. (GAiii) completed289 300 generations in 46hours whereas (GAii) required 12 hours more computation to complete 300290 generations.291
8 Conclusion292
Interpretable and accurate methods are required for widespread application of machine learning293 to real-world regression problems. To automate the training of neural networks so that the result294 is interpretable, three different genetic algorithm approaches are compared: one to minimise the295 Maximum Absolute Error of the networks, which includes standard regularization using l1, l2 and296 dropout; and two which do not use any network regularisation, one minimising the Mean Fit to297 Median and one to minimise the Maximum Absolute Error. The results show that all three approaches298 give similar Mean Absolute Errors from networks on their Pareto fronts, from 2.9% for the approach299 with regularisation to 3.1% for the approach minimising Maximum Absolute Error. However, the300 Mean Fit to the Median approach shows a considerably better interpretability, or fit to the ground301 truth, with a spread in predicted input-output curves of 3% compared to a spread of 6% for the302 approach using regularisation and 10% when minimising the Maximum Absolute Error.303
References304 Battaglia, P. W., Hamrick, J. B., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V., Malinowski, M.,305 Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., Gulcehre, C., Song, F., Ballard, A., Gilmer,306 J., Dahl, G., Vaswani, A., Allen, K., Nash, C., Langston, V., Dyer, C., Heess, N., Wierstra, D.,307
Kohli, P., Botvinick, M., Vinyals, O., Li, Y. & Pascanu, R. (2018), ‘Relational inductive biases,308 deep learning, and graph networks’.309
Bishop, C. (1995), Neural Networks for Pattern Recognition, Oxford University Press, chapter 6,310 pp. 194–225.311
Chollet, F. et al. (2015), ‘Keras’, https://keras.io.312
Dozat, T. (2016), ‘Incorporating nesterov momentum into adam’, International Conference on313 Learning Representations 2016 .314
Duchi, J., Hazan, E. & Singer, Y. (2011), ‘Adaptive subgradient methods for online learning and315 stochastic optimization’, Journal of Machine Learning Research 12(61), 2121–2159.316
Grudniewski, P. A. & Sobey, A. J. (2019), Do general genetic algorithms provide benefits when317 solving real problems?, in ‘2019 IEEE Congress on Evolutionary Computation (CEC)’, IEEE,318 pp. 1822–1829.319
Grudniewski, P. A. & Sobey, A. J. (2021), ‘cMLSGA: a co-evolutionary multi-level selection genetic320 algorithm for multi-objective optimization’.321
Hanson, S. & Burr, D. (1987), Minkowski-r back-propagation: learning in connectionist models with322 non-euclidian error signals., in ‘Neural Information Processing Systems (NIPS 1987)’.323
Hinton, G., Srivastava, N. & Swersky, K. (2012), ‘Lecture 6a:overview of mini-batch gradient descent’. URL: http://www.cs.toronto.edu/ tijmen/csc321/slides/lectureslideslec6.pdf
Jeon, M., Noh, Y., Shin, Y., Lim, O., Lee, I. & Cho, D. (2018), ‘Prediction of ship fuel consumption324 by using an artificial neural network’, Journal of Mechanical Science and Technology 32(12), 5785–325 5796.326
Jin, Y., Okabe, T. & Sendhoff, B. (2004), Neural network regularization and ensembling using327 multi-objective evolutionary algorithms, in ‘Proceedings of the 2004 Congress on Evolutionary328 Computation (IEEE Cat. No.04TH8753)’, Vol. 1.329
Kingma, D. P. & Ba, J. (2014), ‘Adam: A method for stochastic optimization’, arXiv preprint330 arXiv:1412.6980 .331
Kumar, P., Batra, S. & Raman, B. (2021), ‘Deep neural network hyper-parameter tuning through332 twofold genetic approach’, Soft Computing pp. 1–25.333
Le, L., Lee, G., Park, K. & Kim, H. (2020), ‘Neural network-based fuel consumption estimation for334 container ships in korea’, Maritime Policy & Management pp. 1–18.335
Leifsson, L., Sævarsdóttir, H., Sigurðsson, S. & Vésteinsson, A. (2008), ‘Grey-box modeling of an336 ocean vessel for operational optimization’, Simulation Modelling Practice and Theory 16(8), 923–337 932.338
Liang, Q., Tvete, H. A. & Brinks, H. W. (2019), Prediction of vessel propulsion power using machine339 learning on ais data, ship performance measurements and weather data, in ‘Journal of Physics:340 Conference Series’, Vol. 1357, p. 012038.341
Luketina, J., Berglund, M., Greff, K. & Raiko, T. (2016), ‘Scalable gradient-based tuning of continu-342 ous regularization hyperparameters’.343
Nowlan, S. J. & Hinton, G. E. (1992), ‘Simplifying neural networks by soft weight sharing’, Neural344 Computation .345
Parkes, A. I., Savasta, T. D., Sobey, A. J. & Hudson, D. A. (2019), Efficient vessel power prediction346 in operational conditions using machine learning., in ‘Practical Design of Ships and Other Floating347 Structures(PRADS), September 2019, Yokohama, Japan’.348
Parkes, A. I., Sobey, A. J. & Hudson, D. A. (2018), ‘Physics-based shaft power prediction for large349 merchant ships using neural networks’, Ocean Engineering 166, 92–104.350
Parkes, A. I., Sobey, A. J. & Hudson, D. A. (2021), ‘Towards error measures which influence a351 learners inductive bias to the ground truth’.352
Pedersen, B. P. & Larsen, J. (2009), Prediction of full-scale propulsion power using artificial neural353 networks, in ‘Proceedings of the 8th international conference on computer and IT applications in354 the maritime industries (COMPIT’09), Budapest, Hungary May’, pp. 10–12.355
Petersen, J. P., Jacobsen, D. J. & Winther, O. (2012), ‘Statistical modelling for ship propulsion356 efficiency’, Journal of marine science and technology 17(1), 30–39.357
Smith, C. & Jin, Y. (2014), ‘Evolutionary multi-objective generation of recurrent neural network358 ensembles for time series prediction’, Neurocomputing 143, 302–311.359
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. (2014), ‘Dropout:360 a simple way to prevent neural networks from overfitting’, The Journal of Machine Learning361 Research 15(1), 1929–1958.362
Tani, L., Rand, D., Veelken, C. & Kadastik, M. (2021), ‘Evolutionary algorithms for hyperparameter363 optimization in machine learning for application in high energy physics’, The European Physical364 Journal C 81(2), 1–9.365
Wager, S., Wang, S. & Liang, P. (2013), ‘Dropout training as adaptive regularization’, arXiv preprint366 arXiv:1307.1493 .367
Wang, B., Sun, Y., Xue, B. & Zhang, M. (2019), Evolving deep neural networks by multi-objective368 particle swarm optimization for image classification, in ‘Proceedings of the Genetic and Evolution-369 ary Computation Conference’, GECCO ’19, p. 490–498.370
Willard, J., Jia, X., Xu, S., Steinbach, M. & Kumar, V. (2020), ‘Integrating physics-based modeling371 with machine learning: a survey’.372
Yang, S., Tian, Y., He, C., Zhang, X., Tan, K. C. & Jin, Y. (2021), ‘A gradient-guided evolutionary373 approach to training deep neural networks’, IEEE Transactions on Neural Networks and Learning374 Systems pp. 1–15.375
Zeiler, M. D. (2012), ‘Adadelta: an adaptive learning rate method’.376
Checklist377
The checklist follows the references. Please read the checklist guidelines carefully for information on378 how to answer these questions. For each question, change the default [TODO] to [Yes] , [No] , or379 [N/A] . You are strongly encouraged to include a justification to your answer, either by referencing380 the appropriate section of your paper or providing a brief inline description. For example:381
• Did you include the license to the code and datasets? [Yes] See Section ??.382 • Did you include the license to the code and datasets? [No] The code and the data are383 proprietary.384 • Did you include the license to the code and datasets? [N/A]385
Please do not modify the questions and only use the provided macros for your answers. Note that the386 Checklist section does not count towards the page limit. In your paper, please delete this instructions387 block and only keep the Checklist section heading above along with the questions/answers below.388
1. For all authors...389 (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s390 contributions and scope? [Yes] The scope of the paper (regression problem automation)391 is made clear in the abstract. The claims of improved interpretability using the Fit to392 Median Error measure are reflected in Figure 5, as well as the body of text.393
(b) Did you describe the limitations of your work? [Yes] The limitations of the Fit to394 Median are discussed in Section 7, and the limitations relating to compute are discussed395 in Section 3.396
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See397 Section 6398
(d) Have you read the ethics review guidelines and ensured that your paper conforms to399 them? [Yes]400
2. If you are including theoretical results...401 (a) Did you state the full set of assumptions of all theoretical results? [No] Citation of402 (Bishop 1995) is included instead of explicit statement of all assumptions, although the403 most pertinent assumptions are discussed in Section 1404
(b) Did you include complete proofs of all theoretical results? [N/A] No theorems are405 posited, the paper explores a practical comparison of common methods.406
3. If you ran experiments...407 (a) Did you include the code, data, and instructions needed to reproduce the main ex-408 perimental results (either in the supplemental material or as a URL)? [Yes] Instruc-409 tions needed to replicate the results on a different dataset are provided in the main410 text; Sections 3 and 2 specify the expermental setup. The code is based on the411 cMLSGA algorithm, available at https://www.bitbucket.org/******* (redacted412 for anonymity) which is provided in Section 3. The data is proprietary so is not pro-413 vided.414
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they415 were chosen)? [Yes] See Sections 3 and 2416
(c) Did you report error bars (e.g., with respect to the random seed after running ex-417 periments multiple times)? [Yes] All results are reported to plus/minus the standard418 deviation, and all figures show multiple runs with respect to either a random seed or an419 evolutionary method.420
(d) Did you include the total amount of compute and the type of resources used (e.g., type421 of GPUs, internal cluster, or cloud provider)? [Yes] See Section 3422
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...423 (a) If your work uses existing assets, did you cite the creators? [Yes] The creators of the424 genetic algorithim cMLSGA are cited in Section 3 and the data is acknowledged to425 belong to Silverstream Technologies Ltd in Section 4426
(b) Did you mention the license of the assets? [Yes] The code is licensed under the GNU427 General Public License. The data is not licensed.428
(c) Did you include any new assets either in the supplemental material or as a URL? [No]429 (d) Did you discuss whether and how consent was obtained from people whose data430 you’re using/curating? [No] The work is performed in collaboration with Silverstream431 Techonologies Ltd.432
(e) Did you discuss whether the data you are using/curating contains personally identifiable433 information or offensive content? [No] The shaft power variable is normalised to434 remove any possibility of identifying the vessel in question.435
5. If you used crowdsourcing or conducted research with human subjects...436 (a) Did you include the full text of instructions given to participants and screenshots, if437 applicable? [N/A]438 (b) Did you describe any potential participant risks, with links to Institutional Review439 Board (IRB) approvals, if applicable? [N/A]440 (c) Did you include the estimated hourly wage paid to participants and the total amount441 spent on participant compensation? [N/A]442

Review summaries:
This paper studies the interpretability of neural networks. Specifically, the authors use four approaches with different objectives and regularizations and use cMLSGA, a genetic optimization method to optimize these problems. Eventually, the authors conduct experiments to validate that Fit to Median Error measure possesses better interpretability compared to L1, L2 and dropout regularizations since it regularizes the learnt input-output relationships to the conditional median. 

The paper claims that building ML models through automation can become a black-box approach that results in lower errors but also lower interpretability. Also, minimizing traditional error measures cannot guarantee an accurate approximation of the ground truth. The paper studies using fit to Median Error measure to produce better model with regard to interpretability of models.   The paper uses ship power prediction problem that aims to reduce fuel consumption, a challenging regression problem, as an application where fit to Median Error measure is used. The paper compares four different genetic algorithms settings: minimizing the Maximum Absolute Error of the networks, minimizing the Maximum Absolute Error and minimizing the Mean Fit to Median.   The experiment results demonstrate that all four settings produced similar Mean Absolute Errors from networks on their Pareto fronts. But the Mean Fit to the Median setting obtained much better interpretability (fit to the ground truth).