Automation for Interpretable Machine Learning Through a Comparison of Loss Functions to Regularisers
Authors: Amy Parkes
Adam Sobey
Josef Camilleri
Dominic Hudson
Venue: NeurIPS 2021 Submitted (2021)
Keywords: regularisation
interpretability
evolutionary computation
neural networks

Abstract: To increase the ubiquity of machine learning it needs to be automated. Automation is cost-effective as it allows experts to spend less time tuning the approach, which leads to shorter development times. However, while this automation produces highly accurate architectures, they can be uninterpretable, acting as `black-boxes' which produce low conventional errors but fail to model the underlying input-output relationships---the ground truth. This paper explores the use of the Fit to Median Error measure in machine learning regression automation, using evolutionary computation in order to improve the approximation of the ground truth. When used alongside conventional error measures it improves interpretability by regularising learnt input-output relationships to the conditional median. It is  compared to traditional regularisers to illustrate that the use of the Fit to Median Error produces regression neural networks which model more consistent input-output relationships. The problem considered is ship power prediction using a fuel-saving air lubrication system, which is highly stochastic in nature. The networks optimised for their Fit to Median Error are shown to approximate the ground truth more consistently, without sacrificing conventional Minkowski-r error values.

Automation for Interpretable Machine Learning Through a Comparison of Loss Functions to Regularisers
Anonymous Author(s) Affiliation Address
email
Abstract
To increase the ubiquity of machine learning it needs to automated. Automation is cost-effective as it allows experts to spend less time tuning the approach, which leads to shorter development times. However, while this automation produces highly accurate architectures, they can be uninterpretable, acting "black-boxes which produce low conventional errors but fail to model the underlying input-output relationships the ground truth. This paper explores the use of the Fit to Median Error measure machine learning regression automation, using evolutionary computation in order to improve the approximation of the ground truth. When used alongside conventional error measures it improves interpretability by regularising learnt input-output relationships to the conditional median. It is compared traditional regularisers to illustrate that the use of the Fit Median Error produces regression neural networks which model more consistent input-outpur relationships. The problem considered is ship power prediction using a fuel-saving air lubrication system, which is highly stochastic in nature. The networks optimised for their Fit to Median Error are shown to approximate the ground truth more consistently, without sacrificing conventional Minkowski-r error values.
11
13
15
Development of Interpretable Machine Learning
17
18 Machine learning regression models are increasingly being used in industrial and engineering con- texts for high-stakes decision making, automation and control. These methods often produce low conventional error values, yet are known to produce physically inconsistent results which cannot generalise off test set and cannot be relied upon to model the ground truth of the system. The
20
21
22 models are designed to accurate, not interpretable, and so human cannot understand how changes in the inputs change the prediction. In real-world applications, where the output can have direct effect on human life or the environment, model accuracy alone is not sufficient.
24
25 Trust is increased if a trained model approximates the true input-output relationships, performing accurately within the bounds of the training data set and beyond it. has been demonstrated that for many applications minimising traditional eтror measures cannot guarantee an accurate approximation of the ground truth (Willard et al. 2020). This is due to a poor inductive bias, the inherent prioritisation
26
27
28
29 of one solution over another (Battaglia et al. 2018), produced by conventional error measures which
30 are based on Minkowski-r metrics (Hanson & Burr 1987).
31 This trust can be increased by manually tuning to remove overfitting or to provide solution that
32 makes more sense the user. However, the expert knowledge and domain experience required to
33 properly tune a machine learning method manually are not always available in industry. Genetic
34 algorithms are therefore increasingly used search method's hyperparameter space more efficiently
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

35
(Yang et al. 2 (Kumar et al. 202T]; which minimise conventional error measures on test set, often combined with lowering the complexity of the network. This automation exacerbates the lack of interpretability, as models have large flexibility, and prediction accuracy is prioritised, a low conventional error is achieved without certainty that the method has modelled the correct internal functions. Regularisation hyperparameters can be optimised alongside other neural network parameters (Tani ct al. 2021) (Luketina et al. 2016), which increases the search space and creates more flexibility for methods to produce "accurate" predictions and avoid overfitting.
36
37
38
39
40
41
42
Common regression regularisation methods are 11 and 12 regularisation and dropout. For 11 and 12 regularisation, large network weights are penalised in the loss function (Nowlan & Hinton 1992). The absolute value of the weights is penalised in 11 regularisation and the squared value in 12, meaning Il encourages weights towards zero and 12 encourages weights to be small but non-zero. The 11, 12 and elastic net (1I++1) regularisers improve a networks generality, increasing the applications where the trained methods can be applied, by penalising complexity. Dropout, where randomly selected subset of weights are optimised at each epoch rather than the full set, improve the generality of the trained models by preventing co-adaption of weight values (Srivastava et al. 2014). Dropout has been shown to be equivalent to 12 regularisation after scaling by Fisher information (Wager et al. 2013)). suggesting that the two should not be used in unison. The neural network regularisation methods discussed above aim to improve generality, reducing overfitting by simplifying the relationships modelled by the networks.
43
44
45
46
47
48
19
50
51
52
54
Regularisers improve the modelling of the ground truth in scenarios adhering to the assumptions in the proof in |Bishop 1995), under which minimum Minkowski-r error values approximate the conditional average of the dataset. This is because the inductive bias from the loss function guides the input- output relationships towards the conditional average, while the regularisation stops overfitting by simplfying the input-output relationships being modelled. However, these assumptions are restrictive and it is noted that few regression applications adhere to them. For example, one assumption is that the dataset is homoscedastic. In scenarios not adhering to these assumptions, network regularisation simplifies the relationships being modelled but this does not necessarily improve the generality, or model the ground truth.
55
56
57
58
60
61
62
63
The Fit to Median Error measure (Parkes et al. (2021) produces more interpretable regression, when used in conjunction with conventional error measures. This is achieved by regularising the learnt input-output relationships to the conditional median of the training dataset: the median output value, conditioned on each isolated input variable in turn (Bishop) 1995). For many regression applications the conditional medians are good approximation of the ground truth input-output relationships but as yet it has not been explored as part of an automated approach.
64
65
66
68
69 A challenging regression problem is ship power prediction for vessel using air lubrication to reduce fuel consumption. It is chosen to be used in this study as violates the assumptions in Bishop (1955). 71 where the noise in the output space is non- non-Gaussian and heteroscedastic. In this situation, correctly
70
72 modelling the ground truth and accurate prediction is required but there limited understanding
73 of that ground truth (Parkes et al. (018)) The literature shows that shaft powering of vessel can
74 be predicted with average accuracies of between 1.5-5% error with the use of regression neural network trained with high frequency data from the vessel (Pedersen & Larsen 2009), (Petersen et al. 2012), (Le al. 2020). (Jeon et al. 2018), (Liang al. 2019). All neural network applications to ship power prediction in the literature use a combination of local searches and domain knowledge to identify hyperparameter values. The addition of an air lubrication device increases the complexity of the regression problem, as the system interacts with number of interrelated input variables.
75
76
77
78
80 This paper explores the automation of neural network training to a new problem, with a focus 81 on producing a network which accurately models the ground truth. It compares the ground truth
82 representation of a neural network when genetic algorithm optimises the network's 's hyperparameters
83 to reduce the Mean Fit to Median Error measure and compares it to standard regularization using 11,
84 12 and dropout, and to a network optimised to minimise the Maximum Absolute Error. It is illustrated
85
that neural network regularisation methods (I1, 12 and dropout) can be replaced by the use of the
86 Mean Fit to Median performance measure as an objective the genetic algorithm, reducing the complexity of the search space and producing networks which more consistently model the ground
88 truth.
2

---

2
Neural Networks Parameters
89
Previous applications of neural networks to ship power prediction use between 1 and 3 hidden layers [eiisson al. 2008) (Parkes al. 2019), and between 5 and 300 neurons in each hidden layer (Jeon 92 et al. 2018). To provide sufficiently large search space to allow verification, or otherwise, of these parameters maximum of hidden layers and 1000 neurons in each layer are used. The majority of the literature treats the problem as time- invariant and use feed-forward networks, so no recurrent parameters are optimised. As the optimiser or activation functions are rarely documented in the 96 literature, the state-of-the-ard optimisers and activation functions available in the Keras framework (Chollet et al. 2015) are used in the optimisation, Table T
90
91
93
94
85
97
Table I: Selected Neural Network Hyperparameters Value or set
Hyperparameter
Layers
[1,4]
Neurons in each layer
[11000]]
Epochs
Increasing from 1-20 for increasing generations
Early stopping patience
5
Loss function
Mean Absolute Error
Performance measures
Mean Absolute Relative Error, Maximum Absolute Relative Error,
Mean Fit to Median Error
Optimiser
SGD,, Adam (Kingma & Ba 2014)), Nadam (Dozat 2016).
RMSprop (Hinton et al. 20T2), Adagrad (Duchi et al. 201
Adadelta (Zeleer2012), Adamax (Kingma & Ba 2014)
Activation function 11 & 12 Rates
ReLU, sigmoid, softmax, softplus, softsign, tanh, selu, elu
0, 0,00.0..001.000001.00000001
Dropout
(0,0.9)
Initialiser
Random Normal (μ 0, 0.1)
88
The number of epochs and early stopping procedure are not optimised, as there was a need for predictable compute requirements and allowing the optimisation of these parameters leads unpre- dictable run times. The number of epochs to train each network increases for increasing generation number in the genetic algorithm, from epoch in the first 15 generations to 20 in the final 15. This was also implemented to reduce compute and it was validated that when more than 20 epochs were allowed, that the early stopping, with a patience of 5, stopped the training within 20 epochs for the majority of networks. The loss function is similarly not optimised, the Mean Absolute Error is used, as the conditional medians are closer to the ground truth input-output relationships in these datasets than the conditional means.
99
100
101
102
103
104
105
108
The performance measures, or the genetic algorithm's fitness functions, are the Mean Absolute Relative Error, the Maximum Absolute Relative Error and the Mean Fit to Median Error. Different combinations of these, alongside the use of regularisation parameters in the search space are compared to illustrate the effect of different types of regularisation.
108
109
110
cMLSGA Parameters
Table 2: Selected cMLSGA Hyperparameters
Hyperparameter
Value set
Algorithm Individual Level
HEIA, IBEA
Crossover Type & Rate
SBX & DE, I
Mutation Type & Rate
Polynomial, 0.08
Number of eliminated collectives
Generations between elimination
10
Population size
1000
Generations
300
Proportion elite
10%
3

---

112
In this study cMLSGA is selected as it shows the top performance on a range of evolutionary benchmarking problems (Grudniewski & Sobey
113
and practical problems (Grudniewski & Sobey 2019). Genetic algorithms are increasing used to tune neural network hyperparameters including regularisation parameters for use on new problems (Tin et al. 2004). Many approaches have multiple genetic algorithm objectives, although these all minimise an error measure and a measure of network complexity (Wang et al. 2019) and (Smith & Jin 2014). The use of multiple different performance
114
115
116
117
measures objectives is yet to be explored in the literature.
Table 3: Genetic Algorithm Approaches
Approach
Objective(s)
Network Regularisation
GAI
Mean Absolute Error
11, 12 and dropout
GAii
Mean Absolute Error
11, 12 and dropout
Maximum Absolute Error
GAiii
Mean Fit to Median Error
None
Mean Absolute Error
GAiv
Mean Absolute Error
None
Maximum Absolute Error
119
Four approaches are investigated in this study, summarised in Table for approach (GAi) and (GAii) the genetic algorithm eMLSGA optimises all variables in Table 2 including the 11 and 12 regularisation rate and the dropout rate of the networks. Although it is advised that 12 regularisation and dropout are not used in the same network the genetic algorithms are provided with zero options for all regularisation parameters, identify if one preferable in this scenario.
120
122
123
124
Approach (GAi) is a single objective genetic algorithm optimising the Mean Absolute Error which is compared to multi-objective formulation where the (GAii) approach optimises both Mean Absolute Error and Maximum Absolute Error. For approaches (GAiii) and (GAiv) no network regularisation parameters are optimised: 11, 12 and dropout rates all set permanently to zero They avoid producing networks that have overfitted by the use of two performance metrics as multi-objectives, (GAii)) uses the Mean Fit to Median and Mean Absolute Errors to be minimised and (GAiv) uses the Maximum Absolute and Mean Absolute. All approaches use 40 CPUs with 2.0 GHz Intel Skylake
126
128
129
130
processors and 192 GB of DDR4 memory, and take less than 3 days, this setup may not be feasible for widespread industrial application, although it is suggested it is within reach of some industries.
132
133
4
Data
134
The data used in this study are from a large vessel equipped with the Silverstream@ Air Lubrication System. The air lubrication system works through use of fluid sheering to create an air microbubble carpet directly captured within the boundary layer on the ship hull bottom. The bubble carpet reduces the frictional resistance thereby increasing the speed and reducing the shaft power. Compressors provide a constant supply of air the hull bottom to maintain a uniform bubble carpet operated at the optimal compressor power that maximises the energy balance. The study is performed on both system on and system off datasets, however for brevity only results for system off are presented as they show similar performance. Thhis prediction is required for baseline determination of how the system is working, but the relationships between the power, weather, ocean and operating conditions are complex and difficult to model.
135
136
137
138
139
140
142
144
The variables considered in this study are the shaft power, speed through water, relative wind speed and direction, draught and trim, with shaft power the target variable. These are selected based on a detailed study into variable selection for shaft power prediction (Parkes et al. 2019). The speed through water is selected over the speed over ground, for use as an input variable, as it is more hydrodymanically relevant and its accuracy is validated by comparison
145
146
147
148
the speed over ground. The dataset is cleaned by removing rows with missing or non-physical values and all datapoints below 0.05 normalised shaft power are removed. The dataset is split into two using the air lubrication system status: system on and system off, where system on is defined as air lubrication system power greater than zero The system on dataset contains 352,690 datapoints and system off contains 237,962. The data is split into training, testing and validation sets of 70%. 15% and 15% respectively. Each
149
150
151
152
The code for cMLSGA is available at
www.bbbbbbcket.oo/tt.......

---

Number of datapoints
41 to 2,961
0.8
2.961 to 5.881
5,881 to 8,801
Nomalised Shaft Power
8,801 to 11,721
0.6
11.721 14.842
14,642
20,482 23,402
0.4
23,402 26,323
0.2
0.0
21
Speed Through Water (knots)
Figure 1: The distribution of the observed shaft powers for half knot bins of speed through the water for dataset where the system is off. In the box and whisker plots the boxes contain 50% of the distribution and the whiskers extend to the datum which is at 1.5 times the interquartile range.
154 network in the genetic algorithm trains on a randomly sampled 35,000 datapoints from the training
set and uses randomly sampled sets of size 7,500 from validation and testing sets for validation during training, and testing to produce the fitness of the network for the genetic algorithm. The errors stated in the paper are from networks on the Pareto fronts of each approach, which are validated on the full testing set.
156
158
159
The datasets contain large regions of sparse data in all input variable domains, this is exemplified by the ship speed domain where each half-kno interval below 16 knots contains less than 0.8% of the data, which accounts for more than half the speed domain, Figure [ In addition, the boxplot ranges and outliers show high heteroscedicity with idiosyneratic noise caused by situations where the angle of the propeller blades is varied to achieve the required speed. This highlights the complexity developing models the powering of this vessel, as the dataset also contains the effects from other latent variables, such as piloting behaviour and route taken.
160
162
184
165
Optimisation including regularisation parameters: (GAi) and (GAii)
166
Previous studies predicting ship powering using neural networks report that 11, 12 and elastic net increase both test set and off-test set errors and that optimal values for both 11 and 12 are zero. Therefore the genetic algorithm setup is biased towards low and zero values of regularisation rates by using a set of exponentially decreasing values and an explicit zero option.
188
170
The single objective (GAi) fails to identify that zero regularisation rates produce the lowest errors, favouring networks with the highest possible rate 12 (0.01), Figure 2b (GAi) produces networks with the highest Mean Absolute Relative Errors of all the approaches, (5.19 19 + 0.00) % from Figure In contrast, (GAii) favours lower and rates
172
174
or 0.00001, Figures 2c and 2d which results networks with the lowest Mean Absolute Relative Errors of all four approaches, average, with value of (2.87 + 0.45)%, shown in Figure 4a This is around 0.5% higher than the lowest documented error for ship power prediction.
4a
176
178
posited that the high enror for the single objective problem directly related to the use of large 12 regularisation rate, as noted in previous studies for ship power prediction. It is possible that the use multi-objective search algorithm for a single-objective problem means that optimal hyperparameters can't be found, resulting in large errors. The implementation also requires restrictions in the number of epochs used for training in the initial generations, it is possible this biases (GAi) towards certain size networks, where higher 12 rates preferable. This hypothesis is
179
180
182
5

---

regularisation rate
12 reqularisation rate
reqularisation rate
12 reqularisation rate
0.01
0.01
0.01
0.01
0.001
0.001
0.001
0.001
0.0001
0.0001
0.0001
0.0001
1o-05
1o-05
0.0
0.0
0.0
0.0
(a) (GAi)
(b) (GAi)
(c) (GAti)
(d) (GAii)
Figure 2: Distribution of regularisation rates for networks in the last 15 generations of (GAi) cMISGA with multi-objectivee of minimising Maximum and Mean Absolute Error for (a) 11 and (b) 12 and (GAli) eMLSGA with the single objective of minimising Mean Absolute Error for (c) and (d) 12
1000
Number of neurons in each layer
0.8
800
Drogout Rate
0.6
600
0.4
400
200
GAlii
GAiv
GAi Genetic Algorithm Approach
GAii
(1 layer) (1 layer)
(3 layers)
(3 layers)
Genetic Algorithm Approach (modal number of layers]
(a)
(b)
Figure 3: (a) Dropout rate for networks in the last 15 generations of cMLSGA with (GAi) the single objective of minimising Mean Absolute Error and (GAii) multi-objectives of minimising Maximum and Mean Absolute Error and (b) the number of neurons in each layer for networks in the last 15 generations of cMLSGA with (GAi), (GAii) and (GAiv).
184
supported by the fact that 74.4% of networks the first 15 generations of (GAi) have hidden layer, and that over 99.8% of the networks in the final 15 generations have hidden layer, with 880 + 17 neurons in this layer, Figure 3b This is significantly more neurons than those in the hidden layer of networks in the final 15 generations of (GAii) which range from 3-952 with a median value of 709, Figure 36 The added objective of minimising Maximum Absolute Error in (GAii) may cause these slightly smaller networks to be more attractive they are in sense regularised by their size, as they have reduced modelling flcxibility therefore are less likely to overfit and produce high Maximum Absolute Errors.
185
196
188
190
192
Another explanation for the difference in 12 rates chosen by (GAi) and (GAii) is the equivalence of 12 and dropout. Since 12 and dropout are equivalent up to Fisher transformation, their use in conjunction not recommended. The evidence for this is that (GAi) favours the highest 12 rate and has a median dropout rate in the final 15 generations of 0.1 whereas (GAii) favours the zero 12 rate and has a median dropout rate of 0.624, Figure 3a This illustrates that the genetic algorithms will chose either 12 or dropout to minimise the Mean Absolute Relative Error. The I1 rates also support this hypothesis, as chosen rates for 11 regularisation the final generations are more comparable
193
194
195
196
198
for (GAi) and (GAii).

---

Optimisation using multiple performance measures: (GAili) and (GAiv)
200
201
For approaches (GAili) and (GAiv) all neural network regularisation parameters are set to zero. The regularisation is performed by minimising different network performance measures, the Mean Absolute and Mean Fit to Median for (GAiii), and the Mean Absolute and Maximum Absolute for (GAiv). The trade-off between the two objectives produces regularised neural networks, without explicitly changing the architecture or loss function. The Mean Fit to Median is chosen as it indicates
203
204
205
208
how close the relationships modelled by a network are to the conditional averages of the dataset, in
207
many regression examples this is akin to the ground truth input-output relationships (Parkes et al. 2021). The Maximum Absolute chosen as for many industrial applications of machine learning the maximum prediction error is more pertinent than the mean error. The Mcan Absolute Error is used
208
209
210
instead of the Mean Squared Error both approaches, as the conditional medians are closer to the ground truth input-output relationships in these datasets than the conditional means.
Differently shaped networks are favoured by (GAiii) and (GAiv), compared to (GAi) and (GAii), focusing on networks with 3 hidden layers and on average less than 400 neurons in each layer. Figure 3b These networks have 51 times the number of connections than the networks chosen in (GAi) and (GAit), Apart from (GAi), (GAiii) has the most consistently sized networks in the final 15 generations, with an interquartile range of 46 neurons, compared (GAiv) which have an interquartile range of 131 neurons. It is suggested that as the Mean Fit to Median Error biases networks towards specific input-output relationships, there a smaller range of potential network architectures which habitually model these relationships. Whereas networks which minimise the Maximum Absolute Error are less restricted and can model a wider range of input and output relationships.
213
214
215
216
217
218
220
Maximum Relalive Absolutte Error (56)
65.0
Mean Relative Absoluute Error (%6
5.0
4.5
60.0
4.0
57.5
3.5
52.5
50.0
2.5
47.5
GA
GAii
GAiii
GAIN
GAi
GAii
GAii
GAiv
Genetic Algorithm Approach
Genetic Algorithm Approach
(b)
Figure 4: Mean Relative Absolute Error (a) and Maximum Absolute Error (b) from cMLSGA with (GAi) the single objective of minimising Mean Absolute Error and (GAii) objei-vbss of minimising Maximum and Mean Absolute Error, both optimising the parameters for 11, 12 regularisation and dropout in the networks, and (GAiii) and (GAiv) which do not use network regularisation but minimise Mean Fit to Median and Maximum Absolute Error respectively, alongside Mean Absolute Error
221
The Mean Absolute Relative Errors from networks in the Pareto fronts are (2.97 - 0.25) % for (GAiii) and (3.10 + 0.28)% for (GAiv). It is expected that (GAiv) would produce higher Mean Absolute Relative Errors as discussed above, minimising the Maximum Absolute Error should bias predictions towards the midpoint of the conditional output distributions, whereas minimising the Mean Absolute Error should bias predictions towards the median of these distributions. As it is established that noise
222
223
224
225
226
in the output distribution is non- non-Gaussian, Figure [] these values will not align so some sacrifice in Mean Absolute Error is expected from (GAiv). Both (GAiii) and (GAiv) produce comparable Maximum Absolute Errors, of (49.5 1.1) % and (51.9 + 4.5)%. is suggested that this is because, although the conditional median output value and conditional midpoint output value do not align for the majority of the input domain, they are sufficiently close to produce comparable Maximum
227
228
230
Absolute Errors.
7

---

232
Across all four approaches, the genetic algorithm producing networks with the highest Mean Absolute Error is the approach which does not provide extra weighting to sparse areas of data. The approaches minimising Maximum Absolute Error are implicitly biased away from networks which predict the majority of the testing datapoints correctly, but predict one datapoint poorly, favouring networks which predict all testing datapoints to moderate degree of error. Approach (GAiii) more explicitly weights prediction in sparse areas of data by favouring networks which model the conditional median of the dataset across all input domains, irrespective of the quantity of data across each input domain. The regression problem
233
235
237
239
ship power prediction is chosen in part because of it's irregular data distribution; more than 9% of the dataset lies in less than 0.5 knot interval of ship speed. Figure [] This provides an explanation for the high testing errors from (GAi), where only the Mean Absolute Error is minimised, there is little incentive for the genetic algorithm produce networks which generalise across the full range of the input domain well.
241
243
7
Comparison of the interpretability
245
To asses the interpretability of the networks selected by the four different approaches the learnt relationship between an input, the ship speed, and the output, shaft power, for the networks in the Pareto front of each approach are visualised, Figure [ These are extracted with the following procedure: set all but one input variable be constant the mode; cycle the remaining variable from its minimum to its maximum recorded values with 150 points evenly spaced along the domain and run the new dataset through the trained network.
248
247
248
250
Range of predictions GAI (Mean Absolute single-dkjective)
Range predictions
GAIl (Maximum Absolute objective with network regulansation
Noralied Shelt Power
Noraal lsed Shaft Power
0.6
0.6
0.4
0.4
0.0
20
0.0
20
Speed Through Water (knots)
Through Water (knots)
(a) (GAi)
(b) (GAii)
Range predictions
Range predictions
GAii (fit Median objectivel
GAVv (Maximum Absolute objective)
Normalised Shaft Power
0.8
Normalised Stafft Power
0.8
0.6
0.6
0.0
10
20
0.0
10
20
Speed Through Water (knots)
Speed Through Water (krots)
(c) (GAili)
(d) (GAiv)
Figure 5: The learnt speed- speed-power curves from 5 networks on the Pareto fronts of (GAii), (GAiii), (GAiv) and the 5 networks producing lowest Mean Absolute Relative Error from (GAi).
251 The approach which produces the most consistent speed- power relationships is (GAi), with an average variation of 1.8% Figure 5a However, the relationship modelled by the 5 networks with the lowest 253 Mean Absolute Relative Error in (GAi) all approximate a piece-wise wise linear relationship which clearly underfits the dataset in Figure T The expected trend between ship speed through the water and shaft
2 Average variation in just the speed-powe curves are discussed in this section, but it is verified that all input-power curves follow the same trends with variation around 0.5% across input variables.

---

255
power is cubic polynomial, therefore as well as producing the highest Mean Absolute Relative Errors, networks chosen by (GAi) model the ground truth input- relationships the worst out of the four approaches. Both (GAii) and (GAīv) produce 5 fairly consistent speed-pows curves, with average variations of 5.9% and 10% respectively, Figures 55 and 5d Both approaches approximate smooth polynomial curves, although the degrees of the polynomials might differ, as multiple curves intersect at various points along the speed axis. The spread of learnt relationships is greater at the highest and lowest speeds for (GAii), with a decrease in spread for speeds of around 15 knots, where many of the curves intersect. The curves from (GAiv) show equal spread across the speed domain.
256
257
258
259
260
262
263
The approach with both accurate and consistent learnt speed- pooeer curves is (GAiii,, with limited intersections of curves and an average spread of 3.0%. It suggested that the reason using the Mean Fit to Median Error as an objective in multi-objective genetic algorithm produces more interpretable results, or more consistent learnt relationships, is because instead of encouraging the networks to model more simple relationships. It encourages the networks to model the conditional median functions of the dataset, supported by the increase network connections. Whereas the other approaches leave room for networks to fail to model the conditional averages, especially in irregularly distributed and non-normally distributed datasets. The Mean Absolute Error values from networks selected by (GAiii) are on average 0.1 1% higher than those from (GAii), and the Maximum Absolute Error values are 1.6% higher.
264
285
266
267
268
270
272
273 A limitation of the approach is that the Fit to Median Error measure will likely perform best at improving interpretability on datasets which violate the assumptions in Bishop (1995); the ship powering example is chosen to illustrate this as it provides a clearly heteroscedastic dataset. For applications where noise profiles are Gaussian, and there no effect from latent or interrelated input variables, the Fit to Median Error will not improve interpretability, but will perform the same as conventional Minkowski-r metries, either Mean Squared or Mean Absolute Error depending on the convexity of input-output relationships.
274
275
276
277
278
280
Interestingly. the approach producing the lowest Mean Absolute and Maximum Absolute Errors
281
does not model the ground truth the most accurately. This creates a potential for negative societal impacts, as the standard performance metrics for regression neural networks do not provide a full
282
283
picture of performance or expected behaviour. Interpretability of trained methods is essential for safe application of machine learning in the real world, especially when automated methods are used to replace experienced professionals. (GAii) demonstrates the same accuracy of approach as those with standard network regularisation, but with a better fit to the ground truth. This approach bypasses the need to use, and therefore to optimise the parameters of the regularisation methods. evolutionary computation is already being used to optimise network parameters, then compute is saved by removing the network regularisation parameters 11, 12 and dropout. (GAiii) completed 300 generations in 46hours whereas (GAii) required 12 hours more computation to complete 300 generations.
284
285
286
286
290
8 Conclusion
292
293
Interpretable and accurate methods are required for widespread application of machine learning to real-world regression problems. To automate the training of neural networks so that the result is interpretable, three different genetic algorithm approaches are compared: one to minimise the Maximum Absolute Error of the networks, which includes standard regularization using 11, 12 and
294
295
296
297
dropout; and two which do not use any network regularisation, one minimising the Mean Fit to Median and one to minimise the Maximum Absolute Error. The results show that all three approaches give similar Mean Absolute Errors from networks on their Pareto fronts, from 2.9% for the approach with regularisation to 3.1% for the approach minimising Maximum Absolute Error. However, the
298
299
300
Mean Fit to the Median approach shows a considerably better interpretability. or fit to the ground truth, with a spread in predicted onput-output curves of 3% compared to a spread of 6% for the approach using regularisation and 10% when minimising the Maximum Absolute Error.
302
References
304
Battaglia, P. W., Hamrick, J. B., Bapst, V., Sanchez- A., Zambaldi, V., Malinowski, M., Tacchetti, A., Raposo, D., Santoro, A., Faulkner, R., Gulcehre, C., Song, F., Ballard, A., Gilmer, J. Dahl, G., Vaswani, A., Allen, K., Nash, C., Langston, V., Dyer, C., Heess, N., Wierstra, D.,
306
9

---

308
Kohli, P., Botvinick, M., Vinyals, O., Li, Y. & Pascanu, R. (2018), 'Relational inductive biases, deep learning, and graph networks'.
309
310
Bishop, C. (1995), Neural Networks for Pattern Recognition, Oxford University Press, chapter pp. 194-225.
312
Chollet, F. et al. (2015), 'Keras',
Dozat, T. (2016), 'Incorporating nesterov momentum into adam', International Conference
314
Learning Representations 2016.
315
Duchi, J., Hazan, E. & Singer, Y. (2011), 'Adaptive subgradient methods for online learning and stochastic optimization' Journal of Machine Learning Research 12(61), 2121-2159.
Grudniewski, P. A. & Sobey, A. J. (2019), Do general genetic algorithms provide benefits when solving real problems?, in '2019 IEEE Congress on Evolutionary Computation (CEC)', IEEE, pp. 1822 1829.
318
Grudniewski, P. A. & Sobey, A. (2021), 'cMLSGA: co-evolutionary multi-level selection genetic algorithm for multi-objective optimization'.
321
322
Hanson, S. & Burr, D. (1987), Minkowski-r paccppropagatiin: learning in connectionist models with
323
non-euclidian error signals., in 'Neural Information Processing Systems (NIPS 1987)'
Hinton, G., Srivastava, N. & Swersky, K. (2012), 'Lecture 6a:overview of mini-batch gradient
descent'.
URL: http://www.cs.toronto.oo
324
Jeon, M., Noh, Y., Shin, Y., Lim, O., Lee, Cho, D. (2018), 'Prediction of ship fuel consumption by using artificial neural network'. Journal of Mechanical Science and Technology 32((2), 5785- 5796.
325
326
327
Jin, Y., Okabe, T. Sendhoff, (2004), Neural network regularization and ensembling using multi-objective evolutionary algorithms, in 'Proceedings of the 2004 Congress on Evolutionary Computation (IEEE Cat. No.04TH8753)', Vol. 1.
329
330
Kingma, D. & Ba, J. (2014), 'Adam: A method for stochastic optimization', arXiv preprint a1Xiv:1412.6980.
Kumar, P., Batra, S. & Raman, B. (2021), 'Deep neural network hyper-parameter tuning through
333
twofold genetic approach', Soft Computing pp. 1-25.
334
Le, L.,, Lee, G., Park, K. & Kim, H. (2020), 'Neural network-based fuel consumption estimation for container ships in korea', Maritime Policy & Management pp- 1-18.
Leifsson, I.,, Sævarsdóttir, H., Sigurösson, S. & Vésteinsson, A. (2008), 'Grey-box modeling of an ocean vessel for operational optimization', Simulation Modelling Practice and Theory 16(8), 923- 932.
337
I Liang, Q., Tvete, H. A. & Brinks, H. W. (2019), Prediction of vessel propulsion power using machine learning on ais data, ship performance measurements and weather data, in 'Journal of Physics: Conference Series', Vol. 1357, p. 012038.
340
341
342
Luketina, J., Berglund, M.. Greff, & Raiko, (2016), 'Scalable gradient-based tuning of continu- ous regularization hyperparameters'.
343
344
Nowlan, S. J. & Hinton, G. E. (1992), 'Simplifying neural networks by soft weight sharing', Neural
345
Computation.
346
Parkes, A. I., Savasta, D., Sobey, A. Hudson, D. A. (2019), Efficient vessel power prediction in operational conditions using machine learning., in 'Practical Design of Ships and Other Floating Stractures(PRADDS) September 2019, Yokohama, Japan'.
347
348
349
Parkes, A. I., Sobey, A. J. Hudson, D. A. (2018), Physics-based shaft power prediction for large
merchant ships using neural networks', Ocean Engineering 166, 92-104.
10

---

351
Parkes, A. I., Sobey. A. J. & Hudson, D. A. (2021), 'Towards error measures which influence learners inductive bias the ground truth'.
352
353
Pedersen, B. & Larsen, J. (2009), Prediction of full-scale propulsion power using artificial neural networks, in 'Proceedings of the 8th international conference on computer and IT applications in the maritime industries (COMPIT'O9) 09),, Budapest, Hungary May'. pp. 12-12.
354
355
356
Petersen, P., Jacobsen, D. J. & Winther, O. (2012), 'Statistical modelling for ship propulsion efficiency', Journal of marine science and technology 17(1), 30-39.
357
358
Smith, C. & Jin, Y. (2014), Evvoltionary multi-objective generation of recurrent neural network ensembles for time series prediction', Neurocomputing 143, 302-311.
359
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. (2014), 'Dropout: a simple way to prevent neural networks from overfitting', The Journal of Machine Learning Research 15(1), 1929-1958.
361
Tani, L., Rand, D., Veelken, C. & Kadastik, M. (2021), 'Evolutionary algorithms for hyperparameter optimization in machine learning for application in high energy physics', The European Physical Journal C (1)). 1-9.
364
Wager, S., Wang, S. & Liang, P. (2013), 'Dropout training as adaptive regularization', arXiv preprint 0rXiv:1307.1493
387
368
Wang, B., Sun, Y., Xue, & Zhang. M. (2019), Evolving deep neural networks by multi-objective particle swarm optimization for image classification, in 'Proceedings of the Genetic and Evolution- ary Computation Conference', GECCO '19, p. 490-498.
370
371
Willard, J., Jia, X., Xu, S., Steinbach, M. & Kumar, V. (2020), 'Integrating physics-based modeling with machine learning: a survey'.
372
373
Yang, S., Tian, Y., He, C., Zhang, X., Tan, K. C. & Jin, Y. (2021). "A muddeett-guider evolutionary approach to training deep neural networks', IEEE Transactions on Neural Networks and Learning
374
375
Systems pp. 1-15.
376
Zeiler, M. D. (2012), 'Adadelta: an adaptive learning rate method'.
Checklist
377
378
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A]. . You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing brief inline description. For example:
380
382
Did you include the license to the code and datasets? [Yes] See Section ??.
383
Did you include the license the code and datasets?? [No] The code and the data are proprictary.
384
385
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the
387
Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
388
389
1. For all authors...
390
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The scope of the paper (regression problem automation) made clear in the abstract. The claims of improved interpretability using the Fit to Median Error measure are reflected in Figure 5 as well as the body of text.
381
382
383
(b) Did you describe the limitations of your work? [Yes] The limitations of the Fit to Median are discussed in Section and the limitations relating to compute are discussed in Section
385
11

---

397
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See S
396
399
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
401
2. If you are including theoretical results...
402
(a) Did you state the full set of assumptions of all theoretical results? [No] Citation of (Bishop [995) is included instead of explicit statement of all assumptions, although the most pertinent assumptions are discussed in Section |
403
404
405
(b) Did you include complete proofs of all theoretical results? [N/A] No theorems are posited, the paper explores practical comparison of common methods.
406
407
3. If you ran experiments..
(a) Did you include the code. data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] Instruc- tions needed to replicate the results on different dataset are provided in the main text; Sections 3 and [ specify the expermental setup. The code is based on the cMLSGA algorithm, available https: / /wwww//wwwwwoo bitbucket. wwwpicket.rg//*..ee org/******* (redacted for anonymity) which is provided in Section 3 The data is proprietary so is not pro- vided.
409
411
413
414
415
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See 3ections
416
(c) Did you report error bars (e.g., with respect the random seed after running ex- periments multiple times)? [Yes] All results are reported to plus/minus the standard deviation, and all figures show multiple runs with respect to either random seed or an evolutionary method.
418
420
421
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 3
422
428
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
424
(a) If your work uses existing assets, did you cite the creators? [Yes] The creators of the genetic algorithim cMLSGA are cited in Section 3 and the data is acknowledged to
425
belong to Silverstream Technologies Ltd in Section 4
(b) Did you mention the license of the assets? General Public License. The data is not licensed.
[Yes] The code is licensed under the GNU
428
429
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [No] The work is performed in collaboration with Silverstream Techonologies Ltd.
431
433
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] The shaft power variable is normalised to remove any possibility of identifying the vessel in question.
434
435
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
438
439
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
441
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
442
12


Reviews:
[{"review_id": "k2NMOSGhKKr", "reviewer": "Reviewer_5hCU", "summary": "The paper claims that building ML models through automation can become a black-box approach that results in lower errors but also lower interpretability. Also, minimizing traditional error measures cannot guarantee an accurate approximation of the ground truth. The paper studies using fit to Median Error measure to produce better model with regard to interpretability of models. \n\nThe paper uses ship power prediction problem that aims to reduce fuel consumption, a challenging regression problem, as an application where fit to Median Error measure is used. The paper compares four different genetic algorithms settings: minimizing the Maximum Absolute Error of the networks, minimizing the Maximum Absolute Error and minimizing the Mean Fit to Median. \n\nThe experiment results demonstrate that all four settings produced similar Mean Absolute Errors from networks on their Pareto fronts. But the Mean Fit to the Median setting obtained much better interpretability (fit to the ground truth). \n", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe paper is mostly about applying existing techniques to an application. The GA, fully-connected neural network, and different measures are not new ideas. The paper cites many other works.\n\nThe paper makes quite many claims. Some of them are quite difficult to comprehend, For instance, in page 8, the paragraph states \"Across all four approaches, the genetic algorithm producing networks with the highest Mean Absolute Error is the approach which does not provide extra weighting to sparse areas of data.\" What is \"extra weighting\"? The conclusion that GAiii is the approach with both accurate and consistent learnt speed-power curves is a bit stretchy. \n\n\nIt is interesting to read the paper. However, it can be better written. For example, the section title of \"3 cMLSGA Parameters\" can be \"Genetic Algorithm Software cMLSGA Parameter Setting\"\n\nThe paper uses an application to prove that fit to Median Error measure produces more interpretable model. I found it is not too convincing. The title of paper is about automating interpretable ML through a comparison of loss functions to regularizers. Based on the methodology and experiment results, it is not clear to me that the goal is achieved. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 "}, "extra_text": {}}, {"review_id": "R69VREGT0_X", "reviewer": "Reviewer_vvdn", "summary": "This paper studies the interpretability of neural networks. Specifically, the authors use four approaches with different objectives and regularizations and use cMLSGA, a genetic optimization method to optimize these problems. Eventually, the authors conduct experiments to validate that Fit to Median Error measure possesses better interpretability compared to L1, L2 and dropout regularizations since it regularizes the learnt input-output relationships to the conditional median. ", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a pretty empirical work. The structure of this paper is not conventional but still retains its own logic.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors want to validate that the Fit to Median Error is more interpretable by comparing its performance to other different measures since the authors think that more consistent or accurate models have more interpretability. This logic may not be valid since an interpretability model should tell us why this model makes this prediction. Lipton [1] discussed that interpretability can be achieved if we human can simulate the decision-making process of the model in a limited time. Kim et al. [2] stated that interpretability is the degree to which a human can consistently predict the model’s result. There are also other definitions of interpretability such as interpretability is the degree to which a human can understand the cause of a decision [3]. Theses three are all consistent with what I mentioned, i.e., “an interpretable model should tell us why this model makes this prediction”. For example, a decision tree will tell you why it makes that decision since the final decision is based on the splitting of features on the node of the tree. Merely making the prediction more accurate is not enough for interpretability.\n\nIn addition, I think only one dataset may not be sufficient to convince the readers. The authors should also provide more details of the model and method by such as providing formulas and structure of the model.\n\n[1] Z. C. Lipton, “The mythos of model interpretability,” CoRR, vol. abs/1606.03490, 2016.\n[2] B. Kim, O. Koyejo, and R. Khanna, “Examples are not enough, learn to criticize! criticism for interpretability,” in NIPS, 2016, pp. 2280–2288.\n[3] T. Miller, “Explanation in artificial intelligence: Insights from the social sciences,” Artificial Intelligence, vol. 267, pp. 1–38, 2019.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Nxc-ApVSl_5", "reviewer": "Reviewer_vAik", "summary": "In this paper the authors utilize a new Fit to Median Error measure for a practical machine learning problem of predicting the shaft powering of a vessel using genetic algorithms. They complete an empirical study using different combinations of loss functions and genetic algorithms to demonstrate that the choice of Fit to Median error improves both model accuracy and interpretability. \n", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents a complete empirical study. That said, the reviewer considers the paper lack of novelty and is discussing a very specific applied problem that might not be of the general interests of NeurIPS audience. \n\nThe major reasons that lead to the reviewer's rating are as follows.\n\n1. The paper does not define a clear research question. \n\n2. The paper lacks proper related research. Two key concepts covered in the paper are Fit to Median Error and Genetic Algorithms (GAi-Gv) whereas neither of them is novel or clearly elaborated. Therefore it is hard to justify the use of this new error measure, hence the novelty of the paper. \n\n3. The paper does not have enough details to demonstrate how the conclusions or techniques in the paper can be generalized to other machine learning questions. '\n\n====\nPost rebuttal:\n\nThe reviewer would like to thank the authors for their response, and would like to keep the initial rating.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper makes a mistake about interpretable machine learning, which is quite often on this topic, in that the interpretation involves many aspects other than accuracy. It requires a major revision of the paper to better reflect the content, thus cannot be accepted in the current form.
Decision: Reject