# Hyperparameter Tuning of Machine Learning Algorithms Using Response Surface Methodology: A Case Study of ANN, SVM, and DBN

Authors: Warut Pannakkong; Kwanluck Thiwa-Anont; Kasidit Singthong; Parthana Parthanadee; Jirachai Buddhakulsomsiri
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

This study applies response surface methodology (RSM) to the hyperparameter fine-tuning of three machine learning (ML) algorithms: artificial neural network (ANN), support vector machine (SVM), and deep belief network (DBN). The purpose is to demonstrate RSM effectiveness in maintaining ML algorithm performance while reducing the number of runs required to reach effective hyperparameter settings in comparison with the commonly used grid search (GS). The ML algorithms are applied to a case study dataset from a food producer in Thailand. The objective is to predict a raw material quality measured on a numerical scale.K-fold cross-validation is performed to ensure that the ML algorithm performance is robust to the data partitioning process in the training, validation, and testing sets. The mean absolute error (MAE) of the validation set is used as the prediction accuracy measurement. The reliability of the hyperparameter values from GS and RSM is evaluated using confirmation runs. Statistical analysis shows that (1) the prediction accuracy of the three ML algorithms tuned by GS and RSM is similar, (2) hyperparameter settings from GS are 80% reliable for ANN and DBN, and settings from RSM are 90% and 100% reliable for ANN and DBN, respectively, and (3) savings in the number of runs required by RSM over GS are 97.79%, 97.81%, and 80.69% for ANN, SVM, and DBN, respectively.


---

## Body

## 1. Introduction

Nowadays, machine learning (ML) algorithms have become an important part of various industries. ML algorithms provide a significant capability to perform or facilitate various tasks. In manufacturing, a company can gain some benefits from ML in terms of performance and efficiency in a wide variety of aspects. For example, ML can be used to reduce the labor cost, human error, and number of product defects or to increase the production rate. Another advantage is that ML algorithms can handle a large amount of input data for model training [1]. The main reason for ML algorithms’ growing use is that ML algorithms can improve productivity and efficiency by automating them in the usage environment [1, 2]. Also, ML algorithms can learn from previous experience by discovering patterns in existing data and using those patterns to develop and/or improve their knowledge over time [3]. These benefits of ML algorithms can lead to business revenue and growth.There are various tasks that ML algorithms can perform. Among them, the most common tasks are classification, clustering, and prediction [4]. Classification is supervised learning that involves predicting or labeling a data class. Training an ML algorithm for classification requires an input dataset with predefined classes. Clustering, an unsupervised machine learning task, involves grouping data into an unknown number of clusters according to some common aspects. Prediction involves estimating some quantitative (numerical) label of future observations by using a mathematical or empirical model. Due to the highly competitive environment of today's business, an accurate prediction model is needed to support the decision-making process, which impacts the business performance. Prediction is used in a wide range of applications today. To select a prediction model, its accuracy is one of the most important criteria for a modeler. For ML algorithms, an important issue in constructing effective prediction models is that the hyperparameters of the algorithms must be appropriately set [5, 6].Hyperparameter settings are a prespecified set of decisions for ML models that directly affect the training process and the prediction result, which reflects an ML’s model performance [6, 7]. Model training is a learning process so that a model can recognize patterns in the training data and predict the output of new data based on these patterns. In addition to hyperparameter setting, model architecture, which indicates the complexity of the model, would also directly impact the execution time to train and test a model. Due to their impacts on the model performance and that the best set of values is unknown, hyperparameter setting has become an important and challenging problem in ML algorithm usage. In the literature, there are several methods for tuning hyperparameters. These methods are briefly discussed below.Every hyperparameter has a wide range of values for a modeler to select. Without using an efficient tuning strategy, simple approaches are to use the default hyperparameter values suggested by an ML algorithm developer or the recommended values from other research studies in the literature. Also, a manual search can be performed, or values based on the modeler’s experience can be used [5, 8]. While using default values and suggested values from previous studies require little effort for hyperparameter tuning, these values may not perform well on different sets of input data [5]. This is because every ML algorithm, when it is applied to a specific dataset, has its own best set of hyperparameter values that may change upon different or updated input data [6, 7, 9, 10]. Therefore, most researchers state that ML algorithms are more effective if their hyperparameters have been tuned [11].In addition, using values based on a modeler’s expertise requires the modeler to have some practical experience in using particular ML algorithms and that the modeler possesses enough knowledge of the application in consideration. In most cases, these requirements are difficult to fulfill. Therefore, efficient and systematic step-by-step tuning strategies that even an inexperienced modeler can follow are clearly more desirable than depending on the modeler’s experience.A sophisticated approach is to find optimal hyperparameter settings for the learning process of the ML algorithms using some optimization techniques. This approach treats hyperparameter setting as an optimization problem, where the objective function is to maximize the ML algorithm performance, i.e., minimizing the errors from ML algorithms, while treating hyperparameters as decision variables. Examples of these methods are genetic algorithm (GA) [11–16], particle swarm optimization (PSO) [17–19], Bayesian optimization [18, 20–22]. Although this approach can yield high ML algorithm performance, such optimization or search algorithms can be difficult to use and may not be available to inexperienced modelers.Many recent studies propose various techniques to find the set of optimal hyperparameters. These include grid search (GS) [23], random search (RS) [8], or Latin hypercube sampling. Among them, GS is simple, easy to use, and requires little experience from the modeler [8, 11, 12]. With GS, all possible combinations of hyperparameter values are tested to find the best settings based on the specified lower and upper bounds of each hyperparameter, along with a specified step size that forms the hyperparameter space [11].Since GS executes all possible combinations, many research articles regard this method as an exhaustive search. A weakness of GS is that the combination of hyperparameters increases multiplicatively as the number of hyperparameters increases. This makes GS extremely time-consuming and requires a high computational cost [11–13, 19, 24, 25]. In most situations with limited computational resources, GS may not be an appropriate technique for hyperparameter setting because of its computational requirements.In addition to the inefficiency of GS, another issue of using GS is that for some ML algorithms, e.g., artificial neural network (ANN) and deep belief network (DBN), rerunning the algorithms with the same hyperparameter settings yields different prediction results. This implies that the best hyperparameter settings found from performing GS one time are likely different from the best hyperparameter settings from another round of GS. This uncertainty is caused by the nature of the ML algorithms. In this study, we refer to this issue as the reliability of hyperparameter setting. To obtain reliable hyperparameter settings, some studies implementk-fold cross-validation as a means to protect against randomness in the ML training process. Moreover, the k-fold cross-validation can also avoid the bias in data separation (i.e., training, validation, and test sets).An alternative to GS is to perform a random search (RS), which randomly selects the values of hyperparameters [8, 18, 19]. Based on the principle that among a large number of hyperparameters, there are only a few important tunable ones, and RS has been shown to perform better than GS by testing some ML algorithms on some datasets [8]. However, applying this finding to other ML algorithms, applications, and datasets may be speculative.To overcome the inefficiency of GS, designs of experiment (DOE) techniques have been proposed to find the optimal hyperparameter values of ML algorithms [7, 26, 27]. DOE is a mathematical and statistical approach, which is used to evaluate the effects of multiple experimental factors at the same time. DOE involves planning a series of experiments for hyperparameter testing, where each experiment is a set of experimental runs at different values of hyperparameters that should be tested together. After the experiments are run, a statistical analysis of experimental data is performed to estimate the effects of the hyperparameters on the performance of the ML algorithms. In other words, an empirical model is constructed that relates the performance of ML algorithms, e.g., prediction errors (as response variable) to hyperparameters (as predictors of ML performance). Analyzing the results from an experiment can identify significant factors and their interactions, leading to subsequent experiments that should be run [28, 29]. Some important designs include factorial experiments for a small number of hyperparameters, fractional factorial experiments for a larger number of hyperparameters (up to 15), and response surface methodology (RSM) for numerical hyperparameters [7]. RSM is an efficient technique in DOE that can be used for optimizing the values of hyperparameters [30]. By using RSM, optimal (or near-optimal) settings can be found with a small number of runs [31]. This is because RSM can provide information on an effective search path that can move toward the optimal region [32].In this study, we propose using RSM withk-fold cross-validation to search for optimal (or near-optimal) hyperparameters of ML algorithms. We hypothesize that RSM is similar to GS in terms of ML performance and can give reliable hyperparameter settings. It is also more efficient in terms of the number of experimental runs. Both GS and RSM are performed to find the set of optimal hyperparameters of the three ML prediction models, ANN, support vector machine (SVM), and DBN, using a dataset obtained from an industrial user. The ML model performance, including prediction accuracy and the number of runs, is then compared. The contributions of our study are that we demonstrate from extensive testing the following advantages of using RSM: (1) the approach is applicable to important ML algorithms with numerical hyperparameters, and their infrastructure is specified by the number of neurons in the hidden layer, which is also a numerical factor, (2) it is a step-by-step procedure that does not depend on the ML modeler’s experience and knowledge in hyperparameter setting.


## 2. Problem Statement

This article demonstrates the process of hyperparameter tuning using RSM for ML algorithms for predicting a numerical response variable. The tuning process is a sequential experimentation, consisting of designing a set of initial experiments and experimental data analysis, leading to subsequent experiments. The process continues until an optimal (or near-optimal) set of hyperparameter values is obtained. The hyperparameter tuning process using RSM is also compared with grid search (GS) for the ML algorithm performance, a number of experimental runs, and computational time. Both techniques are applied with three different ML algorithms: an artificial neural network (ANN), support vector machine (SVM), and deep belief network (DBN). Descriptions of the algorithms are briefly discussed next.


### 2.1. Artificial Neural Network (ANN)

Artificial neural network (ANN) was originated in 1943 by McCulloch and Pitts [33]. They proposed an ANN in terms of a mathematical model imitating the brain’s neuron connections. A feed-forward neural network (FFNN), a well-known and widely used neural network, has a structure that usually consists of three types of layers: input, hidden, and output layers. The feed-forward term represents one-way direction information flow from the input layer to the output layer. Each node is connected with the other nodes in the different layers by weights.The learning mechanism of FFNN is the backpropagation algorithm, as shown in Figure1. The weights and the biases are changed by recommendations from the gradient descent algorithm. The number of cycles that the weights and biases are optimized is called the training cycles [34, 35].Figure 1
Backpropagation algorithm.The hyperparameters of FFNN are the numbers of input nodes, numbers of hidden layers and their respective hidden nodes, learning rate, and training cycles. A greater number of layers and nodes can capture more complex relationships between inputs and an output; however, this requires a time-consuming training process. In addition, more complex relationships need a smaller learning rate and greater training cycles, and vice versa. The duration of the training process is proportional to the training cycles.


### 2.2. Support Vector Machine (SVM)

Support vector machine (SVM) was introduced by Vapnik in 1995 [36]. SVM is a supervised ML algorithm for classification and regression problems in various industries such as medicine, banking, beverages, and traffic flow [37–39] because it promises outstanding performance and avoids the overfitting problem that can occur in an ANN.Originally, SVM could only handle linear problems as it constructs a linear hyperplane to classify the dataset objects. However, nonlinear problems can be analyzed by SVM after data transformation by a kernel function (transformation function). This function is used to transform the points of data to the required form (Figure2). Nevertheless, the kernel function must be properly selected according to the type of dataset. Otherwise, SVM cannot provide an accurate prediction [40, 41].Figure 2
Diagram of (a) SVM with nonlinear hyperplane; (b) SVM with linear hyperplane (source: Alisneaky, 2011).The hyperparameters of SVM are complexity constant (C) and convergence epsilon (Conv). The complexity constant (C) represents the trade-off between minimization of the misclassified observations and margin maximization. The higher the value of C, the narrower the margin that causes less misclassification. Conversely, the lower the value of C, the wider the margin that leads to higher misclassification. The convergence epsilon (Conv) influences the precision on the Karush–Kuhn–Tucker (KKT) condition that is the stopping criteria of the hyperplane optimization. The lower the value of Conv, the more the precision of KKT condition, and the more the iterations of hyperplane optimization.


### 2.3. Deep Belief Network (DBN)

A deep belief network (DBN) was created by Hinton in 2006 [42], which is a combination of an artificial neural network (ANN) and a restricted Boltzmann machine (RBM). DBN can be applied as a prediction model in various fields (i.e., prediction for reservoir landslide displacement and prediction for deformed coal) and some other applications such as image recognition, speech recognition, and clustering. [43, 44].The structure of a DBN is similar to an ANN, but its initial weights are initialized by RBM in the pretraining stage before the fine-tuning stage (backpropagation algorithm). The weights from RBM can shorten the computational time in the fine-tuning stage because the computational time of the backpropagation algorithm depends on the initial values of connection weights which are randomly defined in a typical ANN.The hyperparameters of RBM in the pretraining stage are the learning rate and training cycles of RBM. The hyperparameters (same as in ANN) consist of the number of input nodes, a number of hidden layers and their respective hidden nodes, learning rate, and training cycles of the backpropagation algorithm in the fine-tuning stage. Typically, the learning rate of RBM is larger than the backpropagation algorithm, but the number of training cycles of RBM is smaller than the backpropagation algorithm, as RBM uses a greedy algorithm [45]. The architectures of DBN and RBM are shown in Figure 3.Figure 3
Diagram of (a) deep belief network (DBN) architecture and (b) restricted Boltzmann machines (RBM) (source: Shao et at., 2018).


### 2.4. ML Algorithm Performance Measure

We evaluate and compare the model performance of the two methods for the best hyperparameter settings. This involves using response surface methodology (RSM) and grid search (GS) by comparing the accuracy of the prediction model. In this study, the mean absolute error (MAE) is a key factor to measure and compare the performance between these two approaches. The MAE can be expressed as follows:(1)MAE=∑i=1Nyi−y^iN,where yi and y^i denote the actual value and predicted value of observation i, respectively, ei denotes the prediction error of observation i, and N denotes the total number of observations in the data. The lower the MAE, the better the model performance.


### 2.5. Data Partitioning and Cross-Validation

In general, constructing an ML model involves the following steps: constructing an initial model, setting the hyperparameter values, and testing the model performance. To perform these steps, an input dataset is usually partitioned into three datasets. These datasets are the training set for initial model construction, validation set for the tuning hyperparameter, and testing set for evaluating the model performance.In addition,k-fold cross-validation is applied for the data partitioning process. k-fold cross-validation is a widely used technique that is applied to building and testing models in machine learning. This technique randomly splits a dataset into k subsets of the same size. The value of k is specified by the modeler. Each subset takes turn to be in the training, validation, and testing sets. In other words, the training set contains k−2 subsets, and the validation and testing sets require one subset each. For k-fold cross-validation, an ML algorithm is constructed, validated, and tested k times.


## 2.1. Artificial Neural Network (ANN)

Artificial neural network (ANN) was originated in 1943 by McCulloch and Pitts [33]. They proposed an ANN in terms of a mathematical model imitating the brain’s neuron connections. A feed-forward neural network (FFNN), a well-known and widely used neural network, has a structure that usually consists of three types of layers: input, hidden, and output layers. The feed-forward term represents one-way direction information flow from the input layer to the output layer. Each node is connected with the other nodes in the different layers by weights.The learning mechanism of FFNN is the backpropagation algorithm, as shown in Figure1. The weights and the biases are changed by recommendations from the gradient descent algorithm. The number of cycles that the weights and biases are optimized is called the training cycles [34, 35].Figure 1
Backpropagation algorithm.The hyperparameters of FFNN are the numbers of input nodes, numbers of hidden layers and their respective hidden nodes, learning rate, and training cycles. A greater number of layers and nodes can capture more complex relationships between inputs and an output; however, this requires a time-consuming training process. In addition, more complex relationships need a smaller learning rate and greater training cycles, and vice versa. The duration of the training process is proportional to the training cycles.


## 2.2. Support Vector Machine (SVM)

Support vector machine (SVM) was introduced by Vapnik in 1995 [36]. SVM is a supervised ML algorithm for classification and regression problems in various industries such as medicine, banking, beverages, and traffic flow [37–39] because it promises outstanding performance and avoids the overfitting problem that can occur in an ANN.Originally, SVM could only handle linear problems as it constructs a linear hyperplane to classify the dataset objects. However, nonlinear problems can be analyzed by SVM after data transformation by a kernel function (transformation function). This function is used to transform the points of data to the required form (Figure2). Nevertheless, the kernel function must be properly selected according to the type of dataset. Otherwise, SVM cannot provide an accurate prediction [40, 41].Figure 2
Diagram of (a) SVM with nonlinear hyperplane; (b) SVM with linear hyperplane (source: Alisneaky, 2011).The hyperparameters of SVM are complexity constant (C) and convergence epsilon (Conv). The complexity constant (C) represents the trade-off between minimization of the misclassified observations and margin maximization. The higher the value of C, the narrower the margin that causes less misclassification. Conversely, the lower the value of C, the wider the margin that leads to higher misclassification. The convergence epsilon (Conv) influences the precision on the Karush–Kuhn–Tucker (KKT) condition that is the stopping criteria of the hyperplane optimization. The lower the value of Conv, the more the precision of KKT condition, and the more the iterations of hyperplane optimization.


## 2.3. Deep Belief Network (DBN)

A deep belief network (DBN) was created by Hinton in 2006 [42], which is a combination of an artificial neural network (ANN) and a restricted Boltzmann machine (RBM). DBN can be applied as a prediction model in various fields (i.e., prediction for reservoir landslide displacement and prediction for deformed coal) and some other applications such as image recognition, speech recognition, and clustering. [43, 44].The structure of a DBN is similar to an ANN, but its initial weights are initialized by RBM in the pretraining stage before the fine-tuning stage (backpropagation algorithm). The weights from RBM can shorten the computational time in the fine-tuning stage because the computational time of the backpropagation algorithm depends on the initial values of connection weights which are randomly defined in a typical ANN.The hyperparameters of RBM in the pretraining stage are the learning rate and training cycles of RBM. The hyperparameters (same as in ANN) consist of the number of input nodes, a number of hidden layers and their respective hidden nodes, learning rate, and training cycles of the backpropagation algorithm in the fine-tuning stage. Typically, the learning rate of RBM is larger than the backpropagation algorithm, but the number of training cycles of RBM is smaller than the backpropagation algorithm, as RBM uses a greedy algorithm [45]. The architectures of DBN and RBM are shown in Figure 3.Figure 3
Diagram of (a) deep belief network (DBN) architecture and (b) restricted Boltzmann machines (RBM) (source: Shao et at., 2018).


## 2.4. ML Algorithm Performance Measure

We evaluate and compare the model performance of the two methods for the best hyperparameter settings. This involves using response surface methodology (RSM) and grid search (GS) by comparing the accuracy of the prediction model. In this study, the mean absolute error (MAE) is a key factor to measure and compare the performance between these two approaches. The MAE can be expressed as follows:(1)MAE=∑i=1Nyi−y^iN,where yi and y^i denote the actual value and predicted value of observation i, respectively, ei denotes the prediction error of observation i, and N denotes the total number of observations in the data. The lower the MAE, the better the model performance.


## 2.5. Data Partitioning and Cross-Validation

In general, constructing an ML model involves the following steps: constructing an initial model, setting the hyperparameter values, and testing the model performance. To perform these steps, an input dataset is usually partitioned into three datasets. These datasets are the training set for initial model construction, validation set for the tuning hyperparameter, and testing set for evaluating the model performance.In addition,k-fold cross-validation is applied for the data partitioning process. k-fold cross-validation is a widely used technique that is applied to building and testing models in machine learning. This technique randomly splits a dataset into k subsets of the same size. The value of k is specified by the modeler. Each subset takes turn to be in the training, validation, and testing sets. In other words, the training set contains k−2 subsets, and the validation and testing sets require one subset each. For k-fold cross-validation, an ML algorithm is constructed, validated, and tested k times.


## 3. Hyperparameter Tuning with RSM

RSM is a design of experiments (DOE) technique that generates a small number of experimental runs to evaluate the causal effects of experimental factors on a response variable [46]. Experimental data are used to fit a regression model that relates the response variable to the experimental factors. Information from the regression model would guide the search process in subsequent experiments toward the region of factor values that optimize the response variable. For ML algorithms, the response variable is the ML performance measure, i.e., MAE in this study, and experimental factors are the hyperparameters of the ML algorithms. The procedure for using RSM for hyperparameter settings implemented in this study is shown in Figure 4.Figure 4
General procedure of RSM.In Step 1, a list of hyperparameters and their ranges is specified for each ML algorithm. Then, in Step 2, an experiment is designed to generate a number of experimental runs. Each run consists of a set of hyperparameter values that are used to construct the ML model in Step 3, where the prediction performance of the ML model is measured for the training, validation, and test sets. In Step 4, the regression model that relates the prediction performance to the hyperparameters is fitted. For fine-tuning, only the prediction performance of the validation set is used in this step. If the regression model contains only significant linear terms, then the tuning process has not reached the optimal region of hyperparameters. In this case, the path of steepest descent that would improve the ML model performance is from the coefficients of the variables representing the hyperparameters in the regression model. Subsequent experiments that move the hyperparameter values along this path are run until no further improvement can be obtained (then, go to Step 2). However, if the regression model is a second-order model, then the regression model is optimized to find a set of optimal hyperparameter values. Finally, some confirmation runs are performed to verify the ML model performance at the optimal settings.In this study, the experimental design that is applied in Step 2 is the central composite design (CCD). The CCD is an efficient design commonly used for constructing a quadratic (second-order) model. Using the CCD, a small number of experiments are performed to optimize the response variable. A CCD generally contains three levels of each factor, as shown in Figure5. The three types of design points are the factorial points (±1) or corner points, axial points (±α), and some replications of the center points (0).Figure 5
Central composite design (CCD) with three factors: (a) factorial points, (b) axial points and center points, and (c) central composite design.
(a)(b)(c)From Figure5, each factor in a CCD has five levels. Two levels are associated with factorial points, two levels with axial points, and one level at the center. A design space, bounded by factorial points and axial points, is the region of hyperparameters under study. The design space that contains a second-order model of the response variable is the region that has the minimum, maximum, or saddle point.For each factor, factorial points are low and high values of interest for the factor. Axial points are design points that set all factors at the center, except for one factor. With axial points, a quadratic model of the response variable can be constructed, and the curvature effects of experimental factors can be estimated [30,47]. Setting axial points requires the modeler to set an alpha (α) value. This value is the distance between each axial point and the center point. In this study, the α value is specified as 1.682 for a three-factor experiment (ANN), and 1.414 and 2.378 for a two-factor experiment (SVM) and a six-factor experiment (DBN), respectively. The value of α, which mainly depends on the number of factors, can be found using the following equation:(2)α=number of experimental runs1/4.For the center points, the value is set halfway between the low and high levels of each factor. Replications at the center point are usually performed to provide a measure of the variability of the prediction model, as well as to check for the curvature of the response surface.In this study, the total number of experimental runs of each ML algorithm depends on the number of hyperparameters. For two-factor CCD (the number of hyperparametersk = 2 for SVM), there are 2k= 4 factorial points (full factorial), 2k= 4 axial points, and usually five replications at the center point, for a total of 13 design points for each kernel function. For three-factor CCD (k = 3 for ANN), there are 20 design points: 2k=8 factorial points (full factorial), 2k= 6 axial points, and six replications of center points. For six-factor half-fraction CCD (k = 6 for DBN), there are a total of 53 runs, which are 2k−1=32 factorial points (half fraction of a full factorial), 2k= 12 axial points, and nine center points. In comparison, the number of runs for two-factor, three-factor, and six-factor experiments by GS, with five levels in each factor, is 52=25 runs, 53=125 runs, and a prohibitive 56=15,625 runs, respectively. It can be seen that the CCD offers significant savings in the number of experimental runs when compared to GS. Specifically, a general k-factor CCD contains 2k factorial points, 2k axial points, and n replications at the center, for a total of (2k + 2k + n) runs, rather than the 5k runs in GS.


## 4. Computational Experiment

### 4.1. Industrial Application and Dataset

A computational experiment is conducted on a dataset obtained from an industrial user, the largest producer and distributor in the canned fruit industry in Thailand. The data are collected at the raw material (RM) receiving operation from one of the main products. This operation is located at the beginning of the production line. The dataset contains observational data over a period of one year from over two thousand batches of RM.To the user, an important characteristic of the packing medium, which must be prepared in advance, is that it has to match the degree of Brix of the incoming RM. The degree of Brix is, therefore, the dependent variable, of which an accurate prediction would lead to significantly less packing medium preparation cost and lead time. A number of factors can be observed before the arriving RMs are put into production including RM harvest month, color, geographical variables, supplier, and source data. These factors are used as independent variables in the ML prediction models. The dataset is partitioned into 10 subsets for cross-validation purposes, where eight subsets are used as the training set and one subset each as the validation and test sets.


### 4.2. Hyperparameter Ranges for GS

First, the list of hyperparameters for the three ML algorithms is given in Table1, along with their initial input ranges. All experimental runs are based on this information.Table 1
List of ANN hyperparameters and their ranges.
MLHyperparameterDescriptionLowerUpperNo. of stepsANNHNNo. of hidden nodes11010TCNo. of training cycles10100021LRLearning rate0.00010.121SVMCComplexity constant−1100102ConvConvergence epsilon0.0010.111DBNHNNo. of hidden nodes1103niNo. of pretraining cycles1010003nNo. of training cycles1010003LRRBMPretraining learning rate0.00010.13LRLearning rate0.00010.13NbBatch size35017003ANN hyperparameters include the number of hidden nodes (HN), number of training cycles (TC), and learning rate (LR). Using GS, the value of each hyperparameter starts at its lower bound and increases with a fixed step size until reaching the upper bound. With the given number of steps, the total number of experimental runs from GS with 10-fold cross-validation is (10 × 21 × 21 × 10) = 44,100 runs.The SVM contains two hyperparameters, which are the complexity constant and convergence epsilon. In addition, a user can specify the type of kernel function of SVM, so it is also treated as a factor in our experiment. Four types of kernel functions are considered, including ANOVA, dot, radial, and Epanechnikov. For SVM, the total number of experimental runs from GS with 10-fold cross-validation is (102 × 11 × 4 × 10) = 44,880 runs.For DBN, the hyperparameters include the number of hidden nodes (HN), number of training cycles during pretraining (ni) and training (n), learning rate of pretraining and training (LRRBM and LR, respectively), and batch size (Nb). Since there are six hyperparameters, the number of steps for each hyperparameter is set to 3, and the total number of experimental runs from GS with 10-fold cross-validation is equal to (36 × 10) = 7,290 runs.


### 4.3. GS Results

The 10-fold cross-validation requires that all experimental runs for the three ML algorithms are to be performed 10 times. For each fold, the best hyperparameter settings for each algorithm are the one that results in the minimum MAE. Tables2–4 contain the best sets of hyperparameters from all 10-fold cross-validation results for ANN, SVM, and DBN, respectively.Table 2
Best hyperparameter setting and the test performance of ANN.
Grid searchConfirmation runsPI coverageFoldHNTCLRMAE (validation)Avg MAE (validation)SD of MAE (validation)95% PI half-width176040.0050950.95131.05430.06050.1517Yes268020.0050951.03331.14610.05710.1432Yes392080.0050950.96371.03270.0320.0803Yes498520.0050950.97431.07060.04580.1149Yes5510000.0050950.96211.07310.01880.0472No63100.010090.95691.02650.04410.1106Yes796540.010091.11911.19560.04310.1081Yes865050.020081.01431.20140.07130.1788No979510.0050950.98941.09380.05380.1349Yes1026040.0150851.06331.21260.06060.152YesBold values indicate the best set of hyperparameters from Grid Search that leads to the minimal MAE (validation).Table 3
Best hyperparameter setting and the test performance of SVM.
FoldKernel functionCConvMAE (validation)1ANOVA10.01091.06682Radial30.03071.12373ANOVA20.07031.0524Radial750.0011.04925ANOVA10.11.0616Dot−10.0010.99717ANOVA10.04061.16488ANOVA−10.09011.1059ANOVA−10.02081.069310ANOVA10.08021.1319Table 4
Best hyperparameter setting and the test performance of DBN.
GSConfirmation runsFoldHNninLRRBMLRNbMAE (validation)Avg MAE (validation)SD of MAE (validation)95% PI half-widthPI coverage11010005050.10.0500510250.9871.03690.03180.0832Yes2101000100.050050.110251.09061.1190.03830.1002Yes310100010000.050050.050053500.97181.010.02190.0573Yes4105055050.10.0500510251.00921.06470.03520.0921Yes51050510000.10.0500517000.99781.04620.04560.1193Yes6101000100.10.110250.93131.00630.02460.0644No71050510000.10.110251.1121.17370.0380.0994Yes8105055050.10.0500510251.03611.07360.03330.0871Yes91050510000.10.110250.98931.06750.02770.0725No105.510005050.050050.050053501.04461.10910.02510.0657YesIn the table, the MAEs of the validation dataset of all 10 folds are given. The reported hyperparameter values for each fold represent the best setting among 4,410 experimental runs. The best values of HN and TC can vary from the lower bound to the upper bound, i.e., HN from 2 to 9 and TC from 10 to 1,000 (most are 500 or more). LR is the only hyperparameter where the best values are clustered around the lower bound, from 0.005 to 0.02, with the most frequent value at 0.005. The results suggest that the best and reliable setting of LR is relatively low, while the best values of the HN and TC are inconclusive.It is important to note that each value of MAE (from cross-validation) is based on one experimental run at their respective settings. Also, rerunning an ANN at the same hyperparameter setting gives different MAE results due to the nature of ANNs. Therefore, to test the reliability of the best settings found from the 10-fold cross-validation, an ANN with each of these settings is rerun as confirmation. The number of confirmation runs is selected such that the 95% confidence interval of the average MAE of the confirmation runs is within 5% of the mean, which results in eight confirmation runs.After performing the confirmation runs, the average MAE of the validation set and its SD are reported for each hyperparameter setting. It was found that the average MAEs from confirmation runs are not as good as the MAEs value obtained from cross-validation GS due to the natural variability of the ML algorithm performance. As seen in Table2, the minimum value of MAE (0.95) from the 1st fold of cross-validation is different from the average of the MAE (1.03) from eight confirmation runs. To properly evaluate the reliability of the hyperparameters from GS, we perform “prediction interval (PI) coverage” in this study. The PI coverage uses the half-width of the 95% PI. We define the performance of GS as reliable if the PI coverage is satisfied, i.e., the MAE from one run (cross-validation) is within the half-width of the 95% PI from the average MAE of the confirmation runs. Finally, the PI coverage of the hyperparameter settings found from GS is satisfied in 8 out of 10 folds. This indicates reasonable (80%) reliability of the ANN hyperparameter values found from GS.Table3 provides the best hyperparameter settings from 10-fold cross-validation of SVM. In 7 out of 10 folds, setting the kernel function as ANOVA and a low value of the complexity constant (−1 or 1) is reliable, whereas the best coverage values tend to vary. SVM is an ML algorithm based on statistical learning, where performing a rerun of SVM at the same hyperparameter setting would give the same prediction value, i.e., the same prediction performance. Therefore, there is no need to do the confirmation runs.Table4 provides DBN results in the same fashion as Table 2. Most of the best hyperparameter settings are an HN of 10 (high value), ni and n are either 500 or 1000 (medium or high value), LRRBM and LR are either 0.05 or 0.10 (medium or high value), and the batch size is 1,025 (medium value). In 8 out of 10 folds, the PI coverage is satisfied, which indicates 80% reliability of hyperparameter settings of DBN found from GS.As mentioned earlier, due to GS multiplicative nature, the required number of runs is large especially whenk-fold cross-validation is used. For an ANN with three hyperparameters, the 10-fold cross-validation in our testing requires 44,100 runs. The GS search for the ANN is reasonable because each hyperparameter is searched in over 10 steps. However, for a DBN with six hyperparameters, the same level of detial for GS search is prohibited, i.e., more than 106 runs for each fold. The same level of detail for GS search is prohibited, i.e., more than 106 runs for each fold. That means the search space of GS for DBN is not well explored. In other words, GS may be ineffective for an ML algorithm with many hyperparameters due to computational resource limitations.


### 4.4. RSM Results

#### 4.4.1. Artificial Neural Network

From the initial ranges of hyperparameters, listed in Table1, the search process begins at the low values of these hyperparameters, i.e., near the origin point of the hyperparameter space. The lower bound and upper bound are specified for the axial points of the CCD. Since HN and TC are integers, the values of the hyperparameters at factorial points are rounded to the nearest integer.Experiment 1.
Identifying the steepest descent direction
A three-factor CCD with 20 design points is performed for each data fold. Each row in the table represents an experimental run, where point type of 1, −1, and 0 indicates that the experiment is a factorial point, axial point, and center point, respectively. Note that, this CCD is constructed by specifying the axial points for each hyperparameter as follows: 1 and 3 for HN, 10 and 100 for TC, and 0.0001 and 0.001 for LR. Also, the values of HN of some design points are rounded to the nearest integer since HN cannot be a fractional number. The MAEs of the validation set of all 10 folds of data are shown in Table5.
Analysis of variance (ANOVA) is performed on each data fold separately. The steps in the data analysis of the second fold are described as an example. First, the regression model begins as a full-quadratic model, which contains the main effects, squared terms, and two-factor interaction terms. Then, insignificant terms are eliminated from the regression model, one at a time, until the final model is obtained. The regression model fitting process is conducted, while maintaining the hierarchy of the model, i.e., keeping lower-order term(s) in the presence of a significant higher-order term.
The ANOVA from the final model of the second data fold is shown in Table6. The linear term and the quadratic term of TC are significant, while the HN and LR effects are not. This may be caused by the ranges of HN and LR being too narrow. Further analysis as shown in Figure 6 suggests that a higher value of HN could reduce the MAE, and a high value of LR may already be effective. Therefore, the search direction of subsequent experiments needs to increase HN and LR, while fixing TC. A similar analysis is performed on the other nine data folds, which results in nine additional ANOVA tables and regression models. After examining the results of all 10 folds, four patterns of hyperparameter effects on the MAE of the validation set are summarized in Table 7, along with a suggested search direction.
We denote a quadratic effect of a hyperparameter as convex if the value of the hyperparameter that is associated with a low value of MAE occurs in the middle of the input range. A concave pattern indicates the opposite hyperparameter effect where low values of MAE occur at the boundaries of the input range. In addition, there are two linear trends, a negative trend and a positive trend, which point to a low value of MAE occurring at the upper bound and lower bound, respectively. These patterns also suggest the search directions. For a convex pattern and linear positive trend, the hyperparameter values remain unchanged at the value that gives the minimum MAE and at the lower bound, respectively, in subsequent experiments. However, for a concave pattern and linear negative trend, the search direction is to increase the hyperparameter value. The search direction is defined by the coded coefficients of the final regression model. For example, the final regression model from the second fold of data is as follows:(3)MAE=1.1343−0.0006HN−0.0085TC−0.0016LR+0.0050TC2.
Based on the coded coefficients in the model, every coded unit increase in HN is associated with (-0.0000/-0.0016) = 2.6667 coded units increase in LR. The 2.6667 coded units have to be transformed to an original scale. Recall that in the original scale, LR = 0.00028 is coded as −1 at the factorial point, and LR = 0.00055 is coded as 0 at the center points. In other words, a coded unit increase is equivalent to (0.00055–0.00028) = 0.00027 units in LR. Therefore, in the steepest descent direction, an increase of 2.6667 coded units becomes a (0.00027 × 2.6667) = 0.000714 increase in LR.Table 5
Design and experiment results of Experiment1.
Point typeHyperparameterMAE (validation)HNTCLRFold 1Fold 2Fold 3Fold 4Fold 5Fold 6Fold 7Fold 8Fold 9Fold 1011280.000281.13321.14651.12541.09841.10711.02671.26921.11731.08741.163613280.000281.11461.15021.13341.09321.09681.02021.28431.12941.09791.202811820.000281.12711.13581.0941.07361.09640.99941.24691.11841.0641.139613820.000281.09681.13011.0791.0761.08651.00241.23771.12031.06731.151511280.000821.19431.13551.16341.07191.10131.02671.25631.11921.07411.153413280.000821.11261.13061.07351.07531.08911.02021.26271.11661.06331.142211820.000821.17851.13061.13991.08151.10010.99941.24331.11871.07571.139313820.000821.10021.13381.04091.09361.05761.00241.22821.08571.08491.1382−11550.000551.15681.13441.11771.07281.09981.00911.24981.11861.07151.1446−13550.000551.09941.1281.06691.07661.08211.01071.24961.11021.06691.1451−12100.000551.13341.17781.17291.12571.13361.03871.28841.1641.11211.2212−121000.000551.07831.12751.07861.10461.08680.9781.22321.11271.07931.1793−12550.00011.07831.12751.07861.10461.08680.9781.22321.11271.07931.1793−12550.0011.08521.13351.08731.11121.09740.97731.24381.14511.08331.212302550.000551.09251.14831.17931.08041.09691.00051.25111.23371.15781.143802550.000551.09251.13031.10461.10781.08650.97271.27711.21831.08331.179302550.000551.08821.14331.10251.09481.07020.99041.23961.13371.06751.180602550.000551.09471.13051.07411.08391.08861.01111.23671.19171.07421.137302550.000551.20471.1291.1071.08891.10360.97561.23241.1091.06731.14302550.000551.07771.15291.06631.091.0960.97681.28281.12031.08221.1441Table 6
Analysis of variance of the second fold.
SourceDfAdj SSAdj MSF-valueP valueModel40.0014180.0003553.560.031Linear30.0010430.0003483.490.042HN10.000010.000010.10.754TC10.0009980.00099810.020.006LR10.0000350.0000350.350.562Square10.0003750.0003753.770.071TC × TC10.0003750.0003753.770.071Error150.0014940.0001Lack-of-fit100.0009470.0000950.870.606Pure error50.0005470.000109Total190.002912Figure 6
The response optimizer from the second fold.Table 7
Patterns of hyperparameter effects and search direction.
FoldHidden nodesTraining cyclesLearning rateSearch direction1ConvexLinear (−)Linear (+)Fix HC, TC at minimum MAE and increase TC2Linear (−)ConvexLinear (-)Increase HN, LR together and fix TC at minimum MAE3ConvexLinear (−)Linear (−)Fix HN at minimum MAE and increase TC and LR together4ConcaveConvexConvexIncrease HN and fix TC5Linear (−)ConvexLinear (−)Increase HN and LR together6ConvexConvexLinear (−)Increase LR7Linear (−)ConvexConvexIncrease HN8Linear (−)Linear (−)ConvexIncrease HN and TC9Linear (−)ConcaveLinear (+)Increase HN10ConcaveConvexConvexIncrease HNExperiment 2.
Searching along the steepest descent direction
Search results along the steepest descent direction are given in Figure7. From the results, an HN of 8 and LR of 0.0117 seem to be effective. Therefore, these settings become a new center point for the next experiment.
The process of searching along the path of steepest descent is repeated for each data fold. This process stops when the best hyperparameter values on this path are found. These settings become the center point for further experiments.Figure 7
The search direction from the second fold.Experiment 3.
Finding optimal setting
For the second data fold, another CCD is conducted. The results are shown in Figure8.
The regression model fitting process is performed based on the MAE results and hyperparameter values in Figure8. The ANOVA of the final model is shown in Table 8. The linear terms and quadratic terms of HN and TC are significant, while the LR effects are not significant. The final regression model is then optimized with respect to HN and TC. The best hyperparameter settings from the second data fold are HN = 8, TC = 194, and LR = 0.005095 with the expected value of MAE equal to 1.0900. Similar to the GS, eight confirmation runs are performed.
Experiment3 is repeated for the other nine data folds. A summary of the results is shown in Table 9. The table contains the best values of the hyperparameters for all 10 folds, the validation MAE at these settings from the experiment, and the results from the confirmation run. In the last column, PI coverage indicates whether or not the 95% prediction intervals of MAE contain the validation MAE from the RSM experiments. The results show that in 9 out of 10 folds, the 95% PIs contain the validation MAE. This indicates a 90% reliability of the hyperparameter values obtained from RSM.
Among different folds, HN varies from 4 to 10, and TC varies from 10 to 951. The best LR is the same for all folds, which is 0.005095. Although the best settings of HN and TC are inconclusive, it is noticeable that there is a mild trend between HN and TC, i.e., lower HN tends to be associated with higher TC and vice versa.Figure 8
Design and experiment results of Experiment3.Table 8
ANOVA of the final model of the second data fold.
SourceDfAdj SSAdj MSF-valueP valueModel40.128620.03215510.62<0.001HN10.011480.0114813.790.070TC10.089530.08953429.56<0.001HN × HN10.017620.0176175.820.029TC × TC10.014830.0148354.90.043Error150.045430.003028Lack-of-fit100.033470.0033471.40.373Pure error50.011950.002391Total190.17405Table 9
Best hyperparameter setting and the validation performance of ANN using RSM.
RSMConfirmation runsFoldHNTCLRMAE (validation)Avg MA (validation)ESD of MAE (validation)95% PI half-widthPI coverage148020.0050951.0121.22550.05670.1422No281940.0050951.0871.13640.05030.1262Yes349510.0050951.0341.07810.02630.066Yes452580.0050951.0951.11790.05180.1299Yes510600.0050951.0351.15350.06110.1532Yes69100.0050950.981.08180.05330.1337Yes751090.0050951.1691.27940.06750.1693Yes884560.0050951.0621.1980.06430.1613Yes972080.0050951.0511.10910.03480.0873Yes1041090.0050951.1131.21190.05790.1452YesBold values indicate that this is the combination of hyperparameters that give the minimal average MAE (validation) among all folds.


#### 4.4.2. Support Vector Machine (SVM)

Similar to an ANN, the search starts at low values of the initial range of hyperparameter (see Table1). Since the type of kernel function is a categorical factor, the first experiment is conducted separately for the four types of kernel functions.Experiment 4.
Identifying the steepest descent direction
Using a two-factor CCD, a data fold has 13 experimental runs for each kernel function. The total number of initial experimental runs to find paths of steepest descent is equal to 13 × 10 × 4 = 520 runs. After running all experiments, the results in Table10 reveal that dot and radial kernel functions are more effective than ANOVA and Epanechnikov functions, which, therefore, are not further considered.
The same process of RSM described in the previous section is conducted with SVM. Patterns of hyperparameter effects on MAE fordot and radial functions are summarized in Table 11, along with the data fold(s) associated with each pattern in the last two columns.
For subsequent experiments, search along the steepest descent direction.
The process of searching along the steepest descent directions to find the appropriate number of steps to move and to find the optimal settings for all 10 folds is performed in the same manner as that of the ANN. Detailed results are omitted to save space. A summary of the optimal hyperparameter settings for all 10 folds is provided in Table12. Overall, the values of C for dot and radial functions are relatively low, while Conv is inconclusive. As mentioned previously, rerunning SVM at the same settings would give the same prediction value. Therefore, confirmation runs are not performed.
After applying the previous steps to all 10 folds, the best hyperparameters fordot and radial kernel functions are shown in Table 12. The minimum value of MAE is 1.053 for both kernel functions. For the dot function, the best hyperparameter settings are equal to 9 and 0.0307 for the C and Conv, respectively. For the radial function, C and Conv are 6 and 0.001, respectively.Table 10
Experiment 4 results for the four kernel functions.
Kernel functionAverage MAE from the validation setFold 1Fold 2Fold 3Fold 4Fold 5Fold 6Fold 7Fold 8Fold 9Fold 10ANOVA1.19641.29691.16071.17631.20611.22831.30411.29121.24321.249Dot1.10211.15731.10561.10241.18031.08651.321.20161.20161.1701Radial1.21241.21241.11361.07581.12151.15651.2581.18211.14791.1373Epanechnikov1.19631.18851.12721.08871.14851.12711.29441.21571.14941.2193Bold values indicate the minimal average MAE for each data fold (Fold 1 to Fold 10).Table 11
Patterns of hyperparameter effects and search direction fordot and radial kernel function.
CConvSearch directiondotradialLinear (−)Linear (−)Increase C and Conv togetherFold: 1, 5, 6, 10Fold: 1ConcaveLinear (−)Fix C at minimum MAE and increase ConvFold: 2, 7, 9Linear (+)Linear (−)Fix C at minimum MAE and increase ConvFold: 3, 8Linear (+)Linear (+)Fix C and Conv at minimum MAEFold: 4ConvexLinear (+)Fix C and Conv at minimum MAEFold: 2Linear (+)Linear (-)Fix C at lower bound and increase ConvFold: 3ConcaveLinear (+)Fix C and Conv at minimum MAEFold: 4, 5, 9Linear (−)Linear (+)Increase C and fix Conv at minimum MAEFold: 7ConcaveLinear (−)Fix C at lower bound and increase ConvFold: 8, 10Table 12
Best hyperparameter settings fordot and radial kernel functions using RSM.
FoldDotRadialCConvergenceMAE (validation)CConvergenceMAE (validation)1120.0011.1680120.01091.14552−10.11.186830.0011.12373−10.04061.1102−10.07031.15754-10.0011.169460.0011.05295110.01091.120990.0011.1137690.03071.0528170.02081.08927−10.11.2577110.0011.24528−10.05051.123430.01091.16049−10.09011.076590.0011.098810100.04061.192370.11.1827


#### 4.4.3. Deep Belief Network (DBN)

The initial ranges of hyperparameters in the first experiment for DBN are listed in Table1. These ranges are also chosen to start at low values. The lower and upper bounds in Table 11 are values for the axial points. Since HN, n, and ni are integers, their values at factorial points are rounded to the nearest integers.Experiment 5.
: A six-factor half-fraction CCD with 53 design points is performed for each data fold. For 10 folds, a total of 53 × 10 = 530 runs are conducted to find the paths of the steepest descent. As an example, ANOVA of the final model from the 2nd data fold is given in Table13. The obtained regression model provides the steepest descent direction to increase LR together with n, while holding the other hyperparameters fixed at their boundaries or at the point where MAE is minimized if the squared term is significant.
The RSM process is conducted with DBN on 10 data folds. Patterns of hyperparameter effects on MAE and the search directions are summarized in Table14. Moving along these steepest descent directions toward the optimal region gives the center points for the subsequent experiments. Then, running another CCD around these center points leads to the final regression models that yield the optimal settings for all 10 data folds. The best hyperparameter settings obtained from the final regression models and the validation MAE at these settings for DBN using RSM are provided in Table 15. The results from confirmation runs, including the average MAE, SD, half-width of 95% PI, and coverage of PI, are also shown. The PI coverages of the DBN hyperparameter settings from RSM indicate 100% reliability based on our testing.Table 13
Analysis of variance of the second fold.
SourceDfAdj SSAdj MSF-valueP valueModel8363.1845.39714.5<0.001HN10.4460.4460.140.708LRRBM10.6520.6520.210.650LR1137.73137.7343.98<0.001N114.96814.9684.780.034ni1153.99153.9949.17<0.001Nb128.91128.9119.230.004HN × HN111.96911.9693.820.057LR ×n114.5114.514.630.037Error44137.83.132Lack-of-fit36111.063.0850.920.603Pure error826.7383.342Total52500.98Table 14
Patterns of hyperparameter effects and search direction for DBN.
FoldHNLRRBMnniLRNbSearch direction1Linear (−)ConcaveLinear (−)Linear (+)Linear (−)Linear (+)Increase HN, N, LR, and fix LRRBM, ni, Nbat min. MAE2ConvexLinear (+)Linear (−)Linear (+)Linear (−)Linear (+)Increase N, LR, and fix HN, LRRBM, ni, Nbat min. MAE3Linear (−)Linear (−)Linear (−)Linear (+)Linear (−)Linear (+)Increase HN, LRRBM, N, LR, and fix ni, Nbat min. MAE4Linear (−)Linear (−)Linear (−)Linear (−)Linear (−)Linear (+)Increase HN, LRRBM, N, ni, LR, and fix, Nbat min. MAE5Linear (+)Linear (−)Linear (−)Linear (−)Linear (−)Linear (+)Increase LRRBM, N, ni, LR, and fix HN, Nbat min. MAE6Linear (−)Linear (+)Linear (−)Linear (−)Linear (−)Linear (+)Increase HN, N,ni, LR, and fix LRRBM, Nbat min. MAE8Linear (−)ConvexLinear (−)ConvexLinear (−)ConcaveIncrease HN, N, LR, and fix LRRBM, ni, Nbat min. MAE9Linear (−)Linear (−)Linear (−)Linear (+)Linear (−)Linear (+)Increase HN, LRRBM, N, LR, and fix ni, Nbat min. MAE10Linear (+)Linear (+)Linear (−)Linear (+)Linear (−)Linear (+)Increase N, LR, and fix HN, LRRBM, ni, Nbat min. MAETable 15
Best hyperparameter setting and the validation performance of DBN using RSM.
Cross-validationConfirmation runsFoldHNninLRRBMLRNbMAEAvg MAESD (MAE)95% PI half-widthPI coverage17389100.00010.006910251.0811.0970.0320.109Yes23530100.0010.010710251.1661.1680.0090.03Yes38575100.0010.00673501.0731.0830.0110.036Yes43254100.0010.006810251.0831.0960.0130.042Yes53209100.0010.007317001.0871.0930.0090.031Yes65209100.0010.00810251.0000.9980.0130.045Yes74434100.0010.009410251.2231.2320.0180.061Yes87344100.0010.007110251.121.140.0350.118Yes96620100.0010.010110251.1311.0890.0280.095Yes106620100.0010.01013501.2991.2030.0640.215YesBold values emphasize that this is the combination of hyperparameters that results in the minimal average MAE (0.998).


### 4.5. Comparison between GS and RSM

ANOVA is performed to compare the confirmation MAE between GS and RSM (denoted as Method) for ANN, SVM, and DBN (denoted as ML) while treating data folds as blocks (see Table16). The interaction between method and fold is mildly significant (P value = 0.094), which indicates that the average MAE of validation set between GS and RSM is statistically different in some data folds. In Table 17, further analysis using Tukey’s multiple comparisons shows that the difference is significant only for the first data fold, i.e., the hyperparameters from GS give statistically better results (on average) than those of RSM. For the other nine data folds, the hyperparameter settings from GS and RSM perform statistically the same.Table 16
ANOVA of confirmation MAE of validation set by hyperparameter tuning methods.
SourceDfAdj SSAdj MSF-valueP valueFold90.1734340.0192734.08<0.001ML20.0192890.00964517.06<0.001Method10.0331920.03319258.7<0.001Fold × method90.0093150.0010351.830.094Error380.0214860.000565Total590.256716Table 17
Tukey’s multiple comparison of the confirmation MAE between GS and RSM.
Fold 1Fold 2Fold 3Fold 4Fold 5Fold 6Fold 7Fold 8Fold 9Fold 10GSAAAAAAAAAARSMBAAAAABAAANote: Means that do not share the same letter are significantly different.To investigate the reliability of using the best hyperparameters from RSM to predict unforeseen data, the three ML models with the best hyperparameters from GS and RSM are applied to predict the results in the testing dataset. Then, ANOVA is conducted to compare the average MAE of the testing set between GS and RSM (see Table18). The ANOVA shows that there is no significant difference in the average MAE between the two hyperparameter tuning methods (i.e., GS and RSM). Therefore, this implies that the hyperparameter setting from RSM is reliable in the testing set as well.Table 18
ANOVA on MAE of testing set by hyperparameter tuning methods: GS and RSM.
SourceDfAdj SSAdj MSF-valueP valueFold90.17230.019110.47<0.001ML20.03670.018310.03<0.001Tuning method10.00440.00442.410.127Error470.0859690.001829Total590.299374


### 4.6. Discussion

Grid search (GS) is widely used in several studies as a common hyperparameter tuning method. As shown in section4.3, implementing GS in ANN, SVM, and DBN requires many experiments, i.e., 44,100 runs, 44,880 runs, and 7,290 runs, respectively. The results in section 4.5 indicate that on average, the prediction performance of hyperparameter settings obtained from GS and RSM is statistically the same in all three ML models, while RSM requires a smaller number of runs. Specifically, with RSM, the number of runs for ANN, SVM, and DBN is 976 runs (97.79% saving over GS), 984 runs (97.81% saving), and 1,408 runs (80.69% saving), respectively. In other words, the benefit from using RSM has been demonstrated that it can significantly reduce the number of experiments to determine the suitable hyperparameter setting while maintaining the prediction performance.Finally, it is worth noting that RSM, by nature, involves sequential experimentation. That is, RSM requires a modeler to run some experiments and perform data analysis to determine whether the subsequent experiment is needed. If it is the case, then findings from one experiment will guide the search in a subsequent experiment. A limitation of RSM is that this search process is not automated or has not been integrated as a hyperparameter tuning process in standard ML software.


## 4.1. Industrial Application and Dataset

A computational experiment is conducted on a dataset obtained from an industrial user, the largest producer and distributor in the canned fruit industry in Thailand. The data are collected at the raw material (RM) receiving operation from one of the main products. This operation is located at the beginning of the production line. The dataset contains observational data over a period of one year from over two thousand batches of RM.To the user, an important characteristic of the packing medium, which must be prepared in advance, is that it has to match the degree of Brix of the incoming RM. The degree of Brix is, therefore, the dependent variable, of which an accurate prediction would lead to significantly less packing medium preparation cost and lead time. A number of factors can be observed before the arriving RMs are put into production including RM harvest month, color, geographical variables, supplier, and source data. These factors are used as independent variables in the ML prediction models. The dataset is partitioned into 10 subsets for cross-validation purposes, where eight subsets are used as the training set and one subset each as the validation and test sets.


## 4.2. Hyperparameter Ranges for GS

First, the list of hyperparameters for the three ML algorithms is given in Table1, along with their initial input ranges. All experimental runs are based on this information.Table 1
List of ANN hyperparameters and their ranges.
MLHyperparameterDescriptionLowerUpperNo. of stepsANNHNNo. of hidden nodes11010TCNo. of training cycles10100021LRLearning rate0.00010.121SVMCComplexity constant−1100102ConvConvergence epsilon0.0010.111DBNHNNo. of hidden nodes1103niNo. of pretraining cycles1010003nNo. of training cycles1010003LRRBMPretraining learning rate0.00010.13LRLearning rate0.00010.13NbBatch size35017003ANN hyperparameters include the number of hidden nodes (HN), number of training cycles (TC), and learning rate (LR). Using GS, the value of each hyperparameter starts at its lower bound and increases with a fixed step size until reaching the upper bound. With the given number of steps, the total number of experimental runs from GS with 10-fold cross-validation is (10 × 21 × 21 × 10) = 44,100 runs.The SVM contains two hyperparameters, which are the complexity constant and convergence epsilon. In addition, a user can specify the type of kernel function of SVM, so it is also treated as a factor in our experiment. Four types of kernel functions are considered, including ANOVA, dot, radial, and Epanechnikov. For SVM, the total number of experimental runs from GS with 10-fold cross-validation is (102 × 11 × 4 × 10) = 44,880 runs.For DBN, the hyperparameters include the number of hidden nodes (HN), number of training cycles during pretraining (ni) and training (n), learning rate of pretraining and training (LRRBM and LR, respectively), and batch size (Nb). Since there are six hyperparameters, the number of steps for each hyperparameter is set to 3, and the total number of experimental runs from GS with 10-fold cross-validation is equal to (36 × 10) = 7,290 runs.


## 4.3. GS Results

The 10-fold cross-validation requires that all experimental runs for the three ML algorithms are to be performed 10 times. For each fold, the best hyperparameter settings for each algorithm are the one that results in the minimum MAE. Tables2–4 contain the best sets of hyperparameters from all 10-fold cross-validation results for ANN, SVM, and DBN, respectively.Table 2
Best hyperparameter setting and the test performance of ANN.
Grid searchConfirmation runsPI coverageFoldHNTCLRMAE (validation)Avg MAE (validation)SD of MAE (validation)95% PI half-width176040.0050950.95131.05430.06050.1517Yes268020.0050951.03331.14610.05710.1432Yes392080.0050950.96371.03270.0320.0803Yes498520.0050950.97431.07060.04580.1149Yes5510000.0050950.96211.07310.01880.0472No63100.010090.95691.02650.04410.1106Yes796540.010091.11911.19560.04310.1081Yes865050.020081.01431.20140.07130.1788No979510.0050950.98941.09380.05380.1349Yes1026040.0150851.06331.21260.06060.152YesBold values indicate the best set of hyperparameters from Grid Search that leads to the minimal MAE (validation).Table 3
Best hyperparameter setting and the test performance of SVM.
FoldKernel functionCConvMAE (validation)1ANOVA10.01091.06682Radial30.03071.12373ANOVA20.07031.0524Radial750.0011.04925ANOVA10.11.0616Dot−10.0010.99717ANOVA10.04061.16488ANOVA−10.09011.1059ANOVA−10.02081.069310ANOVA10.08021.1319Table 4
Best hyperparameter setting and the test performance of DBN.
GSConfirmation runsFoldHNninLRRBMLRNbMAE (validation)Avg MAE (validation)SD of MAE (validation)95% PI half-widthPI coverage11010005050.10.0500510250.9871.03690.03180.0832Yes2101000100.050050.110251.09061.1190.03830.1002Yes310100010000.050050.050053500.97181.010.02190.0573Yes4105055050.10.0500510251.00921.06470.03520.0921Yes51050510000.10.0500517000.99781.04620.04560.1193Yes6101000100.10.110250.93131.00630.02460.0644No71050510000.10.110251.1121.17370.0380.0994Yes8105055050.10.0500510251.03611.07360.03330.0871Yes91050510000.10.110250.98931.06750.02770.0725No105.510005050.050050.050053501.04461.10910.02510.0657YesIn the table, the MAEs of the validation dataset of all 10 folds are given. The reported hyperparameter values for each fold represent the best setting among 4,410 experimental runs. The best values of HN and TC can vary from the lower bound to the upper bound, i.e., HN from 2 to 9 and TC from 10 to 1,000 (most are 500 or more). LR is the only hyperparameter where the best values are clustered around the lower bound, from 0.005 to 0.02, with the most frequent value at 0.005. The results suggest that the best and reliable setting of LR is relatively low, while the best values of the HN and TC are inconclusive.It is important to note that each value of MAE (from cross-validation) is based on one experimental run at their respective settings. Also, rerunning an ANN at the same hyperparameter setting gives different MAE results due to the nature of ANNs. Therefore, to test the reliability of the best settings found from the 10-fold cross-validation, an ANN with each of these settings is rerun as confirmation. The number of confirmation runs is selected such that the 95% confidence interval of the average MAE of the confirmation runs is within 5% of the mean, which results in eight confirmation runs.After performing the confirmation runs, the average MAE of the validation set and its SD are reported for each hyperparameter setting. It was found that the average MAEs from confirmation runs are not as good as the MAEs value obtained from cross-validation GS due to the natural variability of the ML algorithm performance. As seen in Table2, the minimum value of MAE (0.95) from the 1st fold of cross-validation is different from the average of the MAE (1.03) from eight confirmation runs. To properly evaluate the reliability of the hyperparameters from GS, we perform “prediction interval (PI) coverage” in this study. The PI coverage uses the half-width of the 95% PI. We define the performance of GS as reliable if the PI coverage is satisfied, i.e., the MAE from one run (cross-validation) is within the half-width of the 95% PI from the average MAE of the confirmation runs. Finally, the PI coverage of the hyperparameter settings found from GS is satisfied in 8 out of 10 folds. This indicates reasonable (80%) reliability of the ANN hyperparameter values found from GS.Table3 provides the best hyperparameter settings from 10-fold cross-validation of SVM. In 7 out of 10 folds, setting the kernel function as ANOVA and a low value of the complexity constant (−1 or 1) is reliable, whereas the best coverage values tend to vary. SVM is an ML algorithm based on statistical learning, where performing a rerun of SVM at the same hyperparameter setting would give the same prediction value, i.e., the same prediction performance. Therefore, there is no need to do the confirmation runs.Table4 provides DBN results in the same fashion as Table 2. Most of the best hyperparameter settings are an HN of 10 (high value), ni and n are either 500 or 1000 (medium or high value), LRRBM and LR are either 0.05 or 0.10 (medium or high value), and the batch size is 1,025 (medium value). In 8 out of 10 folds, the PI coverage is satisfied, which indicates 80% reliability of hyperparameter settings of DBN found from GS.As mentioned earlier, due to GS multiplicative nature, the required number of runs is large especially whenk-fold cross-validation is used. For an ANN with three hyperparameters, the 10-fold cross-validation in our testing requires 44,100 runs. The GS search for the ANN is reasonable because each hyperparameter is searched in over 10 steps. However, for a DBN with six hyperparameters, the same level of detial for GS search is prohibited, i.e., more than 106 runs for each fold. The same level of detail for GS search is prohibited, i.e., more than 106 runs for each fold. That means the search space of GS for DBN is not well explored. In other words, GS may be ineffective for an ML algorithm with many hyperparameters due to computational resource limitations.


## 4.4. RSM Results

### 4.4.1. Artificial Neural Network

From the initial ranges of hyperparameters, listed in Table1, the search process begins at the low values of these hyperparameters, i.e., near the origin point of the hyperparameter space. The lower bound and upper bound are specified for the axial points of the CCD. Since HN and TC are integers, the values of the hyperparameters at factorial points are rounded to the nearest integer.Experiment 1.
Identifying the steepest descent direction
A three-factor CCD with 20 design points is performed for each data fold. Each row in the table represents an experimental run, where point type of 1, −1, and 0 indicates that the experiment is a factorial point, axial point, and center point, respectively. Note that, this CCD is constructed by specifying the axial points for each hyperparameter as follows: 1 and 3 for HN, 10 and 100 for TC, and 0.0001 and 0.001 for LR. Also, the values of HN of some design points are rounded to the nearest integer since HN cannot be a fractional number. The MAEs of the validation set of all 10 folds of data are shown in Table5.
Analysis of variance (ANOVA) is performed on each data fold separately. The steps in the data analysis of the second fold are described as an example. First, the regression model begins as a full-quadratic model, which contains the main effects, squared terms, and two-factor interaction terms. Then, insignificant terms are eliminated from the regression model, one at a time, until the final model is obtained. The regression model fitting process is conducted, while maintaining the hierarchy of the model, i.e., keeping lower-order term(s) in the presence of a significant higher-order term.
The ANOVA from the final model of the second data fold is shown in Table6. The linear term and the quadratic term of TC are significant, while the HN and LR effects are not. This may be caused by the ranges of HN and LR being too narrow. Further analysis as shown in Figure 6 suggests that a higher value of HN could reduce the MAE, and a high value of LR may already be effective. Therefore, the search direction of subsequent experiments needs to increase HN and LR, while fixing TC. A similar analysis is performed on the other nine data folds, which results in nine additional ANOVA tables and regression models. After examining the results of all 10 folds, four patterns of hyperparameter effects on the MAE of the validation set are summarized in Table 7, along with a suggested search direction.
We denote a quadratic effect of a hyperparameter as convex if the value of the hyperparameter that is associated with a low value of MAE occurs in the middle of the input range. A concave pattern indicates the opposite hyperparameter effect where low values of MAE occur at the boundaries of the input range. In addition, there are two linear trends, a negative trend and a positive trend, which point to a low value of MAE occurring at the upper bound and lower bound, respectively. These patterns also suggest the search directions. For a convex pattern and linear positive trend, the hyperparameter values remain unchanged at the value that gives the minimum MAE and at the lower bound, respectively, in subsequent experiments. However, for a concave pattern and linear negative trend, the search direction is to increase the hyperparameter value. The search direction is defined by the coded coefficients of the final regression model. For example, the final regression model from the second fold of data is as follows:(3)MAE=1.1343−0.0006HN−0.0085TC−0.0016LR+0.0050TC2.
Based on the coded coefficients in the model, every coded unit increase in HN is associated with (-0.0000/-0.0016) = 2.6667 coded units increase in LR. The 2.6667 coded units have to be transformed to an original scale. Recall that in the original scale, LR = 0.00028 is coded as −1 at the factorial point, and LR = 0.00055 is coded as 0 at the center points. In other words, a coded unit increase is equivalent to (0.00055–0.00028) = 0.00027 units in LR. Therefore, in the steepest descent direction, an increase of 2.6667 coded units becomes a (0.00027 × 2.6667) = 0.000714 increase in LR.Table 5
Design and experiment results of Experiment1.
Point typeHyperparameterMAE (validation)HNTCLRFold 1Fold 2Fold 3Fold 4Fold 5Fold 6Fold 7Fold 8Fold 9Fold 1011280.000281.13321.14651.12541.09841.10711.02671.26921.11731.08741.163613280.000281.11461.15021.13341.09321.09681.02021.28431.12941.09791.202811820.000281.12711.13581.0941.07361.09640.99941.24691.11841.0641.139613820.000281.09681.13011.0791.0761.08651.00241.23771.12031.06731.151511280.000821.19431.13551.16341.07191.10131.02671.25631.11921.07411.153413280.000821.11261.13061.07351.07531.08911.02021.26271.11661.06331.142211820.000821.17851.13061.13991.08151.10010.99941.24331.11871.07571.139313820.000821.10021.13381.04091.09361.05761.00241.22821.08571.08491.1382−11550.000551.15681.13441.11771.07281.09981.00911.24981.11861.07151.1446−13550.000551.09941.1281.06691.07661.08211.01071.24961.11021.06691.1451−12100.000551.13341.17781.17291.12571.13361.03871.28841.1641.11211.2212−121000.000551.07831.12751.07861.10461.08680.9781.22321.11271.07931.1793−12550.00011.07831.12751.07861.10461.08680.9781.22321.11271.07931.1793−12550.0011.08521.13351.08731.11121.09740.97731.24381.14511.08331.212302550.000551.09251.14831.17931.08041.09691.00051.25111.23371.15781.143802550.000551.09251.13031.10461.10781.08650.97271.27711.21831.08331.179302550.000551.08821.14331.10251.09481.07020.99041.23961.13371.06751.180602550.000551.09471.13051.07411.08391.08861.01111.23671.19171.07421.137302550.000551.20471.1291.1071.08891.10360.97561.23241.1091.06731.14302550.000551.07771.15291.06631.091.0960.97681.28281.12031.08221.1441Table 6
Analysis of variance of the second fold.
SourceDfAdj SSAdj MSF-valueP valueModel40.0014180.0003553.560.031Linear30.0010430.0003483.490.042HN10.000010.000010.10.754TC10.0009980.00099810.020.006LR10.0000350.0000350.350.562Square10.0003750.0003753.770.071TC × TC10.0003750.0003753.770.071Error150.0014940.0001Lack-of-fit100.0009470.0000950.870.606Pure error50.0005470.000109Total190.002912Figure 6
The response optimizer from the second fold.Table 7
Patterns of hyperparameter effects and search direction.
FoldHidden nodesTraining cyclesLearning rateSearch direction1ConvexLinear (−)Linear (+)Fix HC, TC at minimum MAE and increase TC2Linear (−)ConvexLinear (-)Increase HN, LR together and fix TC at minimum MAE3ConvexLinear (−)Linear (−)Fix HN at minimum MAE and increase TC and LR together4ConcaveConvexConvexIncrease HN and fix TC5Linear (−)ConvexLinear (−)Increase HN and LR together6ConvexConvexLinear (−)Increase LR7Linear (−)ConvexConvexIncrease HN8Linear (−)Linear (−)ConvexIncrease HN and TC9Linear (−)ConcaveLinear (+)Increase HN10ConcaveConvexConvexIncrease HNExperiment 2.
Searching along the steepest descent direction
Search results along the steepest descent direction are given in Figure7. From the results, an HN of 8 and LR of 0.0117 seem to be effective. Therefore, these settings become a new center point for the next experiment.
The process of searching along the path of steepest descent is repeated for each data fold. This process stops when the best hyperparameter values on this path are found. These settings become the center point for further experiments.Figure 7
The search direction from the second fold.Experiment 3.
Finding optimal setting
For the second data fold, another CCD is conducted. The results are shown in Figure8.
The regression model fitting process is performed based on the MAE results and hyperparameter values in Figure8. The ANOVA of the final model is shown in Table 8. The linear terms and quadratic terms of HN and TC are significant, while the LR effects are not significant. The final regression model is then optimized with respect to HN and TC. The best hyperparameter settings from the second data fold are HN = 8, TC = 194, and LR = 0.005095 with the expected value of MAE equal to 1.0900. Similar to the GS, eight confirmation runs are performed.
Experiment3 is repeated for the other nine data folds. A summary of the results is shown in Table 9. The table contains the best values of the hyperparameters for all 10 folds, the validation MAE at these settings from the experiment, and the results from the confirmation run. In the last column, PI coverage indicates whether or not the 95% prediction intervals of MAE contain the validation MAE from the RSM experiments. The results show that in 9 out of 10 folds, the 95% PIs contain the validation MAE. This indicates a 90% reliability of the hyperparameter values obtained from RSM.
Among different folds, HN varies from 4 to 10, and TC varies from 10 to 951. The best LR is the same for all folds, which is 0.005095. Although the best settings of HN and TC are inconclusive, it is noticeable that there is a mild trend between HN and TC, i.e., lower HN tends to be associated with higher TC and vice versa.Figure 8
Design and experiment results of Experiment3.Table 8
ANOVA of the final model of the second data fold.
SourceDfAdj SSAdj MSF-valueP valueModel40.128620.03215510.62<0.001HN10.011480.0114813.790.070TC10.089530.08953429.56<0.001HN × HN10.017620.0176175.820.029TC × TC10.014830.0148354.90.043Error150.045430.003028Lack-of-fit100.033470.0033471.40.373Pure error50.011950.002391Total190.17405Table 9
Best hyperparameter setting and the validation performance of ANN using RSM.
RSMConfirmation runsFoldHNTCLRMAE (validation)Avg MA (validation)ESD of MAE (validation)95% PI half-widthPI coverage148020.0050951.0121.22550.05670.1422No281940.0050951.0871.13640.05030.1262Yes349510.0050951.0341.07810.02630.066Yes452580.0050951.0951.11790.05180.1299Yes510600.0050951.0351.15350.06110.1532Yes69100.0050950.981.08180.05330.1337Yes751090.0050951.1691.27940.06750.1693Yes884560.0050951.0621.1980.06430.1613Yes972080.0050951.0511.10910.03480.0873Yes1041090.0050951.1131.21190.05790.1452YesBold values indicate that this is the combination of hyperparameters that give the minimal average MAE (validation) among all folds.


### 4.4.2. Support Vector Machine (SVM)

Similar to an ANN, the search starts at low values of the initial range of hyperparameter (see Table1). Since the type of kernel function is a categorical factor, the first experiment is conducted separately for the four types of kernel functions.Experiment 4.
Identifying the steepest descent direction
Using a two-factor CCD, a data fold has 13 experimental runs for each kernel function. The total number of initial experimental runs to find paths of steepest descent is equal to 13 × 10 × 4 = 520 runs. After running all experiments, the results in Table10 reveal that dot and radial kernel functions are more effective than ANOVA and Epanechnikov functions, which, therefore, are not further considered.
The same process of RSM described in the previous section is conducted with SVM. Patterns of hyperparameter effects on MAE fordot and radial functions are summarized in Table 11, along with the data fold(s) associated with each pattern in the last two columns.
For subsequent experiments, search along the steepest descent direction.
The process of searching along the steepest descent directions to find the appropriate number of steps to move and to find the optimal settings for all 10 folds is performed in the same manner as that of the ANN. Detailed results are omitted to save space. A summary of the optimal hyperparameter settings for all 10 folds is provided in Table12. Overall, the values of C for dot and radial functions are relatively low, while Conv is inconclusive. As mentioned previously, rerunning SVM at the same settings would give the same prediction value. Therefore, confirmation runs are not performed.
After applying the previous steps to all 10 folds, the best hyperparameters fordot and radial kernel functions are shown in Table 12. The minimum value of MAE is 1.053 for both kernel functions. For the dot function, the best hyperparameter settings are equal to 9 and 0.0307 for the C and Conv, respectively. For the radial function, C and Conv are 6 and 0.001, respectively.Table 10
Experiment 4 results for the four kernel functions.
Kernel functionAverage MAE from the validation setFold 1Fold 2Fold 3Fold 4Fold 5Fold 6Fold 7Fold 8Fold 9Fold 10ANOVA1.19641.29691.16071.17631.20611.22831.30411.29121.24321.249Dot1.10211.15731.10561.10241.18031.08651.321.20161.20161.1701Radial1.21241.21241.11361.07581.12151.15651.2581.18211.14791.1373Epanechnikov1.19631.18851.12721.08871.14851.12711.29441.21571.14941.2193Bold values indicate the minimal average MAE for each data fold (Fold 1 to Fold 10).Table 11
Patterns of hyperparameter effects and search direction fordot and radial kernel function.
CConvSearch directiondotradialLinear (−)Linear (−)Increase C and Conv togetherFold: 1, 5, 6, 10Fold: 1ConcaveLinear (−)Fix C at minimum MAE and increase ConvFold: 2, 7, 9Linear (+)Linear (−)Fix C at minimum MAE and increase ConvFold: 3, 8Linear (+)Linear (+)Fix C and Conv at minimum MAEFold: 4ConvexLinear (+)Fix C and Conv at minimum MAEFold: 2Linear (+)Linear (-)Fix C at lower bound and increase ConvFold: 3ConcaveLinear (+)Fix C and Conv at minimum MAEFold: 4, 5, 9Linear (−)Linear (+)Increase C and fix Conv at minimum MAEFold: 7ConcaveLinear (−)Fix C at lower bound and increase ConvFold: 8, 10Table 12
Best hyperparameter settings fordot and radial kernel functions using RSM.
FoldDotRadialCConvergenceMAE (validation)CConvergenceMAE (validation)1120.0011.1680120.01091.14552−10.11.186830.0011.12373−10.04061.1102−10.07031.15754-10.0011.169460.0011.05295110.01091.120990.0011.1137690.03071.0528170.02081.08927−10.11.2577110.0011.24528−10.05051.123430.01091.16049−10.09011.076590.0011.098810100.04061.192370.11.1827


### 4.4.3. Deep Belief Network (DBN)

The initial ranges of hyperparameters in the first experiment for DBN are listed in Table1. These ranges are also chosen to start at low values. The lower and upper bounds in Table 11 are values for the axial points. Since HN, n, and ni are integers, their values at factorial points are rounded to the nearest integers.Experiment 5.
: A six-factor half-fraction CCD with 53 design points is performed for each data fold. For 10 folds, a total of 53 × 10 = 530 runs are conducted to find the paths of the steepest descent. As an example, ANOVA of the final model from the 2nd data fold is given in Table13. The obtained regression model provides the steepest descent direction to increase LR together with n, while holding the other hyperparameters fixed at their boundaries or at the point where MAE is minimized if the squared term is significant.
The RSM process is conducted with DBN on 10 data folds. Patterns of hyperparameter effects on MAE and the search directions are summarized in Table14. Moving along these steepest descent directions toward the optimal region gives the center points for the subsequent experiments. Then, running another CCD around these center points leads to the final regression models that yield the optimal settings for all 10 data folds. The best hyperparameter settings obtained from the final regression models and the validation MAE at these settings for DBN using RSM are provided in Table 15. The results from confirmation runs, including the average MAE, SD, half-width of 95% PI, and coverage of PI, are also shown. The PI coverages of the DBN hyperparameter settings from RSM indicate 100% reliability based on our testing.Table 13
Analysis of variance of the second fold.
SourceDfAdj SSAdj MSF-valueP valueModel8363.1845.39714.5<0.001HN10.4460.4460.140.708LRRBM10.6520.6520.210.650LR1137.73137.7343.98<0.001N114.96814.9684.780.034ni1153.99153.9949.17<0.001Nb128.91128.9119.230.004HN × HN111.96911.9693.820.057LR ×n114.5114.514.630.037Error44137.83.132Lack-of-fit36111.063.0850.920.603Pure error826.7383.342Total52500.98Table 14
Patterns of hyperparameter effects and search direction for DBN.
FoldHNLRRBMnniLRNbSearch direction1Linear (−)ConcaveLinear (−)Linear (+)Linear (−)Linear (+)Increase HN, N, LR, and fix LRRBM, ni, Nbat min. MAE2ConvexLinear (+)Linear (−)Linear (+)Linear (−)Linear (+)Increase N, LR, and fix HN, LRRBM, ni, Nbat min. MAE3Linear (−)Linear (−)Linear (−)Linear (+)Linear (−)Linear (+)Increase HN, LRRBM, N, LR, and fix ni, Nbat min. MAE4Linear (−)Linear (−)Linear (−)Linear (−)Linear (−)Linear (+)Increase HN, LRRBM, N, ni, LR, and fix, Nbat min. MAE5Linear (+)Linear (−)Linear (−)Linear (−)Linear (−)Linear (+)Increase LRRBM, N, ni, LR, and fix HN, Nbat min. MAE6Linear (−)Linear (+)Linear (−)Linear (−)Linear (−)Linear (+)Increase HN, N,ni, LR, and fix LRRBM, Nbat min. MAE8Linear (−)ConvexLinear (−)ConvexLinear (−)ConcaveIncrease HN, N, LR, and fix LRRBM, ni, Nbat min. MAE9Linear (−)Linear (−)Linear (−)Linear (+)Linear (−)Linear (+)Increase HN, LRRBM, N, LR, and fix ni, Nbat min. MAE10Linear (+)Linear (+)Linear (−)Linear (+)Linear (−)Linear (+)Increase N, LR, and fix HN, LRRBM, ni, Nbat min. MAETable 15
Best hyperparameter setting and the validation performance of DBN using RSM.
Cross-validationConfirmation runsFoldHNninLRRBMLRNbMAEAvg MAESD (MAE)95% PI half-widthPI coverage17389100.00010.006910251.0811.0970.0320.109Yes23530100.0010.010710251.1661.1680.0090.03Yes38575100.0010.00673501.0731.0830.0110.036Yes43254100.0010.006810251.0831.0960.0130.042Yes53209100.0010.007317001.0871.0930.0090.031Yes65209100.0010.00810251.0000.9980.0130.045Yes74434100.0010.009410251.2231.2320.0180.061Yes87344100.0010.007110251.121.140.0350.118Yes96620100.0010.010110251.1311.0890.0280.095Yes106620100.0010.01013501.2991.2030.0640.215YesBold values emphasize that this is the combination of hyperparameters that results in the minimal average MAE (0.998).


## 4.4.1. Artificial Neural Network

From the initial ranges of hyperparameters, listed in Table1, the search process begins at the low values of these hyperparameters, i.e., near the origin point of the hyperparameter space. The lower bound and upper bound are specified for the axial points of the CCD. Since HN and TC are integers, the values of the hyperparameters at factorial points are rounded to the nearest integer.Experiment 1.
Identifying the steepest descent direction
A three-factor CCD with 20 design points is performed for each data fold. Each row in the table represents an experimental run, where point type of 1, −1, and 0 indicates that the experiment is a factorial point, axial point, and center point, respectively. Note that, this CCD is constructed by specifying the axial points for each hyperparameter as follows: 1 and 3 for HN, 10 and 100 for TC, and 0.0001 and 0.001 for LR. Also, the values of HN of some design points are rounded to the nearest integer since HN cannot be a fractional number. The MAEs of the validation set of all 10 folds of data are shown in Table5.
Analysis of variance (ANOVA) is performed on each data fold separately. The steps in the data analysis of the second fold are described as an example. First, the regression model begins as a full-quadratic model, which contains the main effects, squared terms, and two-factor interaction terms. Then, insignificant terms are eliminated from the regression model, one at a time, until the final model is obtained. The regression model fitting process is conducted, while maintaining the hierarchy of the model, i.e., keeping lower-order term(s) in the presence of a significant higher-order term.
The ANOVA from the final model of the second data fold is shown in Table6. The linear term and the quadratic term of TC are significant, while the HN and LR effects are not. This may be caused by the ranges of HN and LR being too narrow. Further analysis as shown in Figure 6 suggests that a higher value of HN could reduce the MAE, and a high value of LR may already be effective. Therefore, the search direction of subsequent experiments needs to increase HN and LR, while fixing TC. A similar analysis is performed on the other nine data folds, which results in nine additional ANOVA tables and regression models. After examining the results of all 10 folds, four patterns of hyperparameter effects on the MAE of the validation set are summarized in Table 7, along with a suggested search direction.
We denote a quadratic effect of a hyperparameter as convex if the value of the hyperparameter that is associated with a low value of MAE occurs in the middle of the input range. A concave pattern indicates the opposite hyperparameter effect where low values of MAE occur at the boundaries of the input range. In addition, there are two linear trends, a negative trend and a positive trend, which point to a low value of MAE occurring at the upper bound and lower bound, respectively. These patterns also suggest the search directions. For a convex pattern and linear positive trend, the hyperparameter values remain unchanged at the value that gives the minimum MAE and at the lower bound, respectively, in subsequent experiments. However, for a concave pattern and linear negative trend, the search direction is to increase the hyperparameter value. The search direction is defined by the coded coefficients of the final regression model. For example, the final regression model from the second fold of data is as follows:(3)MAE=1.1343−0.0006HN−0.0085TC−0.0016LR+0.0050TC2.
Based on the coded coefficients in the model, every coded unit increase in HN is associated with (-0.0000/-0.0016) = 2.6667 coded units increase in LR. The 2.6667 coded units have to be transformed to an original scale. Recall that in the original scale, LR = 0.00028 is coded as −1 at the factorial point, and LR = 0.00055 is coded as 0 at the center points. In other words, a coded unit increase is equivalent to (0.00055–0.00028) = 0.00027 units in LR. Therefore, in the steepest descent direction, an increase of 2.6667 coded units becomes a (0.00027 × 2.6667) = 0.000714 increase in LR.Table 5
Design and experiment results of Experiment1.
Point typeHyperparameterMAE (validation)HNTCLRFold 1Fold 2Fold 3Fold 4Fold 5Fold 6Fold 7Fold 8Fold 9Fold 1011280.000281.13321.14651.12541.09841.10711.02671.26921.11731.08741.163613280.000281.11461.15021.13341.09321.09681.02021.28431.12941.09791.202811820.000281.12711.13581.0941.07361.09640.99941.24691.11841.0641.139613820.000281.09681.13011.0791.0761.08651.00241.23771.12031.06731.151511280.000821.19431.13551.16341.07191.10131.02671.25631.11921.07411.153413280.000821.11261.13061.07351.07531.08911.02021.26271.11661.06331.142211820.000821.17851.13061.13991.08151.10010.99941.24331.11871.07571.139313820.000821.10021.13381.04091.09361.05761.00241.22821.08571.08491.1382−11550.000551.15681.13441.11771.07281.09981.00911.24981.11861.07151.1446−13550.000551.09941.1281.06691.07661.08211.01071.24961.11021.06691.1451−12100.000551.13341.17781.17291.12571.13361.03871.28841.1641.11211.2212−121000.000551.07831.12751.07861.10461.08680.9781.22321.11271.07931.1793−12550.00011.07831.12751.07861.10461.08680.9781.22321.11271.07931.1793−12550.0011.08521.13351.08731.11121.09740.97731.24381.14511.08331.212302550.000551.09251.14831.17931.08041.09691.00051.25111.23371.15781.143802550.000551.09251.13031.10461.10781.08650.97271.27711.21831.08331.179302550.000551.08821.14331.10251.09481.07020.99041.23961.13371.06751.180602550.000551.09471.13051.07411.08391.08861.01111.23671.19171.07421.137302550.000551.20471.1291.1071.08891.10360.97561.23241.1091.06731.14302550.000551.07771.15291.06631.091.0960.97681.28281.12031.08221.1441Table 6
Analysis of variance of the second fold.
SourceDfAdj SSAdj MSF-valueP valueModel40.0014180.0003553.560.031Linear30.0010430.0003483.490.042HN10.000010.000010.10.754TC10.0009980.00099810.020.006LR10.0000350.0000350.350.562Square10.0003750.0003753.770.071TC × TC10.0003750.0003753.770.071Error150.0014940.0001Lack-of-fit100.0009470.0000950.870.606Pure error50.0005470.000109Total190.002912Figure 6
The response optimizer from the second fold.Table 7
Patterns of hyperparameter effects and search direction.
FoldHidden nodesTraining cyclesLearning rateSearch direction1ConvexLinear (−)Linear (+)Fix HC, TC at minimum MAE and increase TC2Linear (−)ConvexLinear (-)Increase HN, LR together and fix TC at minimum MAE3ConvexLinear (−)Linear (−)Fix HN at minimum MAE and increase TC and LR together4ConcaveConvexConvexIncrease HN and fix TC5Linear (−)ConvexLinear (−)Increase HN and LR together6ConvexConvexLinear (−)Increase LR7Linear (−)ConvexConvexIncrease HN8Linear (−)Linear (−)ConvexIncrease HN and TC9Linear (−)ConcaveLinear (+)Increase HN10ConcaveConvexConvexIncrease HNExperiment 2.
Searching along the steepest descent direction
Search results along the steepest descent direction are given in Figure7. From the results, an HN of 8 and LR of 0.0117 seem to be effective. Therefore, these settings become a new center point for the next experiment.
The process of searching along the path of steepest descent is repeated for each data fold. This process stops when the best hyperparameter values on this path are found. These settings become the center point for further experiments.Figure 7
The search direction from the second fold.Experiment 3.
Finding optimal setting
For the second data fold, another CCD is conducted. The results are shown in Figure8.
The regression model fitting process is performed based on the MAE results and hyperparameter values in Figure8. The ANOVA of the final model is shown in Table 8. The linear terms and quadratic terms of HN and TC are significant, while the LR effects are not significant. The final regression model is then optimized with respect to HN and TC. The best hyperparameter settings from the second data fold are HN = 8, TC = 194, and LR = 0.005095 with the expected value of MAE equal to 1.0900. Similar to the GS, eight confirmation runs are performed.
Experiment3 is repeated for the other nine data folds. A summary of the results is shown in Table 9. The table contains the best values of the hyperparameters for all 10 folds, the validation MAE at these settings from the experiment, and the results from the confirmation run. In the last column, PI coverage indicates whether or not the 95% prediction intervals of MAE contain the validation MAE from the RSM experiments. The results show that in 9 out of 10 folds, the 95% PIs contain the validation MAE. This indicates a 90% reliability of the hyperparameter values obtained from RSM.
Among different folds, HN varies from 4 to 10, and TC varies from 10 to 951. The best LR is the same for all folds, which is 0.005095. Although the best settings of HN and TC are inconclusive, it is noticeable that there is a mild trend between HN and TC, i.e., lower HN tends to be associated with higher TC and vice versa.Figure 8
Design and experiment results of Experiment3.Table 8
ANOVA of the final model of the second data fold.
SourceDfAdj SSAdj MSF-valueP valueModel40.128620.03215510.62<0.001HN10.011480.0114813.790.070TC10.089530.08953429.56<0.001HN × HN10.017620.0176175.820.029TC × TC10.014830.0148354.90.043Error150.045430.003028Lack-of-fit100.033470.0033471.40.373Pure error50.011950.002391Total190.17405Table 9
Best hyperparameter setting and the validation performance of ANN using RSM.
RSMConfirmation runsFoldHNTCLRMAE (validation)Avg MA (validation)ESD of MAE (validation)95% PI half-widthPI coverage148020.0050951.0121.22550.05670.1422No281940.0050951.0871.13640.05030.1262Yes349510.0050951.0341.07810.02630.066Yes452580.0050951.0951.11790.05180.1299Yes510600.0050951.0351.15350.06110.1532Yes69100.0050950.981.08180.05330.1337Yes751090.0050951.1691.27940.06750.1693Yes884560.0050951.0621.1980.06430.1613Yes972080.0050951.0511.10910.03480.0873Yes1041090.0050951.1131.21190.05790.1452YesBold values indicate that this is the combination of hyperparameters that give the minimal average MAE (validation) among all folds.


## 4.4.2. Support Vector Machine (SVM)

Similar to an ANN, the search starts at low values of the initial range of hyperparameter (see Table1). Since the type of kernel function is a categorical factor, the first experiment is conducted separately for the four types of kernel functions.Experiment 4.
Identifying the steepest descent direction
Using a two-factor CCD, a data fold has 13 experimental runs for each kernel function. The total number of initial experimental runs to find paths of steepest descent is equal to 13 × 10 × 4 = 520 runs. After running all experiments, the results in Table10 reveal that dot and radial kernel functions are more effective than ANOVA and Epanechnikov functions, which, therefore, are not further considered.
The same process of RSM described in the previous section is conducted with SVM. Patterns of hyperparameter effects on MAE fordot and radial functions are summarized in Table 11, along with the data fold(s) associated with each pattern in the last two columns.
For subsequent experiments, search along the steepest descent direction.
The process of searching along the steepest descent directions to find the appropriate number of steps to move and to find the optimal settings for all 10 folds is performed in the same manner as that of the ANN. Detailed results are omitted to save space. A summary of the optimal hyperparameter settings for all 10 folds is provided in Table12. Overall, the values of C for dot and radial functions are relatively low, while Conv is inconclusive. As mentioned previously, rerunning SVM at the same settings would give the same prediction value. Therefore, confirmation runs are not performed.
After applying the previous steps to all 10 folds, the best hyperparameters fordot and radial kernel functions are shown in Table 12. The minimum value of MAE is 1.053 for both kernel functions. For the dot function, the best hyperparameter settings are equal to 9 and 0.0307 for the C and Conv, respectively. For the radial function, C and Conv are 6 and 0.001, respectively.Table 10
Experiment 4 results for the four kernel functions.
Kernel functionAverage MAE from the validation setFold 1Fold 2Fold 3Fold 4Fold 5Fold 6Fold 7Fold 8Fold 9Fold 10ANOVA1.19641.29691.16071.17631.20611.22831.30411.29121.24321.249Dot1.10211.15731.10561.10241.18031.08651.321.20161.20161.1701Radial1.21241.21241.11361.07581.12151.15651.2581.18211.14791.1373Epanechnikov1.19631.18851.12721.08871.14851.12711.29441.21571.14941.2193Bold values indicate the minimal average MAE for each data fold (Fold 1 to Fold 10).Table 11
Patterns of hyperparameter effects and search direction fordot and radial kernel function.
CConvSearch directiondotradialLinear (−)Linear (−)Increase C and Conv togetherFold: 1, 5, 6, 10Fold: 1ConcaveLinear (−)Fix C at minimum MAE and increase ConvFold: 2, 7, 9Linear (+)Linear (−)Fix C at minimum MAE and increase ConvFold: 3, 8Linear (+)Linear (+)Fix C and Conv at minimum MAEFold: 4ConvexLinear (+)Fix C and Conv at minimum MAEFold: 2Linear (+)Linear (-)Fix C at lower bound and increase ConvFold: 3ConcaveLinear (+)Fix C and Conv at minimum MAEFold: 4, 5, 9Linear (−)Linear (+)Increase C and fix Conv at minimum MAEFold: 7ConcaveLinear (−)Fix C at lower bound and increase ConvFold: 8, 10Table 12
Best hyperparameter settings fordot and radial kernel functions using RSM.
FoldDotRadialCConvergenceMAE (validation)CConvergenceMAE (validation)1120.0011.1680120.01091.14552−10.11.186830.0011.12373−10.04061.1102−10.07031.15754-10.0011.169460.0011.05295110.01091.120990.0011.1137690.03071.0528170.02081.08927−10.11.2577110.0011.24528−10.05051.123430.01091.16049−10.09011.076590.0011.098810100.04061.192370.11.1827


## 4.4.3. Deep Belief Network (DBN)

The initial ranges of hyperparameters in the first experiment for DBN are listed in Table1. These ranges are also chosen to start at low values. The lower and upper bounds in Table 11 are values for the axial points. Since HN, n, and ni are integers, their values at factorial points are rounded to the nearest integers.Experiment 5.
: A six-factor half-fraction CCD with 53 design points is performed for each data fold. For 10 folds, a total of 53 × 10 = 530 runs are conducted to find the paths of the steepest descent. As an example, ANOVA of the final model from the 2nd data fold is given in Table13. The obtained regression model provides the steepest descent direction to increase LR together with n, while holding the other hyperparameters fixed at their boundaries or at the point where MAE is minimized if the squared term is significant.
The RSM process is conducted with DBN on 10 data folds. Patterns of hyperparameter effects on MAE and the search directions are summarized in Table14. Moving along these steepest descent directions toward the optimal region gives the center points for the subsequent experiments. Then, running another CCD around these center points leads to the final regression models that yield the optimal settings for all 10 data folds. The best hyperparameter settings obtained from the final regression models and the validation MAE at these settings for DBN using RSM are provided in Table 15. The results from confirmation runs, including the average MAE, SD, half-width of 95% PI, and coverage of PI, are also shown. The PI coverages of the DBN hyperparameter settings from RSM indicate 100% reliability based on our testing.Table 13
Analysis of variance of the second fold.
SourceDfAdj SSAdj MSF-valueP valueModel8363.1845.39714.5<0.001HN10.4460.4460.140.708LRRBM10.6520.6520.210.650LR1137.73137.7343.98<0.001N114.96814.9684.780.034ni1153.99153.9949.17<0.001Nb128.91128.9119.230.004HN × HN111.96911.9693.820.057LR ×n114.5114.514.630.037Error44137.83.132Lack-of-fit36111.063.0850.920.603Pure error826.7383.342Total52500.98Table 14
Patterns of hyperparameter effects and search direction for DBN.
FoldHNLRRBMnniLRNbSearch direction1Linear (−)ConcaveLinear (−)Linear (+)Linear (−)Linear (+)Increase HN, N, LR, and fix LRRBM, ni, Nbat min. MAE2ConvexLinear (+)Linear (−)Linear (+)Linear (−)Linear (+)Increase N, LR, and fix HN, LRRBM, ni, Nbat min. MAE3Linear (−)Linear (−)Linear (−)Linear (+)Linear (−)Linear (+)Increase HN, LRRBM, N, LR, and fix ni, Nbat min. MAE4Linear (−)Linear (−)Linear (−)Linear (−)Linear (−)Linear (+)Increase HN, LRRBM, N, ni, LR, and fix, Nbat min. MAE5Linear (+)Linear (−)Linear (−)Linear (−)Linear (−)Linear (+)Increase LRRBM, N, ni, LR, and fix HN, Nbat min. MAE6Linear (−)Linear (+)Linear (−)Linear (−)Linear (−)Linear (+)Increase HN, N,ni, LR, and fix LRRBM, Nbat min. MAE8Linear (−)ConvexLinear (−)ConvexLinear (−)ConcaveIncrease HN, N, LR, and fix LRRBM, ni, Nbat min. MAE9Linear (−)Linear (−)Linear (−)Linear (+)Linear (−)Linear (+)Increase HN, LRRBM, N, LR, and fix ni, Nbat min. MAE10Linear (+)Linear (+)Linear (−)Linear (+)Linear (−)Linear (+)Increase N, LR, and fix HN, LRRBM, ni, Nbat min. MAETable 15
Best hyperparameter setting and the validation performance of DBN using RSM.
Cross-validationConfirmation runsFoldHNninLRRBMLRNbMAEAvg MAESD (MAE)95% PI half-widthPI coverage17389100.00010.006910251.0811.0970.0320.109Yes23530100.0010.010710251.1661.1680.0090.03Yes38575100.0010.00673501.0731.0830.0110.036Yes43254100.0010.006810251.0831.0960.0130.042Yes53209100.0010.007317001.0871.0930.0090.031Yes65209100.0010.00810251.0000.9980.0130.045Yes74434100.0010.009410251.2231.2320.0180.061Yes87344100.0010.007110251.121.140.0350.118Yes96620100.0010.010110251.1311.0890.0280.095Yes106620100.0010.01013501.2991.2030.0640.215YesBold values emphasize that this is the combination of hyperparameters that results in the minimal average MAE (0.998).


## 4.5. Comparison between GS and RSM

ANOVA is performed to compare the confirmation MAE between GS and RSM (denoted as Method) for ANN, SVM, and DBN (denoted as ML) while treating data folds as blocks (see Table16). The interaction between method and fold is mildly significant (P value = 0.094), which indicates that the average MAE of validation set between GS and RSM is statistically different in some data folds. In Table 17, further analysis using Tukey’s multiple comparisons shows that the difference is significant only for the first data fold, i.e., the hyperparameters from GS give statistically better results (on average) than those of RSM. For the other nine data folds, the hyperparameter settings from GS and RSM perform statistically the same.Table 16
ANOVA of confirmation MAE of validation set by hyperparameter tuning methods.
SourceDfAdj SSAdj MSF-valueP valueFold90.1734340.0192734.08<0.001ML20.0192890.00964517.06<0.001Method10.0331920.03319258.7<0.001Fold × method90.0093150.0010351.830.094Error380.0214860.000565Total590.256716Table 17
Tukey’s multiple comparison of the confirmation MAE between GS and RSM.
Fold 1Fold 2Fold 3Fold 4Fold 5Fold 6Fold 7Fold 8Fold 9Fold 10GSAAAAAAAAAARSMBAAAAABAAANote: Means that do not share the same letter are significantly different.To investigate the reliability of using the best hyperparameters from RSM to predict unforeseen data, the three ML models with the best hyperparameters from GS and RSM are applied to predict the results in the testing dataset. Then, ANOVA is conducted to compare the average MAE of the testing set between GS and RSM (see Table18). The ANOVA shows that there is no significant difference in the average MAE between the two hyperparameter tuning methods (i.e., GS and RSM). Therefore, this implies that the hyperparameter setting from RSM is reliable in the testing set as well.Table 18
ANOVA on MAE of testing set by hyperparameter tuning methods: GS and RSM.
SourceDfAdj SSAdj MSF-valueP valueFold90.17230.019110.47<0.001ML20.03670.018310.03<0.001Tuning method10.00440.00442.410.127Error470.0859690.001829Total590.299374


## 4.6. Discussion

Grid search (GS) is widely used in several studies as a common hyperparameter tuning method. As shown in section4.3, implementing GS in ANN, SVM, and DBN requires many experiments, i.e., 44,100 runs, 44,880 runs, and 7,290 runs, respectively. The results in section 4.5 indicate that on average, the prediction performance of hyperparameter settings obtained from GS and RSM is statistically the same in all three ML models, while RSM requires a smaller number of runs. Specifically, with RSM, the number of runs for ANN, SVM, and DBN is 976 runs (97.79% saving over GS), 984 runs (97.81% saving), and 1,408 runs (80.69% saving), respectively. In other words, the benefit from using RSM has been demonstrated that it can significantly reduce the number of experiments to determine the suitable hyperparameter setting while maintaining the prediction performance.Finally, it is worth noting that RSM, by nature, involves sequential experimentation. That is, RSM requires a modeler to run some experiments and perform data analysis to determine whether the subsequent experiment is needed. If it is the case, then findings from one experiment will guide the search in a subsequent experiment. A limitation of RSM is that this search process is not automated or has not been integrated as a hyperparameter tuning process in standard ML software.


## 5. Conclusion

Hyperparameter tuning is an essential process in ML algorithms. To obtain proper hyperparameter settings, GS is the common approach in many applications due to its simplicity to implement but with a drawback of many experimental runs. In this study, we propose a hyperparameter tuning process using the RSM approach. RSM is compared with GS on three ML algorithms: ANN, SVM, and DBN. A 10-fold cross-validation is performed to make sure that the ML performance is robust to the data partitioning process. The dataset used in the computational test is from an industrial user in the food industry. The purpose of the analysis is to predict the quality of the raw material in the production process. The MAE is considered a key measure. The MAE from the validation set is used to evaluate the prediction performance between the two hyperparameter tuning processes.The comparison results indicate that the widely used GS method can give hyperparameter settings that are 80% reliable for both ANN and DBN, whereas the hyperparameter tuning process by RSM can give 90% and 100% reliability for ANN and DBN, respectively. In addition, the prediction performance from GS and RSM is directly compared using statistical analysis. The results show that the hyperparameter settings obtained from RSM can give statistically similar performance to those from GS in 9 out of 10 data folds and 7 out of 10 data folds for ANN and DBN, respectively. Although the performance from GS is slightly better than RSM, the savings in the number of runs from GS are 97.79%, 97.81%, and 80.6% for ANN, SVM, and DBN, respectively. Therefore, this demonstrates the effectiveness of using RSM as the hyperparameter tuning process for the three ML algorithms tested in this study.We have demonstrated the robustness of the proposed method’s effectiveness by testing it with three machine learning models having different characteristics: ANN with numerical hyperparameters, SVM with a mix between numerical hyperparameter and categorical (kernel function), and DBN with many numerical hyperparameters. The fundamental result is that the proposed method can be applied to ML algorithms with various characteristics for predicting a numerical response. This fundamental result also indicates the limitation of RSM that it cannot be applied to other types of responses such as binary and categorical responses found in classification problems. Directions for future research are to test the RSM tuning process in other ML algorithms and/or in other industrial applications to see whether our findings can be useful for other ML algorithms as well as other industries. [48].



---

*Source: 8513719-2022-01-17.xml*