# Optimizing the Parameters of Long Short-Term Memory Networks Using the Bees Algorithm
## 3 Proposed Novel Hybrid BA-CNN-LSTM Algorithm
### CNN Architecture  
The features in artificial porosity images were extracted using the CNN, which uses filters in the convolutional layer to activate some features in the images. The architecture of the CNN consists of nineteen layers, starting with five convolutional layers that are each accompanied by a rectified linear unit layer and a batch normalization layer with four average pooling layers in between.  
The filter in the convolutional layers is a matrix that contains weights. This matrix is multiplied by the input matrix that will be described in the next section, which results in a matrix called a feature map [31]. Each convolutional layer is accompanied by a rectified linear unit layer that maps the negative values to zero and keeps the positive values for faster effective training [32]. Also, each convolutional layer is followed by a batch normalization layer to minimize the issue of overfitting [33]. The pooling layers come between the convolutions in order to reduce the output dimensions while keeping the features of interest in the images [34]. This reduction contributes to minimizing the computational cost.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Hybrid Algorithm**} & \multirow{2}{*}{**Purpose**} & **Original** & \multirow{2}{*}{**Hybrid Algorithm Accuracy**} & \multirow{2}{*}{**RMSE Difference**} & \multirow{2}{*}{**Reference**} \\  & & **Algorithm** & & & \\  & & **Accuracy** & & & \\ \hline \multirow{3}{*}{GA-LSTM} & Predictive & - & 98.14\% & - & [22] \\  & maintenance & & 56\% & - & [23] \\  & Word prediction & - & - & 0.0007 & [24] \\ \hline \multirow{3}{*}{PSO-LSTM} & Pollution level & - & - & 0.0007 & [24] \\  & prediction & - & 95.02\% & - & [25] \\ \cline{1-1}  & New intrusion & & 90\% & 90\% & - & [26] \\ \cline{1-1}  & detection & & 95\% & 97\% & - & [29] \\ \hline \hline \end{tabular}
\end{table}
Table 1: State-of-the-art studies.  
The normal filter size is 3 \(\times\) 3 or 5 \(\times\) 5, as advised in [32]. After trying both sizes, the 5 \(\times\) 5 size was selected for all five convolutional layers, and the number of filters was 8, 16, 32, 64, and 128, so each layer is double the previous filter number, as advised in [10]. The edges of the images were detected using padding with a stride value of one, so the pixel shift is one cell. The pooling size is 4 \(\times\) 4 with a four-stride value to reduce the computational time, since the image size is large (650 \(\times\) 630 \(\times\) 3). There are two types of pooling. Average pooling calculates the average pixel brightness value of the four numbers in the pooling matrix. It is compatible with lighter backgrounds, so it was used in the architecture rather than the max pooling type, which selects the maximum pixel value by taking the most activated feature, which works better with a dark background [35].  
The most popular training algorithm is the stochastic gradient descent with momentum (SGDM) [33], so it was used to train on the artificial porosity images with 20 epochs [36]. A section depth value of 1 was selected for network depth controlling, with a 0.0101 initial learning rate value that allowed for feature learning, a 0.9568 momentum value for parameters updating, and a 0.0097 regularization value to mitigate the overfitting risk [37]. They were the same set of parameters found using the BA in the hybrid Bees Algorithm Regression Convolutional Neural Network (BA-RCNN) developed in [38]. The hybrid BA-RCNN was used to predict the porosity percentage in each layer of the SLM part. Since the layers are sequential, the LSTM was added after the CNN to deal with sequential data for better prediction accuracy, as will be described in the following subsection.