# Optimizing the Parameters of Long Short-Term Memory Networks Using the Bees Algorithm
## 3 Proposed Novel Hybrid BA-CNN-LSTM Algorithm
### BA Process  
The BA is a swarm-based optimization technique that performs global and local searches to find the optimal solution. It is an iterative process that starts with performing a global search using scout bees. They use the fitness value (validation error) to evaluate different positions. Since the objective function of this optimization problem is to minimize the validation error, it is a complex function that cannot be formulated. After that, the local search selects the best sites and abandons the remaining sites. Then, the intense search selects the best among the selected sites to recruit more bees in the most promising sites based on the neighborhood size and there being fewer bees in the other sites to conduct the local search that is performed simultaneously with the global search. The process is terminated if the optimal solution is reached, the number of iterations is exceeded, or if there is no improvement over sequential specified iterations [42].  
The following Table 2 shows the values of BA parameters, which were assigned based on the capability of the computer and using the equations stated in [43].  
The challenge is in using the BA process to find the optimal values for nine LSTM parameters to improve the performance of three gates and the cell state, which is specifically achieved by optimizing the learning rate factor so that each part has its own learning rate that is determined based on the global learning rate. Having a more optimum learning rate means more optimum updates for the network weight described in Section 3.2[11]. The bidirectional LSTM layer was used, as described in Section 3.2, which trains the input as is and on the reverse copy of the input [30], this resulted in four parameters related for the forward side, the other four parameters related to the backward side, and one parameter related to the fully connected layer. Consequently, the nine optimization variables became the following:  
* Learning rate factor for input gate (Forward);
* Learning rate factor for forget gate (Forward);
* Learning rate factor for cell candidate (Forward);
* Learning rate factor for output gate (Forward);
* Learning rate factor for input gate (Backward);
* Learning rate factor for forget gate (Backward);
* Learning rate factor for cell candidate (Backward);
* Learning rate factor for output gate (Backward);
* Learning rate factor for fully connected layer.  
The optimal adjustment factors obtained by the BA (ranging between 0.9 and 1.1) were multiplied by the global learning rate of 0.0101 [38] to result in a more optimum learning rate for each part stated above so that the network weights described in Section 3.2 were updated with more optimum values to improve the performance of the LSTM.  
The pseudo-code for the proposed novel hybrid BA-CNN-LSTM algorithm is presented in the following Figure 6. Reference [38] was used to develop this figure.  
\begin{table}
\begin{tabular}{c c c c} \hline
**BA Parameter** & **Symbol** & **Equation** & **Value** \\ \hline Scout bees & n & - & 4 \\ Selected bees & m & (1/2) x m & 2 \\ Elite bees & e & - & 1 \\ Recruited bees for elite sites & nep & 2 x m & 4 \\ Recruited bees for other sites & nsp & (1/2) x n & 2 \\ Neighborhood size & ngh & 0.1 x (Maximum – Minimum) = 0.1 x (1.1 – 0.9) & 0.02 \\ \hline \end{tabular}
\end{table}
Table 2: The values of BA parameters.  
In addition to the pseudo-code, the following Figure 7 shows a flow chart for the workflow diagram for the proposed novel hybrid BA-CNN-LSTM algorithm.  
Figure 6: The pseudo-code for the proposed novel hybrid BA-CNN-LSTM algorithm.  
Figure 7: Workflow diagram for the proposed novel hybrid BA-CNN-LSTM algorithm.