# Optimizing the Parameters of Long Short-Term Memory Networks Using the Bees Algorithm
## 2 Long Short-Term Memory (LSTM)
### State-Of-The-Art Studies  
Nature-inspired algorithms have a great impact on optimizing the parameters of the DL models. The need for trial and error when selecting the parameters is reduced, as the nature-inspired algorithms recognize the parameters automatically [21].  
The Genetic Algorithm (GA) is one of the nature-inspired algorithms used to find the optimal parameters in the LSTM network for predictive maintenance [22]. The authors optimized the time steps, the number of LSTM layers, and the number of hidden neurons in each layer by using the GA. They suggested a procedure that starts with population initialization, followed by fitness computation for chromosomes, and finally concludes with genetic operator applications for new population creation, if needed. The design of the GA is based on chromosome structure, fitness function, crossover operators, mutation operators, and population updating and termination [22]. The proposed model achieved an accuracy of 98.14%.  
Similarly, another study used the GA to optimize five parameters related to LSTM hidden layer size, the number of hidden layers, batch size, the number of times steps, and the number of epochs. The hybrid GA-LSTM was used to predict the next word in the sentence, which achieved an accuracy of 56% [23].  
Furthermore, a swarm-based metaheuristic optimization algorithm called Particle Swarm Optimization (PSO) was applied to improve the performance of the LSTM network by optimizing hidden layers, the number of neurons, activation function, the loss function, optimizers, batch size, and the number of epochs. The hybrid PSO-LSTM was applied to predict a pollution level based on a weather dataset; it achieved a lower RMSE than the original LSTM by a value of 0.0007 [24].  
A study used the Artificial Bee Colony (ABC) to optimize the weights of an ANN [25]. The author used the hybrid algorithm to propose a new intrusion detection system that achieved an accuracy of 95.02% [25]. Another study used the BA to train an RNN for sentiment classification, and it improved the accuracy from 60% for the traditional RNN to 90% for the BA-RNN algorithm [26]. Additionally, the ABC was used to optimize LSTM parameters (window size, LSTM units, dropout probability, number of epochs, batch size, and global learning rate) [27]. They used the hybrid ABC-LSTM algorithm for stock market prediction, which achieved a lower RMSE by 5.6836 [27].  
In addition, the LSTM can be integrated with another DL network through the CNN, which is used if the input data are images in order to extract features and feed LSTMs that deal with sequential data [28]. A hybrid CNN-LSTM analyzed motion data in a video to recognize unsafe actions performed by workers. Only applying the CNN resulted in an accuracy of 82%, but adding the LSTM improved the model accuracy to 97%, as it stored the information for a long period, which made it possible to consider the long-term dependencies [28]. The hybrid CNN-LSTM was integrated as well with the ABC to optimize the type of network, the number of epochs, the LSTM hidden units, the global learning rate activation function, the step size, the fully connected layer, and the pooling size [29]. The authors used the hybrid ABC-CNN-LSTM algorithm to detect fake reviews of a product and resulted in an accuracy of 97% compared to 95% for the CNN-LSTM algorithm [29].  
The following Table 1 summarizes the state-of-the-art studies. It shows the purpose of the study, along with the available model accuracy or RMSE difference between the original and hybrid algorithms.  
When looking at the literature, there is a lack of optimization of the learning rate adjustment factor for each gate in the LSTM network, which is an important gap, as the learning rate controls the weight update. Having a more optimum learning rate means more optimum updates for the network weight [11]. The following Section 3 will explain a hybrid BA-CNN-LSTM algorithm that addresses this gap in detail.