# Optimizing the Parameters of Long Short-Term Memory Networks Using the Bees Algorithm
## 1 Introduction  
Artificial intelligence (AI) facilitates intelligent systems improvement and leads to an increase in the effectiveness and efficiency of processes [1]. AI has different techniques that can be applied in different contexts in real life, and the most popular ones are Artificial Neural Networks (ANNs), which are inspired by the human brain. ANNs can model high dimensional data, extract implicit patterns, and investigate complex relationships to predict a complex systems' future state [2]. Additionally, they can deal with nonlinear dynamic problems [3]. Deep Learning (DL) is an extension of ANN with better learning capability,as it extracts the features automatically using nonlinear filters [4]. It can be applied in different fields, such as in manufacturing, thereby enabling advanced analytics for big data that result in smart manufacturing systems, whether during processing by diagnosing processes conditions to predict the future state of the products, or during postprocessing to the finished parts [5]. In addition, it can be applied in the pharmaceutical field to diagnose cancers and predict aqueous solubility [6]. On the other hand, there are some disadvantages of adopting DL. The most important one is the acquisition of datasets, as DL requires a large amount of data, so the availability is not guaranteed. In addition, applying data preprocessing for big data is challenging, especially if many irrelevant observations need to be removed, and applying improper data mining may affect the performance of the DL model [3]. Long Short-Term Memory (LSTM) is one of the DL networks that deals with time series or sequential data. It is an advanced Recurrent Neural Network (RNN) that retains information for a long period so that it can remember long-term dependencies [7]. The RNN can retain the information as well, but for a shorter period, so it cannot remember the dependencies in the long term. LSTM was established to address this problem in the RNN [8]. It consists of three gates: the first one is the forget gate that decides if it is necessary to remember the information coming from the previous time scale or not; the second one is the input gate that learns information from the cell input; and the last gate is the output gate, wherein the cell transfers the updated information to the next time cycle. In addition, it has a cell state that carries the information, along with all cycles, and a hidden state for short-term memory [9]. Improving the performance of the LSTM network is an ongoing challenge [10]. The traditional approach of assigning the parameters using trial and error is less accurate, since it depends on the user experience. However, using a nature-inspired algorithm to automatically select the optimum parameter values may improve the performance of the network [10]. The existing studies addressed optimizing hidden layers, the number of neurons, activation function, loss function, optimizers, batch size, and the number of epochs using different nature-inspired algorithms, as will be shown in Section 2.3. This study addresses optimizing the learning rate in each of the three gates and the cell candidate so that each step has a customized learning rate for more optimum performance. In particular, the weights update in each of the three gates and cell candidate depends on the learning rate value [9], so using a customized learning rate for each part would result in more optimal updates for the weights [10]. A study proposed a nature-inspired algorithm that utilized the power of global, local, and intense searches in the BA to optimize the learning rate factor for the convolutional layers and the fully connected layer in order to adjust the global learning for a CNN, which resulted in a more optimum weights update [10]. When the proposed hybrid algorithm was applied to 'Cifar10 DataDir', the validation accuracy was improved from 80.72% to 82.22% [10].