# Data-Driven Predictive Modeling of Neuronal Dynamics Using Long Short-Term Memory
## 2 Neural Network Architecture, Algorithm and Approach
### Deep LSTM Neural Network Architecture  
Long short-term memory (LSTM) neural networks [29] are a particular type of recurrent neural networks (RNNs) which mitigate the vanishing or exploding gradient problem during the network training while capturing both the long-term and the short-term temporal features in sequential time-series data processing [19]. Specifically, LSTM uses multiple gating variables that control the flow of information of a hidden cell state and assign temporal importance to the dynamical features that are present in the time series data flowing through the cell state. Figure 2 shows a schematic illustrating the internal gating operation in a single LSTM cell. A forward pass of information through a single LSTM cell is described by the following cell and gating state equations (reference):  
\[c_{t}=f_{t}\circ c_{t-1}+i_{t}\circ\sigma_{c}(W_{c}x_{t}+U_{c}h_{t-1}+b_{c}), \tag{1a}\] \[h_{t}=o_{t}\circ\sigma_{c}(c_{t}). \tag{1b}\]  
Figure 2: A schematic illustrating the internal gating operation in a single LSTM cell. The “\(+\)” represents an additive operation and the “\(\circ\)” represents a multiplicative operation. \(\sigma_{g}\) is the sigmoidal activation function and \(\sigma_{c}\) is the hyperbolic tangent activation function. \[f_{t} =\sigma_{g}(W_{f}x_{t}+U_{f}h_{t-1}+b_{f}), \tag{1c}\] \[i_{t} =\sigma_{g}(W_{i}x_{t}+U_{i}h_{t-1}+b_{i}),\] (1d) \[o_{t} =\sigma_{g}(W_{o}x_{t}+U_{o}h_{t-1}+b_{o}), \tag{1e}\]  
in Equations (1a)- (1e), \(c_{t}\in\mathbb{R}^{h}\) and \(h_{t}\in\mathbb{R}^{h}\) represent the cell state vector and the hidden state vector, respectively, at time \(t\). \(f_{t}\in\mathbb{R}^{h}\), \(i_{t}\in\mathbb{R}^{h}\), and \(o_{t}\in\mathbb{R}^{h}\) are the \"forget gate\", \"input gate\", and \"output gate\" activation vector, respectively, at time \(t\). \(x_{t}\in\mathbb{R}^{d}\) is the input vector to the LSTM unit at time \(t\), and \(h_{t-1}\in\mathbb{R}^{h}\) is the previous time step hidden state vector passed back into the LSTM unit at time \(t\). The matrices \(W_{f}\), \(W_{i}\), and \(W_{o}\) represent the input weights for the \"forget gate\", \"input gate\", and \"output gate\", respectively. The matrices \(U_{f}\), \(U_{i}\) and \(U_{o}\) represent the weights of the recurrent connections for the \"forget gate\", \"input gate\", and \"output gate\", respectively. The vectors \(b_{f}\)\(b_{i}\), and \(b_{o}\) represent the \"forget gate\", \"input gate\", and \"output gate\" biases, respectively. \(\circ\) represents the element-wise multiplication. The function \(\sigma_{g}\) represents the sigmoidal activation function, and \(\sigma_{c}\) is the hyperbolic tangent activation functions. In this paper, we use stacked LSTM network integrated with a fully connected feedforward output layer to make multi-timestep state predictions. The use of a single feedforward dense output layer allows the network to effectively learn the static input-output features, while the stacked LSTM network captures the temporal dynamical features. To appropriately select the optimum dimensionality of the hidden states in a single hidden layer, we systematically varied the number of hidden states in a sequence of {\(n\), \(n^{2}\), \(2n^{2}\), \(4n^{2}\), \(\cdots\)}, where \(n\) is the dimension of the system's state and evaluated the training performance for each case. We found that for our application (\(n=9\)), a hidden state dimensionality of \(4n^{2}=324\) was optimal in learning dynamical behaviors while avoiding overfitting. To select the number of hidden layers, we systematically increased the number of hidden layers of identical hidden state dimensionality (i.e., 324 states) and compared the network performance during the training. We found that increasing the number of hidden layers beyond 3 layers did not improve the network performance on the training and validation dataset. Thus, we fixed the number of hidden layers to 3 in our study.