# Data-Driven Predictive Modeling of Neuronal Dynamics Using Long Short-Term Memory
## 2 Neural Network Architecture, Algorithm and Approach
### Network Training  
We formulated the following optimization problem to train a set of network weights \(\theta\):  
\[\theta^{*}=\operatorname*{arg\,min}_{\theta}\mathcal{L}(\theta), \tag{3}\]  
Figure 5: Diversity in the spiking patterns of hippocampal CA1 pyramidal neurons to applied currents. (a) Regular bursting in response to the external current of 0.23 nA. (b) Irregular bursting in response to the external current of 1.0 nA. (c) Plateau potentials followed by regular spiking in response to the external current of 3.0 nA.  
where the loss function \(\mathcal{L}(\theta)\) is given by  
\[\mathcal{L}(\theta)=\frac{1}{N_{P}}\sum_{k=0}^{N_{P}}(\vec{x}(k)-\hat{x}(k|\theta ))^{T}(\vec{x}(k)-\hat{x}(k|\theta)). \tag{4}\]  
Here \(N_{P}\) represents the length of horizon over which the predictions are made, \(\vec{x}(k)\) is the known state vector at time step \(k\), and \(\hat{x}(k|\theta)\) is the neural network's prediction of the state vector at time \(k\), given \(\theta\).  
To solve the optimization problem (3) and (4), we used the standard supervised backpropagation learning algorithm [42; 43; 44] along with the Adaptive Moment Estimation (Adam) method [45]. The Adam method is a first-order gradient-based optimization algorithm and uses lower-order moments of the gradients between layers to optimize a stochastic objective function.  
Given the network parameter \(\theta^{(i)}\) and the loss function \(\mathcal{L}(\theta)\), where \(i\) represents the algorithm's training iteration, the parameter update is given by Reference [45]  
\[m_{\theta}^{(i+1)}\leftarrow\beta_{1}m_{\theta}^{(i)}+(1-\beta_{ 1})\
abla_{\theta}\mathcal{L}^{(i)}, \tag{5}\] \[v_{\theta}^{(i+1)}\leftarrow\beta_{2}m_{\theta}^{(i)}+(1-\beta_ {2})(\
abla_{\theta}\mathcal{L}^{(i)})^{2},\] (6) \[\hat{m}_{\theta}=\frac{m_{\theta}^{(i+1)}}{1-(\beta_{1})^{i+1}},\] (7) \[\hat{v}_{\theta}=\frac{v_{\theta}^{(i+1)}}{1-(\beta_{2})^{(i+1)}},\] (8) \[\theta^{(i+1)}\leftarrow\theta^{(i)}-\eta\frac{\hat{m}_{\theta}} {\sqrt{\hat{v}_{\theta}+\epsilon}}, \tag{9}\]  
where \(m_{\theta}\) is the first moment of the weights in a layer, \(v_{\theta}\) is the second moment of the weights in a layer, \(\eta\) is the learning rate, \(\beta_{1}\) and \(\beta_{2}\) are the exponential decay rates for the moment estimates, \(\
abla\) is the differential gradient operator, and \(\epsilon\) is a small scalar term to help numerical stability. Throughout this work, we used \(\beta_{1}=0.9\), \(\beta_{2}=0.999\) and \(\eta=0.001\)[45].  
It should be noted that there is a trade-off between the predictive time-horizon of deep LSTM neural network and the computational cost involved in training the network over the predictive horizon. As the predictive horizon increases, the computational cost of training the network over that horizon increases significantly for an equivalent number of examples. To keep the computational tractability in our simulations, all networks with long predictive horizons (i.e., \(N_{P}=50,100,200\)) were trained for 200 epochs except the one-step predictive network, which was trained for 1000 epochs.  
For all training sets throughout this paper, we used the validation to training data ratio as \(1/32\). We set the minibatch size for training to 32. We performed all the training and computation in the TensorFlow computational framework on a discrete server running CentOS 7 with twin Nvidia GTX 1080 Ti GPUs equipped with \(11\) Gb of VRAM.