# An Implementation of Actor-Critic Algorithm on Spiking Neural Network Using Temporal Coding Method
## 2 Spike Neuron Model  
The neuronal dynamics can be conceived as a combination of an integration process and a firing process. The neuron's membrane accepts external stimuli and accumulates all the influences caused by external stimuli. As a result, the membrane potential adds up with external stimuli reaching the membrane. However, the membrane potential does not add up forever; it is limited by a voltage threshold. When the membrane potential increases and reaches the threshold, the neuron releases the accumulated electronic charges and fires a spike. By firing regular-shaped spikes, the neuron turns the received external stimuli into a series of spikes and encodes the information in the form of moments when the spikes are fired. A complete spike neuron model should include two parts: the integration part and the firing part. In this paper, an approximate equation is used to describe the evolution of the membrane potential and the threshold is set to be a constant. This kind of model is called a leaky integrate-and fire (LIF) model [29]. The leaky integration part includes two kinds of external stimuli: spikes that other neurons fire and DC stimuli directly applied to the membrane. First, a kernel function is used instead of the single exponential function to describe the membrane's response to the received spikes [30]. The kernel function is:  
\[K(t-t_{k})=\frac{1}{\kappa}\bigg{[}\exp\bigg{(}-\frac{t-t_{k}}{\tau_{0}}\bigg{)} -\exp\bigg{(}-\frac{t-t_{k}}{\tau_{1}}\bigg{)}\bigg{]}, \tag{1}\]  
where \(t_{k}\) is the time that the received spike arrives at the neuron membrane, \(\kappa\) is a constant to make the maximum of the kernel function 1, \(\tau_{0}\) is the membrane potential integration constant and \(\tau_{1}\) is the spike currency constant. Here, we decide \(\tau_{0}=4\tau_{1}\). As is shown in Figure 1, the kernel function rises gradually before it reaches the maximum. After that, it decays more slowly than it rises. This feature of a slow rise and slower decay enables the kernel function to imitate the true circumstance of animal neurons. The figure also shows different shapes of the kernel function with different time constants. In all three functions, \(\tau_{1}\) is set to be \(\tau_{0}/4\), and the maximums of the three functions are all normalized to 1. The maximum of the kernel function comes first when \(\tau_{0}=150\) ms and comes last when \(\tau_{0}=350\) ms. With \(\tau_{0}\) getting larger, the rising period gets longer and the rising and decaying rates get slower. Figure 1: Kernel function. Second, an exponential function is used to describe the membrane's response to the DC stimulus. Here, the DC stimulus is regarded as a constant so that the non-spike potential of the membrane will regress to a constant. When there is no DC stimulus, the non-spike potential of the membrane will regress to 0. Combining the two parts, the evolution of membrane potential is:  
\[V_{mem}(t)=\sum_{i}W_{i}\sum_{t_{k}<t}K(t-t_{k})+V_{regress}(t), \tag{2}\]  
where \(W_{i}\) is the weight value of the synapse linking two neurons. In the firing part, the threshold of the membrane potential is set to be a constant \(\theta\). When \(V_{mem}\) crosses the threshold from small to large, the neuron fires a spike. In the meantime, all the accumulated electronic charges are cleared out and the membrane potential is reset at \(V_{reset}\) to imitate the true discharging process. \(V_{reset}\) is smaller than 0 and is a non-spike potential, so it will regress to 0, influenced by DC stimuli even when there is no DC stimulus. In this paper, input neurons are separated from output neurons by the difference of these two membrane potential resources. Neurons that only accept DC stimuli are input neurons. Influenced by the DC stimulus, the membrane potential rises by exponential regression. Furthermore, limited by the voltage threshold, input neurons keep generating spikes regularly and the membrane potential recovers from \(V_{reset}\) all the time (see in Figure 2a). Neurons that only accept spikes are output neurons.