# An Implementation of Actor-Critic Algorithm on Spiking Neural Network Using Temporal Coding Method
###### Abstract
These works attempt to figure out the relationship between RL and STDP and focus on the applications of STDP. [PERSON] made an attempt at spiking neural network reinforcement learning based on STDP for the first time in 2007 [19]. [PERSON] and colleagues completed a spiking neural network model that implemented actor-critic TD learning by combining local plasticity rules with a global reward signal [20]. [PERSON] and colleagues introduced an end-to-end learning approach of spiking neural networks for a lane-keeping vehicle [21]. These works are enlightening but do not dig down inside of the actor-critic algorithm. Many spiking models used now adopt a rate coding scheme, although more information is hidden in the timing of neuronal spikes [22; 23; 24]. In the nervous system, the relative firing time of neurons is concerned with information delivered between neurons [25]. In order to make full use of the information hidden in the firing time of neurons, a differentiable relationship between spike time and synaptic weights is needed, which means information should be encoded in the temporal domain [26]. Several works have been done about temporal coding. The SpikeProp model is provided in Ref. [27]. In this model, neurons are only allowed to generate single spikes to encode the XOR problem in the temporal domain. Other works include [PERSON]'s and [PERSON]'s works [20; 28]. They both tried solving temporally encoded XOR and MNIST problems. [PERSON] tried encoding image information in the temporal domain and achieved some excellent results. However, these works are mostly concentrated on supervised learning and the temporal coding method has not been used in actor-critic networks for decision-making. The objective of this paper is to study a novel spiking neural network reinforcement learning method using actor-critic architecture and temporal coding. The improved leaky integrate-and-fire (LIF) model is used to describe the behavior of the spike neuron. The actor-critic network structure with temporally encoded information is then provided by combining with the improved LIF model. The simulation and actual flight test will be conducted to test the provided method. The main contribution of this paper is to use the temporal coding method to implement an actor-critic TD learning agent on a spiking neural network. The actor-critic learning agent could realize more potential by encoding its output information in the temporal domain. In this way, the scale of the neural network is reduced so that the decision-making process will be faster and consume less power. This article introduces a lightweight network using a simple improved leaky integrate-and-fire (LIF) model and small-scale spiking actor-critic network with only 2-3 layers to realize complex decision-making tasks such as UAV window crossing and avoiding dynamic obstacles. The main contributions of this paper are as follows:  
* It implements an actor-critic TD learning agent on a spiking neural network using the temporal coding method. * The actor-critic learning agent could realize more potential by encoding its output information in the temporal domain. By this way, the scale of the neural network is reduced so that the decision-making process will be faster and consume less power. * The simulation and actual flight test are conducted. The results show that the method proposed has the advantages of faster decision-making speed, less resource consumption and better biological interpretability than the traditional RL method. This paper is organized as follows: In Section 2, the spike neuron model is presented. In Section 3, the actor-critic network structure and the update formulas using temporally encoded information are presented. In Section 4, the network is tested in a gridworld task, a UAV flying through a window task and a UAV avoiding a flying basketball task.