# An Implementation of Actor-Critic Algorithm on Spiking Neural Network Using Temporal Coding Method
## 4 Results and Discussion
### UAV Avoiding Flying Basketball Task  
The task of UAV fast obstacle avoidance is conducted to verify the performance of the algorithm. We choose a standard basketball with a radius of about 12.3 cm. The basketball is thrown from 3-5 m in front of the UAV and the speed is 3-5 m/s. The sensor selected by the UAV is a monocular camera with a frame rate of 31 fps. The simulation and the true flight test are based on the above simple experimental settings. The task does not consider the dynamic background due to the differential encoding of the camera input. In the dynamic background, the decision accuracy will be affected. Moreover, in order to facilitate the training and calculation of the network, the hovering UAV can only choose three actions, left, right and hovering, to complete the obstacle avoidance task (see in Figure 11). The maximum flight speed of the UAV is limited to 3 m/s during the experiment. In this experiment, the basketball's flight trajectory will mostly pass through the initial position of the UAV. In order to prevent a reduction in the training effect, the UAV should be kept in its initial position as much as possible and return again after avoiding. Based on the above characteristics, the goal of the training network is to prolong the survival time of the UAV in the initial position and keep hovering. Therefore, we assume that the reward signal has the following characteristics:  
1. The longer the UAV survives, the greater the reward;
2. The farther the UAV is from the initial position, the smaller the reward;
3. If the UAV fails to avoid the flying basketball, it will be punished. Therefore, the basic reward \(R_{0}\) that varies with the distance of the UAV from the initial position is designed. Note that the distance between the current position of the UAV and the initial position is \(d\), and the time that the drone has survived so far is \(t_{s}\). In the case of the right decision, the reward signal would be:  
\[R=\sum_{i=0}^{t_{s}/\Delta t}R_{0}(d,i\Delta t) \tag{22}\]  
\[R_{0}(d,i\Delta t)=R_{0}(0,t)\exp\left(-\frac{d}{D}\right) \tag{23}\]  
where \(D\) is a constant representing the level to which the reward signal decreases with the distance. If the wrong decision is made, the UAV will be punished. The signal is defined as:  
\[R=-pR_{0}(d,t) \tag{24}\]  
where \(p\) is the penalty factor and is large enough that the reward previously obtained by the UAV is completely cancelled. In the simulation environment, dynamic obstacles are designed to train spiking neural networks in the XTDrone platform. The structure of the spiking AC network is shown in Figure 12. The number of neurons in the input layer is the  
Figure 11: UAV avoiding flying basketball task. same as the number of pixels in a monocular camera. The input camera data are processed by differential coding and the neuron corresponding to the pixel with a significant change in brightness will generate a pulse. The hidden layer of the actor network and critic network consists of 16*16 spiking neurons. While the output layer of the actor network has three neurons, representing left, right and hovering, the output layer of the critic network has one neuron, which represents the state value function. The initial parameters are selected before training. The key parameters are given in Table 1.  
Among them, the first nine parameters are selected based on [30]. While the other parameters are ultimately selected by testing several times. The actor-critic network parameters \(\theta\) and \(\omega\) are initialized randomly. These initial parameters execute the iterative optimization during the training. After that, the true flight test is conducted using the trained actor network parameters. During the simulation, the overall time of the spiking neural network is discrete, and the time step is one millisecond. A throw of the ball is regarded as one episode. Generally, the length of one episode is about one second.