# An Implementation of Actor-Critic Algorithm on Spiking Neural Network Using Temporal Coding Method
## 4 Results and Discussion
### UAV Avoiding Flying Basketball Task
Overall, the cumulative reward signal obtained by the UAV is on the rise, indicating that the training is effective and the survival time is increasing (see in Figure 13). Several typical scenarios in which the UAV avoids the flying basketball during the simulation experiment are shown in Figure 14.  
In order to facilitate the optimization and improvement of the network, small parameters are selected for initialization. Therefore, at the beginning of training, it takes a long time for the UAV to make a decision. The time it takes for the UAV to make a decision then gradually decreases during the training (see in Figure 15). At the end of the training, the average time spent by the UAV to make a decision drops to 30.7 milliseconds. \begin{table}
\begin{tabular}{l l l} \hline \hline
**Notation** & **Definition** & **Value** \\ \hline \(\tau_{0}\) & The membrane potential integration constant of the kernel function & 150 \\ \(\zeta\) & Neuron firing threshold & 50 \\ \(u_{rest}\) & Neuron resting potential & 0 \\ \(u_{r}\) & Neuron recovery potential & \(-\)15 \\ \(\beta\) & Neuron firing probability & 50 \\ \(\alpha_{\theta},\alpha_{\omega}\) & Update rate & 0.01 \\ \(\gamma\) & Discount factor & 0.9 \\ \(\lambda\) & Damping decrement & 0.8 \\ \(\Delta L\) & Log-intensity threshold & 0.25 \\ \(R_{0}(0,t)\) & Initial reward & 50 \\ \(D\) & Reward decay rate & 3 \\ \(p\) & Penalty factor & 1000 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Summary of parameters. Figure 12: Actor-critic network structure. Comparative experiments with traditional AC networks are also carried out to verify the rapidity of the algorithm. One hundred basketball-avoiding experiments were performed in the simulation platform. The UAV used the spiking AC algorithm and the traditional AC algorithm to make decisions, respectively. Figure 16a shows the distribution of decision times for one of the basketball avoidances. It is noted that the decision time of the UAV using the spiking AC network is mainly concentrated in the interval of 25 ms to 35 ms, while the decision time of using the traditional AC network is concentrated in the interval of 55 to 65 milliseconds. The decision-making algorithm proposed in this paper is significantly faster, so the number of decisions in one episode is also significantly larger. Figure 16b shows the average decision time for one hundred basketball avoidance experiments. As shown in the figure, the average decision-making time of the UAV using the algorithm proposed in this paper is about 30 milliseconds, while the average decision-making time of the UAV based on the traditional AC network is about 60 milliseconds. Therefore, the UAV decision-making algorithm proposed in this paper greatly speeds up the UAV's decision-making. Figure 14: Six typical images during UAV’s moving: **(a)** The basketball is flying from the front left of the UAV, so the UAV moves right; **(b)** The UAV moves right; **(c)** The UAV avoids the basketball successfully; **(d)** The basketball is flying towards the front of the UAV; **(e)** The UAV moves left; **(f)** The UAV avoids the basketball successfully. Figure 13: Cumulative reward signal obtained during training. The algorithm is then tested through the actual flight test of the UAV avoiding flying basketballs. A F450 quadrotor UAV carrying the TX2 computing platform is used to conduct the task and the maximum flight speed of it is 3 m/s. A scene of the experiment is shown in Figure 17. In this experiment, the basketball is thrown from 3-5 m on the left front of the UAV. When the basketball approaches, the UAV makes a decision to fly to the left and successfully avoids collision. We repeated the above experiment 100 times and the success rate of obstacle avoidance was 96%. It can be observed that the randomness of the actor network does not have a significant impact on the UAV's decision-making. The average decision time of 100 times is 41.3 ms, which is slightly longer than the simulation experiment.