# An Implementation of Actor-Critic Algorithm on Spiking Neural Network Using Temporal Coding Method
## 4 Results and Discussion  
### Gridworld Task  
The algorithm is tested in a gridworld task. The gridworld is a good approximation of real world. The environment is divided into regular grids so that the relationship between the environment and the agent is shown clearly. It is a common test for reinforcement learning algorithms since it provides a good way to implement a finite Markov decision process. In this case, it is convenient to use the gridworld to show the agent's states. The gridworld is a map on which an agent can move from one state to the next by taking four different actions: up, down, left and right (see in Figure 5). A random state \(S_{d}\) is selected as the destination (state D in Figure 5).  
Since the agent can only take four directions, the distance on the map is defined as vertical distance plus horizontal distance, which is shown as follows:  
\[d(S,S^{\prime})=\big{|}x-x^{\prime}\big{|}+\big{|}y-y^{\prime}\big{|}, \tag{17}\]  
Figure 5: The gridworld with a \(5\times 5\) grid map.  
where \(S=(x,y)\) and \(S^{\prime}=(x^{\prime},y^{\prime})\). \(x\), \(y\) and \(x^{\prime}\), \(y^{\prime}\) are integers between 1 and 5.  
The agent is supposed to learn the quickest way to reach the destination. To ensure that, the reward at the destination is one and the reward at other states is zero. In the meantime, the value function at the destination is set to zero.  
To measure the agent's performance, the latency of the agent moving from the initial state to the destination state is designed. Assuming \(n\) is the number of moves that the agent takes from the initial state to the destination state, latency is defined as:  
\[latency=n-d(S_{initial},S_{d}). \tag{18}\]  
When the agent reaches the destination, the initial state is updated with a random state on the map. During the learning process, each state has a latency value. We choose one state's latency value as a target, for example, state S in Figure 5. The agent won't stop learning until the maximum is small enough. Figure 6 shows the latency change during learning. When the latency converges to zero, the quickest way from state S to state D is found.  
After an agent's learning, a color map is used to show states' values in Figure 7a. With states getting farther away from the destination, states' values decay. Since we set \(\gamma\) = 0.9 in Equation (7), under the ideal circumstance the states' values would be:  
\[V_{ideal}(S)=0.9^{d(S,S_{d})}. \tag{19}\]  
A root mean square error can be obtained to describe the performance of value learning:  
\[\sigma_{V}=\sqrt{\frac{\sum_{S}{(V(s)-V_{ideal}(s))^{2}}}{N_{S}}}. \tag{20}\]  
Here, \(N_{S}\) is the number of states. For the values showed in Figure 7a, \(\sigma_{V}=0.05347\). Considering the computing error, this result is acceptable.  
A probability arrow \(\overrightarrow{p}(S)\) is used for each state to judge the agent's policy learning, which is:  
\[\overrightarrow{p}(S)=\begin{pmatrix}\pi(up|S)-\pi(down|S)\\ \pi(left|S)-\pi(right|S)\end{pmatrix}. \tag{21}\]  
The probability arrow map can show us the agent's learning achievements in a direct way.  
Except for the destination, all states' probability arrows point to the states which have the largest values in their neighbors, so that the quickest way from each state to the destination is found. However, a part of the probability arrows is largely different  
Figure 6: State S’s latency change during learning.  
from the ideal circumstance. For example, the three downward arrows in the first left row completely abandon the probability to take the right direction, which is obviously not ideal. This situation may be related to the synaptic weight change when the synaptic weight is small enough. Overall, the synaptic weight change tends to be larger when it is smaller. Therefore, it is possible that the synaptic weight gets too small to make the output neuron fire a spike if only another action with an advantage is chosen continuously. In that way, although this action is also advantageous, it might be abandoned.