## 8 Results  
The model of our study was implemented using the real-time development platform Unity. The source code is available at [[https://github.com/trinhthanhtrung/unity-pedestrian-rl](https://github.com/trinhthanhtrung/unity-pedestrian-rl)]([https://github.com/trinhthanhtrung/unity-pedestrian-rl](https://github.com/trinhthanhtrung/unity-pedestrian-rl)) (accessed on 30 April 2021), by opening the scene _PathPlanningTask_ within the _Scene_ folders. Figure 7 presents a screenshot of our implemented application. For the training task of the pedestrian agent, we adopted the reinforcement learning library ML-Agents [44], which acts as a communicator between Unity and Python machine learning code. In each training episode, the information of the model, consisting of the agent's observations and actions, and the cumulative reward value, is sent to Python. The information is subsequently used for training the agent's policy in a neural network using the PPO algorithm, then the updated policy will be sent back to the pedestrian agent. Because the environment's states are moderately noisy, it is recommended to train the agent with a large batch size. Furthermore, multiple instances of the same training environment are created to speed up the training process. We have been able to successfully get the cumulative reward to converge after three million steps with a learning rate of \(1.5\times 10^{-3}\). For a smoother navigation path, we used the mean of the agent's actions in multiple episodes of the same environment's state. Figure 8 shows the statistics of the training process in TensorFlowBoard. Figure 7: Path-planning task implementation screenshot. Figure 9 shows the planned path of the agent in different situations in our implementations. In these figures, the actor model at the bottom is the pedestrian agent, the red point on the top is the agent's destination, the black circle represents the predicted POC by the agent, and the red circle (covered by the POC in (b) and (c)) is the current obstacle. Figure 8: Cumulative reward statistics. Figure 9: Agent’s planned path in different situations: (**a**) no obstacle; (**b**) with a static obstacle with a low danger level; (**c**) with a static obstacle with a high danger level; (**d**) with an obstacle moving straight in one direction away from the agent (**e**) with an obstacle moving straight in one direction toward the agent; (**f**) with a pedestrian obstacle. Figure 10 shows how the implementation of our model compares to SFM. In each figure, our implementation is on the left and the SFM implementation is on the right (darker environment). From observation, in Figure 9, the agent could be able to plan a sufficiently realistic path to the destination, while also considering the rules and following common conventions like walking to the left side and naturally changing direction. This can be seen in the situation (a), where the agent has chosen to walk on the left side of the road and gradually move toward the destination when needed, instead of walking straight to the destination. Although the planned path was constructed from the outputs of the neural networks, it is shown to be remarkably stable. The agent has also shown its capability to avoid the obstacle, as the planned path does not collide with the obstacle or its prediction in most situations. Furthermore, its planned path also seems to be adapted to the risk from the obstacle. This is observable by comparing the paths planned by the agent in (b) and (c). We implemented the obstacles to have the same properties in both situations; however, the obstacle in (c) has a much higher danger level than the obstacle in (b). The result is that, in Figure 9b, the agent only almost avoided colliding with the obstacle, while in (c), the agent chose a path that steers much further away from the obstacle than in (b). This resembles actual human thinking when planning a navigation path where there is a dangerous obstruction on the road. Figure 9d,e demonstrate how the agent adopts the prediction process into path planning.