# Machine Learning for Decision Support and Automation in Games: A Study on Vehicle Optimal Path
## 4 Results
### Results Analysis  
In the current phase of our project, where the objective is for the car to reach the target regardless of its position on or off the road, we have observed promising yet imperfect  
Figure 5: Real and virtual A-MoVeR motorcycle side by side. Figure 6: Motorcycle navigating on the new environment. results. The car is generally successful in reaching the target with PPO, demonstrating its capability to learn and adapt effectively to the task. However, with SAC, the car rarely reaches the target, indicating significant challenges in its learning process. The following analysis presents a comprehensive comparison of various metrics for PPO (represented in purple) and SAC (represented in orange). The actions in this experiment were implemented as continuous, and the training was conducted over 8 million steps. These visualizations provide valuable insights into the performance, stability, and learning efficiency of the two algorithms under the specified conditions. Additionally, Table 2 is presented, comparing key metrics between both algorithms. * **Training Time** As anticipated, the training duration for PPO was 13.6 h, while SAC required a significantly longer training period of 1.541 days. This disparity in training times highlights the differences in the complexity and update mechanisms of the two algorithms. * **Cumulative Reward Trend** The graph in Figure 7 illustrates the cumulative reward over time for PPO and SAC. The cumulative reward for PPO starts low but shows a consistent upward trend, stabilizing around a cumulative reward of 30 after approximately 3 million time steps. The curve is relatively smooth with some early fluctuations but becomes more stable as training progresses. In contrast, the cumulative reward for SAC starts low and initially rises but stabilizes at a much lower level compared to PPO. The SAC curve shows significant fluctuations throughout the training period, oscillating around \(-10\) to \(0\) without a clear upward trend after the initial increase. * **Learning Rate** Figure 8 shows the policy learning rate over time for both PPO and SAC. PPO starts with a higher learning rate that decreases more rapidly compared to SAC. This indicates a more aggressive learning approach initially, which then stabilizes as training progresses. Figure 8: Policy learning rate over time for PPO (purple) and SAC (orange). Figure 7: Cumulative reward over time for PPO (purple) and SAC (orange). * **Policy Loss** The policy loss over time for both algorithms is shown in Figure 9. PPO maintains a consistently low policy loss throughout the training period, indicating stable policy updates. On the other hand, SAC exhibits a high and increasing policy loss, suggesting instability and inefficiency in policy updates. * **Cumulative Reward Distribution** Figures 10 and 11 present the cumulative reward distribution for PPO and SAC, respectively. PPO shows a more concentrated reward distribution towards higher rewards, while SAC's distribution is more spread out and centered around lower rewards. * **Value Loss** Figure 12 illustrates the value loss over time for both algorithms. PPO maintains a relatively low value loss with some fluctuations, whereas SAC shows a consistently low value loss, indicating different stability characteristics. Figure 11: Cumulative reward distribution for SAC. Figure 10: Cumulative reward distribution for PPO. Figure 9: Policy loss over time for PPO (purple) and SAC (orange). * **Episode Length** The episode length over time is depicted in Figure 13. PPO shows a decreasing trend in episode length, stabilizing at a lower value, which suggests efficient learning and faster convergence to the target. SAC, however, maintains a relatively high and stable episode length, indicating inefficiency in learning to reach the target. The results suggest that PPO is more suitable for this environment or task, given its higher and more stable cumulative reward, as well as its faster training time.