# Machine Learning for Decision Support and Automation in Games: A Study on Vehicle Optimal Path
## 3 Environment
### Phase 1: Basic Navigation
#### 3.2.2 Hyperparameters  
The hyperparameters are fine-tuned to optimize the performance of the autonomous car in reaching its target efficiently. These parameters are configured in a YAML file, provided by the ML-Agents' flexibility, which allows for easy modification and experimentation. In this initial phase the configurations used were based on the default ML-Agents hyperparameters, providing a solid foundation for further refinement [35; 36]. Each hyperparameter is described and can be consulted in the documents provided by Unity Technologies [37], but there are some that require particular attention due to their significant impact on training performance, these are described below. Shared Hyperparameters  
The **learning rate** is the step size for gradient descent updates. A higher learning rate can speed up training but may cause instability, whereas a lower learning rate leads to more stable training but requires more iterations. The typical range for the learning rate is \(1\times 10^{-5}\) to \(1\times 10^{-3}\), with a default value of \(3\times 10^{-4}\). The **batch size** refers to the number of training samples per mini-batch. Larger batch sizes can provide more accurate gradient estimates but require more memory and computational power. The typical range for batch size is from 32 to 512, with default values being 64 for PPO and 256 for SAC. The **gamma** parameter is the discount factor for future rewards. A higher gamma value puts more weight on future rewards, promoting long-term strategies. The typical range is from 0.8 to 0.995, with a default value of 0.99. Proximal Policy Optimization (PPO)  
For PPO, the **clip ratio** is a crucial parameter. It is used to clip the policy objective to prevent large updates, helping to maintain stable updates by limiting the extent to which the policy is changed in a single update. This parameter typically ranges from 0.1 to 0.3, with a default value of 0.2. The number of **epochs** refers to how many times the learning algorithm will work through the entire training dataset. More epochs mean more thorough learning from the collected data but increase computational time. The typical range is from 3 to 10, with a default value of 10. The **GAE lambda** is a smoothing parameter for Generalized Advantage Estimation (GAE). It balances bias and variance in advantage estimation, where higher values provide lower variance but higher bias. The typical range for GAE lambda is from 0.9 to 0.95, with a default value of 0.95. The **entropy coefficient** is used for entropy regularization, encouraging exploration by adding an entropy term to the loss function. This prevents the policy from converging prematurely to suboptimal solutions. The entropy coefficient typically ranges from 0.01 to 0.1, with a default value of 0.01. Soft Actor-Critic (SAC)  
For SAC, the **tau** parameter is the interpolation parameter for the soft update of target network parameters. It affects the update speed of the target networks, balancing stability and responsiveness. The typical range for tau is from 0.005 to 0.05, with a default value of 0.005. The **alpha** parameter is the entropy regularization coefficient, which determines the balance between exploration and exploitation. Higher values encourage more exploration, which can help in finding better policies but may also slow down convergence. The typical range for alpha is from 0.1 to 0.2, with a default value of 0.2. The **target update interval** specifies the number of steps between target network updates. Frequent updates can make the training more responsive, while less frequent updates can enhance stability. The typical range is from 1 to 1000, with a default value of 1. Finally, the **gradient steps** refer to the number of gradient steps performed after each environment step. More gradient steps can lead to better policy updates per interaction with the environment but also increase computational load. The typical range is from 1 to 4, with a default value of 1.