# Machine Learning for Decision Support and Automation in Games: A Study on Vehicle Optimal Path
## 3 Environment
### Phase 1: Basic Navigation
#### 3.2.3 Implementation Overview  
To provide a comprehensive understanding of the implementation, this section details the key steps involved in the car's decision-making and learning process. The following components form the core of our implementation: initialization, action selection, reward application, and environment interaction. The flowchart, Figure 3, illustrates the sequence of these steps, followed by pseudocode snippets that correspond to critical parts of the algorithm.  
The flowchart highlights the following key steps:  
* **Initialization**: Set up the environment, define hyperparameters, and initialize the agent (car) with its PPO or SAC algorithm (see Algorithm 1).
* **Action Loop**: For each episode, the agent senses its environment using the 360-degree sensor and selects an action based on the current policy (see Algorithm 2).
* **Environment Interaction**: The selected action is applied to the environment, and the agent moves according to the chosen action (see Algorithm 2).
* **Reward Calculation**: The reward function is evaluated based on the agent's new state, considering proximity to the target, penalties for undesirable actions, and the time taken (see Algorithm 3).
* **Policy Update**: Using the collected rewards and experiences, the agent updates its policy through gradient descent (see Algorithm 4).
* **Repeat**: The process repeats until the agent converges on an optimal policy or the training session ends (see Algorithm 5).  
```
1:while not done do
2:\(action\gets agent\_select\_action(state)\)\(\triangleright\) Choose action using current policy
3:\(next\_state,reward,done\gets env.step(action)\)\(\triangleright\) Apply action and get new state
4:\(total\_reward\gets total\_reward+reward\)\(\triangleright\) Accumulate reward
5: Store transition:\(agent\_memory.add(state,action,reward,next\_state,done)\)
6:\(state\gets next\_state\)\(\triangleright\) Update state
7:endwhile
```
**Algorithm 2** Action Selection and Environment Interaction  
The reward function was designed to balance exploration and goal-oriented behavior, guiding the agent to navigate efficiently towards the target. Reaching the target yields a significant reward (base_reward + time_bonus), incentivizing success and efficient trajectories. Proximity rewards (proximity_reward \(\times\) angle_reward) encourage steady progress while ensuring directional alignment.  
Penalties discourage undesired behaviors, such as moving away from the target (penalty_for_bad_action) or hitting environment boundaries (boundary_penalty), reflecting real-world constraints. The modular design allows for easy adjustment of weights for different scenarios, ensuring flexibility and adaptability.  
This function was iteratively refined during training to effectively incentivize desired behaviors, aligning with real-world applications.
```
1:if done then
2:\(agent.update\_policy()\)\(\triangleright\) Perform PPO updates using stored transitions
3:endif
4:Log episode results:\(episode\_rewards.append(total\_reward)\)
```  
**Algorithm 3** Reward Calculation