# Machine Learning for Decision Support and Automation in Games: A Study on Vehicle Optimal Path
## 2 Machine Learning Models
### Training Methods
#### 2.2.1 Proximal Policy Optimization  
PPO is an on-policy optimization algorithm that employs a neural network to approximate an optimal function, mapping an agent's observations to the best possible actions in a given state. Integrated with frameworks such as TensorFlow, it interacts with Python processes to optimize step updates through a gradient descent framework. This method has shown to be more effective than traditional gradient descent, resulting in more stable and faster training processes [21]. It functions within an actor-critic framework, where the \"actor\" is responsible for selecting actions, and the \"critic\" evaluates them. This approach integrates current experiences with critiques to refine the policy, avoiding the large, destructive updates typical of traditional policy gradient methods. It accomplishes this by alternating between sampling data from environmental interactions and optimizing a clipped policy surrogate. The resulting balance of simplicity, computational efficiency, and effective sample utilization solidifies its position as a state-of-the-art method [22].  
The core idea of PPO is to optimize a surrogate objective function while ensuring that the new policy does not deviate excessively from the old policy [23]. The key mathematical formula used in PPO is the objective function (1)  
\[L^{\text{CLIP}}(\theta)=\mathbb{E}_{t}\big{[}\min\big{(}r_{t}(\theta)\hat{A}_{t},\text{clip}(r_{t}(\theta),1-e,1+e)\hat{A}_{t}\big{)}\big{]} \tag{1}\]  
Here each component is defined as follows:  
* \(\theta\) represents the parameters of the policy (neural network) being optimized.
* \(\mathbb{E}_{t}\) denotes the empirical expectation over a finite batch of samples.
* \(r_{t}(\theta)\) is the probability ratio between the new policy and the old policy, defined as \[r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{\text{old}}}(a_{ t}|s_{t})}\] (2) where \(\pi_{\theta}(a_{t}|s_{t})\) is the probability of taking action \(a_{t}\) given state \(s_{t}\) under the new policy, and \(\pi_{\theta_{\text{old}}}(a_{t}|s_{t})\) is the probability under the old policy.
* \(\hat{A}_{t}\) is the advantage estimate at time step \(t\), which measures how much better or worse the action \(a_{t}\) is compared to the expected action under the current policy.
* \(\epsilon\) is a small hyperparameter that defines the clipping range.
* \(\text{clip}(r_{t}(\theta),1-e,1+\epsilon)\) clips the probability ratio \(r_{t}(\theta)\) to be within the range \([1-e,1+e]\).  
The objective function \(L^{\text{CLIP}}(\theta)\) ensures that the new policy does not deviate significantly from the old policy by taking the minimum of the unclipped objective \(r_{t}(\theta)\hat{A}_{t}\) and the clipped objective \(\text{clip}(r_{t}(\theta),1-e,1+\epsilon)\hat{A}_{t}\). This conservative update helps maintain a balance between exploration and exploitation, leading to more stable and reliable learning [23; 24].  
In summary, the PPO objective function is designed to penalize large changes in the policy, ensuring smooth and stable policy updates while optimizing performance.