## 4 COIN-LEO Algorithm
### Deep Reinforcement Learning Using the PPO Algorithm
#### 4.2.4 Proximal Policy Optimization (PPO) Algorithm  
We use the proximal policy optimization (PPO) algorithm for reinforcement learning [26]. PPO is a commonly used reinforcement learning algorithm designed to address the policy optimization problem. The main feature of the PPO algorithm is to update the policy through proximal clipping, ensuring that policy improvements are not too drastic and avoiding instability. The goal of the PPO algorithm is to improve the performance of the current policy as much as possible while maintaining the similarity of the policy and avoiding significant changes during policy updates. The objective of PPO is to maximize the expected cumulative reward, denoted by J(\(\theta\)), where \(\theta\) represents the policy parameters. This can be formulated as follows:  
\[\text{J(\theta)}=\text{E}[\text{R(t)}], \tag{2}\]  
where R(t) represents the cumulative reward from time step 0 to time step t.  
To optimize the objective function J(\(\theta\)), we use the PPO algorithm for policy updates. The core idea of PPO is to constrain the policy updates using proximal clipping to enhance algorithm stability. Specifically, the PPO algorithm uses two different objective functions for policy updates, namely the clipped surrogate objective and PPO-clip objective. 1. Clipped Surrogate Objective  
The Clipped Surrogate Objective is the core objective function of the PPO algorithm, used for computing the policy gradient. When updating the policy, the PPO algorithm calculates the policy gradient by comparing the probability ratio between the new policy and the old policy, then constrains it within a certain range, which is called proximal clipping. The purpose of this is to ensure that the improvement of the new policy relative to the old policy is not too large, thereby enhancing the stability of the algorithm. The clipped surrogate objective is used to calculate the policy gradient and is defined as:  
\[L^{\theta}(s,a)=\min\left(\frac{\pi^{\prime}(a|s)}{\pi(a|s)}\cdot A^{\pi}(s,a),clip\left(\frac{\pi^{\prime}(a|s)}{\pi(a|s)},1-\epsilon,1+\epsilon\right) \cdot A^{\pi}(s,a)\right) \tag{3}\]  
where \(\pi^{\prime}(a\,|\,s)\) represents the probability of selecting action a under the old policy in state \(s\), \(\pi(a\,|\,s)\) represents the probability of selecting action a under the new policy in state \(s\) and \(A^{\pi}(s,\,a)\) represents the advantage of selecting action \(a\) in state \(s\). \"clip\" refers to the operation of bounding or limiting a value within a specified range. The purpose of using clipping in the PPO algorithm is to control the magnitude of policy updates to enhance the stability of the learning process. 2. PPO-Clip Objective  
The PPO-clip objective is a variant of the clipped surrogate objective, used to compute the optimization objective for policy updates. The PPO-clip objective is the minimum value between the clipped surrogate objective and the KL divergence. It constrains the policy update using proximal clipping and KL divergence to ensure that the magnitude of policy updates remains within a reasonable range, thereby preventing the algorithm from over-optimizing. \[L^{\theta}(s)=\left[\min\left(L^{\theta}(s,a),clip\left(\frac{\pi^{\prime}(a |s)}{\pi(a|s)},1-\epsilon,1+\epsilon\right)\right)\right] \tag{4}\]where \(L^{\theta}(s)\) represents the PPO-clip objective in state \(s\), \(\pi^{\prime}(a\,|\,s)\) represents the probability of selecting action a under the old policy in state \(s\) and \(\pi(a\,|\,s)\) represents the probability of selecting action \(a\) under the new policy in state \(s\). With the policy updates using proximal clipping, the PPO algorithm can achieve good performance and stability in large-scale and complex reinforcement learning tasks. This makes PPO one of the widely used algorithms in the field of reinforcement learning. Due to the policy update method of proximal clipping, the PPO algorithm exhibits good stability and is suitable for large-scale and complex reinforcement learning tasks.