## 2 Double-Layer Reinforcement Learning Algorithm Design
### Policy Gradient Methods  
Policy gradient methods are a type of algorithm that solves reinforcement learning problems by directly optimizing the policy function \(\pi\). Unlike value iteration methods, policy gradient methods do not estimate the value function directly but instead optimize the policy itself to maximize the expected cumulative reward. These methods are widely used for reinforcement learning tasks with continuous action spaces and high-dimensional state spaces. The basic idea of policy gradients is to compute the gradient ascent to directly optimize the policy parameters, maximizing the cumulative reward. The core of policy gradient methods is the policy gradient theorem, which provides the method for computing the gradient of the expected cumulative reward with respect to the policy parameters. Let \(\pi_{\theta}(a|s)\) be a parameterized probability distribution representing the probability of taking action \(a\) in state \(s\), with parameters \(\theta\). The policy gradient theorem gives the gradient of the expected cumulative reward with respect to the policy parameters \(\theta\):  
\[\
abla_{\theta}J(\theta)=\mathbb{E}_{\pi_{\theta}}\
abla_{\theta}\log\pi_{ \theta}(a|s)Q^{\pi_{\theta}}(s,a) \tag{2}\]  
In Formula (2), the term \(\
abla_{\theta}J(\theta)\) represents the gradient of the expected cumulative reward \(J(\theta)\) with respect to the policy parameters \(\theta\), indicating how to adjust \(\theta\) to increase the reward. The expectation \(\mathbb{E}_{\pi_{\theta}}\) takes the average over all possible state-action pairs \((s,a)\) that could occur under the current policy, capturing the average behavior of the policy. The term \(\
abla_{\theta}\log\pi_{\theta}(a|s)\) is the gradient of the log probability of taking action \(a\) in state \(s\), which measures how sensitive the action probabilities are to changes in \(\theta\). This helps in simplifying the gradient computation for policy updates. Finally, \(Q^{\pi_{\theta}}(s,a)\) is the state-action value function, representing the expected cumulative reward from state \(s\) after taking action \(a\) and following the policy \(\pi_{\theta}\). This quantifies how beneficial a certain action is in a given state. Together, this formula directs the policy update towards actions that both are probable and lead to high rewards. The steps of a policy gradient method are illustrated as Algorithm 1:  
```
1:Initialize policy parameters \(\theta\) and other related parameters. 2:Sampling: Interact with the environment using the current policy \(\pi_{\theta}\) to generate state-action-reward sequences \((s_{0},a_{0},r_{0}),(s_{1},a_{1},r_{1}),\ldots\). 3:Compute Gradients: Compute the gradient of the expected cumulative reward with respect to the policy parameters \(\theta\) according to the policy gradient theorem: \[\
abla_{\theta}J(\theta)=\mathbb{E}_{\pi_{\theta}}[\
abla_{\theta}\log\pi_{ \theta}(a|s)Q^{\pi_{\theta}}(s,a)]\]
4:Update Policy: Update the policy parameters \(\theta\) using gradient ascent: \[\theta\leftarrow\theta+\alpha\
abla_{\theta}J(\theta)\] where \(\alpha\) is the learning rate. ```  
**Algorithm 1**Policy Gradient Method  
PPO is a widely used policy gradient algorithm. PPO balances the speed and stability of policy improvement by limiting the step size of policy updates, addressing the instability issues often encountered in policy gradient methods. The core idea of PPO is to introduce a clipped probability ratio objective function, which limits the extent of change between the new and old policies, ensuring stable updates. PPO optimizes the following objective function:  
\[L^{CLIP}(\theta)=\mathbb{E}_{t}\big{[}\min(r_{t}(\theta)\hat{A}_{t},\text{ clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t})\big{]} \tag{3}\]  
In Formula (3), \(r_{t}(\theta)=\frac{\pi_{\theta}(a|s_{t})}{\pi_{\theta_{\text{old}}}(a|s_{t})}\) represents the probability ratio of the new policy to the old policy, indicating how the new policy differs from the previous one.