# DRL-Based Dynamic SFC Orchestration Algorithm for LEO Satellite Networks
## 3 System Model
### PPO Algorithm  
Proximal Policy Optimization (PPO) represents a significant advancement in policy gradient methods for reinforcement learning. As a member of the actor-critic family, PPO addresses the challenge of determining appropriate step sizes for policy updates while maintaining stability during training. The algorithm introduces a new objective function that effectively constrains policy updates within a trust region by clipping the probability ratio between old and new policies, without requiring complex second-order optimization techniques. In the reinforcement learning environment, our objective is to find a policy \(\pi_{\theta}\) that maximizes the expected cumulative discounted reward:  
\[J(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}}[\sum_{t=0}^{T}\gamma^{t}r_{t}] \tag{26}\]  
where \(\tau\) represents the trajectory, \(\gamma\in[0,1]\) is the discount factor, and \(r_{t}\) represents immediate reward at time step \(t\). Taking the derivative of the objective function with respect to policy parameters \(\theta\), we can use gradient ascent to maximize this objective. According to the policy gradient theorem, the gradient can be expressed as follows:  
\[\
abla_{\theta}J(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}}[\sum_{t=0}^{T} \
abla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})Q^{\pi_{\theta}}(s_{t},a_{t})] \tag{27}\]  
where \(Q^{\pi_{\theta}}(s_{t},a_{t})\) is the action-value function. To reduce variance, a state value function baseline \(V^{\pi_{\theta}}(s_{t})\) is introduced:  
\[\
abla_{\theta}J(\theta)=\mathbb{E}_{\tau\sim\pi_{\theta}}[\sum_{t=0}^{T} \
abla_{\theta}\log\pi_{\theta}(a_{t}|s_{t})\hat{A}_{t}] \tag{28}\]  
where \(\hat{A}_{t}=Q^{\pi_{\theta}}(s_{t},a_{t})-V^{\pi_{\theta}}(s_{t})\) is the advantage function. In PPO's actor-critic architecture, the actor network \(\pi_{\theta}(a|s)\) maps states to action distributions, whereas the critic network \(V_{\theta}(s)\) learns the value function. Based on the above derivation, an actor-critic architecture is proposed, where the actor network is responsible for policy improvement, while the critic network supports policy updates by providing more accurate value estimates. This structure with a clear division of responsibilities not only improves the algorithm's sample efficiency but also enables solutions to stability issues in policy optimization. In the AC architecture of the PPO algorithm, the actor network (policy network) \(\pi_{\theta}(a|s)\) with parameters \(\theta\) maps states to action distributions, while the critic network (value network) \(V_{\phi}(s)\) with parameters \(\phi\) learns the value function to assist the actor network in policy updates. For the policy gradient objective, PPO introduces a clipped surrogate objective:  
\[L^{CLIP}(\theta)=\mathbb{E}_{t}[\min(r_{t}(\theta)\hat{A}_{t},\text{clip}(r_{ t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t})] \tag{29}\]  
where \(r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{\text{old}}}(a_{t }|s_{t})}\) is the probability ratio between new and old policies for the same state-action pair; \(\epsilon\) is the clipping parameter that constrains the objective function to ensure the difference between new and old parameters is not too large. The value function is trained by minimizing the following loss:  
\[L^{VF}(\phi)=\mathbb{E}_{t}[(V_{\phi}(s_{t})-V_{target})^{2}] \tag{30}\]  
where \(V_{target}\) is calculated through GAE. The final optimization objective combines these components:  
\[L^{TOTAL}(\theta,\phi)=\mathbb{E}_{t}[L^{CLIP}(\theta)-c_{1}L^{VF}(\phi)+c_{2} S[\pi_{\theta}](s_{t})] \tag{31}\]  
where \(c_{1},c_{2}\) are coefficients, and \(S[\pi_{\theta}]\) represents the entropy bonus term. PPO uses generalized advantage estimation (GAE) to calculate advantages:  
\[ \begin{split}\hat{A}_{t}&=\delta_{t}+(\gamma \lambda)\delta t+1+...+(\gamma\lambda)^{T-t+1}\delta_{T-1}\\ &=\sum_{l=0}^{T-t-1}(\gamma\lambda)^{l}\delta_{t+l}\end{split}