# Load-Balanced Dynamic SFC Migration Based on Resource Demand Prediction
## 4 Algorithm Design
### PPO-Based SFC Migration Algorithm
#### 4.2.2 Proximal Policy Optimization Algorithm  
Proximal Policy Optimization is a widely used policy optimization algorithm in reinforcement learning. PPO aims to stabilize the training process and reduce fluctuations and uncertainties during policy updates by limiting the difference between new and old policies [15]. PPO is mainly implemented through the following key steps. First, it interacts with the environment using the current policy to collect a batch of trajectory data, including states \(s_{t}\), actions \(a_{t}\), rewards \(r_{t}\), and next states \(s_{t+1}\). To improve sample efficiency, data are usually collected simultaneously in multiple parallel environments. These data are used for subsequent updates of the policy and value function. Next, the Generalized Advantage Estimation (GAE) method is used to estimate the advantage function \(\hat{A}_{t}\). The advantage function measures the superiority of a particular action relative to the average policy:  
\[\hat{A}_{t}=\delta_{t}+(\gamma\lambda)\delta_{t+1}+\ldots+(\gamma\lambda)^{T-t+1 }\delta_{T-1} \tag{36}\]  
where \(\delta_{t}=r_{t}+\gamma V(s_{t+1})-V(s_{t})\), \(\gamma\) is the discount factor, and \(\lambda\) is the GAE decay parameter. Next, for each sample point, the probabilities of selecting action \(a_{t}\) under state \(s_{t}\) for both the new and old policies, \(\pi_{\theta_{\text{old}}}(a_{t}|s_{t})\) and \(\pi_{\theta}(a_{t}|s_{t})\), are calculated. The probability ratio \(r(\theta)\) indicates the relative change between the new and old policies. A ratio greater than 1 indicates that the new policy is more inclined to choose action \(a_{t}\) in this state, while a ratio less than 1 indicates the opposite:  
\[r(\theta)=\pi_{\theta}(a_{t}|s_{t})/\pi_{\theta_{\text{old}}}(a_{t}|s_{t}) \tag{37}\]  
To limit the extent of policy updates and avoid \(r(\theta)\) being too large or too small, PPO introduces a clipping operation. The clipped objective function is defined as follows:  
\[L^{\text{CLIP}}(\theta)=E_{t}\big{[}\min\big{(}r_{t}(\theta)\hat{A}_{t},\; \text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}\big{)}\big{]} \tag{38}\]  
where \(\epsilon\) is a small positive number, typically between 0.1 and 0.3. The clipping operation constrains \(r_{t}(\theta)\) within the range \([1-\epsilon,\;1+\epsilon]\), preventing excessive policy updates that lead to unstable training. In addition, PPO uses mean squared error to minimize the difference between the value function and the target value:  
\[L^{\text{VF}}(\theta)=E_{t}\bigg{[}\Big{(}V_{\theta}(s_{t})-V_{t}^{\text{ target}}\Big{)}^{2}\bigg{]} \tag{39}\]  
The value function \(V_{\theta}(s_{t})\) estimates the expected return after executing the policy in state \(s_{t}\), while the target value \(V_{t}^{\text{target}}\) estimates the actual return, which can be calculated using Monte Carlo or temporal difference methods. To encourage exploration and prevent premature convergence to local optima, an entropy regularization term is added. The entropy regularization term rewards policies with higher entropy, i.e., more random policies, which helps ensure adequate exploration during the early stages of training:  
\[S[\pi_{\theta}](s_{t})=-\sum_{a}\pi_{\theta}(a|s_{t})\log\pi_{\theta}(a|s_{t}) \tag{40}\]  
Combining these components, the total loss function \(L(\theta)\) is defined as follows:  
\[L(\theta)=E_{t}\Big{(}L^{\text{CLIP}}(\theta)-c_{1}L^{\text{VF}}(\theta)+c_{2 }S_{\pi\theta}\Big{)} \tag{41}\]  
where \(c_{1}\) and \(c_{2}\) are weights used to balance the importance of policy loss, value function loss, and the entropy regularization term. To optimize the total loss function \(L(\theta)\), PPO typically uses stochastic gradient descent algorithms (such as the Adam optimizer) for parameter updates. First, based on the current batch of sample data, calculate the gradients of the total loss function \(L(\theta)\) with respect to the policy parameters and value function parameters, \(\
abla_{\theta_{\epsilon}}L(\theta)\) and \(\
abla_{\theta_{\epsilon}}L(\theta)\).