## 4 Optimal Frame Generation Time Threshold Calculation
### Algorithm Design
#### 4.2.3 Policy Entropy Reward Function  
The policy entropy reward function \(\kappa^{S}(\theta)\) defined in this paper is  
\[\kappa^{S}(\theta)=\mathbb{E}_{t}[\mathcal{H}(\pi_{\theta}(\cdot|s_{t}))] \tag{22}\]  
The policy entropy reward function can improve the degree of divergence of reinforcement learning policy exploration and achieve optimal decision generation through a more comprehensive decision test. \(\mathcal{H}(\pi_{\theta}(\cdot|s_{t}))\) represents the entropy of the policy \(\pi_{\theta}\) under the state \(s_{t}\), which represents the randomness of the policy. The larger the entropy, the higher the randomness of the policy and the stronger the exploration.  
Finally, the loss function combines these three parts of functions to obtain  
\[\kappa(\theta)=-\kappa^{CLIP}(\theta)+c_{1}\kappa_{\text{critic}}(\theta)-c_{2 }\kappa^{S}(\theta) \tag{23}\]  
In this formula, \(c_{1}\) and \(c_{2}\) are hyperparameters that weigh the value function loss and entropy reward, respectively, and \(c_{1}\) and \(c_{2}\) control the weight of the value function loss and entropy penalty term, respectively. The overall goal of the algorithm is to maximize the expected return by optimizing the proximal policy objective and the entropy term, while minimizing the value function loss. At the same time, the policy updates are constrained to ensure stable training.  
The entropy-based regularization and its integration into the total loss function are designed by the authors to balance exploration and stability in the decision-making process.  
The pseudo-code of the optimal frame generation time threshold calculation algorithm is as shown in Algorithm 1.  
```
0:PPO hyperparameters, number of iterations
0:Optimal frame generation time threshold \(T_{h}\)  Initialize network parameters \(\theta\), and buffer \(\mathcal{D}\) for each episode do  Initialize PPO hyperparameters for each Delay Sensitive Queue do  Construct the state space \(s_{k}\) for current time slot  According to \(s_{k}\), the action \(a_{k}\) is selected by \(\pi_{old}\), and save action probability \(P(\pi_{old}(a_{k}\mid s_{k}))\) if Execute \(a_{k}\), \(r_{t}(\theta)>1\)then  Abandon the threshold value result derived from this action  Break  Execute \(a_{k}\), \(s_{k}\gets s_{k+1}\), obtain the reward function  Store the current dataset \(\{s_{k},a_{k},P(\pi_{old}(a_{k}\mid s_{k})),r_{k}\}\) in buffer \(\mathcal{D}\) if\(a_{k}\
eq a_{1}\) (not the first action) then for\(n_{update}\)do  if time step reaches \(update\_timestep\)then  Calculate objective function \(\kappa^{\text{CLIP}}(\theta)\) using Equation (18), update PPO  policy loss function  Calculate the mean squared error using Equations (20) and (21), update the critic network endif if action space is sufficient & time step = action std decay freq (action standard deviation decay frequency) then  Update action standard deviation according to Equation \(\sigma_{t+1}=\max(\sigma_{t}-\text{Y},\sigma_{\text{min}})\) endif  Complete a series of updates of the agent endfor  Calculate cumulative reward \(sum(r_{k})\) and store it endif if the amount of data in buffer \(\mathcal{D}\) exceeds maximum capacity \(>K\)then  Update network parameters \(\theta_{\text{old}}\leftarrow\theta\), clear buffer \(\mathcal{D}\) endif \(s_{k}\gets s_{k+1}\) endif endfor
```  
**Algorithm 1** Optimal Frame Generation Time Threshold Calculation