## 3 Meta-PPO Anti-Jamming Intelligent Decision Algorithm
### Base Learner
First of all, during the updating process, the temporal difference error \(\delta_{T}\) is calculated according to Equation (6):  
\[\delta_{T}=R_{T}+\gamma V_{\mu}(S_{T+1})-V_{\mu}(S_{T}) \tag{6}\]  
In the equation, \(\delta_{T}\) represents the single-step temporal difference error at time \(T\). \(r_{T}\) represents the immediate reward at time \(T\). \(\mu\) represents the parameters of the Critic network. \(V_{\mu}(S_{T+1})\) is the estimated state value of state \(S_{T+1}\) at time \(T+1\), output by the Critic network. \(V_{\mu}(S_{T+1})\) is the estimated state value of the next state \(S_{T+1}\) at time \(T+1\), and \(V_{\mu}(S_{T})\) is the output of the Critic network. After calculating the temporal difference error for each time step, the target for updating the Critic network is calculated according to Equation (7). The mean squared error is used as the loss function for updating the Critic network. The equation for calculating the Critic loss function can be expressed as Equation (8). Then, the Critic network is updated through backpropagation. \[y_{T}=A_{T}+V_{\mu}(S_{T}) \tag{7}\]  
In the equation, \(A_{T}\) represents the estimated advantage function at time \(T\). \[L(\mu)=\frac{1}{\text{batch\_size}}\underset{T}{\sum}\left(y_{T}-V_{\mu}(S_{T })\right)^{2} \tag{8}\]  
In the equation, \(y_{T}\) represents the update target of the Critic network. \(V_{\mu}(S_{T})\) represents the estimated state value of state \(S_{T}\) at time \(T\). The loss function calculation equation for the Actor network is shown in Equation (9). The Actor network parameters are updated by minimizing the loss function. \[L(\theta)=\frac{1}{\text{batch\_size}}{\sum_{T}min(r_{T}(\theta)A_{T},clip(r_{T} (\theta)1-\varepsilon,1+\varepsilon)\,A_{T})} \tag{9}\]  
In the equation, \(r_{T}(\theta)=\frac{\pi_{0}(a_{T}|S_{T})}{\pi_{0}(a_{T}|S_{T})}\) represents the ratio of the new policy to the old policy. \(\theta\) represents the parameters of the new policy network. \(\theta_{1}\) represents the parameters of the old policy network. \(A_{T}\) represents the estimated advantage function at time \(T\). In the inner loop, after updating the model parameters \(k\) times, the process will enter the outer loop. In the outer loop, the Meta-PPO algorithm will calculate the update gradient according to Equations (10) and (11) and update the model parameters again to minimize the expected loss function over the task distribution, aiming to find a set of initial parameters that can quickly adapt to new tasks. \[\mu=\mu+\varepsilon\Big{(}\widetilde{\mu}-\mu\Big{)} \tag{10}\]  
In the equation, \(\mu\) represents the parameters of the Critic network before learning, \(\widetilde{\mu}^{\prime}\) represents the parameters after learning on task \(\tau\), and \(\varepsilon\) is the update step size. \[\theta=\theta+\varepsilon\Big{(}\widetilde{\theta}-\theta\Big{)} \tag{11}\]  
In the equation, \(\theta\) represents the parameters of the Critic network before learning, \(\widetilde{\theta}^{\prime}\) represents the parameters after learning on task \(\tau\), and \(\varepsilon\) is the update step size. For the assumed communication environment in our research, the decision model of the basic learner based on the proposed Meta-PPO algorithm is shown in Figure 5. When a communication process starts, the Actor network of the PPO algorithm receives the communication environment values perceived by the upper layer as its input, and then it outputs action commands to control the decisions made by the communication system. During the interaction with the external environment, the generated data \((S_{t},A_{t},R_{t},S_{t+1})\) will be stored in the experience pool for subsequent model learning.