## 3 Meta-PPO Anti-Jamming Intelligent Decision Algorithm
### Base Learner  
In the proposed Meta-PPO algorithm, the base learner learns from communication anti-jamming tasks, which are similar but are independent from each other. All tasks are finished on the same communication system, facing similar jamming environments. The only difference is that the jamming parameters used by the jammer for each task are different. Therefore, the intelligent anti-jamming strategies used by the algorithm for each task are similar. The basic functions of the base learner performed in each task are as follows:  
(1) As shown in Figure 4, for the current task, using the PPO algorithm to find the pattern of the jamming signals from the jammer and obtaining the optimal communication strategy under the current communication environment. (2) Obtaining the experience from the meta-learner that is helpful for completing the current task, including the initial model, initial parameters, etc. (3) Feeding the learned model and parameters back to the meta-learner after the current task learning is completed. The proposed algorithm adopted the idea used in MAML meta-learning. During the base learner's learning process, the methods reported in [26; 27; 28; 29; 30] were adopted and the concepts used in the PPO deep reinforcement learning algorithm such as the experience replay, valuation neural network, and target neural network were also adopted. During the algorithm learning phase, the base learner uses the \(\varepsilon-greedy\) strategy for learning updates. Under this strategy, the action with the highest reward is selected with a  
Figure 4: Decision model of the basic learner based on the PPO algorithm. Figure 3: Meta-PPO algorithm framework. probability of \(1-\varepsilon\), and actions are randomly selected with a probability of \(\varepsilon\), as shown in Equation (4):  
\[\pi(a|s)=\left\{ \begin{aligned} & argmaxQ(s,a)\,\ 0<& x\leq 1- \varepsilon\\ &\text{random selection}\,\ 1-\varepsilon<x<1\end{aligned} \right. \tag{4}\]  
The goal of adopting meta-learning in the proposed algorithm is to let its learning process quickly adapt to new tasks. In order to achieve this, the Meta-PPO algorithm uses a few samples of multiple similar tasks as the input data for the learning process to reduce the expected loss of the algorithm on multiple tasks. And by doing so, the direction is set for the parameter updates. Thus, a set of model parameters, adapting to a new task, can be acquired in a faster manner. Therefore, the optimized objective function of the proposed algorithm can be expressed with Equation (5):  
\[min_{\theta}E_{\tau\sim p(\tau)}\left[L_{\tau}\left(U_{\tau}^{k}(\theta) \right)\right] \tag{5}\]  
In the equation, \(p(\tau)\) represents the task distribution. \(\theta\) represents the parameters of the model. \(U_{\tau}^{k}(\theta)\) indicates that the model parameters are updated \(k\) times using a small amount of data collected from task \(\tau\). \(L_{\tau}\left(U_{\tau}^{k}(\theta)\right)\) represents the loss function of the model on task \(\tau\). The parameter updating process in the meta-training phase of the proposed algorithm can be divided into two parts, an inner loop and an outer loop, according to Equation (6). In the inner loop, the Meta-PPO algorithm uses a small amount of data from a randomly chosen task \(\tau\) as the learning data to update the model parameters, reducing the model's loss on task \(\tau\). In this loop, the model parameter updating process is the same as the PPO algorithm proposed in [26; 27; 28; 29; 30]. The neural network of the algorithm learns from several batches of data on the randomly chosen tasks. In each round, the agent performs actions according to the policy of the Actor network, interacting with the environment to collect experience, and then saves the collected experience to an experience pool. When the number of collected experiences meets the model update threshold, the model parameters will be updated.