## 4 Simulations and Analyses
Figure 9: Comparison of anti-jamming performance of different algorithms. The clipping ratio \(r(\theta)\) of a new and an old policy is an important hyperparameter for Meta-PPO algorithms that is used to limit the size of policy updates, ensuring that the difference between these two policies is not too large. If \(r(\theta)\) is too small, then beneficial policy updates might be hindered, while if \(r(\theta)\) is too large, then overly aggressive and potentially harmful updates might happen. Therefore, this hyperparameter must be chosen properly to optimize the algorithm performance. Figure 10 shows the normalized network output under different clipping ratios while other parameters remain unchanged for the simulation analyses. As can be seen, when clipping ratio \(r(\theta)\) of the new and old policies is modified, the speed at which the Meta-PPO algorithm learns the optimal policy remains unchanged. The optimal policy appears after about 100 episodes. When the clipping ratio is chosen to be \(r(\theta)=0.2\), the algorithm approaches its optimal upper limit. At this point, the Meta-PPO algorithm allows the policy to make necessary updates without over-updating. Therefore, the best clipping ratio choice is \(r(\theta)=0.2\). The MAML gradient learning update rate, denoted as \(\alpha\), is another crucial hyperparameter in the Meta-PPO algorithm, determining the update rate of parameters in the inner loop. The purpose of the inner loop in the MAML algorithm is to learn a specific task rapidly, while the outer loop aims for generalization across multiple tasks. As shown in Figure 11, when other parameters remain unchanged and only the MAML gradient learning update rate \(\alpha\) is modified, the speed at which the Meta-PPO algorithm learns the optimal policy remains unchanged, converging around 100 episodes. The highest normalized throughput is achieved when \(\alpha=0.1\). At this point, in the inner loop, the model parameter updates are moderate, allowing the model to adapt to the specific tasks and environments of the basic learner without over-fitting. This prevents a reduction in the generalization capability in the outer loop. Figure 12a illustrates the curve of the action loss for the Meta-PPO algorithm varying with the number of episodes. Typically, a loss function signifies a negative return of a strategy as expected, which means that a loss function should have negative values. The optimization process is expected to maximize this return, which can be translated to minimizing the negative value of this loss function. As seen from Figure 12a, during the strategy optimization process of the proposed algorithm, the strategy loss steadily rises from a notably large negative value and then starts to converge to a stable value after the 100 th episode. This suggests that the strategy has been refined and has reached an optimal point. Figure 10: Comparison of normalization throughput at different clipping ratios. Figure 12b presents the curve of the Meta-PPO algorithm's strategy entropy varying with the number of episodes. Entropy is widely regarded as a measure of the randomness or uncertainty of a strategy. Enhancing the entropy during a strategy's optimization process means exploration in a wider action space is encouraged, which is beneficial to the strategy optimization since a strategy with a high-entropy optimization process means that the strategy is optimized over a wider range of actions rather than only over a few concentrated ones. As seen from Figure 12b, during the algorithm's policy optimization, the entropy escalates from 0 to approximately 2.71 after the 100 th episode and subsequently oscillates between 2.7 and 2.71. Notably, even when the policy loss stabilizes around the 100 th episode, the policy entropy does not plummet to zero or near-zero values. This indicates that the Meta-PPO's learning trajectory initially transitions from a deterministic strategy, diversifying over time, and then stabilizing while retaining a degree of exploration.