## 4 Experiment
### Results  
Figure 8 describes the corresponding results in three cases, respectively. The figures on the left show the tuning steps during training, while the figures on the right show the tuning steps during testing (i.e., simply run after every four training episodes). The vertical axes show the number of tuning steps in each episode. At the beginning of the explorations, the agent can hardly make the filter be fully tuned in 200 steps. As the exploration goes on, the agent takes fewer steps to succeed. In other words, the fewer steps in one episode, the better; further, the fewer episodes used to converge, the better. When applying the knowledge-inspired shaping reward \(r_{t}^{\text{direction}}\), the algorithm almost succeeded in finding the optimal tuning strategy in about 100 episodes. In contrast, using the Euclidean distance reward \(r_{t}^{\text{distance}}\) and without any shaping reward (i.e., \(r\) consists only of \(r^{\text{goal}}\) and \(r^{\text{limit}}\)), it takes more than 250 episodes for the algorithm to converge to an optimal policy. The results indicate the following: (i) By applying the DDPG algorithm with and without reward shaping, the cavity filter can both be fully tuned. This indicates that the proposed RL method is effective to learn how to tune a detuned filter to its target from scratch. (ii) By using a carefully prescribed shaping reward \(r_{t}^{\text{distance}}\), the RL agent takes fewer episodes to converge to the optimal policy than using no shaping reward. This is because, without any shaping reward, the agent can only receive feedback when the cavity filter is finely tuned, which always takes a long period of time. On the contrary, adding an extra shaping reward provides instant feedback at each training step; thus, the total reward will not be sparse anymore and the learning is accelerated. (iii) An inappropriate shaping reward will not accelerate learning. Simply computing the Euclidean distance to the target for shaping the reward has no obvious advantage over without any shaping reward. Although it seems reasonable to use the distance for evaluating the tuning state, it is not good at handling the cases when the S-parameters curve has a horizontal shift. Figure 9 illustrates the computation of the Euclidean distance of three different S-parameter curves. Only the points above \(-20\) dB are considered for computing the distance, which have been marked red. In Figure 9a, the curve is shifted away from the target vertically. A total of 114 points are included for calculation, and the computed Euclidean distance is 76.11. In Figure 9b, the curve almost reaches the target, and only 13 points are included for calculation, with a final distance value of 4.21. In Figure 9c, the curve also seems to be approaching the target and only 18 points do not meet the requirement, but the corresponding distance value reaches 51.11, which is relatively high. Compared with the curve in Figure 9b, the shape of the curve in Figure 9c is almost the same, but has a slight shift to the right. This indicates that a small horizontal shift will cause a large distance value. Therefore, simply using the Euclidean distance for evaluating the reward is not always appropriate. Figure 10 shows the learning curves of the RL process. Figure 10a shows the accumulated rewards. We can find that with knowledge-inspired reward \(t_{l}^{\text{direction}}\) or the Euclidean distance reward \(r_{l}^{\text{distance}}\), the reward value is accumulated faster than without any shaping reward because the agent obtains shaping reward values more frequently, and a sparser reward value of \(+10\) will be added if the filter is successfully tuned. A steep rise in accumulated reward denotes a more frequently added \(+10\). We also reserved a set of randomly obtained states and computed their average \(Q\) values after each training episode. The blue and green curve in Figure 10b increase more quickly and steadily than the red one, which also reflects the effectiveness of the shaping reward.