## 4 Experiment  
### Settings  
To evaluate the proposed tuning algorithm, simulation experiments are carried out on a four-screw cavity filter. Since frequent adjustment of screws on real products can cause damage to filter products and the tuning robot, we use a data-driven simulation model for experiments instead. The screw positions and S-parameters data collected randomly by the automatic tuning robot were used to build the model (Figure 6). In total, 2400 data pairs were collected. The robot tuning system is composed of a three-axis robotic arm, an industrial personal computer, and a control cabinet. Five servomotors are used for moving in the x, y, and z-direction, as well as the rotation for tuning the screw and the screw-nut. The robot can output specified rotation angles, so that desired tuning amount on the filter can be recorded. The \(S_{11}\) curve is observed from the VNA as the state for RL. To calculate the shaping reward, the central region of the \(S_{11}\) curve is first truncated. The cavity filter model is a combiner with four channels, and only one channel is considered for the test  
Figure 5: The policy generalization framework: (**a**) The learning phase: learning the filter mapping model. (**b**) The transferring phase: transferring the learned RL policy to a new individual with the mapping model. (Figure 7). The channel has four cavities with seven tuning screws, but for the sake of simplicity, only the tuning screws of four resonators are used in the experiment, indicating that the action dimension of RL is 4. Other screws remain in their standard position. The DDPG model used in our experiments is similar to the low-dimensional network structure in [26]. We extracted the S-parameters into a 20-dimensional feature set with Principle Component Analysis (PCA) as the input to the value network (critic) and the policy network (actor). The output of the policy network was a 4-dimensional set of continuous actions, representing actions taken on four screws. Both networks are feed-forward neural networks with the extracted feature set as the input. Both of them have two hidden layers, with 300 units on the first hidden layer and 600 units on the second. The activation function for the two hidden layers is Relu, while the output layer uses tanh as the activation function. The learning rate is 0.0001 for the actor network and 0.001 for the critic network. Weights are randomly initialized. For the filter product we used, the effective screw insertion position values were normalized and limited to \([-3,+3]\), representing the rotation performed to the screw from a standard position by \([-540,+540]\) degrees. For each tuning step, the action value output by the policy network is multiplied by a small size length to calculate the final tuning amount on each screw, i.e., the update of the screw positions would be  
\[z_{t+1}^{(i)}:=z_{t}^{(i)}+a_{t}^{(i)}\cdot\Delta z^{(i)}, \tag{14}\]  
where \(z_{t}^{(i)}\) is the position of the \(i\)-th tuning screw at time step \(t\), \(a_{t}^{(i)}\) is the output action of the policy network at time step \(t\), and \(\Delta z^{(i)}\) is a small step-size. The \(\Delta z^{(i)}\) of all four screws were set to the value of 0.25 in the experiment of this proposed work. To avoid getting stuck, we only trained the DDPG model with a maximum of 200 steps per episode and tested the model after every four episodes. In other words, if the algorithm has already tuned the filter 200 times but the filter has not yet been fully tuned, this episode will be stopped. At the beginning of each episode, the screw insertion positions were randomly initialized. The ratios for the three reward terms were set with \(\lambda_{1}=\lambda_{2}=1\),  
Figure 6: The automatic tuning robot system used for experiment. Figure 7: The combiner with four channels to be tuned in the experiment. The red spline denotes the channel used for experiment. \(\lambda_{3}=10\). The ratio values were selected based on experience or practical tests.