## 2 Related Work  
### Automatic Cavity Filter Tuning  
Research on automatic tuning algorithms has been going on for a long time. Studies in [3; 10] utilized the time-domain response of the input reflection characteristics of the filter. This technique requires an optimal response as a template and relies on accurate estimates of features, which is not realistic in some cases. A sequential adjusting method based on Group Delay was proposed by [11]. This technique depends on the hypothetical computation of the group delay for each resonator. Notwithstanding, when processing the group delay of some resonator, other resonators should be short-circuited, which is infeasible in practice. Other methods based on model parameter extraction [4], poles and zeros [12], coupling matrix decomposition and extraction [13], etc. are also theoretical but almost invalid in most practical cases. Another set of technologies utilize state-of-the-art artificial intelligence. [PERSON] proposed a novel tuning strategy building a data-driven model for the cavity filter product [5]. A feed-forward neural network was constructed to model the relationship between S-parameters and the screw deviations \(\Delta Z\), i.e., to train a nonlinear mapping \(f:S\rightarrow\Delta Z\). When tuning a detuned filter, the model is used to predict the screw deviations so that the desired S-parameters are approximated. Similarly, [PERSON] et al. exploited Support Vector Regression (SVR) to construct metamodels between screw rotations and coupling matrix change [6; 14]; the screw rotations are not measured but obtained from two optimization procedures, and the calculation of coupling matrix also brings uncertainties. Furthermore, fuzzy logic control [7; 15], neural-fuzzy control [16], Extreme Learning Machine (ELM) [17], and regularized deep belief network (R-DBN) [18] have also been applied in modeling cavity filters. These data-driven methods learn models simply from pregathered data without analyzing the complex theoretical and physical characteristics of the filter product, eliminating the error produced by the difference between the ideal model and the real product. However, these data-driven modeling techniques have constraints in generalization. In other words, to build a data-driven model of a cavity filter, a large number of data pairs should be collected; further, for every filter product type, data pairs should be recollected. Furthermore, the trained model cannot always adapt to the difference between individuals of the same product type, which limits the practicability. [PERSON] et al. proposed new methods by integrating intelligent algorithms with filter theories, which also had good tuning effects [19; 20], but the essence of the method is still fitting the screw and the scattering parameters, which might have the same problem as previous methods. Different from the literature, we do not focus on any specific cavity filter model but make more efforts to study the tuning process. The tuning process of an expert is a typical exploration and exploitation process. The tuning expert interacts with the environment by adjusting the screws and observes the resulting S-parameter curves from the VNA. Then, a new tuning action is taken and the S-parameters are changed accordingly. Iterationsare repeated until the filter is fully tuned. In [9], reinforcement learning (RL) approach was first tried to model the tuning process in the framework of Deep Q-Learning (DQN). However, this work only considered two screws, and the action was discretized for Q-learning. In [21], a novel automatic fine-tuning approach using DQN was proposed for planar bandpass filters. A forward model was trained by supervised learning to construct the environment and then interact with the agent in a DQN setting. As an improvement of DQN in handling with continuous state and action spaces, DDPG has been applied for tuning cavity filters with more tuning elements [22]. In [23], a new learning algorithm called Deep Imitation-Reinforcement learning for Tuning (DIRT) was proposed.