## 3 Intelligent Tuning Based On DDPG
### Formulations  
The RL method in this work follows the basic Markov Decision Process (MDP) settings. At each time step \(t\), the agent observes a state \(s_{t}\) from the environment. Then, it takes an action \(a_{t}\) to interact with the environment and the state of the agent is transferred to \(s_{t+1}\). At the same time, the reward \(r_{t}\) reflecting the current state and action pairs is obtained from the environment or specified by a human teacher. With the transition set \((s_{t},a_{t},r_{t},s_{t+1})\), the agent learns the policy with some optimization process, so that the expected future reward is maximized. In our previous work, the framework of DQN was applied. A value function \(Q\) is defined based on the Bellman equation to compute the expected future reward. Then, the optimal action to take at each time step is the one that maximizes the \(Q\) values,  
\[Q(s_{t},a_{t})=r_{t}+\gamma\max_{a_{t+1}}Q(s_{t+1},a_{t+1}), \tag{1}\]  
where \(s_{t},a_{t}\), and \(r_{t}\) are the state, action, and reward, respectively, at step \(t\); \(\gamma\) is a discounting factor. In DQN, a replay memory \(D\) is used to save transition sequences state, action, reward, new state. At each time step, a minibatch of transitions will be randomly selected from the replay buffer \(D\) for training the Q-network by minimizing the distance between the target Q-network and the current hypothesis model:  
\[\min_{\theta}(y_{i}-Q(s_{i},a_{t})|\theta), \tag{2}\]  
where the target is  
\[y_{i}=r_{i}+\gamma\max_{a_{t}^{\prime}}\hat{Q}\big{(}s_{i+1},a^{\prime};\theta \big{)}. \tag{3}\]  
Figure 3: S-parameter curves: the states observed during tuning. In this example, the design specifications require that, within the passband (731\(\sim\)747 MHz), the \(S_{11}\) curve should be less than \(-18\) dB (Return Loss), and the \(S_{12}\) curve should be no less than \(-2\) db (Insertion Loss). For the Out-Of-Band Rejection, it requires that the \(S_{12}\) curve should be less than \(-80\) dB within the range (704\(\sim\)716 MHz). These are the targets for cavity filter tuning. Instead of using Deep Convolutional Neural Networks to approximate the value function, we extracted a feature set from the original S-parameters as the state. We follow the Experience Replay and the Target Network of DQN to ensure the stability of the algorithm. We also used the Euclidean distance between the S-parameters curves as a reward function. In particular, if the current curve is getting closer to the target, the reward is large, and vice versa. This definition of this reward is somewhat coarse and should be modified in the light of human experience. In this work, the tuning algorithm is based on another important RL framework--DDPG--in which continuous actions are valid. Experience replay and Target Q-Network are preserved from DQN. DDPG is based on the actor-critic mechanism, where a critic function \(Q(s,a)\) and an actor function \(\mu(s|\theta^{\mu})\) are parameterized by two separate neural networks. The actor network is also called the policy network, which updates the policy with policy gradient. The critic network is similar to the value function in Q-learning, which evaluates the action from the actor network by updating the Q-values [26]. We define the tuning task as an MDP with the following settings. At each time step \(t\), the agent is in a state \(s_{t}\in\mathcal{S}\). In the tuning problem, the state \(s_{t}\) is defined as the S-parameters (\(S_{11}\)) curve (or some extracted features) drawn from the VNA. The agent performs an action \(a_{t}\in\mathcal{A}\) at each time step \(t\) to transfer state \(s_{t}\) to a new state \(s_{t+1}\). The action is defined as the tuning amount we can take, i.e., the adjustment of each screw element. Unlike our previous work [9], screw lengths are continuous values instead of discrete values, which is more accurate and reasonable. Thus, the action at any time step can be represented as a vector of dimension \(m\), where \(m\) is the number of valid tuning screws.