## 3 Overview of the Design Method
### Reinforcement Learning Approach for Optimizing Hairpin Filters  
In this study, we utilized the PPO reinforcement learning framework within a continuous action space to execute the task. Herein, we delineate the reinforcement learning configuration employed for the hairpin filter design task, ensuring the reinforcement learning methodology is suited to the task at hand. The reinforcement learning environment is defined by **states**, **actions**, and **rewards**, with a specific focus on the reward definition. A well-structured reward can steer the agent towards learning more effective strategies and directly influence the speed of convergence and the final performance of the algorithm. The amalgamation of reinforcement learning and artificial neural network techniques markedly diminishes the time spent in the iterative search for the optimal policy. Nonetheless, it remains vital to accurately assess the agent's behavior within the environment. Constructing a logical reward function that aligns with the practical design requirements is a fundamental aspect of reinforcement learning. The ANN model is deployed as a substitute for an EM simulator, such as the Ansys High-Frequency Structure Simulator (HFSS), as discussed in the preceding section. The ANN model provides a more rapid response to the agent's actions than solvers, thereby facilitating more efficient learning. **State**: The state is represented by concatenating two types of feature vectors: (1) a vector comprising geometric design parameters of the hairpin filter, namely, [\(W\), \(Lt\), \(L_{1}\), \(L_{2}\), \(d_{1}\), \(d_{2}\), \(tw\), \(sc\)\(w\)\(sg\)], which are normalized based on the upper and lower limits of each parameter's design space; and (2) the S-parameters (S11 and S21) of the corresponding designed filter at specific frequency points within the simulation frequency range. In this instance, we design within a simulation frequency band from 60 GHz to 125 GHz with a 1 GHz interval, resulting in a total of 66 frequency points for both S11 and S21, amounting to a total of 132 frequency points. ```
1:Initializing actor policy network parameters \(\pi_{\theta}\) and \(\pi_{\theta_{old}}\); critic value network parameters \(V(w)\). 2:Initialize the replay buffer D.
3:Randomly initializing the design of the hairpin filter. 4:for eposide = 1 to \(M\)do
5:Calling the forward network of ANN to predict the S-parameters of the current design of the hairpin filter. 6:Initializing the state \(s_{1}\). 7:for step = 1 to \(T\)do
8: Select action \(a_{t}\) at according to the current policy \(\pi_{\theta}\). 9: Adjusting the design of the hairpin filter to perform the corresponding action. 10: Calculating the S-parameters of the newly designed hairpin filter based on the ANN model. 11: Generating the new state \(s_{t+1}\), and computing the reward \(r_{t}\). 12: Store the transition (\(s_{t}\), \(a_{t}\), \(r_{t}\), \(s_{t+1}\)) in D.
13:if data volume of D \(\geq\) minimum batch size then
14: Estimate advantages \(\hat{A}_{t}=\sum_{t^{\prime}>t}\gamma^{t^{\prime}-t}r_{t^{\prime}}-V(s_{t};w)\). 15:for k = 1 to K do
16:\(\pi_{\theta_{old}}\leftarrow\pi_{\theta}\)
17: Calculate \(L(\theta)\) based on Equation (6). 18: Update the parameters of the policy network \(\pi_{\theta}\):\(\theta\leftarrow\theta+\pi\
abla\theta L(\theta)\). 19: Update the critic network by minimizing the loss \(L(w)=\frac{1}{t}\sum_{t=1}^{T}(\hat{A}_{t})^{2}:w\gets w-\beta\
abla wL(w)\)
20:endfor
21:endif
22:endfor
23:endfor
```  
**Algorithm 1** Automatic design of hairpin filters based on PPO. **Action**: The agent's actions govern its interaction with the environment. This framework employs a continuous action space defined as \([-1,1]^{m}\), where m corresponds to the number of geometric parameters for designing the hairpin filter. In this paper, \(m=8\). For instance, when \(a_{t}^{i}=1\), it signifies an increase in the upper limit value of the ith geometric parameter size in the current design space of the hairpin filter.