## 3 Overview of the Design Method
### Reinforcement Learning Approach for Optimizing Hairpin Filters
**Reward**: The design of a hairpin filter is typically associated with electrical characteristics such as passband range, insertion loss, and return loss. Consequently, the reward function must reflect the quality of the relevant design criteria. Using return loss as an example, it gauges the extent to which the filter aligns with the signal transmission within the passband. Generally, designers aim for the return loss of the filter to be below a certain threshold within the passband and above a certain threshold within the stopband, as depicted in Figure 4:  
The assessment of return loss (S11) can be achieved by assigning weighted scores to the performance at each frequency point in both the passband and the stopband. Specifically, within the passband, a straightforward approach involves calculating the score for each frequency point by determining the distance between a specific standard value \(l\) (for example, \(-15\) dB) and the S11 at the corresponding frequency point. If the S11 surpasses the standard value at this point, the score is set to be negative; conversely, if the S11 is below the standard value, the score is positive. To avoid the tendency of the system to overly focus on individual frequency points with high scores while neglecting the overall performance across the entire frequency range, we can apply the function \(g(x)=1-e^{-(1+x)}\) to scores greater than 0 to limit score inflation. Consequently, the score \(F_{p}(f)\) for a single frequency point \(f\) within the passband can be represented by Equation (7):  
\[F_{p}(f)=\left\{ \begin{matrix}l-S_{11}(f)&\text{if }S_{11}(f)>l\\ g(l-S_{11}(f))&\text{if }S_{11}(f)\leq l\end{matrix} \right. \tag{7}\]where \(S11(f)\) denotes the dB value of S11 corresponding to the frequency point \(f\), \(l\) represents the standardized metric within the passband, and \(F_{p}(f)\) stands for the evaluation score for a single frequency point \(f\). The overall performance score of the entire passband, represented by \(F_{p}\), can be derived by calculating the average score of all frequency points within the passband, as demonstrated in Equation (8):  
\[F_{p}\ =\ \frac{1}{n}\sum_{i=0}^{n-1}F_{p}(f_{i}), \tag{8}\]  
where \(n\) denotes the number of frequency points on the passband, \(i\) is the index of the frequency point, \(F_{p}(f_{i})\) represents the score estimation at the frequency point \(f_{i}\), and \(F_{p}\) signifies the overall score estimation after taking into account all the frequency points on the passband. Similarly, we can calculate the performance score \(F_{s}\) within the stopband. By combining these two scores with appropriate weights, we can compute the overall performance score \(F_{S_{11}}\) for the return loss:  
\[F_{S_{11}}=\beta_{1}F_{p}\ +\beta_{2}F_{s}, \tag{9}\]  
where \(\beta_{1}\) and \(\beta_{2}\) are the weighting coefficients, \(F_{p}\) represents the composite scores on the passband, \(F_{s}\) represents the composite scores on the stopband, and \(F_{S_{11}}\) represents the composite scores on the S11 after combining both the stopband and passband scores. If we want to consider the performance score of the insertion loss (S21) as well, a similar method can be applied and the detailed process is omitted here. Therefore, we can obtain the overall performance score of the filter as \(F_{total}=\gamma_{1}F_{S_{11}}+\gamma_{2}F_{S_{21}}\), where \(F_{S_{21}}\) represents the score of the insertion loss, taking into account both the stopband and passband performance. Here, \(\gamma_{2}\) and \(\gamma_{2}\) are weighting coefficients. Additionally, we introduce the concepts of short-term objective and ultimate objective to provide additional rewards \(r_{extra}\):  
\[r_{extra}=\left\{ \begin{array}{ll}10&if\ achieve\ E_{S}\\ 1&if\ achieve\ E_{s},\\ 0&otherwise\end{array} \right. \tag{10}\]  
where \(E_{s}\) represents the short-term objective and \(E_{S}\) represents the ultimate objective.