## 3 Overview of the Design Method
### The PPO in the Reinforcement Learning Algorithm  
This section primarily elucidates the foundational concepts of reinforcement learning and the rudimentary principles of the PPO algorithm. Reinforcement learning can be conceptualized as a Markov Decision Process (MDP), wherein the current decision is exclusively contingent upon the present state, devoid of any influence from previous states. Thus, in the reinforcement learning process, we define a sequence of states, actions, and rewards at each time step, akin to \(\tau=\{s_{1},a_{1},r_{1},s_{2},a_{2},r_{2},\ldots,s_{T},a_{T},r_{T}\}\), where \(T\) denotes the termination time step of the learning process. The objective of reinforcement learning is to maximize future rewards by discounting them using a discount factor \(\gamma\). We define the discounted return as \(R_{t}=\sum_{T}^{t^{\prime}}\gamma^{t^{\prime}-t}r_{t^{\prime}}\). PPO is a type of policy gradient algorithm. Its goal is to maximize the discounted return by directly enhancing the parameters, denoted as \(\theta\), of the policy denoted as \(\pi_{\theta}\). Presently, the estimators that are widely used for computing policy gradients can be expressed using the following Equation (3):  
\[\hat{g}=\hat{\mathbb{E}}_{t}[\
abla_{\theta}\log\pi_{\theta}(a_{t}\mid s_{t}) \hat{A}_{t}], \tag{3}\]  
where \(\hat{A}_{t}\) represents the estimate of the advantage function at the current time step, \(E_{t}[\ldots]\) signifies the empirical average of a finite batch of samples, and \(\hat{g}\) can be obtained by differentiating the loss function \(L^{\text{PG}}(\theta)\) with respect to the parameter \(\theta\):  
\[L^{\text{PG}}(\theta)=\hat{\mathbb{E}}_{t}[\log\pi_{\theta}(a_{t}\mid s_{t}) \hat{A}_{t}], \tag{4}\]  
Figure 3: Schematic diagram that describes the process of modeling the S-parameters of a hairpin filter using artificial neural network technology.  
where the meanings of the various symbolic variables are the same as in Equation (3): \(\pi_{\theta}(a_{t}\mid s_{t})\) represents the possibility of taking action \(a_{t}\) under state \(s_{t}\), and \(\hat{A}_{t}\) represents the advantage function at the current time step.  
However, this updating method of policy gradients is susceptible to excessively updating the policy parameters, which disrupts the stability of the training process. To address this issue, PPO introduces a clipping mechanism to limit the magnitude of updates between the new and old policies, enabling a more stable policy update. Initially, we calculate the probability ratio between the new and old policies, denoted as \(r_{t}(\theta)\), with the following Formula (5):  
\[r_{t}(\theta)=\ \frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{old}}(a_{t}|s_{t})}, \tag{5}\]  
where \(\pi_{\theta}(a_{t}|s_{t})\) represents the probability of selecting action \(a_{t}\) by the current policy in state \(s_{t}\), while \(\pi_{\theta_{old}}(a_{t}|s_{t})\) represents the probability of selecting action \(a_{t}\) by the original policy before the update in state \(s_{t}\). Based on \(r_{t}(\theta)\), PPO introduces a new objective function, denoted as \(L^{CLIP}(\theta)\):  
\[L^{CLIP}(\theta)=\ \hat{\mathbb{E}}_{t}\big{[}min\big{(}r_{t}(\theta)\hat{A}_{t },clip(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}\big{)}\big{]}, \tag{6}\]  
where \(\epsilon\) is a hyperparameter. The second term in \(min\), \(clip(r_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t}\) modifies the surrogate objective by clipping the probability ratio, eliminating incentives outside the range of \([1-\epsilon,1+\epsilon]\) for \(r_{t}(\theta)\). Finally, the minimum value between the pre-clip and the post-clip is taken. This method is easier to implement and more effective compared to the approach proposed by TRPO [38].  
In Algorithm 1, we provide the pseudocode for the specific steps of designing a Kalman filter using the PPO reinforcement learning algorithm. We also provide additional details in Algorithm 1 regarding the implementation and deployment of the ANN forward model in our proposed method.