## 2 Proposed Approach
### RL-Based Optimization Method
#### 2.2.1 Policy Generation
Afterward, to update the policy at a given iteration, we compute the action of a particle by sampling from a normal distribution, where the standard deviation is a function of the particle's distance from the global best particle's distance, described as  
\[\Delta x_{i,j}=\mathcal{N}_{\pi_{g}}(\zeta_{i,j},\psi(x_{i,j}-x_{g,j}), \tag{6}\]  
where \(g\) is the index of the globally best particle, and \(\psi\colon\mathbb{R}\to\mathbb{R}\) is a linear function provided by  
\[\psi(x_{i,j}-x_{g,i})=a+b(x_{i,j}-x_{g,j}). \tag{7}\]  
In Equation (7), the standard deviation for the update of particle \(i\) is a linear function of its distance from the particle that has the best location globally. In this work, the constants \(a=0.002\) and \(b=0.18\) produced good results over a range of training tests and thus were used for all the results shown in Section 4. Thus, the stochastic component of a particle's movement is large when far away from the global best to encourage exploration, but its update becomes more deterministic as the design optimum is approached to promote better exploitation. At each timestep, the particles take actions that are steps within the domain of the objective function. The reward function is designed such that actions that lead to subsequently poor objective value are penalized, while those that lead to adequate and fast convergence are rewarded. The reward function of particle \(i\) at the \(k\) iteration during a training episode is defined as  
\[\mathcal{R}_{i}^{k}=\left(\mathcal{R}_{i,1}^{k}+\mathcal{R}_{i,2}^{k}\right) \times\frac{k}{L}. \tag{8}\]  
In Equation (8), \(L\) is the total number of iterations that are allowed per episode, while the first reward component, \(\mathcal{R}_{i,1}^{k}\), represents an adjusted logarithmic reward function for the \(i\)th particle at iteration \(k\), described as  
\[\mathcal{R}_{i,1}^{k}=-\log\left(\alpha+|\hat{f}(\mathbf{x}_{i})^{k}|\right), \tag{9}\]  
where \(\alpha\) is a constant base sensitivity adjustment that controls the sensitivity of the reward structure as the particles approach regions close to the global optimum, set to \(10^{-11}\) in this study. In situations where high precision is required in the final objective value obtained, \(\alpha\) can be set to lower values. \(\hat{f}(\mathbf{x})^{k}\) is the normalized function value of each particle at the current iteration, defined as \(\hat{f}(\mathbf{x})^{k}=\frac{f(\mathbf{x})^{k}}{W}\), where \(W\) is the lowest observed value of \(f(\mathbf{x})\) observed through all the iterations in a given episode. The second component of the reward in Equation (8), \(\mathcal{R}_{i,2}^{k}\), represents a direct improvement reward structure, described as  
\[\mathcal{R}_{i,2}^{k}=\frac{f(\mathbf{x}_{i})^{k-1}-f(\mathbf{x}_{i})^{k}}{f( \mathbf{x}_{i})^{k}}. \tag{10}\]  
In two parts: the computation of the reward based on the current (\(f(\mathbf{x})^{k}\)) and previous function values (\(f(\mathbf{x})^{k-1}\)), and an adjusted logarithmic reward based on the normalized value \(\hat{f}(\mathbf{x})^{k}\). The adjusted logarithmic reward function \(R_{1}\) is defined as follows:  
\[R_{1}=-\log\left(\alpha\times 10^{-5}+|f^{*}(\mathbf{x})^{k}|\right) \tag{11}\]  
where \(\alpha\) is the base sensitivity adjustment  
\[f^{*}(\mathbf{x})^{k}=\frac{f(\mathbf{x})^{k}}{W} \tag{12}\]The reward based on the current and previous values, \(R_{2}\), is provided by  
\[\Delta x_{i,j}=\mathcal{N}_{\pi_{q}}(\zeta_{i,j},\sigma_{0}). \tag{5}\]0  
The final reward, adjusted for the progress through the episode and post-processed for specific conditions (e.g., a particle becoming stuck in a local optimum), is provided as  
\[\Delta x_{i,j}=\mathcal{N}_{\pi_{q}}(\zeta_{i,j},\sigma_{0}). \tag{5}\]1  
where \(W\) and \(L\) represent the worst observed objective value and the total length of the episode.