## 2 Proposed Approach
### RL-Based Optimization Method
#### 2.2.1 Policy Generation  
As with every reinforcement learning problem, the environment is the agent's world. An agent cannot influence the rules of the environment by its actions. Nevertheless, it can interact with the environment by performing actions within the environment and receiving feedback in the form of a reward. In this work, the environment is customized from the Open-AI gym library [33]. The problem is framed as an episodic task, where each episode consists of 25 updates to the state of the swarm (i.e., design iterations), after which the optimization attempt is terminated regardless of whether the swarm converges or not. At the beginning of each episode, the particles take random positions within the domain of interest. These positions and objective function values are scaled to fall within a \([0,1]\) interval, which enables the resulting policy to apply to problems where the order of magnitude of the domain bounds and objective function are different. However, in many cases, the minimum and maximum values of the objective function may not be known a priori. Therefore, at a given iteration, the range chosen for normalization is based on the currently known global best and global worst objective function values. We denote the position vector for the \(i\)th particle in a swarm of \(N\) particles as \(\mathbf{x}_{i}=[x_{i,1},x_{i,2},\ldots,x_{i,D}]\), where \(D\) is the number of dimensions (or independent variables) in the objective function, \(f\). The elements of \(\mathbf{x}_{i}\), denoted \(x_{i,j}\), represent the position of particle \(i\) in the \(j\)th dimension. At a given iteration, each particle communicates with a random neighboring particle from the swarm [34], denoted as \(\mathbf{x}_{ni}\), the immediate previous location represented as \(\mathbf{x}_{ni}\), and the best particle in its neighborhood [35], \(\mathbf{x}_{i}^{*}\). Finally, \(\mathbf{\bar{x}}_{i}\) is a vector that denotes the historical best position of particle \(i\), i.e., the location of the maximum objective value that particle \(i\) has reached since the swarm was initialized. The observation vector for particle \(i\) in the \(j\)th dimension is provided by  
\[\boldsymbol{\zeta}_{i,j}=\Big{[}x_{i,j}-\bar{x}_{i,j},x_{i,j}-x_{ni,j},x_{i,j}- x_{pi,j},x_{i,j}-x_{ij}^{*}\Big{]}. \tag{4}\]  
At every training epoch, the observation, \(\boldsymbol{\zeta}\), is fed to the actor neural network, which outputs a mean action. The particles' actions are then sampled from a distribution created with the mean action and a set standard deviation. The policy generation has two phases. During the first stage, the environment is explored more aggressively so that the particles are exposed to a wide range of states within the environment. During this stage, the position of particle \(i\) in the \(j\)th dimension, \(x_{i,j}\), is updated using a normal distribution with the mean obtained from the PPO policy and a fixed standard deviation of \(\sigma_{0}\):  
\[\Delta x_{i,j}=\mathcal{N}_{\pi_{q}}(\zeta_{i,j},\sigma_{0}). \tag{5}\]  
Figure 1: Description of the policy generation and deployment phases of Deephire. In this work, the value of \(\sigma_{0}=0.2\) was selected based on empirical testing as it provided a good balance between exploration and exploitation during the initial training phase. Equation (5) is used to update the position vector during the first 2500 iterations.