## 2 Proposed Approach
### RL-Based Optimization Method
#### 2.2.1 Policy Generation
By learning to maximize reward, the particles learn to take actions that lead to high rewards, namely actions that find regions close to the global optimum on the surface. In this study, all the particles used for policy generation share the same policy for choosing their actions within the design space. Furthermore, each dimension within the design space is updated separately. Therefore, the dimensionality of the search space and the number of particles can be readily varied during model deployment to solve a global optimization problem. Policy generation is performed using the 2-dimensional sphere function and 10 particles. The 2-dimensional sphere function was selected for policy generation due to its simplicity and well-understood properties. This allows us to focus on the learning process of the algorithm without introducing complexities from more elaborate functions. The neural network serves as the actor-critic architecture in our implementation of the proximal policy optimization (PPO) algorithm. It consists of 2 hidden layers, each containing 32 neurons, activated using the hyperbolic tangent function. This architecture was chosen to provide sufficient complexity for both the actor (policy) and critic (value function) components of PPO. The actor part of the network learns to generate actions that guide the optimization process, while the critic part estimates the value function, which is crucial for computing advantages in the PPO algorithm. This setup enables the network to learn effective optimization strategies that balance exploration and exploitation based on the current state of the swarm. The training was performed for 2000 episodes (which is equivalent to 40,000 iterations). Figure 2 shows the training progression of reinforcement learning policy, i.e., the average return against the number of episodes. In the figure, the average return is the mean cumulative reward received by the particles over an entire training episode. Initially, the graph exhibits a high degree of variability, as indicated by the pronounced spikes, especially noticeable in the early episodes. This variability can be attributed to the particles' random initialization, which can occasionally place them in advantageous positions within the search space, resulting in higher returns for those episodes. As training progresses, we observe a discernible upward trend in both the raw and smoothed average return, suggesting that the particles are learning and improving their policies over time. The variance in returns, especially noticeable in the initial episodes, indicates that, while the particles occasionally stumble upon highly rewarding states due to their random initialization, their policy has not yet converged, and their performance is not consistently high. Importantly, the reduced variability and higher average returns in later episodes signify that the particles are beginning to understand the structure of the problem. It shows  
\[\Delta x_{i,j}=\mathcal{N}_{\pi_{q}}(\zeta_{i,j},\sigma_{0}). \tag{5}\]2 Average return obtained during the policy generation phase of Deephive. that the particles are less reliant on the chance of a favorable initialization and more on the strength of their learned policies. This trend is indicative of the particles transitioning from excessive exploration to exploitation, refining their strategies to consistently navigate towards more rewarding states as they gain experience. Moreover, the convergence pattern towards the later episodes, with less-pronounced spikes and a more stable average return, hints at the particles achieving a balance between exploration and exploitation. The particles are no longer experiencing dramatic shifts in performance from one episode to the next, which can be taken as a sign of policy maturation. The learning process has enabled them to reliably identify and execute actions that lead to favorable outcomes regardless of their starting positions in the search space.