## 2 Proposed Approach  
### Reinforcement Learning  
Reinforcement learning (RL) involves an interaction between an agent and its environment, wherein the agent receives a reward (or punishment) based on the quality of its actions. The agent takes an action that alters the state of an environment, and a reward is provided based on the quality or fitness of the action performed. Each agent tries to maximize its reward over several episodes of interaction with the environment and by so doing learns to select the best action sequence. Typically, the reinforcement learning problem is expressed as a Markov decision process (MDP). The state and action space in a continuous setting of a finite-horizon MDP is defined by the tuple \((S,\mathcal{A},\mathcal{S}\ p_{0},p,c,\gamma)\), where \(S\) represents the states, \(\mathcal{A}\) is the set of actions, and \(p_{0}\) is the probability density function over initial states \(p_{0}\colon\mathbb{S}\longrightarrow\mathbb{R}^{+}\). \(p\) is the conditional probability density over successor states given the current state and action, \(p\colon S\times A\times S\rightarrow\mathbb{R}^{+}\), \(c\) is the function that maps states to reward \(c\colon S\rightarrow\mathbb{R}\), and \(\gamma\in(0,1]\) is the discount factor [22]. Specifically, we employ the PPO algorithm, which belongs to a class of RL methods called policy gradient methods [27], to train the policy (denoted \(\pi_{\theta}\)) in this work. The policy gradient methods operate by directly optimizing the policy, which is modeled by a parameterized function of \(\theta\), i.e., \(\pi_{\theta}(a|s)\). The policy is then optimized using a gradient ascent algorithm, which seeks to maximize reward. In the original policy gradient approach [28], the goal is to maximize the surrogate objective defined as  
\[\mathcal{L}^{CPI}(\theta)=\mathbb{E}\bigg{[}\frac{\pi_{\theta}(a|s)}{\pi_{ \theta_{old}}(a|s)}\cdot\hat{A}\bigg{]}=\mathbb{E}\big{[}r(\theta)\hat{A} \big{]} \tag{1}\]  
where \(r(\theta)=\frac{\pi_{\theta}(a|s)}{\pi_{\theta_{old}}(a|s)}\) is the ratio of the current policy to an old policy (or baseline policy) and is called the probability ratio. The superscript \"CPI\" stands for \"conservative policy iteration\", \(\mathbb{E}\) is the expectation or expected value, \(\theta\) represents the weights or the policy network parameters, and \(\hat{A}\) is the advantage that measures how good an action is in each state. During training, excessive policy updates can sometimes occur, leading to \begin{table}
\begin{tabular}{l l l l} \hline \hline
**Approach** & **Base Algorithm** & **RL Integration** & **Policy Discovery** \\ \hline ACO & Ant behavior & None & Fixed rules \\ ABC & Bee behavior & None & Fixed rules \\ CSA & Cuckoo behavior & None & Fixed rules \\ GSO & Glowworm behavior & None & Fixed rules \\ PSO & Bird flocking & None & Fixed rules \\ TLBO & Teaching–Learning & None & Fixed rules \\ SSA & Salp chains & None & Fixed rules \\ RLPSO & PSO & Parameter tuning & Partial \\ NRLPSO & PSO & Velocity update & Partial \\ RLMPSO & PSO & Operation selection & Partial \\ Deephive & N/A & Full integration & Direct discovery \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison of Deephive with previous SI- and RL-enhanced optimization approaches. the selection of a poor policy from which the agent is unable to recover. To reduce the occurrence of such actions, trust region optimization approaches, such as PPO, have been developed. PPO prevents excessive policy updates by applying a clipping function that limits the size of the updates as follows:  
\[\mathcal{L}^{CLIP}(\theta)=\hat{\mathbb{E}}\big{[}\min\big{(}r(\theta)\hat{A}, \ \text{clip}(r(\theta),1-\epsilon,1+\epsilon)\hat{A}\big{)}\big{]} \tag{2}\]  
The first term on the right-hand side of Equation (2) is \(\mathcal{L}^{CPI}(\theta)\), as defined in Equation (1). The second term, \(\text{clip}(r(\theta),1-\epsilon,1+\epsilon)\hat{A}\), is the modification that clips the probability ratio to the interval \([1-\epsilon,1+\epsilon]\).