## 1 Introduction
[PERSON] and [PERSON] [18] proposed the use of REINFORCE, an RL algorithm for a gradient-based parameter adjustment method, showing promising outcomes when applied to the Differential Evolution algorithm. Additionally, [PERSON] et al. [19] developed an NRLPSO, a reinforcement-learning-based PSO variant that generates velocity vectors for particles at each timestep through a unique reinforcement-learning-based strategy. Furthermore, reinforcement learning has been applied to optimize other algorithms, such as Differential Evolution [20] and the firefly algorithm [21], indicating its wide-ranging utility in enhancing algorithmic performance across various domains. Ref. [22] developed a framework for learning how to perform gradient-based optimization using reinforcement learning. Ref. [23] presented a variant of PSO based on the reinforcement learning methods with superior performance over various state-of-the-art variants of PSO. In their approach, the particles act as independent agents, choosing the optimal communication topology using a Q-learning approach during each iteration. Ref. [24] developed a reinforcement-learning-based memetic Particle Swarm Optimizer (RLMPSO), where each particle is subjected to five possible operations, which are exploitation, convergence, high jump, low jump, and local fine-tuning. These operations are executed based on the RL-generated actions. In this study, we introduce Deephyive, a new approach to developing swarm-based design optimizers, using a direct integration of reinforcement learning. \"Deep\" refers to the deep neural network trained via reinforcement learning to implement the optimization policy, while \"hive\" draws an analogy to the collective behavior of the particles in our method, similarly to how the bees in a hive work together to achieve a common goal. Reinforcement learning is a branch of artificial intelligence where an agent learns an optimal policy by interacting with the environment for sequential decision-making problems [25]. The approach proposed in this paper differs from the previous approaches in the following ways. Firstly, unlike those prior studies that focus on optimizing the components of an existing optimization algorithm, our proposed approach directly discovers an optimization policy within a continuous action space. As opposed to some previous studies that learn the parameters within the framework of the existing swarm-based approaches, Deephyive learns to optimize with minimal assumptions about the functional form of the particle update policy. In other words, in the other papers, RL is applied within the framework of the existing optimization approaches (e.g., PSO, firefly, etc.) to enhance the performance as compared to the base optimization algorithm, while the proposed approach learns a policy, modeled using a deep neural network, from scratch. Secondly, in contrast to those existing studies where the generalization of the proposed machine learning methods is unclear [17], we demonstrate that Deephyive displays acceptable generalization characteristics to unseen functions and higher dimensions, as will be shown in Section 4. In many of the previous studies mentioned, the RL training is embedded in the optimization of the objective function, and, as a result, the learned parameters cannot be expected to necessarily generalize to new functions [19; 26]. Table 1 details the differences between Deephive and the other common swarm-based optimization algorithms. The remainder of the paper is laid out as follows: Section 2.1 provides a quick overview of reinforcement learning and its relationship to the discovery of swarm optimization policies, including a description of the proximal policy optimization (PPO) algorithm employed in this study. The proposed optimization technique is discussed in detail in Section 2.2. A brief explanation of the benchmark objective functions tested and the results obtained using Deephive are presented and analyzed in Sections 3 and 4, respectively. The paper ends with some concluding remarks in Section 5.