## 3 Research Infrastructure and Technology
\begin{table}
\begin{tabular}{c c c} \hline \hline \multirow{2}{*}{**Algorithm**} & **Discrete/ Continuous Action Set** & **Information** \\ \hline \multirow{2}{*}{A2C} & \multirow{2}{*}{YES/YES} & Advantage Actor Critic (A2C) is an algorithm [31] that is based on the actor-critic mechanism. The actor decides what actions to take, while the critic evaluates these actions and provides feedback. This algorithm uses the advantage function, which evaluates how well the actor’s decision compares to expectations. The goal is to maximize the advantage value. \\ \hline \multirow{2}{*}{PPO} & \multirow{2}{*}{YES/YES} & Proximal Policy Optimization (PPO) is an algorithm [32] that operates more securely and stably than other reinforcement learning methods. Limiting policy changes allows for more efficient and predictable updates to the agent’s strategy. PPO has been one of the most commonly used algorithms in research due to its flexibility and stability. \\ \hline \multirow{2}{*}{DQN} & \multirow{2}{*}{YES/NO} & Deep Q-Network (DQN) is based on the classical Q-learning algorithm [33] but extends its capabilities using neural networks. In this way, solving more complex problems, such as processing images from cameras used in drone simulation tests, is possible. This algorithm used the experience from previous episodes to improve its policy. \\ \hline \multirow{2}{*}{DDPG} & \multirow{2}{*}{NO/YES} & Deep Deterministic Policy Gradient DDPG) is an algorithm based on the actor-critic principle [34] but specializes in problems where actions are continuous, not discrete (as in the case of drone control). Unlike DQN, DDPG operates in a space of constant actions, which allows for more precise control \\ \hline \multirow{2}{*}{SAC} & \multirow{2}{*}{NO/YES} & Soft Actor Critic (SAC) is an advanced algorithm [35] that differs from others in that it maximizes not only the reward but also the entropy of the policy. That is, it aims to make the agent’s policy as unpredictable as possible, which supports the exploration of the environment. This allows the algorithm to avoid getting stuck in a local optimum and is more robust to problems resulting from the need for appropriate training data. \\ \hline \multirow{2}{*}{TD3} & \multirow{2}{*}{NO/YES} & Twin Delayed DDPG (TD3) is an improved version of DDPG [36] that minimizes some of the problems in the classic DDPG. A delay in policy updates is introduced, allowing for more stable results. Additionally, the algorithm uses two Q-function networks, reducing the risk of errors related to incorrect estimation. \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Algorithms and their operation in action sets. include a standard interface, for which it does not matter which algorithm is being tested. The libraries included in the OpenAI Gym environment package support algorithmic, 2D and 3D, and text tasks (Figure 3). OpenAI Gym allows you to use one environment to test many different solutions, compare them thoroughly, and find the optimal result. The environment created according to the OpenAI Gym interface rules is based on defining what happens when the agent (algorithm) performs subsequent actions. Each step has four values returned after its execution (Table 2). The PyBullet [38] library, written in Python v3.12.3, was used as a simulator. It allows you to run simulations for machines, robots, and games and visually verify the effects of their operation. The library allows you to load objects from various files, including the most popular URDF (Universal Robotic Description Format) [39]. This enables you to create your own structures based on two components--link and joints. The structure described in the appropriate XML format is loaded into the simulator and controlled using built-in physical engines. It also has a built-in solution such as a Quadroper or a humanoid four-legged robot. It is possible to use several simulated machines simultaneously, which allows you, for example, to create a flying swarm of drones and teach them to fly synchronously.