## 2 Problem Specification
### Reinforcement Learning Architecture
As previously mentioned, the perception of the swarm at a given moment was directly related to the task to be carried out. It was assumed that the position of the object \(obj\) to be pushed and the position of the \(goal\) are known. The swarm knew the rotation \(fr_{rot}\) and the scaling \(fr_{sca}\) of the formation used in this behavior. It is important to highlight that \(obj\) could not be unequivocally located by the sensors of the robots in the formation, which could perceive only the distances of the robots from the environment but not the type of objects that generated this perception. \(S=(obj_{x},obj_{y},goal_{x},goal_{y},fr_{rot},fr_{sca})\)  
**Action Space**. The actions to be carried out by the swarm were defined by the formation of the swarm. In this case, the swarm could move the formation relative to its current position. This could be achieved by moving it at linear (\(v_{trns}\)) and rotational (\(v_{rot}\)) speed, as well as scaling it by (\(v_{sca}\)). Therefore, only three continuous actions allowed the entire swarm to coordinate, regardless of the number of individuals that formed it; \(A=(v_{trns},v_{rot},v_{sca})\). **Network Architecture**. Given an observation entry at time \(t\) that characterized the state of the swarm \(S_{t}\) and its output \(A_{t}\), we defined the network that developed the policy to generate \(A_{t}\) from \(S_{t}\). Thus, a three-layer network was designed (see Figure 3). There were six network inputs, corresponding to the distance and angle of the swarm to the box to be pushed, the distance and angle of the box with respect to the goal and the rotation and scale of the current formation. The hidden layers contained 64 neurons with tanh activation. The last layer was the one that connected directly with the actions of the swarm. These outputs defined the relative changes of the formation: its translation, rotation and scaling. Because we were working with a continuous policy, the output layer was linear, with one neuron for each output. It is worth mentioning that several tests were carried out with different architectures and activation functions. The network presented worked best for the training process and subsequent operation through the algorithm; in addition, it was very contained in size. This aspect allowed us to train it not only by gradient descent algorithms but also using evolutionary techniques. For illustrative purposes, Figure 4 shows the average reward obtained in several training sessions depending on the training algorithm, architecture and activation function. The foraging task defined in this article had several issues that made it difficult to learn. On the one hand, each execution of the swarm could be potentially different even under identical or similar conditions, given the noise of both the sensors and actuators. In addition, the swarm needed to learn to avoid pushing the object in a direction opposite to that of the goal and therefore often needed to surround the object to find the optimal push direction. Reward is a fundamental part of the design of the MDP, since it is the assessment that guides the achievement of the policy. Therefore, this reward must appropriately contain both positive and negative feedback to avoid the previous problems and enable effective and rapid learning. In this way, we designed a continuous reward that assessed the performance of the behavior at each step. In Equation (7), the used reward function is presented. There were basically three positive rewards: if the distance from the swarm to the box was improved, the reward was 1; once the swarm reached the box, it was rewarded with 5; as soon as it reached the goal, a reward of 100 was given. Any other case was valued with a negative reward of \(-1\) to prioritize behaviors that solved the task in the shortest possible way. It is worth mentioning that, to prevent the swarm from heading straight towards the box in the initial phases, a potential function \(U\) was introduced, which is discussed below.