<figure><img src="image_1.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# ScaRLib: A Framework for Cooperative Many Agent Deep Reinforcement learning in Scala*

Davide Domini¹[0009-0006-8337-8990], Filippo Cavallari¹[0009-0006-4050-9989], Gianluca Aguzzi¹[0000-0002-1553-4561], and Mirko Viroli¹[0000-0003-2702-5702]

Alma Mater Studiorum – Università di Bologna

{filippo.cavallari2}@studio.unibo.it

{davide.domini2,gianluca.aguzzi,mirko.viroli}@unibo.it

**Abstract.** Multi Agent Reinforcement Learning (MARL) is an emerging field in machine learning where multiple agents learn, simultaneously and in a shared environment, how to optimise a global or local reward signal. MARL has gained significant interest in recent years due to its successful applications in various domains, such as robotics, IoT, and traffic control. Cooperative Many Agent Reinforcement Learning (CMARL) is a relevant subclass of MARL, where thousands of agents work together to achieve a common coordination goal.

In this paper, we introduce ScaRLib, a Scala framework relying on state-of-the-art deep learning libraries to support the development of CMARL systems. The framework supports the specification of centralised training and decentralised execution, and it is designed to be easily extensible, allowing to add new algorithms, new types of environments, and new coordination toolchains.

This paper describes the main structure and features of ScaRLib and includes basic demonstrations that showcase binding with one such toolchain: ScaFi programming framework and Alchemist simulator can be exploited to enable learning of field-based coordination policies for large-scale systems.

**Keywords:** Many Agent Reinforcement Learning · Deep Learning · Aggregate Computing.

## 1 Introduction

Recent advances in machine learning have led to the development of Multi Agent Reinforcement Learning (MARL) [9], in which multiple agents learn simultaneously within a shared environment to optimise either a global or local reward signal. This area of research has gained significant interest in recent

<footnote>* Supported by Department of Computer Science and Engineering @ Alma Mater Studiorum - University of Bologna.</footnote>
years due to its successful applications in various domains, such as robotics [21],
IoT [19], and traffic control [13]. MARL provides a powerful approach to tack-
ling complex problems that cannot be easily solved by single-agent reinforce-
ment learning. Most specifically, Cooperative Many Agent Reinforcement Learn-
ing (CMARL) [33,16] is a relevant subclass of MARL, where thousands of agents
work together to achieve a common coordination goal. CMARL finds several ap-
plications in contexts like swarm robotics [17], wireless sensor networks, and
smart grid management. CMARL offers significant opportunities in the design
of large-scale systems requiring agents to coordinate and collaborate effectively,
even in environments with partial observability and intrinsic unpredictability.
Therefore, there is a need for effective frameworks (and tools) that can foster
CMARL adoption.

However, although several frameworks exist both for describing and solving CMARL problems (Ray [24]) and for using and defining multi-agent environments (PettingZoo [31]) they generally lack the following aspects: (i) setting up complex environments (and hence simulation scenarios) is particularly difficult, and (ii) they are typically tailored for handling a limited number of agents, and for non-collaborative tasks. To start addressing these problems, in this work we present ScaRLib, a framework for the design of effective CMARL systems. ScaRLib offers several key features, including support for centralised training and decentralised execution, easy extensibility, and a Domain Specific Language (DSL) for expressing complex cooperative scenarios. Additionally, ScaRLib integrates with the Alchemist[25] simulator for large-scale pervasive computing systems and provides the ability to express field-based coordination problems through its integration with ScaFi[11]. The latter, in particular, provides a high-level language for distributed computing that provides a declarative and compositional ways of expressing complex coordination tasks.

The remainder of this paper is organised as follows: Section 2 provides key background in the context of learning and the Alchemist/Scafi toolchain, Section 3 presents the ScaRLib tool, Section 4 presents demos of ScaRLib, Section 5 discuss some state-of-the-art solutions, and finally Section 6 concludes and presents future works.

2 Background

## 2.1 Reinforcement Learning

Reinforcement Learning (RL) is a subfield of machine learning, other than su-
pervised and unsupervised learning, that focuses on solving sequential decision
problems: giving an *agent* that can interact with an *environment*, the RL goal is
to *learn* a policy (i.e., the action to take in a certain *state* of the environment)
that maximises a *reward* signal. Most specifically, the dynamics of these agent-
environment interactions are modelled in discrete steps (i.e., *t* = 0, 1, 2, ...).
At each step, the agent receives an *observation* *S*<sub>*t*</sub> ∈ *S* (*S* is the set of all the
possible states) from the environment and takes an action *A*<sub>*t*</sub> ∈ *A*(*S*<sub>*t*</sub>) (*A*(*S*<sub>*t*</sub>)
is the action space from the state *S*<sub>*t*</sub>). One time step later, the agent receives
<figure><img src="image_2.png" /><figcaption>Fig. 1: Q-Learning and Deep Q-Learning visual comparison</figcaption></figure>

a reward $R_{t+1} \in \mathbb{R}$ and the new state $S_{t+1}$. The agent decisions are based on a probabilistic policy $\pi_t(a|s)$, which indicates the probability of choosing action $a$ from the state $s$ at time $t$. In order for RL to be effective, two conditions are fundamental: (i) all is meant by goals/purposes/success can be well thought of as the maximization of the expected value of the cumulative sum of the reward (i.e., a scalar signal) – called the reward hypothesis; and (ii) the environment state should summarise the past compactly so that future states only depend on the current state (and not on past states) – the markov property.

In literature, several algorithms can be used to solve RL problems. In this work, we will focus on two of them: the *Q-Learning* [32] algorithm and the *Deep Q-Learning* [22] algorithm. The core part of both of these algorithms is the *Q-function*, which maps each state-action pair to a value that represents the expected future reward of taking that action in that state. Using a modified Bellman equation update, the *Q-function* is iteratively updated based on the rewards received by the agent as it takes actions in the environment. From the *Q-function*, the agent can follow both an *exploration policy* during the training phase and a *behavioural policy* once the learning phase is complete. The *behavioural policy* is a greedy policy, in which the agent chooses the action with the highest *Q-value* in a given state. This policy ensures that the agent always chooses the best action to maximise its expected future reward. On the other hand, during the exploration phase, the agent uses an $\epsilon$-greedy policy, where it chooses a random action with probability $\epsilon$ and the best action (according to the *Q-table*) with probability $1-\epsilon$. This allows the agent to explore the environment and learn from new experiences.

The main difference between the two algorithms is that in Q-Learning the *Q-table* is represented as a table, while in Deep Q-Learning it is approximated by a neural network. Therefore, the first approach works well for simple problems, but it struggles to scale to complex problems due to the explosion of the state space; the second approach, on the other hand, addresses large-scales but requires much data to train the neural network.

## 2.2 Multi Agent Reinforcement Learning

MARL is an extension of RL where multiple agents interact with one another and with the environment. Usually, MARL is modelled as a *Markov Game* (or *Stochastic Game*) [20] in which we have:
- A tuple $S = \langle N, S, \{A^i\}, P, \{R^i\} \rangle$ with $i \in 1...N$

- The number of agents $N > 1$

- The action space of the i-th agent $A^i$. The global action space is defined as $A = A^1 \times A^2 \times \dots \times A^N$

- A function describing the transition dynamics $P: S \times A \to \mathcal{P}(S)$

- The reward function $R^i: S \times A \times S \to \mathbb{R}$ for each agent $i$

Based on the reward function used by the agents, MARL can be divided into two categories: i) *cooperative*, where all the agents trying to maximise the same reward function (e.g., a group of robots trying to clean a room); ii) *competitive*, where, potentially, each agent has its own reward function that is conflictual with the other (e.g., a rock-paper-scissor game). Cooperative MARL can be further divided into two additional categories (based on the policy), namely: i) *homogeneous*, where all the agents have the same capabilities, i.e., they use the same policy ii) *heterogeneous*, where each agent may have its own policy

In this work, we focus on a subset of MARL, namely: *Many Agent Reinforcement Learning* [33]. The only difference between the two approaches is in the number of agents involved. Typically, in Many Agent Reinforcement Learning the number of agents may range from a hundred to one or two thousand whereas, in Multi Agent Reinforcement Learning, there are only a few tens [28,6]. Moreover, we focus on cooperative homogeneous and heterogeneous learning.

## 2.3 Alchemist

Alchemist<fnref n="1" /> [25] is meta-simulator mainly designed for simulating complex distributed systems in a rich variety of scenarios like swarm robotics [12], large-scale sensor networks [2], crowd simulation [7], path planning, and even morphogenesis of multi-cellular systems. The simulator is *meta* in nature, as it is based on general abstractions that can be mapped to specific use cases (i.e., *incarnations*). Inspired by biochemistry, the meta-model consists of a set of *nodes* that exist in an *environment* and are linked together by *relationship* rules. Each node contains a sequence of *molecules* and *reactions*. A *molecule* represents a variable, which acts as a container for data. *Reactions* instead are events that occur based on a set of *conditions*, and are fired according to a time distribution, producing an effect that is described as an action. This abstraction allows the simulator to be flexible and adaptable to a variety of use cases and node numbers (it could support thousands of nodes), while maintaining a consistent underlying structure.

The Alchemist simulator features four incarnations: biochemistry, Sapere, Protelis, and ScaFi, each with a different way of modelling molecules and actions. Moreover, the simulator offers an effortless method for loading simulations. The process requires a YAML file that includes essential parameters, such as the incarnation type, neighbour connection model, and node deployment. In Figure 2, we have provided an example YAML file that creates a simulation using

<footnote n="1">¹ http://alchemistsimulator.github.io/</footnote>
the ScaFi incarnation (first row). It also defines the neighbourhood relationship based on fixed distances (0.5 in this case), placing nodes in a fixed grid of size 10x10 starting at -(5,5) and ending at (5,5), with a node-to-node distance of 0.25. Finally, it loads the ScaFi program called “program”, which is evaluated at each node with a frequency of 1Hz.

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 2: An Alchemist simulation example. The simulation result on the right is obtained by running the simulation described on the left.</figcaption></figure>

## 2.4 ScaFi

ScaFi (Scala Fields <fnref n="2" />) [11] is a scala framework for developing large-scale distributed applications and simulating systems of networked agents. ScaFi is designed for Aggregate Programming [7], which is a top-down global-to-local macro-programming [10] paradigm for distributed systems where the focus is on the collective behaviour of the system rather than on the individual agents that make up the system. One of the unique features of aggregate computing is its support for self-organising systems – that are systems that can adapt to changes in the environment and maintain their functionality without requiring centralised control.

In ScaFi, agents are represented as nodes in a logical network, and the interactions between agents are modelled as the exchange of messages over edges in the graph. ScaFi provides a set of primitives for expressing distributed algorithms, which can all be interpreted as field-based coordination policies, namely,

<footnote n="2">² https://scafi.github.io/</footnote>
<figure><img src="image_5.png" /><figcaption>Fig. 3: ScaRLib main modules</figcaption></figure>

distributed computations of maps from nodes to values (i.e. *fields*). These primitives are designed to be composable and reusable, allowing programmers to build complex distributed algorithms from simple building blocks.

## 3 ScaRLib

ScaRLib <fnref n="3" /> <fnref n="4" /> is a research Scala framework designed to support the development of CMARL systems by JVM-based high-level specification, and with learning performed under the hood by PyTorch<fnref n="5" />. This project aims to provide a tool that allows easy and powerful system specification. To meet this purpose we have designed many abstractions, that model high-level aspects of the CMARL domain, without caring about low-level implementation details. Basically, ScaR-Lib is composed of three main modules (Figure 3), namely: i) **scarlib-core** that implements the main abstractions over the CMARL domain, ii) **dsl-core** that provides a high-level language to specify the system, and iii) **alchemist-scafi** that provides bindings between ScaRLib and the two tools Alchemist and ScaFi. It is important to note that ScaRLib is not limited to the Alchemist-Scafi combination, since it is possible to implement other bindings to other tools (e.g., by replacing Alchemist with some other simulator, for example, FLAME GPU [27]).

### 3.1 Core abstraction

The module **scarlib-core** implements the core functionalities and abstractions of the framework, such as the definition of the main data structures and the implementation of the main algorithms. All the abstractions (Figure 4) are built around a bunch of concepts. The key element is the **System**, which is a collection of agents that interact within a shared environment and that are trained to optimise a reward signal expressed by a reward function. The tool comes with two types of systems already implemented that are very common in literature [14],

<footnote n="3">³ Tool available on GitHub at https://github.com/ScaRLib-group/ScaRLib</footnote>

<footnote n="4">⁴ demo video at: https://github.com/ScaRLib-group/ScaRLib-demo-video</footnote>

<footnote n="5">⁵ https://pytorch.org/</footnote>
<figure><img src="image_6.png" /><figcaption>Fig. 4: ScaRLib core architecture</figcaption></figure>

i.e., centralised training and decentralised execution (*CTDESystem*) and decentralised training and execution (*DTDESystem*). Furthermore, an implementation of the DQN algorithm [23] is provided and used to train agents. The end-user who wants to run a learning process only has to implement four elements to define his own system with the desired collective goal, which are: i) the environment, ii) the agent state space, iii) the action space and, iv) the reward function. Only by using this module, it is possible to run a simple learning process in a simulated environment based on our platform.

To better understand the dynamics of the system it is useful to explain some of the internals. Both systems utilise a training process that consists of multiple *epochs*, with each epoch comprising a set of *episodes*. During each episode, the agents receive the current state as input and execute an action based on that state. This collective action causes the environment to move to the next state, advancing the simulation to the next episode. At the end of an epoch, the environment is reset and the agents are trained using the collected experience. Most specifically, if the chosen system is a *CTDESystem* (Figure 5a) the agents are trained in a centralised way, for that reason, there is a single central dataset, where the global experience of all the agents is stored, and a single central learner that is responsible for the training process and for the improvement of the policy neural network. The system is also responsible for the execution of all the agents and the notification of the updated policy. In this way, it is possible to easily extend the system in order to modify the execution flow, e.g., if a concurrent and distributed execution is needed. The *DTDESystem* (Figure 5b) works similarly, the only difference is that every agent has its own dataset and learner.

Regarding the training process, since the tool aims to support neural-network-based RL algorithms (like DQN), we chose to use the current de facto standard
<figure><img src="image_7.png" /><figcaption>Fig. 5: Examples of developed System dynamics. On the left, there is the cen-
tralised system, where a learner with a global view of the system updates the
policy shared with all agents. On the right, there is a decentralised system, where
each agent has a local policy and a local policy.</figcaption></figure>

framework for building neural networks, which is PyTorch—alternatives include
DL4J <fnref n="6" />, which could be subject of future investigation.

One way to integrate this library into a JVM environment could be to rely
on its native core (LibTorch) using JNI – as was done in scala_torch <fnref n="7" /> project.
In ScaRLib, we chose a convenient approach that allowed us not only to access
PyTorch but also all the libraries connected to it (e.g., torch geometric [15], etc.),
which is to use ScalaPy [18] to interact directly with the Python API of these
libraries. This integration generally involves: i) setting up a Python environment
in which the libraries of interest are instantiated, and ii) creating a Scala API
that isolates what is necessary to access the Python ecosystem. In this case,
we have isolated everything in DQN, which is, therefore, the entry point for
accessing Torch.

<footnote n="6">⁶ https://deeplearning4j.konduit.ai/</footnote>

<footnote n="7">⁷ https://github.com/microsoft/scala_torch</footnote>
<figure><img src="image_8.png" /><figcaption>Fig. 6: ScaRLib `alchemist-scafi` architecture. A `ScafiProgram` should be passed to the `AlchemistEnvironment` in order to start the learning process.</figcaption></figure>

## 3.2 ScaFi-Alchemist integration:

In addition to the core, we have implemented another module called `alchemist-scafi` (Figure 6) in which there is integration with the two tools: Alchemist [25] and ScaFi [11]. Such integration enables the possibility to run the learning process in an aggregate computing context. This is a key part of this contribution. In fact, although Alchemist has been used for CMARL with ScaFi in the past [5,3], ad-hoc solutions were always created that were difficult to *reuse*, *rigid*, *untested*, and had *interoperability issues* between Alchemist and the chosen native libraries. With this integration, we want to provide a usable system once and for all, to bring the Many-agent RL community closer to the use of this simulator and this paradigm, which has proven flexible in describing the most diverse environments – from robotic swarms [12] to data centres.

The specification of a learning system does not change, only two new elements are added: the specification of the Alchemist simulation and the implementation of the ScaFi-based logic. In particular, the Alchemist simulation will be defined as shown in Figure 2, by passing a ScaFi class as a program, which contains aggregate programming code. In order to advance the training process, a molecule with the current action to be taken (a subtype of `Action` class) will be present in the ScaFi program. This will be injected by a learner that contains the RL policy. Moreover, the aggregate program will evaluate the environment state (which must be a subtype of `State`) using `computeState` and insert it into the `state` molecule, that will be used by the learner to update the policy.

## 3.3 DSL for learning configurations

The latest module developed is an internal DSL that allows for agile and flexible creation of CMARL training systems. We made this choice to bridge the gap between the MARL system designer's idea and the actual training system. Additionally, by using a system like Scala, creating a typed DSL allows for capturing errors during compilation, rather than waiting for the actual system runs to intercept simple configuration errors.

The exposed DSL is a simple facade to the abstractions shown in the `scarlib-core` module. Therefore, if a developer wants to start their simulation they must
first define a reward function that indicates how good is the current state of a certain agent is compared to the current environmental condition.

```groovy
class MyRewardFunction extends CollectiveRewardFunction:
    override def computeReward(
        state: State, action: Action, nextState: State
    ): Double = ...
```

Consequently, they must decide which actions are supported by the agents living in the chosen system. Since we are talking about CMARL systems, we suppose that each agent has the same action space. Thus, it is possible to define a set of actions as a product type:

```groovy
sealed trait MyAction extends Action
object MyAction:
    case object A extends MyAction
    case object B extends MyAction
    case object C extends MyAction
    def all: Seq[MyAction] = Seq(A, B, C)
```

Final refinements required include: i) choosing the class of the Alchemist environment to instantiate, ii) defining the number of agents living in the chosen environment, and iii) defining the size of the buffer in which the memory will be stored, expressed as follows:

```groovy
val system = learningSystem {
    rewardFunction { new MyRewardFunction() }
    actions { MyAction.all}
    dataset { ReplayBuffer[State, Action](10000) }
    agents { 50 } // select the number of agent
    environment {
        // select a specific environment
        "it.unibo.scarlib.experiments.myEnvironment"
    }
}
```

## 3.4 Tool usage

The tool is published on Maven Central and it is possible to include it in your project, for example, through a build system. In the case of Gradle, for instance, you will need to add the following instructions:

```groovy
implementation("io.github.davidedomini:scarlib-core:1.5.0")
implementation("io.github.davidedomini:dsl-core:1.5.0")
```

At this point, it will be possible to create your own training system as shown in the DSL section. To start the training, you will then need to write:

```groovy
learningSystem.train(episodes = 1000, episodeLength = 100)
```
Of course, the system can also be used to verify a certain policy that has been learned during a training process. To do this, first, you will need to load the neural network extracted during training:

val network = PolicyNN(path, inputSize = ..., hiddenSize = ...)

Then you can execute the test in the following way:

system.runTest(episodeLength = 100, network)

For further details on how to specify simulations and environments, please refer to the repository README, the presentation video and the developed simulation (following section).

# 4 Experiments

## 4.1 Scenario description

To test ScarLib's functionality, we develop an experiment<fnref n="8" /> involving a relatively large number of agents and non-trivial coordination tasks. We aim to create a flock of drones that moves to avoid collisions with each others, by learning a policy by which each agents decide how to move based on neighbors relative position. This is a well-known problem, and various models and algorithms exist which we draw upon [26,36]. In this case, we assumed that agents position themselves in an unlimited 2D environment with a fixed neighborhood (the closest five, in our experiments, though this is a simulation parameter) and have the ability to perform movement steps in the 8 directions of a square grid (horizontally, vertically, or diagonally). The environment state, as perceived by the single agent, is the relative distance to the closest neighbors. Particularly, it was expressed through ScaFi as:

val state = foldhoodPlusSeq.empty)(_ ++ _)(Set(nbrVector))

where `nbrVector` is the vector representing the relative position of the neighbor. `foldhoodPlus` is a ScaFi function that allows to fold over the neighborhood and `++` is the concatenation operator for sequences.

The crucial point for this task is the definition of the reward function. In this simulation, we based it on `collision` and `cohesion` factors. We aim to learn a policy by which agents, initially spread in a very sparse way in the environment, move toward each other until reaching approximate $\delta$ distance without colliding, ultimately forming one or many close groups.

The collision factor comes into play when the distance is less than $\delta$, and exponentially weighs the distance $d$ relative to its closest neighbour:

$$ collision = \begin{cases} 0 & \text{if } d > \delta \\ \exp(-\frac{d}{\delta}) & \text{otherwise} \end{cases} \qquad (1) $$

<footnote n="8">⁸ repository available at https://github.com/ScaRLib-group/ScaRLib-flock-demo</footnote>
<figure><img src="image_9.png" /><figcaption>Fig. 7: Cohesion-Collision reward function: the red vertical line represents the target distance *d*. The portion of the graph to the right of the red line represents the influence of the cohesion term, while the left one represents the influence of the collision term.</figcaption></figure>

In this way, when the negative factor is taken into account: the system will tend to move nodes away from each other.

However, if only this factor were used, the system would be disorganised. This is where the cohesion factor comes in. Given the neighbor with the maximum distance *D*, it linearly adjusts the distance relative to the node being evaluated by function:

$$ \text{cohesion} = \begin{cases} 0 & \text{if } d < \delta \\ -(D - \delta) & \text{otherwise} \end{cases} \qquad (2) $$

The overall reward function is defined as the sum of these two factors (cohesion + collision) as shown in Figure 7.

## 4.2 Results

To verify the functionality of the described simulation, we divided the evaluation into two parts. In the first part, we trained the system for a total of 1000 epochs, each consisting of 100 episodes (or steps). For each epoch, we randomly place 50 agents in a grid large 50x50 meters. We set the target distance *δ* at 2 meters.

Given the flexibility of ScaRLib, we tested the training with both CDTE and DTDE processes to ensure that the system could produce policies capable of solving the described task in both cases. With the homogeneous policies found (i.e., the one extracted from the CTDE process), we verified that the system's behavior was consistent with what was learned by varying the initial seed in 16 simulations. With the CDTE policy, since we considered the system homogeneous, we also verified the behavior as the number of nodes varied, expecting similar performance as the nodes increased.

The graphs shown in Figure 8 demonstrate the multi-objective nature of the problem. In fact, cohesion and collision are two contrasting signals, and the system had to find a balance between these two values. The graphs show that DQN can generally optimise one signal at a time, with cohesion tending towards
<figure><img src="image_10.png" /><figcaption>Fig. 8: Cohesion and collision experiment results. The y-axis represents the reward value. The x-axis represents the total number of episodes. The first three graphs show the results of the CDTE learning process, while the last three show the results of the DTDE learning process.</figcaption></figure>

zero and collision increasing. Nonetheless, after 500 epochs in CTDE simulation, we see that the system had already found a balance between these two factors. In the case of DTDE learning, we observe that convergence is achieved in fewer steps (50). This is because there is a greater number of policies and therefore greater overall complexity compared to a single homogeneous policy.

During the testing phase (Figure 9 shows a series of snapshots of the learned policy), we observed that the system is capable of maintaining a distance of approximately $\delta$, both in the CDTE and DTDE cases. Most specifically, we note that the homogeneous policy is generally a winning choice for homogeneous CMARL tasks. Increasing the number of agents (from 50 to 200), we can observe that collective performances are similar to those with few agents (Figure 10).

# 5 Related work

MARL has gained significant interest in the past decade, leading to the development of several frameworks for use in both research and industry communities. Here, we highlight current state-of-the-art solutions for MARL problems and compare them to the tools presented in this work.

**Many Agent simulators:** unlike supervised learning, where a large dataset is required to improve neural network performance, in RL, algorithms require a simulator to gain experience. One such comprehensive solution for MARL is PettingZoo [31], which provides both competitive and cooperative settings for simulations with multiple agents. Another option for many-agent scenarios
<figure><img src="image_11.png" /><figcaption>Fig. 9: Snapshots of the learned policy, the time flow is from left to right. In the first row, there are 50 agents, whereas in the second row, there are 200 agents. In the last step of the simulation, the agents converged to a distance of approximately $\delta$.</figcaption></figure>

is NeuralMMO [30], a GPU-optimised simulator for MMO-like games that is designed to handle large-scale simulations of thousands of agents. Vectorized Multi-agent Simulator [8] is another promising solution, as it is optimised for collective tasks through GPU computation, and it can be extended with additional environments. While ScarLib is not directly linked to any simulator, its main abstraction can be potentially linked to both JVM-based simulators and gym-based Python environments. Our choice of Alchemist was mainly due to its ability to express CMARL settings easily, but potentially it can be used with any of the above-described solutions.

**Multi-Agent Deep RL libraries:** since the importance of multi-agent settings several libraries have been developed in recent years. Ray [24] is one of the most comprehensive frameworks, originally designed for single-agent RL but now integrated with basic concepts for MARL solutions thanks to MALib. It offers various MARL algorithms, supports different gym-like environments, and is highly customizable through configuration files. PyMARL [29] is one of the first solutions in Python for MARL, though it is limited to specific algorithms (like VDN and QMIX), and it is not generalizable. ScaRLib is more similar to the first framework, even though it is primarily designed for cooperative applica-
<figure><img src="image_12.png" /><figcaption>Fig. 10: The performance of the learned policy. The y-axis represents the distance between the agents. The x-axis represents the time. The green line is equal to $\delta$. In the charts, as the number of agents varies, the performance of the learned policy is similar. Moreover, the minimum (blue line) distance between the agents is always greater than $\delta$. The average distance (orange line) stays close to $2 * \delta$ (after convergence).</figcaption></figure>

However, since it was developed specifically for CMARL, it includes some abstractions and configurations that are not present in Ray, such as the concept of a collective reward function and the configuration for CTDE. This reduces the time required to use ScarLib compared to Ray. Additionally, ScarLib has a simple DSL that is easier to use than Ray's configuration system and is aided by the type system.

Finally, some innovative approaches aim to scale solutions to large populations of cooperative agents, such as mean-field RL. However, only a few implementations currently exist, and they are not considered to be general-purpose. ScarLib, on the other hand, offers a practical, simple implementation that can be leveraged in CMARL settings.

Overall, CMARL is a high-level framework that reduces the effort required for developers and practitioners to define and implement CMARL problems when compared to current state-of-the-art solutions.
**Many-Agent Proof of Concept:** the above RL library solutions are mainly created for multi-agent systems, so they generally do not scale well with large populations of agents. Novel approaches aim to scale the solution to potentially infinite populations of cooperative agents. In particular, mean-field RL [34] is probably one of the most interesting solutions in this context, as it abstracts over the entire agent population by considering only the average response of the neighborhood. Currently, however, only a few implementations exist <fnref n="9" /> and they are not considered to be general-purpose. ScaRLib instead is practical and already provides a simple implementation that can be leveraged in CMARL settings.

# 6 Conclusion and Future work

In this paper, we presented ScaRLib: a collaborative many-agent deep reinforcement learning framework that integrates the functionalities of ScaFi and Alchemist. The framework enables the definition of simulations of large-scale distributed scenarios and the creation of complex scenarios with ease through its exposed DSL. With ScaRLib, developers can effectively and efficiently simulate and experiment with different reinforcement learning algorithms, thereby providing a valuable tool for the advancement of coordination and multi-agent systems research. Looking forward, ScaRLib presents a promising solution for expressing collaborative multi-agent reinforcement learning applications – like hybrid aggregate computing solutions [1,5,4,3]. However, further development and integration is necessary for it to be more easily adopted by non-expert users. One area of improvement would be to integrate additional learning algorithms such as MAPPO [35] and mean field reinforcement learning [34], as DQN is only a baseline approach. Another aspect to be considered is the potential bottleneck that Alchemist may create during the learning phase since it is a JVM-based systems. To address this, we propose integrating a new environment, FLAME GPU [27], which has the ability to run entirely on GPU, thus providing faster learning and reducing the computational load. This integration would further enhance the capabilities of ScaRLib and make it a more practical and efficient tool for CMARL research.

# Data Availability Statement

The artifact is available in the Zenodo repository:

doi:10.5281/zenodo.7831045

# References

1. Aguzzi, G.: Research directions for aggregate computing with machine learning. In: 2021 IEEE International Conference on Autonomic Computing and Self-

<footnote n="9">⁹ https://github.com/mlii/mfrl</footnote>
Organizing Systems Companion (ACSOS-C). IEEE (Sep 2021). https://doi.org/10.1109/acsos-c52956.2021.00078, https://doi.org/10.1109/acsos-c52956.2021.00078

2. Aguzzi, G., Casadei, R., Pianini, D., Viroli, M.: Dynamic decentralization domains for the internet of things. IEEE Internet Computing **26**(6), 16–23 (nov 2022). https://doi.org/10.1109/mic.2022.3216753

3. Aguzzi, G., Casadei, R., Viroli, M.: Addressing collective computations efficiency: Towards a platform-level reinforcement learning approach. In: Casadei, R., Nitto, E.D., Gerostathopoulos, I., Pianini, D., Dusparic, I., Wood, T., Nelson, P.R., Pournaras, E., Bencomo, N., Götz, S., Krupitzer, C., Raibulet, C. (eds.) IEEE International Conference on Autonomic Computing and Self-Organizing Systems, ACSOS 2022, Virtual, CA, USA, September 19-23, 2022. pp. 11–20. IEEE (2022). https://doi.org/10.1109/ACSOS55765.2022.00019, https://doi.org/10.1109/ACSOS55765.2022.00019

4. Aguzzi, G., Casadei, R., Viroli, M.: Machine learning for aggregate computing: a research roadmap. In: 2022 IEEE 42nd International Conference on Distributed Computing Systems Workshops (ICDCSW). IEEE (Jul 2022). https://doi.org/10.1109/icdcsw56584.2022.00032, https://doi.org/10.1109/icdcsw56584.2022.00032

5. Aguzzi, G., Casadei, R., Viroli, M.: Towards reinforcement learning-based aggregate computing. In: Lecture Notes in Computer Science, pp. 72–91. Springer International Publishing (2022). https://doi.org/10.1007/978-3-031-08143-9_5, https://doi.org/10.1007/978-3-031-08143-9_5

6. Baker, B., Kanitscheider, I., Markov, T., Wu, Y., Powell, G., McGrew, B., Mordatch, I.: Emergent tool use from multi-agent autocurricula (2019). https://doi.org/10.48550/ARXIV.1909.07528, https://arxiv.org/abs/1909.07528

7. Beal, J., Pianini, D., Viroli, M.: Aggregate programming for the internet of things. Computer **48**(9), 22–30 (Sep 2015). https://doi.org/10.1109/mc.2015.261, https://doi.org/10.1109/mc.2015.261

8. Bettini, M., Kortvelesy, R., Blumenkamp, J., Prorok, A.: Vmas: A vectorized multi-agent simulator for collective robot learning. The 16th International Symposium on Distributed Autonomous Robotic Systems (2022)

9. Busoniu, L., Babuska, R., Schutter, B.D.: A comprehensive survey of multiagent reinforcement learning. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) **38**(2), 156–172 (Mar 2008). https://doi.org/10.1109/tsmcc.2007.913919, https://doi.org/10.1109/tsmcc.2007.913919

10. Casadei, R.: Macroprogramming: Concepts, state of the art, and opportunities of macroscopic behaviour modelling. ACM Computing Surveys (Jan 2023). https://doi.org/10.1145/3579353, https://doi.org/10.1145/3579353

11. Casadei, R., Viroli, M., Aguzzi, G., Pianini, D.: ScaFi: A scala DSL and toolkit for aggregate programming. SoftwareX **20**, 101248 (Dec 2022). https://doi.org/10.1016/j.softx.2022.101248, https://doi.org/10.1016/j.softx.2022.101248

12. Casadei, R., Viroli, M., Audrito, G., Pianini, D., Damiani, F.: Engineering collective intelligence at the edge with aggregate processes. Engineering Applications of Artificial Intelligence **97**, 104081 (Jan 2021). https://doi.org/10.1016/j.engappai.2020.104081, https://doi.org/10.1016/j.engappai.2020.104081

13. Chu, T., Wang, J., Codecà, L., Li, Z.: Multi-agent deep reinforcement learning for large-scale traffic signal control (2019). https://doi.org/10.48550/ARXIV.1903.04527, https://arxiv.org/abs/1903.04527

14. Du, W., Ding, S.: A survey on multi-agent deep reinforcement learning: from the perspective of challenges and applications. Artificial Intelligence Review **54**(5), 3215–3238 (Nov 2020). https://doi.org/10.1007/s10462-020-09938-y, https://doi.org/10.1007/s10462-020-09938-y
15. Fey, M., Lenssen, J.E.: Fast graph representation learning with pytorch geometric (2019)

16. He, K., Doshi, P., Banerjee, B.: Many agent reinforcement learning under partial observability (2021). https://doi.org/10.48550/ARXIV.2106.09825, https://arxiv.org/abs/2106.09825

17. Hüttenrauch, M., Adrian, S., Neumann, G., et al.: Deep reinforcement learning for swarm systems. Journal of Machine Learning Research 20(54), 1–31 (2019)

18. Laddad, S., Sen, K.: ScalaPy: seamless python interoperability for cross-platform scala programs. In: Proceedings of the 11th ACM SIGPLAN International Symposium on Scala. ACM (Nov 2020). https://doi.org/10.1145/3426426.3428485, https://doi.org/10.1145/3426426.3428485

19. Lei, L., Tan, Y., Zheng, K., Liu, S., Zhang, K., Shen, X.: Deep reinforcement learning for autonomous internet of things: Model, applications and challenges. IEEE Communications Surveys Tutorials 22(3), 1722–1760 (2020). https://doi.org/10.1109/comst.2020.2988367, https://doi.org/10.1109/comst.2020.2988367

20. Littman, M.L.: Markov games as a framework for multi-agent reinforcement learning. In: Cohen, W.W., Hirsh, H. (eds.) Machine Learning Proceedings 1994, pp. 157–163. Morgan Kaufmann, San Francisco (CA) (1994). https://doi.org/10.1016/B978-1-55860-335-6.50027-1, https://www.sciencedirect.com/science/article/pii/B9781558603356500271

21. Long, P., Fanl, T., Liao, X., Liu, W., Zhang, H., Pan, J.: Towards optimally decentralized multi-robot collision avoidance via deep reinforcement learning. In: 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE (May 2018). https://doi.org/10.1109/icra.2018.8461113, https://doi.org/10.1109/icra.2018.8461113

22. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M.: Playing atari with deep reinforcement learning (2013). https://doi.org/10.48550/ARXIV.1312.5602, https://arxiv.org/abs/1312.5602

23. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A.A., Veness, J., Bellemare, M.G., Graves, A., Riedmiller, M., Fidjeland, A.K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., Hassabis, D.: Human-level control through deep reinforcement learning. Nature 518(7540), 529–533 (Feb 2015). https://doi.org/10.1038/nature14236, https://doi.org/10.1038/nature14236

24. Moritz, P., Nishihara, R., Wang, S., Tumanov, A., Liaw, R., Liang, E., Elibol, M., Yang, Z., Paul, W., Jordan, M.I., Stoica, I.: Ray: A distributed framework for emerging ai applications (2017). https://doi.org/10.48550/ARXIV.1712.05889, https://arxiv.org/abs/1712.05889

25. Pianini, D., Montagna, S., Viroli, M.: Chemical-oriented simulation of computational systems with ALCHEMIST. J. Simulation 7(3), 202–215 (2013). https://doi.org/10.1057/jos.2012.27, https://doi.org/10.1057/jos.2012.27

26. Reynolds, C.W.: Flocks, herds and schools: A distributed behavioral model. In: Stone, M.C. (ed.) Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1987, Anaheim, California, USA, July 27-31, 1987. pp. 25–34. ACM (1987). https://doi.org/10.1145/37401.37406, https://doi.org/10.1145/37401.37406

27. Richmond, P., Coakley, S., Romano, D.M.: A high performance agent based modelling framework on graphics card hardware with cuda. In: Proceedings of The 8th International Conference on Autonomous Agents and Multiagent Systems - Volume 2. p. 1125–1126. AAMAS ’09, International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC (2009)
28. Samvelyan, M., Rashid, T., de Witt, C.S., Farquhar, G., Nardelli, N., Rudner, T.G.J., Hung, C.M., Torr, P.H.S., Foerster, J., Whiteson, S.: The star-craft multi-agent challenge (2019). https://doi.org/10.48550/ARXIV.1902.04043, https://arxiv.org/abs/1902.04043

29. Samvelyan, M., Rashid, T., de Witt, C.S., Farquhar, G., Nardelli, N., Rudner, T.G.J., Hung, C.M., Torr, P.H.S., Foerster, J., Whiteson, S.: The StarCraft Multi-Agent Challenge. CoRR abs/1902.04043 (2019)

30. Suarez, J., Du, Y., Isola, P., Mordatch, I.: Neural mmo: A massively multiagent game environment for training and evaluating intelligent agents (2019). https://doi.org/10.48550/ARXIV.1903.00784, https://arxiv.org/abs/1903.00784

31. Terry, J., Black, B., Grammel, N., Jayakumar, M., Hari, A., Sullivan, R., Santos, L.S., Dieffendahl, C., Horsch, C., Perez-Vicente, R., Williams, N., Lokesh, Y., Ravi, P.: Pettingzoo: Gym for multi-agent reinforcement learning. In: Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems. vol. 34, pp. 15032–15043. Curran Associates, Inc. (2021), https://proceedings.neurips.cc/paper/2021/file/7ed2d3454c5eea71148b11d0c25104ff-Paper.pdf

32. Watkins, C.J.C.H., Dayan, P.: Q-learning. Machine Learning 8(3-4), 279–292 (May 1992). https://doi.org/10.1007/bf00992698, https://doi.org/10.1007/bf00992698

33. Yang, Y.: Many-agent Reinforcement Learning. Ph.D. thesis, UCL (University College London) (2021)

34. Yang, Y., Luo, R., Li, M., Zhou, M., Zhang, W., Wang, J.: Mean field multi-agent reinforcement learning (2018). https://doi.org/10.48550/ARXIV.1802.05438, https://arxiv.org/abs/1802.05438

35. Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A., Wu, Y.: The surprising effectiveness of ppo in cooperative, multi-agent games (2021). https://doi.org/10.48550/ARXIV.2103.01955, https://arxiv.org/abs/2103.01955

36. Šošić, A., KhudaBukhsh, W.R., Zoubir, A.M., Koepp, H.: Inverse reinforcement learning in swarm systems (2016). https://doi.org/10.48550/ARXIV.1602.05450, https://arxiv.org/abs/1602.05450