# Experimenting with Agent-Based Model Simulation Tools
## 5 ABM Tools Evaluation
### Performance Evaluation
The implementations of all models and the benchmark are freely available on the following GitHub repository: [[https://github.com/isislab-unisa/ABM_Comparison](https://github.com/isislab-unisa/ABM_Comparison)]([https://github.com/isislab-unisa/ABM_Comparison](https://github.com/isislab-unisa/ABM_Comparison)) (accessed on 15 December 2022). **Benchmark configurations. ** All experiments were performed on the same Ubuntu 22.04 LTS x86_64 machine with kernel version 5.15.0-48-generic and equipped with an Intel i7-8700T (12) @ 4.000 GHz CPU, an NVIDIA GeForce GTX 1050 Mobile GPU, and 16 GB RAM. The performance of each framework was tested with different models configurations, starting with a field of size \(100\times 100\), \(1000\) agents, and \(200\) steps, while maintaining an agent density of \(\cong 10\%\), calculated as \(\frac{\text{width}\times\text{height}}{\text{number of agents}}\). We obtained the other configurations by doubling the number of agents and changing the field dimension to preserve the agent density. Table 8 lists all experiment configurations. **Results. ** Figure 1 depicts the running times of all the benchmarked ABM tools varying the computational load over the four models. The missing data in the graphs means that the specific tool failed in performing the simulation with the given configuration. Generally, most tools achieve the same performance stated in the referring articles. Nevertheless, we can note a few exceptions. Better-than-expected results come from ActressMAS, reaching execution times comparable with tools specifically designed for \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**\(\downarrow\) Tool/Model \(\rightarrow\)** & **Flockers** & **Schelling** &
\begin{tabular}{c} **Wolf, Sheep,** \\ **and Grass** \\ \end{tabular} & **ForestFire** \\ \hline \hline ActressMAS & \(\bigcirc\) & \(\bigcirc\) & \(\bigcirc\) & \(\bigcirc\) \\ AgentPy & \(\bigcirc\) & \(\bigcirc\) & \(\bigcirc\) & \(\bigcirc\) \\ Agents.jl & \(\bigcirc\) & \(\bigcirc\) & \(\bigcirc\) & \(\bigcirc\) \\ CppyABM & \(\bigcirc\) & \(\bigcirc\) & \(\bigcirc\) & \(\bigcirc\) \\ GAMA & \(\bigcirc\) & \(\bigcirc\) & \(\bigcirc\) & \(\bigcirc\) \\ krABMaga & \(\bigcirc\) & \(\bigcirc\) & \(\bigcirc\) & \(\bigcirc\) \\ MASON & \(\bigcirc\) & \(\bigcirc\) & \(\bigcirc\) & \(\bigcirc\) \\ Mesa & \(\bigcirc\) & \(\bigcirc\) & \(\bigcirc\) & \(\bigcirc\) \\ NetLogo & \(\bigcirc\) & \(\bigcirc\) & \(\bigcirc\) & \(\bigcirc\) \\ Repast & & & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Summary of whether each model was available for a given platform (\(\blacksquare\)) or has been developed from scratch (\(\bigcirc\)). We only considered the ABM tools that could be installed and used. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**\#** & **Agents** & **Field Size** & **\#** & **Agents** & **Field Size** \\ \hline
1 & 1000 & \(100\times 100\) & 5 & 16,000 & \(400\times 400\) \\
2 & 2000 & \(141\times 141\) & 6 & 32,000 & \(565\times 565\) \\
3 & 4000 & \(200\times 200\) & 7 & 64,000 & \(800\times 800\) \\
4 & 8000 & \(282\times 282\) & 8 & 128,000 & \(1131\times 1131\) \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Experiment configurations for evaluating ABM tools’ performance. obtaining high performance. This outcome probably derives from the efficiency of C# in managing the simple data structures required to manage the simulation. The platform only struggles with the WSG model, which includes more complex data structures and requires multi-agent management. Likewise, NetLogo performs better than expected in all models, providing good efficiency and scalability. Despite not being designed for simulating complex models, NetLogo grants medium to low execution times and handles intensive workloads. Conversely, CppyABM struggled with huge workloads, incurring execution failures or delivering high execution times. This behavior led to higher computational times than awaited in all models but Flockers.