Simulating Multi-Agent Swarming Events in Python | by Juan Nathaniel | Towards Data Science

Swarming occurs frequently in nature: from a group of migratory birds to a collection of drones performing a particular mission. Thus, being able to simulate how swarming works will enable you to build your own multi-agent reinforcement learning algorithm that closely mimics real-world scenarios. To give you a glimpse of what we are building today, the GIF below will be the output of our swarming simulation model using an example of migratory birds. Without further ado, let’s begin!
Note: You can find the full code at the end of the post.
Why is simulation important in multi-agent Reinforcement Learning (RL) problems? It is primarily because simulation is one of the Exploratory Data Analysis (EDA) methods in RL that allows us to visualize the interrelatedness of different agents within a system. In this post, we are going to discuss swarming events because they are some of the many examples of multi-agent reinforcement learning, and I have compiled below how they act in real life!
Swarming of birdsDrone swarmsFish schoolAnd many others!
Swarming of birds
Drone swarms
Fish school
And many others!
We will be implementing the Viscek model (1995) as our multi-agent swarming baseline algorithm where many more advanced implementations have since been built on top.
Imagine you are trying to simulate the swarming of 300 birds (N=300), where each bird is indexed i=0...(N-1). A bird will only change its direction and velocity vectors when it interacts with another one within a circle of radius, R.
Let’s start by importing the required Python libraries and initializing the above constants.
import matplotlib.pyplot as pltimport numpy as npN = 300 # number of birdsR = 0.5 # radius of effect# Preparing the drawingL = 5   # size of drawing boxfig = plt.figure(figsize=(6,6), dpi=96)ax = plt.gca()
For each time step, n, the position of each bird along the x- and y-axes (r) is updated as follows:
And this is implemented in Python below.
Nt = 200 # number of time stepsdt = 0.1 # time step# bird positions (initial)x = np.random.rand(N,1)*Ly = np.random.rand(N,1)*L
v is the velocity vector with a constant speed, v0, and an angle Θi (read: theta). We will also introduce an angle perturbation (read: eta) to mimic the natural stochasticity of birds’ movement.
The velocity vector is initialized in Python below.
v0 = 0.5 # constant speedeta = 0.6 # random fluctuation in angle (in radians)theta = 2 * np.pi * np.random.rand(N,1)# bird velocities (initial)vx = v0 * np.cos(theta)vy = v0 * np.sin(theta)
Now we will come to the main for loop. For each timestep, the position, r, is updated as:
# Update positionx += vx*dty += vy*dt
And the v is updated by finding the (1) mean angle within R, (2) adding random perturbation to generate a new theta, and (3) updating the new velocity vector, v:
(1) Mean angle within R
mean_theta = theta# Mean angle within Rfor b in range(N):    neighbors = (x-x[b])**2+(y-y[b])**2 < R**2    sx = np.sum(np.cos(theta[neighbors]))    sy = np.sum(np.sin(theta[neighbors]))    mean_theta[b] = np.arctan2(sy, sx)
(2) Adding a random perturbation to the velocity angle. The random term is generated by sampling from a normal distribution within [-0.5, 0.5], multiplying it by a pre-defined eta, and adding this perturbation term with the calculated mean theta.
# Adding a random angle perturbationtheta = mean_theta + eta*(np.random.rand(N,1)-0.5)
(3) Updating the velocity vector with a new theta:
# Updating velocityvx = v0 * np.cos(theta)vy = v0 * np.sin(theta)
Overall, the updates for r and v will be done across dt * Nt time steps.
Finally, now that everything is in order, let’s finish up by plotting how the updates will look like!
plt.cla()plt.quiver(x,y,vx,vy,color='r')ax.set(xlim=(0, L), ylim=(0, L))ax.set_aspect('equal')plt.pause(0.001)
If everything works out, you will be greeted with this on your screen. Congratulations! You can experiment around and change some of the parameters to see how your simulation model varies.
Well done if you have reached the end. You can find the full code here:
Simulation is one of the Exploratory Data Analysis (EDA) steps in Reinforcement Learning (RL). Once you have a clearer idea of how your real-world scenario behaves mathematically, you will be able to design more robust RL algorithms. If you enjoy this post, I will make many more real-world simulations as part of my multi-agent RL EDA series. So stay tuned!
Do subscribe to my Email newsletter: https://tinyurl.com/2npw2fnz where I regularly summarize AI research papers in plain English and beautiful visualization.
Lastly, you can find more resources surrounding the topic of RL and simulation here: