Repository: projectmesa/mesa
Pull request #1955 (2024-01-11 08:40:42)

intro tutorial: Analysing model reporters
Some enhancements to the Mesa Introductory Tutorial:

1. **Visualization Library Update**: Updated the docs to mention Seaborn instead of matplotlib for data visualization.
2. **Model Efficiency Improvement**: Reduced the number of agents in batch runs, resulting in quicker model executions and generating more insightful data without compromising the educational value.
3. **New Agent Reporter**: Introduced a new agent reporter `steps_not_given`, which tracks the number of consecutive steps an agent hasn't transacted. This addition enriches the tutorial's analytical depth, demonstrating how to handle multiple reporters.
4. **Improved Tutorial Layout and Structure**: Enhanced the text layout and structure of the batch run section for better readability and understanding.
5. **General Steps for Analysis**: Added a section outlining general steps for analyzing model results, providing a structured approach for new users to follow and apply in their modeling endeavors.

It also fixes Readthedocs not having enough runtime to finish running the notebook, by focussing the batch_run on runs with fewer agents (which run faster).

Tested and renders correctly in Readthedocs. Please review and merge. Feel free to squash while merging.

codecov[bot]: ## [Codecov](https://app.codecov.io/gh/projectmesa/mesa/pull/1955?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=projectmesa) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Comparison is base [(`a15ce9f`)](https://app.codecov.io/gh/projectmesa/mesa/commit/a15ce9f0c384780037ec81ba76686595e9f1a10f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=projectmesa) 79.45% compared to head [(`2d7a6cd`)](https://app.codecov.io/gh/projectmesa/mesa/pull/1955?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=projectmesa) 79.45%.
> Report is 3 commits behind head on main.


<details><summary>Additional details and impacted files</summary>


```diff
@@           Coverage Diff           @@
##             main    #1955   +/-   ##
=======================================
  Coverage   79.45%   79.45%           
=======================================
  Files          15       15           
  Lines        1285     1285           
  Branches      285      285           
=======================================
  Hits         1021     1021           
  Misses        225      225           
  Partials       39       39           
```



</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/projectmesa/mesa/pull/1955?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=projectmesa).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=projectmesa).

rht: I set the PR to ready for review so that I can read the rendered tutorial on the RTD CI.
EwoutH: @rht There's a `KeyboardInterrupt` in the `batch_run`. Was already there on the current docs, and I can't reproduce it locally. Do you know what's happening there? Maybe something with the multithreading?
EwoutH: Maybe the runtime is just too long for Read the Docs 
Corvince: Also also: Can you fix the reference to the matplotlib installation in the beginning? The tutorial uses seaborn. And we have to install seaborn manually for the colab version. 
EwoutH: Okay fixed all the stuff (see commit messages), ready for another round of review!

Patch:
diff --git a/.github/workflows/build_lint.yml b/.github/workflows/build_lint.yml
index d4e1a9d7..df5916ec 100644
--- a/.github/workflows/build_lint.yml
+++ b/.github/workflows/build_lint.yml
@@ -77,3 +77,37 @@ jobs:
     - if: matrix.os == 'ubuntu'
       name: Codecov
       uses: codecov/codecov-action@v3
+
+  examples:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python
+      uses: actions/setup-python@v5
+      with:
+        python-version: "3.12"
+    - uses: actions/cache@v3
+      with:
+        path: ${{ env.pythonLocation }}
+        key: test-examples-pip-${{ env.pythonLocation }}-${{ hashFiles('setup.py') }}
+    - name: Install dependencies
+      # Only if the cache misses
+      # Based on https://github.com/pypa/pip/issues/8049#issuecomment-633845028
+      # read_requirements.py should be removed once
+      # https://github.com/pypa/pip/issues/11440 is resolved.
+      if: steps.cache.outputs.cache-hit != 'true'
+      run: |
+        pip install toml
+        python tests/read_requirements.py > requirements.txt
+        pip install -r requirements.txt
+    - name: Install Mesa
+      run: pip install --no-deps .
+    - name: Checkout mesa-examples
+      uses: actions/checkout@v4
+      with:
+        repository: projectmesa/mesa-examples
+        path: mesa-examples
+    - name: Test examples
+      run: |
+        cd mesa-examples
+        pytest test_examples.py
diff --git a/docs/tutorials/intro_tutorial.ipynb b/docs/tutorials/intro_tutorial.ipynb
index f130dbcc..69e9b8fb 100644
--- a/docs/tutorials/intro_tutorial.ipynb
+++ b/docs/tutorials/intro_tutorial.ipynb
@@ -74,10 +74,10 @@
     "pip install jupyter\n",
     "```\n",
     "\n",
-    "Install Visualization Tools:\n",
+    "Install [Seaborn](https://seaborn.pydata.org/) (which is used for data visualization):\n",
     "\n",
     "```bash\n",
-    "pip install matplotlib\n",
+    "pip install seaborn\n",
     "```\n",
     "\n",
     "\n",
@@ -144,14 +144,7 @@
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:41.694690Z",
-     "start_time": "2023-04-25T18:23:41.573145Z"
-    },
-    "collapsed": false,
-    "jupyter": {
-     "outputs_hidden": false
-    }
+    "collapsed": false
    },
    "outputs": [],
    "source": [
@@ -190,14 +183,7 @@
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:41.696198Z",
-     "start_time": "2023-04-25T18:23:41.693472Z"
-    },
-    "collapsed": false,
-    "jupyter": {
-     "outputs_hidden": false
-    }
+    "collapsed": false
    },
    "outputs": [],
    "source": [
@@ -232,12 +218,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:41.700374Z",
-     "start_time": "2023-04-25T18:23:41.698198Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "class MoneyModel(mesa.Model):\n",
@@ -269,12 +250,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:41.705824Z",
-     "start_time": "2023-04-25T18:23:41.703377Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "class MoneyAgent(mesa.Agent):\n",
@@ -337,12 +313,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:41.711368Z",
-     "start_time": "2023-04-25T18:23:41.706825Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "starter_model = MoneyModel(10)\n",
@@ -353,14 +324,7 @@
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:41.713554Z",
-     "start_time": "2023-04-25T18:23:41.711368Z"
-    },
-    "collapsed": false,
-    "jupyter": {
-     "outputs_hidden": false
-    }
+    "collapsed": false
    },
    "outputs": [],
    "source": [
@@ -380,14 +344,7 @@
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:41.716975Z",
-     "start_time": "2023-04-25T18:23:41.714555Z"
-    },
-    "collapsed": false,
-    "jupyter": {
-     "outputs_hidden": false
-    }
+    "collapsed": false
    },
    "outputs": [],
    "source": [
@@ -418,14 +375,7 @@
    "cell_type": "code",
    "execution_count": null,
    "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:41.719722Z",
-     "start_time": "2023-04-25T18:23:41.716975Z"
-    },
-    "collapsed": false,
-    "jupyter": {
-     "outputs_hidden": false
-    }
+    "collapsed": false
    },
    "outputs": [],
    "source": [
@@ -459,12 +409,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:41.723390Z",
-     "start_time": "2023-04-25T18:23:41.721719Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "import copy\n",
@@ -509,12 +454,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:41.790021Z",
-     "start_time": "2023-04-25T18:23:41.724975Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "model = MoneyModel(10)\n",
@@ -526,38 +466,15 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Next, we need to get some data out of the model. Specifically, we want to see the distribution of the agent's wealth. We can get the wealth values with list comprehension, and then use matplotlib (or another graphics library) to visualize the data in a histogram."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "If you are running from a text editor or IDE, you'll also need to add this line, to make the graph appear.\n",
-    "\n",
-    "```python\n",
-    "import matplotlib.pyplot as plt\n",
-    "plt.show()\n",
-    "```"
+    "Next, we need to get some data out of the model. Specifically, we want to see the distribution of the agent's wealth. We can get the wealth values with list comprehension, and then use seaborn (or another graphics library) to visualize the data in a histogram."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:41.882917Z",
-     "start_time": "2023-04-25T18:23:41.727492Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
-    "# For a jupyter notebook add the following line:\n",
-    "%matplotlib inline\n",
-    "\n",
-    "# The below is needed for both notebooks and scripts\n",
-    "import matplotlib.pyplot as plt\n",
-    "\n",
     "agent_wealth = [a.wealth for a in model.schedule.agents]\n",
     "# Create a histogram with seaborn\n",
     "g = sns.histplot(agent_wealth, discrete=True)\n",
@@ -583,12 +500,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:42.027373Z",
-     "start_time": "2023-04-25T18:23:41.886918Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "all_wealth = []\n",
@@ -642,12 +554,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:42.030776Z",
-     "start_time": "2023-04-25T18:23:42.028374Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "class MoneyModel(mesa.Model):\n",
@@ -733,12 +640,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:42.036559Z",
-     "start_time": "2023-04-25T18:23:42.033158Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "class MoneyAgent(mesa.Agent):\n",
@@ -798,12 +700,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:42.042251Z",
-     "start_time": "2023-04-25T18:23:42.040741Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "model = MoneyModel(100, 10, 10)\n",
@@ -815,18 +712,13 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Now let's use matplotlib and numpy to visualize the number of agents residing in each cell. To do that, we create a numpy array of the same size as the grid, filled with zeros. Then we use the grid object's `coord_iter()` feature, which lets us loop over every cell in the grid, giving us each cell's positions and contents in turn."
+    "Now let's use seaborn and numpy to visualize the number of agents residing in each cell. To do that, we create a numpy array of the same size as the grid, filled with zeros. Then we use the grid object's `coord_iter()` feature, which lets us loop over every cell in the grid, giving us each cell's positions and contents in turn."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:42.207690Z",
-     "start_time": "2023-04-25T18:23:42.043252Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "agent_counts = np.zeros((model.grid.width, model.grid.height))\n",
@@ -857,12 +749,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:42.215443Z",
-     "start_time": "2023-04-25T18:23:42.214438Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "def compute_gini(model):\n",
@@ -945,12 +832,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:42.232493Z",
-     "start_time": "2023-04-25T18:23:42.216442Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "model = MoneyModel(100, 10, 10)\n",
@@ -968,12 +850,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:42.372795Z",
-     "start_time": "2023-04-25T18:23:42.233495Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "gini = model.datacollector.get_model_vars_dataframe()\n",
@@ -992,12 +869,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:42.385021Z",
-     "start_time": "2023-04-25T18:23:42.372795Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "agent_wealth = model.datacollector.get_agent_vars_dataframe()\n",
@@ -1014,12 +886,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:42.558338Z",
-     "start_time": "2023-04-25T18:23:42.383031Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "last_step = agent_wealth.index.get_level_values(\"Step\").max()\n",
@@ -1043,12 +910,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:42.702477Z",
-     "start_time": "2023-04-25T18:23:42.520333Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "# Get the wealth of agent 14 over time\n",
@@ -1125,12 +987,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:42.711736Z",
-     "start_time": "2023-04-25T18:23:42.702477Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "# save the model data (stored in the pandas gini object) to CSV\n",
@@ -1146,25 +1003,25 @@
    "source": [
     "### Batch Run\n",
     "\n",
-    "Like we mentioned above, you usually won't run a model only once, but multiple times, with fixed parameters to find the overall distributions the model generates, and with varying parameters to analyze how they drive the model's outputs and behaviors. Instead of needing to write nested for-loops for each model, Mesa provides a [`batch_run`](https://github.com/projectmesa/mesa/blob/main/mesa/batchrunner.py) function which automates it for you."
+    "Like we mentioned above, you usually won't run a model only once, but multiple times, with fixed parameters to find the overall distributions the model generates, and with varying parameters to analyze how they drive the model's outputs and behaviors. Instead of needing to write nested for-loops for each model, Mesa provides a [`batch_run`](https://github.com/projectmesa/mesa/blob/main/mesa/batchrunner.py) function which automates it for you.\n",
+    "\n",
+    "The batch runner also requires an additional variable `self.running` for the MoneyModel class. This variable enables conditional shut off of the model once a condition is met. In this example it will be set as True indefinitely."
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "The batch runner also requires an additional variable `self.running` for the MoneyModel class. This variable enables conditional shut off of the model once a condition is met. In this example it will be set as True indefinitely."
+    "#### Additional agent reporter\n",
+    "To make the results a little bit more interesting, we will also calculate the number of consecutive time steps an agent hasn't given any wealth as an agent reporter.\n",
+    "\n",
+    "This way we can see how data is handled when multiple reporters are used."
    ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:23:42.716831Z",
-     "start_time": "2023-04-25T18:23:42.714736Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "def compute_gini(model):\n",
@@ -1179,6 +1036,7 @@
     "    \"\"\"A model with some number of agents.\"\"\"\n",
     "\n",
     "    def __init__(self, N, width, height):\n",
+    "        super().__init__()\n",
     "        self.num_agents = N\n",
     "        self.grid = mesa.space.MultiGrid(width, height, True)\n",
     "        self.schedule = mesa.time.RandomActivation(self)\n",
@@ -1194,53 +1052,71 @@
     "            self.grid.place_agent(a, (x, y))\n",
     "\n",
     "        self.datacollector = mesa.DataCollector(\n",
-    "            model_reporters={\"Gini\": compute_gini}, agent_reporters={\"Wealth\": \"wealth\"}\n",
+    "            model_reporters={\"Gini\": compute_gini},\n",
+    "            agent_reporters={\"Wealth\": \"wealth\", \"Steps_not_given\": \"steps_not_given\"},\n",
     "        )\n",
     "\n",
     "    def step(self):\n",
     "        self.datacollector.collect(self)\n",
-    "        self.schedule.step()"
+    "        self.schedule.step()\n",
+    "\n",
+    "\n",
+    "class MoneyAgent(mesa.Agent):\n",
+    "    \"\"\"An agent with fixed initial wealth.\"\"\"\n",
+    "\n",
+    "    def __init__(self, unique_id, model):\n",
+    "        super().__init__(unique_id, model)\n",
+    "        self.wealth = 1\n",
+    "        self.steps_not_given = 0\n",
+    "\n",
+    "    def move(self):\n",
+    "        possible_steps = self.model.grid.get_neighborhood(\n",
+    "            self.pos, moore=True, include_center=False\n",
+    "        )\n",
+    "        new_position = self.random.choice(possible_steps)\n",
+    "        self.model.grid.move_agent(self, new_position)\n",
+    "\n",
+    "    def give_money(self):\n",
+    "        cellmates = self.model.grid.get_cell_list_contents([self.pos])\n",
+    "        if len(cellmates) > 1:\n",
+    "            other = self.random.choice(cellmates)\n",
+    "            other.wealth += 1\n",
+    "            self.wealth -= 1\n",
+    "            self.steps_not_given = 0\n",
+    "        else:\n",
+    "            self.steps_not_given += 1\n",
+    "\n",
+    "    def step(self):\n",
+    "        self.move()\n",
+    "        if self.wealth > 0:\n",
+    "            self.give_money()\n",
+    "        else:\n",
+    "            self.steps_not_given += 1"
    ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
    "source": [
+    "#### Batch run\n",
+    "\n",
     "We call `batch_run` with the following arguments:\n",
     "\n",
     "* `model_cls`\n",
-    "<br/><br/>\n",
-    "The model class that is used for the batch run.\n",
-    "<br/><br/>\n",
-    "\n",
+    "  The model class that is used for the batch run.\n",
     "* `parameters`\n",
-    "<br/><br/>\n",
-    "A dictionary containing all the parameters of the model class and desired values to use for the batch run as key-value pairs. Each value can either be fixed ( e.g. `{\"height\": 10, \"width\": 10}`) or an iterable (e.g. `{\"N\": range(10, 500, 10)}`). `batch_run` will then generate all possible parameter combinations based on this dictionary and run the model `iterations` times for each combination.\n",
-    "<br/><br/>\n",
-    "\n",
+    "  A dictionary containing all the parameters of the model class and desired values to use for the batch run as key-value pairs. Each value can either be fixed ( e.g. `{\"height\": 10, \"width\": 10}`) or an iterable (e.g. `{\"N\": range(10, 500, 10)}`). `batch_run` will then generate all possible parameter combinations based on this dictionary and run the model `iterations` times for each combination.\n",
     "* `number_processes`\n",
-    "<br/><br/>\n",
-    "If not specified, defaults to 1. Set it to `None` to use all the available processors.\n",
-    "<br/><br/>\n",
-    "Note: Multiprocessing does make debugging challenging. If your parameter sweeps are resulting in unexpected errors set `number_processes = 1`.\n",
-    "<br/><br/>\n",
-    "\n",
+    "  If not specified, defaults to 1. Set it to `None` to use all the available processors.\n",
+    "  Note: Multiprocessing does make debugging challenging. If your parameter sweeps are resulting in unexpected errors set `number_processes=1`.\n",
     "* `iterations`\n",
-    "<br/><br/>\n",
-    "The number of iterations to run each parameter combination for. Optional. If not specified, defaults to 1.<br/><br/>\n",
+    "  The number of iterations to run each parameter combination for. Optional. If not specified, defaults to 1.\n",
     "* `data_collection_period`\n",
-    "<br/><br/>\n",
-    "The length of the period (number of steps) after which the model and agent reporters collect data. Optional. If not specified, defaults to -1, i.e. only at the end of each episode.\n",
-    "<br/><br/>\n",
-    "\n",
+    "  The length of the period (number of steps) after which the model and agent reporters collect data. Optional. If not specified, defaults to -1, i.e. only at the end of each episode.\n",
     "* `max_steps`\n",
-    "<br/><br/>\n",
-    "The maximum number of time steps after which the model halts. An episode does either end when `self.running` of the model class is set to `False` or when `model.schedule.steps == max_steps` is reached. Optional. If not specified, defaults to 1000.\n",
-    "<br/><br/>\n",
-    "\n",
+    "  The maximum number of time steps after which the model halts. An episode does either end when `self.running` of the model class is set to `False` or when `model.schedule.steps == max_steps` is reached. Optional. If not specified, defaults to 1000.\n",
     "* `display_progress`\n",
-    "<br/><br/>\n",
-    "Display the batch run progress. Optional. If not specified, defaults to `True`."
+    "  Display the batch run progress. Optional. If not specified, defaults to `True`."
    ]
   },
   {
@@ -1251,10 +1127,10 @@
     "\n",
     "We want to keep track of\n",
     "\n",
-    "1. the Gini coefficient value and\n",
-    "2. the individual agent's wealth development.\n",
+    "1. the Gini coefficient value at each time step, and\n",
+    "2. the individual agent's wealth development and steps without giving money.\n",
     "\n",
-    "Since for the latter changes at each time step might be interesting, we set `data_collection_period = 1`.\n",
+    "**Important:** Since for the latter changes at each time step might be interesting, we set `data_collection_period=1`. By default, it only collects data at the end of each episode.\n",
     "\n",
     "Note: The total number of runs is 245 (= 49 different populations * 5 iterations per population). However, the resulting list of dictionaries will be of length 6186250 (= 250 average agents per population * 49 different populations * 5 iterations per population * 101 steps per iteration). "
    ]
@@ -1269,20 +1145,15 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:24:02.422337Z",
-     "start_time": "2023-04-25T18:23:42.717833Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
-    "params = {\"width\": 10, \"height\": 10, \"N\": range(10, 500, 10)}\n",
+    "params = {\"width\": 10, \"height\": 10, \"N\": range(5, 100, 5)}\n",
     "\n",
     "results = mesa.batch_run(\n",
     "    MoneyModel,\n",
     "    parameters=params,\n",
-    "    iterations=5,\n",
+    "    iterations=7,\n",
     "    max_steps=100,\n",
     "    number_processes=1,\n",
     "    data_collection_period=1,\n",
@@ -1300,12 +1171,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:24:10.090556Z",
-     "start_time": "2023-04-25T18:24:02.423340Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "results_df = pd.DataFrame(results)\n",
@@ -1322,12 +1188,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:24:10.237362Z",
-     "start_time": "2023-04-25T18:24:10.090556Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "# Filter the results to only contain the data of one agent (the Gini coefficient will be the same for the entire population at any time) at the 100th step of each episode\n",
@@ -1343,6 +1204,33 @@
     ");"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "We can create different kinds of plot from this filtered DataFrame. For example, a point plot with error bars."
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "# Create a point plot with error bars\n",
+    "g = sns.pointplot(data=results_filtered, x=\"N\", y=\"Gini\", linestyle='none')\n",
+    "g.figure.set_size_inches(8, 4)\n",
+    "g.set(\n",
+    "    xlabel=\"Number of agents\",\n",
+    "    ylabel=\"Gini coefficient\",\n",
+    "    title=\"Gini coefficient vs. number of agents\",\n",
+    ");"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "execution_count": null
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -1356,12 +1244,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:24:10.257241Z",
-     "start_time": "2023-04-25T18:24:10.239363Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "# First, we filter the results\n",
@@ -1369,7 +1252,7 @@
     "# Then, print the columns of interest of the filtered data frame\n",
     "print(\n",
     "    one_episode_wealth.to_string(\n",
-    "        index=False, columns=[\"Step\", \"AgentID\", \"Wealth\"], max_rows=25\n",
+    "        index=False, columns=[\"Step\", \"AgentID\", \"Wealth\"], max_rows=10\n",
     "    )\n",
     ")\n",
     "# For a prettier display we can also convert the data frame to html, uncomment to test in a Jupyter Notebook\n",
@@ -1387,20 +1270,168 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "metadata": {
-    "ExecuteTime": {
-     "end_time": "2023-04-25T18:24:10.314122Z",
-     "start_time": "2023-04-25T18:24:10.258232Z"
-    }
-   },
+   "metadata": {},
    "outputs": [],
    "source": [
     "results_one_episode = results_df[\n",
     "    (results_df.N == 10) & (results_df.iteration == 1) & (results_df.AgentID == 0)\n",
     "]\n",
-    "print(results_one_episode.to_string(index=False, columns=[\"Step\", \"Gini\"], max_rows=25))"
+    "print(results_one_episode.to_string(index=False, columns=[\"Step\", \"Gini\"], max_rows=10))"
    ]
   },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "### Analyzing model reporters: Comparing 5 scenarios\n",
+    "Other insight might be gathered when we compare the Gini coefficient of different scenarios. For example, we can compare the Gini coefficient of a population with 25 agents to the Gini coefficient of a population with 400 agents. While doing this, we increase the number of iterations to 25 to get a better estimate of the Gini coefficient for each population size and get usable error estimations."
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "params = {\"width\": 10, \"height\": 10, \"N\": [5, 10, 20, 40, 80]}\n",
+    "\n",
+    "results_5s = mesa.batch_run(\n",
+    "    MoneyModel,\n",
+    "    parameters=params,\n",
+    "    iterations=100,\n",
+    "    max_steps=120,\n",
+    "    number_processes=1,\n",
+    "    data_collection_period=1,  # Important, otherwise the datacollector will only collect data of the last time step\n",
+    "    display_progress=True,\n",
+    ")\n",
+    "\n",
+    "results_5s_df = pd.DataFrame(results_5s)"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "execution_count": null
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "# Again filter the results to only contain the data of one agent (the Gini coefficient will be the same for the entire population at any time)\n",
+    "results_5s_df_filtered = results_5s_df[(results_5s_df.AgentID == 0)]\n",
+    "results_5s_df_filtered.head(3)"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "execution_count": null
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "# Create a lineplot with error bars\n",
+    "g = sns.lineplot(\n",
+    "    data=results_5s_df,\n",
+    "    x=\"Step\",\n",
+    "    y=\"Gini\",\n",
+    "    hue=\"N\",\n",
+    "    errorbar=(\"ci\", 95),\n",
+    "    palette=\"tab10\",\n",
+    ")\n",
+    "g.figure.set_size_inches(8, 4)\n",
+    "plot_title = \"Gini coefficient for different population sizes\\n(mean over 100 runs, with 95% confidence interval)\"\n",
+    "g.set(title=plot_title, ylabel=\"Gini coefficient\");"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "In this case it looks like the Gini coefficient increases slower for smaller populations. This can be because of different things, either because the Gini coefficient is a measure of inequality and the smaller the population, the more likely it is that the agents are all in the same wealth class, or because there are less interactions between agents in smaller populations, which means that the wealth of an agent is less likely to change."
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "### Analyzing agent reporters\n",
+    "From the agents we collected the wealth and the number of consecutive rounds without a transaction. We can compare the 5 different population sizes by plotting the average number of consecutive rounds without a transaction for each population size.\n",
+    "\n",
+    "Note that we're aggregating multiple times here: First we take the average of all agents for each single replication. Then we plot the averages for all replications, with the error band showing the 95% confidence interval of that first average (over all agents). So this error band is representing the uncertainty of the mean value of the number of consecutive rounds without a transaction for each population size."
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "# Calculate the mean of the wealth and the number of consecutive rounds for all agents in each episode\n",
+    "agg_results_df = (\n",
+    "    results_5s_df.groupby([\"iteration\", \"N\", \"Step\"])\n",
+    "    .agg({\"Wealth\": \"mean\", \"Steps_not_given\": \"mean\"})\n",
+    "    .reset_index()\n",
+    ")\n",
+    "agg_results_df.head(3)"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "execution_count": null
+  },
+  {
+   "cell_type": "code",
+   "outputs": [],
+   "source": [
+    "# Create a line plot with error bars\n",
+    "g = sns.lineplot(\n",
+    "    data=agg_results_df, x=\"Step\", y=\"Steps_not_given\", hue=\"N\", palette=\"tab10\"\n",
+    ")\n",
+    "g.figure.set_size_inches(8, 4)\n",
+    "g.set(\n",
+    "    title=\"Average number of consecutive rounds without a transaction for different population sizes\\n(mean with 95% confidence interval)\",\n",
+    "    ylabel=\"Consecutive rounds without a transaction\",\n",
+    ");"
+   ],
+   "metadata": {
+    "collapsed": false
+   },
+   "execution_count": null
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "It can be clearly seen that the lower the number of agents, the higher the number of consecutive rounds without a transaction. This is because the agents have fewer interactions with each other and therefore the wealth of an agent is less likely to change."
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  },
+  {
+   "cell_type": "markdown",
+   "source": [
+    "#### General steps for analyzing results\n",
+    "\n",
+    "Many other analysis are possible based on the policies, scenarios and uncertainties that you might be interested in. In general, you can follow these steps to do your own analysis:\n",
+    "\n",
+    "1. Determine which metrics you want to analyse. Add these as model and agent reporters to the datacollector of your model.\n",
+    "2. Determine the input parameters you want to vary. Add these as parameters to the batch_run function, using ranges or lists to test different values.\n",
+    "3. Determine the hyperparameters of the batch_run function. Define the number of iterations, the number of processes, the number of steps, the data collection period, etc.\n",
+    "4. Run the batch_run function and save the results.\n",
+    "5. Transform, filter and aggregate the results to get the data you want to analyze. Make sure it's in long format, so that each row represents a single value.\n",
+    "6. Choose a plot type, what to plot on the x and y axis, which columns to use for the hue. Seaborn also has an amazing [Example Gallery](https://seaborn.pydata.org/examples/index.html).\n",
+    "7. Plot the data and analyze the results."
+   ],
+   "metadata": {
+    "collapsed": false
+   }
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
diff --git a/mesa/agent.py b/mesa/agent.py
index 0b204c22..109a9dcd 100644
--- a/mesa/agent.py
+++ b/mesa/agent.py
@@ -251,7 +251,12 @@ class AgentSet(MutableSet, Sequence):
         Returns:
             AgentSet | list[Any]: The results of the method calls if return_results is True, otherwise the AgentSet itself.
         """
-        res = [getattr(agent, method_name)(*args, **kwargs) for agent in self._agents]
+        # we iterate over the actual weakref keys and check if weakref is alive before calling the method
+        res = [
+            getattr(agentref(), method_name)(*args, **kwargs)
+            for agentref in self._agents.keyrefs()
+            if agentref()
+        ]
 
         return res if return_results else self
 


Test patch:
diff --git a/tests/test_agent.py b/tests/test_agent.py
index 3cc8f9ad..5861038d 100644
--- a/tests/test_agent.py
+++ b/tests/test_agent.py
@@ -11,6 +11,26 @@ class TestAgent(Agent):
         return self.unique_id
 
 
+class TestAgentDo(Agent):
+    def __init__(
+        self,
+        unique_id,
+        model,
+    ):
+        super().__init__(unique_id, model)
+        self.agent_set = None
+
+    def get_unique_identifier(self):
+        return self.unique_id
+
+    def do_add(self):
+        agent = TestAgentDo(self.model.next_id(), self.model)
+        self.agent_set.add(agent)
+
+    def do_remove(self):
+        self.agent_set.remove(self)
+
+
 def test_agent_removal():
     model = Model()
     agent = TestAgent(model.next_id(), model)
@@ -164,6 +184,31 @@ def test_agentset_do_method():
     with pytest.raises(AttributeError):
         agentset.do("non_existing_method")
 
+    # tests for addition and removal in do
+    # do iterates, so no error should be raised to change size while iterating
+    # related to issue #1595
+
+    # setup
+    n = 10
+    model = Model()
+    agents = [TestAgentDo(model.next_id(), model) for _ in range(n)]
+    agentset = AgentSet(agents, model)
+    for agent in agents:
+        agent.agent_set = agentset
+
+    agentset.do("do_add")
+    assert len(agentset) == 2 * n
+
+    # setup
+    model = Model()
+    agents = [TestAgentDo(model.next_id(), model) for _ in range(10)]
+    agentset = AgentSet(agents, model)
+    for agent in agents:
+        agent.agent_set = agentset
+
+    agentset.do("do_remove")
+    assert len(agentset) == 0
+
 
 def test_agentset_get_attribute():
     model = Model()