Repository: projectmesa/mesa

Issue:
BatchRunner swaps content of variable_params columns, only in random proportion of executions
**Describe the bug**
When using BatchRunner, and calling get_model_vars_dataframe(), the contents of the columns in the output  of two variable parameters are swapped: the column heads stay the same, but matches with the parameter values of another parameter. I cannot tell yet whether it is just the columns of the DataFrame that is wrong, or also the batch run with the wrong values.

The remarkable thing is: when I execute my script several times, with the same code, it only happens in a number of executions. Maybe the behaviour of the bug is dependent on a random value?

**Expected behavior**
I expect the BatchRunner model vars dataframe to give back the variable parameter values that correspond with a certain parameter. Instead, the parameter values of one parameter are swapped with the parameter values of another parameter.

**To Reproduce**
I do not have a minimum working example. I have some code which depends on my specific model:

```
from mesa.batchrunner import BatchRunner
from constants import HEIGHT, WIDTH, PROPORTION_L2, SUFFIX_PROB, CAPACITY_L1, CAPACITY_L2, \
                        DROP_SUBJECT_PROB, DROP_OBJECT_PROB, MIN_BOUNDARY_FEATURE_DIST
from Model import Model

fixed_params = {
    "height": HEIGHT,
    "width": WIDTH,
    "proportion_l2": PROPORTION_L2,
    "suffix_prob": SUFFIX_PROB,
    "capacity_l1": CAPACITY_L1,
    "capacity_l2": CAPACITY_L2,
    "drop_subject_prob": DROP_SUBJECT_PROB,
    "drop_object_prob": DROP_OBJECT_PROB,
    "min_boundary_feature_dist": MIN_BOUNDARY_FEATURE_DIST
}

variable_params = {"proportion_l2":[0.0, 0.2, 0.5], "capacity_l1": [0.0,1.0,2.0,5.0]}

for k in variable_params:
    del fixed_params[k]

stats = {"global_filled_prefix_l1": lambda m: m.global_filled_prefix_l1,
         "global_filled_suffix_l1": lambda m: m.global_filled_suffix_l1,
         "global_filled_prefix_l2": lambda m: m.global_filled_prefix_l2,
         "global_filled_suffix_l2": lambda m: m.global_filled_suffix_l2}

batch_run = BatchRunner(
    Model,
    variable_params,
    fixed_params,
    iterations=1,
    max_steps=1,
    model_reporters=stats
)

batch_run.run_all()
cols = list(variable_params.keys()) + list(stats.keys())
run_data = batch_run.get_model_vars_dataframe()
run_data = run_data[cols]
print(run_data)
run_data.to_csv("evaluation.tsv", sep="\t")
```

It gives the following output:
```
12it [00:02,  5.13it/s]
    proportion_l2  capacity_l1  global_filled_prefix_l1  global_filled_suffix_l1  global_filled_prefix_l2  global_filled_suffix_l2
0             0.0          0.0                      0.0                      0.0                      0.0                      0.0
1             1.0          0.0                      0.0                      0.0                      0.0                      0.0
2             2.0          0.0                      0.0                      0.0                      0.0                      0.0
3             5.0          0.0                      0.0                      0.0                      0.0                      0.0
4             0.0          0.2                      0.0                      0.0                      0.0                      0.0
5             1.0          0.2                      0.0                      0.0                      0.0                      0.0
6             2.0          0.2                      0.0                      0.0                      0.0                      0.0
7             5.0          0.2                      0.0                      0.0                      0.0                      0.0
8             0.0          0.5                      0.0                      0.0                      0.0                      0.0
9             1.0          0.5                      0.0                      0.0                      0.0                      0.0
10            2.0          0.5                      0.0                      0.0                      0.0                      0.0
11            5.0          0.5                      0.0                      0.0                      0.0                      0.0
```

In the output, the values of ther parameters `capacity_l1` and `proportion_l2` have been swapped, if you compare them how these have been initialized.

**Additional context**
The bug occurs on two different Ubuntu machines, with mesa 0.8.7 or 0.8.8. The bug doesn't occur in mesa 0.8.6.



Discussion:
I came here to report the very same behavior. I was getting strange behavior of BatchRunner `get_model_vars_dataframe` when passing a list of non-integers for variable_params. Decided to test on the Money Model to make sure it wasn't something in my model. Added a new model parameter `initial_wealth` and then ran via: 

```
fixed_params = {"width": 10,
               "height": 10}

# The following leads to column names being swapped (N ends up as col name for
# the initial wealth values and vice versa) in the dataframe returned by get_model_var_dataframe()
variable_params = {"N": range(10, 30, 10),
                  "initial_wealth": [1.0, 1.5, 2.0]}

#If I use this version and simply swap the dict entries, the column names weren't swapped
#variable_params = {"initial_wealth": [1.0, 1.5, 2.0],
#                   "N": range(10, 30, 10)}

batch_run = BatchRunner(MoneyModel,
                        variable_params,
                        fixed_params,
                        iterations=5,
                        max_steps=100,
                        model_reporters={"Gini": compute_gini})
batch_run.run_all()

run_data = batch_run.get_model_vars_dataframe()
print(run_data.head(20))
```

Results in:
```
initial_wealth    N  Run      Gini  width  height
0               10  1.0    0  0.580000     10      10
1               10  1.0    1  0.620000     10      10
2               10  1.0    2  0.540000     10      10
3               10  1.0    3  0.580000     10      10
...
```
The use of `set` seems to be the issue, [here](https://github.com/projectmesa/mesa/compare/v0.8.6...v0.8.7#diff-8bc53945291b175d4d90954a9ced3d6b9dce4b7d3975948b315e4d09bb217aa4R228).
This seems to have been introduced by #651

This (reasonably naive) diff seems to do the trick, when I test @misken's reproducing example:

```diff
diff --git a/mesa/batchrunner.py b/mesa/batchrunner.py
index 0c43c58..45467b8 100644
--- a/mesa/batchrunner.py
+++ b/mesa/batchrunner.py
@@ -257,10 +257,10 @@ class FixedBatchRunner:
         column as a key.
         """
         extra_cols = ["Run"] + (extra_cols or [])
-        index_cols = set()
+        index_cols = []
         for params in self.parameters_list:
-            index_cols |= params.keys()
-        index_cols = list(index_cols) + extra_cols
+            index_cols.extend(key for key in params.keys() if key not in index_cols)
+        index_cols = index_cols + extra_cols
 
         records = []
         for param_key, values in vars_dict.items():
```

I don't fully understand the subtleties of the code (and the `dict(zip(index_cols, param_key))` that follows), so I'm not yet submitting this as PR. But as far as I can tell, this fixes the issue.

Resolving patch:
diff --git a/mesa/batchrunner.py b/mesa/batchrunner.py
--- a/mesa/batchrunner.py
+++ b/mesa/batchrunner.py
@@ -100,6 +100,11 @@ def __init__(
         self.iterations = iterations
         self.max_steps = max_steps
 
+        for params in self.parameters_list:
+            if list(params) != list(self.parameters_list[0]):
+                msg = "parameter names in parameters_list are not equal across the list"
+                raise ValueError(msg)
+
         self.model_reporters = model_reporters
         self.agent_reporters = agent_reporters
 
@@ -257,10 +262,10 @@ def _prepare_report_table(self, vars_dict, extra_cols=None):
         column as a key.
         """
         extra_cols = ["Run"] + (extra_cols or [])
-        index_cols = set()
-        for params in self.parameters_list:
-            index_cols |= params.keys()
-        index_cols = list(index_cols) + extra_cols
+        index_cols = []
+        if self.parameters_list:
+            index_cols = list(self.parameters_list[0].keys())
+        index_cols += extra_cols
 
         records = []
         for param_key, values in vars_dict.items():


Test patch:
diff --git a/tests/test_batchrunner.py b/tests/test_batchrunner.py
--- a/tests/test_batchrunner.py
+++ b/tests/test_batchrunner.py
@@ -8,7 +8,7 @@
 from mesa import Agent, Model
 from mesa.time import BaseScheduler
 from mesa.datacollection import DataCollector
-from mesa.batchrunner import BatchRunner, ParameterProduct, ParameterSampler
+from mesa.batchrunner import BatchRunner, FixedBatchRunner, ParameterProduct, ParameterSampler
 
 
 NUM_AGENTS = 7
@@ -120,7 +120,7 @@ def launch_batch_processing(self):
 
     def launch_batch_processing_fixed(self):
         # Adding second batchrun to test fixed params increase coverage
-        batch2 = BatchRunner(
+        batch = BatchRunner(
             self.mock_model,
             fixed_parameters={"fixed": "happy"},
             iterations=4,
@@ -129,15 +129,31 @@ def launch_batch_processing_fixed(self):
             agent_reporters=None,
         )
 
-        batch2.run_all()
-        return batch2
+        batch.run_all()
+        return batch
+
+    def launch_batch_processing_fixed_list(self):
+        batch = FixedBatchRunner(
+            self.mock_model,
+            parameters_list=self.variable_params,
+            fixed_parameters=self.fixed_params,
+            iterations=self.iterations,
+            max_steps=self.max_steps,
+            model_reporters=self.model_reporters,
+            agent_reporters=self.agent_reporters,
+        )
+        batch.run_all()
+        return batch
 
     @property
     def model_runs(self):
         """
         Returns total number of batch runner's iterations.
         """
-        return reduce(mul, map(len, self.variable_params.values())) * self.iterations
+        if isinstance(self.variable_params, list):
+            return len(self.variable_params) * self.iterations
+        else:
+            return reduce(mul, map(len, self.variable_params.values())) * self.iterations
 
     def test_model_level_vars(self):
         """
@@ -151,6 +167,11 @@ def test_model_level_vars(self):
         )  # extra column with run index
         self.assertEqual(model_vars.shape, (self.model_runs, expected_cols))
         self.assertEqual(len(model_collector.keys()), self.model_runs)
+        for var, values in self.variable_params.items():
+            self.assertEqual(set(model_vars[var].unique()), set(values))
+        if self.fixed_params:
+            for var, values in self.fixed_params.items():
+                self.assertEqual(set(model_vars[var].unique()), set(values))
 
     def test_agent_level_vars(self):
         """
@@ -169,6 +190,8 @@ def test_agent_level_vars(self):
         assert "agent_id" in list(agent_collector[(0, 1, 1)].columns)
         assert "Step" in list(agent_collector[(0, 1, 5)].index.names)
         assert "nose" not in list(agent_collector[(0, 1, 1)].columns)
+        for var, values in self.variable_params.items():
+            self.assertEqual(set(agent_vars[var].unique()), set(values))
 
         self.assertEqual(
             agent_collector[(0, 1, 0)].shape, (NUM_AGENTS * self.max_steps, 2)
@@ -224,6 +247,39 @@ def test_model_with_variable_and_fixed_kwargs(self):
             model_vars["reported_fixed_param"].iloc[0], self.fixed_params["fixed_name"]
         )
 
+    def test_model_with_variable_kwargs_list(self):
+        self.variable_params = [
+            {"variable_model_param": 1, "variable_agent_param": 1},
+            {"variable_model_param": 2, "variable_agent_param": 1},
+            {"variable_model_param": 2, "variable_agent_param": 8},
+            {"variable_model_param": 3, "variable_agent_param": 8},
+        ]
+        n_params = len(self.variable_params[0])
+        batch = self.launch_batch_processing_fixed_list()
+
+        model_vars = batch.get_model_vars_dataframe()
+        expected_cols = n_params + len(self.model_reporters) + 1
+        self.assertEqual(model_vars.shape, (self.model_runs, expected_cols))
+
+        agent_vars = batch.get_agent_vars_dataframe()
+        expected_cols = n_params + len(self.agent_reporters) + 2
+        self.assertEqual(
+            agent_vars.shape, (self.model_runs * NUM_AGENTS, expected_cols)
+        )
+
+    def test_model_with_variable_kwargs_list_mixed_length(self):
+        self.variable_params = [
+            {"variable_model_param": 1},
+            {"variable_model_param": 2},
+            {"variable_model_param": 2, "variable_agent_param": 8},
+            {"variable_model_param": 3, "variable_agent_param": 8},
+            {"variable_agent_param": 1},
+        ]
+        # This is currently not supported. Check that it raises the correct error.
+        msg = "parameter names in parameters_list are not equal across the list"
+        with self.assertRaises(ValueError, msg=msg):
+            self.launch_batch_processing_fixed_list()
+
 
 class TestParameters(unittest.TestCase):
     def test_product(self):