Repository: popsim-consortium/stdpopsim (language: python)

Issue:
recapitation fails with new SLiM because of mismatching population names
Using the development version of SLiM, running the example from the `analysis2/README.md`:
```
import numpy as np
import stdpopsim

species = stdpopsim.get_species("HomSap")

# Adding a specific Distribution of fitness effects
dfe = species.get_dfe("Gamma_K17")

contig = species.get_contig("chr20")
model = species.get_demographic_model("OutOfAfrica_3G09")
samples = model.get_samples(100, 100, 100)  # YRI, CEU, CHB

# Extracting gene annottation
exons = species.get_annotations("ensembl_havana_104_exons")
exon_intervals = exons.get_chromosome_annotations("chr20").astype("int")
contig.add_dfe(intervals=exon_intervals, DFE=dfe)

# Engine used for simulations
engine = stdpopsim.get_engine("slim")

# Simulate
ts = engine.simulate(
    model,
    contig,
    samples,
    seed=236,
    slim_scaling_factor=100,
    slim_burn_in=10,
)
```

... gets the following error:
```
Traceback (most recent call last):
  File "/home/peter/projects/popsim-consortium/analysis2/test.py", line 22, in <module>
    ts = engine.simulate(
  File "/home/peter/.local/lib/python3.9/site-packages/stdpopsim-0.1.3b1-py3.9.egg/stdpopsim/slim_engine.py", line 1281, in simulate
    ts = self._recap_and_rescale(
  File "/home/peter/.local/lib/python3.9/site-packages/stdpopsim-0.1.3b1-py3.9.egg/stdpopsim/slim_engine.py", line 1394, in _recap_and_rescale
    ts = msprime.sim_ancestry(
  File "/home/peter/.local/lib/python3.9/site-packages/msprime/ancestry.py", line 1181, in sim_ancestry
    sim = _parse_sim_ancestry(
  File "/home/peter/.local/lib/python3.9/site-packages/msprime/ancestry.py", line 973, in _parse_sim_ancestry
    raise ValueError(
ValueError: Population names in the input demography and the initial state population table must be equal: pop_0 ≠ p0
```

I'm not clear yet if this is a problem with SLiM, with tskit udpates, or with stdpopsim.


Discussion / hints:
This is definitely on us. Older version of SLiM did not have population names and the new one does. By default, SLiM now adds names to the populations in the treesequence and uses `p0`, `p1` and so on.

In doing the recapitation, we rely on `msprime.Demography.from_old_style` to create the `Demography` object that is used in the recapitation (see [here](https://github.com/popsim-consortium/stdpopsim/blob/1c489090a6dde9414dc1b8825cdb2d0d540dbe3c/stdpopsim/slim_engine.py#L1382)). By default, msprime defines population names as `pop_0`, `pop_1` and so on, which conflicts with the names given by SLiM.

Idk what to do though. Did we update the requirements to use the new SLiM version? Or did this happen just because you had an up-to-date SLiM, @petrelharp?

@grahamgower any ideas? Ideally we should move towards having the SLiM engine dealing with msprime.Demography or demes objects, I think?

Does it work with the stable slim release?
The appropriate fix is for us to set the population names in the SLiM code we're generating (so, `p1.name = "foo"`) and make sure we set those names on the demography object that is passed into msprime for recapitation. We'll need to depend on SLiM 3.7 to do the former.

> Does it work with the stable slim release?

Yeah, no problems there.
Once the names match, there's an additional problem. See https://github.com/tskit-dev/msprime/issues/1947

Fix patch:
diff --git a/.circleci/config.yml b/.circleci/config.yml
index 1cf527eed..3794b97ae 100644
--- a/.circleci/config.yml
+++ b/.circleci/config.yml
@@ -29,8 +29,8 @@ jobs:
             git clone --bare https://github.com/MesserLab/SLiM /tmp/SLiM_top/SLiM
             cd /tmp/SLiM_top/SLiM
             git describe --tags $(git rev-list --tags --max-count=1) > ${TAG_FILE}
-            # Pinning to SLiM 3.6; comment out the next line to use latest tag.
-            echo v3.6 > ${TAG_FILE}
+            # Pinning to SLiM 3.7; comment out the next line to use latest tag.
+            echo v3.7 > ${TAG_FILE}
             echo Using SLiM version $(cat /tmp/SLiM_tag)
 
       - restore_cache:
@@ -52,7 +52,7 @@ jobs:
                 echo "Cache found!"
             fi
       - save_cache:
-          key: SLiM-v2-{{ checksum "/tmp/SLiM_tag" }}
+          key: SLiM-{{ checksum "/tmp/SLiM_tag" }}-build-0
           paths:
             - "/usr/local/bin"
       - persist_to_workspace:
@@ -74,7 +74,6 @@ jobs:
           name: Install Python dependencies
           command: |
             pip install --upgrade setuptools pip wheel twine
-            pip install numpy==1.19.2
             pip install -r requirements/CI/requirements.txt
       - save_cache:
           key: stdpopsim-v3-{{ .Branch }}-{{ checksum "requirements/CI/requirements.txt" }}
diff --git a/requirements/CI/conda.txt b/requirements/CI/conda.txt
index b42959884..8c0a03587 100644
--- a/requirements/CI/conda.txt
+++ b/requirements/CI/conda.txt
@@ -1,1 +1,1 @@
-msprime==1.0.4
+msprime==1.1.0
diff --git a/requirements/CI/requirements.txt b/requirements/CI/requirements.txt
index e1414cb90..610a4ad74 100644
--- a/requirements/CI/requirements.txt
+++ b/requirements/CI/requirements.txt
@@ -10,15 +10,15 @@ sphinx-argparse==0.3.1
 sphinx-issues==1.2.0
 sphinx_rtd_theme==1.0.0
 sphinxcontrib-programoutput==0.17
-msprime==1.0.4
+msprime==1.1.0
 attrs==21.2.0
 appdirs==1.4.4
 humanize==3.13.1
-pyslim==0.600
+pyslim==0.700
 pandas==1.3.4
 numpy==1.21.4
 scikit-allel==1.3.5
 zarr==2.10.3
 biopython==1.79
-demes==0.2.0
+demes==0.2.1
 demesdraw==0.2.0
diff --git a/requirements/development.txt b/requirements/development.txt
index ec586817d..aedb42b56 100644
--- a/requirements/development.txt
+++ b/requirements/development.txt
@@ -13,7 +13,7 @@ sphinx-argparse
 sphinx-issues
 sphinx_rtd_theme
 sphinxcontrib-programoutput
-msprime>=1.0.4
+msprime>=1.1.0
 attrs
 appdirs
 humanize
diff --git a/stdpopsim/slim_engine.py b/stdpopsim/slim_engine.py
index fd95473e8..9f8060056 100644
--- a/stdpopsim/slim_engine.py
+++ b/stdpopsim/slim_engine.py
@@ -249,8 +249,10 @@
     for (i in 0:(num_populations-1)) {
         if (N[0,i] > 0) {
             check_size(i, N[0,i], sim.generation);
-            dbg("sim.addSubpop("+i+", "+N[0,i]+");");
-            sim.addSubpop(i, N[0,i]);
+            dbg("p = sim.addSubpop("+i+", "+N[0,i]+");");
+            p = sim.addSubpop(i, N[0,i]);
+            dbg("p.name = '"+pop_names[i]+"';");
+            p.name = pop_names[i];
         }
     }
 
@@ -341,7 +343,8 @@
             check_size(newpop, size, g);
             sim.registerLateEvent(NULL,
                 "{dbg(self.source); " +
-                "sim.addSubpopSplit("+newpop+","+size+","+oldpop+");}",
+                "p = sim.addSubpopSplit("+newpop+","+size+","+oldpop+"); " +
+                "p.name = '"+pop_names[newpop]+"';}",
                 g, g);
         }
     }
@@ -1381,7 +1384,9 @@ def _recap_and_rescale(
 
         population_configurations = [
             msprime.PopulationConfiguration(
-                initial_size=pop.start_size, growth_rate=pop.growth_rate
+                initial_size=pop.start_size,
+                growth_rate=pop.growth_rate,
+                metadata={"name": pop.name},
             )
             for pop in recap_epoch.populations
         ]


Test patch:
diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml
index b34e02af9..a2af70341 100644
--- a/.github/workflows/tests.yml
+++ b/.github/workflows/tests.yml
@@ -20,7 +20,7 @@ jobs:
         os: [ubuntu-18.04, macos-10.15, windows-latest]
         python: [3.7, "3.10"]
     env:
-      SLIM_TAG: 3.6
+      SLIM_TAG: 3.7
       CONDA_ENV_NAME: stdpopsim
 
     steps:
@@ -48,7 +48,7 @@ jobs:
         uses: actions/cache@v2
         env:
           # Increase this to reset the cache if the key hasn't changed.
-          CACHE_NUM: 0
+          CACHE_NUM: 1
         with:
           path: |
             ${{ steps.find-conda.outputs.CONDA }}/envs/${{ env.CONDA_ENV_NAME }}
diff --git a/tests/test_slim_engine.py b/tests/test_slim_engine.py
index 565942759..99e21255f 100644
--- a/tests/test_slim_engine.py
+++ b/tests/test_slim_engine.py
@@ -120,7 +120,6 @@ def test_script_generation(self):
         )
         assert "sim.registerLateEvent" in out
 
-    @pytest.mark.filterwarnings("ignore::msprime.IncompletePopulationMetadataWarning")
     @pytest.mark.filterwarnings("ignore:Recombination map has length:UserWarning")
     def test_recombination_map(self):
         engine = stdpopsim.get_engine("slim")
@@ -135,7 +134,6 @@ def test_recombination_map(self):
             slim_burn_in=0.1,
         )
 
-    @pytest.mark.filterwarnings("ignore::msprime.IncompletePopulationMetadataWarning")
     @pytest.mark.filterwarnings("ignore::stdpopsim.SLiMScalingFactorWarning")
     def test_simulate(self):
         engine = stdpopsim.get_engine("slim")
@@ -153,7 +151,6 @@ def test_simulate(self):
         assert ts.num_samples == 10
         assert all(tree.num_roots == 1 for tree in ts.trees())
 
-    @pytest.mark.filterwarnings("ignore::msprime.IncompletePopulationMetadataWarning")
     @pytest.mark.filterwarnings("ignore::stdpopsim.SLiMScalingFactorWarning")
     def test_simulate_verbosity(self):
         engine = stdpopsim.get_engine("slim")
@@ -173,7 +170,6 @@ def test_simulate_verbosity(self):
             assert ts.num_samples == 10
             assert all(tree.num_roots == 1 for tree in ts.trees())
 
-    @pytest.mark.filterwarnings("ignore::msprime.IncompletePopulationMetadataWarning")
     @pytest.mark.filterwarnings("ignore::stdpopsim.SLiMScalingFactorWarning")
     @pytest.mark.filterwarnings(
         "ignore:.*model has mutation rate.*but this simulation used.*"
@@ -275,7 +271,6 @@ def test_script_generation(self):
         out, _ = self.docmd("--slim-script HomSap -d AmericanAdmixture_4B11")
         assert "sim.registerLateEvent" in out
 
-    @pytest.mark.filterwarnings("ignore::msprime.IncompletePopulationMetadataWarning")
     @pytest.mark.filterwarnings("ignore::stdpopsim.SLiMScalingFactorWarning")
     @pytest.mark.filterwarnings("ignore:.*has only.*individuals alive")
     def test_simulate(self):
@@ -926,7 +921,6 @@ def get_example_dfes(self):
         ]
         return example_dfes
 
-    @pytest.mark.filterwarnings("ignore::msprime.IncompletePopulationMetadataWarning")
     def test_slim_simulates_dfes(self):
         contig = get_test_contig()
         contig.mutation_rate = 10 * contig.mutation_rate
@@ -1003,7 +997,7 @@ def verify_mutation_rates(self, contig, ts):
             self.verify_rates_match(mut_rate, intervals, slim_rates, slim_ends)
 
         # also verify any bits not covered by DFEs have no mutations
-        empty_intervals = np.empty((0, 2))
+        empty_intervals = np.empty((0, 2), dtype="int")
         for intervals in contig.interval_list:
             empty_intervals = stdpopsim.utils.mask_intervals(
                 empty_intervals,
@@ -1050,7 +1044,6 @@ def verify_genomic_elements(self, contig, ts):
                 for a, b in zip(mt.distribution_args, slim_mt["distributionParams"]):
                     assert a == b
 
-    @pytest.mark.filterwarnings("ignore::msprime.IncompletePopulationMetadataWarning")
     def test_no_dfe(self):
         contig = get_test_contig()
         contig.clear_dfes()
@@ -1063,8 +1056,6 @@ def test_no_dfe(self):
                 samples=self.samples,
             )
 
-    @pytest.mark.skip(reason="TODO wait for SLiM v3.7")
-    @pytest.mark.filterwarnings("ignore::msprime.IncompletePopulationMetadataWarning")
     def test_default_dfe(self):
         contig = get_test_contig()
         assert len(contig.dfe_list) == 1
@@ -1078,8 +1069,6 @@ def test_default_dfe(self):
         self.verify_genomic_elements(contig, ts)
         self.verify_mutation_rates(contig, ts)
 
-    @pytest.mark.skip(reason="TODO wait for SLiM v3.7")
-    @pytest.mark.filterwarnings("ignore::msprime.IncompletePopulationMetadataWarning")
     def test_multiple_dfes(self):
         contig = get_test_contig()
         L = contig.length
@@ -1099,8 +1088,6 @@ def test_multiple_dfes(self):
         self.verify_genomic_elements(contig, ts)
         self.verify_mutation_rates(contig, ts)
 
-    @pytest.mark.skip(reason="TODO wait for SLiM v3.7")
-    @pytest.mark.filterwarnings("ignore::msprime.IncompletePopulationMetadataWarning")
     def test_unused_dfe(self):
         # test that even if a DFE ends up being unused then it'll still be there
         contig = get_test_contig()
@@ -1118,8 +1105,6 @@ def test_unused_dfe(self):
         self.verify_genomic_elements(contig, ts)
         self.verify_mutation_rates(contig, ts)
 
-    @pytest.mark.skip(reason="TODO wait for SLiM v3.7")
-    @pytest.mark.filterwarnings("ignore::msprime.IncompletePopulationMetadataWarning")
     def test_same_dfes(self):
         # test that we can add multiple DFEs with the same name
         # (maybe the same, maybe not)
@@ -1154,7 +1139,6 @@ def test_same_dfes(self):
         self.verify_genomic_elements(contig, ts)
         self.verify_mutation_rates(contig, ts)
 
-    @pytest.mark.skip(reason="TODO wait for SLiM v3.7")
     @pytest.mark.filterwarnings("ignore::stdpopsim.SLiMScalingFactorWarning")
     def test_slim_produces_mutations(self):
         contig = get_test_contig()