Repository: acorg/slurm-pipeline

Issue:
Add --firstStep and --lastStep options
These will restrict the steps that are executed in the pipeline. Or at least they will give the step scripts information (via an environment variable) that will allow them to skip doing any work.

Discussion:

Patch:
diff --git a/.gitignore b/.gitignore
index 1c7b5ad..11a2926 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,10 +1,16 @@
+*.pyc
 _trial_temp
-examples/word-count/output
 __pycache__
 dist
 slurm_pipeline.egg-info
+examples/word-count/output
 examples/blast/x??
 examples/blast/x??.blast-out
 examples/blast/BEST-HITS
 examples/blast/pipeline.log
-*.pyc
+examples/blast/specification-post-run.json
+examples/blast-with-force-and-simulate/x??
+examples/blast-with-force-and-simulate/x??.blast-out
+examples/blast-with-force-and-simulate/BEST-HITS
+examples/blast-with-force-and-simulate/pipeline.log
+examples/blast-with-force-and-simulate/specification-post-run.json
diff --git a/README.md b/README.md
index bdbb407..8c8f9c3 100644
--- a/README.md
+++ b/README.md
@@ -123,14 +123,54 @@ been scheduled, their SLURM job ids, script output, timestamps, etc. See
 the `word-count` example below for sample output (and the TODO section for
 plans for using the updated specification).
 
-#### Script environment variables
-
-Two environment variables are set when a step's script is exectued:
+### slurm-pipeline.py options
+
+`slurm-pipeline.py` accepts the following options:
+
+* `--specification filename`: as described above.
+* `--force`: Will cause `SP_FORCE` to be set in the environment of step
+  scripts with a value of `1`. It is up to the individual scripts to notice
+  this and act accordingly. If `--force` is not used, `SP_FORCE` will be
+  set to `0`.
+* `--firstStep step-name`: Step scripts are always run with an environment
+  variable called `SP_SIMULATE`. Normally this will be set to `0` for all
+  steps. Sometimes though, you may want to start a pipeline from one of its
+  intermediate steps and not re-do the work of earlier steps. Using
+  `--firstStep step-name` Will cause `step-name` to be run with
+  `SP_SIMULATE=0` in its environment. Earlier steps in the pipeline will be
+  run with `SP_SIMULATE=1`. It is up to the scripts to decide how to act
+  when `SP_SIMULATE=1`.
+* `--lastStep step-name`: Corresponds to `--firstStep`, except it turns
+  simulation back on (i.e., sets `SP_SIMULATE=1`) after the named step.
+  Note that `--firstStep` and `--lastStep` may specify the same step (to
+  just run that step) and that `--firstStep` may be given without also
+  giving `--lastStep`.
+
+Note that all script steps are *always* executed, including when
+`--firstStep` is used. It is up to the scripts to decide what to do.
+Scripts that are simulating will normally want to emit task names, as
+usual, but without doing any work. In that case they can emit the task name
+with no job id, so the later steps in the pipeline will not need to wait.
+
+It is cleaner to implement partial pipeline operation like this because it
+would otherwise be unclear how to invoke intermediate step scripts if the
+earlier scripts had not emitted any task names.  Simulated steps may still
+want to log the fact that they were run in simulated mode, etc. And it's
+conceptually easier to know that `slurm-pipeline.py` always runs all
+pipeline step scripts (see the Separation of concerns section below).
+
+### Step script environment variables
+
+The following environment variables are set when a step's script is
+exectued:
 
 * `SP_ORIGINAL_ARGS` contains the (space-separated) list of arguments
   originally passed to `slurm-pipeline.py`. Most scripts will not need to
-  know this information, but it might be useful. Scripts with no
-  dependencies get these arguments on the command line too.
+  know this information, but it might be useful. Scripts that have no
+  dependencies will be run with these arguments on the command line too.
+  Note that if an original command-line argument contained a space, and you
+  split `SP_ORIGINAL_ARGS` on spaces, you'll have two strings instead of
+  one.
 * `SP_FORCE` will be set to `1` if `--force` is given on the
   `slurm-pipeline.py` command line. This can be used to inform step scripts
   that they may overwrite pre-existing result files if they wish. If
@@ -139,6 +179,15 @@ Two environment variables are set when a step's script is exectued:
   invokes `sbatch` to guarantee that the execution of the script does not
   begin until after the tasks from all dependent steps have finished
   successfully.
+* `SP_SIMULATE` will be set to `1` if the step should be simulated, and `0`
+  if not. In simulating a step, a script should just emit its task name(s)
+  as usual, but without job ids. The presumption is that a pipeline is
+  being re-run and that the work that would normally be done by a step that
+  is now being simulated has already been done. A script that is called
+  with `SP_SIMULATE=1` might want to check that its regular output does in
+  fact already exist, but there's no need to exit if not. The entire
+  pipeline might be simulated, in which case there is no issue if
+  intermediate results are never computed.
 
 The canonical way to use `SP_DEPENDENCY_ARG` in a step script is as
 follows:
@@ -334,6 +383,29 @@ are removed along the way (uncomment the clean up `rm` line in
 `2-run-blast.sh` and `3-collect.sh` and re-run to see the 200 intermediate
 files).
 
+#### Simulated BLAST with --force and --firstStep
+
+`examples/blast-with-force-and-simulate` simulates the running of
+[BLAST](https://blast.ncbi.nlm.nih.gov/Blast.cgi) on a FASTA file, as
+above.
+
+In this case the step scripts take the values of `SP_FORCE` and
+`SP_SIMULATE` into account.
+
+As in the `examples/blast` example, all the action takes place in the same
+directory, but intermediate files are not removed along the way. This
+allows the step scripts to avoid doing work when `SP_SIMULATE=1` and to
+detect whether their output already exists, etc. A detailed log of the
+actions taken is written to `pipeline.log`.
+
+The `Makefile` has some convenient targets: `run`, `rerun`, and `force`.
+You can (and should) of course run `slurm-pipeline.py` from the command
+line yourself, with and without `--force` and using `--firstStep` and
+`--lastStep` to control which steps are simulated (i.e., receive
+`SP_SIMULATE=1` in their environment) and which are not.
+
+ Use `make clean` to get rid of the intermediate files.
+
 #### A more realistic example
 
 Another example can be seen in another of my repos,
@@ -347,14 +419,9 @@ the `examples` directory.
 
 * Make `slurm-pipeline.py` able to accept an in-progress specification
   so it can report on output, tasks, job ids, completion, timing, etc.
-* Add `--first-step` and `--last-step` (or similar) options to
-  `slurm-pipeline.py` to have it only execute a subset of the steps in a
-  specification. This will require an in-progress specification because
-  earlier processing may still be in process, and so dependency information
-  is needed. The job ids of dependent tasks are in the specification.
-* Make it possible to pass the names of the two environment variables
-  (currently hard-coded as `SP_ORIGINAL_ARGS` and `SP_DEPENDENCY_ARG`) to
-  the `SlurmPipeline` constructor.
+* Make it possible to pass the names (or at least the prefix) of the two
+  environment variables (currently hard-coded as `SP_ORIGINAL_ARGS` and
+  `SP_DEPENDENCY_ARG`) to the `SlurmPipeline` constructor.
 * Make it possible to pass a regex for matching task name and job id lines
   in a script's stdout to the `SlurmPipeline` constructor (instead of
   using the currently hard-coded `TASK: name jobid1 jobid2`).
diff --git a/bin/slurm-pipeline.py b/bin/slurm-pipeline.py
index 225ee62..971d493 100755
--- a/bin/slurm-pipeline.py
+++ b/bin/slurm-pipeline.py
@@ -19,9 +19,25 @@ parser.add_argument(
     help=('If specified, indicate to step scripts that they may overwrite '
           'pre-existing results files.'))
 
+parser.add_argument(
+    '--firstStep', metavar='step-name', default=None,
+    help=('The name of the first specification step to execute. Earlier '
+          'steps will actually be executed but they will have SP_SIMULATE=1 '
+          'in their environment, allowing them to skip doing actual work '
+          '(while still emitting task names without job numbers so that '
+          'later steps receive the correct tasks to operate on).'))
+
+parser.add_argument(
+    '--lastStep', metavar='step-name', default=None,
+    help=('The name of the last specification step to execute. See the '
+          'help text for --first-step for how this affects the environment '
+          'in which step scripts are executed.'))
+
 args, scriptArgs = parser.parse_known_args()
 
-sp = SlurmPipeline(args.specification, args.force, scriptArgs=scriptArgs)
+sp = SlurmPipeline(args.specification, force=args.force,
+                   firstStep=args.firstStep, lastStep=args.lastStep,
+                   scriptArgs=scriptArgs)
 sp.schedule()
 
 print(sp.toJSON())
diff --git a/examples/blast-with-force-and-simulate/0-start-log.sh b/examples/blast-with-force-and-simulate/0-start-log.sh
new file mode 100755
index 0000000..2b11ce8
--- /dev/null
+++ b/examples/blast-with-force-and-simulate/0-start-log.sh
@@ -0,0 +1,6 @@
+#!/bin/bash
+
+echo "Running (fake) BLAST pipeline at `date`" > pipeline.log
+echo "Input FASTA is $@" >> pipeline.log
+echo "SP_FORCE is $SP_FORCE" >> pipeline.log
+echo "SP_SIMULATE is $SP_SIMULATE" >> pipeline.log
diff --git a/examples/blast-with-force-and-simulate/1-split-fasta.sh b/examples/blast-with-force-and-simulate/1-split-fasta.sh
new file mode 100755
index 0000000..025bb65
--- /dev/null
+++ b/examples/blast-with-force-and-simulate/1-split-fasta.sh
@@ -0,0 +1,36 @@
+#!/bin/bash
+
+echo >> pipeline.log
+echo "In `basename $0`" >> pipeline.log
+echo "SP_FORCE is $SP_FORCE" >> pipeline.log
+echo "SP_SIMULATE is $SP_SIMULATE" >> pipeline.log
+
+prexistingCount=`ls x?? 2>/dev/null | wc -l | awk '{print $1}'`
+echo "There are $prexistingCount pre-existing split files." >> pipeline.log
+
+if [ $SP_SIMULATE = "0" ]
+then
+    echo "This is not a simulation." >> pipeline.log
+    if [ $prexistingCount -ne 0 ]
+    then
+        if [ $SP_FORCE = "1" ]
+        then
+            echo "Pre-existing split files exist, but --force was used. Overwriting." >> pipeline.log
+            split -l 2 $1
+        else
+            echo "Will not overwrite pre-existing split files. Use --force to make me." >> pipeline.log
+            echo "`basename $0`: Will not overwrite pre-existing split files. Use --force to make me." >&2
+            exit 1
+        fi
+    else
+        echo "No pre-existing split files exist, splitting." >> pipeline.log
+        split -l 2 $1
+    fi
+else
+    echo "This is a simulation. Will re-emit task names (the already existing split file names)." >> pipeline.log
+fi
+
+for task in x??
+do
+    echo TASK: $task
+done
diff --git a/examples/blast-with-force-and-simulate/2-run-blast.sh b/examples/blast-with-force-and-simulate/2-run-blast.sh
new file mode 100755
index 0000000..e82f96e
--- /dev/null
+++ b/examples/blast-with-force-and-simulate/2-run-blast.sh
@@ -0,0 +1,10 @@
+#!/bin/bash
+
+echo >> pipeline.log
+echo "In `basename $0`" >> pipeline.log
+echo "SP_FORCE is $SP_FORCE" >> pipeline.log
+echo "SP_SIMULATE is $SP_SIMULATE" >> pipeline.log
+
+./blast.sh -query $1 -outfmt "6 bitscore sseqid qseqid"
+
+echo "TASK: $1"
diff --git a/examples/blast-with-force-and-simulate/3-collect.sh b/examples/blast-with-force-and-simulate/3-collect.sh
new file mode 100755
index 0000000..96c42a8
--- /dev/null
+++ b/examples/blast-with-force-and-simulate/3-collect.sh
@@ -0,0 +1,38 @@
+#!/bin/bash
+
+echo >> pipeline.log
+echo "In `basename $0`" >> pipeline.log
+echo "SP_FORCE is $SP_FORCE" >> pipeline.log
+echo "SP_SIMULATE is $SP_SIMULATE" >> pipeline.log
+
+# This script could be improved by checking to see if the input files it
+# depends on are actually present. E.g.:
+#
+#   prexistingCount=`ls x??.blast-out 2>/dev/null | wc -l | awk '{print $1}'`
+#   echo "There are $prexistingCount pre-existing BLAST files." >> pipeline.log
+#
+# And checking $prexistingCount in the code below.
+
+out=BEST-HITS
+
+if [ $SP_SIMULATE = "0" ]
+then
+    echo "This is not a simulation." >> pipeline.log
+    if [ -f $out ]
+    then
+        if [ $SP_FORCE = "1" ]
+        then
+            echo "Pre-existing result file ($out) exists, but --force was used. Overwriting." >> pipeline.log
+            cat x??.blast-out | sort -nr | head -n 100 > $out
+        else
+            echo "Will not overwrite pre-existing result file ($out). Use --force to make me." >> pipeline.log
+            echo "`basename $0`: Will not overwrite pre-existing result file ($out). Use --force to make me." >&2
+            exit 1
+        fi
+    else
+        echo "No pre-existing result file ($out) exists, making it." >> pipeline.log
+        cat x??.blast-out | sort -nr | head -n 100 > $out
+    fi
+else
+    echo "This is a simulation." >> pipeline.log
+fi
diff --git a/examples/blast-with-force-and-simulate/4-end-log.sh b/examples/blast-with-force-and-simulate/4-end-log.sh
new file mode 100755
index 0000000..c3086ff
--- /dev/null
+++ b/examples/blast-with-force-and-simulate/4-end-log.sh
@@ -0,0 +1,9 @@
+#!/bin/bash
+
+echo >> pipeline.log
+echo "In `basename $0`" >> pipeline.log
+echo "SP_FORCE is $SP_FORCE" >> pipeline.log
+echo "SP_SIMULATE is $SP_SIMULATE" >> pipeline.log
+echo "BLAST pipeline finished at `date`" >> pipeline.log
+echo "Top hit is against `head -n 1 BEST-HITS | cut -f2 -d' '`" >> pipeline.log
+echo "Full output is in BEST-HITS" >> pipeline.log
diff --git a/examples/blast-with-force-and-simulate/Makefile b/examples/blast-with-force-and-simulate/Makefile
new file mode 100644
index 0000000..e181085
--- /dev/null
+++ b/examples/blast-with-force-and-simulate/Makefile
@@ -0,0 +1,13 @@
+.PHONY: run clean
+
+run:
+	../../bin/slurm-pipeline.py -s specification.json viruses.fasta > specification-post-run.json
+
+force:
+	../../bin/slurm-pipeline.py -s specification.json --force viruses.fasta > specification-post-run.json
+
+rerun:
+	../../bin/slurm-pipeline.py -s specification.json --force --firstStep collect viruses.fasta > specification-post-run.json
+
+clean:
+	rm -f x?? x??.blast-out BEST-HITS pipeline.log specification-post-run.json
diff --git a/examples/blast-with-force-and-simulate/blast.sh b/examples/blast-with-force-and-simulate/blast.sh
new file mode 100755
index 0000000..7d2eabe
--- /dev/null
+++ b/examples/blast-with-force-and-simulate/blast.sh
@@ -0,0 +1,41 @@
+#!/bin/bash
+
+echo >> pipeline.log
+echo "In `basename $0` $@" >> pipeline.log
+echo "SP_FORCE is $SP_FORCE" >> pipeline.log
+echo "SP_SIMULATE is $SP_SIMULATE" >> pipeline.log
+
+# $1 = "--query" (to simulate running BLAST), which we just ignore.
+# $3 = "--outfmt" (ditto).
+
+# $2 is given to us by run-blast.sh (it's one of the x?? FASTA files). Pull
+# out the query id so we can make fake BLAST output for it.
+task=$2
+queryId=`head -n 1 $task | cut -c2-`
+
+out=$task.blast-out
+
+if [ $SP_SIMULATE = "0" ]
+then
+    echo "This is not a simulation." >> pipeline.log
+
+    if [ -f $out ]
+    then
+        if [ $SP_FORCE = "1" ]
+        then
+            echo "Pre-existing BLAST file ($out) exists, but --force was used. Overwriting." >> pipeline.log
+            echo "$RANDOM subject-$RANDOM $queryId" > $out
+        else
+            echo "Will not overwrite pre-existing BLAST file ($out). Use --force to make me." >> pipeline.log
+            exit 0
+        fi
+    else
+        echo "No pre-existing BLAST file ($out) exists. Running BLAST." >> pipeline.log
+        # Emit fake BLAST output: bitscore, subject id, query id (taken from the FASTA).
+        echo "$RANDOM subject-$RANDOM $queryId" > $out
+    fi
+else
+    echo "This is a simulation. Will re-emit task name ($task)." >> pipeline.log
+fi
+
+echo "TASK: $task"
diff --git a/examples/blast-with-force-and-simulate/specification.json b/examples/blast-with-force-and-simulate/specification.json
new file mode 100644
index 0000000..aa5c947
--- /dev/null
+++ b/examples/blast-with-force-and-simulate/specification.json
@@ -0,0 +1,28 @@
+{
+    "steps": [
+        {
+            "name": "start-log",
+            "script": "0-start-log.sh"
+        },
+        {
+            "name": "split",
+            "script": "1-split-fasta.sh"
+        },
+        {
+            "dependencies": ["split"],
+            "name": "blast",
+            "script": "2-run-blast.sh"
+        },
+        {
+            "collect": true,
+            "dependencies": ["blast"],
+            "name": "collect",
+            "script": "3-collect.sh"
+        },
+        {
+            "dependencies": ["collect"],
+            "name": "end",
+            "script": "4-end-log.sh"
+        }
+    ]
+}
diff --git a/examples/blast-with-force-and-simulate/viruses.fasta b/examples/blast-with-force-and-simulate/viruses.fasta
new file mode 100644
index 0000000..357f7cd
--- /dev/null
+++ b/examples/blast-with-force-and-simulate/viruses.fasta
@@ -0,0 +1,200 @@
+>gi|526245010|ref|NC_021865.1| Paenibacillus phage phiIBB_Pl23, complete genome
+CCATCGTTTACACCAAATATAAATTTTTCATGAAAGGGGGGGAACCTGAAATGGCGGTACCTACTTCCAAACTCATCAGG
+>gi|460042095|ref|NC_020479.1| Bacillus phage Curly, complete genome
+CCCCCACCCCGATGAGGTCGCCTTCTCGTTCCCTGACACCAGAAGCTAATAAGGAGGAATACATATGAGTAAAAGTTGGA
+>gi|526244636|ref|NC_021857.1| Shigella phage SfII, complete genome
+TAATAATTCTCAACTGCTTGAGGGGAGGGGCGGGTCAAATCCCTGCGGCCTGACGTCTTCCGGACTGCCCGCCCCATCGT
+>gi|526003444|ref|NC_021772.1| Salmonella phage FSL SP-058, complete genome
+CCCCCCCCCCATACTTCACACATCCATGAGTCTTCGACTCTAGTGGAGCAATGGTGCTCTACTTAATCATGATTAGGAGT
+>gi|525972736|ref|NC_021782.1| Salmonella phage FSL SP-076, complete genome
+ATCAAATATATCCCAGATTGTTAATAACAAGGTGGATAGATGAACAATGATTTATTCTTAATTCTTAGTTCTATTAAATC
+>gi|548558394|ref|NC_022518.1| Human endogenous retrovirus K113 complete genome
+TGTGGGGAAAAGCAAGAGAGATCAGATTGTTACTGTGTCTGTGTAGAAAGAAGTAGACATAGGAGACTCCATTTTGTTAT
+>gi|537711871|ref|NC_022230.1| Tomato mottle mosaic virus isolate MX5, complete genome
+GTATTTATTATTACAACAATTACCAACAACAACAACAAACAACAACAACATTACATTTTACAATCTACAACTACAATGGC
+>gi|498907904|ref|NC_021203.1| Porcine stool-associated circular virus 2 isolate f, complete genome
+AAAGTGTACAATGAGGTATAGAAATGTACAAAAGGAGATTTTACAGAAGGAAGTCGTATGGTCGTAGGAACAGAAGGTTC
+>gi|498907907|ref|NC_021204.1| Porcine stool-associated circular virus 3 isolate 3L7, complete genome
+GAAATGTACATTGAGGTATGAAAATGTATAGAAGGAGATTCTACAACAGAAGGAAGTCGTATGGTCGCAGAAATAGGAGA
+>gi|475155116|ref|NC_020896.1| Sweet potato latent virus, complete genome
+AAATTAACAAAACTCAATACAACAACAGAAGAATCAAGCAAACAAAAACAAACTCAAGTTTTTATAAGCAATTGAAATTT
+>gi|408905858|ref|NC_018863.1| Bacillus phage B4, complete genome
+ATGGTAGGGTTTGGTATCGCATCACTTTTAAGTCTGGGAGGTATCCAAGTAGAGCGTACAACACTGTCATACTTTATTAC
+>gi|530787625|ref|NC_022091.1| Pseudomonas phage MPK7, complete genome
+TCCCGACTCTGATCAGCCCACCCTTCCCCCGTAGGCCCTCCGCCTGGTGGGCCATCCCTCGCGTTGCCTGGGTGTTGCTG
+>gi|552558691|ref|NC_022580.1| Drosophila obscura sigma virus 10A, complete genome
+GCGGAGATAAACAAAACATACCGATATAGAAGTCAGATAGAAAAAGTAACAGATAATCTATTATTTATCTACCAATTTTT
+>gi|552558699|ref|NC_022581.1| Eel Virus European X complete genome, viral cRNA, isolate 153311
+ACGAGAAAAACAAAACCCATTAAACGACTTAATACTTTTCTCTGCCTTTTAAGGCTATTCTTTAACAGACATCTGTTTGT
+>gi|446730228|ref|NC_019782.1| Lactobacillus phage AQ113, complete genome
+ATGTCCGAAAAAGGGGCTTTTCAAAAATTAAATAAAAAGCATCAAAAGGCTGTAAAAATGATATTTAAGCATCGTTTTAC
+>gi|538602652|ref|NC_022249.1| Feline astrovirus 2 strain 1637F, complete genome
+GGGCCAACTGGTTGAAGAGGTACCAAAGTATGTTCTACTACAACGACGCAGACCGCGGCATCTGCTCTGGCAGTATGCGT
+>gi|113200776|ref|NC_008305.1| Euphorbia mosaic virus-Yucatan Peninsula DNA B, complete sequence
+ACCGGATGGCCGCGATTTTTTTGGTGTCCCCGTTCTCGGCCCAGGCCCAACCGCATTTTACACACTCAAAACGCTCCGTT
+>gi|529218053|ref|NC_021923.1| Hemileuca sp. nucleopolyhedrovirus, complete genome
+ATGTATACTCGCTACAGTTATAGTCCTTCGTTGGGGCGTACATATGTCTATGACAACAAGTATTACAAAAACCTTGGTCA
+>gi|526178036|ref|NC_021801.1| Staphylococcus phage SA12, complete genome
+AAATTTAGAATCGCTAAAGAAAAAAGTAAATTAAAACTCAATTTACTAAAACATGCAAACAGTAATTTAGAAACAAGAAA
+>gi|554967333|ref|NC_018872.2| Pokeweed mosaic virus isolate PkMV-PA, complete genome
+AAAATAAAACAAAACATCACAACATAACACAAAACAACACATCCTATCTCAAGCTCGCAGACTTTCAAAGCATTTACAAA
+>gi|525972083|ref|NC_021776.1| Vibrio phage vpms1, complete genome
+TGACACCTATATAATTATAGCTACTATATAGGTGCCACACAGTGTAAATACTTAGGTATGTAGCCTACAAACCAAGGTAA
+>gi|448260147|ref|NC_020077.1| Sulfolobus virus STSV2, complete genome
+CTACCTTTTTCTTAGTCCTGCTATCCTCTTACGTTATACCACCCCTTTTTACGTAAATTTTACGTTTTCCCCTTATTTTG
+>gi|555928253|ref|NC_022616.1| Red clover cryptic virus 1 isolate IPP_Nemaro segment 1 RNA-dependent RNA polymerase gene, complete cds
+CAGCTGCTCTACCTCAACTTAATACCTACTCTAAATGGATTACTTCATATCCGCATTTAACCGAATCACACATTGGTTTA
+>gi|555928255|ref|NC_022617.1| Red clover cryptic virus 1 isolate IPP_Nemaro segment 2 coat protein gene, complete cds
+CAACTGCTCTACCTCAACTTTTTACTTACTCTACCGCTTTAACAAGCCGCTCGATAACCATGAATCACAACACTCCTCCT
+>gi|555928257|ref|NC_022619.1| Rhizoctonia cerealis endornavirus 1, complete genome
+ACTATTTAAATTAGTCATGCAAACCAACCAAAACACTACACAAAATACAATATCATTTAATGAACAAATGCGGGCCATAC
+>gi|555928279|ref|NC_022630.1| Razdan virus strain LEIV-Arm2741 segment L, complete sequence
+CAGCATGGAAACAAGGATACAGGATACTCGAATAAACTTTCCAAGGAGAGATGAGCTAACTGCTGAGGGCATTCTGTACC
+>gi|555928281|ref|NC_022631.1| Razdan virus strain LEIV-Arm2741 segment M, complete sequence
+ACACAAAGACGCCAGCTATCAAGATGATGTTCTCACATGTACTGATGCTGGCACTGATTTGTGCTGTAGCTTGCGATGAT
+>gi|476572429|ref|NC_019419.2| Enterobacteria phage JL1, complete genome
+GGTTTCCTTAGTAGAAGTCTACGGTTGGAACAACGGCGATCGGACGAATCACAGGGAACAAGTGCGCAATAGGGTAGCCT
+>gi|556503837|ref|NC_022647.1| Mesocricetus auratus papillomavirus 1 complete genome, isolate APV10
+ATGATTGTTAACAACAGTCATCCCTTTGAAATTTTCGAAGGCGGAGCCGTTAACGGTCGTATAAATATAGGCGGTCGAGA
+>gi|556549034|ref|NC_022646.1| Clostera anastomosis granulovirus isolate CaLGV-Henan, complete genome
+ATGGGATACAACAAATCTTTGAGGTATAGCCGACACGAGGGTACCACTTGTGTAATCGACAACCACCACCTTAAAAGTCT
+>gi|82503188|ref|NC_007605.1| Human herpesvirus 4 complete wild type genome
+AGAATTCGTCTTGCTCTATTCACCCTTACTTTTCTTCTTGCCCGTTCTCTTTCTTAGTATGAATCCAGTATGCCTGCCTG
+>gi|401829604|ref|NC_018459.1| Aino virus Gn-Gc-NSm gene for M polyprotein, segment M, genomic RNA, isolate 38K
+GTGTACTACCACAAACAACATGTTTTTCAAAGTGATATTGATCTTCATTCTATTCAAGCAATGCAAGCAAATCCCATTGC
+>gi|401829606|ref|NC_018460.1| Aino virus N and NSs genes, segment S, genomic RNA, isolate 38K
+CTCCACTATAGAACAAAGCTTTTAAATGGCAAACCAATTTATTTTCCAAGATGTTCCTCAACGGAATCTCGCTACGTTTA
+>gi|401829609|ref|NC_018461.1| Sathuperi virus RdRp gene for RNA-dependent RNA polymerase, segment L, genomic RNA
+GCCCTAATTACAATCACTATGGAGACATACAAAATAAACATCTTTAGAGATAGGATCAATCAATGTCGAAGTGCTGAAGA
+>gi|401829611|ref|NC_018462.1| Sathuperi virus N and NSs genes, segment S, genomic RNA
+CTGACATATTTCAGTAGTTAATAGTGGAATACACTACTGAAATATGTCAAGCCAATTCATTTTTGAAGATGTACCACAAC
+>gi|401829614|ref|NC_018463.1| Shamonda virus RdRp gene for RNA-dependent RNA polymerase, segment L, genomic RNA, isolate Ib An 5550
+GCCCCTAATTACAATCACTATGGAGACATACAAGATCAATATTTTTAGAGACAGGATCAACCAGTGTCGAAGTGCTGAAG
+>gi|401829616|ref|NC_018464.1| Shamonda virus N and NSs genes, segment S, genomic RNA, isolate Ib An 5550
+AGTAGTGAATCCACTATTAACTACAGAAATATGTCAAGCCAATTCATTTTTGAAGATGTACCACAACGGAATGCAGCTAC
+>gi|401829619|ref|NC_018465.1| Aino virus RdRp gene for RNA-dependent RNA polymerase, segment L, genomic RNA, isolate 38K
+CTGGGCTGCAGTTCCTCACAGTGAAGACAGCGCCAGGTGGCAAGAAGTCAAAAGGTATATCTTGAGTAGTGTACCCCTAG
+>gi|401829621|ref|NC_018466.1| Sathuperi virus Gn-Gc-NSm gene for M polyprotein, segment M, genomic RNA
+ATCAAAATGCTTCTCAACATTATCTTGATATCGAACCTAGCCTATTTGGCTTTTACACTCCCACTTAAGGAAGGCACTAG
+>gi|401829623|ref|NC_018467.1| Shamonda virus Gn-Gc-NSm gene for M polyprotein, segment M, genomic RNA, isolate Ib An 5550
+AGTAGTGAACTACCAAAATGAAAACAATCCTTCGTATCGCCTCAATTTTAGCACAATGTGCTATCATGATATGCTTACCA
+>gi|401829625|ref|NC_018476.1| Simbu virus RdRp gene for RNA-dependent RNA polymerase, segment L, genomic RNA, isolate SA Ar 53
+AGCCCCTAATATAACAATTCAATGGATAGACCAACAATAAATGCATTTAGGGATCGCATTAATGCATGCAATGATCCTGA
+>gi|401829627|ref|NC_018477.1| Simbu virus N and NSs genes, segment S, genomic RNA, isolate SA Ar 53
+CTCCACTATTAACTATATTATTAAATGGCAAACCAATTCATTTTTGAGGATGTACCTCAACGGAACTTATCTACATTTAG
+>gi|401829630|ref|NC_018478.1| Simbu virus Gn-Gc-NSm gene for M polyprotein, segment M, genomic RNA, isolate SA Ar 53
+TACCACAAATAACATGCCTTCTTACGTTGGATTTACAATTCTCTTGTGTGCCACAATTGTGGCCTCATTACCATTAAGAG
+>gi|508177721|ref|NC_021298.1| Streptomyces phage Lika, complete genome
+AGCGGCCAAGATCGCGCCACGGTTCCAGGGTCGATGAGGGTGGTGGGGGCCTGCGCTCCCAGAGGGCCGATGCAGAACCG
+>gi|508178941|ref|NC_021304.1| Streptomyces phage Sujidade, complete genome
+AGCGGCCAAGATCGCGCCACGGTTCCAGGGTCGATGAGGGTGGTGGGGGCCTGCGCTCCCAGAGGGCCGATGCAGAACCG
+>gi|509141363|ref|NC_021339.1| Streptomyces phage Zemlya, complete genome
+AGCGGCCAAGATCGCGCCACGGTTCCAGGGTCGATGAGGGTGGTGGGGGCCTGCGCTCCAAGAGGGCCGATGCAGAACCG
+>gi|557307365|ref|NC_022744.1| Erwinia phage FE44, complete genome
+TCTCATAGTTCATACAAAGTGTTTACCCCCTATGGCCTCCTTCAGTTACCCAAAGAGTCGGCCCCTTATGGTAACTTATG
+>gi|557307417|ref|NC_022745.1| Tall oatgrass mosaic virus isolate Benesov, complete genome
+AAATTAACCTCGTCCATAGCTTCTCTGATCAGCCTATAGCCATCTCGTTTTAAGCCTTCAAGCTTTCGAGTGCTCTTCTT
+>gi|557307421|ref|NC_022746.1| Pseudomonas phage MPK6, complete genome
+CCCTCCCGACTCCAACACGCCCATCCTTCCCCCGTAGGCCCTCCGCTCTGCGGTGCAGCCCTTGGTCTCCCCAGCTTCGC
+>gi|557307474|ref|NC_022747.1| Vibrio phage VPUSM 8, complete genome
+TCCTCTACACCCGCCCGCGCGGTTTTTGGATCACTTTCGAAATTTTGGTTTAGCGAAAAACGCCACACACCATCGAAACG
+>gi|557307519|ref|NC_022748.1| Solenopsis invicta densovirus isolate SiDNV-Arg, complete genome
+CCCACTTCCTAACCGAGAGGAGATAGAGGCTCCTCGAGGTTAGGTAGAGGGGGTCGACTAGTGCCCCATCTACGCCTGAC
+>gi|557307526|ref|NC_022749.1| Shigella phage SfIV, complete genome
+TCCGGACTGCCCGCCCCATCGTTTTTTTATACCCGCGAAAAATGAAATTTAACCAGGAGTGCCGCATATGGCTGGAACGG
+>gi|557307628|ref|NC_022751.1| Cyanophage PP, complete genome
+TATAGTATGTGTATGTGTAGTGTGTTCCCCCCGTTGGTGTACTCACCTTTAGTGATGGTGTGGTGGTCTAGTGATGACTG
+>gi|557307672|ref|NC_022752.1| Salmonella phage SETP13, complete genome
+ATGCGCTATGATGGTCATACGCATTTAAGAGAGGATTGCATGAAACTCAAATTAAAACAGCCTGACGCCGAAGTAGTGGC
+>gi|557307743|ref|NC_022753.1| Mycobacterium phage Fredward, complete genome
+TGCGCCTCCACGGATTTGTACGGGTTTCCAAGTCGATGAACCTTCGGAGGCCGCGCTAGAGCGCCCCTGCGCGTGGCTGT
+>gi|557428123|ref|NC_022776.1| Streptococcus phage TP-778L complete genome
+TCAAATTTTATATATTTTCGTTTTGACATGCACATTAAAACGCTCGACAGTATCTTGAGCGTGAGTTTTAGCGTAACTAT
+>gi|526118633|ref|NC_021858.1| Pandoravirus dulcis, complete genome
+GCGCAAAATTTTTTGCAGAGTCTTTGGGGTCTTCTTGTTGTCGTCGTCCCGCGCGTCGCCGTGTTGTTGTCGCGCTGCCA
+>gi|531034792|ref|NC_022098.1| Pandoravirus salinus, complete genome
+GCGACCTCGCGAAAAGGACGCCCCTTTTTTGCGTCGCCCTTTTTCTCTCTTGTCTCTTTTTTTCTTTGGAGCCGCGGCCC
+>gi|557745621|ref|NC_022788.1| Gyrovirus Tu243, complete genome
+CTTGTGTACAGGCTCCACTGTTTGTATACTTACACTTGTTGGGTACCACGGCTTGTTGCGATTGTCGCGCTGGGTGATGA
+>gi|557745625|ref|NC_022789.1| Gyrovirus Tu789, complete genome
+CCCCCCCATACCCCCCCCACGGGGGGGAACTTCCCCCCCGTACCCCCCATCTGCTAATAAATTACTTACGCGCACCAGAT
+>gi|557745636|ref|NC_022791.1| Streptococcus phage phiBHN167 complete genome
+TCACCTCGTGCACCTTGCAAGCCTTGTAAACCTTGTAATCCACGTTCTCCTTTTTCCCCGTGCACACCAATAACAGTAGG
+>gi|550455633|ref|NC_022564.1| Acheta domesticus mini ambidensovirus isolate Kalamazoo, complete genome
+TCCCACTTCTGAGCCCCGGCGCAGTGCCCGCTCGTGACCCCCGCCCTGCGCGTAGCGCGCCTTCGGCAGGCGGGGGTCAC
+>gi|538397808|ref|NC_021773.2| Staphylococcus phage JS01, complete genome
+GAGATGAGAGTGTGATACATGTTTTAGATTTTAACGACAAGATTATAGATTTCCTTTCTACTGATGACCCTTCCTTAGTT
+>gi|557940284|ref|NC_022800.1| Feline bocavirus 2 isolate POR1, complete genome
+CAGGTTAATCTAACGCGCGCGCGAAGCGCGCGGCGGCGAAGCCGCCGGCGTTATTTAACCGGCGCGATGACGTGTCAGTG
+>gi|559797726|ref|NC_022892.1| Human papillomavirus type 167 isolate KC10, complete genome
+AATTGTTGTTGGCAACAATCATTATACTATGTTTTTTTGCATACCAGAAGAGGTATATATAATTGAGGAGGAAACTTAGG
+>gi|559797734|ref|NC_022894.1| Donkey orchid symptomless virus isolate Mariginiup11, complete genome
+CCAACCCATACTACTTCGACGATGCGGGAAAACCAACCGAATTCCAGCATAACAGCAGCAAACGAAACTCGCAGCCCCTT
+>gi|559797746|ref|NC_022896.1| Sclerotinia sclerotiorum hypovirus 2 isolate 5472, complete genome
+TTTTTGGGGATGGTACTCTCAGGTTTGATCTTTGTAGATCCTAACTAAGAGGAAACATTGCCCTGTTTCGTGATGATTTC
+>gi|560186020|ref|NC_022915.1| Ralstonia phage RSK1 DNA, complete genome
+GGATGGAGGCAAACGAAGAATTTGCTGCACGCATCGCGCGCGCAAGAATGGTTGGTAGAGAGGCGATCCTCGAGGATACG
+>gi|560186125|ref|NC_022917.1| Ralstonia phage RSB3 DNA, complete genome
+TCCGACCCCAAGGCGCTCGCATTCCCCCGTGGGTACCCTCCTCGCTCTAGCCCGGACTCGCGCTGGAAATAACGAGGGTG
+>gi|548894527|ref|NC_022519.1| African elephant polyomavirus 1, complete genome
+TTTGCCTCTGAAGAAGCCTCTGAAAAGCCTCTCATTACTTTAAGAATTAGAAGGAGGCTTAAGCCTCAGGCCTCCTTATA
+>gi|560185957|ref|NC_022914.1| Staphylococcus phage phiRS7, complete genome
+ATGGCTAGACCAAGAAAATTAAACGCAGCTAAAGTTGGAAATCACAATAAAGAAGAATTAGAACAGCAAGAATTACGTGA
+>gi|560186075|ref|NC_022916.1| Burkholderia phage JG068, complete genome
+TCTGCGACAAATCCAGTCTCGCGCGATCCCCCGTATGGGGTGCCGTCGTTGCAGCGGCAGGGGAGGGGAGTGCCTCGAAG
+>gi|561142790|ref|NC_022961.1| Garlic virus D isolate GarVD-SW10, complete genome
+GAAAACTCAAACTACAACAAACTACACTGCACAGCACAAACTTTAACAACACTTGATAAGGACACCTAGCAAGGCCCTGT
+>gi|561142803|ref|NC_022962.1| Olive viral satellite RNA isolate Caltabellotta1 from Italy, complete genome
+AAATTTCTTTAAAAATTGACCGTTAGGGTCGGCGCAGCCGATTCGAAGCGGGCACTGACAGTGCTGATGGTTACTCACTG
+>gi|407868409|ref|NC_018668.1| Bovine hungarovirus 1 strain BHUV1/2008/HUN, complete genome
+TCGCCCCTCCTTTCCCCCCTTATTTTCCCTTTTCCCTCTCTCCCTCCCTTTGTAACTTAACGTTTGGCAAGAACCCTCAC
+>gi|294719514|ref|NC_009029.2| Kokobera virus, complete genome
+AGATGTTCACCTGTGTGAACTAACCAGACAGATCGAAGTTAGGTGATTACATAACACAGTGTGAACAAGTTTTTTGAACA
+>gi|563354756|ref|NC_018705.3| Ntaya virus isolate IPDIA, complete genome
+AGAAGTTCATCTGTGTGAACTTCGTGATTGACAGCTCAACACGAGTGCGGGCAACCGTAAACACAGTTTGAACGTTTTTT
+>gi|456351080|ref|NC_020415.1| Rosa rugosa leaf distortion virus isolate MN-3, complete genome
+CAAACTATGAGTTTGTTTTTCGAACTTGCCTACCTGACAGCCAAGGCTGGGTTTGTGGTAGCTAAGGAGTTGGCGCTCCT
+>gi|563397006|ref|NC_022965.1| Mycobacterium phage CloudWang3, complete genome
+TGCGCCCGGACCATCCTGTACGGGTTTACAAGTCGAAGGAACGAAGGGGCCGTCTCCAGAGCACCTATTCCTGGCCCTGT
+>gi|563397110|ref|NC_022966.1| Pseudomonas phage PAK_P5, complete genome
+ATGACTAAGACAGTCATCGGTCCTAAGTCTCCGAAGCAGGAAGCGTTTATCAACTCCACTGCGAACATCACCGTGTTCGG
+>gi|563397275|ref|NC_022967.1| Pseudomonas phage PAK_P2, complete genome
+ATGGAAAACACTAAGGTTCCTGATAAAAAACTCGGACCAATCGCGCTAGCGCTTAAAGAGGCCAGAGAGAATTTCGATCT
+>gi|563397724|ref|NC_022969.1| Mycobacterium phage PhatBacter, complete genome
+GGCTTTCGTGTGAGCCGTGACGTTTTCACAAATATGTGCCCCACCTGGGGAAATGGGGCATCTGCCCTGCATATCTATGC
+>gi|563397873|ref|NC_022970.1| Pseudomonas phage PAK_P3, complete genome
+ATGACTAAGACAGTCATCGGTCCTAAGTCTCCGAAGCAGGAAGCGTTTATCAACTCCACTGCGAACATCACCGTGTTCGG
+>gi|563398178|ref|NC_022973.1| Mycobacterium phage Conspiracy, complete genome
+TGCGCCTCCACGGTCCGCTACGGGTTTCCAAGTCGATCACCAGGGACGAGCCGCACCAGGACCCACCGTACTGGGCTCTG
+>gi|563398265|ref|NC_022974.1| Pseudomonas phage CHA_P1, complete genome
+ATGACTAAGACAGTCATCGGTCCTAAGTCTCCGAAGCAGGAAGCGTTTATCAACTCCACTGCGAACATCACCGTGTTCGG
+>gi|563398432|ref|NC_022975.1| Mycobacterium phage HanShotFirst, complete genome
+TGCGGCTGCCAGATCGTGTACGGGTTTGGAAGTCGACGGAGAGAACAGCGCGGGCCTAGAAGGCCCCGTAATGCCCCCTG
+>gi|563398524|ref|NC_022976.1| Mycobacterium phage Nala, complete genome
+GGCTTTCGTGTGAGCCGTGACGTTTTCACAAATATGTGCCCCACCTGGGGAAATGGGGCATCTGCCCTGCATATCTATGC
+>gi|563398672|ref|NC_022977.1| Mycobacterium phage Artemis2UCLA, complete genome
+TGCGCCCGGACCATCCTGTACGGGTTTACAAGTCGAAGGAACGAAGGGGCCGTCTCCAGAGCACCTATTCCTGGCCCTGT
+>gi|563398775|ref|NC_022978.1| Carnation yellow fleck virus, complete genome
+CATTTTTTATATCATCCTAATACTAGACGTACGTACGTACACCACCTAGATCTTTTTATTACCTTCCCAGAAGCCCCTCT
+>gi|563398785|ref|NC_022979.1| Mycobacterium phage Graduation, complete genome
+TGCGGCTGCCAGATTTTGTACGGGTTTGGAAGTCGACGGAGAGAACAGCGCGGGCCTAGAAGGCCCCGTATTGCCCCCTG
+>gi|563398883|ref|NC_022981.1| Mycobacterium phage HufflyPuff, complete genome
+GGCTTTCGTGTGAGCCGTGACGTTTTCACGAATATGTGCCCCACCTGGGGAAATGGGGCACATGCCCTGCATATCTATGC
+>gi|563399183|ref|NC_022984.1| Mycobacterium phage Jovo, complete genome
+TGCGCCTCCACGGTCCGCTACGGGTTTCCAAGTCGATCACCAGGGACGAGCCGCACCAGGACCCACCGTACTGGGCTCTG
+>gi|563399268|ref|NC_022985.1| Mycobacterium phage Zaka, complete genome
+TGCGCCCGGACCATCCTGTACGGGTTTACAAGTCGAAGGAACGAAGGGGCCGTCTCCAGAGCACCTATTCCTGGCCCTGT
+>gi|563399370|ref|NC_022986.1| Pseudomonas phage PAK_P4, complete genome
+ATGGAAAACACTAAGGTTCCTGATAAAAAACTCGGACCAATCGCGCTAGCGCTTAAAGAGGCCAGAGAGAATTTCGATCT
+>gi|563399598|ref|NC_022988.1| Mycobacterium phage Bruin, complete genome
+GGCTTTCGTGTGAGCCGTGACGTTTTCACAAATATGTGCCCCACCTGGGGAAATGGGGCATCTGCCCTGCATATCTATGC
+>gi|563399740|ref|NC_022990.1| Zamilon virophage complete genome
+CAAAAAGACATTTTTTTTGGGATATTAAGATTTCCTAGGATAGAAATCAAATTATTAGATAATTTTTGTATATATTTTAA
+>gi|563448535|ref|NC_015294.2| Pseudomonas phage PAK_P1, complete genome
+ATGGAAAACACGCAGGTTCCTGATAAAAAACTCGGACCAATCGCGCTAGCGCTTAAAGAGGCCAGAGAGAATTTCGATCT
+>gi|557307584|ref|NC_022750.1| Enterobacteria phage fiAA91-ss, complete genome
+GGCGAGGCGGGGAAAGCACTGCGCGCTGGCGGTGGTGCTGATTTTATTTTTTCAGCGTCTGAGCGCGTCGTGATGGCGTT
+>gi|564014228|ref|NC_022989.1| Lactobacillus phage LL-Ku, complete genome
+TAACACCCGAACCAAAAAACACACAATTTATTTTTTCGCATGGTTTTTTAGAAAAAGGTTTATGGAGCATACAAAAGCGA
+>gi|564102869|ref|NC_009232.2| Salmonella phage SETP3, complete genome
+ATGCGTTATGATGGTCATACGCATTTAAGAGAGGATTGCATGAAACTCAAATTAAAACAGCCGGACGCCGAAGTAGTGGC
diff --git a/examples/blast/blast.sh b/examples/blast/blast.sh
index b9af12e..4943ed0 100755
--- a/examples/blast/blast.sh
+++ b/examples/blast/blast.sh
@@ -1,6 +1,11 @@
 #!/bin/bash
 
-# Our $2 is given to us by run-blast.sh (it's one of the x?? FASTA files).
+
+# $1 = "--query" (to simulate running BLAST), which we just ignore.
+# $3 = "--outfmt" (ditto).
+
+# $2 is given to us by run-blast.sh (it's one of the x?? FASTA files). Pull
+# out the query id so we can make fake BLAST output for it.
 queryId=`head -n 1 $2 | cut -c2-`
 
 # Emit fake BLAST output: bitscore, subject id, query id (taken from the FASTA).
diff --git a/setup.py b/setup.py
index dc7a73c..58daefc 100644
--- a/setup.py
+++ b/setup.py
@@ -3,7 +3,7 @@
 from setuptools import setup
 
 setup(name='slurm-pipeline',
-      version='1.0.6',
+      version='1.0.7',
       packages=['slurm_pipeline'],
       include_package_data=True,
       url='https://github.com/acorg/slurm-pipeline',
diff --git a/slurm_pipeline/pipeline.py b/slurm_pipeline/pipeline.py
index f89ed74..1b99ba0 100644
--- a/slurm_pipeline/pipeline.py
+++ b/slurm_pipeline/pipeline.py
@@ -27,10 +27,19 @@ class SlurmPipeline(object):
 
     @param specification: Either a C{str} giving the name of a file containing
         a JSON execution specification, or a C{dict} holding a correctly
-        formatted execution specification.
+        formatted execution specification. Note that in the latter case the
+        passed specification C{dict} will be modified by this script.
     @param force: If C{True}, step scripts will be told (via the environment
         variable SP_FORCE=1) that they may overwrite pre-existing result files.
         I.e., that --force was used on the slurm-pipeline.py command line.
+    @param firstStep: If not C{None}, the name of the first specification step
+          to execute. Earlier steps will actually be executed but they will
+          have SP_SIMULATE=1 in their environment, allowing them to skip doing
+          actual work (while still emitting task names without job numbers so
+          that later steps receive the correct tasks to operate on.
+    @param lastStep: If not C{None}, the name of the last specification step
+          to execute. See above docs for C{firstStep} for how this affects the
+          calling of step scripts.
     @param scriptArgs: A C{list} of C{str} arguments that should be put on the
         command line of all steps that have no dependencies.
     """
@@ -41,7 +50,8 @@ class SlurmPipeline(object):
     # job ids. The following regex just matches the first part of that.
     TASK_NAME_LINE = re.compile('^TASK:\s+(\S+)\s*')
 
-    def __init__(self, specification, force=False, scriptArgs=None):
+    def __init__(self, specification, force=False, firstStep=None,
+                 lastStep=None, scriptArgs=None):
         if isinstance(specification, string_types):
             specification = self._loadSpecification(specification)
         # self.steps will be keyed by step name, with values that are
@@ -49,12 +59,17 @@ class SlurmPipeline(object):
         # access to steps by name. It is initialized in
         # self._checkSpecification.
         self.steps = {}
+        self.firstStep = firstStep
+        self.lastStep = lastStep
         self._checkSpecification(specification)
         self.specification = specification
         self._scriptArgs = scriptArgs
         self._scriptArgsStr = (
             ' '.join(map(str, scriptArgs)) if scriptArgs else '')
         environ['SP_FORCE'] = str(int(force))
+        self.specification['force'] = force
+        self.specification['firstStep'] = firstStep
+        self.specification['lastStep'] = lastStep
 
     def schedule(self):
         """
@@ -64,14 +79,42 @@ class SlurmPipeline(object):
             raise SchedulingError('Specification has already been scheduled')
         else:
             self.specification['scheduledAt'] = time()
+
+            firstStepFound = lastStepFound = False
+
             for step in self.specification['steps']:
-                self._scheduleStep(step)
+                if self.firstStep is not None:
+                    if firstStepFound:
+                        if self.lastStep is not None:
+                            if lastStepFound:
+                                simulate = True
+                            else:
+                                if step['name'] == self.lastStep:
+                                    simulate = False
+                                    lastStepFound = True
+                                else:
+                                    simulate = True
+                        else:
+                            simulate = False
+                    else:
+                        if step['name'] == self.firstStep:
+                            simulate = False
+                            firstStepFound = True
+                        else:
+                            simulate = True
+                else:
+                    simulate = False
 
-    def _scheduleStep(self, step):
+                self._scheduleStep(step, simulate)
+
+    def _scheduleStep(self, step, simulate):
         """
         Schedule a single execution step.
 
         @param step: A C{dict} with a job specification.
+        @param simulate: If C{True}, this step should be simulated. The step
+            script is still run, but with SP_SIMULATE=1 in its environment.
+            Else, SP_SIMULATE=0 will be in the environment.
         """
         assert 'scheduledAt' not in step
         assert 'tasks' not in step
@@ -94,6 +137,7 @@ class SlurmPipeline(object):
                 # it about all job ids for all tasks that are depended on.
                 env = environ.copy()
                 env['SP_ORIGINAL_ARGS'] = self._scriptArgsStr
+                env['SP_SIMULATE'] = str(int(simulate))
                 dependencies = ','.join(
                     sorted(('afterok:%d' % jobId)
                            for jobIds in taskDependencies.values()
@@ -109,6 +153,7 @@ class SlurmPipeline(object):
                 for taskName in sorted(taskDependencies):
                     env = environ.copy()
                     env['SP_ORIGINAL_ARGS'] = self._scriptArgsStr
+                    env['SP_SIMULATE'] = str(int(simulate))
                     jobIds = self.steps[stepName]['tasks'][taskName]
                     dependencies = ','.join(sorted(('afterok:%d' % jobId)
                                                    for jobId in jobIds))
@@ -130,6 +175,7 @@ class SlurmPipeline(object):
                 args = list(map(str, self._scriptArgs))
             env = environ.copy()
             env['SP_ORIGINAL_ARGS'] = self._scriptArgsStr
+            env['SP_SIMULATE'] = str(int(simulate))
             env.pop('SP_DEPENDENCY_ARG', None)
             self._runStepScript(step, args, env)
 
@@ -212,6 +258,8 @@ class SlurmPipeline(object):
         if not isinstance(specification['steps'], list):
             raise SpecificationError('The "steps" key must be a list')
 
+        firstStepFound = lastStepFound = False
+
         for count, step in enumerate(specification['steps'], start=1):
             if not isinstance(step, dict):
                 raise SpecificationError('Step %d is not a dictionary' % count)
@@ -249,6 +297,16 @@ class SlurmPipeline(object):
 
             self.steps[step['name']] = step
 
+            if step['name'] == self.firstStep:
+                firstStepFound = True
+
+            if step['name'] == self.lastStep:
+                if self.firstStep is not None and not firstStepFound:
+                    raise SpecificationError(
+                        'The lastStep %r occurs before the firstStep %r in '
+                        'the specification' % (self.lastStep, self.firstStep))
+                lastStepFound = True
+
             if 'dependencies' in step:
                 dependencies = step['dependencies']
                 if not isinstance(dependencies, list):
@@ -262,6 +320,16 @@ class SlurmPipeline(object):
                             'Step %d depends on a non-existent (or '
                             'not-yet-defined) step: %r' % (count, dependency))
 
+        if self.firstStep is not None and not firstStepFound:
+            raise SpecificationError(
+                'The firstStep %r was not found in the specification' %
+                self.firstStep)
+
+        if self.lastStep is not None and not lastStepFound:
+            raise SpecificationError(
+                'The lastStep %r was not found in the specification' %
+                self.lastStep)
+
     def toJSON(self):
         """
         Return the specification as a JSON string.


Test patch:
diff --git a/test/test_pipeline.py b/test/test_pipeline.py
index c017300..db474fc 100644
--- a/test/test_pipeline.py
+++ b/test/test_pipeline.py
@@ -296,6 +296,65 @@ class TestRunner(TestCase):
                               ]
                           })
 
+    @patch('os.access')
+    @patch('os.path.exists')
+    def testNonexistentFirstStep(self, existsMock, accessMock):
+        """
+        If SlurmPipeline is passed a firstStep value that doesn't match
+        any of the specification steps, a SpecificationError must be raised.
+        """
+        error = "^The firstStep 'xxx' was not found in the specification$"
+        assertRaisesRegex(self, SpecificationError, error, SlurmPipeline,
+                          {
+                              'steps': [
+                                  {
+                                      'name': 'name1',
+                                      'script': 'script',
+                                  },
+                              ]
+                          }, firstStep='xxx')
+
+    @patch('os.access')
+    @patch('os.path.exists')
+    def testNonexistentLastStep(self, existsMock, accessMock):
+        """
+        If SlurmPipeline is passed a lastStep value that doesn't match
+        any of the specification steps, a SpecificationError must be raised.
+        """
+        error = "^The lastStep 'xxx' was not found in the specification$"
+        assertRaisesRegex(self, SpecificationError, error, SlurmPipeline,
+                          {
+                              'steps': [
+                                  {
+                                      'name': 'name1',
+                                      'script': 'script',
+                                  },
+                              ]
+                          }, lastStep='xxx')
+
+    @patch('os.access')
+    @patch('os.path.exists')
+    def testLastStepBeforeFirstStep(self, existsMock, accessMock):
+        """
+        If SlurmPipeline is passed a lastStep value that occurs before the
+        first step, a SpecificationError must be raised.
+        """
+        error = ("^The lastStep 'name1' occurs before the firstStep 'name2' "
+                 "in the specification$")
+        assertRaisesRegex(self, SpecificationError, error, SlurmPipeline,
+                          {
+                              'steps': [
+                                  {
+                                      'name': 'name1',
+                                      'script': 'script',
+                                  },
+                                  {
+                                      'name': 'name2',
+                                      'script': 'script',
+                                  },
+                              ]
+                          }, firstStep='name2', lastStep='name1')
+
     @patch('os.access')
     @patch('os.path.exists')
     def testSpecificationIsStored(self, existsMock, accessMock):
@@ -319,6 +378,13 @@ class TestRunner(TestCase):
         mockOpener = mockOpen(read_data=data)
         with patch.object(builtins, 'open', mockOpener):
             runner = SlurmPipeline('file')
+            # Add keys to the expected specification to match what
+            # SlurmPipeline.__init__ does.
+            specification.update({
+                'force': False,
+                'firstStep': None,
+                'lastStep': None,
+            })
             self.assertEqual(specification, runner.specification)
 
     def testScheduledTime(self):
@@ -934,3 +1000,149 @@ class TestRunner(TestCase):
 
             env = mockMethod.mock_calls[0][2]['env']
             self.assertEqual('1', env['SP_FORCE'])
+
+    @patch('os.access')
+    @patch('os.path.exists')
+    def testFirstStep(self, existsMock, accessMock):
+        """
+        If firstStep is specified for a SlurmPipeline the correct SP_SIMULATE
+        value must be set in the environment.
+        """
+        with patch.object(
+                subprocess, 'check_output', return_value='') as mockMethod:
+            runner = SlurmPipeline(
+                {
+                    'steps': [
+                        {
+                            'name': 'name1',
+                            'script': 'script1',
+                        },
+                        {
+                            'name': 'name2',
+                            'script': 'script2',
+                        },
+                    ],
+                }, firstStep='name2')
+            runner.schedule()
+
+            mockMethod.assert_has_calls([
+                call(['script1'], cwd='.', universal_newlines=True,
+                     stdin=subprocess.DEVNULL, env=ANY),
+                call(['script2'], cwd='.', universal_newlines=True,
+                     stdin=subprocess.DEVNULL, env=ANY),
+            ])
+
+            # Check that the SP_SIMULATE environment variable is correct in
+            # all calls.
+            env1 = mockMethod.mock_calls[0][2]['env']
+            self.assertEqual('1', env1['SP_SIMULATE'])
+
+            env2 = mockMethod.mock_calls[1][2]['env']
+            self.assertEqual('0', env2['SP_SIMULATE'])
+
+    @patch('os.access')
+    @patch('os.path.exists')
+    def testFirstStepAndLastStepDifferent(self, existsMock, accessMock):
+        """
+        If firstStep and lastStep are specified for a SlurmPipeline and the
+        steps are not the same, the correct SP_SIMULATE value must be set
+        correctly in the environment.
+        """
+        with patch.object(
+                subprocess, 'check_output', return_value='') as mockMethod:
+            runner = SlurmPipeline(
+                {
+                    'steps': [
+                        {
+                            'name': 'name1',
+                            'script': 'script1',
+                        },
+                        {
+                            'name': 'name2',
+                            'script': 'script2',
+                        },
+                        {
+                            'name': 'name3',
+                            'script': 'script3',
+                        },
+                        {
+                            'name': 'name4',
+                            'script': 'script4',
+                        },
+                    ],
+                }, firstStep='name2', lastStep='name3')
+            runner.schedule()
+
+            mockMethod.assert_has_calls([
+                call(['script1'], cwd='.', universal_newlines=True,
+                     stdin=subprocess.DEVNULL, env=ANY),
+                call(['script2'], cwd='.', universal_newlines=True,
+                     stdin=subprocess.DEVNULL, env=ANY),
+                call(['script3'], cwd='.', universal_newlines=True,
+                     stdin=subprocess.DEVNULL, env=ANY),
+                call(['script4'], cwd='.', universal_newlines=True,
+                     stdin=subprocess.DEVNULL, env=ANY),
+            ])
+
+            # Check that the SP_SIMULATE environment variable is correct in
+            # all calls.
+            env1 = mockMethod.mock_calls[0][2]['env']
+            self.assertEqual('1', env1['SP_SIMULATE'])
+
+            env2 = mockMethod.mock_calls[1][2]['env']
+            self.assertEqual('0', env2['SP_SIMULATE'])
+
+            env3 = mockMethod.mock_calls[2][2]['env']
+            self.assertEqual('0', env3['SP_SIMULATE'])
+
+            env4 = mockMethod.mock_calls[3][2]['env']
+            self.assertEqual('1', env4['SP_SIMULATE'])
+
+    @patch('os.access')
+    @patch('os.path.exists')
+    def testFirstStepAndLastStepSame(self, existsMock, accessMock):
+        """
+        If firstStep and lastStep are specified for a SlurmPipeline and the
+        steps are the same, the correct SP_SIMULATE value must be set
+        correctly in the environment.
+        """
+        with patch.object(
+                subprocess, 'check_output', return_value='') as mockMethod:
+            runner = SlurmPipeline(
+                {
+                    'steps': [
+                        {
+                            'name': 'name1',
+                            'script': 'script1',
+                        },
+                        {
+                            'name': 'name2',
+                            'script': 'script2',
+                        },
+                        {
+                            'name': 'name3',
+                            'script': 'script3',
+                        },
+                    ],
+                }, firstStep='name2', lastStep='name2')
+            runner.schedule()
+
+            mockMethod.assert_has_calls([
+                call(['script1'], cwd='.', universal_newlines=True,
+                     stdin=subprocess.DEVNULL, env=ANY),
+                call(['script2'], cwd='.', universal_newlines=True,
+                     stdin=subprocess.DEVNULL, env=ANY),
+                call(['script3'], cwd='.', universal_newlines=True,
+                     stdin=subprocess.DEVNULL, env=ANY),
+            ])
+
+            # Check that the SP_SIMULATE environment variable is correct in
+            # all calls.
+            env1 = mockMethod.mock_calls[0][2]['env']
+            self.assertEqual('1', env1['SP_SIMULATE'])
+
+            env2 = mockMethod.mock_calls[1][2]['env']
+            self.assertEqual('0', env2['SP_SIMULATE'])
+
+            env3 = mockMethod.mock_calls[2][2]['env']
+            self.assertEqual('1', env3['SP_SIMULATE'])