Repository: acorg/slurm-pipeline

Issue:
Add skip directive and command-line option
To allow the skipping of a step, either via the specification file or on the command line. This is slightly different from simulating a run. But we should be very clear about the difference and think more to be convinced that these two are actually different.

In simulate, a step is still in the pipeline, it's just being asked to do absolutely nothing (its work was already done in a previous run and its output is in place).  A skipped step is not part of the pipeline workflow. But we don't want the pipeline to break, so the skipped step has to make sure its post-run expectations are met. So for example it might copy its input file(s) unchanged from the usual location to the place where it would normally put its output. Then the subsequent step can continue as normal, unaware that the skipped step didn't do anything.

Discussion:

Patch:
diff --git a/.gitignore b/.gitignore
index c8fe7d1..2cf9758 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,9 +1,11 @@
 *.pyc
+.tox
 _trial_temp
 __pycache__
 dist
 slurm_pipeline.egg-info
 examples/word-count/output
+examples/word-count-with-skipping/output
 examples/blast/x??
 examples/blast/x??.blast-out
 examples/blast/BEST-HITS
@@ -14,4 +16,3 @@ examples/blast-with-force-and-simulate/x??.blast-out
 examples/blast-with-force-and-simulate/BEST-HITS
 examples/blast-with-force-and-simulate/pipeline.log
 examples/blast-with-force-and-simulate/specification-post-run.json
-.tox
diff --git a/Makefile b/Makefile
index 21e46fa..99c405f 100644
--- a/Makefile
+++ b/Makefile
@@ -24,7 +24,9 @@ clean:
 	python setup.py clean
 	rm -fr slurm_pipeline.egg-info dist
 	make -C examples/word-count $@
+	make -C examples/word-count-with-skipping $@
 	make -C examples/blast $@
+	make -C examples/blast-with-force-and-simulate $@
 
 # The upload target requires that you have access rights to PYPI.
 upload:
diff --git a/README.md b/README.md
index fc4d6d4..a4f9daf 100644
--- a/README.md
+++ b/README.md
@@ -3,8 +3,6 @@
 A Python class for scheduling [SLURM](http://slurm.schedmd.com/)
 ([wikipedia](https://en.wikipedia.org/wiki/Slurm_Workload_Manager)) jobs.
 
-## Operation
-
 This repo contains a Python script, `bin/slurm-pipeline.py` that can
 schedule programs to be run in an organized pipeline fashion on a Linux
 cluster that uses SLURM as a workload manager. Here "pipeline" means with
@@ -39,7 +37,7 @@ repo. Here's an example,  `examples/word-count/specification.json`:
 The specification above contains almost everything you need to know to set
 up your own pipeline, though you of course still have to write the scripts.
 
-### Steps
+## Steps
 
 A pipeline run is organized into a series of conceptual *steps*. These are
 scheduled in the order they appear in the specification file. The actual
@@ -65,7 +63,7 @@ directory that the script should be invoked from. If no directory is given,
 the script will be run in the directory where you invoke
 `slurm-pipeline.py`.
 
-### Tasks
+## Tasks
 
 A step may choose to emit one or more *task*s. It does this by writing
 lines such as `TASK: xxx 39749 39750` to its standard output (this may
@@ -123,7 +121,7 @@ been scheduled, their SLURM job ids, script output, timestamps, etc. See
 the `word-count` example below for sample output (and the TODO section for
 plans for using the updated specification).
 
-### slurm-pipeline.py options
+## slurm-pipeline.py options
 
 `slurm-pipeline.py` accepts the following options:
 
@@ -145,21 +143,66 @@ plans for using the updated specification).
   Note that `--firstStep` and `--lastStep` may specify the same step (to
   just run that step) and that `--firstStep` may be given without also
   giving `--lastStep`.
+* `--skip`: Used to tell `slurm-pipeline.py` to tell a step script that it
+  should be skipped. In this case the script should act as though it is not
+  even in the pipeline. Commonly this will mean just taking its expected
+  input file and copying it unchanged to the place where it normally puts
+  its output. This allows the next script in the pipeline to run without
+  error. This argument may be repeated to skip multiple steps. See also the
+  `skip` directive that can be used in a specification file for more
+  permanent disabling of a script step.
 
 Note that all script steps are *always* executed, including when
-`--firstStep` is used. It is up to the scripts to decide what to do.
-Scripts that are simulating will normally want to emit task names, as
-usual, but without doing any work. In that case they can emit the task name
-with no job id, so the later steps in the pipeline will not need to wait.
+`--firstStep` or `--skip` are used. It is up to the scripts to decide what
+to do.
+
+### Simulating versus skipping
 
-It is cleaner to implement partial pipeline operation like this because it
+Scripts that are simulating will normally want to emit task names, as
+usual, but without doing any work *because the work has already been done
+in a previous run*. In that case they can emit the task name with no job
+id, so the later steps in the pipeline will not need to wait.  In
+simulation (as opposed to skipping), the script has already done its work
+(its output file(s) are already in place) and simply does nothing.
+
+Skipping refers to when a step is really no longer in the pipeline. A step
+that skips will normally just want to pass along its input to its output
+unchanged. The step still needs to be run so that it can make sure that its
+subsequent step(s) do not fail. It can be more convenient to add `"skip":
+true` to a specification file to completely get rid of a step rather than
+changing the subsequent step to take its input from the location the
+stepped script would use. Being able to use `--skip step-name` on the
+command line provides an easy way to skip a step.
+
+In summary, a simulated step doesn't do anything because its work was
+already done on a previous run, but a skipped step pretends it's not there
+at all (normally by copying its input to its output unchanged). The skipped
+step *never* does anything, the simulated step has *already* done its work.
+
+### Why are all scripts always executed?
+
+It is cleaner to implement partial pipeline operation as above because it
 would otherwise be unclear how to invoke intermediate step scripts if the
 earlier scripts had not emitted any task names.  Simulated steps may still
 want to log the fact that they were run in simulated mode, etc. And it's
 conceptually easier to know that `slurm-pipeline.py` always runs all
 pipeline step scripts (see the Separation of concerns section below).
 
-### Step script environment variables
+## Specification file directives
+
+You've already seen most of the specification file directives above. Here's
+the full list:
+
+* `name`: the name of the step (required).
+* `script`: the script to run (required).
+* `cwd`: the directory to run the script in.
+* `collect`: for scripts that should run only when all tasks from all their
+  prerequisites have completed.
+* `dependencies`: a list of previous steps that a step depends on.
+* `skip`: if `true`, the step script will be run with `SP_SKIP=1` in its
+  environment. Otherwise, `SP_SKIP` will always be set and will be `0`.
+
+## Step script environment variables
 
 The following environment variables are set when a step's script is
 exectued:
@@ -205,7 +248,7 @@ exectued:
 
     A task name may be emitted multiple times by the same script.
 
-### Separation of concerns
+## Separation of concerns
 
 `slurm-pipeline.py` doesn't interact with SLURM at all. Actually, the
 *only* thing it knows about SLURM is how to construct a dependency argument
@@ -230,11 +273,11 @@ The scripts in the examples all do their work synchronously (they emit fake
 SLURM job ids using the Bash shell's `RANDOM` variable, just to make it
 look like they are submitting jobs to `sbatch`).
 
-### Examples
+## Examples
 
 There are some simple examples in the `examples` directory:
 
-#### Word counting
+### Word counting
 
 `examples/word-count` (whose `specification.json` file is shown above) ties
 together three scripts to find the most commonly used long words in three
@@ -367,7 +410,16 @@ tasks the step depends on.  Note that the job ids will differ on your
 machine due to the use of the `$RANDOM` variable to make fake job id
 numbers in the pipeline scripts.
 
-#### Simulated BLAST
+### Word counting, skipping the long words step
+
+The `examples/word-count-with-skipping` example is exactly the same as
+`examples/word-count` but provides for the possibility of skipping the step
+that filters out short words. If you execute `make run` in that directory,
+you'll see `slurm-pipeline.py` called with `--skip long-words`. The
+resulting `output/MOST-FREQUENT-WORDS` output file contains typical
+freqeunt (short) English words such as `the`, `and`, etc.
+
+### Simulated BLAST
 
 `examples/blast` simulates the running of
 [BLAST](https://blast.ncbi.nlm.nih.gov/Blast.cgi) on a FASTA file. This is
@@ -384,7 +436,7 @@ are removed along the way (uncomment the clean up `rm` line in
 `2-run-blast.sh` and `3-collect.sh` and re-run to see the 200 intermediate
 files).
 
-#### Simulated BLAST with --force and --firstStep
+### Simulated BLAST with --force and --firstStep
 
 `examples/blast-with-force-and-simulate` simulates the running of
 [BLAST](https://blast.ncbi.nlm.nih.gov/Blast.cgi) on a FASTA file, as
@@ -407,7 +459,7 @@ line yourself, with and without `--force` and using `--firstStep` and
 
  Use `make clean` to get rid of the intermediate files.
 
-#### A more realistic example
+### A more realistic example
 
 Another example can be seen in another of my repos,
 [eske-pipeline-spec](https://github.com/acorg/eske-pipeline-spec). You wont
diff --git a/bin/slurm-pipeline.py b/bin/slurm-pipeline.py
index ea55018..8ea1640 100755
--- a/bin/slurm-pipeline.py
+++ b/bin/slurm-pipeline.py
@@ -33,6 +33,10 @@ parser.add_argument(
           'help text for --first-step for how this affects the environment '
           'in which step scripts are executed.'))
 
+parser.add_argument(
+    '--skip', metavar='step-name', default=None, action='append',
+    help='Name a step that should be skipped. May be repeated.')
+
 parser.add_argument(
     '--sleep', type=float, default=0.0,
     help=('Specify the (floating point) number of seconds to sleep for '
@@ -46,6 +50,7 @@ args, scriptArgs = parser.parse_known_args()
 sp = SlurmPipeline(args.specification, force=args.force,
                    firstStep=args.firstStep, lastStep=args.lastStep,
                    sleep=args.sleep, scriptArgs=scriptArgs)
-sp.schedule()
+
+sp.schedule(skip=(set(args.skip) if args.skip else None))
 
 print(sp.toJSON())
diff --git a/examples/blast-with-force-and-simulate/Makefile b/examples/blast-with-force-and-simulate/Makefile
index e181085..4f5a497 100644
--- a/examples/blast-with-force-and-simulate/Makefile
+++ b/examples/blast-with-force-and-simulate/Makefile
@@ -1,4 +1,4 @@
-.PHONY: run clean
+.PHONY: run, force, rerun, clean
 
 run:
 	../../bin/slurm-pipeline.py -s specification.json viruses.fasta > specification-post-run.json
diff --git a/examples/blast/Makefile b/examples/blast/Makefile
index f3ab365..b4282ab 100644
--- a/examples/blast/Makefile
+++ b/examples/blast/Makefile
@@ -1,4 +1,4 @@
-.PHONY: run clean
+.PHONY: run, clean
 
 run:
 	../../bin/slurm-pipeline.py -s specification.json viruses.fasta > specification-post-run.json
diff --git a/examples/word-count-with-skipping/Makefile b/examples/word-count-with-skipping/Makefile
new file mode 100644
index 0000000..51d11fd
--- /dev/null
+++ b/examples/word-count-with-skipping/Makefile
@@ -0,0 +1,12 @@
+.PHONY: run, sh, clean
+
+run:
+	@test -e output || mkdir output
+	rm -f output/*
+	../../bin/slurm-pipeline.py -s specification.json --skip long-words texts/*.txt > output/specification-post-run.json
+
+sh:
+	cat texts/*.txt | tr ' ' '\012' | sort | uniq -c | sort -nr | head -n 10
+
+clean:
+	rm -fr output
diff --git a/examples/word-count-with-skipping/scripts/long-words-only.sh b/examples/word-count-with-skipping/scripts/long-words-only.sh
new file mode 100755
index 0000000..30827c0
--- /dev/null
+++ b/examples/word-count-with-skipping/scripts/long-words-only.sh
@@ -0,0 +1,20 @@
+#!/bin/bash
+
+task=$1
+out=output/long-words-$task.out
+
+echo "Args are $@" > $out
+echo "SP_* env vars are:" >> $out
+set | egrep '^SP_' | sed 's/^/  /' >> $out
+
+if [ $SP_SKIP = '1' ]
+then
+    # Act as though we don't exist. We could also make a symbolic link
+    # instead of doing the actual copying.
+    cp output/$task.words output/$task.long-words
+else
+    awk 'length($0) > 5' < output/$task.words > output/$task.long-words
+fi
+
+# Pretend that we started a SLURM job to do more work on this task.
+echo "TASK: $task $RANDOM"
diff --git a/examples/word-count-with-skipping/scripts/one-word-per-line.sh b/examples/word-count-with-skipping/scripts/one-word-per-line.sh
new file mode 100755
index 0000000..5146b2e
--- /dev/null
+++ b/examples/word-count-with-skipping/scripts/one-word-per-line.sh
@@ -0,0 +1,20 @@
+#!/bin/bash
+
+out=output/one-word-per-line.out
+
+echo "Args are $@" > $out
+echo "SP_* env vars are:" >> $out
+set | egrep '^SP_' | sed 's/^/  /' >> $out
+
+# This is the first script to be run. On its command line it gets the
+# (non-specification) arguments that were given to
+# ../../bin/slurm-pipeline.py by the Makefile (i.e., texts/*.txt).
+
+for file in "$@"
+do
+    task=`basename $file | cut -f1 -d.`
+    tr ' ' '\012' < $file > output/$task.words
+
+    # Pretend that we started a SLURM job to do more work on this task.
+    echo "TASK: $task $RANDOM"
+done
diff --git a/examples/word-count-with-skipping/scripts/summarize.sh b/examples/word-count-with-skipping/scripts/summarize.sh
new file mode 100755
index 0000000..7487bf8
--- /dev/null
+++ b/examples/word-count-with-skipping/scripts/summarize.sh
@@ -0,0 +1,18 @@
+#!/bin/bash
+
+out=output/summarize.out
+
+echo "Args are $@" > $out
+echo "SP_* env vars are:" >> $out
+set | egrep '^SP_' | sed 's/^/  /' >> $out
+
+files=
+
+for base in "$@"
+do
+    files="$files output/$base.long-words"
+done
+
+echo "Files we operate on:$files" >> $out
+
+cat $files | sort | uniq -c | sort -nr | head -n 10 > output/MOST-FREQUENT-WORDS
diff --git a/examples/word-count-with-skipping/specification.json b/examples/word-count-with-skipping/specification.json
new file mode 100644
index 0000000..c5fa341
--- /dev/null
+++ b/examples/word-count-with-skipping/specification.json
@@ -0,0 +1,19 @@
+{
+    "steps": [
+        {
+            "name": "one-per-line",
+            "script": "scripts/one-word-per-line.sh"
+        },
+        {
+            "dependencies": ["one-per-line"],
+            "name": "long-words",
+            "script": "scripts/long-words-only.sh"
+        },
+        {
+            "collect": true,
+            "dependencies": ["long-words"],
+            "name": "summarize",
+            "script": "scripts/summarize.sh"
+        }
+    ]
+}
diff --git a/examples/word-count-with-skipping/texts/1-karamazov.txt b/examples/word-count-with-skipping/texts/1-karamazov.txt
new file mode 100644
index 0000000..a3c27e1
--- /dev/null
+++ b/examples/word-count-with-skipping/texts/1-karamazov.txt
@@ -0,0 +1,59 @@
+The History Of A Family
+
+Chapter I. Fyodor Pavlovitch Karamazov
+
+Alexey Fyodorovitch Karamazov was the third son of Fyodor Pavlovitch
+Karamazov, a land owner well known in our district in his own day, and
+still remembered among us owing to his gloomy and tragic death, which
+happened thirteen years ago, and which I shall describe in its proper
+place. For the present I will only say that this “landowner”—for so we used
+to call him, although he hardly spent a day of his life on his own
+estate—was a strange type, yet one pretty frequently to be met with, a type
+abject and vicious and at the same time senseless. But he was one of those
+senseless persons who are very well capable of looking after their worldly
+affairs, and, apparently, after nothing else. Fyodor Pavlovitch, for
+instance, began with next to nothing; his estate was of the smallest; he
+ran to dine at other men's tables, and fastened on them as a toady, yet at
+his death it appeared that he had a hundred thousand roubles in hard
+cash. At the same time, he was all his life one of the most senseless,
+fantastical fellows in the whole district. I repeat, it was not
+stupidity—the majority of these fantastical fellows are shrewd and
+intelligent enough—but just senselessness, and a peculiar national form of
+it.
+
+He was married twice, and had three sons, the eldest, Dmitri, by his first
+wife, and two, Ivan and Alexey, by his second. Fyodor Pavlovitch's first
+wife, Adelaïda Ivanovna, belonged to a fairly rich and distinguished noble
+family, also landowners in our district, the Miüsovs. How it came to pass
+that an heiress, who was also a beauty, and moreover one of those vigorous,
+intelligent girls, so common in this generation, but sometimes also to be
+found in the last, could have married such a worthless, puny weakling, as
+we all called him, I won't attempt to explain. I knew a young lady of the
+last “romantic” generation who after some years of an enigmatic passion for
+a gentleman, whom she might quite easily have married at any moment,
+invented insuperable obstacles to their union, and ended by throwing
+herself one stormy night into a rather deep and rapid river from a high
+bank, almost a precipice, and so perished, entirely to satisfy her own
+caprice, and to be like Shakespeare's Ophelia. Indeed, if this precipice, a
+chosen and favorite spot of hers, had been less picturesque, if there had
+been a prosaic flat bank in its place, most likely the suicide would never
+have taken place. This is a fact, and probably there have been not a few
+similar instances in the last two or three generations. Adelaïda Ivanovna
+Miüsov's action was similarly, no doubt, an echo of other people's ideas,
+and was due to the irritation caused by lack of mental freedom. She wanted,
+perhaps, to show her feminine independence, to override class distinctions
+and the despotism of her family. And a pliable imagination persuaded her,
+we must suppose, for a brief moment, that Fyodor Pavlovitch, in spite of
+his parasitic position, was one of the bold and ironical spirits of that
+progressive epoch, though he was, in fact, an ill-natured buffoon and
+nothing more. What gave the marriage piquancy was that it was preceded by
+an elopement, and this greatly captivated Adelaïda Ivanovna's fancy. Fyodor
+Pavlovitch's position at the time made him specially eager for any such
+enterprise, for he was passionately anxious to make a career in one way or
+another. To attach himself to a good family and obtain a dowry was an
+alluring prospect. As for mutual love it did not exist apparently, either
+in the bride or in him, in spite of Adelaïda Ivanovna's beauty. This was,
+perhaps, a unique case of the kind in the life of Fyodor Pavlovitch, who
+was always of a voluptuous temper, and ready to run after any petticoat on
+the slightest encouragement. She seems to have been the only woman who made
+no particular appeal to his senses.
diff --git a/examples/word-count-with-skipping/texts/2-trial.txt b/examples/word-count-with-skipping/texts/2-trial.txt
new file mode 100644
index 0000000..cf7ae4c
--- /dev/null
+++ b/examples/word-count-with-skipping/texts/2-trial.txt
@@ -0,0 +1,121 @@
+The Trial
+
+Franz Kafka
+
+Translation Copyright (C) by David Wyllie
+Translator contact email: dandelion@post.cz
+
+
+Chapter One
+Arrest - Conversation with Mrs. Grubach - Then Miss Bürstner
+
+Someone must have been telling lies about Josef K., he knew he had
+done nothing wrong but, one morning, he was arrested.  Every day at
+eight in the morning he was brought his breakfast by  Mrs. Grubach's
+cook - Mrs. Grubach was his landlady - but today she didn't come.  That
+had never happened before.  K. waited a little while, looked from his
+pillow at the old woman who lived opposite and who was watching him with
+an inquisitiveness quite unusual for her, and finally, both hungry and
+disconcerted, rang the bell.  There was immediately a knock at the door
+and a man entered.  He had never seen the man in this house before.  He
+was slim but firmly built, his clothes were black and close-fitting,
+with many folds and pockets, buckles and buttons and a belt, all of
+which gave the impression of being very practical but without making it
+very clear what they were actually for.  "Who are you?" asked K.,
+sitting half upright in his bed.  The man, however, ignored the question
+as if his arrival simply had to be accepted, and merely replied, "You
+rang?"  "Anna should have brought me my breakfast," said K.  He tried to
+work out who the man actually was, first in silence, just through
+observation and by thinking about it, but the man didn't stay still to
+be looked at for very long.  Instead he went over to the door, opened it
+slightly, and said to someone who was clearly standing immediately
+behind it, "He wants Anna to bring him his breakfast."  There was a
+little laughter in the neighbouring room, it was not clear from the
+sound of it whether there were several people laughing.  The strange man
+could not have learned anything from it that he hadn't known already,
+but now he said to K., as if making his report "It is not possible."
+"It would be the first time that's happened," said K., as he jumped out
+of bed and quickly pulled on his trousers.  "I want to see who that is
+in the next room, and why it is that Mrs. Grubach has let me be
+disturbed in this way."  It immediately occurred to him that he needn't
+have said this out loud, and that he must to some extent have
+acknowledged their authority by doing so, but that didn't seem important
+to him at the time.  That, at least, is how the stranger took it, as he
+said, "Don't you think you'd better stay where you are?"  "I want
+neither to stay here nor to be spoken to by you until you've introduced
+yourself."  "I meant it for your own good," said the stranger and opened
+the door, this time without being asked.  The next room, which K.
+entered more slowly than he had intended, looked at first glance exactly
+the same as it had the previous evening.  It was Mrs. Grubach's living
+room, over-filled with furniture, tablecloths, porcelain and
+photographs.  Perhaps there was a little more space in there than usual
+today, but if so it was not immediately obvious, especially as the main
+difference was the presence of a man sitting by the open window with a
+book from which he now looked up.  "You should have stayed in your room!
+Didn't Franz tell you?"  "And what is it you want, then?" said K.,
+looking back and forth between this new acquaintance and the one named
+Franz, who had remained in the doorway.  Through the open window he
+noticed the old woman again, who had come close to the window opposite
+so that she could continue to see everything.  She was showing an
+inquisitiveness that really made it seem like she was going senile. "I
+want to see Mrs. Grubach ...," said K., making a movement as if tearing
+himself away from the two men - even though they were standing well away
+from him - and wanted to go.  "No," said the man at the window, who
+threw his book down on a coffee table and stood up.  "You can't go away
+when you're under arrest."  "That's how it seems," said K.  "And why am
+I under arrest?" he then asked.  "That's something we're not allowed to
+tell you.  Go into your room and wait there.  Proceedings are underway
+and you'll learn about everything all in good time.  It's not really
+part of my job to be friendly towards you like this, but I hope no-one,
+apart from Franz, will hear about it, and he's been more friendly
+towards you than he should have been, under the rules, himself.  If you
+carry on having as much good luck as you have been with your arresting
+officers then you can reckon on things going well with you."  K. wanted
+to sit down, but then he saw that, apart from the chair by the window,
+there was nowhere anywhere in the room where he could sit.  "You'll get
+the chance to see for yourself how true all this is," said Franz and
+both men then walked up to K.  They were significantly bigger than him,
+especially the second man, who frequently slapped him on the shoulder.
+The two of them felt K.'s nightshirt, and said he would now have to wear
+one that was of much lower quality, but that they would keep the
+nightshirt along with his other underclothes and return them to him if
+his case turned out well.  "It's better for you if you give us the
+things than if you leave them in the storeroom," they said.  "Things
+have a tendency to go missing in the storeroom, and after a certain
+amount of time they sell things off, whether the case involved has come
+to an end or not.  And cases like this can last a long time, especially
+the ones that have been coming up lately.  They'd give you the money
+they got for them, but it wouldn't be very much as it's not what they're
+offered for them when they sell them that counts, it's how much they get
+slipped on the side, and things like that lose their value anyway when
+they get passed on from hand to hand, year after year."  K. paid hardly
+any attention to what they were saying, he did not place much value on
+what he may have still possessed or on who decided what happened to
+them.  It was much more important to him to get a clear understanding of
+his position, but he could not think clearly while these people were
+here, the second policeman's belly - and they could only be policemen -
+looked friendly enough, sticking out towards him, but when K. looked up
+and saw his dry, boney face it did not seem to fit with the body.   His
+strong nose twisted to one side as if ignoring K. and sharing an
+understanding with the other policeman.  What sort of people were these?
+What were they talking about?  What office did they belong to?  K. was
+living in a free country, after all, everywhere was at peace, all laws
+were decent and were upheld, who was it who dared accost him in his own
+home?  He was always inclined to take life as lightly as he could, to
+cross bridges when he came to them, pay no heed for the future, even
+when everything seemed under threat.  But here that did not seem the
+right thing to do.  He could have taken it all as a joke, a big joke set
+up by his colleagues at the bank for some unknown reason, or also
+perhaps because today was his thirtieth birthday, it was all possible of
+course, maybe all he had to do was laugh in the policemen's face in some
+way and they would laugh with him, maybe they were tradesmen from the
+corner of the street, they looked like they might be - but he was
+nonetheless determined, ever since he first caught sight of the one
+called Franz, not to lose any slight advantage he might have had over
+these people.  There was a very slight risk that people would later say
+he couldn't understand a joke, but - although he wasn't normally in the
+habit of learning from experience - he might also have had a few
+unimportant occasions in mind when, unlike his more cautious friends, he
+had acted with no thought at all for what might follow and had been made
+to suffer for it. He didn't want that to happen again, not this time at
+least; if they were play-acting he would act along with them.
diff --git a/examples/word-count-with-skipping/texts/3-ulysses.txt b/examples/word-count-with-skipping/texts/3-ulysses.txt
new file mode 100644
index 0000000..5f84428
--- /dev/null
+++ b/examples/word-count-with-skipping/texts/3-ulysses.txt
@@ -0,0 +1,58 @@
+Stately, plump Buck Mulligan came from the stairhead, bearing a bowl of
+lather on which a mirror and a razor lay crossed. A yellow dressinggown,
+ungirdled, was sustained gently behind him on the mild morning air. He
+held the bowl aloft and intoned:
+
+—Introibo ad altare Dei.
+
+Halted, he peered down the dark winding stairs and called out coarsely:
+
+—Come up, Kinch! Come up, you fearful jesuit!
+
+Solemnly he came forward and mounted the round gunrest. He faced about
+and blessed gravely thrice the tower, the surrounding land and the
+awaking mountains. Then, catching sight of Stephen Dedalus, he bent
+towards him and made rapid crosses in the air, gurgling in his throat
+and shaking his head. Stephen Dedalus, displeased and sleepy, leaned his
+arms on the top of the staircase and looked coldly at the shaking
+gurgling face that blessed him, equine in its length, and at the light
+untonsured hair, grained and hued like pale oak.
+
+Buck Mulligan peeped an instant under the mirror and then covered the
+bowl smartly.
+
+—Back to barracks! he said sternly.
+
+He added in a preacher's tone:
+
+—For this, O dearly beloved, is the genuine Christine: body and soul and
+blood and ouns. Slow music, please. Shut your eyes, gents. One moment. A
+little trouble about those white corpuscles. Silence, all.
+
+He peered sideways up and gave a long slow whistle of call, then paused
+awhile in rapt attention, his even white teeth glistening here and there
+with gold points. Chrysostomos. Two strong shrill whistles answered
+through the calm.
+
+—Thanks, old chap, he cried briskly. That will do nicely. Switch off the
+current, will you?
+
+He skipped off the gunrest and looked gravely at his watcher, gathering
+about his legs the loose folds of his gown. The plump shadowed face and
+sullen oval jowl recalled a prelate, patron of arts in the middle ages.
+A pleasant smile broke quietly over his lips.
+
+—The mockery of it! he said gaily. Your absurd name, an ancient Greek!
+
+He pointed his finger in friendly jest and went over to the parapet,
+laughing to himself. Stephen Dedalus stepped up, followed him wearily
+halfway and sat down on the edge of the gunrest, watching him still as
+he propped his mirror on the parapet, dipped the brush in the bowl and
+lathered cheeks and neck.
+
+Buck Mulligan's gay voice went on.
+
+—My name is absurd too: Malachi Mulligan, two dactyls. But it has a
+Hellenic ring, hasn't it? Tripping and sunny like the buck himself. We
+must go to Athens. Will you come if I can get the aunt to fork out
+twenty quid?
diff --git a/examples/word-count/Makefile b/examples/word-count/Makefile
index abf7484..c28975f 100644
--- a/examples/word-count/Makefile
+++ b/examples/word-count/Makefile
@@ -1,4 +1,4 @@
-.PHONY: run clean
+.PHONY: run, sh, clean
 
 run:
 	@test -e output || mkdir output
diff --git a/slurm_pipeline/pipeline.py b/slurm_pipeline/pipeline.py
index 5353c2f..34df6bc 100644
--- a/slurm_pipeline/pipeline.py
+++ b/slurm_pipeline/pipeline.py
@@ -14,15 +14,15 @@ except ImportError:
 
 
 class SlurmPipelineError(Exception):
-    'Base class of all SlurmPipeline exceptions'
+    'Base class of all SlurmPipeline exceptions.'
 
 
 class SchedulingError(SlurmPipelineError):
-    'An error in scheduling execution'
+    'An error in scheduling execution.'
 
 
 class SpecificationError(SlurmPipelineError):
-    'An error was found in a specification'
+    'An error was found in a specification.'
 
 
 class SlurmPipeline(object):
@@ -38,18 +38,18 @@ class SlurmPipeline(object):
         variable SP_FORCE=1) that they may overwrite pre-existing result files.
         I.e., that --force was used on the slurm-pipeline.py command line.
     @param firstStep: If not C{None}, the name of the first specification step
-          to execute. Earlier steps will actually be executed but they will
-          have SP_SIMULATE=1 in their environment, allowing them to skip doing
-          actual work (while still emitting task names without job numbers so
-          that later steps receive the correct tasks to operate on.
+        to execute. Earlier steps will actually be executed but they will
+        have SP_SIMULATE=1 in their environment, allowing them to not do
+        actual work (while still emitting task names without job numbers so
+        that later steps receive the correct tasks to operate on.
     @param lastStep: If not C{None}, the name of the last specification step
-          to execute. See above docs for C{firstStep} for how this affects the
-          calling of step scripts.
+        to execute. See above docs for C{firstStep} for how this affects the
+        calling of step scripts.
     @param sleep: Gives the C{float} number of seconds to sleep for between
-          running step scripts. This can be used to allow a distributed file
-          system to settle, so that jobs that have been scheduled can be seen
-          when used as dependencies in later invocations of sbatch. Pass 0.0
-          for no sleep.
+        running step scripts. This can be used to allow a distributed file
+        system to settle, so that jobs that have been scheduled can be seen
+        when used as dependencies in later invocations of sbatch. Pass 0.0
+        for no sleep.
     @param scriptArgs: A C{list} of C{str} arguments that should be put on the
         command line of all steps that have no dependencies.
     """
@@ -82,15 +82,32 @@ class SlurmPipeline(object):
         self.specification['firstStep'] = firstStep
         self.specification['lastStep'] = lastStep
 
-    def schedule(self):
+    def schedule(self, skip=None):
         """
         Schedule the running of our execution specification.
+
+        @param skip: A C{set} of step names that should be skipped. Those step
+            scripts will still be run, but will have C{SP_SKIP=1} in their
+            environment. Steps may also be skipped by using C{skip: "true"} in
+            the pipeline specification file.
+        @raise SchedulingError: If the specification has already been scheduled
+            or if asked to skip a non-existent step.
         """
         if 'scheduledAt' in self.specification:
             raise SchedulingError('Specification has already been scheduled')
         else:
             self.specification['scheduledAt'] = time.time()
 
+            if skip:
+                unknownSteps = skip - set(self.steps)
+                if unknownSteps:
+                    raise SchedulingError(
+                        'Unknown step%s (%s) passed to schedule' % (
+                            '' if len(unknownSteps) == 1 else 's',
+                            ', '.join(sorted(unknownSteps))))
+            else:
+                skip = set()
+
             firstStepFound = lastStepFound = False
             nSteps = len(self.specification['steps'])
 
@@ -117,14 +134,16 @@ class SlurmPipeline(object):
                 else:
                     simulate = False
 
-                self._scheduleStep(step, simulate)
+                self._scheduleStep(
+                    step, simulate,
+                    step.get('skip') or step['name'] in skip)
 
                 # If we're supposed to pause between scheduling steps and
                 # this is not the last step, then sleep.
                 if self._sleep > 0.0 and stepIndex < nSteps - 1:
                     time.sleep(self._sleep)
 
-    def _scheduleStep(self, step, simulate):
+    def _scheduleStep(self, step, simulate, skip):
         """
         Schedule a single execution step.
 
@@ -132,6 +151,10 @@ class SlurmPipeline(object):
         @param simulate: If C{True}, this step should be simulated. The step
             script is still run, but with SP_SIMULATE=1 in its environment.
             Else, SP_SIMULATE=0 will be in the environment.
+        @param skip: If C{True}, the step should be skipped, which will be
+            indicated to the script by SP_SKIP=1 in its environment. SP_SKIP
+            will be 0 in non-skipped steps. It is up to the script, which is
+            run in either case, to decide how to behave.
         """
         assert 'scheduledAt' not in step
         assert 'tasks' not in step
@@ -155,6 +178,7 @@ class SlurmPipeline(object):
                 env = environ.copy()
                 env['SP_ORIGINAL_ARGS'] = self._scriptArgsStr
                 env['SP_SIMULATE'] = str(int(simulate))
+                env['SP_SKIP'] = str(int(skip))
                 dependencies = ','.join(
                     sorted(('afterok:%d' % jobId)
                            for jobIds in taskDependencies.values()
@@ -171,6 +195,7 @@ class SlurmPipeline(object):
                     env = environ.copy()
                     env['SP_ORIGINAL_ARGS'] = self._scriptArgsStr
                     env['SP_SIMULATE'] = str(int(simulate))
+                    env['SP_SKIP'] = str(int(skip))
                     jobIds = self.steps[stepName]['tasks'][taskName]
                     dependencies = ','.join(sorted(('afterok:%d' % jobId)
                                                    for jobId in jobIds))
@@ -193,6 +218,7 @@ class SlurmPipeline(object):
             env = environ.copy()
             env['SP_ORIGINAL_ARGS'] = self._scriptArgsStr
             env['SP_SIMULATE'] = str(int(simulate))
+            env['SP_SKIP'] = str(int(skip))
             env.pop('SP_DEPENDENCY_ARG', None)
             self._runStepScript(step, args, env)
 


Test patch:
diff --git a/test/test_pipeline.py b/test/test_pipeline.py
index b4e200c..3a40033 100644
--- a/test/test_pipeline.py
+++ b/test/test_pipeline.py
@@ -1252,3 +1252,139 @@ class TestRunner(TestCase):
         runner.schedule()
 
         self.assertFalse(sleepMock.called)
+
+    @patch('subprocess.check_output')
+    @patch('os.access')
+    @patch('os.path.exists')
+    def testSkipNonexistentStep(self, existsMock, accessMock, subprocessMock):
+        """
+        If the passed skip argument contains a non-existent step name, a
+        SchedulingError must be raised.
+        """
+        error = '^Unknown step \(xxx\) passed to schedule$'
+        runner = SlurmPipeline(
+            {
+                'steps': [
+                    {
+                        'name': 'name',
+                        'script': 'script',
+                    },
+                ]
+            })
+        assertRaisesRegex(self, SchedulingError, error, runner.schedule,
+                          skip={'xxx'})
+
+    @patch('subprocess.check_output')
+    @patch('os.access')
+    @patch('os.path.exists')
+    def testSkipNonexistentSteps(self, existsMock, accessMock, subprocessMock):
+        """
+        If the passed skip argument contains two non-existent step names, a
+        SchedulingError must be raised.
+        """
+        error = '^Unknown steps \(xxx, yyy\) passed to schedule$'
+        runner = SlurmPipeline(
+            {
+                'steps': [
+                    {
+                        'name': 'name',
+                        'script': 'script',
+                    },
+                ]
+            })
+        assertRaisesRegex(self, SchedulingError, error, runner.schedule,
+                          skip={'xxx', 'yyy'})
+
+    @patch('os.access')
+    @patch('os.path.exists')
+    def testSkipNone(self, existsMock, accessMock):
+        """
+        If no steps are skipped, the SP_SKIP environment variable must be 0
+        in each step script.
+        """
+        with patch.object(
+                subprocess, 'check_output', return_value='') as mockMethod:
+            runner = SlurmPipeline(
+                {
+                    'steps': [
+                        {
+                            'name': 'name1',
+                            'script': 'script1',
+                        },
+                        {
+                            'name': 'name2',
+                            'script': 'script2',
+                        },
+                        {
+                            'name': 'name3',
+                            'script': 'script3',
+                        },
+                    ],
+                })
+            runner.schedule()
+
+            mockMethod.assert_has_calls([
+                call(['script1'], cwd='.', universal_newlines=True,
+                     stdin=DEVNULL, env=ANY),
+                call(['script2'], cwd='.', universal_newlines=True,
+                     stdin=DEVNULL, env=ANY),
+                call(['script3'], cwd='.', universal_newlines=True,
+                     stdin=DEVNULL, env=ANY),
+            ])
+
+            # Check that the SP_SKIP environment variable is 0 in all calls.
+            env1 = mockMethod.mock_calls[0][2]['env']
+            self.assertEqual('0', env1['SP_SKIP'])
+
+            env2 = mockMethod.mock_calls[1][2]['env']
+            self.assertEqual('0', env2['SP_SKIP'])
+
+            env3 = mockMethod.mock_calls[2][2]['env']
+            self.assertEqual('0', env3['SP_SKIP'])
+
+    @patch('os.access')
+    @patch('os.path.exists')
+    def testSkipTwo(self, existsMock, accessMock):
+        """
+        If two steps are skipped, the SP_SKIP variable in their environments
+        must be set to 1.
+        """
+        with patch.object(
+                subprocess, 'check_output', return_value='') as mockMethod:
+            runner = SlurmPipeline(
+                {
+                    'steps': [
+                        {
+                            'name': 'name1',
+                            'script': 'script1',
+                        },
+                        {
+                            'name': 'name2',
+                            'script': 'script2',
+                        },
+                        {
+                            'name': 'name3',
+                            'script': 'script3',
+                        },
+                    ],
+                })
+            runner.schedule(skip={'name2', 'name3'})
+
+            mockMethod.assert_has_calls([
+                call(['script1'], cwd='.', universal_newlines=True,
+                     stdin=DEVNULL, env=ANY),
+                call(['script2'], cwd='.', universal_newlines=True,
+                     stdin=DEVNULL, env=ANY),
+                call(['script3'], cwd='.', universal_newlines=True,
+                     stdin=DEVNULL, env=ANY),
+            ])
+
+            # Check that the SP_SKIP environment variable is 0 in all calls.
+            env1 = mockMethod.mock_calls[0][2]['env']
+            self.assertEqual('0', env1['SP_SKIP'])
+
+            env2 = mockMethod.mock_calls[1][2]['env']
+            self.assertEqual('1', env2['SP_SKIP'])
+
+            env3 = mockMethod.mock_calls[2][2]['env']
+            self.assertEqual('1', env3['SP_SKIP'])