Implement #28 (failFast option)

I decided to make it a regular option after all, since this seems to be
the easiest way.