Add spurious failure detection

Unfortunately, it becomes inevitable that spurious failures occur within
a test suite. Even more unfortunate is that they are usually difficult
to track down and are often extremely difficult to reproduce.

However, if a test spuriously fails less than some percentage of time,
then we'd rather not invest the time to fix it, so long as we're
convinced that when it passes in isolation it's because it is doing what
we expect it to.

In order to save engineering time in dealing with these spurious
failures, add support for spurious failure detection to Buffet.

This adds support for a single threshold customization representing how
many times we allow a spec to fail before we consider it an actual
failure.

When at least one example in a spec fails, we re-run the spec and see if
the example fails again. We keep doing this until all examples pass or
all the currently failing examples have exceeded the threshold for the
number of allowed failures.

The default (and recommended value) is 2, as that basically says "you
only get one do-over".

Note that with this implementation, the worst-case number of times a
spec will be re-run is N*T, where N is the number of examples in the
spec and T is the failure threshold. Practically, this will never
happen, as the chances of every example in the spec failing at least T
times independently each spec run is incredibly unlikely, and is
probably a sign that your test suite is borked in a really bad way.

For normal test suites, this should work as expected as most of the time
the vast majority of the spec suite is green. In the event you somehow
break every example in your suite, it will at most double the length of
your test run as they all run twice.

We collect data on each spec run because we can't assume that an example
can be uniquely identified by its file and line number. For example:

    [1, 2, 3].each do
      it { ... }
    end

This will have multiple examples reported for the same file/line number
but they need to be treated as distinct.

If we make the assumption that the number and order of examples that run
never changes across multiple spec runs, e.g. we don't have

    [1, 2, (3 if (rand < 0.5))].compact.each do
      it { ... }
    end

or

    [1, 2, 3].shuffle.each do
      it { ... }
    end

then we can still detect spurious failures.

The insight is that if you run examples A, B, and C in a spec,
then over multiple spec runs you will see a history of results like:

    [A, B, C, A, B, C, ...]

Thus if we store the history of every example that is run over the
course of all spec runs, and keep track of how many spec runs we've had,
we can still keep track of individual examples A, B, and C and their
individual success/failure status history.

Change-Id: I1bd9e30a83f4cda4dfcbea2f61d566891f89c859
Reviewed-on: https://gerrit.causes.com/27926
Tested-by: Shane da Silva <shane@causes.com>
Reviewed-by: Shane da Silva <shane@causes.com>
Reorder Buffet output to be more useful

99% of the time, when you look at the output of Buffet, you're looking
for spec failures in Jenkins, which displays output from the top, not
the bottom like in the console. The current ordering of output doesn't
match this 99% use case.

This commit changes the order to display the summary stats immediately
followed by any spec failures.

Change-Id: I606fd35f849125a5dfd589b5e6b041ffd3fae1b8
Reviewed-on: https://gerrit.causes.com/27990
Tested-by: Shane da Silva <shane@causes.com>
Reviewed-by: Shane da Silva <shane@causes.com>