Benchmarks: Fail orphaned benchmarks

When all the executor nodes for a running benchmark drop out of the
cluster, we need to force the benchmark to fail and return to the
caller.

Also fixes a bug that could sometimes cause listener.onResponse() to be
called twice