addthis/hydra — pull request #92 by mspiegel (2014-05-21 23:28:28+00:00)
Title: MarkDB database errors should error the job.

Previously markDB database errors were silent. They were logged in
the standard output and job continued to execute normally. Now these
errors cause the job to exit abnormally. This commit touches a lot of
files but most files have minor changes. The logic for erroring a job
is buried in TaskFeeder. I had to pass around an AtomicBoolean so that
all threads involved would be reading and/or writing the same state
to determine if a job has errored. To engage the old behavior of silent
failure set 'hydra.markdb.error.ignore' to True.

I have tested this change on local stack by creating a split job with a novel encoding scheme, and then downgrading the hydra job so that the encoding scheme is unrecognized. Previously the job would run to completion. Now the job errors. A split job must be used because a map job would successfully error as PageDB errors in the 'map' section of the job will cause the job to error.