Findwise/Hydra — pull request #264 by laserval (2013-10-25 12:48:52+00:00)
Title: AbstractProcessStage timeouts and cleaner shutdown

This is a continuation of @ssimon's work in #250, which fixes #247, #248 and #249.

Processing stages now create a separate processing thread, which can have an optional timeout set. If the processing takes longer than the timeout, the processing thread will be cancelled. If the thread can't cancel, the stage will destroy the JVM to prevent memory leaks and avoid corrupt state. Core should handle this as any other stage crash and restart the stage.

The motivation for this is that there is no way of knowing if a thread will exit if it can't be cancelled, which means it will take up memory and possibly continue doing work. To protect the rest of the stage JVM, the only way to stop the thread is to destroy the whole JVM.

The tests for `AbstractProcessStage` are now improved and no longer rely on timing to pass.

I couldn't come up with a good test for one case, so I created #263 for that.