totalcareauto/basic-consumer — pull request #1 by scizo (2013-12-04 08:09:07+00:00)
Title: Refactor from event based to pipeline process

This turned into quite a large refactor of the codebase that started out
primarily as an attempt to not delete the retry files until after they
were processed. Dealing with the evented model became more complex as
more was added and I think this is a significantly easier model to
reason about and deal with.

The new process is basically a pipeline of workers each with their own
thread or thread pool communicating coordinating with each other with
queues. There are two producers, the Broker, which produces messages as
they are pushed from the broker, and the RetryLoader, which loads
messages that have been persisted to disk because of previous failures.
These messages are sent to the Handler which has a configurable thread
pool and runs the main user code. The results are then optionally sent to
the RetryScheduler which will persist failed messages for a retry in the
future. Messages are finally sent to a cleanup thread that depends on
their origins (Broker, RetryLoader) where the artifacts are cleaned up
accordingly (acked, file deleted).

This does introduce some boilerplate that could be cleaned up each time
a new thread is created. Overall each section of the app is synchrounous
code which I think is significantly easier to follow then the previous
event based model.