too. One option is to retain a buffer of messages that will only be transmitted once the input message has been successfully processed. Keeping your development team sane(ish) While the above topics have focussed on data processing performance and integrity, it's also important to consider developmental requirements. While your pipeline may be fast and scalable, it should also be easy to develop and maintain. Here are a few suggestions for how to make life easier: Logs everywhere. We try to log every significant event, often as informational messages but sometimes as debug messages for low level events. The combination of thorough logging with a log collection service such as greylog2 makes tracking and problem determination easier. Use a task management framework. The AWS SDK for Java makes it easy to interact with AWS services, but it doesn't provide a framework for task management or message processing. We developed a simple framework for pipeline tasks which handles all SQS related activity such as reading, writing, and deleting from queues. Having a single, unified abstraction for task processing makes it easy to apply wide-reaching, fundamental changes to your core aspects of your pipeline. Some languages (such as Erlang) provide this for free, though choosing a less mainstream language often means integration with third party software will be harder. Start with a zero-tolerance approach to exceptions - only add more lenient exception handling when your system is running reliably. Most exceptions in SureChemOpen are re-thrown as a runtime exceptions, causing message processing failure.