At [Quantifind](http://www.quantifind.com), we use Kafka across our data processing pipeline. Kafka is a distributed message queuing system and, with a simple setup, it provides us with durability and fault-tolerance. In our proessing pipeline, we need to be sure we are not leaking data and that every message sent from one component to the other will be received. This is why we like Kafka, as it guarantees that consumers will see each published message _at least once_.

However, as we'll see, while Kafka keeps this guarantee, if the consumer doesn't interact with the queue properly, messages can easily be dropped in some cases (in particular when the application fails).