TheClimateCorporation/mandoline — pull request #13 by leon-barrett (2014-12-03 06:24:40+00:00)
Title: Dedup chunks

At present, writing chunks may cause the same chunk to be written many times, for instance if the data is full of zeros. This can cause heavy throttling with the DynamoDB backend, since all those writes hit the same machines in DynamoDB.

This change adds deduplication of chunks; once a chunk is written, mandoline will avoid writing it again.