cantino/huginn — pull request #627 by dsander (2014-11-08 22:31:59+00:00)
Title: Added DeDuplicationAgent

A simple approach that should work for #600 and #555, even though it will be get slow if you want to de-duplicate against every event ever checked, as the checked properties (or the CRC32 hash of the payload) are stored in the agents memory.

My initial plan to use postgresql's JSON type kind of conflicts with liquid interpolation. We can not really be sure the value is just a 'JSONPath' to the property, the user could also use a liquid filter which has to be interpolated and thus can not be compared to the values in the database directly.

Maybe using a [bloomfilter](http://en.wikipedia.org/wiki/Bloom_filter) would be an option for de-duplicating a _lot_ (10,000+) events.