cantino/huginn — pull request #154 by Alex-Ikanow (2014-01-30 22:16:25+00:00)
Title: #135 #141 2 deduplication fixes for the website agent

Fix 1) #135 update events' expiry date on deduplication

Previously events would age out even when being de-duplicated
frequently, resulting in spurious identical events making their way to
the next agent in the pipeline. This isn't the most efficient code (same
as previously), but not worth optimizing further until we've decided
what to do with a more generic deduplication module

Fix 2) #135 set deduplication "look back" to be configurable (and
default to a large number)

I think this is preferable default behavior? Potential performance
issues should be addressed via an efficient deduplication library using
indexed DB fields