alphagov/support — pull request #144 by benilovj (2014-03-03 13:52:55+00:00)
Title: Anonymous feedback de-duplication batch processing

## Why is this needed?

We're seeing a considerable (~10%) rate of duplicates in the various anonymous feedback submissions (such as service feedback, problem reports, etc) across the site. Data analysis suggests that these duplicates are caused by duplicate submissions from specific browsers (IE9 and above) but we haven't been able to replicate and fix these issues in the front-end up to now.

Duplication has been manageable up to now - since problem reports are handled manually, dupes are just filtered out. However, in order to be able to aggregate service feedback stats, the data quality needs to be cleaned up, and we've run out of time to fix it in the front-end. This change introduces a daily cron job which processes the previous day's feedback submissions and "hides" the dupes.

This is hopefully a temporary solution until we can fix the frontend and stop these duplicates entering the system in the first place. When that happens, we can remove this.

## Technical details

De-duplication is done by:
- fetching all the records from the database in creation order
- iterating through each one
- looking for the same record being created multiple times within a 5 second period.

The `DuplicateDetector` class keeps a rolling window of unique records for performance reasons.

@bradleywright suggested trying to make this more performant by running the uniqueness logic in the database; however I'm not convinced that this is possible without [windowing functions](http://docs.oracle.com/cd/E11882_01/server.112/e26088/functions004.htm#SQLRF06174) (that MySQL doesn't provide, to my knowledge).