CruGlobal/mpdx — pull request #218 by draffensperger (2014-12-30 02:29:52+00:00)
Title: MPDX-186 Improve Find Duplicates

As per the JIRA ticket, when people import data from a variety of sources like Tnt, Facebook, donor services, etc. there can be a lot of duplicate people and contacts.

This improves the find duplicates feature to be a two step process where first you merge duplicate contacts:

![screenshot-1419889455](https://cloud.githubusercontent.com/assets/2855507/5575167/3e030120-8fa0-11e4-8744-2ba73eadcfef.png)

Then when there are no duplicate contacts left to merge, it prompts you to merge duplicate people within contacts:

![screenshot-1419889569](https://cloud.githubusercontent.com/assets/2855507/5575168/3fa9551a-8fa0-11e4-9783-ae340c6b6677.png)

I rewrote the code to detect duplicates and added a lot more logic to it. Specifically, it will now match by more methods like parts of a name, initials, middle names, emails, phone and address. I also pulled in a nicknames table to match on nickname, and a table of male/female ratios of first names to help identify cases where we are trying to match a man and a woman who have the same contact info (usually spouses). The gender field itself can often be wrong, so that data is a big help. I also made it do some tracking of people merges so we can add more nicknames over time based on what people are actually merging.

I use SQL queries and temporary tables which were much faster than corresponding Ruby code to compare everything outside of the database. Not all the commit messages are relevant because I had some issues with juggling different branches, but the overall changes are what I intended.

I will have Stacie review it in stage, but comments about the code and the feature itself are welcome.