Performance improvement for removing duplicates. Still slow to populate the DB.