Only retain elements in sets for same indexes if they have equivalent reason for matching.