deduplicate scored_pairs as we go