I left in the code to be able to use the old method even when we are not deduping, in case we wanted to be able to add a hint (if there are very few dups it's  slightly more efficient).

Might be good to avoid a second compare on the dedup'ed case though