Additional improvements to the de-dupe logic for faster processing.