refactor the duplicate detection codes