update duplicate detection to be more detailed