Detect duplicates and improve performance