Duplicate detection, character encoding checking