added comments about the checks to handle different character encodings