check for invalid UTF encoding