correct IO encoding handling
overhauled tokenization with optional normalization