natural language tokenization using various types of tokenizer, including a state function lexer, unicode matcher, and basic white space splitter