Benchmark for preprocess function
preprocess function in one pass
Reduce match branches
Remove redundant UTF-8 check
Simplify match block removing code for multi-byte chars
Merge pull request #66 from ayosec/faster-preprocess

Function tokenizer::preprocess 3x faster