Reworked the default tokenizer to work in a single loop instead of using split operations.