Finished the tokenizer basic logic, excluding json, regex and the creation of tokens itself.