Change tokenizer to use IEnumerable<char> instead of string

This will allow a custom file reader to be used