Now tracks the tokens as a Token with a String and an index into where exactly in the complete string the token starts.