Updated all of the next_token code in accord with the changes in the previous commit.