Avoid many redundant tests
Handle custom tokenization correctly when at EOS