Allowing plain tokens constructed by lexer to also work.  Thanks to mye on IRC for pointing this omission.