Fix a bug aborting tokenizing unexpectedly.