Consolidated tokenization to one function.

#29