Complete implementation of tokenize().