fixed and made work the new version of tokenize