corrected tokenize implementation