improved tokenize functionality in processor