profiling and fixes for new tokenize