some improvements in the tokenization process