tokenization-scorer (pypi): Package for evaluating text tokenizations.
Keywords: bpe|segmentation|subword|tokenization