GitHub repository: kacperlukawski/real-wordpiece
Primary language: Python
Description: A score-based implementation of WordPiece tokenization training, compatible with HuggingFace tokenizers.