GitHub repository: Hk669/bpetokenizer
Primary language: Jupyter Notebook
Created: 2024-05-26T19:02:37Z
Description: (py package) train your own tokenizer based on BPE algorithm for the LLMs (supports the regex pattern and special tokens)