Model: Max1798/my-tokenizer
Author: Max1798
Tags: bpe, region:us
Card metadata: task: token-classification
widget: text: Hello, this is a test of my BPE tokenizer.
example_title: 英文分词示例, text: 我爱自然语言处理，这是我的分词器测试。
example_title: 中文分词示例
Likes: 0 | All-time downloads: 7
Created: 2025-07-17 03:16:47