YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
Vietnamese GPT-2 Tokenizer
A ByteLevel BPE tokenizer trained on the wheevu/ct219-vietnamese-raw-400k Vietnamese corpus.
Configuration
- Model: BPE
- Vocabulary size: 30,000
- Pre-tokenizer: ByteLevel
- Decoder: ByteLevel
- Minimum frequency: 2
- Language: Vietnamese
Purpose
This tokenizer is designed to improve Vietnamese text tokenization for GPT-2 fine-tuning.
The original GPT-2 tokenizer produces inefficient tokenization for Vietnamese text because GPT-2 was originally trained primarily on non-Vietnamese text.
This tokenizer is trained specifically on Vietnamese text.
Training data
Dataset: wheevu/ct219-vietnamese-raw-400k
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support