⚡ KordAI TokenizerFast

KordAI TokenizerFast is a fast Hugging Face tokenizer designed for efficient text preprocessing for Thai and English language models.

It is fully compatible with the 🤗 Transformers ecosystem and can be used for both training and inference with compatible models.

---

✨ Features

• 🚀 Fast tokenizer implementation
• 🇹🇭 Thai language support
• 🇺🇸 English language support
• 🤗 Compatible with Hugging Face Transformers
• ⚡ Efficient tokenization and decoding
• 💻 Suitable for both training and inference

---

📋 Model Details

Item	Value
Repository	KordAI/TokenizerFast
Type	Fast Tokenizer
Languages	Thai, English
Library	Transformers
License	Apache-2.0

---

🎯 Intended Use

This tokenizer is designed for:

• Thai text tokenization
• English text tokenization
• Multilingual NLP pipelines
• Dataset preprocessing
• Language model training
• Language model inference

---

🐍 Usage

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "KordAI/TokenizerFast"
)

text = "สวัสดี Hello!"

tokens = tokenizer(text)

print(tokens)

decoded = tokenizer.decode(tokens["input_ids"])
print(decoded)

---

⚠️ Notes

• This repository contains only the tokenizer.
• It should be used together with a compatible language model.
• Performance depends on the model paired with this tokenizer.

---

🙏 Acknowledgments

Special thanks to:

• Hugging Face for the Transformers ecosystem.
• KordAI for developing and distributing the tokenizer.
• The open-source AI community for advancing multilingual NLP.

---

📖 Citation

@misc{kordaitokenizerfast2026,
  title={KordAI TokenizerFast},
  author={KordAI},
  year={2026},
  publisher={Hugging Face},
  howpublished={https://huggingface.co/KordAI/TokenizerFast}
}