Nuri
Collection
Nuri research model family: dense and MoE encoders, instruction-tuned masked generators, sentence embeddings and a shared tokenizer. • 6 items • Updated
Nuri tokenizer
The shared 65,536-token byte-level BPE tokenizer for the Nuri model family. Korean and English are primary; some Japanese is supported. It is a tokenizer, not a standalone language model.
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("alice-noa-chan/nuri-tokenizer")
ids = tokenizer("Hello, Nuri.", add_special_tokens=False).input_ids
text = tokenizer.apply_chat_template(
[{"role": "user", "content": "Hello."}],
tokenize=False, add_generation_prompt=True,
)
For IT prompts, apply the chat template and tokenize the rendered string with add_special_tokens=False. It ends with the assistant tag; the answer goes on a separate masked canvas. Each model repository includes the same tokenizer files, so a separate download is optional.
| Model | Parameters | Purpose |
|---|---|---|
| nuri-base | 302.3M | Pretraining; explicit generation canvas length |
| nuri-base-it | 302.3M | Instruction tuning; predicted generation length |
| nuri-base-moe | 527.4M total /302.5M active | Pretraining; explicit generation canvas length |
| nuri-base-moe-it | 527.4M total /302.5M active | Instruction tuning; predicted generation length |
| nuri-base-embed | 302.3M | Contrastive sentence embeddings |
| nuri-tokenizer | 65,536 tokens | Shared byte-level BPE tokenizer |
License
MIT. Training data sources are not disclosed or distributed.
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support