Enjoy — We extend the JiRack Models Ecosystem! 🚀
JiRack Utra Pro Tokenizer 512
JiRack Utra Pro Tokenizer - 347 active language editions of Wikipedia
High-performance production-grade Byte-Level BPE tokenizer developed as part of the JiRack Ternary Models ecosystem.
This is the Ultra Pro version designed for maximum quality, advanced compression vs 256k Ultra version for very large models , and precision in complex real-world applications.
- JiRackTernary_1b model https://huggingface.co/kgrabko/JiRackTernary_1b
Open Robot platform
- Tiangong : https://english.www.gov.cn/english.www.gov.cn/news/202411/13/content_WS673406e2c6d0868f4e8ece33.html
- Unitree g1 https://a.co/d/0e4A8YVc
- LimX Oli https://www.limxdynamics.com/en/products/oli?channel=option_google_advertising__c-
- ubtrobot https://www.ubtrobot.com/en/
- x-humanoid https://www.x-humanoid.com/detail/hskw.html
Key Features
- Algorithm: Byte-Level BPE
- Vocabulary Size: 524,288 tokens — excellent balance between precision and efficiency
- Multilingual & Technical Strength: Optimized for English, Russian, code, scientific literature, and technical documentation
- Domain Specialization: Strong performance on programming languages, engineering, robotics, and scientific texts
Special Tokens Support
- Full ChatML dialogue format (
<|im_start|>,<|im_end|>) - FIM (Fill-in-the-Middle) support for code generation
- Rich set of domain routing tokens (
__CODING__,__PYTHON__,__ROBOTICS__,__SCIENCE__, etc.) - Extended robotics and control tokens
CMS Manhattan Service & Support
- Jirack patent guards your technology for competitors
- Redesign Llama , Qwen , Gemma to Ternary model
- Re-tain and replace embeddings for Llama , Qwen , Gemma to extend langeages to 347
- Accelerate inference via high compression 512K tokenizer and replace multiplication with sum operations via Ternary weights
Usage
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("CMSManhattan/JiRack-UltraPro-Tokenizer-512K")
text = "__CODING__ __PYTHON__ Write a merge sort function in Python."
tokens = tokenizer.tokenize(text)
token_ids = tokenizer.encode(text)
print("Tokens:", tokens)
print("Token IDs:", token_ids)
Vocab size: 524288
pad_token_id: 2
eos_token_id: 1
JiRack Pretrain Dataset
https://huggingface.co/datasets/CMSManhattan/JiRack-Pretrain-Dataset
python train_jirack_accelerate.py
Processing jirack_pretrain_chunk_0.pt: 27%|█████████████████████▋ | 268/1000 [22:02:45<60:12:33, 296.11s/it, loss=6.3145, avg_loss=7.0132, ppl=1111.16]
Processing jirack_pretrain_chunk_0.pt: 87%|███████████████████████████████████████████████████████████████████████ | 866/1000 [75:57:47<13:13:29, 355.29s/it, loss=2.8616, avg_loss=5.9877, ppl=398.52]
Benchmark for tokens quality .
=== Text after ChatML Template ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Write a merge sort function in Python.<|im_end|>
=== Tokens (IDs) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 31576, 522, 66472, 6176, 2037, 576, 7637, 269, 4, 454]
=== Decoding Token by Token ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
31576 -> ' Write'
522 -> ' a'
66472 -> ' merge'
6176 -> ' sort'
2037 -> ' function'
576 -> ' in'
7637 -> ' Python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
=== ChatML 模板处理后的文本 ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ 用 Python 写一个归并排序函数。<|im_end|>
=== Token (ID) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 196893, 7637, 476, 24410, 16482, 61950, 14333, 295880, 92252, 870, 4, 454]
=== 逐个 Token 解码 ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
196893 -> ' 用'
7637 -> ' Python'
476 -> ' '
24410 -> '写'
16482 -> '一个'
61950 -> '归'
14333 -> '并'
295880 -> '排序'
92252 -> '函数'
870 -> '。'
4 -> '<|im_end|>'
454 -> '
'
== Texte après le modèle ChatML ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Écris une fonction de tri fusion en Python.<|im_end|>
=== Tokens (IDs) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 170797, 3433, 4484, 56203, 595, 3102, 34759, 720, 7637, 269, 4, 454]
=== Décodage token par token ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
170797 -> ' Éc'
3433 -> 'ris'
4484 -> ' une'
56203 -> ' fonction'
595 -> ' de'
3102 -> ' tri'
34759 -> ' fusion'
720 -> ' en'
7637 -> ' Python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
=== Text nach ChatML-Template ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Schreibe eine Merge-Sort-Funktion in Python.<|im_end|>
=== Token (IDs) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 115144, 18528, 6806, 256728, 268, 144869, 268, 386798, 592, 576, 7637, 269, 4, 454]
=== Dekodierung Token für Token ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
115144 -> ' Schre'
18528 -> 'ibe'
6806 -> ' eine'
256728 -> ' Merge'
268 -> '-'
144869 -> 'Sort'
268 -> '-'
386798 -> 'Funkt'
592 -> 'ion'
576 -> ' in'
7637 -> ' Python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
=== النص بعد تطبيق قالب ChatML ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ اكتب دالة فرز بالدمج (merge sort) بلغة بايثون.<|im_end|>
=== الرموز (IDs) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 33139, 9120, 360514, 209685, 6513, 5867, 1187, 580, 121802, 6176, 264, 474504, 84721, 2518, 1435, 269, 4, 454]
=== فك الترميز رمزا برمز ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
33139 -> ' اك'
9120 -> 'تب'
360514 -> ' دالة'
209685 -> ' فرز'
6513 -> ' بال'
5867 -> 'دم'
1187 -> 'ج'
580 -> ' ('
121802 -> 'merge'
6176 -> ' sort'
264 -> ')'
474504 -> ' بلغة'
84721 -> ' باي'
2518 -> 'ث'
1435 -> 'ون'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
=== Текст после ChatML шаблона ===
<|im_start|>system
You are a precise router model.<|im_end|>
<|im_start|>user
__CODING__ __PYTHON__ Напиши функцию сортировки слиянием на python.<|im_end|>
=== Токены (ID) ===
[5, 454, 3285, 934, 522, 21133, 112585, 6457, 269, 4, 454, 6, 454, 73, 476, 88, 35013, 91654, 492868, 430356, 174599, 302515, 467815, 874, 66017, 269, 4, 454]
=== Декодирование по токенам ===
5 -> '<|im_start|>system'
454 -> '
'
3285 -> 'You'
934 -> ' are'
522 -> ' a'
21133 -> ' precise'
112585 -> ' router'
6457 -> ' model'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
6 -> '<|im_start|>user'
454 -> '
'
73 -> '__CODING__'
476 -> ' '
88 -> '__PYTHON__'
35013 -> ' Нап'
91654 -> 'иши'
492868 -> ' функцию'
430356 -> ' сорти'
174599 -> 'ровки'
302515 -> ' слия'
467815 -> 'нием'
874 -> ' на'
66017 -> ' python'
269 -> '.'
4 -> '<|im_end|>'
454 -> '
'
📧 Contact & Licensing
For joint ventures, hardware integration, or licensing inquiries:
- Email: grabko@cmsmanhattan.com
- Phone: +1 (516) 777-0945
- Location: New York, USA