jarvis-138m
jarvis-138m is a compact, decoder-only causal language model focused primarily on Russian text and dialogue. Its architecture, tokenizer, data pipeline, training loop and local generation code are implemented in lubenkov/jarvis.
The repository contains the source code; this Hugging Face repository contains the fine-tuned PyTorch checkpoint and the tokenizer required to use it. The checkpoint is intended for local experimentation, learning and research.
Model details
| Property | Value |
|---|---|
| Model type | Autoregressive decoder-only Transformer |
| Primary language | Russian (ru) |
| Parameter count | Approximately 138.4M (113.3M without embeddings) for the main preset |
| Layers | 16 |
| Attention heads | 12 |
| Hidden size | 768 |
| Feed-forward size | 2,048 (SwiGLU) |
| Context length | 1,024 tokens |
| Vocabulary | 32,768-token byte-level BPE (NFC, per-digit splitting, <|system|>) |
| Position encoding | RoPE |
| Normalization | RMSNorm |
| Embeddings | tied input/output |
| Initialisation | depth-scaled residual projections |
| Checkpoint | jarvis-138m-sft-fp16.pt |
| Tokenizer | tokenizer.json |
The checkpoint is a PyTorch .pt file saved by the project. It stores the model weights and configuration; it is not a transformers AutoModelForCausalLM checkpoint. Use the custom model code in the GitHub repository to load it. The tokenizer and checkpoint must be kept together.
Intended use
- Local experimentation with a small Russian-language causal language model.
- Educational use: inspecting a Transformer implementation and a full pretraining/SFT pipeline.
- Generating short Russian text or trying a lightweight conversational model.
Out-of-scope use
Do not use this model as the sole basis for medical, legal, financial, employment, educational or other high-impact decisions. It has not been evaluated for high-stakes use, robust instruction following, factual reliability, privacy, or safety. Do not rely on it to provide professional advice, and do not send sensitive personal or confidential information to a deployment that you do not control.
How to use
The easiest supported inference path is the repository's chat.py script. Install a PyTorch build suitable for your machine (use the official PyTorch selector for CUDA), then clone the code and install its requirements:
git clone --branch v2.0.0 --depth 1 https://github.com/lubenkov/jarvis.git
cd jarvis
python -m venv .venv
# Activate the environment for your shell, then:
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip install --upgrade huggingface_hub
Download the checkpoint and matching tokenizer into the repository's ignored checkpoints/ directory:
python - <<'PY'
from huggingface_hub import snapshot_download
snapshot_download(
repo_id="Eragon1/jarvis-138m",
local_dir="checkpoints/hf-jarvis-138m",
allow_patterns=["jarvis-138m-sft-fp16.pt", "tokenizer.json"],
)
PY
Start a local chat:
python chat.py \
--ckpt checkpoints/hf-jarvis-138m/jarvis-138m-sft-fp16.pt \
--tokenizer checkpoints/hf-jarvis-138m/tokenizer.json \
--system "Ты — Jarvis, дружелюбный русскоязычный помощник."
The system prompt is optional and never hardcoded: pass --system "..." or set JARVIS_SYSTEM_PROMPT. Without it the model is prompted as a plain dialogue; about half of the fine-tuning dialogues carried a system prompt, so both paths work.
For one prompt, use generate.py:
python generate.py \
--ckpt checkpoints/hf-jarvis-138m/jarvis-138m-sft-fp16.pt \
--tokenizer checkpoints/hf-jarvis-138m/tokenizer.json \
--prompt "Привет! Расскажи короткую историю."
The scripts select CUDA when available and otherwise run on CPU. Generation supports temperature, top-k/top-p sampling and a repetition penalty; see python chat.py --help and python generate.py --help for options. The model is not currently deployed by an Inference Provider; inference is local through the project code.
Training data and process
The training scripts load source datasets from Hugging Face. The repository does not include the raw corpora.
| Training source | Dataset(s) |
|---|---|
| Russian web text | HuggingFaceFW/fineweb-2, rus_Cyrl/train, shards sampled across the whole dataset |
| Russian articles | IlyaGusev/gazeta, train split |
| Russian encyclopedia | wikimedia/wikipedia, 20231101.ru |
| Dialogue / instructions | IlyaGusev/saiga_scored, Vikhrmodels/GrandMaster-PRO-MAX, Vikhrmodels/Veles-2.5, Den4ikAI/russian_instructions_2, inkoziev/jokes_dialogues, and the Russian subset of OpenAssistant/oasst1 |
The training recipe in model/config.py at the v2.0.0 tag is two-stage:
| Phase | Configured token budget | Sampling mixture | Objective |
|---|---|---|---|
| Pretraining | 3.2B tokens | 88% web, 5% chat, 4% wiki, 3% articles | Next-token prediction |
| SFT | 200M tokens | 95% chat, 5% web | Assistant-response loss for chat examples |
Training ran on two T4 GPUs with fp16 autocast and DDP: micro-batch 8 × gradient accumulation 5 × 2 devices × 1,024 tokens, that is 81,920 tokens per optimizer step, at roughly 20k tokens/s for about 45 GPU-hours in total. Both phases use a cosine schedule with warmup (peak learning rate 6e-4 for pretraining, 3e-4 for SFT).
The preparation pipeline applies source-specific filters, normalizes chat roles, removes duplicate examples and filters for Cyrillic content. The web/wiki/news part is streamed: prepare_stream.py filters and tokenizes in one pass and writes only the uint16 shards, which keeps a multi-billion-token corpus within a 20 GB disk budget. Tokenized validation data is held out per source.
Reproducibility note: the published inference checkpoint contains model weights and configuration, but not the full training log, exact processed-token count, hardware details or dataset revision hashes. The budgets above describe the training configuration used for this checkpoint; they should not be treated as an independently audited count. The preparation script does not pin source dataset revisions, so a later rebuild may use updated source data.
Evaluation
The repository reports bits per character (BPC; lower is better) on a 600,000-character held-out sample from each source, scored with 768-token windows. These are small, domain-specific language-modeling measurements, not a comprehensive evaluation of generated answers.
| Model | web_ru |
texts_ru |
chat_ru |
|---|---|---|---|
| jarvis v2 SFT checkpoint | 1.08 | 1.10 | 0.88 |
| jarvis v2 pretraining checkpoint | 1.02 | 1.01 | 0.91 |
| jarvis v1 SFT checkpoint | 1.14 | 1.14 | 0.92 |
| gzip / xz | 4.07 / 3.08 | 4.19 / 3.03 | 3.57 / 2.68 |
All three checkpoints were scored by the same benchmark on byte-identical text, so the rows are comparable; the v1 release reported 1.22 / 1.07 / 0.91 on its own sample, which is not comparable with this table. Held-out perplexity at the end of pretraining was 16.6 (web_ru), 11.5 (chat_ru), 17.3 (wiki_ru) and 18.6 (texts_ru); after SFT it is 7.6 on dialogue. The full measurements are available in results/bpc_v2.json; evaluation code is in tools/bench_bpc.py.
Limitations and risks
- Small model: at approximately 138M parameters, it has limited capacity compared with modern large language models. It may be incoherent, repetitive or unable to follow multi-step instructions reliably.
- Factuality: generated text can be wrong or fabricated. The model does not provide sources or calibrated confidence, and arithmetic and factual questions remain unreliable.
- Language coverage: training is focused primarily on Russian. Performance in other languages is not established.
- Safety and bias: the model has not undergone a comprehensive safety evaluation. Web and conversational corpora can contain bias, personal information or unsafe material, which may be reflected in outputs.
- Data provenance: source dataset licenses and terms differ. Review each dataset card and its license before use or redistribution. No raw training data is included with the checkpoint.
- Context: the configured context window is 1,024 tokens; long prompts and conversations are truncated by the local chat code.
License
The model repository declares the checkpoint under Apache-2.0, matching the source-code license in lubenkov/jarvis. This declaration does not replace the terms of the upstream training datasets. Users are responsible for reviewing and complying with those source terms.
Citation
If you use or discuss this model, please cite the project repository:
lubenkov. jarvis: GPT-style transformer written from scratch in PyTorch. v2.0.0, 2026.
https://github.com/lubenkov/jarvis/releases/tag/v2.0.0
🇷🇺 По-русски
jarvis-138m — небольшая авторегрессионная Transformer-модель примерно на 138 млн параметров, ориентированная прежде всего на русский текст и диалоги. Веса лежат в jarvis-138m-sft-fp16.pt, совместимый токенизатор — в tokenizer.json. Для запуска используется код из lubenkov/jarvis; формат весов не совместим напрямую с transformers.AutoModelForCausalLM.
Модель подходит для локальных экспериментов и обучения на примере. Она может ошибаться, повторяться и выдавать небезопасный или предвзятый текст; не используйте её для важных решений и не отправляйте ей конфиденциальные данные. Опубликованная конфигурация обучения задаёт 3,2 млрд токенов pretraining и 200 млн токенов SFT на двух T4.
Отличие от первой версии: 16 слоёв вместо 14, контекст 1024 вместо 768, словарь 32 768 вместо 16 384, опциональный системный промпт <|system|> и в 2,5 раза больше данных претрейна. По BPC модель лучше v1 на всех источниках.