ShubhWorking/sarvam-1-smoltalk-instruct

Sarvam-1 SmolTalk Instruct

An independent, experimental English instruction fine-tune of sarvamai/sarvam-1, published by ShubhWorking. This is a full-model checkpoint, with 2,525,087,744 parameters, trained on the pinned SmolTalk all training split.

This is a community derivative, not an official Sarvam AI release. The upstream Sarvam AI Research License governs this model; commercial use requires permission from Sarvam AI. See LICENSE.md and NOTICE.txt.

What changed from the completion model?

The upstream model is documented as a text-completion base model intended for downstream fine-tuning. This checkpoint updates all its weights through supervised fine-tuning on conversations, using loss on assistant responses and the model's native [INST] ... [/INST] template. It is intended to answer user instructions through that template.

Both checkpoints remain autoregressive next-token language models. Architecture, parameter count, tokenizer vocabulary, and the configured 8,192-token context limit are unchanged. SFT changes the weights and learned response behavior. It does not add a search engine, retrieval system, memory service, or an external tool executor.

What the measurements establish

On the same 511 development-validation conversations, assistant prediction loss changed from 0.960699 to 0.516619 (46.2% lower), and teacher-forced mean token accuracy changed from 76.16% to 84.41%.

These measure fit to held-out SmolTalk responses. Token accuracy here is teacher-forced next-token accuracy, not answer correctness or task success. The validation split was used during development and model selection. It is not the official SmolTalk test split or an independent benchmark.

What the examples show

The saved outputs contain improvements and regressions:

• A rectangle-perimeter problem changes from an incorrect length of 15 cm to the correct 11 cm; the requested final-answer format is still missed.
• A negative review changes from a neutral classification to negative; the one-word constraint is still missed.
• Strict JSON-only requests can produce Python code or unsolicited tool-call text.
• A context-only age question produces an invented age after SFT, where the base declines to supply the age.

Read four verbatim paired examples, or inspect all 60 paired development outputs. The examples used greedy decoding, at most 256 new tokens, and the same chat-formatted input for both models. This controls prompt format, but does not establish the best performance attainable from a base model using raw completion prompts. These prompts were visible during development; they are not an independent test.

No IFEval, GSM8K, MATH, HumanEval, MBPP, or multilingual benchmark improvement is claimed. The evidence does not establish that this model is uniformly better than the base model.

Training

• Method: full supervised fine-tuning with TRL SFTTrainer; all parameters trainable.
• Data: HuggingFaceTB/smoltalk, configuration all, revision 5feaf2fd3ffca7c237fc38d1861bc30365d48ffa.
• Base revision: e9607337286ddf496d4a2562b194e489dcf3feea.
• Source train split: 1,043,917 conversations; 512 were reserved before training-pool construction.
• Prepared training conversations: 1,041,705; conversations were deduplicated against held-out hashes and within the training candidates.
• Validation: 511 of the 512 held-out conversations had scoreable assistant targets after truncation.
• Packing: 127,241 blocks of 8,192 tokens, prepared before passing to the trainer. Conversations can span block boundaries; normal causal attention can cross packed conversation boundaries.
• Objective: precomputed assistant masks applied by the collator, with non-assistant labels ignored. assistant_only_loss=False in the high-level trainer config reflects the pretokenized schema; it does not mean user tokens contributed to the loss.
• Schedule: nominal two epochs, capped at 15,904 optimizer updates (reported epoch approximately 1.9998).
• Effective global batch: 16 packed sequences, microbatch 2, accumulation 8, one GPU.
• Learning rate: 1e-4; cosine decay; 4% warmup.
• Optimizer: fused AdamW; weight decay 0; gradient clipping 1.0.
• Precision: BF16 autocast with FP32 trainable parameters. Released weights are FP32 safetensors, approximately 10.11 GB / 9.41 GiB. They can be loaded in BF16 for inference on supporting hardware.
• Seed: 44 for the final training run; data split/shuffle seed: 42.
• Hardware: one NVIDIA H200 NVL per allocation. The first 24-hour job timed out; a continuation restored checkpoint 10,752, including optimizer, scheduler and RNG state. Some updates beyond that saved checkpoint were repeated.
• Training stack: PyTorch 2.9.1+cu128, Transformers 4.57.6, TRL 0.24.0, PEFT 0.17.1.

The earlier LoRA/full-training pilot favored full tuning at 1e-4 on the matched-budget confirmation runs. This is configuration-selection evidence under those budgets, not a general claim that full tuning is always faster or better than LoRA.

See parameters, data manifest, validation row hashes, metrics, and the training source. The source records the original cached-token preparation path; regenerating training data requires the pinned raw dataset plus the same tokenization/mask preparation, or the existing row-aligned cache. Training data and optimizer state are not distributed in this inference repository.

Usage

Use the saved chat template. A raw user string is not the training prompt format. This example uses greedy decoding and explicitly enables the inference KV cache.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "ShubhWorking/sarvam-1-smoltalk-instruct"
device = "cuda" if torch.cuda.is_available() else (
    "xpu" if hasattr(torch, "xpu") and torch.xpu.is_available() else "cpu"
)
dtype = torch.bfloat16 if device != "cpu" else torch.float32
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=dtype, device_map={"": device}, low_cpu_mem_usage=True
).eval()
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain photosynthesis in simple English."}],
    tokenize=True, add_generation_prompt=True, return_dict=True,
    return_tensors="pt",
)
inputs = {key: value.to(device) for key, value in inputs.items()}
with torch.inference_mode():
    output = model.generate(
        **inputs, max_new_tokens=128, do_sample=False, use_cache=True,
        pad_token_id=tokenizer.eos_token_id,
    )
print(tokenizer.decode(output[0, inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

device_map requires Accelerate. CUDA/XPU inference requires a compatible PyTorch build and working device drivers; choose a dtype supported by your device. BF16 weights need roughly 5.1 GB before KV-cache/runtime overhead. The integrated Intel GPU uses shared system memory.

Limitations and intended use

Use this checkpoint for non-commercial research into SFT, response behavior, and model evaluation. English is the post-training focus. Retention of the base model's Indic-language capabilities has not been measured. Tool-call-looking text is generated text, not a validated tool integration. The model can hallucinate, ignore supplied context, violate formatting constraints, produce incorrect code/math, and generate unsafe or biased material. No comprehensive safety alignment or reasoning-specific post-training was performed. SmolTalk includes math and coding sources; this release should not be described as a separately trained reasoning model.

Data and license attribution

The base model is by Sarvam AI. SmolTalk is by HuggingFaceTB and combines newly generated sources with imported public datasets. Its newly generated components use Apache 2.0; imported components retain their source terms. This model is not Apache-2.0 licensed: the base model's Sarvam AI Research License applies. Dataset inclusion does not establish blanket commercial-use permission. Source names and counts are recorded in the data manifest. Upstream dataset documentation contains the source-license links.

Relevant resources:

• Sarvam-1 model card
• Sarvam AI Research License
• SmolTalk dataset card
• SmolLM2: When Smol Goes Big (2025)
• TRL SFTTrainer

Artifact provenance

The model weights are unchanged from the verified final training artifact. The release config sets use_cache=true for inference; it was false in the training artifact. Assistant-mask generation markers in the template do not change its rendered conversation text. The repository contains deployment weights and selected evidence, excluding optimizer tensors, cluster job logs, local credentials and dataset caches.