RuVI-AI/Pragya-EXP3-CHAT

Pragya · EXP3 Chat

A 28.5M-parameter chat model, fine-tuned from Pragya EXP2. For testing, not for production.

⚠️ Not a product

This is a small experimental chat model. It is not aligned, factually reliable, or safe. Its replies are short and often wrong.

---

Summary

Base model	Pragya EXP2
Fine-tuning	Supervised, single-turn chat, completion-only loss
Total parameters	28.5M
Context	1024 tokens
Vocabulary	12,288 byte-level BPE
Reply end marker	*end*
Intelligence Index	14.21 (base EXP2: 12.41)

---

What changed from EXP2

EXP2 is a base model: it continues text but has no way to signal the end of a reply. EXP3 Chat is fine-tuned on single-turn conversations, with each reply closed by the marker *end*. The model learns the marker during fine-tuning, so replies can be cut at it.

The architecture, tokenizer and weights are unchanged, apart from the fine-tuning updates.

---

Fine-tuning

Data	HuggingFaceTB/smol-smoltalk, first 20,000 examples (15,873 kept after length filter)
Format	User: <question>\nAssistant: <reply> *end*
Loss	Reply tokens only (prompt tokens masked)
Loss target	Next-token, shifted by one position
Epochs	1
Learning rate	1e-4, cosine decay, 38 warm-up steps
Batch	1 sequence × 16 gradient accumulation
Max length	512 tokens
Final training loss	about 1.5 (from 2.87 at step 50)
Precision	FP32

Training used batch size 1 because the model rejects padded batches. Gradient checkpointing is off, as the model does not support it.

---

Results

BananaMind Base Bench 1.1 (official runner)

Overall Elo 976. Raw accuracy 174/350 (49.71%), weighted accuracy 44.89%, against 25% chance.

Category	EXP2 base	EXP3 Chat
Language completion	92%	88%
Commonsense	56%	56%
World knowledge	56%	62%
Context tracking	30%	42%
Quantitative	18%	20%
Logical reasoning	46%	46%
Code completion	56%	34%

Context tracking and world knowledge improved. Code completion dropped by 22 points, which fits chat data containing little code.

lm-evaluation-harness (0-shot, acc_norm)

Benchmark	EXP2 base	EXP3 Chat	Chance
HellaSwag	29.76%	29.82%	25%
ARC-Easy	31.57%	38.85%	25%
ARC-Challenge	23.98%	23.46%	25%
PIQA	61.86%	63.06%	50%
ArithMark-3	38.30%	37.80%	25%

Intelligence Index: 14.21 (base 12.41), using the same formula as before:

N(s, c) = 100 × (s − c) ÷ (100 − c)
Index = (N(HellaSwag, 25) + N(ARC, 25) + N(PIQA, 50) + 0.65 × N(ArithMark-3, 25)) ÷ 3.65

The ARC-Easy gain drives most of the index change. Treat the index change as suggestive until the embedding check above is resolved.

AxiomicLabs reasoning benchmarks

Benchmark	EXP2 acc / acc_norm	EXP3 Chat acc / acc_norm	Items
LogicMark	42.74% / 42.80%	43.88% / 44.18%	5,000
Tiny Theory of Mind	35.60% / 31.85%	36.25% / 32.80%	2,000

Report acc_norm for Tiny Theory of Mind, since its card specifies length-normalised scoring.

---

Quick Architecture

## Quick start

pip install torch transformers safetensors xformers

The model loads through transformers with remote code. Generate one prompt at a time, since padded batches are not supported.

from transformers import AutoTokenizer, AutoModelForCausalLM

repo = "RuVI-AI/Pragya-EXP3-Chat"   # or a local folder
tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True).to("cuda:0").eval()

def chat(question, max_new_tokens=200):
    prompt = f"User: {question.strip()}\nAssistant:"
    enc = tok(prompt, return_tensors="pt").to("cuda:0")
    out = model.generate(
        **enc,
        max_new_tokens=max_new_tokens,
        do_sample=True,
        temperature=0.7,
        top_k=12,
        top_p=0.95,
        repetition_penalty=1.3,
        no_repeat_ngram_size=3,
        stop_strings=["*end*", "User:"],
        tokenizer=tok,
        pad_token_id=tok.pad_token_id,
    )
    text = tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True)
    return text.split("*end*")[0].split("User:")[0].strip()

print(chat("What is Internet?"))

Replies end at *end* when the model writes it. If the model doesn't write the marker, generation stops at max_new_tokens and the reply is cut at the first User:.

---

Sampling

These settings gave fluent replies in testing:

temperature        0.7
top_k              12
top_p              0.95
repetition_penalty 1.3
no_repeat_ngram    3

Greedy decoding repeats phrases, so use sampling.

---

Capabilities and limitations

It can: answer simple single-turn questions briefly, and close most replies with *end*.

It cannot: reliably answer factual questions, handle multi-turn conversations, write useful code, use context longer than 1024 tokens, or handle non-English input.

• Small scale. Replies are short and often factually wrong.
• Single-turn training only. Multi-turn context is not trained.
• No safety filtering. Replies can be biased, false or toxic.
• EOS is not learned. The end-of-sequence token is still unused. The *end* marker is the stop signal.

---

License

Apache 2.0. The fine-tuning data (HuggingFaceTB/smol-smoltalk) and the base model's training data carry their own licences. Check upstream terms before redistributing derivatives.

---

Citation

@misc{pragya-exp3-chat,
  title  = {Pragya EXP3 Chat: a 28.5M chat model fine-tuned from Pragya EXP2},
  author = {Arush},
  year   = {2026},
  note   = {Experimental checkpoint, not for production use},
  howpublished = {\url{https://huggingface.co/RuVI-AI/Pragya-EXP3-Chat}}
}