SheikhMinds/rudushi-1-1.5b

Rudushi-1 (1.5B)

Rudushi-1 is the first open, Bangla-first instruction model from the Rudushi initiative. It is a QLoRA fine-tune of Qwen/Qwen2.5-1.5B-Instruct trained to answer with structured steps (ধাপ ১ / ধাপ ২ / চূড়ান্ত উত্তর) and to check its own work (খসড়া → যাচাই → সংশোধন).

This repository contains merged fp16 full weights.

🪷 Demo	SheikhMinds/rudushi-1-1.5b-demo
📚 Training data	SheikhMinds/rudushi-bangla-sft
🧪 Checkpoint	rudushi-1-reasoning · 800 steps · merged fp16 full weights

⚠️ Research checkpoint. This is a 1.5B model trained on a single T4. It is useful for Bangla instruction following and for demonstrating structured reasoning, but it is not a general reasoner and it can be confidently wrong. Verify facts, numbers and dates.

Table of Contents

1. Model Summary
2. Quickstart
3. Answer modes
4. Training
5. Evaluation
6. Limitations
7. বাংলায় সংক্ষেপে
8. Jessore dialect
9. License
10. Citation

Model Summary

• Type: causal decoder-only language model (instruction-tuned)
• Architecture: Qwen2.5 — RoPE, SwiGLU, RMSNorm, grouped-query attention (12 query / 2 KV heads), 28 layers
• Parameters: 1.54B total (1.31B non-embedding)
• Base context length: 32,768 tokens (this fine-tune was trained at 512 tokens)
• Languages: Bengali (primary), English
• Release format: sharded safetensors (1 GB shards), fp16
• Fine-tuning: 4-bit NF4 QLoRA, fp16 compute, assistant-only loss

What it is trained to do

capability	how it is trained	honest scope
Bangla instruction following	general Bangla SFT mix	good for common tasks, not expert-level
Structured multi-step answers	synthetic chain-of-thought data	bounded arithmetic/logic patterns
Self-checking	verification/correction traces	format learned; not a correctness guarantee
Jessore dialect	20 draft seed pairs, separate config	not trained in this checkpoint

Quickstart

pip install "transformers>=4.45" torch accelerate safetensors

from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "SheikhMinds/rudushi-1-1.5b"
model = AutoModelForCausalLM.from_pretrained(repo, device_map="auto", torch_dtype="auto")
tok = AutoTokenizer.from_pretrained(repo)

messages = [
    {"role": "user", "content": "একটি দোকানে ৩টি কলমের দাম ২১ টাকা। ৪টির দাম কত? ধাপে ধাপে উত্তর দাও।"}
]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt").to(model.device)

out = model.generate(
    **inputs,
    max_new_tokens=256,
    do_sample=False,
    pad_token_id=tok.pad_token_id,
    eos_token_id=tok.eos_token_id,
)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Expected shape (not guaranteed):

ধাপ ১: একটির দাম = ২১ ÷ ৩ = ৭ টাকা।
ধাপ ২: ৪টির দাম = ৭ × ৪ = ২৮ টাকা।
চূড়ান্ত উত্তর: ২৮ টাকা।

The snippet above loads full weights. If this repository is an adapter-only release, load Qwen/Qwen2.5-1.5B-Instruct first and wrap it with PeftModel.from_pretrained(model, "SheikhMinds/rudushi-1-1.5b").

Answer modes

The demo Space exposes three modes, and you can reproduce them in code:

mode	prompt	behaviour
Direct	question as-is	short answer, no steps
Step-by-step	prepend a request for ধাপ ১ / ধাপ ২ / চূড়ান্ত উত্তর	structured solution
Self-check	step-by-step draft → critique prompt → revise	draft, critique, final answer

The self-check mode is inference-time reflection: the weights do not change, the model simply gets a second pass over its own draft. It sometimes helps and sometimes hurts — both numbers are reported below.

Training

Data

SheikhMinds/rudushi-bangla-sft contains the published artifacts. The full training mix is built deterministically by the Rudushi data pipeline:

source	rows	notes
iamshnoo/alpaca-cleaned-bengali	12,975	translated general instructions
synthetic chain-of-thought / self-check	7,000	generated in Python, exact labels
curated Bangla seed	25	facts, translation, writing, coding
total (train / val)	19659 / 199	deduplicated, script-filtered

Reasoning generators cover: unit price, discount, linear equations, averages, ratios, unit conversion, arithmetic sequences, work-rate, syllogisms, and verification traces.

Recipe

item	value
method	4-bit NF4 QLoRA (fp16 compute)
LoRA rank / alpha	16 / 32
max sequence length	512
optimiser steps	800
final train loss	0.6087
training time	1.3 h
hardware	1× Tesla T4 16 GB
loss masking	assistant tokens only

Framework versions used for this run:

package	version
PyTorch	2.14
Transformers	5.22.1
PEFT	0.21.2
datasets	4.8.5
bitsandbytes	0.50.2

Reproduce with the Rudushi repository:

make reasoning-data
make reasoning-train
make reasoning-eval
make base-eval
make merge

Evaluation

Loss is not capability. We report exact-match accuracy on held-out synthetic problems (generated with a different seed than training) and the change caused by reflection.

Held-out synthetic problems, generator seed 777 (unseen in training).

metric	Rudushi-1	base model
problems	40	—
draft accuracy	52.5%	2.5%
after reflection	52.5%	5.0%
reflection gain	+0.0%	+2.5%

By category

category	n	draft	after reflection
average	5	20.0%	20.0%
convert_hour_min	2	50.0%	50.0%
convert_kg_g	1	100.0%	100.0%
convert_litre_ml	1	100.0%	100.0%
discount	5	60.0%	60.0%
linear_equation	5	80.0%	80.0%
ratio	5	40.0%	40.0%
sequence	4	100.0%	100.0%
syllogism	3	0.0%	0.0%
unit_price	5	20.0%	20.0%
work_rate	4	75.0%	75.0%

These are bounded, synthetic distributions. High accuracy means the model learned this format and these operations; it does not establish general mathematical reasoning. A base-model column is included so the fine-tune can be judged against its starting point.

Limitations

• Narrow reasoning data. Arithmetic/logic coverage is synthetic and finite; expect failures outside those patterns.
• Small model. 1.5B parameters cannot match 7B+ models on knowledge or multi-hop tasks.
• Hallucination. Names, numbers, dates and citations can be fabricated.
• No RLHF/DPO. Helpfulness and refusal behaviour are not preference-tuned.
• Dialect coverage. Standard Bangla dominates. Jessore dialect is a separate, untrained run.
• Long context. The base supports 32k tokens, but this fine-tune was trained at 512; long-context quality is not evaluated.
• Not for safety-critical use. No medical, legal, financial or safety advice.

বাংলায় সংক্ষেপে

রুদুষি-১ একটি বাংলা-প্রথম ১.৫বি মডেল। এটি ধাপে ধাপে সমাধান ও নিজের উত্তর যাচাই করার ফরম্যাট শেখে। এটি সাধারণ বুদ্ধিমত্তার মডেল নয় — ছোট, সীমিত ডেটায় প্রশিক্ষিত একটি গবেষণামূলক মডেল। গুরুত্বপূর্ণ তথ্য সবসময় যাচাই করুন।

যশোর অঞ্চলের উপভাষার জন্য আলাদা ডেটা ও আলাদা রান দরকার; এই চেকপয়েন্টটি উপভাষায় বিশেষভাবে প্রশিক্ষিত নয়। বিস্তারিত: Rudushi রিপোর docs/dialect-jessore.md।

Jessore dialect

The Bangla spoken in Jessore (যশোর) is a clear, melodic form of the southwestern Bangla dialect group, described as shaped by both Rarhi and Bangali strands. It is a living regional identity, not "incorrect" Bangla.

item	status
draft seed pairs	20, awaiting native-speaker review
dialect-aware config	configs/rudushi-1-jessore.yaml
dialect-specialised checkpoint	not trained

Example pairs from the draft seed:

standard Bangla	Jessore form
কী করছ?	কিতা করতাছস?
কেমন আছ?	কেমুন আছস?
আজ বাড়ি যাব।	আইজ ঘর যাইবো।

The design goal is optional, context-aware dialect use: if a user writes standard Bangla, the answer should stay standard.

License

• Model weights and training code: Apache-2.0
• Base model: Qwen/Qwen2.5-1.5B-Instruct (Apache-2.0)
• Data: each upstream source keeps its own licence; see the dataset card.

Citation

@misc{rudushi1,
  title        = {Rudushi-1: An Open Bangla-First Instruction Model with Structured Reasoning},
  author       = {Rudushi},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/SheikhMinds/rudushi-1-1.5b}},
  note         = {QLoRA fine-tune of Qwen/Qwen2.5-1.5B-Instruct}
}

Model card contact

Open an issue or discussion on the Rudushi Space.