SupritiVijay/reasoning-distillation-models

Reasoning Distilled Models

A collection of small language models fine-tuned on reasoning traces from multiple teacher models across 4 domains (math, science, if_chat, code).

Repository Structure

qwen2.5_1.5b/                          # Qwen2.5-1.5B-Instruct base
├── {parent}/
│   ├── math/                          # Single-domain (~18k)
│   ├── science/
│   ├── if_chat/
│   ├── code/
│   ├── code(1.5k)/                    # Code-only, data-cardinality splits
│   ├── code(4.5k)/
│   ├── code(9k)/
│   ├── mixed_domain(18k)/             # Mix of code/math/science/if_chat (18k)
│   └── mixed_domain(72k)/             # All 4 domains combined (72k)
└── mixed_parent/
    ├── code(54k)/                     # Code from 3 teachers combined
    └── code(90k)/                     # Code from 5 teachers combined

llama3.2_1b/                           # Llama-3.2-1B-Instruct base
├── {parent}/
│   ├── code/                          # Single-domain (code only)
│   ├── mixed_domain(18k)/             # Mix of code/math/science/if_chat (18k)
│   └── mixed_domain(72k)/             # All 4 domains combined (72k)

qwen2.5_3b/                            # Qwen2.5-3B-Instruct base
├── {parent}/
│   └── code(18k)/                     # Code-only traces (18k)

qwen2.5_7b/                            # Qwen2.5-7B-Instruct base
├── {parent}/
│   ├── code(18k)/                     # Code-only traces (18k)
│   ├── code(1.5k)/                    # Code-only, data-cardinality splits
│   ├── code(4.5k)/
│   └── code(9k)/

Teacher Models (Parents)

Parent	Description
deepseek_distill_llama3_70b	DeepSeek distilled on LLaMA-3 70B
deepseek_distill_qwen_25	DeepSeek distilled on Qwen2.5
deepseekv3_2	DeepSeek-V3
exaone4_32b	EXAONE-4 32B
glm	GLM
gpt_oss_120b	GPT OSS 120B
gpt_oss_20b	GPT OSS 20B
olmo31	OLMo 3.1
qwen3_235B	Qwen3-235B
qwq	QwQ

Training Details

• Epochs: 3
• Learning rate: 1e-4 with 10% warmup
• Max length: 32,768 tokens
• Precision: bf16
• Loss: Completion-only loss (assistant tokens only)
• Hardware: 8× H100 (torchrun; DeepSpeed ZeRO-2 for 1.5B, ZeRO-3 + CPU offload for 7B)

Qwen2.5-1.5B Models

All 10 parents unless noted.

Variant	Models	Notes
math / science / if_chat / code	40	10 parents × 4 single domains, ~18k samples each
code(1.5k)	10	Code-only data-cardinality split
code(4.5k)	10	Code-only data-cardinality split
code(9k)	10	Code-only data-cardinality split
mixed_domain(18k)	10	Mix of all 4 domains (18k)
mixed_domain(72k)	10	All 4 domains (72k)
mixed_parent/code(54k)	1	Code traces mixed across 3 teachers
mixed_parent/code(90k)	1	Code traces mixed across 5 teachers

Data cardinality splits (code(1.5k) / code(4.5k) / code(9k)) hold the domain fixed at code and vary only the number of training traces, isolating the effect of data volume on lineage signal.

Mixed-parent models hold the domain fixed at code and blend traces from multiple teachers into one training set, so no single lineage dominates:

Variant	Teachers	Traces
mixed_parent/code(54k)	gpt_oss_120b, deepseekv3_2, qwen3_235B	3 × 18k = 54k
mixed_parent/code(90k)	the 3 above + exaone4_32b, qwq	5 × 18k = 90k

Llama-3.2-1B Models

• 1 single-domain model: qwen3_235B / code
• 6 mixed_domain(18k) models
• 6 mixed_domain(72k) models

Qwen2.5-3B Models

• 6 code(18k) models: 6 parents × code-only traces (18k)

Qwen2.5-7B Models

Variant	Models	Parents
code(18k)	6	all 6
code(1.5k)	3	deepseekv3_2, qwen3_235B, qwq
code(4.5k)	3	deepseekv3_2, qwen3_235B, qwq
code(9k)	2	qwen3_235B, qwq

The 6 parents for 3B/7B models are: qwen3_235B, deepseekv3_2, gpt_oss_120b, olmo31, deepseek_distill_qwen_25, qwq

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

# Qwen2.5-1.5B — single domain
model = AutoModelForCausalLM.from_pretrained(
    "SupritiVijay/Reasoning-Distilled-Models",
    subfolder="qwen2.5_1.5b/gpt_oss_120b/math",
    torch_dtype="auto",
    device_map="auto",
)

# Qwen2.5-1.5B — code data-cardinality split
model = AutoModelForCausalLM.from_pretrained(
    "SupritiVijay/Reasoning-Distilled-Models",
    subfolder="qwen2.5_1.5b/glm/code(4.5k)",
    torch_dtype="auto",
    device_map="auto",
)

# Qwen2.5-1.5B — mixed domain (72k)
model = AutoModelForCausalLM.from_pretrained(
    "SupritiVijay/Reasoning-Distilled-Models",
    subfolder="qwen2.5_1.5b/qwen3_235B/mixed_domain(72k)",
    torch_dtype="auto",
    device_map="auto",
)

# Qwen2.5-1.5B — mixed parent (90k, code only)
model = AutoModelForCausalLM.from_pretrained(
    "SupritiVijay/Reasoning-Distilled-Models",
    subfolder="qwen2.5_1.5b/mixed_parent/code(90k)",
    torch_dtype="auto",
    device_map="auto",
)

# Llama-3.2-1B — mixed domain (72k)
model = AutoModelForCausalLM.from_pretrained(
    "SupritiVijay/Reasoning-Distilled-Models",
    subfolder="llama3.2_1b/qwen3_235B/mixed_domain(72k)",
    torch_dtype="auto",
    device_map="auto",
)

# Qwen2.5-3B — code (18k)
model = AutoModelForCausalLM.from_pretrained(
    "SupritiVijay/Reasoning-Distilled-Models",
    subfolder="qwen2.5_3b/qwen3_235B/code(18k)",
    torch_dtype="auto",
    device_map="auto",
)

tokenizer = AutoTokenizer.from_pretrained(
    "SupritiVijay/Reasoning-Distilled-Models",
    subfolder="qwen2.5_3b/qwen3_235B/code(18k)",
)

Download a Single Model (CLI)

huggingface-cli download SupritiVijay/Reasoning-Distilled-Models \
    --include "qwen2.5_3b/gpt_oss_120b/code(18k)/*" \
    --local-dir ./model