Sabbir505/Ling-3.0-tiny-math-sft-v1

Ling-3.0-tiny · Math-SFT (QLoRA fine-tune)

A QLoRA fine-tune of inclusionAI/Ling-3.0-tiny (7.9B-parameter MoE, ~1.3B active per token, hybrid KDA linear-attention + MLA architecture), trained to improve math reasoning and instruction-following on consumer-scale hardware.

This is work in progress. We plan to fine-tune it further — a larger and more math-heavy mixture, longer training, and additional post-training (preference/RL) stages — to achieve better results. Expect this repo to be updated.

Benchmark results

All probes use identical data, seeds, and extraction before and after training (100-question MMLU and GSM8K subsets, 50-prompt IFEval-lite with verifiable constraints). Numbers are not leaderboard-comparable; they are controlled A/B measurements of this fine-tune.

Probe	Base	Fine-tuned	Δ
MMLU-100 (knowledge, loglikelihood)	58.0%	59.0%	+1.0
GSM8K-100 (math, no-think)	5.0%	27.0%	+22.0
IFEval-lite-50 (instruction following)	44.0%	44.0%	0.0
GSM8K-30 (think-mode, supplementary)	—	23.3%	new capability

• The GSM8K gain (+22 points, >5×) is the headline result: the model learned to actually solve grade-school math instead of failing almost uniformly.
• MMLU unchanged confirms no catastrophic forgetting of general knowledge.
• The think-mode supplementary eval (GSM8K-30, enable_thinking=True, answer extracted from \boxed{}/####) shows the model adopted the chain-of-thought format it was trained on; 23.3% on a 30-question subset is consistent with the no-think 27% given the small sample.
• IFEval-lite unchanged: instruction-following was already the model's strongest area and 475 IF examples in the mix were not enough to move it. A future run uses a heavier IF share.

Training details

Method	QLoRA (4-bit NF4 base, bf16 compute)
LoRA	r=16, alpha=32, dropout 0.05, all attention + MLP projections (MoE experts included, router frozen)
Trainable params	296,091,648 (3.62% of 8.19B)
Data	2,000 verified examples: 1,210 everyday conversations (smoltalk2/smollm3), 475 instruction-following (Tulu-3 SFT personas), 475 math reasoning with chain-of-thought (OpenThoughts-114k)
Sequence length	384 tokens, no packing
Optimization	lr 5e-5 cosine, 60 warmup steps, effective batch 8, 1 epoch (246 steps), paged AdamW 8-bit
Hardware / time	1× RTX 4090 D, ~4.7 hours

The data was rendered through Ling-3.0's native chat template, preserving its hybrid thinking-mode design: everyday/IF examples use detailed thinking off, math examples use detailed thinking on with real chain-of-thought targets.

Files

File	Size	Use
ling30tiny-ft-Q4_K_M.gguf	~4.7 GB	recommended for local inference
ling30tiny-ft-Q8_0.gguf	~8.4 GB	near-lossless
ling30tiny-ft-Q2_K.gguf	~3.1 GB	ultra-compact
merged-ft-f16.gguf	~15.8 GB	full-precision merged model
adapter/	~600 MB	raw LoRA adapter (apply to the base model)
benchmarks/	—	raw before/after benchmark JSONs

Usage

llama.cpp (GGUF)

llama-server -m ling30tiny-ft-Q4_K_M.gguf --port 8080
# or
llama-cli -m ling30tiny-ft-Q4_K_M.gguf -p "Solve: Natalia sold clips to 48 friends in April and half as many in May. How many altogether?"

For math, wrap the prompt with the model's thinking-mode system prompt (detailed thinking on) or rely on the chat template's enable_thinking flag — the model was trained to reason in <think>...</think> blocks on math problems.

transformers + PEFT (adapter)

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

tok = AutoTokenizer.from_pretrained("inclusionAI/Ling-3.0-tiny", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    "inclusionAI/Ling-3.0-tiny", torch_dtype=torch.bfloat16,
    device_map="auto", trust_remote_code=True)
model = PeftModel.from_pretrained(model, "Sabbir505/Ling-3.0-tiny-math-sft-v1", subfolder="adapter")

Honest limitations

• Trained on a deliberately small (2,000-example), single-epoch run on free/consumer hardware; the gains are real but the model is far from its ceiling.
• The IFEval-lite checker is a simplified subset of the official harness; absolute numbers are not comparable to published IFEval scores, only the before/after delta.
• GSM8K answers were extracted via #### N / last-number fallback; the think-mode supplementary eval used \boxed{}/#### on a 30-question subset.
• In no-think mode the model can be verbose on math prompts (it defaults to think-style reasoning it learned); thinking-mode prompting produces cleaner outputs.

Roadmap

• Larger, more math-heavy mixture (5–10k reasoning examples)
• 2–3 epoch training with evaluation-gated checkpoints
• Longer sequence length (1024+) for full chain-of-thought targets
• Instruction-following heavy rebalance to lift IFEval
• Preference optimization (DPO) on top of this SFT stage

Acknowledgments

Base model by inclusionAI (Apache-2.0). Data from HuggingFaceTB smoltalk2 (Apache-2.0), AI2 Tulu-3 SFT personas (ODC-BY), and OpenThoughts (Apache-2.0). Trained with transformers 4.57.6 + TRL 1.13.0 + PEFT + bitsandbytes + flash-linear-attention.