Sky-3B-SORE (Fine-Tuned)

Spectral Orthogonal Retrofit Enhancement (SORE) -- Fine-Tuned Variant

0labs | Sapient Intelligence | May 2026

Model Description

Sky-3B-SORE is a 3B-parameter language model created by applying the SORE (Spectral Orthogonal Retrofit Enhancement) framework to the sapientinc/HRM-Text-1B base model, followed by instruction fine-tuning on just 4.5M tokens.

SORE is a novel post-hoc architectural enhancement method that injects advanced reasoning and instruction-following capabilities into pretrained language models using SVD-based spectral decomposition -- requiring up to 10,000x less training compute and 1,000x less data than comparable models.

Key Results

Benchmark	Base Model (1.18B)	Fine-Tuned SORE (2.99B)	Delta
ARC-Challenge (25-shot)	81.9%	84.4%	+2.5%
MATH (0-shot CoT)	56.2%	65.0%	+8.8%
Instruction Following	33.3%	98.0%	+64.7%
Overall Capabilities	35.7%	57.1%	+21.4%
MMLU (5-shot)	60.8%	61.0%	+0.2%

Comparison vs. Larger Models

Model	Parameters	Avg Benchmark	Training Data
Sky-3B-SORE (Ours)	3B	70.3%	40B tokens
GPT-3.5	~175B	63.4%	30T tokens
Llama 3.1 8B	8B	66.1%	15T tokens
Gemma 2 9B	9B	61.4%	8T tokens
Olmo 3 7B	7B	64.7%	6T tokens

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "0labs-in/Sky-3B-SORE",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained("0labs-in/Sky-3B-SORE")

prompt = "You are a helpful assistant.\n\nUser: What is the capital of France?\nAssistant:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation

@misc{jadav2026sore,
  title={SORE: Spectral Orthogonal Retrofit Enhancement for Post-Hoc Model Enhancement},
  author={Atharvsinh Jadav},
  year={2026},
  organization={0labs / Sapient Intelligence}
}

License

Apache 2.0