OliviaRossi/UltiMerge

Pure text-causal build. Multi-token-prediction heads and multimodal vision weights were decoupled and stripped before merging. That drops roughly 4.2 GB of VRAM overhead and gives immediate out-of-the-box compatibility with vLLM, SGLang, and llama.cpp (GGUF).

---

Contents

• Overview
• Quad-DNA Architecture
• The GS-MoE Transformation Pipeline
• Mathematical Formulation
• Inside the Merged Weight Manifold
• Deployment & Inference
• Recommended Sampling
• Model Specifications
• Attributions & Citations

---

Overview

UltiMerge is a 35B-A3B mixture-of-experts model built on the Qwen 3.6 hybrid architecture — 30 Gated DeltaNet linear-recurrence layers plus 10 full GQA attention layers, with 256 routed experts (top-8 active) behind one always-on shared expert.

It isn't a fine-tune. It's a geometric merge of four independently trained checkpoints, each strong in a different discipline: world simulation and tool use, repository-scale software engineering, deep chain-of-thought reasoning, and adversarial self-correction. A five-stage pipeline — Geometric Spectral MoE (GS-MoE) — isolates what each donor actually learned, filters out merge noise, and extrapolates the combination along the checkpoints' shared weight manifold, rather than simply averaging them.

The result runs with the compute footprint of a 3.1B-parameter model while carrying the combined skill set of four 35B-class specialists.

---

Quad-DNA Architecture

Source checkpoints

Pillar	Checkpoint	Primary domain	Core strengths
Anchor	Qwen/Qwen-AgentWorld-35B-A3B	World simulator & agent core	Environment state persistence, tool-execution sandbox, structured planning
Donor 1	Kwaipilot/KAT-Coder-V2.5-Dev	SWE-bench autonomous engineering	Multi-file git diffs, AST-aware refactoring, repository bug localization
Donor 2	Jackrong/Qwopus3.6-35B-A3B-Coder	Opus-grade reasoning engine	Recursive problem decomposition, concise zero-boilerplate generation
Donor 3	ornith-ai/Ornith-1.5-35B-A3B	Self-correcting agentic core	Zero-shot constraint checking, adversarial logic audits, multi-paradigm coding

Each checkpoint's task vector is isolated relative to the shared Qwen 3.6 base before any merge operation runs, which is what keeps the fusion from collapsing into an average of four unrelated skills.

---

The GS-MoE Transformation Pipeline

Stage	Operation	Target layers	Function
01	Invariant State Centroiding	SSM recurrence (dt_bias, A_log, in_proj_a/b)	Norm-conserved spherical SLERP that preserves sequence memory
02	Frobenius Gate Normalization	MoE routers (mlp.gate, router)	Magnitude calibration that locks the softmax routing distribution
03	DARE Bernoulli Masking	256 routed experts (gate/up/down_proj)	Drop-and-rescale noise filter at p = 0.20
04	STAR Subspace SVD	2D projections (q/k/v/o_proj, expert MLPs)	Randomized low-rank truncation retaining 90% of spectral energy
05	Model Stock Extrapolation	Full weight manifold	Hyperspherical angle scaling along the merge direction

Recurrence parameters bypass low-rank truncation entirely (stage 01) to avoid degrading memory behavior at long context lengths — everything downstream of that is where the actual skill-blending happens.

---

Mathematical Formulation

1. Invariant SSM layer preservation. Recurrence parameters are simple-averaged across donors to protect linear-attention stability up to 256K context:

$${\mathbf{W}}_{\text{SSM}}=\frac{1}{N}\sum _{i=1}^N{\mathbf{W}}_i\backslash mathbf\{W\}\_\{\backslash text\{SSM\}\}\; =\; \backslash frac\{1\}\{N\}\; \backslash sum\_\{i=1\}^N\; \backslash mathbf\{W\}\_i$$

2. MoE router entropy-locked alignment. Merged gate weights are rescaled against the anchor's Frobenius norm to stop routing entropy from exploding:

$${\mathbf{W}}_{\text{gate}}={\bar{\mathbf{W}}}_{\text{gate}}\cdot \frac{\mathrm{\parallel }{\mathbf{W}}_0{\mathrm{\parallel }}_F}{\mathrm{\parallel }{\bar{\mathbf{W}}}_{\text{gate}}{\mathrm{\parallel }}_F+ϵ}\backslash mathbf\{W\}\_\{\backslash text\{gate\}\}\; =\; \backslash bar\{\backslash mathbf\{W\}\}\_\{\backslash text\{gate\}\}\; \backslash cdot\; \backslash frac\{\backslash |\backslash mathbf\{W\}\_0\backslash |\_F\}\{\backslash |\backslash bar\{\backslash mathbf\{W\}\}\_\{\backslash text\{gate\}\}\backslash |\_F\; +\; \backslash epsilon\}$$

3. DARE subspace noise filtering. Each expert's task delta $\Delta \mathbf{W}_i = \mathbf{W}_i - \mathbf{W}_0$ is sparsified at Bernoulli probability $p = 0.20$ and rescaled:

$${\widetilde{\mathrm{\Delta }\mathbf{W}}}_i=\frac{\mathrm{\Delta }{\mathbf{W}}_i\odot \mathbf{M}}{1-p},\mathbf{M}\sim \text{Bernoulli}(1-p)\backslash widetilde\{\backslash Delta\; \backslash mathbf\{W\}\}\_i\; =\; \backslash frac\{\backslash Delta\; \backslash mathbf\{W\}\_i\; \backslash odot\; \backslash mathbf\{M\}\}\{1\; -\; p\},\; \backslash quad\; \backslash mathbf\{M\}\; \backslash sim\; \backslash text\{Bernoulli\}(1\; -\; p)$$

4. Spectral truncation & rescaling (STAR). Randomized low-rank SVD filters high-frequency merge noise while retaining $\gamma = 0.90$ of total Frobenius energy:

$${\mathbf{\Sigma }}_{:k}^{\mathrm{\prime }}={\mathbf{\Sigma }}_{:k}\cdot \frac{\mathrm{\parallel }\mathbf{\Sigma }{\mathrm{\parallel }}_F}{\mathrm{\parallel }{\mathbf{\Sigma }}_{:k}{\mathrm{\parallel }}_F+ϵ},{\widehat{\mathrm{\Delta }\mathbf{W}}}_i={\mathbf{U}}_{:k}{\mathbf{\Sigma }}_{:k}^{\mathrm{\prime }}{\mathbf{V}}_{:k}^T\backslash mathbf\{\backslash Sigma\}\text{'}\_\{:k\}\; =\; \backslash mathbf\{\backslash Sigma\}\_\{:k\}\; \backslash cdot\; \backslash frac\{\backslash |\backslash mathbf\{\backslash Sigma\}\backslash |\_F\}\{\backslash |\backslash mathbf\{\backslash Sigma\}\_\{:k\}\backslash |\_F\; +\; \backslash epsilon\},\; \backslash qquad\; \backslash widehat\{\backslash Delta\; \backslash mathbf\{W\}\}\_i\; =\; \backslash mathbf\{U\}\_\{:k\}\; \backslash mathbf\{\backslash Sigma\}\text{'}\_\{:k\}\; \backslash mathbf\{V\}\_\{:k\}^T$$

5. Model Stock hyperspherical extrapolation. The merged update is extrapolated along the intersection manifold using the optimal cosine-angle factor $t^*$:

$$t^{\ast }=\frac{N\cos \bar{\theta }}{1+(N-1)\cos \bar{\theta }},{\mathbf{W}}_{\text{final}}={\mathbf{W}}_0+t^{\ast }\cdot \frac{1}{N}\sum _{i=1}^N{\widehat{\mathrm{\Delta }\mathbf{W}}}_{i}t^*\; =\; \backslash frac\{N\; \backslash cos\; \backslash bar\{\backslash theta\}\}\{1\; +\; (N\; -\; 1)\backslash cos\; \backslash bar\{\backslash theta\}\},\; \backslash qquad\; \backslash mathbf\{W\}\_\{\backslash text\{final\}\}\; =\; \backslash mathbf\{W\}\_0\; +\; t^*\; \backslash cdot\; \backslash frac\{1\}\{N\}\backslash sum\_\{i=1\}^N\; \backslash widehat\{\backslash Delta\; \backslash mathbf\{W\}\}\_i$$

---

Inside the Merged Weight Manifold

---

Deployment & Inference

1. Hugging Face transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "OliviaRossi/UltiMerge"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

messages = [
    {"role": "system", "content": "You are UltiMerge, an elite autonomous software engineer and reasoning system."},
    {"role": "user", "content": "Implement a thread-safe, high-concurrency LRU cache in Python using doubly linked lists and hash maps, complete with unit tests."}
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=2048,
    temperature=0.2,
    top_p=0.9,
    repetition_penalty=1.05
)

print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

2. Production server with vLLM

vllm serve OliviaRossi/UltiMerge \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.95 \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --port 8000

3. Local GGUF inference (llama.cpp)

# 1. Convert to BF16 GGUF
python3 convert_hf_to_gguf.py OliviaRossi/UltiMerge \
  --outfile ./ultimerge_35b_f16.gguf \
  --outtype bf16

# 2. Quantize to Q4_K_M
llama-quantize ./ultimerge_35b_f16.gguf ./ultimerge_35b_Q4_K_M.gguf Q4_K_M

# 3. Launch interactive CLI
llama-cli \
  -m ./ultimerge_35b_Q4_K_M.gguf \
  -p "<|im_start|>system\nYou are UltiMerge.<|im_end|>\n<|im_start|>user\nWrite a lock-free queue in Rust.<|im_end|>\n<|im_start|>assistant\n" \
  -n 2048 \
  --temp 0.2 \
  -ngl 99

4. Ollama Modelfile

FROM ./ultimerge_35b_Q4_K_M.gguf

TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""

SYSTEM "You are UltiMerge, an elite software engineering agent and reasoning system."

PARAMETER temperature 0.2
PARAMETER top_p 0.9
PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"

---

Recommended Sampling

Workload	Temperature	Top-P	Min-P	Repetition penalty	Max tokens
Autonomous coding & git patching	0.15	0.90	0.05	1.05	4,096
Step-by-step logic & reasoning	0.30	0.95	0.05	1.02	8,192
Sandbox agent & tool calling	0.20	0.90	0.05	1.05	4,096
Creative refactoring & architecture	0.40	0.95	0.05	1.03	8,192

---

Model Specifications

Metric	Specification
Base architecture	Qwen2MoeForCausalLM (hybrid Gated DeltaNet SSM + GQA)
Total parameters	34,812,940,288 (~34.8B)
Active parameters per token	3,091,841,024 (~3.1B)
Total transformer layers	40 (30 hybrid DeltaNet SSM + 10 full-attention GQA)
Routed MoE experts	256 total (top-8 active per token)
Shared continuous experts	1 dense expert, always active
Context window	32,768 native, extendable to 256,000 via YaRN RoPE
Vocabulary size	152,064 tokens
Hidden dimension	2,048
Intermediate expert dimension	1,408
Attention heads	16 query / 2 key-value (GQA)
Decoupled modules	MTP speculative heads & vision encoders (0% overhead)

---

Attributions & Citations

Source checkpoints

• Alibaba Qwen Team — Qwen-AgentWorld-35B-A3B, Language Models as Interactive World Simulators
• Kwaipilot Team — KAT-Coder-V2.5-Dev, Autonomous Software Engineering with MoE Foundations
• Jackrong — Qwopus3.6-35B-A3B-Coder, Opus-Grade Reasoning Distillation in MoE Models
• Ornith AI — Ornith-1.5-35B-A3B, Recursive Self-Correction and Constraint Alignment

BibTeX

@article{qwen_agentworld_2026,
  title        = {Qwen-AgentWorld: Language Models as Interactive World Simulators},
  author       = {{Qwen Team, Alibaba Group}},
  journal      = {Hugging Face Model Hub},
  year         = {2026},
  url          = {https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B}
}

@article{kwaipilot_kat_coder_2026,
  title        = {KAT-Coder: Autonomous Software Engineering with Mixture-of-Experts},
  author       = {{Kwaipilot Team}},
  journal      = {Hugging Face Model Hub},
  year         = {2026},
  url          = {https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev}
}

@article{jackrong_qwopus_2026,
  title        = {Qwopus-3.6: Distilling Opus-Grade Reasoning into High-Throughput MoE Architectures},
  author       = {Jackrong},
  journal      = {Hugging Face Model Hub},
  year         = {2026},
  url          = {https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder}
}

@article{ornith_ai_2026,
  title        = {Ornith-1.5: Recursive Self-Correction and Constraint Alignment in Hybrid Models},
  author       = {{Ornith AI}},
  journal      = {Hugging Face Model Hub},
  year         = {2026},
  url          = {https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B}
}

@misc{ultimerge_2026,
  title        = {UltiMerge: A Geometric Spectral MoE Fusion of Four 35B-A3B Specialists},
  author       = {Rossi, Olivia},
  journal      = {Hugging Face Model Hub},
  year         = {2026},
  url          = {https://huggingface.co/OliviaRossi/UltiMerge}
}

_{Merged via Geometric Spectral MoE (GS-MoE) · Apache 2.0 License · 2026}