OliviaRossi/DAOS-Fusion

Domain-Aware Orthogonal Subspace Fusion with Spectral-DELLA Consensus

🌐 World Simulation	💻 SWE Autonomous	🧠 Opus Reasoning	🦅 Self-Correction
AgentWorld-35B State persistence & tool runtime	KAT-Coder-V2.5 Repository-level git diffs	Qwopus-3.6 Claude Opus-grade logic	Ornith-1.5 Recursive logic verification

⚡ Pure Text-Causal Decoupling — Multi-Token Prediction (MTP layer 40) and multimodal vision modules have been surgically removed. This eliminates speculative-decoding memory overhead, frees roughly 4.2 GB of GPU VRAM, sidesteps blk.40 GGUF dimension errors, and gives out-of-the-box compatibility with vLLM, SGLang, and llama.cpp.

---

📑 Table of Contents

• 🌟 Executive Overview
• 🧬 Quad-DNA Checkpoint Foundation
• 🔧 Hybrid Architecture
• 🔬 Mathematical Formulation
• 🚀 Deployment & Inference
• ⚙️ Recommended Sampling Matrix
• 📊 Model Specifications
• 🤝 Academic Attributions & Citations

---

🌟 Executive Overview

DAOS-Fusion-35B-A3B is built on the Qwen 3.5-MoE hybrid backbone — 30 Gated DeltaNet linear-recurrence layers interleaved with 10 full GQA attention layers, routing across 256 experts. It executes with the compute footprint of a 3.1B-parameter dense model while carrying 34.8B total parameters.

Using Domain-Aware Orthogonal Subspace Fusion (DAOS-Fusion) — a seven-stage deterministic merge pipeline described below — this checkpoint combines four specialist fine-tunes into a single coherent weight manifold:

• the autonomous software-engineering skill of KAT-Coder
• the recursive chain-of-logic reasoning of Qwopus-3.6
• the environment-sandbox persistence of Qwen-AgentWorld
• the adversarial self-verification of Ornith-1.5

Rather than naïve weight averaging — which causes destructive sign cancellation across domains that never trained together — DAOS-Fusion elects a democratic centroid, filters noise with saliency-aware DARE, resolves conflicting update directions through DELLA sign consensus, denoises with adaptive low-rank SVD, and extrapolates the result onto the true geometric intersection of all four fine-tuning trajectories.

---

🧬 Quad-DNA Checkpoint Foundation

Source Models & Contributions

Pillar	Checkpoint	Primary Specialization	Signature Capability
🌐 World Simulation	Qwen/Qwen-AgentWorld-35B-A3B	World simulation & sandbox agent	Multi-turn environmental persistence, sandbox tool execution, structured planning
💻 SWE Engineering	Kwaipilot/KAT-Coder-V2.5-Dev	Autonomous software engineering	Repository-level bug localization, git-diff patch synthesis, AST-aware refactoring
🧠 Opus Reasoning	Jackrong/Qwopus3.6-35B-A3B-Coder	Opus-grade reasoning engine	Deep recursive problem decomposition, concise execution without boilerplate
🦅 Self-Correction	ornith-ai/Ornith-1.5-35B-A3B	Self-correcting agentic core	Zero-shot logic verification, adversarial constraint checking, multi-paradigm coding

DAOS-Fusion Multi-Stage Transformation Pipeline

Stage	Operation	Target	Mathematical Role
01	Democratic Centroiding	Global weight space	Geometric baseline $\mathbf{W}_{\text{centroid}} = \frac{1}{N}\sum \mathbf{W}_i$ (no anchor bias)
02	Saliency-Guided DARE	256 routed experts	Drops bottom-20% magnitude noise while preserving high-saliency features
03	DELLA Sign Consensus	Attention & expert MLPs	Coordinate-wise majority sign vote; zeros conflicting updates
04	Adaptive STAR SVD	2D projections	Proportional low-rank SVD ($q \propto \min(m,n)$) isolating the noise floor
05	Model Stock Extrapolation	Full weight manifold	Projects the consensus update onto the true intersection hypersphere ($t^*$)
06	Entropy-Locked Calibration	MoE routers (mlp.gate)	Conserves top-8 logit margins, temperature-sharpens to prevent expert blurring
07	Invariant State Centroiding	SSM recurrence (dt_bias, A_log)	Preserves linear-recurrence eigenvalues and sequence-memory stability

---

🔧 Hybrid Architecture

---

🔬 Mathematical Formulation (DAOS-Fusion)

Standard linear averaging or unweighted task arithmetic causes destructive sign cancellation across disparate domains. DAOS-Fusion resolves this by chaining five mathematical techniques.

1. Democratic Symmetric Centroid (No Anchor Dominance)

Deltas are computed relative to the true geometric centroid of all models present for a given tensor, rather than against an arbitrarily chosen base model:

$${\mathbf{W}}_{\text{centroid}}=\frac{1}{N}\sum _{i=1}^N{\mathbf{W}}_i,\mathrm{\Delta }{\mathbf{W}}_i={\mathbf{W}}_i-{\mathbf{W}}_{\text{centroid}}\backslash mathbf\{W\}\_\{\backslash text\{centroid\}\}\; =\; \backslash frac\{1\}\{N\}\backslash sum\_\{i=1\}^N\; \backslash mathbf\{W\}\_i,\; \backslash qquad\; \backslash Delta\; \backslash mathbf\{W\}\_i\; =\; \backslash mathbf\{W\}\_i\; -\; \backslash mathbf\{W\}\_\{\backslash text\{centroid\}\}$$

2. Saliency-Guided DARE (Magnitude-Aware Masking)

Instead of uniform random coin-flips, parameters with the lowest absolute magnitude (noise jitter) are dropped, and survivors are rescaled by $1/(1-p)$ to conserve Frobenius energy:

$${\widetilde{\mathrm{\Delta }\mathbf{W}}}_i=\frac{\mathrm{\Delta }{\mathbf{W}}_i\odot {\mathbf{M}}_{\text{salient}}}{1-p},\text{where }{\mathbf{M}}_{\text{salient}}=\mathbb{I}(\mathrm{\mid }\mathrm{\Delta }{\mathbf{W}}_i\mathrm{\mid }\ge \text{Quantile}(\mathrm{\mid }\mathrm{\Delta }{\mathbf{W}}_i\mathrm{\mid },p))\backslash widetilde\{\backslash Delta\; \backslash mathbf\{W\}\}\_i\; =\; \backslash frac\{\backslash Delta\; \backslash mathbf\{W\}\_i\; \backslash odot\; \backslash mathbf\{M\}\_\{\backslash text\{salient\}\}\}\{1\; -\; p\},\; \backslash quad\; \backslash text\{where\; \}\; \backslash mathbf\{M\}\_\{\backslash text\{salient\}\}\; =\; \backslash mathbb\{I\}\backslash left(|\backslash Delta\; \backslash mathbf\{W\}\_i|\; \backslash ge\; \backslash text\{Quantile\}(|\backslash Delta\; \backslash mathbf\{W\}\_i|,\; p)\backslash right)$$

3. DELLA Elective Direction Sign Consensus

To eliminate destructive cancellation when fine-tunes pull in opposite directions, the majority sign across all models is elected and conflicting signals are zeroed:

$${\text{Sign}}_{\text{elected}}=\text{sign}\left(\sum _{i=1}^N{\widetilde{\mathrm{\Delta }\mathbf{W}}}_i\right)\backslash text\{Sign\}\_\{\backslash text\{elected\}\}\; =\; \backslash text\{sign\}\backslash left(\backslash sum\_\{i=1\}^N\; \backslash widetilde\{\backslash Delta\; \backslash mathbf\{W\}\}\_i\backslash right)$$

$$\mathrm{\Delta }{\mathbf{W}}_{\text{consensus}}=\frac{\sum _{i=1}^N{\widetilde{\mathrm{\Delta }\mathbf{W}}}_i\odot \mathbb{I}(\text{sign}({\widetilde{\mathrm{\Delta }\mathbf{W}}}_i)={\text{Sign}}_{\text{elected}})}{\max (1,\sum _{i=1}^N\mathbb{I}(\text{sign}({\widetilde{\mathrm{\Delta }\mathbf{W}}}_i)={\text{Sign}}_{\text{elected}}))}\backslash Delta\; \backslash mathbf\{W\}\_\{\backslash text\{consensus\}\}\; =\; \backslash frac\{\backslash sum\_\{i=1\}^N\; \backslash widetilde\{\backslash Delta\; \backslash mathbf\{W\}\}\_i\; \backslash odot\; \backslash mathbb\{I\}\backslash left(\backslash text\{sign\}(\backslash widetilde\{\backslash Delta\; \backslash mathbf\{W\}\}\_i)\; =\; \backslash text\{Sign\}\_\{\backslash text\{elected\}\}\backslash right)\}\{\backslash max\backslash left(1,\; \backslash sum\_\{i=1\}^N\; \backslash mathbb\{I\}\backslash left(\backslash text\{sign\}(\backslash widetilde\{\backslash Delta\; \backslash mathbf\{W\}\}\_i)\; =\; \backslash text\{Sign\}\_\{\backslash text\{elected\}\}\backslash right)\backslash right)\}$$

4. Adaptive STAR Low-Rank SVD Denoising

The low-rank projection $q$ is sized dynamically from matrix geometry ($q = \max(16, \min(128, 0.10 \times \min(m,n)))$) to avoid attention-head starvation, retaining $\gamma = 0.90$ of the singular energy:

$${\mathbf{\Sigma }}_{:k}^{\mathrm{\prime }}={\mathbf{\Sigma }}_{:k}\cdot \frac{\mathrm{\parallel }\mathbf{\Sigma }{\mathrm{\parallel }}_F}{\mathrm{\parallel }{\mathbf{\Sigma }}_{:k}{\mathrm{\parallel }}_F+ϵ},\mathrm{\Delta }{\mathbf{W}}_{\text{STAR}}={\mathbf{U}}_{:k}{\mathbf{\Sigma }}_{:k}^{\mathrm{\prime }}{\mathbf{V}}_{:k}^T\backslash mathbf\{\backslash Sigma\}\text{'}\_\{:k\}\; =\; \backslash mathbf\{\backslash Sigma\}\_\{:k\}\; \backslash cdot\; \backslash frac\{\backslash |\backslash mathbf\{\backslash Sigma\}\backslash |\_F\}\{\backslash |\backslash mathbf\{\backslash Sigma\}\_\{:k\}\backslash |\_F\; +\; \backslash epsilon\},\; \backslash qquad\; \backslash Delta\; \backslash mathbf\{W\}\_\{\backslash text\{STAR\}\}\; =\; \backslash mathbf\{U\}\_\{:k\}\; \backslash mathbf\{\backslash Sigma\}\text{'}\_\{:k\}\; \backslash mathbf\{V\}\_\{:k\}^T$$

5. Model Stock Hyperspherical Extrapolation

The filtered update is projected onto the intersection manifold of the fine-tuned hyperspheres using the optimal geometric angle scaling $t^*$:

$$t^{\ast }=\frac{N\cos \bar{\theta }}{1+(N-1)\cos \bar{\theta }},{\mathbf{W}}_{\text{final}}={\mathbf{W}}_{\text{centroid}}+t^{\ast }\cdot \mathrm{\Delta }{\mathbf{W}}_{\text{STAR}}t^*\; =\; \backslash frac\{N\; \backslash cos\; \backslash bar\{\backslash theta\}\}\{1\; +\; (N\; -\; 1)\backslash cos\; \backslash bar\{\backslash theta\}\},\; \backslash qquad\; \backslash mathbf\{W\}\_\{\backslash text\{final\}\}\; =\; \backslash mathbf\{W\}\_\{\backslash text\{centroid\}\}\; +\; t^*\; \backslash cdot\; \backslash Delta\; \backslash mathbf\{W\}\_\{\backslash text\{STAR\}\}$$

6. MoE Router Entropy-Locked Calibration

Router projections determine expert selection. To prevent logit flattening (entropy explosion across 256 experts), gate matrices are magnitude-calibrated with temperature sharpening ($1.05\times$):

$${\mathbf{W}}_{\text{gate}}={\bar{\mathbf{W}}}_{\text{gate}}\cdot \left(\frac{\mathrm{\parallel }{\mathbf{W}}_0{\mathrm{\parallel }}_F}{\mathrm{\parallel }{\bar{\mathbf{W}}}_{\text{gate}}{\mathrm{\parallel }}_F+ϵ}\right)\cdot 1.05\backslash mathbf\{W\}\_\{\backslash text\{gate\}\}\; =\; \backslash bar\{\backslash mathbf\{W\}\}\_\{\backslash text\{gate\}\}\; \backslash cdot\; \backslash left(\backslash frac\{\backslash |\backslash mathbf\{W\}\_0\backslash |\_F\}\{\backslash |\backslash bar\{\backslash mathbf\{W\}\}\_\{\backslash text\{gate\}\}\backslash |\_F\; +\; \backslash epsilon\}\backslash right)\; \backslash cdot\; 1.05$$

---

🚀 Deployment & Inference

1. Hugging Face transformers

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "OliviaRossi/DAOS-Fusion"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

messages = [
    {"role": "system", "content": "You are DAOS-Fusion, an autonomous software architect and reasoning agent."},
    {"role": "user", "content": "Implement a high-throughput, thread-safe asynchronous token bucket rate limiter in Python with Redis backing."}
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=2048,
    temperature=0.2,
    top_p=0.9,
    repetition_penalty=1.05
)

print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

2. Production Server with vLLM

Serve high-throughput OpenAI-compatible endpoints with expert tensor parallelism:

vllm serve OliviaRossi/DAOS-Fusion \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.95 \
  --trust-remote-code \
  --tensor-parallel-size 1 \
  --port 8000

3. Local GGUF Inference (llama.cpp)

# 1. Convert to BF16 GGUF
python3 convert_hf_to_gguf.py OliviaRossi/DAOS-Fusion \
  --outfile ./daos_fusion_f16.gguf \
  --outtype bf16

# 2. Quantize to Q4_K_M
llama-quantize ./daos_fusion_f16.gguf ./daos_fusion_Q4_K_M.gguf Q4_K_M

# 3. Launch interactive CLI
llama-cli \
  -m ./daos_fusion_Q4_K_M.gguf \
  -p "<|im_start|>system\nYou are DAOS-Fusion.<|im_end|>\n<|im_start|>user\nWrite an async lock-free queue in Rust.<|im_end|>\n<|im_start|>assistant\n" \
  -n 2048 \
  --temp 0.2 \
  -ngl 99

4. Ollama Modelfile

FROM ./daos_fusion_Q4_K_M.gguf

TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""

SYSTEM "You are DAOS-Fusion, an elite software engineering agent and reasoning system."

PARAMETER temperature 0.2
PARAMETER top_p 0.9
PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"

---

⚙️ Recommended Sampling Matrix

Workload Domain	Temperature	Top-P	Min-P	Repetition Penalty	Max Tokens
Autonomous Coding & Git Patching	0.15	0.90	0.05	1.05	4,096
Step-by-Step Logic & Math	0.30	0.95	0.05	1.02	8,192
Sandbox Agent & Tool Calling	0.20	0.90	0.05	1.05	4,096
Creative Refactoring & Architecture	0.40	0.95	0.05	1.03	8,192

---

📊 Model Specifications

Metric	Specification
Base Architecture	Qwen3_5MoeForCausalLM (Hybrid Gated DeltaNet SSM + GQA Attention)
Total Parameters	34,812,940,288 (~34.8B)
Active Parameters / Token	3,091,841,024 (~3.1B)
Total Transformer Layers	40 (30 Hybrid DeltaNet SSM + 10 Full Attention GQA)
Routed MoE Experts	256 Total (Top-8 Active per Token)
Shared Continuous Experts	1 Continuous Dense Expert
Native Context Length	32,768 tokens (Extendable to 262,144 via YaRN)
Vocabulary Size	248,320 tokens
Hidden Dimension	2,048
Intermediate Expert Dimension	512
Attention Query / KV Heads	16 Query Heads / 2 Key-Value Heads (GQA)
Decoupled Modules	MTP speculative heads & Vision encoders (0% overhead)
License	Apache 2.0
Tensor Type	BF16

---

🤝 Academic Attributions & Citations

Foundation Checkpoints

• Alibaba Qwen Team: Qwen-AgentWorld-35B-A3B — Language Models as Interactive World Simulators
• Kwaipilot Team: KAT-Coder-V2.5-Dev — Autonomous Software Engineering with Mixture-of-Experts
• Jackrong: Qwopus3.6-35B-A3B-Coder — Opus-Grade Reasoning Distillation in MoE Models
• Ornith AI: Ornith-1.5-35B-A3B — Recursive Self-Correction and Constraint Alignment

📜 Click to view BibTeX citations

@article{qwen_agentworld_2026,
  author  = {{Qwen Team, Alibaba Group}},
  title   = {Qwen-AgentWorld: Language Models as Interactive World Simulators},
  journal = {Hugging Face Model Hub},
  year    = {2026},
  url     = {https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B}
}

@article{kwaipilot_kat_coder_2026,
  author  = {{Kwaipilot Team}},
  title   = {KAT-Coder: Autonomous Software Engineering with Mixture-of-Experts},
  journal = {Hugging Face Model Hub},
  year    = {2026},
  url     = {https://huggingface.co/Kwaipilot/KAT-Coder-V2.5-Dev}
}

@article{jackrong_qwopus_2026,
  author  = {Jackrong},
  title   = {Qwopus-3.6: Distilling Opus-Grade Reasoning into High-Throughput MoE Architectures},
  journal = {Hugging Face Model Hub},
  year    = {2026},
  url     = {https://huggingface.co/Jackrong/Qwopus3.6-35B-A3B-Coder}
}

@article{ornith_ai_2026,
  author  = {{Ornith AI}},
  title   = {Ornith-1.5: Recursive Self-Correction and Constraint Alignment in Hybrid Models},
  journal = {Hugging Face Model Hub},
  year    = {2026},
  url     = {https://huggingface.co/ornith-ai/Ornith-1.5-35B-A3B}
}

---

_{Merged via DAOS-Fusion (S-DELLA + STAR + Model Stock)  •  Apache 2.0 License  •  2026}