TripleTrouble (KAT-Ornith-AgentWorld Merge)

An agentic Mixture-of-Experts (MoE) foundation model synthesized by merging three state-of-the-art specialized variants of the Qwen 35B-A3B architecture using Component-Aware Norm-Balanced Hyperspherical Fusion.

This checkpoint fuses:

1. Autonomous Repository Engineering & SWE Skills from Kwaipilot/KAT-Coder-V2.5-Dev
2. Multi-Turn Tool Orchestration & Deep Reasoning from ornith-ai/Ornith-1.5-35B-A3B
3. Environment Dynamic Simulation & World Modeling from Qwen/Qwen-AgentWorld-35B-A3B

---

Model Architecture Specifications

The base topology utilizes the Qwen 35B-A3B sparse hybrid design:

• Total Parameters: ~34.7 Billion
• Active Parameters per Token: ~3.5 Billion (A3B)
• Decoder Layers: 40
• Attention Mechanism: Hybrid Grouped-Query Attention (GQA) interleaved with Gated DeltaNet (GDN) linear recurrent projections
• MoE Structure:
  • 256 fine-grained routed experts per layer (top-8 activated)
  • 1 shared expert active across all tokens
  • Over 30,000 independent projection matrices across all blocks

---

Lineage & Weight Allocation

Model Identifier	Weight	Primary Domain & Added Inductive Bias
Kwaipilot/KAT-Coder-V2.5-Dev	0.40	Autonomous software engineering, patch generation, Terminal-Bench workflows, and syntax tree diffing.
ornith-ai/Ornith-1.5-35B-A3B	0.35	Tool calling protocol execution, multi-turn plan refinement, competitive algorithmic problem solving.
Qwen/Qwen-AgentWorld-35B-A3B	0.25	Language world model dynamics, environment state transitions across MCP, OS, Browser, and Terminal spaces.

---

The Merge Methodology

Standard linear averaging ($\frac{1}{3}A + \frac{1}{3}B + \frac{1}{3}C$) causes severe activation variance shrinkage in high dimensions ($\mathbb{E}[|\sum w_i \theta_i|] < \mathbb{E}[|\theta_i|]$) and degrades router gate logits, leading to routing entropy collapse where arbitrary experts are activated.

To prevent this, parameters were partitioned into structural categories and processed with dedicated mathematical operators:

1. Routed Experts & GQA/DeltaNet Attention (Norm-Balanced Directional Consensus)

For attention projections and all 256 routed MoE expert matrices (gate_proj, up_proj, down_proj), we interpolate along the hypersphere by enforcing directional consensus while strictly conserving the expected Frobenius norm:

$$\mathbf{D}=\sum _{i=1}^3{w}_i{\theta }_i\backslash mathbf\{D\}\; =\; \backslash sum\_\{i=1\}^3\; w\_i\; \backslash theta\_i$$

$${\theta }_{\text{merged}}=\mathbf{D}\cdot \frac{\sum _{i=1}^3{w}_i\mathrm{\parallel }{\theta }_i{\mathrm{\parallel }}_F}{\mathrm{\parallel }\mathbf{D}{\mathrm{\parallel }}_F}\backslash theta\_\{\backslash text\{merged\}\}\; =\; \backslash mathbf\{D\}\; \backslash cdot\; \backslash frac\{\backslash sum\_\{i=1\}^3\; w\_i\; \backslash |\backslash theta\_i\backslash |\_F\}\{\backslash |\backslash mathbf\{D\}\backslash |\_F\}$$

This prevents the experts from under-activating or falling into dead routing zones.

2. MoE Router Gates (Logit Sharpness Preservation)

MoE routing decisions rely on sensitive softmax logits ($x W_{\text{gate}}^T$). A drop in $|W_{\text{gate}}|_F$ artificially raises routing temperature, causing unconfident top-8 allocation. We apply maximum-norm scaling:

$$W_{\text{gate, merged}}=\left(\sum _{i=1}^3{w}_i{W}_{\text{gate},i}\right)\cdot \frac{\underset{i}{\max }\mathrm{\parallel }{W}_{\text{gate},i}{\mathrm{\parallel }}_F}{{\parallel \sum _{i=1}^3{w}_i{W}_{\text{gate},i}\parallel }_F}W\_\{\backslash text\{gate,\; merged\}\}\; =\; \backslash left(\; \backslash sum\_\{i=1\}^3\; w\_i\; W\_\{\backslash text\{gate\},\; i\}\; \backslash right)\; \backslash cdot\; \backslash frac\{\backslash max\_i\; \backslash |W\_\{\backslash text\{gate\},\; i\}\backslash |\_F\}\{\backslash left\backslash |\; \backslash sum\_\{i=1\}^3\; w\_i\; W\_\{\backslash text\{gate\},\; i\}\; \backslash right\backslash |\_F\}$$

3. Normalization Layers & Embeddings

RMSNorm gains, LayerNorm weights, and biases are merged via standard convex weighted averaging:

$${\theta }_{\text{merged}}=\sum _{i=1}^3{w}_i{\theta }_i\backslash theta\_\{\backslash text\{merged\}\}\; =\; \backslash sum\_\{i=1\}^3\; w\_i\; \backslash theta\_i$$

---

Deployment & Serving

Because only ~3.5B parameters are activated per token, the model delivers 35B-class reasoning and coding capacity at the inference throughput of a sub-4B dense model.

1. High-Throughput Serving with vLLM

Recommended execution command with multi-GPU Tensor Parallelism:

vllm serve OliviaRossi/TripleTrouble \
  --tensor-parallel-size 2 \
  --max-model-len 32768 \
  --dtype bfloat16 \
  --trust-remote-code \
  --enable-chunked-prefill \
  --gpu-memory-utilization 0.92

2. SGLang Engine

python3 -m sglang.launch_server \
  --model-path OliviaRossi/TripleTrouble \
  --tp 2 \
  --mem-fraction-static 0.88 \
  --trust-remote-code

3. Hugging Face Transformers Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "OliviaRossi/TripleTrouble"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)

messages = [
    {
        "role": "system",
        "content": "You are an expert autonomous software engineer and tool-calling agent."
    },
    {
        "role": "user",
        "content": "Inspect the repository structure, diagnose why the asyncio event loop deadlocks during worker pool shutdown, and write a unified diff patch."
    }
]

prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=2048,
    temperature=0.2,
    top_p=0.95,
    do_sample=True
)

print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

---

Prompt Template (ChatML)

The model inherits the standard ChatML dialect:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
{user_query}<|im_end|>
<|im_start|>assistant

Tool / Function Calling Format

<|im_start|>system
You are a helpful assistant with access to the following functions:

{
  "name": "execute_bash",
  "description": "Run shell commands in an isolated Linux environment",
  "parameters": {
    "type": "object",
    "properties": {
      "command": {"type": "string"}
    },
    "required": ["command"]
  }
}
<|im_end|>
<|im_start|>user
Check the remaining disk space on /dev/nvme0n1p1.<|im_end|>
<|im_start|>assistant
<tool_call>
{"name": "execute_bash", "arguments": {"command": "df -h /dev/nvme0n1p1"}}
</tool_call><|im_end|>

---

Hardware Requirements

Precision	VRAM Required (Weights Only)	Recommended GPU Configuration
BF16 / FP16	~70 GB	$2 \times$ A100/H100 (80GB) or $4 \times$ RTX 3090/4090 (24GB)
FP8 / INT8	~36 GB	$1 \times$ A100 (80GB) or $2 \times$ RTX 3090/4090 (24GB)
AWQ / GPTQ (4-bit)	~20 GB	$1 \times$ RTX 3090/4090 (24GB) or $1 \times$ A5000

---

Citation & Acknowledgments

This model is a derivative merge of the following works:

• Kwaipilot/KAT-Coder-V2.5-Dev by Kuaishou Kwaipilot Team
• ornith-ai/Ornith-1.5-35B-A3B by Ornith AI
• Qwen/Qwen-AgentWorld-35B-A3B by Qwen Team, Alibaba Cloud