# 01 — Architecture

All scalar hyperparameters should match [`derived/SPEC.md`](../derived/SPEC.md). If they disagree, trust `derived/` (from Hub config) and re-run `make derive`.

---

## 1. Stack

| Piece | Choice |
|---|---|
| Body | 48-layer decoder, `hidden_size=3072` |
| FFN | Layer 0 dense (`intermediate_size=12288`); layers 1–47 MoE |
| MoE | **256** routed, top-**10**, **1** shared; expert width 1024 |
| Attention | 12 global + 36 SWA (window **512**), ≈ 1:3 |
| Heads | Global 48 Q / 8 KV; SWA 72 Q / 8 KV; `head_dim=128` |
| Norms | RMSNorm; **QK-norm** per head before RoPE |
| Position | YaRN on full attn; default RoPE on SWA |
| Attn gate | Per-head **softplus** before `o_proj` |
| Context | **1,048,576** |
| Vocab | 100,352 |

Official size: **118B total / ~8B active**. BF16 weights ≈ **236 GB** (card).

---

## 2. MoE (code truth)

Hub card: “token-choice router with softplus gating over 256 routed experts plus one shared expert.”

**Code** (`LagunaTopKRouter` / `LagunaSparseMoeBlock`):

```
logits  = W_router h                         # [T, 256]
scores  = σ(logits)                          # sigmoid, not softmax
idx     = topk(scores + bias, k=10)          # aux-loss-free bias on selection only
weights = gather(scores, idx); maybe renorm  # norm_topk_prob=true
y_routed = Experts(h, idx, weights) × 2.5    # moe_routed_scaling_factor
y        = y_routed + SharedMLP(h)           # shared is ADD, not sink mass
```

| Detail | Why it matters |
|---|---|
| Sigmoid router | Distinct from softmax MoE; matches DeepSeek-style aux-free bias path |
| Softplus on **attention** gate | Do not confuse with router scoring |
| Shared expert **add** | Unlike Inkling’s shared-expert **sink** (normalized mass with top-k) |
| `routed_scaling_factor=2.5` | Amplifies routed expert output after combine |
| Dense layer 0 only | `mlp_only_layers=[0]` / first `mlp_layer_types=dense` |

Code: `LagunaTopKRouter`, `LagunaExperts`, `LagunaSparseMoeBlock` — [class index](../derived/class_index.md).

---

## 3. Hybrid attention

From `layer_types`: **full, swa, swa, swa** × 12 → global indices  
`0, 4, 8, 12, 16, 20, 24, 28, 32, 36, 40, 44`.

| | SWA | Global |
|---|---|---|
| Count | 36 | 12 |
| Window | 512 | full |
| Q heads | 72 | 48 |
| KV heads | 8 | 8 |
| RoPE | default θ=10k | YaRN θ=5e5, factor 128, orig 8192, partial 0.5 |
| Mask | sliding causal | full causal |

**Why:** 1M context is only practical if most layers never materialize full KV. Memory notes: [memory_notes.md](../derived/memory_notes.md).

Attention path: Q/K/V proj → reshape → **QK RMSNorm** → RoPE → attn → **softplus gate** → `o_proj`. Scaling is **`1/√d`** (unlike Inkling’s `1/d`).

---

## 4. Position encodings

| Layer type | Config key | Type |
|---|---|---|
| Global | `rope_parameters.full_attention` | `yarn` |
| SWA | `rope_parameters.sliding_attention` | `default` |

YaRN: `factor=128`, `original_max_position_embeddings=8192`, `partial_rotary_factor=0.5` (only half of head_dim rotated on global layers). SWA uses full rotary (`partial=1.0`).

`LagunaModel` keeps **two** rotary modules and builds separate masks for full vs sliding layers.

---

## 5. Thinking (product contract, not weights)

Native interleaved thinking; best with **preserved** `reasoning_content` across multi-turn / tool calls. Toggle: `enable_thinking` via chat template kwargs. Modes at launch: **off** | **max** (default max).

Serve details: [03-serve-and-thinking](../code-study/03-serve-and-thinking.md).

---

## 6. Compare to Inkling / Kimi

| | Laguna S 2.1 | Inkling | Kimi K3 |
|---|---|---|---|
| Total / active | 118B / 8B | 975B / 41B | 2.8T / ? |
| Experts | 10/256+1 add | 6/256+2 sink | 16/896 |
| Layers | 48 · 1:3 G:SWA | 66 · 11:55 G:SWA | TBD |
| PE | YaRN + SWA RoPE | relative (not RoPE) | KDA lineage |
| Attn scale | 1/√d | 1/d | TBD |
| Modality | text | image+audio | vision+video |
| License | OpenMDW-1.1 | Apache-2.0 | API-first |