# 01 — Architecture

All scalar hyperparameters below should match [`derived/SPEC.md`](../derived/SPEC.md). If they disagree, trust `derived/` and re-run `make derive`.

---

## 1. Stack

| Piece | Choice |
|---|---|
| Body | 66-layer decoder-only, `hidden_size=6144` |
| FFN | Dense L0–1 (`I=24576`); MoE L2–65 (`I=3072`) |
| MoE | 256 routed, top-6, 2 shared, sigmoid router, shared-expert sink |
| Attention | Hybrid SWA (window 512, 16 KV) + global (8 KV) |
| Position | Relative bias (`d_rel=16`), **not RoPE** |
| Local mix | SConv kernel 4, **4** states per layer |
| Vision | hMLP, patch 40, temporal 2, 4 layers |
| Audio | dMel → 80 bins × 16 codes |
| Head | μP scale 24; unpadded vocab 200058 of 201024 |
| MTP | 8-layer drafter (separate weights) |

Official size: **975B total / 41B active**. FFN-only accounting in SPEC shows ~936B in expert matrices alone and ~30B active FFN — consistent with the marketing numbers once attention/embeds are included.

---

## 2. MoE (DeepSeek-V3 lineage + sink)

```
logits = W_router h                    # 256 + 2
scores = σ(logits)
idx    = topk(scores_routed + bias, k=6)
weights = normalize_logsigmoid(selected ∪ shared) × route_scale × global_scale
y = Experts(h, idx, w_routed) + SharedExperts(h, w_shared)
```

| Detail | Why it matters |
|---|---|
| Sigmoid + aux-loss-free bias | Load balance without aux loss term |
| Shared-expert sink | Shared experts share **normalized mass** with routed top-k |
| `route_scale=8` | Amplifies routing weights after normalize |
| Extreme sparsity | Capacity at active cost |

Code: `InklingTopkRouter`, `InklingMoE`, `InklingSharedExperts` — see [class index](../derived/class_index.md).

---

## 3. Hybrid attention

From config: **55 SWA + 11 global** (exactly global every 6 layers: 5,11,17,…,65).

| | SWA | Global |
|---|---|---|
| Window | 512 | full |
| KV heads | 16 | 8 |
| Rel extent | 512 | 1024 |
| Mask key | `sliding_attention` | `full_attention` |

**Why:** 1M context is only practical if most layers never materialize full KV. Generated KV estimates: [memory_notes.md](../derived/memory_notes.md).

---

## 4. Relative position (not RoPE)

```
R = r_proj(h)                         # [B,Q,H,d_rel]
profiles = R @ bank                   # bank: [d_rel, rel_extent]
bias = gather(profiles, q_pos-k_pos)  # 0 if distance ∉ [0, extent)
attn_logits = (QK^T)/d + bias + mask
```

| Quirk | Detail |
|---|---|
| Q/K norm | per-head RMSNorm |
| Scale | **`1/d`**, not `1/√d` |
| Long ctx | global: `τ=1+α log(n/n_floor)` for `n>128k`, multiplies Q and bias |
| Flash-Attn | disabled in HF (`position_bias` path) |

Authors’ claim: better long-context extrapolation than RoPE.

---

## 5. SConv

Depthwise causal conv + residual, kernel **4**, fp32 path:

| idx | Site |
|---|---|
| 0 | after K proj |
| 1 | after V proj |
| 2 | attn residual branch |
| 3 | MLP residual branch |

```
h ← h + sconv₂( Attn(LN(h)) )
h ← h + sconv₃( MLP(LN(h)) )
```

Cheap local structure; frees attention/MoE for longer-range work. Decode keeps 4 conv states/layer (small vs KV).

---

## 6. Multimodal early fusion

No heavy external encoders.

| Modality | Front-end | Soft token id |
|---|---|---|
| Text | embed + embed_norm | — |
| Image | 40×40 patches → hMLP → 6144 | `200054` / `<\|unused_200054\|>` |
| Audio | 16 kHz → log-mel → 16-way dMel quant → embed | `200053` / `<\|unused_200053\|>` |

Still images are shaped with **temporal dim 2** (`temporal_patch_size`). Audio hop in processor config: **0.05 s (50 ms)** per frame.

Fusion is `masked_scatter` of tower outputs onto placeholder positions in the text embedding stream — [03-multimodal-io.md](../code-study/03-multimodal-io.md).

---

## 7. Logits

```
logits = lm_head(h / 24)[..., :unpadded_vocab_size]
```

Embed ⟂ unembed (not tied). MTP weights live separately and are ignored on main model load.

---

## 8. What to memorize

1. **55/11** SWA/global, global every 6th from 5  
2. **L0–1 dense**, rest MoE  
3. **6/256 + 2 shared**, sigmoid + sink  
4. **Relative R**, scale `1/d`  
5. **4 SConvs** per layer  
6. **hMLP + dMel** early fusion  

Next: [02-training.md](02-training.md) · code: [modular walkthrough](../code-study/01-modular-walkthrough.md)