# 02 — Modeling walkthrough

Source: [`reference/laguna-code/modeling_laguna.py`](../reference/laguna-code/modeling_laguna.py)  
Line numbers: [`derived/class_index.md`](../derived/class_index.md) (regenerate after refresh).

**Rule:** study the vendored Hub modeling; do not invent layers that are not in the file.

---

## 1. Class map

| Class | Role |
|---|---|
| `LagunaRMSNorm` | T5-style RMSNorm |
| `LagunaRotaryEmbedding` | RoPE / YaRN via `ROPE_INIT_FUNCTIONS` |
| `LagunaMLP` | SwiGLU-style dense FFN (also used as shared expert) |
| `LagunaTopKRouter` | Sigmoid scores + top-k + optional bias |
| `LagunaExperts` | Fused 3D expert weights |
| `LagunaSparseMoeBlock` | Router + experts + shared add |
| `LagunaAttention` | GQA, QK-norm, RoPE, softplus gate |
| `LagunaDecoderLayer` | Pre-norm attn + residual + pre-norm MLP |
| `LagunaModel` | Stack, dual RoPE, dual masks |
| `LagunaForCausalLM` | LM head + generation |

---

## 2. Router (`LagunaTopKRouter`)

```
h → Linear → logits
logits optional soft-cap (tanh) if moe_router_logit_softcapping > 0
scores = sigmoid(logits)
idx = topk(scores + e_score_correction_bias, k)
weights = gather(scores, idx);  # bias does NOT enter weights
if norm_topk_prob: renorm weights over k
```

- Scoring is **sigmoid**, not softmax (docstring is explicit).  
- `e_score_correction_bias`: aux-loss-free load balance (arXiv:2408.15664 style); selection only.  
- Card “softplus gating” is **not** this path — see attention.

---

## 3. Sparse MoE (`LagunaSparseMoeBlock`)

```
shared = SharedMLP(h)                    # LagunaMLP, intermediate=shared_expert_intermediate_size
routed = Experts(h, idx, weights)
if scale != 1: routed *= moe_routed_scaling_factor   # 2.5
out = routed + shared
```

| Contrast | Laguna | Inkling |
|---|---|---|
| Shared expert | **add** after routed | **sink** (share normalized mass with top-k) |
| Router | sigmoid | sigmoid + sink semantics differ |
| Scale | on routed output | route_scale after normalize |

`moe_apply_router_weight_on_input=True` raises `NotImplementedError`.

---

## 4. Attention (`LagunaAttention`)

1. Project Q, K, V (no bias)  
2. Reshape; **RMSNorm Q and K** per head  
3. Apply rotary from layer-type position embeddings  
4. Cache update if past  
5. Attention (`scaling = head_dim ** -0.5`)  
6. If gating: `gate = softplus(g_proj(h))` — **per-head** broadcast across head_dim  
7. `o_proj`  

| Config | Effect |
|---|---|
| `gating="per-head"` | one gate per head (S 2.1) |
| `gating` true / `"per-element"` | one gate per channel |
| `gating` false | no `g_proj` |

SWA layers may have optional attention sinks if `swa_attention_sink_enabled` (default off in this config).

Per-layer head counts come from `num_attention_heads_per_layer[layer_idx]` (72 on SWA, 48 on global).

---

## 5. Decoder layer

Standard pre-norm residual:

```
h = h + Attn(LN(h))
h = h + MLP(LN(h))   # dense or MoE
```

MLP choice: dense if `layer_idx in mlp_only_layers` else sparse when `num_experts > 0` and sparse step matches.

---

## 6. Model forward (dual PE + dual masks)

`LagunaModel` builds:

- Sliding-window causal mask for SWA layers  
- Full causal mask for global layers  
- Separate rotary embeddings for full vs sliding `rope_parameters`  

Each `LagunaDecoderLayer` receives the matching mask and position embeddings for its `attention_type`.

---

## 7. Checks

1. Softplus appears on **attention gate**, not router  
2. Shared expert is **added**, not mass-shared with top-k  
3. Global layers: every 4th starting at 0  
4. Attn scale is **1/√d**, not 1/d