# 01 — Config walkthrough

File: `reference/hub/config.json` · class: `LagunaForCausalLM` / `model_type=laguna`.  
Auto-map: `configuration_laguna.LagunaConfig` · `modeling_laguna.LagunaForCausalLM`.

## Dimensions

| Key | Value |
|---|---|
| `hidden_size` | 3072 |
| `num_hidden_layers` | 48 |
| `vocab_size` | 100352 |
| `max_position_embeddings` | 1048576 (1M) |
| `intermediate_size` | 12288 (dense FFN) |
| `moe_intermediate_size` | 1024 |
| `shared_expert_intermediate_size` | 1024 |
| `rms_norm_eps` | 1e-6 |
| `torch_dtype` | bfloat16 |

## Attention schedule

`layer_types`: pattern **full, swa, swa, swa** × 12 → **12** global + **36** SWA.

| Key | Value |
|---|---|
| `sliding_window` | 512 |
| `num_attention_heads` | 48 (default / global) |
| `num_attention_heads_per_layer` | 48 on global, **72** on SWA |
| `num_key_value_heads` | 8 |
| `head_dim` | 128 |
| `gating` | `"per-head"` |
| `gating_types` | all `"per_head"` (48 entries) |

## RoPE

```json
"rope_parameters": {
  "full_attention": {
    "rope_type": "yarn",
    "rope_theta": 500000.0,
    "factor": 128.0,
    "original_max_position_embeddings": 8192,
    "partial_rotary_factor": 0.5
  },
  "sliding_attention": {
    "rope_type": "default",
    "rope_theta": 10000.0,
    "partial_rotary_factor": 1.0
  }
}
```

## MoE

| Key | Value |
|---|---|
| `num_experts` | 256 |
| `num_experts_per_tok` | 10 |
| `mlp_only_layers` | `[0]` |
| `mlp_layer_types` | dense then 47× sparse |
| `moe_routed_scaling_factor` | 2.5 |
| `norm_topk_prob` | true |
| `router_aux_loss_coef` | 0.0 |
| `moe_apply_router_weight_on_input` | false (unimplemented if true) |
| `moe_router_logit_softcapping` | 0.0 |

Shared expert count is **not** a separate config integer; card + `shared_expert_intermediate_size` + `LagunaSparseMoeBlock.shared_expert` establish **1** shared MLP.

## Tokens

| Key | Value |
|---|---|
| `bos_token_id` | 2 |
| `eos_token_id` | [2, 24] |
| `pad_token_id` | 9 |
| `tie_word_embeddings` | false |

## Next

Open `reference/laguna-code/configuration_laguna.py` then [02-modeling-walkthrough](02-modeling-walkthrough.md).