File size: 16,629 Bytes
dfb775d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 | # Development workflow
Conventions and invariants for working in this repo. Read once before opening a PR.
## Toolchain
- **Python 3.12** (`>=3.12,<3.13`) — pinned; matches `rocm/primus:v26.2`.
- **uv** — single project (no workspace). `uv sync` installs the base deps;
`uv sync --extra <group>` adds optional groups.
- **ruff** — replaces black/isort/flake8/pyupgrade. Config in
[`pyproject.toml`](../pyproject.toml).
- **mypy --strict** — only on `mindxtrain/config` and `mindxtrain/provenance`
(the schemas + manifest paths). Training / eval code is exempt.
- **pytest** + `pytest-asyncio` — fast unit tests; GPU tests are manual on
the MI300X.
- **Foundry** — Solidity contracts in `contracts/`. Installed on the MI300X
droplet for the on-chain anchoring path.
## Optional dependency groups
`pyproject.toml` defines six `[project.optional-dependencies]` groups:
| Group | Adds |
|---------|-------------------------------------------------------------|
| `ml` | trl, transformers, peft, accelerate, datasets |
| `eval` | lm-eval, lighteval, inspect-ai, jinja2 |
| `data` | datasketch, sentence-transformers, faiss-cpu, pyarrow |
| `serve` | vllm |
| `chain` | web3, py-algorand-sdk, huggingface-hub |
| `obs` | opentelemetry-sdk, prometheus-client, psutil |
Plus `all` which pulls everything except `amd-quark` (which ships in the
rocm/primus container, see [HANDOFF.md](HANDOFF.md) §3).
The base install (no extras) is enough for: the CLI, the Coach UI, the
autotune dry-run, manifest verify, the operator FastAPI app, and every
in-process Python utility (registry, hot-swap, agent loop, ContextManager,
data filter, sequence packing). See
[actualization_status.md](actualization_status.md) for the per-module map.
## Lazy-import pattern
Every module that wants an optional dep guards the import inside the
function that needs it:
```python
def run_lm_eval(model_dir: Path, tasks: list[str]) -> Path:
if not _lm_eval_available():
msg = "lm-eval not installed; run `uv sync --extra eval`."
raise RuntimeError(msg)
... # subprocess wrap that uses the dep
```
Two implications:
1. `import mindxtrain.eval.harness` always succeeds even without `--extra eval`.
2. The error message includes the exact `uv sync --extra <group>` to run.
This is the canonical pattern; new modules that take optional deps must
follow it.
## The standard local cycle
```bash
uv sync # base install
uv run ruff check --fix . # lint + auto-fix
uv run mypy mindxtrain/config mindxtrain/provenance # types where strict
uv run pytest -q # → 564 passed in ~5s
```
CI runs the same four commands on Ubuntu 24.04 / Python 3.12 (CPU-only).
See [`.github/workflows/ci.yml`](../.github/workflows/ci.yml).
## Repository layout
```
.
├── pyproject.toml # single project; optional-dep groups
├── README.md # entry doc (the only root .md besides CLAUDE/AGENTS)
├── CLAUDE.md, AGENTS.md # agent-tooling entrypoints (required at root)
├── NOTICE, LICENSE-* # legal
├── Containerfile, compose.yaml # podman entry points
├── docs/ # all documentation (index: docs/NAV.md)
│ ├── NAV.md # docs index
│ ├── HANDOFF.md # operator checklist
│ ├── dcoach.md # the proof loop + decentralized fit
│ ├── CHANGELOG.md
│ └── … # architecture, coach, governance, decentralized, reference
├── mindxtrain/ # the package — 12 subpackages, ~99 modules
│ ├── cli/ # typer CLI (9 verbs)
│ ├── config/ # 10-section Pydantic schema + JSON defaults
│ ├── data/ # curate → dedupe → filter → tokenize → pack → synth → verify
│ ├── models/ # registry + chat templates + 5 base presets
│ ├── train/ # sft, dpo, grpo, rlhf, tool_use, distributed, callbacks, recipes/
│ ├── eval/ # lighteval / inspect_ai / bfcl / persona / agenda / card
│ ├── autotune/ # 60s AOT probe — the differentiator
│ ├── operator/ # FastAPI app, Coach UI, ml-intern patterns
│ ├── storage/ # local_fs / hf_hub / lighthouse / ipfs
│ ├── provenance/ # manifest, hashing, verify, erc8004, algorand, x402
│ ├── deploy/ # registry, hot_swap, ab_test, vllm_launcher, quark
│ └── budget/ # ResourceBudget + cloud-provider stubs
├── contracts/ # Foundry workspace (ERC-8004 attestation)
├── ops/ # containerfiles, compose, k8s, vmm, gensyn
├── examples/ # demo YAMLs
├── tests/ # pytest — 566 tests (CPU-only smoke)
└── docs/
├── *.md # current state (this directory)
└── blueprints/ # source design briefs (frozen)
```
## Reuse boundaries
- **From `/home/hacker/mindX/`** (production codebase): Codephreak persona
JSON loaded at runtime via `MINDXTRAIN_PERSONA_PATH`. Do not copy file
bytes — load via env var.
- **Not** from `/home/hacker/aglm/` — broken per its own README. Use only
for reference to legacy class names mindxtrain2.md flagged as needing
refactor.
## Invariants
These are non-negotiable; violating them is a deployment bug, not a style
preference.
1. **AOT-only.** No `torch.compile(mode="max-autotune")` in production paths.
No JIT autotune in vLLM serving (set `VLLM_USE_TRITON_FLASH_ATTN=0` if
needed). The `autotune.policy: aot_only` field in the YAML is the
contract; tested at
`tests/test_config_schema.py::test_qwen3_8b_sft_lora_validates`.
2. **`hardware.gpus: 1 | 8` only.** 2/4-GPU MI300X FSDP groups hit
asymmetric xGMI; the schema rejects them at parse time. Tested at
`tests/test_config_schema.py::test_xgmi_2gpu_rejected` and
`tests/test_distributed.py`.
3. **Seven MI300X env vars in `train.env`** (defaults, can be overridden by
the autotune plan but never removed): `HSA_NO_SCRATCH_RECLAIM=1`,
`NVTE_CK_USES_BWD_V3=1`, `NVTE_CK_IS_V3_ATOMIC_FP32=1`,
`PRIMUS_TURBO_ATTN_V3_ATOMIC_FP32=1`, `NCCL_MIN_NCHANNELS=112`,
`HIP_FORCE_DEV_KERNARG=1`, `PYTORCH_ROCM_ARCH=gfx942`.
4. **`extra: forbid` on every Pydantic model.** Unknown YAML keys raise
`ValidationError`. Tested at
`tests/test_config_schema.py::test_extra_field_forbidden`.
5. **Configs are immutable once loaded** (`frozen: true`).
6. **Solidity contracts: no proxies, no `Ownable`, no admin keys, no setters.**
`mindxtrain_registry.sol` is write-once. Rotating any parameter requires a
fresh deploy.
7. **Lazy imports for optional deps** — see the pattern above.
## Training lanes (CPU / local-GPU / MI300X)
Three ways to actually run a fine-tune, selected by `train.backend`:
| Lane | Backend | Device | When |
|------|---------|--------|------|
| CPU | `trl_cpu` | CPU, float32 (in-process TRL) | mindX self-training, smoke runs, no GPU |
| Local GPU | `trl_local` | auto: CUDA/ROCm GPU (bf16/fp16) else CPU fallback | consumer Radeon RX / NVIDIA RTX, or a laptop |
| MI300X | `axolotl`/`unsloth`/`torchtune`/`primus` | gfx942 subprocess + 7 env vars | the AOT MI300X target |
`trl_local` is the **device-aware** in-process lane (`backend_trl_cpu.py::run_trl_local`):
it picks the GPU when `torch.cuda.is_available()` (ROCm surfaces through the same API),
else logs `no accelerator detected → CPU fallback` and runs on CPU. The same recipe
(`mindx_fallback_qwen3_1_5b_local`) therefore runs unchanged on a gaming GPU or a laptop.
`trl_cpu` is `run_trl_local(..., force_cpu=True)`; `MINDXTRAIN_FORCE_CPU=1` forces the
fallback anywhere. The in-process lanes never inject the seven MI300X env vars.
Confirm which device a box will use:
```bash
uv run python -c "import torch; print(torch.cuda.is_available(), torch.version.hip)"
```
**Unsupported:** integrated Vega/RDNA APUs (e.g. Ryzen "Raven"/`gfx90c`) are not ROCm
targets and fall back to CPU. A discrete RX 6800/7900 (`gfx1030`/`gfx1100`) or any NVIDIA
RTX is the intended consumer GPU.
## Adding a new recipe
1. Drop a YAML at `mindxtrain/train/recipes/<name>.yaml`. Validate locally:
```bash
uv run python -c "from mindxtrain.config.loader import load_config; load_config('mindxtrain/train/recipes/<name>.yaml')"
```
2. The `tests/test_config_schema.py::test_all_recipes_validate` test will
pick it up automatically — re-run pytest.
3. Add a row to [docs/yaml_schema.md](yaml_schema.md) only if the recipe
exercises a previously-unused field.
## Adding a new training backend
1. Add `mindxtrain/train/backend_<name>.py` exposing a
`run_<name>(cfg, plan, out_dir) -> Path` function (or for in-process TRL
trainers, a `run_<name>(cfg, out_dir) -> Path` function).
2. Wire it into `mindxtrain/train/dispatch.py`'s `if backend == ...` ladder.
3. Add `<name>` to the `TrainingBackend` literal in
`mindxtrain/config/schema.py`.
4. Update [docs/cli.md](cli.md) "Where the verbs live" table.
## Adding a new model backend (operator)
1. Add `mindxtrain/operator/backends/<name>.py` with a `Backend` subclass
decorated `@register_backend("<name>")`.
2. Side-effect import it from `mindxtrain/models/registry.py` so registration
runs on package import.
3. Add a runtime branch in `mindxtrain/operator/app.py::chat_completions` for
the env-var-driven kwargs.
## Adding a new training method
1. Define a `_MethodBase` subclass in `mindxtrain/config/schema.py` with
`kind: Literal["<name>"] = "<name>"` and the method-specific fields.
2. Add it to the `TrainMethod` discriminated union.
3. Add a `mindxtrain/train/<name>.py` runner (TRL wrap or subprocess).
4. Update the dispatch path so a YAML with `train.method.kind == "<name>"`
reaches the runner.
5. Add a recipe under `mindxtrain/train/recipes/` exercising it.
6. Update `docs/yaml_schema.md` "train.method" table.
## Adding a new optional-dep group
1. Add the entry to `[project.optional-dependencies]` in `pyproject.toml`.
2. Add a row to the table in [actualization_status.md](actualization_status.md).
3. Update [development.md](development.md) and [quickstart.md](quickstart.md).
## Adding a new doc
1. Write `docs/<name>.md`.
2. Add a one-line entry to [`docs/NAV.md`](NAV.md) under the appropriate section.
## Live training UI
The Coach UI's "Train" step (`#step-train` in
[`coach/static/index.html`](../mindxtrain/operator/coach/static/index.html))
launches a training run and streams loss / lr / log lines back into the
browser over Server-Sent Events. Architecture:
- **Registry**: `mindxtrain.operator.runs.RunRegistry` is an in-memory
singleton (one per uvicorn process) keyed by `run_id`. Snapshots are
immutable `Run` records (frozen Pydantic); state changes produce new
snapshots via `model_copy`.
- **Event schema**: `TrainEvent` is a tagged union over `StatusEvent`,
`StepEvent`, `EvalEvent`, `LogEvent`, `EnergyEvent` — all with
`extra="forbid", frozen=True`. Wire format: `event: <kind>\ndata:
<event.model_dump_json()>\n\n`.
- **Two ingestion paths**, deduped by `(run_id, step)` in
`RunRegistry.publish`:
1. Subprocess stdout regex (`parse_trainer_log_line`) — works on the
base install, parses HF Trainer's `'loss': … 'learning_rate': …`
log lines.
2. In-process `mindxtrain.train.callbacks.StreamCallback` — POSTs to
`/coach/api/runs/{id}/ingest` (loopback only). Requires `--extra ml`.
- **Subprocess orchestration**: `spawn_subprocess_streaming` uses
`subprocess.Popen(stdout=PIPE, bufsize=1, text=True)` and tees lines
to both `train.log` (the durable on-disk artifact) and
`RunRegistry.publish_threadsafe` from a daemon thread. We use
`Popen` (not `asyncio.create_subprocess_exec`, not `BackgroundTasks`)
so the child outlives the launch HTTP request and `SIGINT`-then-`SIGTERM`
cancellation matches the CLI Ctrl-C path.
### Routes
All under `/coach/api/runs`:
| Verb | Path | Purpose |
|---|---|---|
| POST | `/launch` | Spawn a run; returns `Run` immediately. 503 if `accelerate` is missing. |
| GET | `/` | List active + last 20 runs. |
| GET | `/{id}` | `Run` snapshot. |
| GET | `/{id}/events` | SSE — all event kinds. Replays last 200 buffered on connect. |
| GET | `/{id}/logs` | SSE — `kind="log"` only. |
| POST | `/{id}/cancel` | `SIGINT` then `SIGTERM` after grace. |
| POST | `/{id}/ingest` | Loopback-only — used by `StreamCallback`. |
SSE responses set `Cache-Control: no-cache`, `X-Accel-Buffering: no`,
`Connection: keep-alive` so reverse proxies don't buffer the stream.
### Invariants
- `import mindxtrain.operator.runs` succeeds **without** `--extra ml`. The
in-process `StreamCallback` requires `transformers`; the subprocess-stdout
path does not. UI degrades gracefully.
- `Run` and every `*Event` are `frozen=True, extra="forbid"`.
- The subprocess line reader runs in a daemon thread; events reach the
asyncio loop via `loop.call_soon_threadsafe(registry.publish, …)`.
### Frontend
Vanilla JS, no build step. Live view uses the browser-native `EventSource`:
```js
const es = new EventSource(`/coach/api/runs/${id}/events`);
es.addEventListener("step", e => pushPoint(JSON.parse(e.data)));
es.addEventListener("log", e => appendLog(JSON.parse(e.data)));
es.addEventListener("status", e => updateBadge(JSON.parse(e.data)));
```
**Chart.js is vendored locally** at `coach/static/vendor/chart.umd.min.js`
(pinned to v4.4.0; SHA256 in `coach/static/vendor/VERSIONS.md`). No CDN
dependency at demo time. If the vendored bundle is missing, the page
degrades to a metrics table — `coach.js` checks `typeof Chart === "undefined"`
and shows the table-only fallback.
### Why not Selenium / WebSocket / Streamlit
- **Selenium** is a browser-test framework, not a UI library — it
can't push live data into a browser. (It might appear later as CI
smoke for the dashboard; that's E2E testing, not UI.)
- **WebSocket** is bidirectional; we don't need browser→server streaming.
Held in reserve for v2 "edit hyperparam mid-run."
- **Streamlit / Gradio** each spin up their own ASGI server on a separate
port, which breaks the single-URL operator demo and the lazy-import
invariant. SSE on the existing `:8080` is the right shape.
## Common debugging
| Symptom | Cause |
|------------------------------------------------|----------------------------------------------------------------------------------------------------|
| `ModuleNotFoundError: No module named 'mindxtrain'` | Forgot `uv sync`. Fixed by `uv sync`. |
| `RuntimeError: ... not installed; run uv sync --extra <group>` | Optional dep gating — install the named group. |
| `pydantic.ValidationError: extra keys not permitted` | YAML has a typo or stale field name. Compare to [yaml_schema.md](yaml_schema.md). |
| `ValueError: MI300X xGMI permits only 1 or 8 GPUs` | `hardware.gpus` is 2 or 4. Use 1 or 8. |
| `Failed to download due to network timeout` (uv) | `UV_HTTP_TIMEOUT=120 uv sync`. |
| First-iteration training is 30s slow on MI300X | Cold AITER / MIOpen / Triton caches. Volume-mount `~/.cache/miopen`, `AITER_JIT_DIR`, `TORCH_EXTENSIONS_DIR`. |
| `vllm serve` stalls on first batch | Triton autotune cold-start. Set `VLLM_USE_TRITON_FLASH_ATTN=0` or warm-up batch in `mindxtrain serve`. |
## What not to commit
- `*.safetensors`, `*.bin`, `*.pt`, `*.onnx` (large model weights).
- `out/`, `runs/`, `checkpoints/` (run outputs).
- `.env` (use `.env.example`).
- `contracts/lib/` (Foundry submodules — pulled with `forge install`).
- `.venv/`, `.uv-cache/`, `.cache/`.
All of the above are in [`.gitignore`](../.gitignore).
|