docs/benchmarks.md

Benchmarks

The numbers we are chasing on the hero workload, and the framework comparison that goes in the README.

Hero workload

Qwen3-8B SFT, 1× MI300X, bs=8, seq=4096, BF16, AdamW, 1B tokens.

Metric	Target	Why
Throughput	>15 000 tok/s	Comparable to AMD's published Llama-3.1-8B numbers.
MFU	>40 %	Floor for "MI300X is being exercised, not idled."
Time to eval-loss = 1.5	<90 minutes	Lets the demo video show the loop converging in real time.
Total cost	<$3	$1.99 / hr × 1 GPU × ~1.5 hr × safety margin.
Peak HBM	~80 GB	Headroom on MI300X's 192 GB; impossible on H100 80 GB.

Hit those four and the cost slide writes itself: MI300X $1.99/hr × 1 GPU × 1.5 hr ≈ $3 versus H100 $4/hr × 2 GPUs × 4 hr ≈ $32 — 4× cheaper for the same workload, and the H100 baseline can't even fit BF16 at this batch/seq combo without quantization.

H100 cost baseline

The argument the judges remember is "MI300X is 4× cheaper for this exact workload." The cost numbers come from public list prices and need to hold up under questioning.

GPU	$/hr	Memory	Qwen3-8B BF16 bs=8 seq=4096	Cost for 1B tokens
H100 80 GB	$4.00	80 GB	OOM unless bs/seq cut	~$32 (2× GPUs, 4 hr, FP8 fallback)
H200 141 GB	$6.00	141 GB	Fits, ~12k tok/s	~$24 (1× GPU, 4 hr)
MI300X 192 GB	$1.99	192 GB	Fits with headroom, >15k tok/s	<$3 (1× GPU, 1.5 hr)

Framework comparison (the README differentiator)

This is the table the README prints. mindxtrain is the only row with all seven cells filled — that is the elevator pitch.

Framework	One-cmd ROCm 7.2.1 install	MI300X auto-tune	Qwen3.6 day-zero	FP8 via Quark	x402 micropayments	Decentralized fallback	Training-receipt manifest
Axolotl	⚠ (community fork)	✗	✓	△ (torchao)	✗	✗	✗
LLaMA-Factory	✓ (AMD tutorial)	✗	✓	△	✗	✗	✗
Unsloth	✓ (OneClickAMD)	✗	△ (single-GPU)	✗	✗	✗	✗
torchtune	✓ (AMD CI)	✗	✗ (no recipe)	△	✗	✗	✗
Primus	✓ (rocm/primus:v26.2)	✗	✗ (pretrain only)	✓	✗	✗	✗
mindxtrain	✓	✓ (60s AOT)	✓	✓	✓ (Algorand)	✓ (Bacalhau/Akash)	✓ (BLAKE3 + INFT)

Legend: ✓ = supported · ⚠ = supported via community fork · △ = partial / opt-in · ✗ = not supported.

Capturing the numbers

The mindxtrain CLI emits structured logs that map onto the metrics above. The output tree:

runs/<run_id>/
├── config.yaml              # input, BLAKE3-hashed in the manifest
├── autotune_plan.json       # the AOT plan (the differentiator)
├── train.log                # accelerate stdout/stderr
├── metrics.jsonl            # one record per logging step: tok_per_s, mfu, hbm_gb, watts
├── checkpoint/              # HF safetensors + tokenizer, BLAKE3-hashed
├── quantized/               # Quark FP8 PTPC, vLLM-loadable
├── eval.json                # lm-evaluation-harness output
└── manifest.json            # mindxtrain.provenance.Manifest with BLAKE3 hashes

metrics.jsonl is the source of truth for the benchmark numbers. The cost slide is a one-liner over that file: average tok_per_s × seconds × $1.99 / 3600.

Regression detection

eval.regression.threshold_pct: -1.0 in every recipe means fail the run if any benchmark task drops more than 1 percentage point versus the base model baseline. That keeps a fine-tune that improves the target distribution but breaks general capability from being silently published. The baseline JSON is computed once per base model and cached alongside the run; comparison happens via mindxtrain.eval.persona_regression.regression_score and mindxtrain.eval.agenda_regression.regression_score.

What's not measured (yet)

• Energy (kWh per training run) — mindxtrain.operator.telemetry.energy.sample_power_w wraps rocm-smi --showpower (returns 0.0 W gracefully on a CPU dev box). MI300X power baseline is ~750 W under load; a 90-minute run is ~1.1 kWh. Telemetry collection into metrics.jsonl is wired but the dashboard integration is post-hackathon work.
• Multi-node throughput — out of hackathon scope; the mindxtrain receipt manifest accommodates it (hardware.gpus field), and the autotune rccl_probe is the entry point for the multi-node version.