Tier B run history and errata — tierb-b200-gbs8

The HuggingFace repository accumulates every upload, and this campaign was restarted several times on 2026-08-26 while real bugs in the capacity-selection logic were found and fixed. Result files from those earlier attempts are still present alongside the good ones.

Only results whose ECHO capacity_attempts.tsv contains measure and validate rows are valid. Everything below explains how to tell them apart and why the earlier ones are wrong.

Local result directories were overwritten in place by each rerun, so the local tree holds only the newest data. HuggingFace does not overwrite, so it holds all of it, which is why this file exists.

---

How to tell a valid run from an invalid one

The capacity tag in a directory name (__cap2p00, __cap1p25) is not sufficient. cap2p00 means two different things depending on when it was written: in the old algorithm it was the selected timing capacity, in the current one it is only the measurement point used to observe peak demand.

Use these instead, in capacity_attempts.tsv / capacity_selection.tsv:

signal	meaning
ECHO rows with purpose=measure and purpose=validate	current, valid
ECHO rows with purpose=natural	old selection, invalid
required_capacity exactly 1.0 for every model	fabricated, invalid
required_capacity like 1.0293, varying per model	measured, valid
any cell tagged __cap1p02	crashed before iteration 0, invalid
any cell tagged __cap1p75	abandoned descent search, ignore

---

What was wrong, in the order it was found

1. UltraEP crashed on every model — unexecutable capacities (__cap1p02)

The selection rounded the measured requirement up to the next 0.01. UltraEP's dispatch buffer is int(capacity * seq * mbs * topk) and the kernel asserts it is a multiple of 128, so 1.02 produced 66846 tokens and every rank died before iteration 0:

num_permuted_tokens (66846) must be a multiple of pad_multiple (128)

On this model matrix seq*topk is a large power-of-two multiple of 128, so one 0.01 capacity step is 128/25 of a boundary: only capacities that are multiples of 0.25 are executable at all. The 0.01 rule could essentially never work here.

__cap1p02 cells contain no measurement. They are crash evidence only.

2. QWEN2_57B lost the fp32 router it had always been measured with

Until commit f39dedd, --moe-router-dtype fp32 was passed unconditionally. That commit made it per-model and set QWEN2_57B to the ModelZoo-faithful empty value, which UltraEP cannot run — it asserts on the probs dtype at ultra_ep.cpp:909. Restored to fp32, which is also the configuration every previously published QWEN2_57B number was taken under.

3. A MoonEP OOM halted the whole campaign

MoonEP does not fit QWEN3_235B or QWEN3_30B at GBS8. The Stage 7 gate treated that identically to a crash and aborted the study for all three backends, which contradicts the runbook's own rule that an OOM is data and only BADFLAG/ASSERT are bugs. An OOM is now recorded as DOES_NOT_FIT for that one (model, backend) pair and the campaign continues.

This is a result, not an error. Those two DOES_NOT_FIT rows are real findings and should be reported as such.

4. ECHO's required_capacity was not a measurement

summarise_echo derived the requirement from [MOE_DROP_DEBUG][setup_metadata], which reports what the dispatcher expects before routing: its exact_recv_per_rank is the perfectly uniform [32768]*8. The requirement came out as exactly 1.0 for every model at every capacity — the uniform lower bound, structurally unable to see the imbalance the probe exists to measure.

The observed data is in the [dispatch] rows (actual_valid_recv), the same fields the older summarize_token_drop_log.py keyed on. Reading those gives per-model requirements of 1.023–1.055 instead of a flat 1.0, and makes ECHO's number directly comparable with UltraEP's 1.0101: both are now peak actual demand over the uniform share.

Any result showing ECHO required_capacity = 1.0 predates this fix.

5. ECHO selected the highest capacity that fit, not the lowest that worked

The old loop stopped at the first rung that did not OOM, so 2.00 was chosen for all four models on the first attempt and nothing lower was tried. Because UltraEP is required to time at ECHO's capacity, that inflated the receive buffer for both backends. ECHO now follows UltraEP's procedure — measure, round up to an executable value, validate zero drops there — and the common timing capacity is max(ECHO, UltraEP) per model.

Effect: the timing capacity is 1.25, not 2.00.

6. Drop detection reported "zero drops" exactly when tokens were dropped

observed_loss is expected_recv - actual_valid_recv, not a drop count. When the buffer genuinely saturates, actual_valid_recv is clipped at budget, so observed_loss goes strongly negative and max(0, observed_loss) evaluates to zero. Every earlier zero-drop verdict was therefore produced by a test that could not detect the failure it was checking for.

It was masked because every capacity probed until then was heavily oversized. It became load-bearing as soon as the selection started probing downward toward the saturation point. Drops are now detected by buffer saturation (actual_valid_recv >= budget), which survives clipping.

---

What this means when reading the results

• Selected capacity does not discriminate the backends. ECHO needs ~1.03 and UltraEP ~1.01, but the 128-alignment forces both onto the same 0.25 grid, so both select 1.25. The meaningful comparison is the measured required_capacity, not the selected one.
• Both backends are over-provisioned by roughly 21% at C=1.25 against a real requirement near 1.03. That is a property of the kernel's alignment, not of the routing, and memory results should say so rather than describing the buffer as right-sized.
• Probe peaks taken at different capacities are not comparable. Early ECHO numbers were measured at C=2.00 and UltraEP at C=1.25; the valid comparison is the Stage 4 timing runs, which use one common capacity per model.

---

Provenance

Every step writes provenance.txt next to its results with the driver, the CUDA/NCCL/NVSHMEM versions and the commit of each repository. The fixes above correspond to these commits on granularity-base:

e3f7c0b  Tier B probe: pick capacities UltraEP can execute, and keep QWEN2_57B on fp32
4e4081f  Tier B: an OOM excludes one model on one backend, not the whole campaign
         Measure ECHO's real capacity requirement, and search for the lowest zero-drop rung
         Select ECHO's capacity the way UltraEP does, and time both at max(ECHO, UltraEP)
         Detect ECHO token drops by buffer saturation, not by observed_loss

A result predates a fix if its provenance.txt records a blackwell_run_scripts commit older than the corresponding one above.