TamAko783/anomalies-diffusion-qmr

QMR: a calibration-aligned training objective for brain-MRI anomaly detection (negative result)

Research only. Not a medical device. This model did not work; it is released so the failure is reproducible.

Experimental Machines. We tested whether a restoration network trained on synthetic lesions, whose voxel score is trained to clear the healthy-quantile threshold that a healthy-only conformal procedure later uses ("QMR", quantile-margin restoration), would keep lesion detection alive at a healthy-calibrated operating point. It did not. Companion repos: andi-60k, dae-baseline, calibration.

What was tested

Five detectors, all trained on 319 healthy IXI T2 scans (128x128 slices), thresholded using healthy scans only (conformal risk control at alpha 0.05, 100 calibration scans), and evaluated on 20 MS lesion scans (MSLUB, a different scanner vendor): the DAE baseline, ANDi (60k steps), and three new arms sharing one backbone and one synthetic-lesion generator: SR (restoration residual), SR+BCE (plus a score head trained with class-balanced BCE on the synthetic masks) and QMR (the score head trained so healthy voxels fall below a learned 0.995-quantile tau and synthetic lesions rise above tau plus a margin). Also tested: per-voxel normative standardisation of maps against healthy scans (CSS), and choosing checkpoints/thresholds from synthetic MS-like lesions inserted into healthy scans instead of real lesion labels.

Results (n = 20 test scans; exploratory, the test cohort had been seen before)

Mean Dice at the healthy-calibrated threshold	raw	with CSS
DAE	0.023	0.052
ANDi 60k	0.021	0.023
SR	0.0004	0.0006
SR+BCE	0.0007	0.0017
QMR (proposed)	0.000	0.000

• The proposed objective failed (H1 not supported). QMR minus BCE: -0.0007 (95% CI -0.0021 to -0.000005). During training tau rose to 1.0 and stayed, i.e. healthy scores saturated; Dice was exactly 0 on all 20 scans. Maps are cached as float16, so a numerical contribution to the saturation is not excluded (float32 recomputation was not done). Voxel AP was 0.063 for QMR versus DAE 0.057 (descriptive, no CI): the ranking is not worthless, the saturated scores just break thresholding.
• Normative standardisation helped the DAE: +0.029 Dice (95% CI 0.013 to 0.048; BH-adjusted p = 0.001); no detectable effect for ANDi.
• Synthetic-lesion selection did not replace real labels: versus selection with real lesion labels, Dice at the Dice-optimal threshold was lower by 0.031 (DAE), 0.028 (ANDi) and 0.053 (SR+BCE), all with CIs excluding zero; the pre-specified margin was -0.03. For ANDi both rules chose the same checkpoint. SR and QMR "pass" the margin only because their Dice is about 0.
• Raw-score threshold transfer was negative: every raw-score arm is below the pre-registered Dice floor of 0.03; the DAE with standardisation (0.052) is the one exception.
• The healthy false-positive check (G0) passed for all 15 variants at alpha 0.05 and 0.10 on 86 healthy test scans. That is a healthy-data check; it says nothing about validity on other scanners or lesion scans.

Follow-up (qmr-v2, exploratory): what caused the failure, and a fix attempt

Third look at the same 20 MSLUB scans; nothing here is confirmatory. Aggregates are in v2/summary.json.

• Why the exact zero: QMR's scores were pinned at the top of the float16 sigmoid range (thresholds 1.0 and 0.99951). Re-scoring the same checkpoints as float32 logits gives finite thresholds with the healthy false-positive check intact, but Dice stays about 0.002: the ceiling explains the exact zero, not the poor discrimination.
• Training replay (one seed): the learned threshold tau trailed the healthy 99.5th-percentile score at every logged step while both rose toward 1, then saturated (matches the pre-stated "ratchet" criterion in this run only).
• Pre-specified fix (one-way partial-AUC on logits) failed: its logits collapsed to about 0; MSLUB Dice 0.0001 vs 0.021 for BCE on logits (difference -0.021, 95% CI -0.058 to -0.001).
• BCE on float32 logits reached Dice 0.029 and 0.014 in two seeds (DAE 0.023), but the same objective on the original seed re-scored as logits gave 0.0015, so the recovery is within seed-to-seed spread and its cause is not isolated. Voxel AP 0.058 vs ANDi 0.089.
• Synthetic-lesion checkpoint selection was nearly flat across checkpoints (AP 0.001-0.005): no evidence it can replace real lesion labels for these heads.

Limitations

n = 20; one lesion type; vendor shift between IXI (training) and MSLUB (test) confounds every lesion number; analysis plan is internal (not an independent registration) and the test cohort was examined earlier, so all comparisons are exploratory; the synthetic-lesion generator encodes MS-specific assumptions; affine-only registration limits per-voxel statistics.

Files

best_model.safetensors (QMR arm, 7.8 MB; the pre-specified arm, not chosen on test results), report.pdf, aggregate_report.json, best_model.json. Code is not published yet; the architecture is the DAE U-Net plus a 3-layer convolutional head.

Project results, time and cost (all experiments)

All runs use Runpod: one Pod at a time, one GPU per Pod (an NVIDIA L4, or an equivalent fallback GPU when the L4 was unavailable), controlled from a laptop. "Billed" amounts are from Runpod's billing records for the specific Pods; "est." amounts are wall-clock time x list price where the Pod could not be itemised.

Time and cost

#	Stage	Date	Pod time	Cost (USD)	Basis
1	DAE baseline: preprocess, train 20k steps, benchmark, upload	Nov 1	~42 min	~0.41	est.
2	ANDi 20k steps: train, benchmark, paired comparison with DAE	Nov 1	~1 h 50 min	~1.08	est.
3	Healthy-only calibration audit (DAE, ANDi-20k)	Nov 1	~1 h 11 min	0.68	billed
4	ANDi continued to 60k steps, benchmark	Nov 1-2	~3 h 16 min	1.94	billed
5	qmr-v1 study, first Pod (hit its 4.5 h runtime cap during evaluation)	Nov 2	4 h 30 min	3.26	billed
6	qmr-v1 two resume attempts that exited at once (stale finished-job marker)	Nov 2	~6 min	0.15	billed
7	qmr-v1 resume under a fresh marker (completed)	Nov 2	~51 min	0.57	billed
8	qmr-v2: float32 audit, training replay, pAUC-vs-BCE fix study	Nov 3	~2 h 58 min	1.61	billed
	Total itemised (rows 1-8)		~15.4 h	~9.70	8.21 billed + 1.49 est.

Not itemised and therefore not in the total: several short code-sync Pods (a few cents each), earlier development Pods run before the one-Pod-at-a-time rule was added (some were duplicates), and storage on the shared network volume (Runpod bills the whole volume, which is shared with other uses; its share for this project was not measured). Cost of a single scan at inference on an L4 at list price: ANDi about 4.9 s per volume ($0.0008); DAE 0.23 s ($0.00004).

Results

All lesion results use 20 held-out MS scans (MSLUB, a different scanner vendor from the IXI training data) and are exploratory after the first look.

Experiment	Headline result	Strength
DAE baseline (not a diffusion model)	Voxel AP 0.079, Dice 0.107, scan AUROC 0.600, 0.23 s per volume	benchmark
ANDi diffusion model, 20k steps	AP 0.084, Dice 0.120, AUROC 0.701, 4.84 s per volume	benchmark
ANDi, 60k steps	AP 0.090, Dice 0.125, AUROC 0.707; +0.0056 AP vs 20k (CI +0.002 to +0.010); vs DAE AP +0.011 (CI -0.036 to +0.056, not significant)	small gain; ~21x the DAE's cost
Healthy-only conformal calibration (guarantee)	False-positive rate inside the expected band on 86 healthy test scans for every detector tested	solid (healthy data only)
Group bias audit (site, age, sex)	No confirmed group disparity; one uncorrected nominal signal (ANDi, age 40-59 vs <40)	underpowered
Healthy-calibrated threshold on MS lesions	Dice 0.012 (ANDi-20k) and 0.032 (DAE) in the audit; ~0.02 for raw scores of every detector in qmr-v1	consistent across runs; likely scanner shift (untested)
qmr-v1: calibration-aligned objective (QMR)	Failed: Dice 0.000 at the healthy-calibrated threshold; worse than its BCE ablation (-0.0007)	clear
qmr-v1: per-voxel normalisation (CSS)	DAE Dice 0.023 -> 0.052 (+0.029, CI +0.013 to +0.048); no detectable effect for ANDi	one clear positive, n = 20
qmr-v1: synthetic lesions for model selection	Not a substitute for real labels (Dice 0.03-0.05 lower); selection signal nearly flat	exploratory
qmr-v2: float32 audit	Float16 sigmoid ceiling explains the exact zero (thresholds became finite) but not the poor detection (Dice ~0.002)	clear
qmr-v2: training replay	Learned threshold trailed the healthy tail while both saturated (one seed)	supports the failure mechanism in that run
qmr-v2: pre-specified fix (partial-AUC)	Failed: collapsed to a constant score; MSLUB Dice 0.0001 vs 0.021 for BCE on logits	clear
qmr-v2: BCE on float32 logits	Dice 0.029 and 0.014 (mean 0.021; DAE 0.023); original seed re-scored gave 0.0015	exploratory; seed spread as large as the effect