Add Mean Validation Dice 0.7161 + validation_summary.json (per-case results from checkpoint_best)

README.md

@@ -33,9 +33,10 @@ This is a single-fold pretrain checkpoint, intended as a starting point for down
 | **Loss** | Dice + Cross-Entropy (nnU-Net default), `batch_dice=True` |
 | **Schedule** | 1000 epochs, polynomial LR decay 0.01 → 0, batch size 2, patch `[80, 192, 160]` |
 | **Hardware** | 1× NVIDIA H100 80GB, ~6h wall-time |
-| **Best EMA Pseudo Dice** | **0.8155** (epoch ~755) |
-| **Expected real test Dice** | ~0.82–0.84 via sliding-window inference |
-| **Comparison** | At the top of published nnU-Net Task06 baselines (0.69–0.78) |
+| **Mean Validation Dice** (per-case, sliding-window) | **0.7161** |
+| **Best EMA Pseudo Dice** (in-training proxy) | 0.8155 (epoch ~755) |
+| **Foreground IoU** (per-case avg) | ~0.59 (from `validation_summary.json`) |
+| **Comparison** | Within published nnU-Net Task06 range (0.69–0.78 across various reports) |
 
 ## Files in this repo
 
@@ -133,17 +134,23 @@ Input images should be CT volumes named with the nnU-Net channel suffix: `<case_
 
 ## Evaluation
 
-| Metric | Value |
-|--------|-------|
-| Best EMA Pseudo Dice (fold 0 validation) | **0.8155** |
-| Pseudo Dice raw (jagged) range | 0.50–0.85 |
-| Final-epoch train loss | -0.85 |
-| Final-epoch val loss | -0.75 |
-| Train/val gap | ~0.10 (mild late-stage overfitting; `checkpoint_best` predates this) |
+Two complementary Dice metrics, both honest, computed on the 13 fold-0 validation cases:
 
-The training plot (`progress.png`) shows a smooth Pseudo Dice climb from 0 → 0.7 in the first ~50 epochs and a slow refinement to 0.81 by epoch ~750. After that, train loss continues to drop while val loss plateaus — this is the overfitting signature, and nnU-Net's best-checkpoint mechanism preserves the pre-overfit weights.
+| Metric | Value | What it measures |
+|--------|-------|------------------|
+| **Mean Validation Dice** (per-case, sliding-window) | **0.7161** | Per-case Dice from full-volume `nnUNetv2_predict` inference on each of the 13 val cases, averaged. **Case-weighted** — every scan counts equally regardless of tumor size. *This is the metric most papers report.* |
+| **Best EMA Pseudo Dice** (in-training) | 0.8155 | Voxel-pooled Dice across validation patches during training. **Voxel-weighted** — large tumors dominate. Used by nnU-Net to select `checkpoint_best.pth`. |
+| Pseudo Dice raw (jagged) range | 0.50–0.85 | (peak per-epoch readings during training) |
+| Final-epoch train loss | -0.85 | Mild late-stage overfitting visible in `progress.png`. |
+| Final-epoch val loss | -0.75 | `checkpoint_best.pth` predates this. |
 
-Note that **Pseudo Dice is voxel-pooled across validation patches**, not per-case averaged. Real test-time Dice (per-case, full-volume sliding-window inference) typically lands 0.5–3% higher than Pseudo Dice — so the 0.8155 number translates to roughly **0.82–0.84 real test Dice**, which we expect to confirm via `nnUNetv2_predict` on the 13 fold-0 validation cases.
+The 0.10 gap between Pseudo Dice (0.8155) and Mean Validation Dice (0.7161) is **smaller than for varied-lesion-size datasets** like NLSTseg or Dataset500 (~0.15 gap there). MSD Task06's tumors are uniformly large (median volume 5.22 cm³), so voxel-pooled and per-case Dice are reasonably close. The smaller a dataset's lesions and the wider the size distribution, the bigger the Pseudo–Mean gap.
+
+The training plot (`progress.png`) shows a smooth Pseudo Dice climb from 0 → 0.7 in the first ~50 epochs and slow refinement to 0.81 by epoch ~750, then mild overfitting (train loss continues to drop, val loss plateaus). nnU-Net's best-checkpoint mechanism preserves the pre-overfit weights — that's the model in this repo.
+
+For comparisons against other methods, **cite the Mean Validation Dice (0.7161)**. Pseudo Dice is useful as an in-training monitoring signal but not for cross-method comparison.
+
+Per-case validation results are available in `validation_summary.json` (Dice, IoU, TP/FP/FN counts per case).
 
 ## Limitations
 

validation_summary.json

@@ -0,0 +1,234 @@
+{
+    "foreground_mean": {
+        "Dice": 0.7161166470256257,
+        "FN": 3705.153846153846,
+        "FP": 3262.3076923076924,
+        "IoU": 0.5904215376842531,
+        "TN": 68156469.07692307,
+        "TP": 14168.384615384615,
+        "n_pred": 17430.69230769231,
+        "n_ref": 17873.53846153846
+    },
+    "mean": {
+        "1": {
+            "Dice": 0.7161166470256257,
+            "FN": 3705.153846153846,
+            "FP": 3262.3076923076924,
+            "IoU": 0.5904215376842531,
+            "TN": 68156469.07692307,
+            "TP": 14168.384615384615,
+            "n_pred": 17430.69230769231,
+            "n_ref": 17873.53846153846
+        }
+    },
+    "metric_per_case": [
+        {
+            "metrics": {
+                "1": {
+                    "Dice": 0.8758524796398066,
+                    "FN": 436,
+                    "FP": 1439,
+                    "IoU": 0.7791259276711038,
+                    "TN": 148627159,
+                    "TP": 6614,
+                    "n_pred": 8053,
+                    "n_ref": 7050
+                }
+            },
+            "prediction_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_results/Dataset502_MSDLung/nnUNetTrainer__nnUNetPlans__3d_fullres/fold_0/validation/lung_006.nii.gz",
+            "reference_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_preprocessed/Dataset502_MSDLung/gt_segmentations/lung_006.nii.gz"
+        },
+        {
+            "metrics": {
+                "1": {
+                    "Dice": 0.8516823071641108,
+                    "FN": 1264,
+                    "FP": 7808,
+                    "IoU": 0.7416782938010763,
+                    "TN": 63141585,
+                    "TP": 26047,
+                    "n_pred": 33855,
+                    "n_ref": 27311
+                }
+            },
+            "prediction_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_results/Dataset502_MSDLung/nnUNetTrainer__nnUNetPlans__3d_fullres/fold_0/validation/lung_010.nii.gz",
+            "reference_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_preprocessed/Dataset502_MSDLung/gt_segmentations/lung_010.nii.gz"
+        },
+        {
+            "metrics": {
+                "1": {
+                    "Dice": 0.6412669953682952,
+                    "FN": 19789,
+                    "FP": 1820,
+                    "IoU": 0.4719595337585221,
+                    "TN": 68116517,
+                    "TP": 19314,
+                    "n_pred": 21134,
+                    "n_ref": 39103
+                }
+            },
+            "prediction_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_results/Dataset502_MSDLung/nnUNetTrainer__nnUNetPlans__3d_fullres/fold_0/validation/lung_033.nii.gz",
+            "reference_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_preprocessed/Dataset502_MSDLung/gt_segmentations/lung_033.nii.gz"
+        },
+        {
+            "metrics": {
+                "1": {
+                    "Dice": 0.8905521818952126,
+                    "FN": 1503,
+                    "FP": 1597,
+                    "IoU": 0.8026985743380856,
+                    "TN": 77578912,
+                    "TP": 12612,
+                    "n_pred": 14209,
+                    "n_ref": 14115
+                }
+            },
+            "prediction_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_results/Dataset502_MSDLung/nnUNetTrainer__nnUNetPlans__3d_fullres/fold_0/validation/lung_034.nii.gz",
+            "reference_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_preprocessed/Dataset502_MSDLung/gt_segmentations/lung_034.nii.gz"
+        },
+        {
+            "metrics": {
+                "1": {
+                    "Dice": 0.8732567870652429,
+                    "FN": 10417,
+                    "FP": 8514,
+                    "IoU": 0.7750273327946,
+                    "TN": 62830412,
+                    "TP": 65217,
+                    "n_pred": 73731,
+                    "n_ref": 75634
+                }
+            },
+            "prediction_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_results/Dataset502_MSDLung/nnUNetTrainer__nnUNetPlans__3d_fullres/fold_0/validation/lung_041.nii.gz",
+            "reference_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_preprocessed/Dataset502_MSDLung/gt_segmentations/lung_041.nii.gz"
+        },
+        {
+            "metrics": {
+                "1": {
+                    "Dice": 0.23342576254096295,
+                    "FN": 3563,
+                    "FP": 2519,
+                    "IoU": 0.13213470319634704,
+                    "TN": 32760992,
+                    "TP": 926,
+                    "n_pred": 3445,
+                    "n_ref": 4489
+                }
+            },
+            "prediction_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_results/Dataset502_MSDLung/nnUNetTrainer__nnUNetPlans__3d_fullres/fold_0/validation/lung_042.nii.gz",
+            "reference_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_preprocessed/Dataset502_MSDLung/gt_segmentations/lung_042.nii.gz"
+        },
+        {
+            "metrics": {
+                "1": {
+                    "Dice": 0.8932495470141486,
+                    "FN": 1696,
+                    "FP": 1073,
+                    "IoU": 0.8070920997631322,
+                    "TN": 59230190,
+                    "TP": 11585,
+                    "n_pred": 12658,
+                    "n_ref": 13281
+                }
+            },
+            "prediction_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_results/Dataset502_MSDLung/nnUNetTrainer__nnUNetPlans__3d_fullres/fold_0/validation/lung_046.nii.gz",
+            "reference_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_preprocessed/Dataset502_MSDLung/gt_segmentations/lung_046.nii.gz"
+        },
+        {
+            "metrics": {
+                "1": {
+                    "Dice": 0.8605891315388522,
+                    "FN": 615,
+                    "FP": 483,
+                    "IoU": 0.7552930688656118,
+                    "TN": 84405881,
+                    "TP": 3389,
+                    "n_pred": 3872,
+                    "n_ref": 4004
+                }
+            },
+            "prediction_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_results/Dataset502_MSDLung/nnUNetTrainer__nnUNetPlans__3d_fullres/fold_0/validation/lung_048.nii.gz",
+            "reference_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_preprocessed/Dataset502_MSDLung/gt_segmentations/lung_048.nii.gz"
+        },
+        {
+            "metrics": {
+                "1": {
+                    "Dice": 0.60389494371985,
+                    "FN": 175,
+                    "FP": 2042,
+                    "IoU": 0.43255694906577935,
+                    "TN": 57143485,
+                    "TP": 1690,
+                    "n_pred": 3732,
+                    "n_ref": 1865
+                }
+            },
+            "prediction_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_results/Dataset502_MSDLung/nnUNetTrainer__nnUNetPlans__3d_fullres/fold_0/validation/lung_059.nii.gz",
+            "reference_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_preprocessed/Dataset502_MSDLung/gt_segmentations/lung_059.nii.gz"
+        },
+        {
+            "metrics": {
+                "1": {
+                    "Dice": 0.8012234295990041,
+                    "FN": 5341,
+                    "FP": 7592,
+                    "IoU": 0.6683676085953126,
+                    "TN": 33515434,
+                    "TP": 26065,
+                    "n_pred": 33657,
+                    "n_ref": 31406
+                }
+            },
+            "prediction_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_results/Dataset502_MSDLung/nnUNetTrainer__nnUNetPlans__3d_fullres/fold_0/validation/lung_065.nii.gz",
+            "reference_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_preprocessed/Dataset502_MSDLung/gt_segmentations/lung_065.nii.gz"
+        },
+        {
+            "metrics": {
+                "1": {
+                    "Dice": 0.7780967340085576,
+                    "FN": 862,
+                    "FP": 3235,
+                    "IoU": 0.636790780141844,
+                    "TN": 63165424,
+                    "TP": 7183,
+                    "n_pred": 10418,
+                    "n_ref": 8045
+                }
+            },
+            "prediction_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_results/Dataset502_MSDLung/nnUNetTrainer__nnUNetPlans__3d_fullres/fold_0/validation/lung_066.nii.gz",
+            "reference_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_preprocessed/Dataset502_MSDLung/gt_segmentations/lung_066.nii.gz"
+        },
+        {
+            "metrics": {
+                "1": {
+                    "Dice": 0.5206827309236948,
+                    "FN": 487,
+                    "FP": 4287,
+                    "IoU": 0.35197502375458123,
+                    "TN": 69722937,
+                    "TP": 2593,
+                    "n_pred": 6880,
+                    "n_ref": 3080
+                }
+            },
+            "prediction_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_results/Dataset502_MSDLung/nnUNetTrainer__nnUNetPlans__3d_fullres/fold_0/validation/lung_070.nii.gz",
+            "reference_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_preprocessed/Dataset502_MSDLung/gt_segmentations/lung_070.nii.gz"
+        },
+        {
+            "metrics": {
+                "1": {
+                    "Dice": 0.48574338085539714,
+                    "FN": 2019,
+                    "FP": 1,
+                    "IoU": 0.32078009414929387,
+                    "TN": 65795170,
+                    "TP": 954,
+                    "n_pred": 955,
+                    "n_ref": 2973
+                }
+            },
+            "prediction_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_results/Dataset502_MSDLung/nnUNetTrainer__nnUNetPlans__3d_fullres/fold_0/validation/lung_079.nii.gz",
+            "reference_file": "/proj/rasool_lab_projects/Maaz/cln-segmenter/data/msd_task06_nnunet/nnunet_preprocessed/Dataset502_MSDLung/gt_segmentations/lung_079.nii.gz"
+        }
+    ]
+}