RLSplat — random-allocation pretraining, 40k steps (4x B200, accumulate 2)
Run outputs/resilient/b200_random_up504_4gpu_accum2 (2026-09-11 .. 09-13),
recipe +training=rlsplat_b200_random_up504 at commit fe60687: mixed
RE10K + DL3DV, dynamic 2..24 context views (24 views per GPU per step),
context images upsampled to 504x504 for the DVLT backbone, random
per-anchor Gaussian allocation with a Beta(1,1) per-view budget in
[1024, 16384]. 4 GPUs with trainer.accumulate_grad_batches=2, i.e. the
same effective batch as the earlier 8-GPU runs. Trained from scratch
(backbone from nvidia/dvlt). wandb: https://wandb.ai/know/RLSplat/runs/14a90mi1
This is the base checkpoint for GRPO allocation-policy finetunes
(+training=rlsplat_softmax_grpo model.encoder.pretrained_weights=<this file>).
Files
| file | what |
|---|---|
epoch_0-step_40000.ckpt |
full Lightning checkpoint (weights, optimizer, LR schedule, loop state, data-stream positions) |
config.yaml / overrides.yaml |
composed Hydra config and command-line overrides of the final attempt |
launcher.log |
the resilient launcher's log: every (re)start, the step-18225 hang and its diagnosis |
Validation (fixed 24-scene RE10K yardstick, 2 context views, budgets 1024..16384 x7)
val/v02 PSNR / LPIPS / SSIM, compared with the previous 8-GPU run of this
recipe (b200_random_up504_zipview, wandb y6vbhocn lineage):
| step | this run | 8-GPU zipview run |
|---|---|---|
| 16k | 21.60 / 0.232 / 0.720 | 21.41 / 0.231 / 0.707 |
| 20k | 20.82 / 0.237 / 0.688 | 22.20 / 0.216 / 0.744 |
| 24k | 21.77 / 0.223 / 0.730 | 22.31 / 0.208 / 0.750 |
| 28k | 22.03 / 0.218 / 0.734 | 22.73 / 0.204 / 0.760 |
| 32k | 22.25 / 0.207 / 0.751 | 22.16 / 0.218 / 0.738 |
| 36k | 22.83 / 0.200 / 0.769 | 22.23 / 0.218 / 0.745 |
| 40k (this file) | 22.34 / 0.212 / 0.747 | 22.44 / 0.211 / 0.749 |
The original random 40k checkpoint (b200_random_up504, the base of the
GRPO sweeps) scores 22.77 PSNR on the same yardstick. Validation before
step 13k in the wandb run used a random single scene per round and is not
comparable.
Run history
- Started from scratch 2026-09-11 20:00 KST; switched to the yardstick
validation at step 13000; stalled at step 18225 when rank 0 finished its
RE10K pass first and left the DDP lockstep (fixed by
data_loader.infinite_train_stream, commit 01cebcd); resumed from the step-18000 checkpoint with exact data-stream continuation and finished 2026-09-13 11:48 KST without further interruption.
- Downloads last month
- 11