Ced-Collab/uef-scaling-a2

UEF scaling curve — campaign A2 (d12 / d15 / d28)

Trunk-size scaling rungs for a unified embedding field trained jointly over frozen understanding representations (SigLIP2 so400m-patch14-224 image states + flan-t5-small text states). Every rung is the same recipe at the same data and batch — only the trunk width/depth changes.

These are research checkpoints, not a product release. Nothing here is adjudicated: the numbers below are logged validation losses, not a verdict on the shape of the scaling curve.

Rungs

rung	hidden	depth_double	heads	trainable params	steps released
d12	768	10	12	178,573,312	10k · 20k · 25k · 40k · 45k · 50k · best_val
d15	960	13	15	334,866,688	10k · 20k · 25k · 40k · 45k · 50k
d28	1792	26	28	2,121,404,416	10k · 15k · 20k · 25k

All rungs share text_preamble_depth: 2, head_dim: 64, patch_size: 14, pca_channels: 128, time_cond: in_context, time_tokens: 4.

Shared recipe (identical across rungs)

• global batch 4096, lr 4e-4, 25k steps then extended to 50k
• bf16 autocast, fp32 master weights + fp32 AdamW moments, fused AdamW
• dual EMA (slow + fast), both kept fp32
• union pretrain pool 38.4M pairs, text window 128, wds_shard_seed 4242
• 4 nodes x 4 H200, world size 16

Validation loss (logged, final step of each segment)

rung	step	long	short	jdb	img_t2i (long)
d12	25,000	0.6827	0.6681	0.6059	0.5852
d12	50,000	0.6680	0.6539	0.5968	0.5729
d15	25,000	0.6263	0.6125	0.5708	0.5339
d15	50,000	0.6090	0.5951	0.5609	0.5198
d28	25,000	0.5559	0.5443	0.5513	0.5446

Provenance

Each rung's 0→25k segment and its 25k→50k continuation are separate Slurm jobs; the continuation auto-resumes from checkpoint_025000.pt.

rung	segment	run id	slurm job	commit
d12	0→25k	20260818-190255-a2-d12	40149805	d124497
d12	25k→50k	20260821-032904-a2ext-d12	40509242	fba6fa4
d15	0→25k	20260819-065052-a2-d15	40149806	1bd9777
d15	25k→50k	20260821-152650-a2ext-d15	40509287	fba6fa4
d28	0→25k	20260819-183555-a2-d28	40149811	1bd9777

Every file additionally carries its own identity block (experiment id, segment id, parent segment id, config/dataset identity sha256, world size, global batch) and an export_provenance block naming the exact source checkpoint it came from.

Contents of a checkpoint

Weights-only export — the optimizer state and RNG state have been stripped, so these load for evaluation but are not resumable.

import torch
ck = torch.load("d15/checkpoint_050000.pt", map_location="cpu", weights_only=False)

ck["model"]      # raw trained weights          (fp32)
ck["ema"]        # slow EMA                     (fp32)
ck["ema_fast"]   # fast EMA                     (fp32)
ck["config"]     # full training config
ck["identity"]   # run provenance
ck["representation_manifest"]  # frozen repr specs + schedules
ck["step"], ck["best_val_loss"], ck["export_provenance"]

Tensors are bitwise identical to the training checkpoints they were cut from; the export only drops keys, it does not cast or repack.

d12/best_val.pt is that run's lowest-validation checkpoint, step 47,000 — a genuinely different point from its step-50,000 checkpoint.

d15 has no best_val.pt. Its best-validation step was 50,000, and the file was verified bitwise identical to d15/checkpoint_050000.pt, so the duplicate 4.02 GB of weights is not published. The validation numbers it carried are preserved in d15/best_val_metrics.json.

Caveats — read before using these in a comparison

1. Single seed. One run per rung. No seed repeats, so rung-to-rung gaps carry no error bars.
2. d28 is incomplete. Its 25k→50k continuation was still running when this was published; only the 0→25k segment is here. d28/checkpoint_015000.pt is a rolling checkpoint that the live job would otherwise have deleted.
3. Segmented continuation. Resuming reseeds a fresh global data permutation, so the ≥25k segment is not sample-order-aligned with a hypothetical single 50k run.
4. JourneyDB pool repack. The jdb shards are repacked copies with 14 bad image members and their 14 text partners dropped (28 members total); the jdb manifest count is 4,197,986. Composition-identity against the other site's copy of the pool was not closed.
5. d24 is absent — that rung had not produced a usable checkpoint.
6. Validation losses above are read from the run logs, are computed on small val batches, and are the training-time metric only. No FID / GenEval / DPG numbers are attached to these rungs.

Configs

configs/f25k_scale_d{12,15,28}.yml are the exact configs used, verbatim.