JunYoungLee/ut-depth-probe-artifacts

Depth-recurrent LM probe artifacts — recurrence 1 to 20

Raw artifacts from running a depth-recurrent (weight-tied) language model far past the recurrence it was trained at, and recording what happens inside it.

The model is ByteDance/Ouro-1.4B: 24 decoder layers applied repeatedly to the same residual stream, total_ut_steps times per token. It is trained at 4 steps. These artifacts push it to 20 and keep every intermediate state.

Research artifacts, published as-is. Working files organised for reproducibility, not a curated release. What they carry is provenance: each group ships the scripts that produced it, and the pins below decide which files may be compared with which.

The headline measurement

Accuracy against forced recurrence depth, full splits, no prefix caching:

loops	1	2	3	4	5	6	7	8	9	10	11	12	14	16	20
GSM8K exact_match,flexible-extract	0.2032	0.6346	0.7589	0.7900	0.7756	0.7415	0.7293	0.7074	0.6717	0.5914	0.5133	0.4253	0.2555	0.1077	0.0167
MATH-500 math_verify,none	0.2440	0.6120	0.7120	0.7320	0.7160	0.7200	0.6900	0.6700	0.6280	0.5340	0.4060	0.2860	0.1480	—	—
AIME24 exact_match,none	0.0000	0.0000	0.1000	0.1000	0.0333	0.0667	0.1000	0.0667	0.0333	0.0333	0.0000	0.0000	0.0000	0.0000	0.0000

GSM8K peaks at loop 4 — the trained depth — and then falls monotonically. There is no plateau and no second peak. By loop 16 it has lost most of its ability, and by loop 20 it is near zero: the generations stop terminating and degenerate into repetition. MATH-500 peaks at the same depth. AIME24 is 30 documents wide and its curve is mostly zeros, so it constrains little on its own.

More recurrence is not more thinking. That is what the tensors here let you look into.

Pinned protocol

model / revision	ByteDance/Ouro-1.4B @ e3b1e0993b1231a51d6069a870476dda4162c00a
dtype / decoding	bfloat16 / greedy, temperature 0
lm-eval	commit 64f3d0924fc695efd6d776a5ac91f97138085516
trained depth	total_ut_steps = 4 — every depth above 4 is extrapolation
prefix caching	off
generation budget	GSM8K 1024, MATH-500 2048, AIME24 4096

Two files may only be compared if these agree. Runs under different generation budgets are not comparable, which is why the budget is stamped into every directory name.

Layout

Three groups, each carrying its own scripts/ and results/, because a folder that points at a code repository instead of shipping its scripts stops being reproducible the moment that repository moves on.

loopedtransformer_probe/          per-loop hidden states           [GPU, hours]
loopedtransformer_forced_depth/   depth -> accuracy sweep          [GPU, minutes per depth]
loopedtransformer_analysis/       smoke and preflight records      [CPU]

loopedtransformer_probe/ is the only expensive thing here. The sweep regenerates in minutes per depth; the tensors are hours of GPU time.

One tensor is stored exactly once

The probe ran in batches so a lost pod could not cost the whole sweep, and on disk each batch's tensors also appeared in a -merged and a -full view — three names for one inode. Uploading that tree as-is would have pushed 271 GB of which 109 GB was the same bytes again.

Here every tensor is stored once, in its benchmark's canonical folder. The batch folders are kept under loopedtransformer_probe/results/shards/ with their own prompt and correctness records, and a README pointing at where their tensors actually live. Nothing is duplicated and nothing is lost.

Known bad files

Two probe tensors from an earlier (loops=6) collection were truncated by a pod loss. They are not in this repository, but analysis code that also reads that older set must catch both and report the reduced n rather than quietly shrinking it: probe-math500-100-loops6-full/tensors/doc94.npz (BadZipFile) and probe-aime24-loops6-full/tensors/doc3.npz (opens, then fails CRC on read — wrapping only numpy.load in try does not catch it).