RonBuildsAI/sleep-form-rescue-count

Sleep-form rescue count: 80 tiny research networks

Author: Ronald Lee Hampton

This repository holds 80 tiny research networks from one measured test, and the code to re-measure them. They are research instruments, not usable language models: each holds 633,074 numbers, and its held-out loss is about 4.6 nats per token.

The result

On 40 fresh seeds, the sleeping form ended with 1 switchboard against the plain form's 7 on the same seeds: 7 seeds rescued and 1 harmed, exact one-sided sign test p 0.0352. That passes a bar frozen before the run, on a thin margin (see "What this does not show").

What was measured

The network. A tiny next-word network trained on short synthetic stories at a vocabulary of 4,096 takes the last two words in. A router sends each word to one of 48 experts, each a small two-layer block, with a sharing pull on the router's choice; a shared head guesses the next word. A reasoning core is read through a small integer part with ternary weights, which writes a gain on the experts' output, zero at the start. Per network: 595,568 numbers in the routed base network, 12,416 in the reasoning core and 25,090 in the integer part.

The two forms. Both train 10,400 steps a run, with a first stage of 10,000 steps.

• ST-0, the plain form.
• S-SLEEP, the sleep form: through that first stage, the integer part's write weights keep updating on the language loss at 1/200 of the active rate, with the reasoning core plugged in and held fixed.

The measure: the shuffle cost. Each network is run on 256 held-out windows (16,384 positions), then run again with the reasoning core's output shuffled across positions, everything else held fixed. The shuffle cost is the mean increase in held-out loss, in nats. A core whose output can be shuffled at almost no cost acts as a fixed switchboard, a constant gain on each feature that carries no work word by word. A core whose shuffle costs more is used position by position, as context.

The line and the bar, both frozen before any of these networks trained, with predictions written and sealed before the freeze (the frozen file's sha256 is 814aef7d8e6fe3ad7ec52777ae532dfbf29917067623baa644fc6556d8e6de94; the bars restated in plain words are in results/BARS_frozen-before-the-run.txt):

• A core is a switchboard if its shuffle cost is below 0.025 nats, otherwise a context worker.
• A rescue is a seed whose plain core is a switchboard and whose sleeping core is a context worker. A harm is the reverse.
• The bar: the sleep form ends with fewer switchboards than the plain form, AND the rescues beat the harms by the exact one-sided sign test on the discordant seeds at p at or below 0.05 (with fewer than 5 discordant seeds, not confirmed).

Measured (every per-network check passed): plain form 7 switchboards of 40 (seeds 873, 878, 885, 886, 893, 894, 896); sleep form 1 of 40 (seed 883). 7 rescues, 1 harm, 8 discordant seeds, p = 9/256 = 0.0352. Confirmed.

The 40 seeds

Shuffle cost in nats, rounded to 5 places (full precision in results/measured_table.csv).

seed	ST-0 shuffle cost	ST-0	S-SLEEP shuffle cost	S-SLEEP	outcome
857	0.17191	context worker	0.21677	context worker
858	0.26488	context worker	0.26176	context worker
859	0.28441	context worker	0.36387	context worker
860	0.20116	context worker	0.24776	context worker
861	0.13320	context worker	0.34882	context worker
862	0.20471	context worker	0.23471	context worker
863	0.30946	context worker	0.30965	context worker
864	0.21082	context worker	0.24767	context worker
865	0.05376	context worker	0.15759	context worker
866	0.19909	context worker	0.25653	context worker
867	0.22104	context worker	0.27287	context worker
868	0.15181	context worker	0.23933	context worker
869	0.18681	context worker	0.28792	context worker
870	0.15455	context worker	0.16191	context worker
871	0.25835	context worker	0.32672	context worker
872	0.32314	context worker	0.31415	context worker
873	0.00008	switchboard	0.06655	context worker	rescue
874	0.30618	context worker	0.18058	context worker
875	0.10390	context worker	0.20586	context worker
876	0.20125	context worker	0.30537	context worker
877	0.21909	context worker	0.23828	context worker
878	0.00757	switchboard	0.05345	context worker	rescue
879	0.24069	context worker	0.27381	context worker
880	0.35534	context worker	0.34479	context worker
881	0.07280	context worker	0.10509	context worker
882	0.23871	context worker	0.24113	context worker
883	0.06976	context worker	0.01231	switchboard	harm
884	0.23074	context worker	0.20481	context worker
885	0.01763	switchboard	0.11119	context worker	rescue
886	0.02389	switchboard	0.17428	context worker	rescue
887	0.25048	context worker	0.28333	context worker
888	0.15390	context worker	0.24058	context worker
889	0.24025	context worker	0.33816	context worker
890	0.05059	context worker	0.26321	context worker
891	0.13865	context worker	0.21745	context worker
892	0.18379	context worker	0.29248	context worker
893	0.00767	switchboard	0.06912	context worker	rescue
894	0.00038	switchboard	0.28474	context worker	rescue
895	0.10236	context worker	0.17191	context worker
896	0.00161	switchboard	0.08419	context worker	rescue

What this does not show

• The margin is thin. Plain seed 886 sat 0.0011 under the line (0.02389). Counted as a context worker, the result would be 6 rescues and 1 harm, p 0.0625, above the bar. Under six different shuffles it held its label (0.0221 to 0.0241); so did seed 885 (0.0176 on the scored shuffle).
• The count depends on the instrument. A second instrument, the spread of the core's output across positions (its line set on earlier networks before any of these trained, no bar), calls 3 plain switchboards (873, 894, 896) and 0 sleeping ones. It agrees with the shuffle line on 75 of 80 cores. Its direction is the same, fewer switchboards in sleep; the size of the switchboard group is not.
• One bench, one length, one measure. Tiny networks that see two words of context, trained at one length, read by one measure on a cut line. It says nothing about larger networks or longer context.
• Not a better network. Held-out loss was not part of this bar. In earlier frozen tests on this bench, a lower end loss for the sleep form was not confirmed.
• Earlier seeds. Pooled with 16 earlier seeds of the same bench (841 to 856, not in this release), 11 seeds rescued and 1 harmed.
• Scored positions. 40 target token ids were never a training target and are left out of the score: 16,288 of the 16,384 held-out positions are scored.
• One step went off plan. After training, the scored reader stopped on an error in a reporting line. It was re-run on the same trained networks with that one line corrected. No network was retrained, and the line, the bar and the count code did not change.

Re-check it yourself

Needs Python 3 with numpy, torch and safetensors. CPU only, one thread, about a minute. From this folder:

python -I reader/read_shuffle_cost.py --brains brains --data data --expect results/per_brain_record.csv

It prints only, nothing is written. It prints each network's shuffle cost and label, the two counts, the rescues and harms, the sign test and the bar, then compares every network with the record. Expected: 7 of 40 and 1 of 40, the same 8 discordant seeds, CONFIRMED, labels equal 80 of 80. The last digits of the costs can differ by machine and torch version. On the machine used to prepare this release the largest difference from the record was 7.25e-08 nats, and on that machine the reader matched the run's own network code bit for bit on all 80 networks (results/reproduction_notes.txt, raw output in results/reproduction_output.txt).

To check the files first: sha256sum -c SHA256SUMS.txt (prints OK per file).

Files

• brains/<form>_s<seed>_L10400_end.safetensors: the 80 networks, one per form and seed, at the end of training. Tensor names keep the run's own prefixes: tiny. the routed base network, rc. the reasoning core, cc. the small integer part; y holds the 48 experts' smoothed sizes that the router's pull reads. Metadata: the run's labels (arm, charge, form, seed, steps, sleep_n) and the sha256 of the original checkpoint. One small two-output head of the integer part (cc.Wv, cc.bv), which the measure does not use, was removed from every file before release; every other tensor is byte for byte the trained one. results/head_removed_record.csv lists each file's hash with and without it, and the reader's results are the same either way (7 of 40 and 1 of 40, all 80 labels equal).
• reader/read_shuffle_cost.py: the measure, forward pass only.
• tools/convert_pt_to_safetensors.py: how each checkpoint became a .safetensors file, with its exactness check.
• data/heldout_windows_uint16.npy, data/excluded_target_ids.txt, data/NOTICE_data.txt: the held-out positions and the 40 excluded target ids.
• results/: the 40-seed table, a per-network record with both hashes, the conversion log, the seed list, the bars, the reader's raw output and the reproduction notes.
• The training script is not released.

Training data

The networks trained on TinyStories by Ronen Eldan and Yuanzhi Li (2023), dataset roneneldan/TinyStories, the GPT-4 version: the training file TinyStoriesV2-GPT4-train.txt (sha256 6418d412de72888f52b5142c761ac21a582f7d1166f0bfbdb5f03ccfdec90443) for training and the validation file for the held-out windows. The dataset card describes it as synthetically generated short stories in a small vocabulary, licensed CDLA-Sharing-1.0.

References

• Ronen Eldan and Yuanzhi Li, TinyStories: How Small Can Language Models Be and Still Speak Coherent English? (2023), arXiv:2305.07759.

Citation

If you use these networks or the measure, please cite:

@misc{hampton2026sleepformrescue,
  author       = {Hampton, Ronald Lee},
  title        = {Sleep-form rescue count: 80 tiny research networks},
  year         = {2026},
  month        = oct,
  howpublished = {Hugging Face: https://huggingface.co/RonBuildsAI/sleep-form-rescue-count},
  note         = {Measured 2026-10-26; released 2026-10-27}
}

Licence and patents

• Weights, this card, the results and the excluded-id list: CC BY 4.0 (LICENSE_weights_CC-BY-4.0.txt). That licence does not license patent rights.
• Code: The Clear BSD License (LICENSE_code_BSD-3-Clause-Clear.txt), which grants no patent rights.
• The held-out windows are a changed excerpt of the dataset above and stay under CDLA-Sharing-1.0 (data/NOTICE_data.txt).
• Patent rights are reserved; provisional application filed 2026-10-27.

Drafted and fact-checked with an AI assistant from my own records, measurements and decisions. I read and edited every sentence and I am responsible for its contents.