a-kemeklis/analyzing-reasoning-and-generalization

Analyzing Reasoning and Generalization When Constraining Language Modeling Pretraining Data

Paper, code and results. The paper is paper/Analyzing Reasoning and Generalization When Constraining Language Modeling Pretraining Data.pdf, with its LaTeX source and figures in paper/.

The three models compared:

• TinyStories-33M: roneneldan/TinyStories-33M
• ts300-ph (~7M parameters, placeholders for out-of-lexicon words; paper 1): a-kemeklis/constraining-pretraining-data
• ts300-noph (~7M parameters, same stories without placeholders): a-kemeklis/constraining-pretraining-data-no-placeholders

Word types used throughout: inlex (in-lexicon), oov (real out-of-lexicon), nonsense (invented, pronounceable), unpron (invented, unpronounceable).

Layout

Path	What it is	Paper section
paper/	PDF, main.tex, figures and the scripts that draw them	all
code/harness.py	Loads the three models; scoring and sampling	all
code/probes.py	Word lists and the placeholder converter	3, Appendix A
code/battery.py	Experiment 1 templates and item builder. It contains more skills than the paper reports; the analysis keeps agent tracking, emotion cause and actions (9 templates)	3, 4.1, Appendix B
code/persist.py, code/persist2.py	Experiment 2: story openings and reuse scoring	3, 4.2
code/score_noph.py	Scores the LLM-judge round for ts300-noph	4.3
code/ts300.py	Paper 1 code (model definition, sample and score commands)	2, 4.3
scripts/cleantok.py	Re-encodes text using only token ids seen in training	Appendix C
scripts/rebuild_noph.py, scripts/audit_items.py	Tokenizer audit (needs local clones of the two model repos in hf/)	Appendix C
scripts/rerun.py battery	Re-scores Experiment 1 with the clean encoding	4.1
scripts/rerun_unpron.py, scripts/unpron_v2.py, scripts/unpron_replacements.json	Unpronounceable-word runs, with the q-free word list	4.1
scripts/rerun_hat.py	Object-tracking template change (kept for completeness; object tracking is not reported)	—
scripts/persist_inlex_v2.py, scripts/persist_unpron_v2.py	Experiment 2 runs for in-lexicon and unpronounceable words	4.2
scripts/analyze_v2.py	Recomputes every Experiment 1 and 2 number and the figure data from results/	4.1, 4.2
results/	Every scored item (battery_*rows*.jsonl), every generated continuation (persist2_rows_v2.jsonl), and summary JSONs	4.1, 4.2
results_v1/	Earlier runs that the scripts above start from	—
judge_noph/	Story samples, answer key, all 18 judge replies and computed scores for ts300-noph vs TinyStories-33M	4.3

Reproducing

No models are needed to recompute the numbers and figures from the saved results:

python scripts/analyze_v2.py
cd paper/figs && python make_figs.py && python make_figs_23.py && python make_fig_names.py
python code/score_noph.py

To re-run the models, put the two small models in models/ph and models/noph (the files of the Hugging Face repos above) or set MODEL_DIR. TinyStories-33M downloads automatically. Everything runs on CPU.