ShwayW/shorter-formulae

Shorter Formulae: transformer checkpoints for symbolic regression

This repository holds the weights of the six transformers from our symbolic-regression paper: the main model, the same model before fine-tuning on noisy targets, and four ablation models. Each model maps a set of input/output pairs to a formula in Polish notation. Constants are then refined with BFGS at inference time.

Models

Each model lives in its own folder, <name>_res/.

Folder	Params	Description
145M_80_simp1_float_ftnoise_48h_res	145M	Our transformer ("ours" in every figure and table). Formulae up to 80 tokens, simplified targets, further trained for 48 h on noise-corrupted targets (ε ~ U(0, 0.1), half the formulae noise-free, peak LR 5e-5).
145M_80_simp1_float_res	145M	The same transformer before fine-tuning on noisy targets; the fine-tuning above resumes from it.
prefac_on_t120M_res	89M	Ablation: affine transformation on, simplification on
prefac_on_nosimp_t120M_res	89M	Ablation: affine transformation on, simplification off
prefac_off_t120M_res	89M	Ablation: affine transformation off, simplification on
simp_off_t120M_res	89M	Ablation: affine transformation off, simplification off

The two 145M models store their weights in fixed_weights.pth. The four ablation models store theirs in model_fixed_Van_checkpoint.pth. The ablation models were trained from scratch on up to 120M data instances.

Naming. Folder names read <params>_<max formula length>_simp<0|1>[_float][_ftnoise_...]:

• simp1: targets simplified to canonical form
• float: constants not restricted to the symbolic set
• ftnoise: further trained on noise-corrupted targets
• prefac: the affine transformation

Usage

Download all six models into the checkpoints/ folder of the code repository:

pip install -U huggingface_hub
hf download ShwayW/shorter-formulae --local-dir checkpoints

To fetch a single model instead, pass its folder:

hf download ShwayW/shorter-formulae --include "145M_80_simp1_float_ftnoise_48h_res/*" --local-dir checkpoints

Then evaluate a model with the code repository's evaluation script:

python eval_mymodels.py --model 145M_80_simp1_float_ftnoise_48h                          # SRBench (Feynman)
python eval_mymodels.py --model 145M_80_simp1_float_ftnoise_48h --benchmark llmsrbench   # LLM-SRBench

Each .pth file is a PyTorch checkpoint dictionary. It holds the model configuration (model_config), the weights (Van_state_dict) and, where saved, the token vocabulary (vocab). load_model() in eval_mymodels.py rebuilds the model from it.

Citation

A citation will be added once the arXiv preprint is available.