PyramidFormer base
Small transformer encoder (0.8M parameters) over the 101 single-age tokens of a pyramid, pretrained by hiding 30% of ages or one 5-20 year span and reconstructing them. Held-out masked-age MSE (standardised log-share units): 0.0098 after 2000 steps. The similarity, heaping-detector and forecaster models are each fine-tuned from this backbone and are published as separate repos.
Training run
quick run: 2000 optimisation steps.
Training data
UN World Population Prospects 2024, single-age population by sex, countries/areas, 1950-2023 (estimates), as processed in
SallySims/wpp2024-single-age-pyramids. Split by country: 80% train / 10% validation / 10% test,
so every number above is on countries the model never saw. WPP pyramids are modelled estimates, not raw census counts.
Source: United Nations, DESA, Population Division (2024). World Population Prospects 2024. https://population.un.org/wpp/
Check the UN terms of use before reuse. Research prototype; not for official statistics.
This repo is self-contained: pyramidformer.py (model code), model.safetensors, config.json. There is no hosted inference widget because the architecture is custom; use the snippet below.
Usage
from huggingface_hub import snapshot_download
import importlib.util, os
p = snapshot_download("SallySims/pyramidformer-base")
spec = importlib.util.spec_from_file_location("pyramidformer", os.path.join(p, "pyramidformer.py"))
P = importlib.util.module_from_spec(spec); spec.loader.exec_module(P)
model, norm = P.load_pretrained(p)
# start a new task model from this backbone
new = P.init_from_base(P.PyramidFormer(task="forecast"), model) # or "similarity" / "heaping"
- Downloads last month
- -