alignment-decision-lab/robustness-model-bank

The Pile Information

This model bank is trained on domains from The Pile. The original Pile release has been largely taken down / restricted (mainly over the Books3 component), so not all 22 official components are available today. Of the 22, working data sources exist for 18; the remaining 4 have no confirmed source anywhere in this project.

Tier 1 -- true exact-subset mirrors (timaeus/pile-*):

Component	Source
Pile-CC	timaeus/pile-pile-cc
Github	timaeus/pile-github
PubMed Abstracts	timaeus/pile-pubmed_abstracts
DM Mathematics	timaeus/pile-dm_mathematics
FreeLaw	timaeus/pile-freelaw
ArXiv	timaeus/pile-arxiv
HackerNews	timaeus/pile-hackernews
Enron Emails	timaeus/pile-enron_emails

Tier 2 -- other confirmed working sources (proxy / close-alternative quality):

Component	Source
PubMed Central	datajuicer/the-pile-pubmed-central-refined-by-data-juicer
OpenWebText2	suolyer/pile_openwebtext2
Stack Exchange	flax-sentence-embeddings/stackexchange_title_body_jsonl
USPTO Backgrounds	common-pile/uspto_filtered
Gutenberg (PG-19)	emozilla/pg19
Wikipedia (en)	wikimedia/wikipedia
Ubuntu IRC	common-pile/ubuntu_irc
BookCorpus2	Yuti/bookcorpus (approximate -- likely plain BookCorpus, not confirmed to be specifically "2")
EuroParl	Helsinki-NLP/europarl
YoutubeSubtitles	suolyer/pile_youtubesubtitles

Not available (no source found in this project): Books3, OpenSubtitles, PhilPapers, NIH ExPorter.

The 3 sources used for the KL-DRO model bank below (FreeLaw, PubMed Central, ArXiv) are all Tier 1 or a vetted close alternative.

Robustness Model Bank

KL-DRO-trained checkpoints across model sizes, source datasets, and robustness coefficients, used by diagnostic_experiment/algorithm_2.py and diagnostic_experiment/hierarchical_routing.py for shift-aware model selection and interpolation.

Status: 23/27 gpt2-medium checkpoints trained. See model_bank_metadata.csv for the full grid and per-checkpoint status. Other model sizes (gpt2Tiny/, gpt2Small/, gpt2Large/, gpt2Xlarge/) are placeholders -- no checkpoints trained yet.

Layout

gpt2Medium/          <- gpt2-medium, 23/27 trained
gpt2Tiny/            <- not started
gpt2Small/           <- not started
gpt2Large/           <- not started
gpt2Xlarge/          <- not started

gpt2-medium

Dataset	Lambda	Path	Status
FreeLaw	lambda_0	gpt2Medium/FreeLaw/lambda_0/	trained
FreeLaw	lambda_0.05	gpt2Medium/FreeLaw/lambda_0.05/	trained
FreeLaw	lambda_0.1	gpt2Medium/FreeLaw/lambda_0.1/	trained
FreeLaw	lambda_0.2	gpt2Medium/FreeLaw/lambda_0.2/	trained
FreeLaw	lambda_0.3	gpt2Medium/FreeLaw/lambda_0.3/	pending
FreeLaw	lambda_0.4	gpt2Medium/FreeLaw/lambda_0.4/	pending
FreeLaw	lambda_0.5	gpt2Medium/FreeLaw/lambda_0.5/	trained
FreeLaw	lambda_0.7	gpt2Medium/FreeLaw/lambda_0.7/	trained
FreeLaw	lambda_1	gpt2Medium/FreeLaw/lambda_1/	trained
PubMed Central	lambda_0	gpt2Medium/PubMed_Central/lambda_0/	trained
PubMed Central	lambda_0.05	gpt2Medium/PubMed_Central/lambda_0.05/	trained
PubMed Central	lambda_0.1	gpt2Medium/PubMed_Central/lambda_0.1/	trained
PubMed Central	lambda_0.2	gpt2Medium/PubMed_Central/lambda_0.2/	trained
PubMed Central	lambda_0.3	gpt2Medium/PubMed_Central/lambda_0.3/	pending
PubMed Central	lambda_0.4	gpt2Medium/PubMed_Central/lambda_0.4/	pending
PubMed Central	lambda_0.5	gpt2Medium/PubMed_Central/lambda_0.5/	trained
PubMed Central	lambda_0.7	gpt2Medium/PubMed_Central/lambda_0.7/	trained
PubMed Central	lambda_1	gpt2Medium/PubMed_Central/lambda_1/	trained
ArXiv	lambda_0	gpt2Medium/ArXiv/lambda_0/	trained
ArXiv	lambda_0.05	gpt2Medium/ArXiv/lambda_0.05/	trained
ArXiv	lambda_0.1	gpt2Medium/ArXiv/lambda_0.1/	trained
ArXiv	lambda_0.2	gpt2Medium/ArXiv/lambda_0.2/	trained
ArXiv	lambda_0.3	gpt2Medium/ArXiv/lambda_0.3/	trained
ArXiv	lambda_0.4	gpt2Medium/ArXiv/lambda_0.4/	trained
ArXiv	lambda_0.5	gpt2Medium/ArXiv/lambda_0.5/	trained
ArXiv	lambda_0.7	gpt2Medium/ArXiv/lambda_0.7/	trained
ArXiv	lambda_1	gpt2Medium/ArXiv/lambda_1/	trained

Load a specific checkpoint:

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
    "alignment-decision-lab/robustness-model-bank",
    subfolder="gpt2Medium/<dataset>/<lambda_dir>",
)

Produced by: diagnostic_experiment/models_bank.py, config: configs/diagnostic/models_bank.yaml.