Stale model.safetensors on revision branches silently returns the same weights for many checkpoints
Working on a checkpoint study with pythia-2.8b, I found that from_pretrained(..., revision="stepN") returns identical weights across many different checkpoints. Reporting it here in case it affects others and in case the repository can be corrected.
I have only checked pythia-2.8b. The other sizes in the suite (70m, 160m, 410m, 1b, 1.4b, 6.9b, 12b) appear unaffected.
What happens
Many revision branches carry a model.safetensors whose SHA-256 is identical across branches:
step0 single=ab496f1c3fd79e3c shard1=dee700d6e97069cd
step1 single=ab496f1c3fd79e3c shard1=763509284d151a6f
step2 single=ab496f1c3fd79e3c shard1=ade39f01d5e9d646
step1000 single=ab496f1c3fd79e3c shard1=f6a6c2f8f40825a5
step16000 single=ab496f1c3fd79e3c shard1=28af6ff1fea4e0ea
...
The single file is the same object on every branch (size 5,684,693,096). The sharded files differ at every checkpoint, as genuine checkpoints should.
Where both are present, from_pretrained loads the single file, so the revision argument has no effect on the weights returned.
Note the stale file's size also differs from the genuine final checkpoint (step143000, 5,550,463,728), so it does not appear to be a copy of the final model either.
Demonstration
Same probe, greedy decoding, three checkpoints, loaded two ways:
checkpoint naive revision= load loaded from sharded files
step1000 The capital of France is Paris, and the capital of the United The capital of France is a term of the present invention.
step16000 The capital of France is Paris, and the capital of the United The capital of France is Paris.\n\nThe capital of the
step50000 The capital of France is Paris, and the capital of the United (no sharded files on this branch)
The naive column is byte-identical across 49,000 training steps. The sharded column shows the expected progression — grammatical but factually empty at step1000, correct at step16000.
Availability across the repository
Scanning all step* branches:
range state
step0 – step25000 sharded files present, unique per checkpoint — usable
~step26000 – step53000 stale single file only — no genuine weights available
step54000 – step142000 mostly unique single-file SHAs — usable
Within the third region there is a second repeated SHA, 462f2b960062159c, shared across roughly ten branches near step54000–step64000 and also present on step143000.
What did not work
from_pretrained(..., revision="step0")
deleting the local cache and passing force_download=True
use_safetensors=True
All three returned the stale file.
Workaround
Download the sharded files by explicit filename into a clean directory and load from there:
python
import os, shutil, torch, transformers
from huggingface_hub import hf_hub_download, HfApi
def load_checkpoint(rev, model_id="EleutherAI/pythia-2.8b", workdir="/tmp/ckpt"):
shutil.rmtree(workdir, ignore_errors=True); os.makedirs(workdir, exist_ok=True)
files = HfApi().list_repo_files(model_id, revision=rev)
shards = sorted(f for f in files if '-of-' in f and f.endswith('.safetensors'))
need = (["config.json", "model.safetensors.index.json"] + shards) if shards
else ["config.json", "model.safetensors"]
for f in need:
shutil.copy(hf_hub_download(model_id, f, revision=rev), os.path.join(workdir, f))
return transformers.AutoModelForCausalLM.from_pretrained(workdir, dtype=torch.float16)
Why it is easy to miss
Nothing in downstream numbers flags it. In my case a measurement taken on the false step0 fell comfortably inside the noise band I had already established for the trained model — the value looked entirely reasonable. Only a generation sanity check revealed the problem.
A study plotting any metric across these checkpoints without this workaround would see a flat curve through early training followed by a change at the endpoint, which reads as a transition and would be an artifact of file resolution.
Suggested precaution for anyone using these branches: log a fixed-probe generation at every checkpoint and inspect it, rather than trusting revision= to have taken effect.
Questions
Are the sharded files the canonical weights, or the single files?
Can the missing checkpoints in the ~step26000–step53000 range be restored?
What is 462f2b960062159c, given it appears on both step143000 and several branches near step54000–step64000?
Happy to provide the scanning script or run further checks.