NetoAISolutions

/

TSLAM-4B

like 23

Follow

NetoAI Solutions Private Limited 71

Hi @mcglance , thanks for flagging this and for the clean repro. You're right that it's a rope_scaling problem that breaks loading before generation ever runs. We reproduced it on an A100 with the real 4-bit weights and have shipped a fix. Here's exactly what was going on.

Root cause (a small refinement on the summary): the key isn't actually missing. original_max_position_embeddings is present, just at the top level (4096) rather than nested, so the "missing key" error is a bit of a misnomer. The real trigger is that our config was still using the legacy Phi-3 RoPE alias "type": "su". On the Transformers 5.x line (we reproduced this on 5.14.1), RoPE standardization resolves su → longrope, but from what we can tell it doesn't carry the top-level original_max_position_embeddings into the nested rope params before it validates longrope. That stale alias is really what trips it, surfacing as:

KeyError: "Missing required keys in `rope_parameters` for 'rope_type'='longrope': {'original_max_position_embeddings'}"

Worth noting the boundary looks like 5.x, not ≥4.51: stock 4.51.3 and 4.57.6 load the original config fine (their Phi-3 validator normalizes su internally), while we saw the break on 5.14.1. So it reads as a 5.x RoPE-standardization change rather than anything from 4.51 onward.

Fix (shipped): one line in config.json, rope_scaling.type "su" → "longrope", keeping the top-level original_max_position_embeddings: 4096, weights untouched. Commit: https://huggingface.co/NetoAISolutions/TSLAM-4B/commit/0615bbf4aab2049d7b1f0952cddcee1bf6ce7c9d

We looked hard at the fix you suggested (nesting original_max_position_embeddings inside rope_scaling), and it's a completely sensible first move; it does load on 5.14.1. The catch is it breaks 4.51.3/4.57.6: on those versions that config raised a ValueError ("must have exactly three fields"), since their validators expect only type / short_factor / long_factor. Of the options we tried, the su → longrope rename was the only one that loaded cleanly across every version we tested.

Verified on A100 (NF4 4-bit, real weights), fresh download after the fix. Config-validate + 4-bit load + generation all pass:

(These are the versions we ran individually, spanning both the 4.x and 5.x lines; intermediate 5.x point releases weren't each tested. Full validated stack: torch 2.13.0+cu130, bitsandbytes 0.49.2, accelerate 1.14.0.)

To pick up the fix, please force a re-fetch. The broken config.json is likely cached locally, so a plain from_pretrained may silently reuse the old file:

from huggingface_hub import snapshot_download
snapshot_download("NetoAISolutions/TSLAM-4B", force_download=True)
# or pass force_download=True to AutoConfig/AutoModel.from_pretrained(...)
# or delete ~/.cache/huggingface/hub/models--NetoAISolutions--TSLAM-4B

Quick confirm it's resolved:

from transformers import AutoConfig
AutoConfig.from_pretrained("NetoAISolutions/TSLAM-4B", force_download=True)  # no more KeyError

Any transformers in the tested set works; for the smoothest path we'd suggest pinning transformers>=4.57. One caveat: peak memory was ~2.5-3.6 GiB in these short-generation runs. This model supports up to 131072 tokens, so the KV cache grows substantially near full context; budget extra VRAM (or cap max_position / use cache offloading) if you run at long context.

Let us know if anything still comes up. Thanks again, genuinely helpful report. 🙏