SmolLM-1.7B-base-mla-topk4-rank1024

This repository contains the attention-only incremental weights for an MLAfication conversion of HuggingFaceTB/SmolLM-1.7B. It is not a standalone full-model checkpoint.

Variant

Field	Value
Base model	HuggingFaceTB/SmolLM-1.7B
Base revision	d7449ff7241c863f3e8accc475155f0f97afa011
MLA latent rank	1024
RoPE dimensions per KV head	4
Training checkpoint	step 2400
Training variant	stage1+2-distill-QKV
Delta tensors	168
Delta size	0.67 GiB
SHA-256	0adeacc1259906d2300c2ccf57156be0efb13e99872adf17797910359da89ac9

The training run froze every parameter outside attention and also froze each attention output projection. The uploaded delta therefore contains exactly the parameters matching "attn" in name and "o_proj" not in name; embeddings, MLP, normalization outside attention, output projections, and LM head are omitted. The historical source checkpoint already contains exactly the trainable attention tensors.

Loading

Use the MHA2MLA-V2 implementation to instantiate the MLAfication architecture from the pinned base model, then load model.safetensors with strict=False:

from safetensors.torch import load_file

delta = load_file("model.safetensors")
load_result = mla_model.load_state_dict(delta, strict=False)

The architecture must be patched before loading the delta. Loading this repository directly with vanilla AutoModelForCausalLM.from_pretrained(...) is not supported. config.json contains the layer-wise RoPE indices and sanitized MLAfication metadata.

Files

• model.safetensors: attention-only delta weights
• config.json: base architecture, RoPE indices, and MLAfication metadata
• manifest.json: provenance, byte size, tensor count, and checksum

License

The weights follow the Apache 2.0 license of the SmolLM base model.