Zhongzhu/Inkling-GDN-Linearized

Inkling-975B, 6 layers linearized to Gated DeltaNet (fused checkpoint)

Derivative of thinkingmachines/Inkling (Apache-2.0): the attention of layers [56, 58, 61, 62, 64, 65] (including full-attention layer 65) is replaced with GatedDeltaNet (64 heads × 128, gated, short-conv), trained by end-to-end KL distillation against the frozen base as teacher (1 800 steps, seq-128, 64×H100 EP). Layers were selected by zero-ablation importance ranking (least important first). All other weights are byte-identical to the base model.

eval (Inkling-tokenized wikitext)	teacher (base)	this model	gap
PPL @seq-128	7.15	7.288	+1.9%
PPL @seq-2048 (16× the training length)	2.478	2.526	+1.9%

Format

Self-contained: the 109 original shards (native naming, unchanged) + model-gdn-linearized.safetensors (keys model.layers.{L}.linear_attn.*) + patched model.safetensors.index.json. config.json carries an xorl_linearization section describing the spliced layers.

Usage

Loadable with the xorl in-tree Inkling implementation (xorl/models/transformers/inkling/, togethercomputer/xorl-internal PR #511): build InklingConfig with linear_attn_layers=[56,58,61,62,64,65], linear_attn_type="gdn", kda_num_heads=64 and point weights_path at this repo — the bundled InklingCheckpointHandler maps native keys (incl. gate/up de-interleave and expert EP-sharding) and passes the GDN keys through.