Under The Hood: Inkling from Thinking Machines

Thinking Machines's Inkling is a ~1-trillion-parameter open-weights multimodal Mixture-of-Experts model: 975B total parameters, 41B active per token, a 1M-token context window, and native text, image, and audio inputs. It shipped on July 15, 2026 under Apache 2.0, with full BF16 weights and an NVFP4 deployment checkpoint on Hugging Face.

The headline benchmarks are competitive among open-weights models—97.1% on AIME 2026, 77.6% on SWE-bench Verified, 79.8% on IFBench, 91.4% on VoiceBench—but the numbers are not what make this release worth reading carefully. What matters is the set of architectural departures from the standard DeepSeek-V3-style MoE recipe, and what those choices mean when you read config.json and the Transformers implementation line by line.

This post looks under the hood: the architecture decisions, every significant config parameter, the multimodal towers, multi-token prediction (MTP), training and post-training, NVFP4 quantization exclusions, and how Inkling compares to peer open MoE models.

---

Three checkpoints, one lineage

• Inkling is the flagship open-weight checkpoint: trained from scratch, not a continual-pretrain bolt-on of a third-party base.
• Inkling-NVFP4 is the practical deployment path on Blackwell (W4A4) and Hopper (W4A16), cutting VRAM from ~2 TB BF16 to ~600 GB.
• Inkling-Small is a preview sibling with a much lower active footprint; full weights were still finishing validation at announcement time. Early numbers show it matching or exceeding the large model on several benchmarks—an outcome Thinking Machines attributes to improved pretraining data and recipe on the smaller run.

---

What the architecture does differently

Open the config.json and the first line reveals the class:

"architectures": ["InklingForConditionalGeneration"],
"model_type": "inkling_mm_model"

This is not a Llama, Qwen, or DeepSeek class name with a few knobs turned. Inkling ships as a first-class multimodal model type (inkling_mm_model) with nested text_config, vision_config, audio_config, and mtp_config blocks. The Transformers implementation lives under src/transformers/models/inkling/ (configuration_inkling.py, modeling_inkling.py, image/audio processors), so the forward pass is fully inspectable.

The MoE framework follows DeepSeek-V3 (sigmoid routing, shared experts, auxiliary-loss-free load-balancing bias). The combination of design choices is specific to Inkling:

1. Relative attention instead of RoPE. A fourth projection R produces a per-token, per-head relative feature; a learned bank of distance profiles is mixed into pre-softmax attention logits. No rotary embeddings.
2. Hybrid sliding / global attention at 5:1. Of 66 layers, 55 are local (512-token window) and 11 are global. This is what makes 1M context economically plausible.
3. Short convolutions (SConv) in four places per layer. Depthwise causal convs with kernel size 4 after K/V projections and on attention/MoE residual branch outputs—cheap local mixing that frees attention and experts from short-range work.
4. Shared-expert sink routing. Shared experts participate in score normalization (absorbing probability mass) but are excluded from top-k selection. Every token still sees 6 routed + 2 shared experts.
5. Encoder-free multimodality. Lightweight hierarchical MLP (hMLP) image patchifier and discrete mel (dMel) audio tokens project into the same decoder hidden space—no heavy frozen ViT or Whisper tower.
6. Chained 8-head multi-token prediction (MTP). Speculative decoding is first-class: up to 9 tokens per forward step with draft chains built into the checkpoint.
7. Controllable thinking effort. Continuous effort levels (0.00–0.99) trained via system message + per-token cost in RL, not only a discrete “thinking on/off” switch.

---

Reading the config files

The real story of any model is in its config. Below is what Thinking Machines chose and why, grounded in the released text_config and the open Transformers code.

MoE: the expert configuration

"n_routed_experts": 256,
"num_experts_per_tok": 6,
"n_shared_experts": 2,
"shared_expert_sink": true,
"intermediate_size": 3072,
"dense_intermediate_size": 24576,
"dense_mlp_idx": 2

Each routed expert is a SwiGLU FFN with intermediate dimension 3,072. When 6 fire: $6 \times 3072 = 18{,}432$ effective intermediate width from routed experts alone. Two shared experts add another $2 \times 3072 = 6{,}144$, for 24,576 effective intermediate units when all eight contribute—exactly matching dense_intermediate_size. That is intentional: dense foundation layers and sparse MoE layers keep per-token compute roughly aligned.

dense_mlp_idx: 2 means layers 0 and 1 are dense MLPs; layers 2–65 are sparse MoE. Two dense foundation layers (more than Kimi K2’s single dense layer, fewer than Trinity-Large’s six) stabilize early token representations before extreme expert routing begins.

How the expert configuration compares:

Inkling sits between Kimi’s extreme expert breadth and DeepSeek’s higher active count. Sebastian Raschka notes it activates ~4.2% of total parameters per token (41B/975B), denser than Kimi K2.5’s ~3.2%—a bet on quality-per-token rather than maximum sparsity.

The routing strategy: sigmoid + shared expert sink

"gate_activation": "sigmoid",
"route_scale": 8.0,
"norm_after_topk": true,
"use_global_scale": true,
"use_gate_bias": true,
"shared_expert_sink": true

From InklingTopkRouter in Transformers:

1. Router logits cover 256 routed + 2 shared experts (n_total_experts = 258).
2. Scores use sigmoid, independently per expert (no softmax competition in scoring).
3. Top-6 is selected only over the 256 routed experts, after adding a learned e_score_correction_bias (DeepSeek-style auxiliary-loss-free load balancing).
4. Selected routed logits and both shared logits are concatenated; weights are produced via log-sigmoid → logsumexp normalization over that joint set of 8 scores.
5. Weights are scaled by route_scale * global_scale (route_scale: 8.0; global_scale is a learned scalar parameter).

Shared expert sink is the distinctive piece. vLLM’s write-up states it cleanly: shared experts participate in routing-score computation (absorbing probability mass) but are excluded as candidates from top-6 selection. That differs from the common pattern of “always-on shared expert with fixed residual weight, routed experts normalized separately.” Joint normalization means the router can down-weight shared experts when routed specialists are confident, or allocate more mass to shared experts when routing is uncertain.

route_scale: 8.0 is aggressive relative to peers—other recent sigmoid-routed MoEs sit far lower (Arcee's Trinity technical report, for instance, documents a route scale of ~2.83). Combined with joint normalization over eight experts, it sharpens effective expert weights after the logsigmoid step.

Hybrid attention: the 5:1 sliding / global pattern

"num_hidden_layers": 66,
"sliding_window_size": 512,
"local_layer_ids": [0,1,2,3,4, 6,7,8,9,10, 12, ..., 60,61,62,63,64]

local_layer_ids enumerates every layer that is not a multiple-of-6 global layer. Global layers are indices 5, 11, 17, 23, 29, 35, 41, 47, 53, 59, 65—exactly 11 full-attention layers and 55 sliding-window layers. Pattern: five local, one global, repeating; the final layer (65) is global.

That is more sliding-window-heavy than Trinity-Large’s 3:1 (45 local / 15 global over 60 layers). Inkling’s window is also much tighter: 512 tokens versus Trinity’s 4,096. Local compute is nearly free even at 1M context; long-range dependency is the job of the 11 global layers plus residual accumulation through the stack.

vLLM is explicit: “This heavy use of sliding-window attention is what makes the model’s 1M context length efficient.”

Dual GQA geometry: different KV heads for local vs global

"num_attention_heads": 64,
"num_key_value_heads": 8,
"head_dim": 128,
"swa_num_attention_heads": 64,
"swa_num_key_value_heads": 16,
"swa_head_dim": 128

This is easy to miss. Global layers use 8 KV heads (8:1 GQA). Sliding-window layers use 16 KV heads (4:1 GQA). Query heads stay at 64 in both cases; head dim is 128.

Why dual geometry? Local layers dominate runtime and dominate sequence length under the 512-token window. Giving them more KV heads buys fidelity inside that window at modest cache cost (window is tiny). Global layers dominate memory at long context, so they use more aggressive 8-KV compression.

Attention scaling is also non-standard. Because Q and K are RMS-normalized per head, the code uses:

self.scaling = 1.0 / self.head_dim   # not 1/sqrt(d)

QK-norm changes the typical scale of dot products; dividing by d rather than √d keeps pre-softmax logits well-behaved.

Relative attention (no RoPE)

"d_rel": 16,
"rel_extent": 1024,
"log_scaling_n_floor": 128000,
"log_scaling_alpha": 0.1,
"q_bias": false,
"o_bias": false

Inkling does not use Rotary Positional Embeddings. Attention is:

$$\text{AttnWeights}=\frac{Q{K}^{\mathrm{\top }}}{d}+B_{\text{rel}}(q,k)+\text{mask}\backslash text\{AttnWeights\}\; =\; \backslash frac\{QK^\backslash top\}\{d\}\; +\; B\_\{\backslash text\{rel\}\}(q,\; k)\; +\; \backslash text\{mask\}$$

where $B_{\text{rel}}$ is a content-dependent relative position bias:

1. A linear map r_proj: hidden → num_heads × d_rel produces a per-token, per-head relative feature (d_rel = 16).
2. InklingRelativeLogits holds a learned bank proj of shape [d_rel, rel_extent]—a set of bias-vs-distance profiles.
3. Each query’s relative feature mixes those profiles (relative_states @ proj), then gathers by integer distance q_pos - k_pos.
4. Bias is zero outside $0 \le \text{distance} < \text{rel_extent}$. Future tokens are masked by causality separately.

Extent differs by layer type (from modeling code):

Beyond 1,024 tokens in global layers, attention is effectively content-based with respect to this positional bias—Raschka compares this intuition to NoPE (no positional embeddings) on long-range interactions. Thinking Machines reports that relative embeddings “perform better and extrapolate better to longer sequences than RoPE.”

Log attention scaling on global layers for long context:

"log_scaling_n_floor": 128000,
"log_scaling_alpha": 0.1

When sequence position exceeds 128K, queries and position biases are scaled by:

$$\tau =1+0.1\cdot \log \text{\hspace{0.17em}⁣}\left(\frac{n_{\text{eff}}}{128000}\right)\backslash tau\; =\; 1\; +\; 0.1\; \backslash cdot\; \backslash log\backslash !\backslash left(\backslash frac\{n\_\{\backslash text\{eff\}\}\}\{128000\}\backslash right)$$

This is a mild temperature-style adjustment for very long sequences—orthogonal to YaRN-style RoPE frequency scaling, because there is no RoPE to stretch.

Short convolution (SConv)

"use_sconv": true,
"sconv_kernel_size": 4

Each decoder layer applies four depthwise causal 1D convolutions (kernel size 4, groups = channels, residual around the conv):

From the decoder layer forward pass:

x → RMSNorm → Attention (K/V via SConv) → attn_sconv → + residual
  → RMSNorm → MoE/MLP → mlp_sconv → + residual

SConv is a depthwise Conv1d with padding kernel-1, causal truncation, and a residual connection; Transformers keeps these modules in FP32 (_keep_in_fp32_modules_strict). At decode time, a fused causal_conv1d_update updates a small state of the last W-1 = 3 tokens—cheap compared to attention.

Intuition (HF blog + vLLM): SConv acts like a tiny local attention / FIR filter over recent hidden states. That inductive bias lets global attention focus on long-range content and lets MoE experts specialize without also carrying all short-range token mixing.

Inference engines must track four conv states per layer. vLLM treats the SConv cache as a virtual sliding-window KV cache entry so eviction and prefix caching work through the same machinery.

Normalization and muP-style logit scaling

"rms_norm_eps": 1e-06,
"use_embed_norm": true,
"logits_mup_width_multiplier": 24.0,
"final_logit_softcapping": null

• Embed norm: RMSNorm is applied immediately after token embeddings—separate from pre-attention / pre-MLP norms inside each block. Unusual but explicit in both config and code (self.embed_norm).
• QK-norm: Per-head RMSNorm on queries and keys (see attention section).
• muP width multiplier: logits_mup_width_multiplier: 24.0 scales unembedding logits in the spirit of Maximal Update Parametrization, transferring hyperparameter regimes across widths. Softcapping is disabled (null).

Vocabulary and context

"vocab_size": 201024,
"unpadded_vocab_size": 200058,
"model_max_length": 1048576,
"eos_token_id": 200006

• 201,024 padded vocabulary (hardware-friendly multiple); 200,058 unpadded head rows.
• 1,048,576 max positions (exactly $2^{20}$)—native 1M context in the open weights.
• Special multimodal token IDs (from Transformers config defaults / model): image and audio placeholder / BOS tokens live in the high range of the vocab.

Among large open MoEs, this vocab is on the high side (comparable to Llama 4 Maverick’s ~202K and Trinity’s ~200K; larger than Kimi K2’s 163K or DeepSeek-V3’s 129K).

Hidden width and depth

"hidden_size": 6144,
"num_hidden_layers": 66

Inkling’s 6,144-d hidden size sits between Trinity’s thin-and-wide-experts design and DeepSeek/Kimi’s wider 7,168. Knowledge capacity is split across expert breadth (256), depth (66), and multimodal towers.

---

Multimodal towers: encoder-free by design

Inkling is natively multimodal. There is no giant separate vision transformer bolted on at the end of training. Both media paths are intentionally light and fold into the shared decoder residual stream.

Vision: hierarchical MLP patchifier (hMLP)

"vision_config": {
  "vision_encoder_type": "hmlp",
  "decoder_dmodel": 6144,
  "patch_size": 40,
  "temporal_patch_size": 2,
  "n_channels": 3,
  "n_layers": 4,
  "use_vision_norm": true
}

• Images are split into 40×40 pixel patches (much larger than the classic 14×14 ViT patch—fewer tokens per image).
• A 4-layer hierarchical MLP progressively merges neighboring patches (stack local blocks into the channel dimension, project) until one embedding per final patch remains.
• temporal_patch_size: 2 adds a temporal axis for video processing (two-frame temporal patches). Out-of-the-box video was not heavily evaluated at release; TML flags it as useful for downstream fine-tuning.
• Embeddings are RMS-normalized and projected to the text hidden size 6144.

Citation trail: hMLP follows the hierarchical patch-merging ideas in Touvron et al., “Three things everyone should know about Vision Transformers” (arXiv:2203.09795).

Recommended image size band from the model card: each dimension ideally 40–4096 px.

Audio: discrete mel (dMel)

"audio_config": {
  "decoder_dmodel": 6144,
  "n_mel_bins": 80,
  "mel_vocab_size": 16,
  "dmel_min_value": -7.0,
  "dmel_max_value": 2.0,
  "use_audio_norm": true,
  "audio_mode": "dmel"
}

• Audio → mel spectrogram with 80 mel bins.
• Each bin’s continuous value is quantized into one of 16 discrete levels over $[-7.0, 2.0]$ (dMel tokenization; Bai et al., arXiv:2407.15835).
• Discrete mel tokens are embedded and summed/composed into the shared decoder dimension.
• Model card recommends WAV at 16 kHz, ideally under ~20 minutes for best quality.

There is no separate Whisper-scale encoder. The heavy lifting of audio reasoning happens inside the multimodal decoder, which is why VoiceBench / MMAU scores are meaningful: the same 41B-active stack reasons over audio tokens and text tokens jointly.

Why this matters for interaction models

Thinking Machines positions Inkling as the background reasoning model for their interaction models system—real-time collaboration with voice and vision. Lightweight encoders keep prefill latency low; the MoE decoder carries multimodal reasoning. That product goal shows up directly in the architecture: simple towers, shared residual stream, no modality-specific frozen backbones.

---

Multi-token prediction (MTP)

"mtp_config": {
  "num_nextn_predict_layers": 8,
  "chain_hidden_post_norm": false,
  "local_layer_ids": [0, 2, 4, 5, 6, 7]
}

Inkling ships 8 MTP heads for speculative decoding:

• Each head is a single-layer Transformer with dense MLP (not MoE).
• Heads are chained: head $k$ consumes hidden states and the draft token from head $k-1$.
• Up to 9 tokens per base forward (1 verified + 8 speculative).
• MTP layers use a hybrid of sliding/global attention internally (mtp_config.local_layer_ids); MLP type is always dense.
• MTP weights stay in BF16 even on the NVFP4 checkpoint.

vLLM reports ~380 tok/s/user with MTP8 (mean acceptance length 4.5) versus **140 tok/s/user without MTP** on 4× GB200, measured on SPEED-Bench-style 8K→1K workloads. That is the practical payoff of baking speculative heads into the open release rather than training a separate draft model.

Transformers exposure:

generated = model.generate(**inputs, max_new_tokens=1000, use_mtp=True)

---

Controllable thinking effort

Inkling’s reasoning budget is continuous, not binary. Effort is injected via the chat template / system message as a numeric level:

Training mechanism (official post): system message specifies effort; RL applies a per-token cost so the policy learns to spend more or fewer chain-of-thought tokens depending on the requested budget. Sweeping effort from 0.2 → 0.99 traces a smooth performance-vs-token curve on Terminal Bench 2.1, HLE, and IFBench. Thinking Machines reports matching Nemotron 3 Ultra on Terminal Bench at roughly one-third the tokens.

An emergent side effect of large-scale RL: chains of thought became more telegraphic over training—dropping grammatical overhead while remaining comprehensible—without an explicit compression reward. Efficiency pressure alone drove the style shift (also noted by Cognition on SWE-1.7).

Official benchmarks are reported at effort = 0.99, temperature 1.0, with coding trajectories capped at 256K tokens.

---

Training: 45T multimodal tokens, hybrid optimizers, RL at scale

Pretraining

Coupling weight decay to $\eta^2$ is a deliberate stability trick at long training horizons (related literature: Kosson et al. 2023; Defazio 2025). Combined with Muon on the heavy matrices, it aims to keep the overall scale of weights well-behaved without constant retuning.

Post-training and RL

1. Bootstrap SFT on synthetic data from open-weights teachers (including Kimi K2.5)—small fraction of post-train compute.
2. Large-scale asynchronous RL on math, agentic code & tools, audio, image, chat, and safety environments—majority of post-train compute.
3. >30 million rollouts across two long continuous runs; held-out aggregate of AIME / HLE / GPQA-style evals improved log-linearly throughout.
4. Effort control trained jointly (system message + token cost).
5. Epistemics stack: calibration via proper scoring rules on resolved forecasting questions; dual graders (rubric + claims with agentic web search) for long-form faithfulness; abstention-aware short-form QA; anti-censorship training evaluated by Cognition’s Propaganda and Censorship Eval.

The model is explicitly positioned as a customization base, not a closed “final product” chat model. That shows in product packaging (Tinker fine-tuning platform, self-finetuning demo, open Apache 2.0 weights) as much as in architecture.

---

Quantization: BF16 vs NVFP4

BF16 full weights

~2 TB aggregate VRAM and Model card reference configs: 8× B300 or 16× H200.

NVFP4 (thinkingmachines/Inkling-NVFP4)

From hf_quant_config.json:

"quant_algo": "NVFP4",
"kv_cache_quant_algo": "none",
"group_size": 16

NVFP4 uses FP4 weights and activations with block size 16 and FP8-scale metadata (ModelOpt-style num_bits: [2,1] with dynamic block scales)—hardware-aligned to Blackwell FP4 tensor cores.

What gets quantized vs what stays full precision is the interesting part (vLLM + quant config):

Result: ~600 GB VRAM; W4A4 on 4× B300 (SM100+), or W4A16 on 8× H200. KV cache remains unquantized in this config (kv_cache_quant_algo: "none")—a deliberate tradeoff while relative-attention kernels mature (vLLM’s roadmap mentions FP8 global attention as future work).

Community path for smaller boxes: Unsloth / llama.cpp GGUF quants down to ~1-bit (heavy accuracy tradeoffs on agentic tool use; fine for experimentation).

---

How it compares architecturally

Reading the table, Inkling’s bet is clear:

• Not the sparsest MoE (Trinity / Llama-4-style ultra-sparse).
• Not the MLA KV-compression route (DeepSeek / Kimi).
• Yes to extreme context economics via hybrid attention + tiny 512 windows.
• Yes to abandoning RoPE for learned relative bias with long-range NoPE-like behavior.
• Yes to baking local inductive bias (SConv) and multimodality into the base, not as adapters.
• Yes to shipping deployment machinery (MTP, NVFP4, day-0 vLLM/SGLang/llama.cpp) with the weights.

---

Benchmark snapshot (effort = 0.99)

Selected official numbers (full tables on the model card and HF model page):

Thinking Machines is unusually candid that Inkling is not the strongest model on every axis. The profile is that of a broad generalist base intended for domain adaptation—strong agentic tool use, competitive multimodal open-weights audio/vision, efficient controllable reasoning—rather than a leaderboard-maxed specialist.

Go deploy it from Dell Enterprise Hub https://dell.hf.co

---

Why the design choices fit together

It helps to see the architecture as a coherent stack rather than a bag of tricks:

1. 1M context is only affordable if most layers are local. → 5:1 hybrid attention with a 512-token window.
2. Local layers need positional structure; global layers need long-range content matching. → Relative bias with extent = window on local layers; extent = 1,024 (then content-dominated) on global layers; log scaling past 128K.
3. Attention should not waste capacity on nearest-neighbor mixing. → Four SConvs per layer provide cheap local FIR filtering on K, V, attention residual, and MoE residual.
4. Nearly 1T parameters only make sense if most stay idle. → 256 experts, top-6, 2 shared with sink normalization, ~41B active.
5. Routing under sparsity needs a safety net. → Shared expert sink absorbs probability mass; joint normalization; dense layers 0–1 (and full-precision first MoE layer in NVFP4).
6. Real-time multimodal interaction cannot wait on huge encoders. → hMLP 40×40 patches + dMel discrete audio into a shared decoder.
7. Serving a 41B-active 1T model needs speculative decoding and quantization co-designed with the architecture. → 8 chained MTP heads; NVFP4 on routed experts only; custom FA4 + SConv-aware TP in vLLM.

That is a product-shaped architecture: customization base (Tinker + Apache 2.0), interactive multimodality, and long-context agentic coding—not a pure academic MoE demo.

---

Practical takeaways for practitioners

• Read local_layer_ids and dual GQA settings before estimating KV-cache memory; local and global layers do not share the same KV head count.
• Budget four SConv states per layer in any custom inference stack; treating them as tiny sliding caches (as vLLM does) is the right abstraction.
• Do not assume RoPE/YaRN tooling ports cleanly. Position is a learned relative bias + optional log scale. Kernel support (FA4 sheared bias) matters.
• NVFP4 is not “quantize everything.” Attention, shared experts, gates, SConv, multimodal towers, and MTP stay high precision—same philosophy as other large MoE INT4/FP4 releases, with Inkling-specific exclusions.
• Set reasoning_effort deliberately. Medium (~0.7) is often the quality/token sweet spot in HF vibe evals; max effort is for hard math/agent traces.
• Fine-tune for your domain. The release narrative and Tinker integration treat Inkling as clay, not porcelain.

---

Resources for further reading

• Official release post
• Model card
• Hugging Face model and config.json
• Hugging Face Welcome blog
• Transformers modeling_inkling.py
• vLLM Day-0 architecture notes
• Sebastian Raschka architecture notes

---

Dissection based on publicly released weights, configs, Transformers source, and first-party documentation as of July 2026. Parameter counts and eval harness notes follow Thinking Machines Lab’s model card; always re-verify against the live config and eval YAMLs on the Hub for production decisions.