Spark-X2.6-4B X Cartographics: a mechanistic interpretation of a compact on-device agent by juiceb0xc0de

I ran the Spark-X 4B models through the Cartographics HF space (a space that generates the model atlases referenced throughout this article, on demand) without reading the model cards. The XHToken team invited me to do an interpretation of the Spark-X2.6-4B release, and I decided to go in blind on the first pass. I wanted to know if the atlas gave clear indications of what the Spark-X family was designed for. Would my conclusions match what the model cards claimed or would my readings miss the mark? I went over the instruction-tuned model first, then ran a differential on the 2.6 and 2.5 base models.

My question was simple. What did the XHToken team intend Spark-X to be? A coding agent, or a creative roleplay model for people wanting a romantic Spark with X? I aimed to find out "Who is downloading Spark?"

Spark-X is a name that revealed nothing about the answers I was seeking. So I had to rely on the interpretation. This write-up is in two parts. The first is the blind read of the base-model diff, 2.5 to 2.6, kept the way I wrote it before the model cards, wrong calls and all. The second is the technical pass: the same two bases plus the 2.6 instruct, three atlases on one instrument, every claim tagged by what it rests on. A few of the blind read's calls get checked against the numbers there, and not all of them survive. If you're here for the technical analysis and don't need the blog-style introduction, collapse the first section and skip ahead.

Spark-X2.6-4B: how the base and instruct atlases differ

Three atlases, same 8,965 prompt texts, same 17 labels: Spark-X2.5-4B-Base, Spark-X2.6-4B-Base, Spark-X2.6-4B. The base captures use bare prompts. The instruct capture adds a 67-token chat wrapper ending at the opened <think> prefix, so every instruct input is 70 to 117 tokens against 3 to 50 for the bases [data]. That's the correct operating condition for each model, and it means every base-to-instruct difference below is checkpoint, input format, and their interaction tangled together. Nothing here isolates the fine-tune; what I'm comparing is two operating conditions [inferred].

Setup. Every number is MLP stream, mean_all_real pooling, unless stated. Each prompt's activation is the mean over its real input tokens. A coordinate is selective when its category ANOVA is significant at $q < 0.01$ and exactly one category reaches $|\mathrm{AUROC} - 0.5| \ge 0.25$; positive and negative associations both count. Probe accuracy is multinomial logistic regression on 256 PCs, PCA and scaling fitted inside each training fold, reported as mean held-out accuracy; the majority-class baseline is 6.58%. There are 368,640 MLP layer-coordinate entries per atlas and 576 attention heads (36 layers x 16). Every statement carries a tag. [data] is a direct measurement or calculation from the pinned atlas databases or capture metadata. [inferred] is my interpretation under stated assumptions. [data, inferred] is both. [guess] is a mechanism I haven't tested.

1. Health: clean captures, and the instruct commits to categories far more often

MLP taxonomy counts summed per depth third. Norm is the median across layers of each layer's prompt-median row norm.

No NaNs, no infinities, no atlas-defined dead features in any of the three captures [data]. That's capture health; it says nothing about how the training went.

The instruct's norms run about a third lower everywhere, and its per-layer total variance is a different story: the base-to-instruct ratio has a median of 30.7x across layers, ranging 22.7x to 111.5x, with layer 23 the outlier at the top [data]. My first suspect was token averaging [inferred]. Write the pooled instruct activation for prompt $i$ as a mixture of its user-token mean $u_i$ and its template-token mean $t_i$:

$$y_i={\alpha }_i{u}_i+(1-{\alpha }_i)t_i,{\alpha }_i=\frac{n_i}{n_i+67}y\_i\; =\; \backslash alpha\_i\; u\_i\; +\; (1-\backslash alpha\_i)\backslash ,\; t\_i,\; \backslash qquad\; \backslash alpha\_i\; =\; \backslash frac\{n\_i\}\{n\_i\; +\; 67\}$$

If the template contribution were constant across prompts and $\alpha$ fixed, the covariance would attenuate as

$$\mathrm{C}\mathrm{o}\mathrm{v}(y)={\alpha }^2\mathrm{C}\mathrm{o}\mathrm{v}(u)\backslash mathrm\{Cov\}(y)\; =\; \backslash alpha^2\backslash ,\; \backslash mathrm\{Cov\}(u)$$

which at $n = 16$ is a 26.9x drop, the right order of magnitude [inferred]. But $\alpha_i$ varies with prompt length, the template suffix is contextualised by the user text, and the user-token representations themselves change under the wrapper and the new weights. So pooling is a plausible contributor of about the right size, and that's as far as the equation goes [inferred]. One useful invariant falls out of it anyway: uniform positive scaling leaves participation ratio, ANOVA $F$, and AUROC unchanged, so scaling alone can't explain changes in PR, $F$, or the AUROC-based taxonomy in the sections that follow [inferred]. It does change norms and total variance, which is why those are the two measurements I won't interpret as weight changes on their own.

What does move: the number of coordinates meeting the selectivity criterion goes 2,698 to 8,877, a 3.29x increase, from 0.73% to 2.41% of MLP entries. Inert drops 438 to 91. Most coordinates stay broad in both atlases (99.1% base, 97.3% instruct) [data]. The instruct's selective set is 4,550 positive and 4,327 negative associations, so "selective" here means separates a category in either direction, and nothing stronger [data, inferred]. Proportionally the growth is spread across depth (3.42x, 3.64x, 3.10x by third); in absolute terms the back third adds the most [data].

2. Geometry: the pits at 23 and 34 fill in, and the reason is still a hypothesis

Participation ratio of the MLP covariance, $\mathrm{PR} = (\sum_i \lambda_i)^2 / \sum_i \lambda_i^2$ over covariance eigenvalues $\lambda_i$, with the number of PCs needed for 90% variance alongside.

Layers 23 and 34 have strongly concentrated spectra, with PR near 2.3, even though 245 and 242 PCs are needed to reach 90% variance. PR measures effective rank, not the number of PCs required to explain a specified fraction of variance. In the instruct capture the pits fill in: PR at 23 jumps 2.31 to 64.88 and k90 goes 245 to 755 [data].

The large activation outliers don't all stay put. By median $L_\infty$ the base's top three layers are 35, 34, 23. The instruct's are 34, 35, 22, and layer 23 falls to eighth [data]. So layer 34 keeps its outlier status while its covariance spreads out, and layer 23 loses both [data, inferred]. The explanation I like is a wrapper-associated component with low across-prompt variability parked on the sink positions, contributing to the mean and not the spread [guess]. Token-resolved activations under matched capture conditions would test whether the large components land on wrapper positions and how much they contribute to across-prompt covariance; these atlases are pooled, so it stays a hypothesis.

3. What it distinguishes: selective-label concentration rises between the bases and falls in the instruct capture

Selective MLP coordinates by top label, all 36 layers. Share is each label's fraction of that atlas's selective population.

The base-to-base comparison concentrates the selective-coordinate distribution: tool_use plus ml_ai went from 60.7% of selective MLP coordinates in 2.5 to 72.1% in 2.6 [data]. The instruct goes the other way [data]. Tool_use more than doubles in count and loses 18 points of share, because the remaining selective population grew faster in aggregate. Roleplay, which the pretraining delta spent down to 19 coordinates, comes back at 532. Introspection climbs 98 to 966. Core technical grows 72%, 198 to 340, and its share still halves [data]. Dilution, in other words: core technical didn't get removed; everything else grew faster [inferred].

Tool_use is still the top selective label in 35 of 36 layers [data]. Part one read the base as built for a tool loop, with tool_use second to ml_ai in growth. In the base, tool_use holds a majority of selective coordinates in most layers. In the instruct, from layer 9 onward tool_use stays below 50% and is the largest selective-label group except at layer 14, where roleplay leads [data]. The tool-use emphasis holds in both captures; a majority in the base is a plurality in the instruct [data, inferred].

Held-out category probe accuracy, mean of per-layer accuracies per depth third. Majority-class baseline 6.58%; shuffled-label control stays near 6% everywhere.

Base probe accuracy is nearly flat across thirds. Instruct rises with depth, in step with the selective counts, though not monotonically: it drops at 14 of its 35 adjacent-layer steps [data].

4. Attention: the heads went to prompts about prompting, and the MLPs went their own way

per_head labels each head with the category whose mean output norm lands furthest from the overall mean, in either direction, so a label can be the category that makes a head unusually quiet as well as unusually loud. Counts below are the heads component only, 576 heads.

Niche is the probe set's prompt-and-interaction bucket: system prompt crafting, tool-use prompt design, context window management, memory system design, constitutional rules, metacognitive reflection, with constraint-based creativity and red-teaming alongside. In the base, 6 heads carry it as their top label. In the instruct, 135 do, clustered in layers 12 to 23 [data]. Tool_use and niche together are 275 of 576 heads, 47.7% [data]. Only 127 of 576 heads, 22.0%, keep their base label; the largest single transitions across all five attention streams are ml_ai to niche (89 slots) and introspection to niche (68). Median head-norm $F$ more than doubles, 13.9 to 33.6 [data]. The direction of all that is toward prompts about prompting [data, inferred].

Hold that against the MLP table in section 3. Roleplay MLP coordinates went 19 to 532 while roleplay head labels went 53 to 27. Introspection MLP went 98 to 966 while introspection heads went 136 to 29. Niche MLP went 2 to 34 while niche heads went 6 to 135 [data].

I don't see a contradiction there. A head-norm label is about which category shoves a whole head's output magnitude off its average; coordinate selectivity is about which category one dimension separates. Two statistics on two different objects, and nothing forces them to agree [inferred]. The mismatch doesn't license "roleplay moved from attention into the MLPs" either. Neither statistic tracks a conserved quantity, so opposite movements can't establish a migration between modules and can't rule one out. I haven't tested whether the two properties are related at all; a different measurement could still find a redistribution [inferred].

5. Outliers: the base finds short prompts unusual, the instruct finds long ones unusual

mlp_mahalanobis_z is the layer-average of each prompt's standardised distance from the population centre in that layer's whitened top-256-PC space, so a "4.2" is far out on this scale and not 4.2 literal standard deviations. The tail is the top 5%, 448 prompts.

The two captures have disjoint top-100 Mahalanobis outlier lists. The base's highest scores go to two- and three-word imperatives: "brainstorm features", "poke holes in this", "what am i missing", "document this repo". The instruct's go to long, dense, specific prompts: HNSW vs IVF vs ScaNN at 100M vectors, Timescale vs Influx vs ClickHouse at 50k writes a second, full fine-tune vs LoRA vs QLoRA on one 24GB card. No prompt appears in both top-100 lists [data]. The biggest movers are the terse ones: "write survey questions" scores 4.2 in the base and −2.4 in the instruct, more ordinary than the median prompt [data].

The sign flip on length is the result here, and section 1's pooling equation is the competing explanation for it: a longer user prompt has a larger $\alpha_i$, which changes the balance between user and template contributions. If those contributions differ systematically, this length-dependent mixing can induce an association between prompt length and outlier score [guess]. I haven't measured its contribution to the +0.83. The reversal is also metric-dependent. Cross-capture correlation is $-0.25$ for the layer-averaged Mahalanobis score and $+0.71$ for the neighbourhood-based outlier score [data], so the two captures still largely agree about which prompts are unusual under the neighbourhood metric while disagreeing under the whitened distance. Which makes the reversal instrument-dependent and says nothing about how much of it pooling caused [inferred].

None of the category enrichment below is length-controlled. Mean instruct input runs 81.9 tokens for tool_use to 84.9 for ml_ai [data], which looks like nothing on a 47-token range, but subtract the common wrapper and it's 14.9 against 17.9 user tokens, mixture weights $\alpha \approx 0.18$ against $0.21$ [data]. Similar means also say nothing about the upper tails of the length distributions. How much of the technical-tail enrichment survives length adjustment, I don't know yet [inferred]. Tool_use has the lowest tail-entry rate in the base capture and one of the lowest in instruct, effectively tied with business [data]. Core technical has the highest in the instruct: 12.9% of its prompts enter the tail, it makes up 15.9% of the tail against a 6.1% corpus prevalence, an enrichment of

$$\frac{P(\text{core\_technical}\mid \text{tail})}{P(\text{core\_technical})}=\frac{71\mathrm{/}448}{550\mathrm{/}8965}\approx 2.58\backslash frac\{P(\backslash text\{core\backslash \_technical\}\; \backslash mid\; \backslash text\{tail\})\}\{P(\backslash text\{core\backslash \_technical\})\}\; =\; \backslash frac\{71/448\}\{550/8965\}\; \backslash approx\; 2.58$$

The score measures relative displacement from the evaluation-corpus centre under this pooling and PCA metric. It isn't training-distribution familiarity, difficulty, or answer quality [inferred]. Unusual to this capture [data] is the claim. Hard for this model is a behavioural question, and the atlas can't see behaviour.

6. Verdict

Under their respective capture conditions, the instruct checkpoint has broader single-category MLP separation (3.29x the base's selective coordinates, spread across every depth third) and higher category-probe accuracy, with the largest gains late [data]. Tool_use stays the dominant selective label and gives up 18 points of share to conversational categories, including roleplay and introspection, whose selective counts fell between the base captures [data]. Attention head-norm associations reorganise toward the prompt-and-interaction bucket, 6 to 135 heads, and for three categories the MLP and head statistics move opposite ways, which neither establishes nor rules out a migration between modules [data, inferred]. That, plus the outlier reversal under the Mahalanobis metric, is the strongest statement I'm willing to make from these three atlases [data, inferred].

Core technical selective counts rise 72% but slower than the population, so its share halves [data]. Technical prompts are enriched 2.6x in the instruct's outlier tail [data], and that score is both length-associated and measured under a different pooling context than the base [data, inferred]. Neither one adds up to a technical regression, and nothing here says which part of the fine-tuning mix produced the shift [inferred]. Whether that breadth was bought with engineering ability needs matched behavioural evals on exactly the long, multi-constraint technical tail. A bare-prompt capture of the instruct would give the checkpoint difference under that format; separating checkpoint, wrapper, and interaction takes both checkpoints under both formats.

Part one's tool_use call holds: top selective MLP label in 35 of 36 instruct layers, the largest group at every base layer, and a majority at most of them [data]. The prompt-and-interaction niche bucket, which part one doesn't mention, is the second-largest head-norm group in the instruct, 135 of the 436 non-tool-use heads [data]. "Writing moved out of attention and into the MLPs" is unsupported, and in my opinion it read like a number going straight to a story [inferred].

Three mistakes from the process. I've left them in. The 67-token chat template was assumed present in the base captures during one read and absent in the next; it's present only in the instruct capture, and section 1 is how far that arithmetic gets. I first read the layer 2 participation ratio drop in the base as a near-collapse; k90 only moved 1,066 to 867, TwoNN barely moved, and probe accuracy held, so it's concentration of variance, and on its own that isn't evidence of lost category information, let alone a representational collapse. And a first draft of the head table carried a core_technical count that hadn't been queried, plus transition counts from a join that ignored the component column; section 4 carries the corrected numbers.

Atlases

Spark-X2.5-4B-Base · Spark-X2.6-4B-Base · Spark-X2.6-4B

The category table in section 3:

SELECT top_label, COUNT(*) AS selective
FROM features
WHERE component = 'mlp' AND taxonomy = 'selective'
GROUP BY top_label ORDER BY selective DESC;

The geometry and probe curves in sections 2 and 3:

SELECT layer, geometry_participation_ratio, geometry_k90, geometry_twonn,
       probes_probe_accuracy, probes_probe_control_accuracy
FROM layers
WHERE component = 'mlp' AND position = 'mean_all_real'
ORDER BY layer;

The head table in section 4:

SELECT norm_top_label, COUNT(*) AS heads
FROM per_head WHERE component = 'heads'
GROUP BY norm_top_label ORDER BY heads DESC;

The outlier lists in section 5:

SELECT label, prompt, n_tokens, mlp_mahalanobis_z
FROM prompts ORDER BY mlp_mahalanobis_z DESC LIMIT 20;

Dataset revisions used for every number above: