# How much superposition is there?

*Written as part of MATS 7.1. Math by Claude Opus 4.6.*

I know that models are able to represent exponentially more concepts than they have dimensions by engaging in [superposition](https://transformer-circuits.pub/2022/toy_model/index.html) (representing each concept as a direction, and allowing those directions to overlap slightly), but what does this mean concretely? How many concepts can "fit" into a space of a given size? And how much would those concepts need to overlap?

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/1fda68f46ee8acd88f39ffc57a622db1461f5e38a3a5788e.png)

Superposition interference diagram, from [Toy Models of Superposition](https://transformer-circuits.pub/2022/toy_model/index.html)

This felt especially relevant working on [SynthSAEBench](https://arxiv.org/abs/2602.14687v1), where  we needed to explicitly decide how many features to cram into a 768-dim space. We settled on 16k features to keep the model fast - but does this lead to enough superposition to be realistic? Surely real LLMs have dramatically more features and thus far more superposition?

As it turns out: yes, 16k features is plenty! In fact, as we'll see in the rest of this post, 16k features in a 768-dim space actually leads to *more superposition* than trillions of features in a 4k+ dim space, as is commonly used for modern LLMs.

Personally I found the answers to this fascinating - high dimensional spaces are extremely mind-bending. We'll take a geometric approach and try to answer this question. Nothing in this post is ground-breaking, but I found thinking about these questions enlightening. All code for this post can be found in on [Github](https://github.com/chanind/superposition-exploration) or [Colab](https://colab.research.google.com/github/chanind/superposition-exploration/blob/main/exploration.ipynb).

Quantifying superposition
-------------------------

First, let's define a measure of how much superposition there is in a model. We'll use the metric *mean max absolute cosine similarity*, $\rho_\text{mm}$, defined as follows:

$\rho_\text{mm} = \frac{1}{N} \sum_{i=1}^{N} \max_{j \neq i} \left| \frac{\mathbf{v}_i \cdot \mathbf{v}_j}{\|\mathbf{v}_i\| \|\mathbf{v}_j\|} \right|$

This metric represents a "worst-case" measure of superposition interference for each vector in our space. It's answer the question: on average, what's the most interference (highest absolute cosine similarity) each vector will have with another vector in the space?

**Superposition of random vectors**
-----------------------------------

Perfectly tiling the space with concept vectors is challenging, so let's just consider the superposition from random vectors (We'll see later that this is already very close to perfect tiling). If we have $N$ random unit-norm vectors in a $d$-dimensional space, what should we expect $\rho_\text{mm}$ to be? We can try this out with a simple simulation.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d0991e137541bbd9a60b2bc1570ab1bd65b1e2ac4b8b98e4.png)

Simulation picking N random vectors in a d-dimensional space, and calculating superposition $\rho_\text{mm}$

We vary $d$ from 256 to 1024, and $N$ from 4,096 to 32,768 and calculate $\rho_\text{mm}$, showing the results above. This is still very small-scale though. Ideally, we'd like to know how much superposition we could expect with billions or even trillions of potential concepts, and that's too expensive to simulate. Fortunately, we can find a formula that we can use to directly calculate $\rho_\text{mm}$ without needing to actually run the simulation.

Calculating $\rho_\text{mm}$ directly
-------------------------------------

We can compute the expected $\rho_\text{mm}$ exactly (up to numerical integration) using the known distribution of cosine similarity between random unit vectors.

For two random unit vectors in $\mathbb{R}^d$, the squared cosine similarity follows a Beta distribution: $\cos^2\theta \sim \text{Beta}(1/2, (d-1)/2)$. This means the CDF of $|\cos\theta|$ is the regularized incomplete beta function:

$P(|\cos\theta| \leq t) = I_{t^2}\!\left(\tfrac{1}{2}, \tfrac{d-1}{2}\right)$

For each vector, its max absolute cosine similarity with $N-1$ others has CDF $F(t)^{N-1}$ (treating pairwise similarities as independent, which is excellent for large $d$). The expected value of a non-negative random variable gives us:

$\mathbb{E}[\rho_\text{mm}] = \int_0^1 \left(1 - \left[I_{t^2}\!\left(\tfrac{1}{2}, \tfrac{d-1}{2}\right)\right]^{N-1}\right) dt$

We can calculate this integral using `scipy.integrate`. Let's see how well this matches our simulation:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/2c430e7bb509c48d1ac1e2a230f486a73486b72f08cc40c5.png)

The predicted values exactly match what we simulated!

Scaling to trillions of concepts
--------------------------------

Let's use this to see how much superposition interference we should expect for a some really massive numbers of concepts. We'll go up to 10 trillion concepts (10^13) and 8192 dimensions, which is the hidden size of the largest current open models. 10 trillion concepts seems like a reasonable upper bound for the max number of concepts that could conceivably be possible, since that would be roughly 1 concept per training token in a typical LLM pretraining run.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/21c3e9bdd1137f827f9c18471a54e9d53e4ea18b3c9a5d0b.png)

*10 trillion concepts* in 8192 dimensions has far less superposition interference than just *100K concepts* in 768 dimensions (the hidden dimension of GPT-2)! That's a 100,000,000x increase in number of concept vectors! Even staying at a given dimension, increasing the number of concepts by 100x doesn't really increase superposition interference by all that much.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e090124c0a941dcb0975d79a5a0233fed99c33a91939fd1e.jpg)

At least, I found this mind-blowing

What if we optimally placed the vectors instead?
------------------------------------------------

Everything above assumes random unit vectors. But what if we could arrange them optimally — placing each vector to minimize the worst-case interference? Would we do significantly better?

From [spherical coding theory](https://en.wikipedia.org/wiki/Spherical_code), the answer is: *barely*. The minimum achievable max pairwise correlation for $N$ optimally-placed unit vectors in $d$ dimensions is given by the [spherical cap packing bound](https://en.wikipedia.org/wiki/Spherical_cap#Solid_angle):

$\varepsilon_\text{optimal} \approx \sqrt{1 - N^{-2/(d-1)}}$

The intuition is that each vector "excludes" a spherical cap around itself, and we're counting how many non-overlapping caps fit on the unit sphere in $\mathbb{R}^d$.

When $\ln N \ll d$ (which holds for all practical settings — even $N = 10^{13}$ in $d = 512$ gives $\ln N / d \approx 0.06$), we can Taylor-expand:

$N^{-2/(d-1)} = e^{-2\ln N/(d-1)} \approx 1 - \frac{2\ln N}{d}$

which gives:

$\varepsilon_\text{optimal} \approx \sqrt{\frac{2\ln N}{d}}$

This is exactly the leading-order term of the random vector formula! So random placement is already near-optimal — there's essentially nothing to gain from clever geometric arrangement of the vectors, at least for the $N$ and $d$ values relevant to real neural networks.

This is a remarkable consequence of high-dimensional geometry: in spaces with hundreds or thousands of dimensions, random directions are already so close to orthogonal that you can't do meaningfully better by optimizing.

What does this mean for SynthSAEBench-16k?
------------------------------------------

At the start, I mentioned that we used 16k concept directions in a 768-dim space for the [SynthSAEBench-16k model](https://decoderesearch.github.io/SAELens/latest/synth_sae_bench/#synthsaebench-16k-model). So is this enough superposition interference?

The answer is a resounding: **yes**. The SynthSAEBench-16k model has a $\rho_\text{mm}$ of 0.14, which is still dramatically more superposition interference than *10 trilllion* concept vectors in 8196-dim (the hidden dimension of Llama-3.1-70b). It's roughly equivalent to *1 billion* concept vectors in a 2048-dim space (the hidden size of Gemma-2b).