# (Approximately) Deterministic Natural Latents

*Background:* [*Natural Latents: The Math*](https://www.lesswrong.com/posts/dWQWzGCSFj6GTZHz7/natural-latents-the-math)*,* [*Natural Latents: The Concepts*](https://www.lesswrong.com/posts/mMEbfooQzMwJERAJJ/natural-latents-the-concepts)*,* [*Why Care About Natural Latents?*](https://www.lesswrong.com/posts/RTiuLzusJWyepFpbN/why-care-about-natural-latents)*,* [*the prototypical semantics use-case*](https://www.lesswrong.com/posts/RrQftNoRHd5ya54cb/towards-a-less-bullshit-model-of-semantics#Equivalence_Via_Naturality)*. This post does not assume that you’ve read all of those, or even any of them.*

Suppose I roll a biased die 1000 times, and then roll the same biased die another 1000 times. Then...

*   Mediation: The first 1000 rolls are approximately independent of the second 1000 given the bias (to reasonable precision).
*   Redundancy: I can estimate the die’s  bias (to reasonable precision) with high confidence from either the first or second 1000 rolls.

The die’s bias is therefore a natural latent, which means it has various nice properties.

*   Minimality: The bias is the smallest summary of *all* the information about the first 1000 rolls relevant to the second 1000 (and vice-versa).
*   Maximality: The bias is the largest piece of information which can be calculated from the first 1000 rolls *and also* can separately be calculated from the second 1000 rolls.
*   Any other variable which satisfies the above properties must tell us (approximately) the same information about the die rolls as the bias.

Furthermore, the bias is a(n approximate) *deterministic* natural latent: the die’s bias (to reasonable precision) is approximately determined by[^wo04rhagwi] the first 1000 die rolls, and also approximately determined by the second 1000 die rolls. That implies one more nice property:

*   Uniqueness: The bias is the unique-up-to(-approximate)-isomorphism latent which has the above properties, making it a natural [Schelling point](https://www.lesswrong.com/posts/yJfBzcDL9fBHJfZ6P/nash-equilibria-and-schelling-points) for [communication between agents](https://www.lesswrong.com/posts/RrQftNoRHd5ya54cb/towards-a-less-bullshit-model-of-semantics#Equivalence_Via_Naturality).

We’ve proven all that before, mostly in [Natural Latents: The Math](https://www.lesswrong.com/posts/dWQWzGCSFj6GTZHz7/natural-latents-the-math) (including [the addendum](https://www.lesswrong.com/posts/dWQWzGCSFj6GTZHz7/natural-latents-the-math#Addendum__Added_June_12_2024___Approximate_Isomorphism_of_Natural_Latents_Which_Are_Approximately_Determined_By_X) added six months after the rest of the post). But it turns out that the math is a lot shorter and simpler, and easily yields better bounds, if we’re willing to assume (approximate) determinism up-front. That does lose us some theoretical tools (notably the [resampling construction](https://www.lesswrong.com/posts/NHKCtSXgFieDAyWt2/calculating-natural-latents-via-resampling)), but it gives a cleaner foundation for our expected typical use cases (like e.g. [semantics](https://www.lesswrong.com/posts/RrQftNoRHd5ya54cb/towards-a-less-bullshit-model-of-semantics#Equivalence_Via_Naturality)). The goal of this post is to walk through that math.

Background Tool: Determinism in Diagrams
----------------------------------------

We’re going to use [diagrammatic proofs, specifically using Bayes nets](https://www.lesswrong.com/posts/XHtygebvHoJSSeNPP/some-rules-for-an-algebra-of-bayes-nets). But it’s non-obvious how to express (approximate) determinism using Bayes nets, or what rules diagrams follow when determinism is involved, so we’ll walk through that first.

This diagram says that $Y$ is (approximately) determined by $X$:

![](https://lh7-us.googleusercontent.com/docsz/AD_4nXddao0mNR5zxSEY3fOpwDKyebYlhA1ck8JOvlJycEY_M5bj0ptWve3ZFpRkqM1rbUKFX9OBrTzquj1MX-IWr86RKHcevSq1whmG2uaYzVNXlctB-mRNRWQ8FIWEZpFMkdw2CwusHqecGdeFcMuX1ftNgzDv?key=bSf_4m9LzuqVJBIO8ix53g)

Intuitively, the literal interpretation of the diagram is: $X$ mediates between $Y$ and $Y$, i.e. $Y$ itself tells me nothing more about $Y$ once I already know $X$. That only makes sense if $X$ tells me *everything* there is to know about $Y$, i.e. $Y$ is determined by $X$.

In the approximate case, we express the approximation error of the diagram as a KL-divergence, [same as usual](https://www.lesswrong.com/posts/XHtygebvHoJSSeNPP/some-rules-for-an-algebra-of-bayes-nets):

$\epsilon \geq D_{KL}(P[X=x, Y=y, Y=y’] || P[X=x] P[Y=y|X=x] P[Y=y’|X=x])$

If you get confused later about what it means to have two copies of the same variable in a diagram, go back to that line; that’s the definition of the approximation error of the diagram. (One way to view that definition: there’s actually two variables $Y$ and $Y’$, but $P$ says that $Y$ and $Y'$ always have the same value.)

That approximation error simplifies:

$D_{KL}(P[X=x, Y=y, Y=y’] || P[X=x] P[Y=y|X=x] P[Y=y’|X=x])$

$= D_{KL}(P[X=x, Y=y]I[y=y’] || P[X=x] P[Y=y|X=x] P[Y=y’|X=x])$

$= \sum_{x, y, y’} P[X=x, Y=y]I[y=y’] (\text{log}(P[X=x, Y=y]I[y=y’]) - \text{log}(P[X=x] P[Y=y|X=x] P[Y=y’|X=x]))$

$= \sum_{x, y} P[X=x, Y=y] (\text{log}(P[X=x, Y=y]) - \text{log}(P[X=x] P[Y=y|X=x] P[Y=y|X=x]))$

$= - \sum_{x, y} P[X=x, Y=y] \text{log}(P[Y=y|X=x])$

$= H(Y|X)$

So the diagram says $Y$ is determined by $X$, and the approximation error of the diagram is the entropy $H$ of $Y$ given $X$ \- i.e. the number of bits required on average to specify $Y$ once one already knows $X$. Very intuitive!

### The Dangly Bit Lemma

Intuitively, if $Y$ is determined by $X$, then $X$ always mediates between $Y$ and anything else. So, when working with diagrams, we can always add a copy of $Y$ with only $X$ as a parent. 

![](https://lh7-us.googleusercontent.com/docsz/AD_4nXd7ughwP79hvNTRnh857iB93qqachBuXbtQxWUDRzB223lPGn92n5V7pRnlEjtk1uxfajtcUaVp6OhheDf3mZfNQmj5ZS5pdlFCn9-NBuKZ-qCLRsQHFxvJzqFUOWrM3M9gfqwDdIVjZTzuAKLrOtMVD_V_?key=bSf_4m9LzuqVJBIO8ix53g)

Here’s the lemma:

If $Y \leftarrow X \rightarrow Y$ holds to within $\epsilon$ bits, and any other diagram $D$ involving $X$ holds to within $\epsilon’$ bits, then we can create a new diagram $D’$ which is identical to $D$ but has another copy of $Y$ (the “dangly bit”) as a child of $X$. The new diagram $D’$ will hold to within $\epsilon + \epsilon’$ bits.

+++ Proof (click to expand)

Let $Q[X, Z]$ be the distribution over $X$ and any other variables $Z$ specified by the diagram $D$ (note that $Z$ may include some copies of $Y$). Then $D’$ specifies the distribution $Q[X, Z] P[Y|X]$, so the approximation error for $D’$ is

$D_{KL}(P[X, Y, Z] || Q[X, Z] P[Y|X])$

$= D_{KL}(P[X, Z] || Q[X, Z]) + E_{X, Z}[D_{KL}(P[Y|X, Z] || P[Y|X])]$

$= D_{KL}(P[X, Z] || Q[X, Z]) + I(Y; Z|X)$

$\leq D_{KL}(P[X, Z] || Q[X, Z]) + H(Y|X)$

$\leq \epsilon’ + \epsilon$




+++

A simple example: for any $X$, $Z$ the diagram $X \rightarrow Z$ is always satisfied exactly. So, if

![](https://lh7-us.googleusercontent.com/docsz/AD_4nXddao0mNR5zxSEY3fOpwDKyebYlhA1ck8JOvlJycEY_M5bj0ptWve3ZFpRkqM1rbUKFX9OBrTzquj1MX-IWr86RKHcevSq1whmG2uaYzVNXlctB-mRNRWQ8FIWEZpFMkdw2CwusHqecGdeFcMuX1ftNgzDv?key=bSf_4m9LzuqVJBIO8ix53g)

is satisfied, then

![](https://lh7-us.googleusercontent.com/docsz/AD_4nXcEEz0StPnKHU8c1XPXp7dXnvdrmvyXOxRAQ3YoeLEO4HcPH15YsJ41wMJkqcvfPnuhVnoqkPjalznbfsjBQch1E9Vst13YhN7eWmZ2xVnYwA6HKNlzUyVTM306qT9ZYfgkdY1-kfYtsLLGDdtVwmH22EI?key=bSf_4m9LzuqVJBIO8ix53g)

is satisfied for *any* $Z$. If we write out the approximation errors for these diagrams, this example is equivalent to $H(Y|X) \geq I(Y; Z|X)$.

Deterministic Natural Latents
-----------------------------

### Fundamental Theorem

We’ll start with the core idea of natural latents: if one variable $\Lambda$ mediates between two other (sets of) variables $X_S, X_\bar{S}$, and another variable $\Lambda’$ is redundantly represented in both $X_S$ and $X_\bar{S}$, then $\Lambda’$ is also represented in $\Lambda$. Intuitively: imagine that $\Lambda$ is a pipe between $X_S$ and $X_\bar{S}$, and the only way for information to move between $X_S$ and $X_\bar{S}$ is through that pipe. Then if some information $\Lambda’$ is present in both $X_S$ and $X_\bar{S}$, it must have gone through the pipe.

Diagrammatically, the theorem says:

![](https://lh7-us.googleusercontent.com/docsz/AD_4nXeyd9HbOr6O2i2BsolJ9sHxjbhDNtII0YRdVOrRp_LnD5YktjwPiFwDttPloAwDGLLPfcJxpYZ_aeROUFztvXKuDGJOChA_pAU-Wn272BxI6UdRsaOkvwXWBngMHlpg34dvQLB2cKJrxukLELo9Uivi9jc?key=bSf_4m9LzuqVJBIO8ix53g)

The proof is just two applications of The Dangly Bit Lemma, followed by marginalizing out $X_S$ and $X_\bar{S}$:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/1c32146b06d1d99aca09ba234e606350e53a853d869a3de2.png)

### Minimality, Maximality, and Isomorphism of Deterministic Natural Latents

Now suppose that $\Lambda$ (approximately) satisfies *both* the mediation and redundancy conditions. Then:

*   $\Lambda$ (approximately) satisfies the mediation condition, and
*   For *any* other $\Lambda’$ which (approximately) satisfies the mediation condition, $\Lambda \leftarrow \Lambda’ \rightarrow \Lambda$ (approximately) by the Fundamental Theorem.

So, $\Lambda$ is (approximately) the smallest variable which (approximately) satisfies the mediation condition, in the sense that $\Lambda$ is (approximately) determined by any other variable which (approximately) satisfies the mediation condition. This is the “minimality” property of deterministic natural latents.

![](https://lh7-us.googleusercontent.com/docsz/AD_4nXewiNQw-_sBPBYUFkxj--sqf1urIzaLybl5Z5rkXn0TyNA8t5w-iENyTIHB2kRZLhbvrc5lDlYU7YENKwEZ_H6aWUjBHdB-QnhoA4338FGLFKX0dt4Jb_LTkZ-lqVfy4SHAwOClQlbH4_C4DgAFezy2zQ3x?key=bSf_4m9LzuqVJBIO8ix53g)

Similarly:

*   $\Lambda$ (approximately) satisfies the redundancy condition, and
*   For *any* other $\Lambda’$ which (approximately) satisfies the redundancy condition, $\Lambda’ \leftarrow \Lambda \rightarrow \Lambda’$ (approximately) by the Fundamental Theorem.

So, $\Lambda’$ is (approximately) the largest variable which (approximately) satisfies the redundancy condition, in the sense that any other variable which (approximately) satisfies the redundancy condition is (approximately) determined by $\Lambda$. This is the “maximality” property of deterministic natural latents.

![](https://lh7-us.googleusercontent.com/docsz/AD_4nXcWXcCvlQm8-suSeznz2l9_GfqEH9zxLalKFYTCXd9FNogvIJNTYkdhTyN_3eIhayfPGeBdh_uqHEQgY84hp3GF71REwsdvBlYm4SzncBAHWSXGQUaUEPKXp7oyP0P_0fF_cM1tk4s8n8PNwMopnkmEnGI?key=bSf_4m9LzuqVJBIO8ix53g)

Finally, suppose that $\Lambda$ and $\Lambda’$ *both* satisfy *both* the mediation and redundancy conditions. Then by the Fundamental Theorem, each is (approximately) determined by other; they’re approximately isomorphic. So, the (approximate) deterministic natural latent is (approximately) unique up to isomorphism.

![](https://lh7-us.googleusercontent.com/docsz/AD_4nXfkepS4OTjxjyGc6tnO49O9fAg1uk3JloHolDt5qS8LWfiOgfV7-12_lOAnDkXAXSJxnNh1wP94rwxOnyPdgyEa7przHNEQwBLM4kBW2fbrjvqUN2fcmlBBN-xj8LKJoUstbSC5y6jX_8ZfBDNFM4_fRVG6?key=bSf_4m9LzuqVJBIO8ix53g)

And just like that, we’ve proven the main properties of deterministic natural latents which we typically want to use: (approximate) minimality, (approximate) maximality, and (approximate) uniqueness.

[^wo04rhagwi]: When we say "Y is (approximately) determined by X" in this post, we mean "conditional on X, the entropy of Y is (approximately) zero". This may or may not imply any other notion of "Y is (approximately) determined by X", like e.g. "there exists a deterministic function of X which is equal to Y with high probability".