# A summary of Condensation and its relation to Natural Latents

Short summary of Condensation
=============================

Condensation is a theory of concepts by Sam Eisenstat. The paper can be read [here](https://openreview.net/pdf?id=HwKFJ3odui). Abram wrote [a review](https://www.lesswrong.com/posts/BstHXPgQyfeNnLjjp/condensation) on Lesswrong, and a [followup](https://www.lesswrong.com/posts/2x9yatKKTRMabQAWq/condensation-and-relevance). The paper is very much worth reading, and can be skimmed to just understand the motivation if you’re time constrained.

What is a concept? One rough pointer is "the little packets of information that are the *meaning* of a word or phrase". Another is the clusters found in [the cluster structure of thingspace](https://www.lesswrong.com/posts/WBw8dDkAWohFjWQSk/the-cluster-structure-of-thingspace). Maybe we could think of them as little chunks of a generative model that compose together to make complete model of a complex world.

It often seems to be the case that this kind of chunking is arbitrary, in the sense that there isn’t one way to chunk up a generative model. It just depends on the hypothesis class, prior and implementation details. But some concepts seem to be more objective than this, in that we would expect different agents with different implementation details to use the same concept. One question that Condensation answers is *“under what conditions can concepts be more objective?”*. The theory provides some assumptions and metrics that can be used to select among **latent variable models (LVM)s** to find ones that use more “objective”[^i2gd03a2czq] concepts. If we make these assumptions on two agents’ latent variable models, it tells us when they will (and won’t) use the same latent variables.

This is very similar to the motivation for John Wentworth’s [Natural Latents](https://arxiv.org/abs/2509.03780), as is noted in the Condensation paper, but so far no one has worked through the similarities between the two on a mathematical level. So the main goal of this post is to show some simple relationships between these two theories. A secondary goal is to work through an intuitive example of a non-trivial condensation, and this is where we’ll start.

How does condensation work?
---------------------------

There are multiple ways we could organize a generative model. We’ll call a “way of organizing a generative model” an LVM. Condensation is a method for sorting among these and finding ones that are also likely to be found by other agents who are doing the same thing.

If we have a set of random variables $X_i$, the space of LVMs can be (very roughly) thought of as the space of Bayesian Networks whose leaves are the random variables $X_i$. Except if two latent variables are upstream of the same subset of $X$s, we'll combine them into a single variable. So we can name all the latent variables ($Y$) with the subscript indicating which variables they contribute to.

For example, for variables $X_1,…X_6$, an LVM might look like this:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/2db8e927aab0457b85187459789e5d8e519468b3b2ca1bb0.png)

There is a $Y$ variable for every subset of $X$ variables, but under normal circumstances many of these will be constant random variables, so we can ignore them (they are omitted from the diagram to give a cleaner example).

The central metric of condensation is the conditioned score $\chi(A)$, which is a score for every variable $Y_A$ in an LVM. The idea is that if two LVMs minimize every $\chi(A)$ simultaneously, they end up “sharing the same concepts” (in a certain sense).

### Example application

A simple example where the latent variables are more “objective” is in clustering problems. The “objectively correct” latent variables are cluster parameters. If some clusters share similarities, then we have an “objectively correct” hierarchy of clusters. Take a second to intuitively understand the generating process of this data:

![clustering_hierarchical_discrete.gif](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1772549712/lexical_client_uploads/yn0auhntnvfjzxriaxud.gif)

We start with access to a distribution over a set of observation variables. In this example, we have a sequence of 36 2D observations.

For future use, we will calculate the joint entropy of some (carefully chosen) groups of these variables:

![entropy_diagram.png](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1772550046/lexical_client_uploads/gzcyamc4wcgb0mqmztgf.png)

As you can see if you watch the animation for a while, there is a hierarchical generating process behind this data. There are always four clusters, divided into two groups. Within each cluster, each sample is independent of the others. In each group, the cluster means are relatively close to each other. The shape of each cluster is always roughly the same as the shape of each other cluster.

We have lots of options for modelling this. We could fit an autoregressive sequence model such that it outputs a distribution and samples from it at every timestep. This would look like:

![bayes_net_diagram (Edited 2).png](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1772550643/lexical_client_uploads/pcmjw5maadxfxxln8lbh.png)

Every node has all previous nodes as its parents. Each is labeled with its entropy (conditioned on all parents). All entropies are approximate (and several are underestimated here).

All entropies are approximations estimated by sampling

Each node is labelled with its conditional entropy given its parents. We can use these to see how well this model does according to the $\chi$ scores. To work out a score $\chi(A)$, we look at a subset of $X$ variables and add up the conditional entropy of every $Y$ variable that contributes to that set. We then compare that to $H(X_A)$. $\chi(A)$ will always be bigger than $H(X_A)$, and if they’re close together the score is good. If we try $A = \{1\}$ for the autoregressive model, we get $\chi(\{1\}) = 13.2$. This is the same as $H(X_{\{1\}}) = 13.2$, so the score is perfect at $A = \{1\}$.

In contrast, if we try $A = \{36\}$, we get $\chi(\{36\}) = 187.5$. This is far higher than $H(X_{36})=13.2$, so the score is bad at $A=\{36\}$. Similarly, $\chi(\{27..36\}) = 187.5 >> 50.3 = H(X_{\{27..36\}})$, and most other sets have very bad scores.

So let’s look at another model structure. We could fit a generative model of all datapoints simultaneously using a GAN, VAE or Normalizing Flow. These generative models all involve sampling vector from a simple distribution and then passing it through a neural net to produce a sample from the learned distribution.

![Generative model from noise(1).png](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1772554480/lexical_client_uploads/zwcquykf6xunpaperlaf.png)

This has a bad score for most choices of $A$, especially where $A$ is a small subset. The only good score is $\chi(\{1..36\})$.

In contrast, if we represent it as a Bayes net like this, we can see the structure more clearly:

![bayes_net_diagram.png](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1772554557/lexical_client_uploads/aaiprdlpmjnmzsj8e8o8.png)

$\mu_i$ is a variable that contains the cluster center, $G_i$ contains the group center (a cluster containing two clusters), and $S$ is a 3 outcome random variable which chooses whether the covariance matrix of every $X$ is diagonal right, diagonal left, or round.

This has a much better score for most subsets. For example, $\chi(\{36\}) = 4.14+8.74+12.98+1.58 = 27.44$. Compare this to $H(X_{36}) = 13.2$, it’s not too bad, much closer than previous models. For $\chi(\{10..18\}) = 4.14\times8+8.74+12.98+1.58 = 56.42$, it’s close to $H(X_{\{10..18\}})=50.3$.

You might be wondering why these scores aren’t perfect, i.e. why does $\chi(A)$ not equal $H(X_A)$. The difference will be $H(Y_{\supseteq A}|X_A)$, which is how precisely we can nail down the parameters in $Y_{\supseteq A}$ after observing $X_A$. This should give you some intuition for when it’s possible to make a good condensation: When the relevant latent variables of the model can be precisely estimated from a single observation (or small subset of observations). This is why it works well for clustering parameters. The location parameter of a cluster ($\mu_i$) can be fairly accurately estimated from just a single sample from that cluster.

What we can see is that minimizing the $\chi$ score everywhere requires that we divide up the noise variables in a way that attaches them only to exactly the observation variables that they influence. It also forces each $Y$ variable to have the minimal possible entropy, so any unnecessary entropy will be removed. So we end up with this minimal latent variable model that is factored into meaningful chunks.

Selected Results from Sam’s paper
---------------------------------

### **When is perfect condensation possible?**

Theorem 5.10 in [the paper](https://openreview.net/pdf?id=HwKFJ3odui)[^kc1foyh0oj] gives us the necessary and sufficient properties for a perfect condensation (which is a condensation where every score $\chi(A)$ is equal to $H(X_A)$:

1.  Every $Y_A$ is a function of any $X_i$ if $i\in A$. I.e. only one example suffices to nail down every latent variable above it.
2.  The $Y$ variables satisfy a Bayes net structure[^4i7pjpk5fqc] such that $Y_A$ is a parent of $Y_B$ iff $A$ is a subset of $B$ (i.e. parents are always supersets, children are always subsets).

A third condition to keep in mind is enforced by the definition of a Latent Variable Model (Definition 4.2): Every $X$ variable needs to be recoverable from the $Y$ variables that contribute to it.

### **To what extent will LVMs be approximately equivalent?**

When the scores are perfect there is a kind of exact correspondence (Theorem 5.15). We take a $Y_A$ and all of its parents, grandparents, etc, ($Y_{\supseteq A}$) and consider this combined object to be a concept. Then for any two LVMs with perfect scores, each concept is equivalent in the sense that these random variables are deterministic functions of each other ($H(Y_{\supseteq A} | Z_{\supseteq A}) = 0$).

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/69166b91e408b811a00cdc0f26f402ff17e3e72197fcfc05.png)

$Y_{\supseteq \{3\}}$ is a function of $Z_{\supseteq \{3\}}$ and vice versa, in a perfect condensation. Same for $Y_{\supseteq \{4,5,6\}}$. These are upward cones of latent variables.

Things get more complicated when the score aren’t quite perfect, because the theorem needs to account for information that might be mixed up in surrounding nearby concepts.

Sam has a more general theorem (6.8) for the imperfect case (where the scores aren't perfectly minimized) that extends the above theorem in two ways:

1.  For each $Y_{\supseteq A}$, instead of only having a bound for $H(Y_{\supseteq{A}}|Z_{\supseteq{A}})$, we can bound a more general conditional entropy term $H(Y_{\supseteq{A}}|Z_G)$ where we have some freedom in choosing the set $G$. So we'll usually have $G$ be a slightly larger set than $\supseteq A$ (so it captures most of the information that has "leaked" into surrounding concepts) and then the theorem will show that we can almost recover $Y_{\supseteq{A}}$ from $Z_G$.
2.  Insofar as the "Bayesnet structure" of perfect condensation is violated, we can bound the conditional entropy $H(Y_{\supseteq{A}}|Z_G)$ by "how much the Bayesnet structure is violated", which is measured by some conditional mutual information terms.

Correspondences between Natural Latents and Condensations
=========================================================

Here we have a sequence of theorems that show how closely related the theory of Condensation is to the theory of Natural Latents. They are also helpful for highlighting the differences.

Definitions
-----------

### Natural Latent

Let's first recall the definition of a [Natural Latent](https://arxiv.org/pdf/2509.03780). An $\epsilon$-approximate natural latent $\Lambda$ with respect to variables $X_1,...,X_n$ satisfies two conditions:

**Mediation**: $\text{Med}(\Lambda) = \sum_{i\in 1..n} H(X_i|\Lambda)-H(X_{1..n}|\Lambda) \le \epsilon$ [^yhqdmgxasc]

**Redundancy**: $\forall i\in1..n, \text{Red}_i(\Lambda) = H(\Lambda|X_i)\le \epsilon$ [^32a27gklbh3]

### Condensation

Now let's more formally write out a definition of a Condensation. An $\epsilon$-approximate Condensation with respect to variables $X_1,...,X_n$ is a set of latent variables $Y_A$, where $A$ can represent any subset of $1..n$.

The latent variables must satisfy two conditions:

**Conditioned Score**:

The score $\chi(A) = \sum_{B\cap A} H(Y_B | Y_{\supseteq B})$ is called the conditioned score, where $Y_{\supseteq B}$ is the set of $Y$ variables “above” $B$. It’s bounded below by $H(X_A)$.

We have an $\epsilon$-approximate condensation when the conditioned score is almost minimal:

$\sum_{B\cap A} H(Y_B | Y_{\supseteq B}) - H(X_A) \le \epsilon$

**Latent Variable Model condition (**[**Definition 4.2**](https://openreview.net/pdf?id=HwKFJ3odui)**):** We can exactly recover each $X_i$ from the set of latent variables $Y_{\ni i}$ (all the latent variables “above” $i$).

An approximate condensation can be used to construct an approximate weak natural latent
---------------------------------------------------------------------------------------

If we have an approximate condensation of a set of observables, then we can construct a *weak* natural latent of any subset of those observables. A weak natural latent can’t be recovered from any observable, instead it can be recovered from all-but-one (slightly different from the definition above). We construct the natural latent by collecting together all of the condensation latent variables that contribute to more than one of the observables.

**Setup**

We say that a latent variable model satisfies $\epsilon$-approximate condensation if the following holds:

$\chi_{\mathcal{L}}(A)=\sum_{B\cap A\neq\emptyset} H(Y_B|Y_{\supset B})\leq H(X_A)+\epsilon, \forall A\in P^+ I$ [^al79x8nyk0k]

Suppose that we have an $\epsilon$-approximate condensation $(Y_A,A\in P^+ I)$.

Now fix a subset of observables $A$. We construct a natural latent $\Lambda$ over $X_i, i\in A$ by bundling together all latent variables (from the perfect condensation) $Y_B$ with $|B\cap A|\geq 2$.

**Proof**

We'll start by proving the mediation condition.

Denote $Y'_{\{i\}}=(Y_B,B\cap A=\{i\})$ where $i\in A$.

Each $X_i,i\in A$ is a deterministic function of $(\Lambda,Y'_{\{i\}})$, which means:

$H(X_i|\Lambda)\leq H(Y'_{\{i\}}|\Lambda)$ **(1)**

From our assumptions about the conditioned score we get:

$\sum_{i\in A} H(Y'_{\{i\}}|\Lambda)-\epsilon \leq H(X_A)-H(\Lambda)\leq H(X_A|\Lambda)$[^rc8q1oiohr8] **(2)**

Using **(1)** and **(2)**, we can show that the mediation error of $\Lambda$ is less than $\epsilon$:

$\begin{aligned} \text{Med}(\Lambda)&=\sum_{i\in A} H(X_i|\Lambda)-H(X_A|\Lambda)\ &\leq \sum_{i\in A} H(Y'_{\{i\}}|\Lambda)-\sum_{i\in A} H(Y'_{\{i\}}|\Lambda)+\epsilon\ &=\epsilon \end{aligned}$

Now we'll prove the approximate weak redundancy property.

A latent variable $\Lambda$ satisfies the weak redundancy property for variables in $(X_i,i\in A)$ when

$H(\Lambda|X_{A/\{i\}})\leq \epsilon, \forall i$

We start with $H(Y_{\cap B})\leq \chi_\mathcal{L}(B)$ from Proposition 5.2 (for arbitrary $B$), and apply the definition of $\epsilon$-approximate condensation to get:

$H(Y_{\cap B})\leq H(X_B)+\epsilon$

Since $X_B$ is a deterministic function of $Y_{\cap B}$, we have that $H(Y_{\cap B}|X_B)=H(Y_{\cap B})-I(Y_{\cap B};X_B)=H(Y_{\cap B})-H(X_B)$.

Combining these we have (for any $B$):

$H(Y_{\cap B}|X_B) \le \epsilon$

If we set $B$ to $A/\{i\}$ for any $i$, then this gives us the weak redundancy property:

$H(\Lambda|X_{A/\{i\}})\leq \epsilon$

This follows because we can turn $Y_{\cap (A/\{i\})}$ into $\Lambda$ because $Y_{\cap (A/\{i\})}$ includes $\Lambda$$$ as well as $(Y_{\{j\}},j\in A,j\neq i)$.

When can we recover an approximate strong natural latent?
---------------------------------------------------------

The previous statement was about weak natural latents. This one tells us when we can get the usual definition of natural latents out of a condensation, and it ends up being exactly when there isn't lower level structure inside of that subset. For both this theorem and the above, we are getting lots of natural latents out of a single condensation.

**Setup**

Given an $\epsilon$-approximate condensation $(Y_B,B\in P^+I)$ that satisfies $\chi_{\mathcal{L}}(B)\leq H(X_B)+\epsilon,\forall B$. For any subset $A$, we define $\Lambda=Y_{\supseteq A}$ and set $\Lambda'=(Y_B,B\cap A\neq\emptyset, |B|\geq 2,B\not\supseteq A)$. If we have $H(\Lambda'|\Lambda)< \epsilon$, then $\Lambda$ is an $\epsilon$-approximate natural latent over $X_A$.

Intuitively, the $H(\Lambda'|\Lambda)< \epsilon$ condition means that there’s approximately no lower-level structure among the variables in $A$. Approximately all the shared information between these variables is stored in $Y_{\supseteq A}$.

**Proof**

To show strong redundancy:

For each $i\in A,\Lambda=Y_{\supseteq A}$ is a deterministic function of $Y_{\ni i}$ (because of the Latent Variable Model condition).

And $H(Y_{\ni i}|X_i) \le \epsilon$ by applying the definition of $\epsilon$-approximate condensation with $A=\{i\}$.

Combining these we have for all $i$:

$H(Y_{\supseteq A}|X_i) \le \epsilon$

Which is the strong redundancy condition.

To show Mediation:

$\Lambda^*=(\Lambda,\Lambda')$ is an approximate weak natural latent for $X_A$ as constructed in the previous section

For any random variable $Z$ we have $H(Z|\Lambda)\leq H(Z|\Lambda^*)+H(\Lambda^*|\Lambda)$[^wido4a4m7pb] **(1)**

We also have $ -H(X_A|\Lambda)\leq -H(X_A|\Lambda^*)$[^ugrr6grot5e] **(2)**

Applying **(1)** to each $H(X_i|\Lambda)$ term and **(2)** to $-H(X_A|\Lambda)$, we have

$\text{Med}(\Lambda)=\sum_{i\in A} H(X_i|\Lambda)-H(X_A|\Lambda)$

$\leq \sum_{i\in A} [H(X_i|\Lambda^*)+H(\Lambda^*|\Lambda)]-H(X_A|\Lambda^*)$

$\leq \sum_{i\in A} H(X_i|\Lambda^*)+n\epsilon-H(X_A|\Lambda^*)$

$=\text{Med}(\Lambda^*)+n\epsilon$

From previous section $\Lambda^*$satisfies approximate mediation$\text{Med}(\Lambda^*)\leq\epsilon$ which means $\text{Med}(\Lambda)\leq \epsilon+n\epsilon$

(The bound can be improved a bit but would result in a more complicated expression, intuitively for each $i\in A$, once we condition on $\Lambda$, $H(X_i|\Lambda)$ only contains latent variables in $Y_{\ni i}$ that are not already included in $\Lambda$, whereas $\Lambda^*$includes latent variables not in $Y_{\ni i}$, which makes $H(\Lambda^*|\Lambda)$ larger.)

If we have a natural latent we can construct a condensation
-----------------------------------------------------------

This is the reverse of the above. Any time we have a natural latent, we can use it to construct a condensation. Because a natural latent is a relatively simple structure, we get a condensation with a simple structure (with double the error because we get error contribution from both the mediation and redundancy properties).

**Setup**

Given a $\epsilon$-approximate natural latent $\Lambda$ over $X_A$, we construct an $2\epsilon$-approximate condensation $(Y_B,B\in P^+A)$ over $X_A$ by setting $Y_A=\Lambda$, $Y_{\{i\}}=X_i,i\in A$ and set the rest of the latent variables empty.

**Proof**

The latent variable model condition is satisfied because $Y_{\ni i}$ (all $Y$ variables above $X_i$) is equivalent to $(Y_{\{i\}},Y_A)=(X_i,\Lambda)$ and $(X_i,\Lambda)$ fully determines $X_i$.

The mediation property of natural latents can be rearranged to give $\sum_{i\in A}H(X_i|\Lambda)\leq H(X_A|\Lambda)+\epsilon$.

We substitute that into the conditioned score to get this bound:

 $\begin{aligned} \chi(A) &=H(\Lambda)+\sum_{i\in A} H(Y_{\{i\}}|\Lambda)\ &=H(\Lambda)+\sum_{i\in A} H(X_i|\Lambda) \ &\leq H(\Lambda)+H(X_A|\Lambda)+\epsilon\ &=H(X_A,\Lambda)+\epsilon \end{aligned}$

Then we can use the strong redundancy property $H(\Lambda|X_A)\leq \epsilon$:

$H(X_A,\Lambda)+\epsilon\= H(X_A)+H(\Lambda|X_A)+ \epsilon\\\leq H(X_A)+2\epsilon$

Which is a $2\epsilon$ bound on the conditioned score:

$\chi(A) \le H(X_A) + 2\epsilon$

The conditioned score $\chi(B)$ for each subset $B \subset A$ is also bounded by following the same reasoning.

Comparison of agreement theorems in condensation and natural latents
====================================================================

The condensation agreement says that if we have two perfect condensations $(Y_A,A\in P^+I)$ and $(Z_A,A\in P^+I)$ on a collection of observables $(X_i,i\in I)$, then $Y_A$ will be a deterministic function w.r.t $Z_{\supseteq A}$ for all subsets $A$, similarly $Z_A$ will be a deterministic function w.r.t $Y_{\supseteq A}$ for all subsets $A$.

The natural latent agreement condition says that if we have two natural latents $\Lambda, \Lambda'$ w.r.t a collection of observables $(X_i,i\in I)$, then $\Lambda$ and $\Lambda'$ will be deterministic functions of each other

The natural latent agreement theorem can be thought of as a special case of the condensation agreement theorem. To see this, suppose that we have two natural latents $\Lambda,\Lambda'$ over some observables $(X_i,i\in I)$. We can construct two latent variable models $(Y_A,A\in P^+I)$, $(Z_A,A\in P^+I)$ by setting $Y_I=\Lambda, Z_I=\Lambda'$ and $Y_{\{i\}}=Z_{\{i\}}=X_i,\forall i\in I$. Note that both $Y$ and $Z$ will be perfect condensations since our construction is identical to the theorem in the previous section.

By the condensation agreement theorem, $Y_I$ is a function of$Z_{\supseteq I}$, but $Z_{\supseteq I}=Z_I$ as there is no strict super set of $I$ in $P^+I$,[^al79x8nyk0k] this means that $Y_I$ is a function of $Z_I$, applying this in reverse also gives us $Z_I$ is a function of $Y_I$. Since we initially set $\Lambda=Y_I$ and $\Lambda'=Z_I$, we conclude that $\Lambda$ and $\Lambda'$ are deterministic functions of each other.

In summary, condensation provides agreement result in a more general setting where your latent variable can be decomposed into smaller pieces (instead of just being a single undifferentiated latent variable ), and we can reason about uniqueness of those smaller pieces. Of course, in this more general case we don’t get exact isomorphism of the latent variable model, instead we have the weaker guarantee that $Y_A$ is a function of $Z_{\supseteq A}$.

*This post was written as part of the* [*PIBBSS × Iliad research residency*](https://princint.ai/programs/residency/)*. Daniel C has more results about relationships between Natural Latents and Condensations but we didn't have time to include them all in this post, hopefully they'll be posted separately later.*

[^i2gd03a2czq]: More precisely we should say inter-subjective, since it depends on each agent having similar observations, among other shared assumptions. 

[^kc1foyh0oj]:  

[^4i7pjpk5fqc]: A Bayes net is a set of conditional independence statements. By satisfy I mean that it has to satisfy all of these independence statements, but it might also be compatible with more. 

[^yhqdmgxasc]: Usually written as \(KL[P(X_{1..n},\Lambda)||P(\Lambda)\prod_{i\in1..n} P(X_i|\Lambda)] \le \epsilon\) in John’s posts 

[^32a27gklbh3]: This is the deterministic redundancy condition from here. 

[^al79x8nyk0k]: We're using \(P^+\) for the powerset minus the empty set, and \(I\) is just \(\{1..n\}\). So \(P^+ I\) is the set of all possible latent variable indices.\(\) 

[^rc8q1oiohr8]: We start with \(\sum_{i\in A} H(Y'_{\{i\}}|\Lambda)+H(\Lambda)\leq \sum_{B\cap A\neq\emptyset} H(Y_B|Y_{\supset B})=\chi_{\mathcal{L}}(A)\)(to see this, expand \(H(\Lambda)\) by chain rule, and we can see that the LHS condition on more variables than RHS which only conditions on upstream variables, and conditioning can only reduce entropySimilarly, expand \(H(Y'_{\{i\}}|\Lambda)\) by chain rule, we can see that the LHS condition on more variables than RHS for subsets \(B\) with \(B\cap A=\{i\}\))Applying the definition of \(\epsilon\)-approximate condensation:\(\sum_{i\in A} H(Y'_{\{i\}}|\Lambda)+H(\Lambda)\leq H(X_A)+\epsilon\)which we can rearrange to get:\(\sum_{i\in A} H(Y'_{\{i\}}|\Lambda)-\epsilon \leq H(X_A)-H(\Lambda)\leq H(X_A|\Lambda)\)  

[^wido4a4m7pb]: \(\begin{aligned} H(Z|\Lambda)+H(\Lambda)&=H(Z,\Lambda)\leq H(Z,\Lambda,\Lambda^*)\\ &=H(\Lambda)+ H(\Lambda^*|\Lambda)+H(Z|\Lambda,\Lambda^*)\\ &\leq H(\Lambda)+ H(\Lambda^*|\Lambda)+H(Z|\Lambda^*) \end{aligned}\)Subtracting \(H(\Lambda)\) by both sides gives the desired inequality. 

[^ugrr6grot5e]: \(H(X_A|\Lambda^*)=H(X_A|\Lambda,\Lambda')\leq H(X_A|\Lambda)\)