# Proof Explained: Touchette-Lloyd Theorem

*This is a sequel to our* [*previous post*](https://www.lesswrong.com/posts/6aqPCfp3wCk6CKW5J/) *on the Touchette-Lloyd theorem*[^jlxykvkoz7e]*. The previous post contained some introductory material and motivation for the theorem. Here, we will walk through the proof of the theorem and explore its applications in a few worked examples. It isn't strictly necessary to read that post before this one, but we recommend it if anything in this post seems unclear or if you would like some more background.*

Recap and Setup
===============

The theorem concerns three random variables. These are: the input 'environment' variable denoted by $X$, the 'action' taken by a policy denoted by $A$, and the output environment variable, denoted by $Y$. The environmental output is affected by the action and the initial environmental state. The way in which it is affected is given by expressed by a conditional probability distribution $P(Y|X,A)$, which we will refer to as the 'dynamics' of the system. If $P(Y|X,A)$ is equal to either 0 or 1 for all values of $X, Y, A$ then we will say that the dynamics is deterministic, but we will also allow for stochastic dynamics. As is standard, we will denote specific values that the random variables can take using lower case (eg. $x_1, x_2 ...$ are values which $X$ can take, $a_1, a_2...$ are values which $A$ can take). The variables $X$ and $Y$ both represent the state of the 'environment' at different times (before and after the 'action' is taken). They will therefore both have the same domain (set of possible values) but we will nonetheless use $y_1, y_2...$ (as opposed to $x_1, x_2...$) to denote $Y$-values to avoid confusion over variables.

The way in which the action variable $A$ depends on the input environmental state $X$ is called the 'policy'. The theorem concerns the comparison of two classes of polices: 'Blind' policies where the action cannot depend on (or 'see') the environmental state $X$ and 'sighted' policies where the action taken can depend on $X$. [^gu3nrcoano] These policies can be represented by Bayes nets:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d162c29af2fcaf9611c535dc4553ec610c0040d3dab0c2f7.png)

The theorem statement is the inequality

$$
\Delta H \leq \Delta H_{Blind}^{Max}  + I(X;A)
$$

which relates the entropy change $\Delta H = H(X)-H(Y)$ for a sighted policy with mutual information $I(X;A)$ between the environment and action to the maximum possible entropy change achievable for a blind policy $\Delta H_{Blind}^{Max}$. 

The Proof
=========

We will use the subscript $B$ to indicate a blind policy (so $Y_{B}$ is the environmental output for a blind policy, $A_B$ is the action variable for a blind policy). First, let us look at what the output distribution looks like for a blind policy.  The dynamics $P(Y|X,A)$ is fixed and $X$ and $A$ are independent so the probability distribution over $Y$ will be

$$
P(Y_{B} = y) = \sum_{x,a}P(y|x,a)P(X=x)P(A_B=a) \, .
$$

Let us consider the performance of a policy like this in producing a distribution with a low final entropy. In particular, we can quantify this using the difference between the initial environmental entropy $H(X)$ and the final environmental entropy $H(Y)$:

$$
\Delta H = H(X)-H(Y).
$$

Quantifying entropy reduction in this way is intuitive and captures the fact that (other things being equal) it is more impressive for a agent or controller to achieve a low output entropy from a high starting entropy than to start with a low entropy and maintain it.

For any given initial distribution, there is a maximum $\Delta H$ which can be achieved using a blind policy, but $\Delta H$ depends on the initial distribution. Consider the set $\mathcal{X}$ of all possible initial probability distributions $P(X)$, that is, all distributions over the state space containing $x_0, x_1$, ... etc. For each $P(X)\in \mathcal{X}$ there is a 'best' $\Delta H$ which can be achieved using a blind policy and cannot be surpassed by another blind policy acting on the same input distribution. Furthermore, there is some combination of initial distribution and policy which achieves a value of $\Delta H$ such that there is no other combination of blind policy and initial distribution which achieves a greater $\Delta H$. Let this optimal value of $\Delta H$ achievable by a blind policy be labelled $\Delta H_{Blind}^{Max}$:

$$
\Delta H_{Blind}^{Max} :=  \max_{P(X)\in \mathcal{X}, P(A)\in \mathcal{A}} \Delta H \, ,
$$

where $\mathcal{A}$ is the set of all distributions over actions which correspond to blind policies.

We can ask: is there a combination of initial distribution and *sighted* policy which can outperform this value of $\Delta H$? And if so, by how much does it outperform this value? We will answer this question by investigating some general properties of sighted policies.

For a general 'sighted' policy, the action $A$ will depend on the environmental state $X$, so the output distribution will include this dependency

$$
P(Y = y) = \sum_{x,a}P(y|x,a)P(A=a|X=x)P(X=x) \, .
$$

Notice that we can use Bayes rule to re-write the conditional probabilities relating $X$ and $A$. In particular, we can write

$$
P(A=a|X=x)P(X=x) = P(X=x|A=a)P(A=a)
$$

where

$$
P(A=a) = \sum_x P(A=a|X=x) P(X=x).
$$

Note that this doesn't imply that the *causal* relationship between $X$ and $A$ has changed. We are still considering the case where the policy changes its action depending on what environmental state it 'sees'. We are just re-writing the probabilities.

Substituting this back in and rewriting the order of things, we can write the output distribution for a general policy as

$$
P(Y = y) = \sum_a P(A=a) \Bigg[\sum_xP(y|x,a)P(X=x|A=a)\Bigg] \, .
$$

Notice that this is a weighted average over probability distributions for each $a$-value. If we consider only the cases where the action taken is one particular $a$-value, say $a_0$, we can obtain a conditional probability distribution over $Y$:

$$
P(Y = y|A=a_0) =\sum_xP(y|x,a_0)P(X=x|A=a_0) .
$$

Interestingly, (and this is the key insight of the proof) we can think of this conditional output distribution for a sighted policy as the output distribution of a *blind* policy which always selects $A=a_0$ acting on an input distribution $Q_{a_0}(X) = P(X|A=a_0)$ [^xae73t1gwfd].  This means that the difference between the entropy of $Q_{a_0}(X)$ and $P(Y = y|A=a_0)$ is upper bounded by the blind policy entropy reduction $\Delta H_{Blind}^{Max}$. We can therefore write

$$
H(X|A=a_0) - H(Y|A=a_0) \leq \Delta H_{Blind}^{Max} \, .
$$

Furthermore, since $a_0$ was arbitrary, this inequality holds for every possible $a$-value, meaning that we can sum over the $a$-values used by the sighted policy, weighted by $P(A=a_i)$:

$$
\sum_{i} P(A=a_i)H(X|A=a_i) - \sum_{i} P(A=a_i)H(Y|A=a_i) \leq \Delta H_{Blind}^{Max} \, .
$$

Recall the definition of [conditional entropy](https://en.wikipedia.org/wiki/Conditional_entropy) which allows us to write:

$$
H(X|A) = \sum_{i} P(A=a_i)H(X|A=a_i) \quad \text{ and } \quad H(Y|A) = \sum_{i}P(A=a_i) H(Y|A=a_i) \, .
$$

This allows us to simplify this equation and obtain

$$
H(X|A) - H(Y|A)\leq \Delta H_{Blind}^{Max} \, .
$$

To turn this bound on conditional entropies into a bound on $\Delta H$ for a general policy, we can add the entropy of the initial distribution $H(X)$ to both sides and re-arrange to obtain:

$$
H(X) - H(Y|A)\leq \Delta H_{Blind}^{Max} + H(X) - H(X|A) \, .
$$

Now, note two things. First, since $H(Y|A)<H(Y)$, we can write 

$$
\Delta H = H(X)-H(Y) \leq H(X)-H(Y|A)
$$

Second, note that $H(X) - H(X|A)$, the difference between the entropy of $X$ and the entropy of $X$, conditional on $A$, is the definition of the mutual information between $X$ and $A$ so $H(X) - H(X|A) = I(X;A)$. Combining these two observations, we arrive at the following inequality relating the entropy reduction achievable through sighted policies to the blind entropy reduction:

$$
\Delta H \leq\Delta H_{Blind}^{Max} +  I(X;A)
$$

which is the result!

What does this result tell us?
==============================

From the point of view of the agent structure problem, we can view this result as a [selection theorem](https://www.lesswrong.com/posts/G2Lne2Fi7Qra5Lbuf/selection-theorems-a-program-for-understanding-agents) which tells us that, if we select for (or observe) a policy which achieves an entropy reduction $\Delta H$ which is greater than $\Delta H_{Blind}^{Max}$ then, at minimum, its actions must share an amount of mutual information $\Delta H- \Delta H^{Max}_{Blind}$ with the environment. 

Examples of different dynamics
==============================

To gain some intuition for this result, we will explore its implications through a few examples. These can have very different feels, but in all cases, the dynamics are fully specified by defining the conditional distribution $P(Y|x,a)$ for all $x$ and $a$.

1\. Entropy decreasing dynamics
-------------------------------

First, we will consider an entropy *decreasing* dynamics. In this example the entropy of the environment will naturally tend to decrease, without help from the policy. A simple example of such a dynamics is one where all input states are mapped to the same output, regardless of the action taken by the policy. If we consider each state to be labelled by a 5-bit string the following probability distribution has this property:

$$
P(Y=00000|x,a)=1 \quad \text{for all $x,a$} \, .
$$

If we only observed environmental states, we would see that regardless of the entropy of the initial distribution over input strings, the output distribution would have zero entropy. Naively, one might attribute this entropy reduction to an effective policy using information about its environment. But in fact, the entropy reduction is a property of the environmental dynamics, *not* the policy. This can be seen by the fact that no policy, blind or sighted, can achieve a greater entropy change than that achieved by a blind policy. For this dynamics, $\Delta H_{Blind}^{Max} = 5$, and no policy can achieve a greater entropy reduction. In this sense, $\Delta H_{Blind}^{Max}$ tells us the the degree to which the environment itself contains entropy-reducing dynamics. We can see a slightly more subtle example of this by considering the following environmental dynamics

$$
P(Y=00000|x,A=11111)=1 \quad \text{for all $x$}
$$

$$
P(Y=s| X=s, A\neq11111)=1 \quad \text{for all strings $s$}
$$

In words: the output state is equal to the input state, unless the action taken is $A=11111$, in which case the output state is $Y=00000$. This dynamics contains two subdynamics: one (where $A\neq11111$) which is entropy preserving and one (where $A=11111$) which is entropy decreasing. A blind policy can guide the dynamics to become entropy decreasing (by picking $A=11111$) and thus achieve maximal entropy reduction without requiring knowledge of the input state. In this sense, $\Delta H_{Blind}^{Max}$again captures the entropy-reducing qualities present in the environment itself, as opposed to the entropy-reducing qualities of a policy.

2\. Entropy non-increasing dynamics
-----------------------------------

Let's return to the example given in the [previous post](https://www.lesswrong.com/posts/6aqPCfp3wCk6CKW5J), where the policy has to guess the correct 5-bit string. The dynamics of this example is deterministic and characterised by the following conditional probabilities

$$
P(Y=00000|x,a) = \begin{cases} 1 \text{ if } a = x \ 0 \text{ otherwise}  \end{cases}
$$

$$
P(Y=x|x,a) = \begin{cases} 1 \text{ if } a \neq x \ 0 \text{ if } a=x  \end{cases} \quad \text{ for  $x\neq00000$ } \, .
$$

In words: the output state is $Y=00000$ if the action picked has the same string as the input state. Otherwise the output state is the same as the input state.

First, we will find $\Delta H_{Blind}^{Max}$. We do this by finding a blind policy and input distribution which maximizes the change in entropy between the input distribution and the output distribution. Note that the only way that a policy can reduce the entropy is to guess the correct string and that randomizing which string is guessed can do no better than picking the same (highest probability) string each time. Furthermore, the dynamics are symmetric with respect to all strings except 00000 so without loss of generality, we can assume that the best blind policy involves picking the string $A=11111$ with probability one. Now, we need to find which input distribution $P(X)$ maximizes $\Delta H$ for this policy.

First note that if $P(X)$ is restricted to only having nonzero probabilities over $X=00000$ and $X=11111$, then the final distribution will always have zero entropy. Therefore, in this restricted case, the maximum entropy *reduction* is achieved when the input distribution has maximum entropy (ie. $P(X=00000) = P(X=11111) = \frac{1}{2}$ which has 1 bit of entropy). This leads to an entropy reduction of 1 bit. We will now show that this distribution is in fact optimal and $\Delta H_{Blind}^{Max}=1$.

Note also that the above argument can be extended to show that, for *any* initial distribution, the blind policy will always achieve a greater entropy reduction when the probabilities of 00000 and 11111 are equal compared to when they are not. 

Now, notice that any input distribution can be viewed as a mixture between a distribution over the domain $\{00000,11111\}$ which we will call $P(X_0)$ and some other distribution over a domain not including 00000 and 11111 which we will call $P(X_1)$. Thus we can write the optimal input distribution as:

$$
P(X_{opt}) = \lambda P(X_0) +(1-\lambda)P(X_1)
$$

where $P(X_{opt})$ is the optimal distribution over $X$ and $0\leq \lambda \leq 1$ is a 'mixing probability'. Since this distribution is a mixture of two distributions over non-overlapping domains, its entropy is given by:

$$
H(X_{opt}) = \lambda H(X_0) + (1-\lambda) H(X_1) + h(\lambda)
$$

where $h(\lambda) = \lambda \log(\frac{1}{\lambda}) + (1-\lambda) \log(\frac{1}{1-\lambda})$  is the [binary entropy](https://en.wikipedia.org/wiki/Binary_entropy_function) of the mixing constant (a nice proof of this fact can be found [here](https://math.stackexchange.com/questions/4271669/proving-the-decomposibility-of-entropy/4273155#4273155)). 

Now let us consider the output entropy for this distribution, under a blind policy. With probability $\lambda$, the output will be 00000, since any result in the domain $\{00000,11111\}$ will be mapped to 00000. We will call this outcome $Y_0$ which has zero entropy so $H(Y_0)=0$. With probability $1-\lambda$, the output will be a random variable from the distribution $P(X_1)$. The entropy of the total output can similarly be thought of as a mixture of non-overlapping distributions meaning that we can write it as

$$
H(Y) = \lambda H(Y_0) + (1-\lambda)H(X_1) +h(\lambda)
$$

By subtracting $H(Y)$ from $H(X_{opt})$ we can find the change in entropy between input and output:

$$
\Delta H_{Blind} = H(X_{opt})-H(Y)  = \lambda \Big[ H(X_0)- H(Y_0)\Big]
$$

The terms involving distributions over the domain excluding $\{00000,11111\}$ cancel out. Therefore, the blind entropy reduction is maximized by setting $\lambda=1$ and $P(X)$ as a maximum entropy distribution over the $\{00000,11111\}$ domain. We have therefore proved that $\Delta H_{Blind}^{Max}=1$. This means that if we observe a policy achieving an entropy reduction of greater than 1 bit on some input policy, we know that policy must be sighted-its actions must have some mutual information with the environmental state. In practice, this means that the policy must be taking in information about the environmental state.

Suppose we now observed a policy which achieved an output entropy of 2.83 bits when fed a maximum entropy input distribution with $H(X)=5$ bits. Armed with the Touchette-Lloyd theorem, what can we deduce about this policy?

The entropy reduction achieved by this policy is $\Delta H = 5-2.83=2.17$ bits. Since the inequality tells us

$$
\Delta H-\Delta H_{Blind}^{Max} \leq   I(X;A)
$$

this means that our policy must have at least $2.17 -1 =1.17$ bits of mutual information with the environment. In fact, the policy which achieves this entropy change is the policy described in the [first post](https://www.lesswrong.com/posts/6aqPCfp3wCk6CKW5J), in the section titled 'A Formal Example'. This policy takes the first four bits of the environmental state (and then selects the action $A=****1$ where $****$ are the first four bits of $X$) and thus has four bits of mutual information. This example raises an important point about the Touchette-Lloyd inequality which is that while it is true (4 bits is indeed greater than 1.17 bits) it doesn't always saturate. It would be nice if we could get a higher lower bound on the mutual information. There are two related ways to use the inequality to tighten the bound.

First, note that in the derivation of the inequality, we used the fact that $H(Y|A)<H(Y)$, so unless we have a situation where $H(Y)=H(Y|A)$, we cannot expect the bound to be tight. In a step near the end of our derivation, we obtained the inequality

$$
H(X) - H(Y|A)\leq \Delta H_{Blind}^{Max} + H(X) - H(X|A)
$$

Since $H(X)-H(X|A)=I(X;A)$, we can use this to obtain a tighter bound:

$$
I(X;A)\geq H(X)-H(Y|A) -\Delta H_{Blind}^{Max}
$$

In our case, while $H(Y)=2.83$, we have $H(Y|A) = 1$. This is because we can tell the first four bits of $Y$ by looking at $A$. For a uniform input distribution, given $A=****1$, the probability $P(Y=****0)=\frac{1}{2}$ and $P(Y=00000)=\frac{1}{2}$ .  Using $H(X)=5$ and $\Delta H_{Blind}^{MAX}=1$ yields the improved bound

$$
I(X;A)\geq 5 - 1 -1 = 3 \, .
$$

So in fact, we can improve on the bound provided by the original Touchette-Lloyd inequality, provided that we are willing to replace $\Delta H$ with the slightly less intuitive quantity $H(X)-H(Y|A)$. However, the advantage of using the inequality in terms of $\Delta H$ is that knowing $\Delta H$ involves only observing changes in the environment and does not presume knowing all the actions which the policy takes. After all, if you already know what actions a policy is taking, you can probably calculate the mutual information already. From a selection theorem point of view we are more interested in inferring that a policy is using information from its environment by observing the changes in the environment.

Another way to increase the lower bound on $I(X;A)$ is to observe the policy for different initial distributions. For example, if the policy was run on a distribution which was uniform over all 5-bit strings which end in '1' then the initial entropy would be $H(X)=4$ bits but the policy would reduce the final entropy to $H(Y)=0$, achieving $\Delta H=4$. When plugged into the Touchette-Lloyd inequality (along with $\Delta H_{Blind}^{Max}=1$) this would give a lower bound of 3 bits on the mutual information which is much tighter than the 'unoptimized' value of 1.17 bits (recall, the true mutual information between the action and environment for this policy is 4 bits). 

3\. Entropy increasing dynamics
-------------------------------

We described the dynamics above as 'entropy non-increasing', since it is impossible for any policy to lead to an output distribution with a higher entropy than the input. We can also consider dynamics where entropy can increase and a policy has to be more proactive in fighting this entropy increase.

Consider the following dynamics, again in a situation where both environmental state and the action are described by 5-bit strings. If the action matches the environmental state (ie. if $a=x$), then the output environmental state is $y=00000$. On the other hand, if the action does not match the state (ie. if $a\neq x$) then $Y$ is drawn from a uniform distribution over all 5-bit strings excluding 00000. This has dynamics corresponding to the following distribution

$$
P(Y=00000|x,a) = \begin{cases}1 \text{ if $x=a$} \ 0 \text{ if $x\neq a$}\end{cases}
$$

$$
P(Y=s|x,a) =\begin{cases} 0 \text{ if $x=a$} \ \frac{1}{2^5-1} \text{ if $x\neq a$} \end{cases} \quad\text{ for $s \neq 00000$}
$$

Using a similar argument to the one in the previous section, we can find that the maximum blind policy entropy change is 0 bits, achieved when the initial distribution (and the corresponding action string) is completely concentrated on one environmental state. This is because any initial distribution which allows the policy to guess wrong will be punished by a high-entropy final state leading to a negative $\Delta H$.

Consider the sighted policy from the previous section which can 'see' the first four bits and chooses the action $****1$ in this new environment. For a uniform initial distribution, this leads to a final distribution with $P(Y=00000)=\frac{1}{2}$ and $P(Y=s)=\frac{1}{2(2^5-1)}$ for strings $s \neq 00000$. This output distribution has an entropy of $H(Y)=3.48$ so $\Delta H = 1.52$, leading to the lower bound on the entropy of $I(X;A)\geq 1.52$. 

Similarly to the previous section, we can improve on this bound by changing the initial distribution. If we consider the case where the initial distribution is over all strings which end in the bit '1' then this sighted policy chooses the correct action every time and reduces the final entropy to zero. This corresponds to $\Delta H =4$. Since $\Delta H_{Blind}^{Max}=0$, this gives us the bound $I(X;A)\geq 4$. Recall that this policy does indeed have 4 bits of mutual information with the environment, so this bound cannot be improved upon. 

Moving to an environment which was more hostile to entropy reduction leads to a greater discrepancy between blind and sighted policies, this in turn allows the bound on mutual information provided by the Touchette-Lloyd inequality to be made tighter than in the 'entropy non-increasing' environment of the previous example.

Incidentally, this environment also highlights why, when calculating $\Delta H_{Blind}^{Max}$, the maximum must be taken over all initial distributions. It would be convenient if the inequality was true but $\Delta H^{Max}_{Blind}$ was instead defined as the maximum change in entropy achievable for a blind policy with the same initial distribution as the $\Delta H$ term in the inequality. But this is not the case, instead $\Delta H^{Max}_{Blind}$ is defined as the maximum entropy change achievable for a blind policy over *any* initial distribution. We can see why this must be the case by considering this environment.

If the initial distribution has intermediate entropy (ie. nonzero but not maximal) $H(X)$, then a sighted policy can achieve a zero entropy final distribution if it has full knowledge of the environmental state, meaning $I(X;A)=H(X)$. A blind policy acting on this initial distribution will, most of the time guess the incorrect action to take, leading to the final state being drawn from an almost maximum entropy distribution. This can lead to the maximum blind policy entropy change being *negative, (*ie. the final distribution has a larger entropy than the initial distribution) for intermediate entropy distributions. If we define $\Delta H_{Blind}^P$ to be the maximum entropy change for a blind policy acting on an initial distribution $P$ and consider the $\Delta H$ for a sighted policy acting on the same initial distribution we might conjecture the following inequality

$$
\Delta H \leq \Delta H_{Blind}^P +I(X;A) \, .
$$

(NOTE: THIS INEQUALITY IS NOT TRUE!)

But we have just argued that a sighted policy can achieve $\Delta H = H(X)$ using mutual information $I(X;A)=H(X)$ and that $\Delta H_{Blind}^P$ can be negative for the same initial distribution. Therefore the above inequality cannot hold.

On the other hand, if we define $\Delta H_{Blind}^{Max}$in the way that Touchette and Lloyd do, as the maximum blind entropy change achievable over *all* input distributions, then we can note that for a zero entropy initial distribution (ie. one where $X$ only takes a single value), a blind policy can achieve a zero entropy output distribution by choosing the correct action 100% of the time. This means that while $\Delta H_{Blind}^P$ was negative $\Delta H_{Blind}^{Max}$ will equal zero, making the inequality $\Delta H \leq \Delta H_{Blind}^{Max} +I(X;A)$ valid. Understanding the slightly unintuitive way that $\Delta H_{Blind}^{Max}$ is defined is important for using the Touchette-Lloyd theorem. We highlight this here as it was something that we took a long time to properly appreciate.

Conclusions & Further Work
==========================

We hope that the Touchette-Lloyd theorem is clear to you after reading this. We think that it is pretty interesting result from an Agent Foundations point of view and are interested in seeing if we can extend it. We have some ideas for a few avenues for further research.

*   **Re-frame the result using expected utility.** Reducing entropy and maximizing expected utility are (in some regards) similar in that they are both forms of optimization. If one policy achieves a particular expected utility (for some utility function over $X$) can we ascribe some lower bound on the mutual information between its actions and the environmental state?
*   **Extend the result to a non-Markovian environment.** We assumed here that the environment was Markovian so that X could be treated as a random variable independent of its previous value. But most interesting environments do not have this property. Does a similar result hold when a policy must 'remember' previous environmental states to choose optimally at a given timestep?
*   **Policies which 'model' the dynamics, not just the state.** The Touchette-Lloyd theorem broadly tells us when a policy has 'knowledge' of the environmental *state.* But having mutual information between the state and does not necessarily imply successful entropy reduction. In order to successfully reduce entropy, one must also choose actions which are appropriate for the *dynamics* of the environment. Can we prove a similar which captures whether a policy is modelling the dynamics of the system?
*   **Prove an equivalent result using K-complexity.** Entropy reduction is one hallmark of an optimizer. But if a system reduces in complexity or description length, it could also reasonably be said to be 'being optimized'. Can we prove a similar theorem involving the reduction of K-complexity, instead of entropy?

[^jlxykvkoz7e]: That is, theorem 10 from the paper  Information-theoretic approach to the study of control systems. 

[^gu3nrcoano]: In control theory terminology, these families of policies are known as 'open loop' and 'closed loop' respectively. But as a personal preference we find the names 'blind' and 'sighted' more evocative so we will use them instead. 

[^xae73t1gwfd]: Touchette and Lloyd describe this as "the fact that a closed-loop controller is formally equivalent to an ensemble of open-loop controllers acting on the conditional supports \(\text{supp}(X|c)\) instead of \(\text{supp}(X)\)" (where they are using \(c\) to represent a specific controller action, equivalent to our \(a_0\))