# Improving Internal Model Principle

*Funded by the Advanced Research + Invention Agency (ARIA) through project code MSAI-SE01-P005*

*This post was written during the* [*Dovetail Research*](https://dovetailresearch.org/) *Fellowship. Thanks to* [*Alex*](https://www.lesswrong.com/users/alex_altair) *and* [*Alfred*](https://www.lesswrong.com/users/alfred-harwood?mention=user) *for help during the writing process and to all the fellows for the fruitful discussions.*

*I plan to share extended and more formal version of this post as Arxiv preprint, it will be linked here when its finished. *

Overview
========

The Abstract Internal Model Principle, introduced by Wonham in [Towards Abstract IMP](https://www.researchgate.net/publication/3115882_Towards_an_Abstract_Internal_Model_Principle) as a generalization of a tool from control theory, has received some attention from various fields. 

[@Alfred](https://www.lesswrong.com/users/alfred-harwood?mention=user) and [@Jose](https://www.lesswrong.com/users/josefaustino?mention=user) discussed it in context of AI Safety/Agency in posts:

*   [The Internal Model Principle: A Straightforward Explanation ](https://www.lesswrong.com/posts/fsQzYmuFN65vHcEcH/the-internal-model-principle-a-straightforward-explanation) 
*   [Distilling the Internal Model Principle](https://www.lesswrong.com/posts/g4avk6cLHomHjcFkx/distilling-the-internal-model-principle) 
*   [Distilling the Internal Model Principle part II](https://www.lesswrong.com/posts/NZdWYaFpekQJu6LF5/distilling-the-internal-model-principle-part-ii) 
*   [Seven ways to Improve the Internal Model Principle](https://www.lesswrong.com/posts/5NPLrAt9SKWiKPvnw/seven-ways-to-improve-the-internal-model-principle) 

Last of those posts and Imran Thobani's paper [A triviality worry for the internal model principle](https://www.researchgate.net/publication/382177509_A_triviality_worry_for_the_internal_model_principle) raise issues with applying IMP to agents. The paper [General agents contain world models](https://arxiv.org/abs/2506.01622) was an inspiration for this work. End goal would be to obtain similar results a wide class of systems.

This post has two main parts:

1.  Explanation of IMP, highlighting its issues,
2.  My alternative version of IMP that deals with some of the issues.

I will be using basic notions from dynamical systems theory, everything needed will be defined in the post but for a more complete treatment one might consult Chapter 1.1 of Introduction to Dynamical Systems by Brin and Stuck (or any other textbook).

Explanation
===========

Dynamical systems intro
-----------------------

A Discrete-time Dynamical system consists of a set $X$ and a function $\alpha: X \rightarrow X$ and typically this function is applied many times represented by $\alpha^n:=\alpha \circ \alpha \circ \cdots \circ \alpha$representing time moving in discrete steps.

 If $(X,\alpha), (Y,\beta)$ are dynamical systems, a morphism is a map $\psi:X\to Y$ satisfying$\psi\circ\alpha = \beta\circ\psi$ . The map $\psi$ is called a semiconjugacy if it is surjective, in this case we say that $(Y,\beta)$ is a factor of $(X,\alpha)$ and  $(X,\alpha)$  is an extension of $(Y,\beta)$, $\psi$ is also called a factor map. If $\psi$ is bijective then it is isomorphism (conjugacy) of dynamical systems meaning that those systems are essentially the same. 

For $x\in X$ we denote its orbit by $O(x):=\big(\alpha^i(x)\big)_{i\in\mathbb{N}}$. For function (observation)  $\gamma:X\to W$  , we write the observed orbit as $O_{\gamma}(x):=\big(\gamma(\alpha^i(x))\big)_{i\in\mathbb{N}}\in W^{\mathbb{N}}$. 

A set $X^+\subset X$ is called invariant in dynamical system $(X,\alpha)$ if $\alpha(X^+) \subset X^+$ meaning that points from this set won't ever leave it. Dynamical system $(X,\alpha)$ can be restricted to $(X^+,\alpha|_{X^+})$.$$

### Examples of dynamical systems

**Finite state systems**

The simplest examples have $X$ finite. For instance, let $X = \{0, 1\}$ with $\alpha(0) = 1$ and $\alpha(1) = 0$. The orbits are $O(0) = (1, 0, 1, 0, \ldots)$ and $O(1) = (0, 1, 0, 1, \ldots)$, which are periodic.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/6630a0b2b94ac5884f3f2d4b46aca4f310aa65c2148237cb.png)

**Linear maps **

Consider $X = \mathbb{R}^n$ and $\alpha(x) = Ax$ for a matrix $A \in \mathbb{R}^{n \times n}$. The orbit of any $ x_0$ is $O(x_0) = (x_0, Ax_0, A^2x_0, \ldots)$. The long-term behaviour depends on the eigenvalues of A: trajectories converge to zero if all eigenvalues have modulus less than 1, diverge if any have modulus greater than 1, and exhibit mixed behaviour otherwise.

**Rotations on the circle**

Let $X = \mathbb{R}/\mathbb{Z}$ (the unit circle) and let$ \alpha(x) = x + \theta \pmod{1}$ for some $\theta \in \mathbb{R}$. If $\theta$ is rational, all orbits are periodic. If $ \theta$ is irrational, orbits are dense in the circle and equidistributed.

**Symbolic dynamics**

Given a finite alphabet $A = \{a_1, \ldots, a_k\}$, the full shift $(A^{\mathbb{N}}, \sigma)$ is the dynamical system on all infinite sequences of alphabet symbols, with $\sigma$ shifting each sequence left by one position. Subshifts are closed, shift-invariant subsets of $A^{\mathbb{N}}$, which arise naturally when imposing constraints on allowed symbol sequences. They are examples of restriction of dynamical system.

Symbolic dynamics occur naturally if instead of considering dynamical system $(X,\alpha)$ one considers system $(O(X),\sigma)$ 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/5a70795144c36d07b54386b0924726b62d6216105e0433f2.png)

Abstract IMP
------------

Here is a very high level summary of what IMP is trying to say:

*If a controller can keep system in good states, then it must contain model of a system. *

A more specific phrasing of what it actually says

*If a controller has some structural properties and can keep system in good states, then one can reconstruct dynamics of good part of the system on the controller states.*

Now a formal statement, assume that there exist:

*   **A dynamical system **$(X,\alpha)$ **representing joint environment-controller states,**
*   **A mapping from each joint state to the corresponding controller state **$\gamma: X \rightarrow W$,
*   **A set **$K$** of desirable environment-controller states** **which is a subset of **$X$; **so we have **$K \subseteq X$** ,**
*   **An invariant subset **$X^+$** of these desirable states **$X^+ \subseteq K$** where the controller will keep the system, i.e **$\alpha(X^+) \subseteq X^+$.

Additionally, assume that: 

*   **The set of desired joint environment-controller states **$K$** involves the full set of controller states **$W$**.** This can be expressed by writing $\gamma(K)=W$. 
*   **Feedback Structure on **$K$** :** for $x,y \in K$ $\gamma(x)=\gamma(y)$ implies $\gamma(\alpha(x))=\gamma(\alpha(y))$
*   $X^+$** is detectable relative to **$(\gamma, \alpha)$ meaning $O_\gamma$ (image of orbit via $\gamma$ ) is injective when restricted to $X^+$

Then:

1.  $\gamma$ restricted to $X^+$ (denoted $\gamma^+$)  $\gamma^{+}: X^{+} \rightarrow W$ is injective.
2.  There exists a unique map $\bar{\alpha}: W \rightarrow W$, such that $$
    1.  $\bar{\alpha} \circ \gamma |_K =\gamma \circ \alpha |_K$ . This map is uniquely determined by $\gamma, \alpha, K$ this implies:
    2.  $\bar{\alpha} \circ \gamma |_{X^+} =\gamma \circ \alpha |_{X^+} $   which we can rewrite as  $\bar{\alpha} \circ \gamma^{+}=\gamma^{+} \circ \alpha$ 

I split 2a and 2b despite them being almost the same to highlight the fact that 1 and 2b shows isomorphism (conjugacy) of dynamical systems $(X^+,\alpha), (W^+,\bar{\alpha})$ while $(K,\alpha)$ fails to be a dynamical system because $K$ is not invariant (one of the examples will expand it). 

We can visualize isomorphism between $(X^+,\alpha)$ and $ (W^+,\bar{\alpha})$ on a commutative diagram:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/da693b2f756bf78d27466d51a9c90dce3144f922b6ff4ccf.png)

Examples
--------

### Triviality

Take the dynamical system $(\{0,1\}, (+1 \mod 2)$, i.e. $0 \to 1$ and $1 \to 0$. With $X=K=X^+=\{0,1\}$. Then any injective map from $\{0,1\}$, for example $0 \to a$ and $1 \to b$, would satisfy the assumptions and allow for reconstruction of dynamics. Since we used $\alpha$ to construct $\bar{\alpha}$ this would mean that a map that renames states of the system contains a model of the system.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/fe252ca8f082831a522dfa4a88d5b11b0c1322dbdf7f843e.png)

### Issues with $K$

Second example shows that Feedback on $K$ does not imply feedback on $\alpha(K)$, this is what prevents IMP from giving meaningful results for whole $K$.

Let $X = \{a, b, c, d, e, f,z\}, W = \{0, 1,2\},K = \{a, b, z\}, X^+=\{z\}$ . Values of gamma are shown next to letters.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/a230c5b3e627ab9d4108cf8ff688902a223383da29921857.png)

$c,d$ are in $\alpha(K)$ but they fail to satisfy Feedback Structure condition because $\gamma(c)=\gamma(d)$ but $\gamma(e) \neq \gamma(f)$. So even if we can say what are values of $\gamma$ outside of $X^+$ it is not usable information. Assumption of $\gamma(K)=W$ guarantees that leaving $K$ won't cause to much of an issue but it feels artificial in abstract context. We get isomorphism between singletons $z,2$ but if you assume that a system has stable point then there is no ambiguity in dynamics anyway.$$

Simplified Abstract IMP
-----------------------

As mentioned above, parts of Abstract IMP that refer to set $K$ raise some concerns. The interesting part is finding isomorphic systems and that happens only for $X^+$. In part III of Wonham's  [Towards Abstract IMP](https://www.researchgate.net/publication/3115882_Towards_an_Abstract_Internal_Model_Principle) that apply it to linear systems (original setting of IMP) both $K$ and $X^+$ have natural definitions that give meaning to their desirability and both Feedback and Detectability conditions are common in control theory. We can restate Abstract IMP with $K=X^+$, using two observations: 

1.  Feedback on $K $ implies feedback on $X^+$, 
2.  Feedback+Detectability+Invariance on a set imply injectivity of $\gamma $ for this set. 

First is trivial, second is an exercise (it is discussed in Jose's [post](https://www.lesswrong.com/posts/NZdWYaFpekQJu6LF5/distilling-the-internal-model-principle-part-ii) and is a part of Abstract IMP proof)

Assume there exist:

*   **A dynamical system  **$(X,\alpha)$ representing joint environment-controller states,
*   **A mapping **$\gamma: X \rightarrow W$ from each joint state to the corresponding controller state 
*   **A subset **$X^+$** of "desirable" states **$X^+ \subseteq X$where the controller will keep the system, i.e $\alpha(X^+) \subseteq X^+$, such that $\gamma$ restricted to $X^+$ (denoted $\gamma^+$)  $\gamma^{+}: X^{+} \rightarrow W$ is injective.

Then:

1.  There exists a unique map $\bar{\alpha}:=\gamma^+\circ\alpha: W^+ \rightarrow W^+$ such that $\bar{\alpha} \circ \gamma^{+}=\gamma^{+} \circ \alpha$ . This map is uniquely determined by $\gamma, \alpha, X^+$

When we assume that $\gamma^+$ is injective instead of stating in indirectly with Feedback and Detectability, the theorem looks far less impressive. This is equivalent to saying that if there is injective function from dynamical system, then one can reproduce the dynamics on its image. 

Perhaps instead of thinking about IMP as theorem one could consider it a definition, $(W^+,\bar{\alpha})$ would then be internal model that controller $\gamma$ has of the system $(X,\alpha)$. That definition still has the same issues but it is easier to see them. 

Idea of improvement
===================

I will change the assumptions to explicitly make the controller a part of dynamics, remove any other assumptions on controller properties and change the goal to instead reconstruct the dynamics of the whole system from the dynamics of the controller. This is similar to the approach from [General agents contain world models](https://arxiv.org/abs/2506.01622), but with two differences:

*   **Different setting:** Abstract deterministic dynamical systems instead of Controlled Markov processes
*   **Effectiveness of controller:** There is no explicit assumption that controller achieves anything. It can be taken into account and I will discuss the specifics after formal statement.

Modified IMP
------------

Abstract summary:

Imagine we are looking at the internals of a controller, we can take a state of the system and ask controller how it would act at this state. From this we want to deduce something about dynamics of the system. Information about dynamics of system that we can get from controller is intuitively the amount of what controller "knows" about dynamics.

Assume there exists:

*   **A dynamical system  **$(X,\alpha)$ **representing joint environment-controller states **$\alpha$ is "unknown" we are trying to deduce its properties, 
*   **A set of controller states **$W$,
*   **A mapping from each joint state to the corresponding controller state **$\pi: X \rightarrow W$ representing current controller state,
*   **A mapping from each joint state to the corresponding controller state **$f: X \rightarrow W$ representing the controller plan/policy/algorithm. 
*   **Controller action is a part of dynamics**, i.e. $f(x)=\pi(\alpha(x))$, which means that a controller plan becomes a controller state after evolution of the system.

Then: 

*   Information that can be deduced from the controller i.e. $(X,W,\pi,f)$ is captured by$\alpha(x)\in \pi^{-1}f(x)$
*   if $\pi^{-1}f$ is a well defined function, then controller allows for exact reconstruction of original dynamics, alternatively controller is exact model of original dynamics.

There is nothing to prove here, since all that is done is applying $\pi^{-1}$ to last assumption. Therefore similarly to Simplified IMP one can think of this as a definition. For dynamical system $(X,\alpha)$ and controller $(X,W,\pi,f)$ satisfying assumptions above, the Internal Model of controller is relation $\alpha(x)\in \pi^{-1}f(x)$. 

This can be interpreted as controller "modeling" itself as part of dynamics, which is still a trivially sounding statement. 

### **Effectiveness of controller**

To get interesting results one needs a controller that actually achieves something. In Abstract IMP this was keeping system in the set $X^+$ a subset of desirable states. In [General agents](https://arxiv.org/abs/2506.01622) it was being close to optimal in satisfying goals stated in [LTL](https://en.wikipedia.org/wiki/Linear_temporal_logic) (for example get from state $$$a$ to state $$$b$, stay there for 3 next steps and then move back to $$$a$). Verifying these conditions requires knowledge of the system dynamics. 

If we assume that there is some invariant set $X^+$ subset of $X$, then we can just repeat this approach for restriction of $(X,\alpha)$ to $(X^+,\alpha^+)$. All the assumptions still hold after restriction and $\pi^{-1}f$ might become a well defined function when restricted to smaller subsets (or at least be less ambiguous). 

Examples
--------

Let's analyse some simple examples to get a better intuition. 

### Clock example

Take $$system consisting of two clocks i.e. $X=\{0,1\}\times \{0,1\}$ with action$\alpha((a,b)):=(a+1,b+1) \mod 2$. 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d6cf5ded1a93f8588a22d84369818d9f0e561623251cdd4f.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/6c4d5586c53c07a81ebd6a040016664730db851e3db9fea9.png)

This system has two invariant disjoint subsystems,  

*   clock are aligned: $X_1:=\{(0,0),(1,1)\}$
*   clocks are not aligned: $X_2:=\{(0,1),(1,0)\}$

we can try to consider first clock a controller of whole system both using simplified IMP and modified IMP. Both will mostly tell us the same thing, but will get there in a different way. $\gamma=\pi_1$ will be projection to the first coordinate (first clock) 

Simplified IMP:

Assumptions are satisfied on either of the invariant subsets, lets say clocks are aligned. Using knowledge of $\alpha$ we define $\bar{\alpha}=\gamma\circ\alpha|_{X_1}$ which is exactly dynamics of the first clock. 

Modified IMP  

We assume $f(a,b)=a+1 \mod{2}$  and $f=\pi_1 \circ \alpha$ which is the same as $\bar{\alpha}$ would be for either $X_1$ or $X_2$ but we do it before choosing. Knowing the behaviour of the first clock restricts possible dynamics of whole system, to choosing between pairs of arrows with same letters in the picture

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/821b28c80cd5faf67dc5eb7bd14d3b3a028077b71154aab0.png)

With that knowledge, additional information about set $X_1$ (or $X_2$) being invariant (clocks being aligned or not) is enough to determine dynamics on that set.$$

### Perfect controller

Lets try a bit more complicated setting that might give some intuition for dealing with goals of controller. There will be two controllers in the same setting, that differ in effectiveness

Set:

*   $X=\mathbb{Z} \times \{-1,0,1\}$  where first coordinate is position (p) and second is controller state (c)
*   Dynamics are defined as: $\alpha((p,c)):=(p+c,f(p,c))$ 
*   Where $f$ is some algorithm of a controller, that uses current state and previous controller state to determine current controller state. (There will be two controllers for this setting)

This means that in each step the controller can change the plant position by at most 1, and use the current state of itself and the plant to plan its next move. Without the controller (meaning the controller is always $0$) nothing moves. Controller state function is $\pi_2:X\to C, \pi_2(p,c)=c$

Restating the assumption for this setting:

*   Set $X=\mathbb{Z} \times \{-1,0,1\}$
*   $\alpha: X \rightarrow X$  dynamics of the system with $\alpha((p,c))=(p+c,f(p,c))$ we will try to deduce it from controller
*   Controller states $W=\{-1,0,1\}$, 
*   $\pi: X \rightarrow \{-1,0,1\}$ representing current controller state,
*   $f: X \rightarrow \{-1,0,1\}$ representing the controller plan/policy/algorithm. 
*   Controller is a part of dynamics, i.e. $f(x)=\pi_2(\alpha(x))$, this is always satisfied if $\alpha$ is as above.

There will be two examples of controller for this system, they were designed to get the system close to state $p=0$. I mention this to hint at possible future directions of this approach since it should be visible in examples even if it is not worked out yet.

It is easy to design the perfect controller for getting $p$ to $0$$$ by taking $f(p,c)=-sgn(p+c)$

 (sign of next position) that is: 

$$
f(p,c) = \begin{cases} -1 & \text{if } p+c>0 \ 0 & \text{if } p+c = 0 \ 1 & \text{if } p+c<0 \end{cases}
$$

The following graph shows the behaviour of the system around 0. Arrows represent $\alpha$. Notice that columns contain points with the same $p$ and rows contain points with the same $c$.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/90c1d671e8f401bdc9f58c6dede74ab0f35b08935a036963.png)

The system is split into $3$ parts, by $\pi_2^{-1}f$ marked by colours. For red points controller plans $1$, for blue points $0$ and for green points $-1$. Now consider the case where we additionally know that blue set is invariant. Picture of actual dynamics restricted to this set looks like this:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/81f6ffe2834ea81bcecf294d5a9ecece689a7b25f8249718.png)

We can now fully deduce the dynamics of $\alpha$ on the invariant subset just from the controller behaviour. All arrows must go to $(0,0)$ because all points are from the blue set.

For slightly more interesting example lets assume that the invariant set is as on the picture. 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/505af3c20935606c83910e8f7194320d0be2b1faaf3d3369.png)

Then we can deduce all arrows leaving red points, but arrows leaving blue points could go to either $(0,0)$ or $(-1,0)$ shown by bold arrows, there are four possibilities of $\alpha$ that are compatible with this colouring. Following picture has different arrows (and therefore very different behaviour) but still would result in same colouring. Set considered in pictured above is still invariant here, bold arrows were chosen differently.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/96ce910dd9ad16fad8c9300e6294c818ed6190d692bf091c.png)

### Imperfect controller

For a more interesting example, let's modify the controller a bit. It will still be decent at achieving the goal of staying near 0, but it won't be perfect. Intuitively, this controller avoids changing its state unless its necessary.

$$
f_1(p,c) = \begin{cases} -1 & \text{if } p+c \geq 1 \ c & \text{if } 1 < p+c < -1 \ 1 & \text{if } p+c\leq-1 \end{cases}
$$

Again the picture shows the dynamics around $0$ and colouring that changed slightly.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/4a02197fb2c92afc029b520f4289bf7af99e47fa7b1e6949.png)

Notice that this controller cannot achieve $(0,0)$ if it doesn't start there. It does, however, do a decent job of getting close to 0, every point except $(0,0)$ is attracted to 4-state loop $L:=(0,1)\to (1,-1)\to (0,-1)\to (-1,1)$.

Once again, while colouring gives some information about the possible dynamics, more can be deduced if we get information about invariant subsets. Knowing that $(0,0)$ is invariant wouldn't allow for any more deductions. If a singleton is invariant we already know everything. 

Note, however, that if we knew that a set $X^+=L$ is invariant, then a priori there are $4^4=256$ possible dynamical systems on L. Taking into account information from the controller, we get a diagram of possibilities 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/ef77c741be6c4977505369e8aa8c6d77bea04ef07a76f6ab.png)

All $4^2=16$ possible dynamics can be obtained by choosing one of the two arrows coming out of each node. If we were to additionally know that $\alpha$ is a permutation on $L$, then there would be only 4 possible choices of compatible dynamics from $4!=24$ total. 

See below for few examples of such choices. 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/225d59c7312a3038d7c6b72cedaf912411a18efe4cc43216.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/a34381501c7cd5cb2a7df2e1768fb5151dbc9f43c00c6d19.png)

### Effectiveness

Both perfect and imperfect controllers are pretty good at directing all the points from the system to be near 0. The perfect controller is uniquely defined as getting all points to state $(0,0)$ as fast as possible. Intuitively this gives even more information about the system, for example: **"A positive controller state means that p will increase and negative means that it will decrease"**. Formalizing such intuitions to work in general setting and not for specific toy examples is a crucial continuation of this work.

Conclusion
==========

Improvements
------------

Alfred's listed [Seven ways to Improve the Internal Model Principle](https://www.lesswrong.com/posts/5NPLrAt9SKWiKPvnw/seven-ways-to-improve-the-internal-model-principle), all except 5 and 6 are at least partially addressed: 

1.  Relax the Assumption of Autonomy  
    *Removed it *
2.  Introduce a causal link between World and World Model  
    *World Model is now deduced from behaviour of controller in the World*
3.  Make it harder to Satisfy the Conditions  
    *Controller now needs to be a part of the the system. Conditions are still easy to satisfy but I hope that triviality issues are at least clearer with this approach*
4.  Show that the controller must model the whole world, not just the 'good' states  
    *Controller contains some information about whole world, restricting to 'good' states potentially gives more information*
5.  **Introduce a notion of Optimization**
6.  **Introduce Randomness**
7.  Explicitly include Controller 'Actions'  
    *There are no explicit Controller Actions but Controller is embedded in a system and if it doesn't affect the rest of the world (it always has the same state), then it doesn't convey new information.*

Continuation
------------

This is still a very preliminary work, 

*   **Optimization/Effectiveness:** It was somewhat present in the examples I gave, and it was a main motivation for what was presented here. Ideal theorem would have be something similar to [General Agents](https://arxiv.org/abs/2506.01622) but  
*   Different categories of dynamical systems:
    *   **Random** not only Controlled Markov process
    *   **Continuous time**
*   **Additional structure of the system** This would allow for more specific results, especially with Optimization/defining goals in mind 
    *   Metric or Topology (appeared informally with being near 0)
    *   Measure 
    *   Linearity
    *   etc. 
*   **Imperfect Observations:** Currently both points in $X$ and controller states are observed perfectly. It can be generalized to seeing space and controller states with only limited precision or some random noise. A trivial example is when only controller states of points in $X$ are known, we can get a factor of $(X,\alpha)$, corresponding to controller states.