# Minimal Maps, Semi-Decisions, and Neural Representations

*Epistemological Status: I attempt to distill the [minimal map](https://www.lesswrong.com/posts/Lz2nCYnBeaZyS68Xb/probability-as-minimal-map) argument and provide a natural way to chain a sequence of queries together. I argue that this definition, together with reasonable symmetry constraints on the minimal map, provides an intuitive explanation for the appearance of neural networks.* 

## Minimal Maps

Say we sample an observation $(o,h) \in O \times H$ from a state-observation pair and want to decide based only on $o$ whether $h$ has a property $U$. We can define the function $\chi_U: H \times O \to \lbrace 0, 1 \rbrace$ to indicate whether or not an observation has the property we have in mind. Concretely,
$$
\chi_U(o) = \chi_U(h_o | o) = \begin{cases} 1 \quad h_o \text{ has property } U \\ 0 \quad \text{otherwise} \end{cases}
$$
I've chosen to suppress the reference to the hidden state since it's not an observable. This could alternatively be viewed as a semi-decision that either terminates in finite time with a positive answer or runs forever. The catch is that each observation is partial and dependent on an unobserved hidden state. Here's one natural question: what information summarizes the outcome of the semi-decision? 

**Minimal Map Lemma:** Up to isomorphism, any information in observing the outcome of $\chi_U(o)$ for a given $o \in O$ is equivalent to $P(\chi_U(o) | o)$. Moreover, this map is minimal in the sense that any other representation $g_U(o)$ is either equivalent in information content or lossy.

**Proof:** First, fix some other map $g_U$ and consider the sequence $o \to \chi_{U}(o) \to g_U(o)$. By the [data processing inequality](https://en.wikipedia.org/wiki/Data_processing_inequality) the total mutual information satisfies $I(o; \chi_U(o)) \ge I(o;g_U(o))$. Second, by definition, the event $P(\chi_U(o) | o) = p$ is equivalent to any observation $\hat o \in P^{-1}(p)$ because the outcome of the semi-decision is unaffected. Because we also have $o \in P^{-1}(p)$ we can just write,
$$
P(\chi_U(o) | P(\chi_U(o) | o)) = P(\chi_U(o) | \hat o) = P(\chi_U(o) | o)
$$
Finally, we have for the entropy,
$$
H(\chi_{U}(o) | P(\chi_{U}(o) | o)) = - \sum_{\chi_{U}(o) \in \lbrace 0, 1 \rbrace} P(\chi_{U}(o) | P(\chi_{U}(o) | o)) \cdot \log(P(\chi_{U}(o) | P(\chi_{U}(o) | o))) \\
= - \sum_{\chi_{U}(o) \in \lbrace 0, 1 \rbrace} P(\chi_{U}(o) | o) \cdot \log(P(\chi_{U}(o) | o)) = H(\chi_{U}(o) | o)
$$
Thus, with respect to the semi-decision, the information contained in the probability of a decision is equivalent to that contained in the observation. $\square$

## Semi-Decision Process

At this point, we conclude that it's natural to represent the semi-decision on the observation as a probability. Now we can consider something more familiar. Consider a set of properties $\mathcal{U} = \lbrace U_i \rbrace_{i \in I}$ that satisfy the following for $o \in O$,
$$
\exists i \in I : \chi_{U_i}(o) = 1 \quad \text{(Decidable)} \\
\chi_{U_i}(o) = 1 \rightarrow \chi_{U_j} = 0, \ \forall j \in I \setminus \lbrace{i \rbrace} \quad \text{(Mutually Exclusive)}
$$
If the previous properties are satisfied we'll say that the objects $o \in O$ can be classified by $\mathcal{U}$. Alternatively, we can say $\mathcal{U}$ is a classifier for the objects $o \in O$. It's worth noting that we can always add a 'non of the above' property in the case that our $\mathcal{U}$ is only mutually exclusive which results in a proper classifier. 

**Fact:** Every property $U_i$ can be associated to a classifier $\mathcal{U} = \lbrace U_i, \lnot U_i \rbrace$. 

This essentially is the motivation for the choice of definition.

It’d be nice to have a natural way to compose classifiers together on observations. Why? A central advantage is that if we do this correctly then we would be able to build up complicated classifiers from simpler ones.

With this goal in mind, we’ll first define,
$$
\text{return} :: o \to M o \\
\text{return}(o) = (\phi \circ P)(\chi_{U_i}(o) | o) = \vec p
$$
The $\phi$ is an isomorphism that comes from the 'up to isomorphism' clause of the minimal map lemma. This is actually quite useful as we'll see in a bit. For the moment, our definition of return can be understood as a projection of the object onto a basic collection of minimal maps. After this, we need one more definition,
$$
\text{bind} :: M o \to (o \to M u) \to M u \\
\text{bind}(\vec p,(\phi \circ P)(\chi_{U_j}(\cdot) | \cdot)) = (\phi \circ P) (\chi_{U_j}(\vec p) | \vec p)
$$
This projects the resulting minimal maps onto a new collection of minimal maps. What we're trying to do is allow for queries about query outcomes. This allows us to chain together a sequence of queries. Specifically, there will be some $L$ number of sequential binds with $n_{j}$ queries at each bind for $j \in \lbrace 1, \ldots, n \rbrace$. The sequence of queries at each bind defines a semi-decision process over the observations.

This definition allows us to treat our queries as a [monad](https://en.wikipedia.org/wiki/Monad_(functional_programming)). While the only immediate use is for function composition, which could've been defined without the formality, this approach makes it easier to think about side-effects and related constructs such as the comonad (categorical dual of the monad) based on the exact outcomes of the queries.

This representation might also be useful to anticipate a natural class of minimal maps to consider. This is because our bind maintains the conditional notation and almost appears as $(\chi_{U_j}(\vec p) | \vec p) \sim \langle \vec u_j | \vec p \rangle$ and we could view each query as a projection against the objects.

## Neural Representations

We'll say the observations and queries have (vector) representations that are rotation invariant if we have,
$$
(\phi \circ P)(\chi_{U_i}(\vec p) | \vec p) = \sigma_{\vec u_i}(\vec p)\quad \text{s.t.} \\
R^T R = I \Rightarrow \sigma_{\vec u_i}(\vec p) = \sigma_{R \cdot \vec u_i}(R \cdot \vec p)
$$
The property is telling us that if we rotate our observations *and* our query then the resulting determination should remain the same. We'll say that the observation and queries are dual if we have,
$$
\sigma_{\vec u_i}(\vec p) = \sigma_{\vec p}(\vec u_i)
$$
In other words, if the process is dual then mapping the query over the observations is equivalent to mapping the data over the query. With these two notions, we can state the main insight,

**Proposition:**

If our semi-decision process on the observations is rotation invariant and dual then we have,
$$
\sigma_{\vec u_i}(\vec p) = \sigma(\vec u_i \cdot \vec p, \vec u_i \cdot \vec u_i, \vec p \cdot \vec p) = \sigma(\vec u_i \cdot \vec p)
$$
Moreover, $\sigma$ is a ridge-activation and the outcome of $L$-sequential binds is equivalent to a neural network with $L$-layers.

**Proof:** The first equality follows from the fact the inner-products are the only rotationally invariant quantities that can be made out of the query and the observation. The second inequality follows from symmetry. Only the inner-product between the query and observation is symmetric under swapping.

For the next part we'll relabel $\vec u_i \to \vec w_i$. Say the $j$-th sequential bind has $n_j$ queries. Then we have,
$$
\vec p_{j+1} = \sigma(\vec w_{i_j} \cdot \vec p_{j}), \quad i_j \in \lbrace 1, \ldots, n_j \rbrace \\
\Rightarrow \vec p_{L} = \sigma(\vec w_{i_L} \cdot \sigma(\vec w_{i_{L-1}} \ldots \sigma(w_{i_1} \cdot \vec p_{0}))) \\
\Rightarrow \vec p_L = \sigma(W_L \cdot \sigma(W_{L-1} \ldots \sigma(W_1 \cdot \vec p_{0})))
$$
In the last equation, we've replaced the index to the queries with a matrix for compactness. The last expression is the definition of a multi-layer neural network. $\square$

As an immediate corollary, we see that this covers both [radial and ridge activation functions](https://en.wikipedia.org/wiki/Activation_function). Our activation is fixed as a ridge activations since duality holds, but if we were to relax the condition to just rotation invariance then we'd have the possibility of radial activations. Moreover, if $P$ is monotone in its argument then because $\phi$ is an isomorphism the activation will also be monotone. For example, if $P(v) = \frac{1}{1 + e^{-x}}$ is a sigmoid and $\phi(v) = - \log(v)$ then $\phi \circ P$ is [Softplus](http://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf).

Even if our data *doesn't* satisfy our representation property, every (reasonable) classifier can be approximated as such by the [universal approximation theorem](ttps://en.wikipedia.org/wiki/Universal_approximation_theorem). This means that if we only care about producing approximations of minimal maps we can view the assumptions we made as a [Wittgenstein ladder](https://en.wikipedia.org/wiki/Wittgenstein%27s_ladder). Overall, what we showed is that the minimal map theorem, the monadic representation of a semi-decision process, and basic symmetry assumptions are enough to derive the general form of a neural network.