# Infra-Bayesian physicalism: a formal theory of naturalized induction

*This is joint work by Vanessa Kosoy and Alexander "Diffractor" Appel.*
*For the proofs, see [1](https://www.alignmentforum.org/posts/cj3PRu8QoFm4BA8oc/infra-bayesian-physicalism-proofs-part-i) and [2](https://www.alignmentforum.org/posts/CPr8bRGekTyvh7nGC/infra-bayesian-physicalism-proofs-part-ii).*

**TLDR:** We present a new formal decision theory that realizes
[naturalized induction](https://www.alignmentforum.org/tag/naturalized-induction).
Our agents reason in terms of infra-Bayesian hypotheses, the domain
of which is the cartesian product of computations and physical states,
where the ontology of "physical states" may vary from one hypothesis
to another. The key mathematical building block is the "bridge transform",
which, given such a hypothesis, extends its domain to "physically
manifest facts about computations". Roughly speaking, the bridge
transforms determines which computations are executed by the physical
universe. In particular, this allows "locating the agent in the
universe" by determining on which inputs its own source is executed.

# 0. Background

The "standard model" of ideal agency is Bayesian reinforcement
learning, and more specifically, [AIXI](https://arxiv.org/abs/cs/0004001).
We [challenged](https://www.alignmentforum.org/s/CmrW8fCmSLK7E25sa)
this model before due to its problems with non-realizability, suggesting
infra-Bayesianism as an alternative. Both formalisms assume the "cartesian
cybernetic framework", in which (i) the universe is crisply divided
into "agent" and "environment" and (ii) the two parts interact
solely via the agent producing actions which influence the environment
and the environment producing observations for the agent. This is
already somewhat objectionable on the grounds that this division is
not a clearly well-defined property of the physical universe. Moreover,
once we examine the structure of the hypothesis such an agent is expected
to learn (at least naively), we run into some concrete problems.

The modern understanding of the universe is that no observer plays
a privileged role[^1]. Therefore, the laws of
physics are insufficient to provide a cartesian description of the universe,
and must, to this end, be supplemented with "[bridge rules](https://www.lesswrong.com/posts/ethRJh2E7mSSjzCay/building-phenomenological-bridges)"
that specify the agent's location inside the universe. That is, these
bridge rules need to translate the fundamental degrees of freedom
of a physical theory (e.g. quantum wavefunction) to the agent's observations
(e.g. values of pixels on a camera), and translate the agent's actions
(e.g. signal to robot manipulators) in the other direction[^2].
The cost of this is considerable growth in the description complexity
of the hypothesis.

A possible retort is something like "how many bits can it really take to pick out the computer within the universe state and describe how to translate universe state to observations? Sure, it might be a considerable chunk of data. But, since Solomonoff induction only needs to make a kilobyte worth of predictive mistakes to learn to predict the inputs as well as any thousand-bit predictor, it should learn this sort of stuff pretty fast." This objection is addressed more later on in this post. A first counter-objection is that, for practical algorithms, this can be a bigger obstacle, especially when sample complexity scales superlinearly with description complexity. For example, the [Russo-Van Roy](https://arxiv.org/abs/1403.5341) regret bound for Thompson sampling in multi-armed bandits has the time horizon necessary to get a particular regret bound scale as the square of entropy, and the computational complexity cost can also go way up as the description complexity rises, because you need to evaluate more hypotheses more times to test them. For reinforcement learning it's even worse, as in the case of a game where an agent must enter an n-digit password and if it gets it right, it gets reward, and if it gets it wrong, it does not get reward and can try again. The learning time for this game scales as $\Omega(2^n)$, far superlinearly with description complexity.

Another assumption of AIXI is the simplicity prior, and we expect
some form of this assumption to persist in computable and infra-Bayesian
analogues. This reflects the intuitive idea that we expect the world
to follow simple laws (or at least contain simple patterns). However,
from the cartesian perspective, the "world" (i.e. the environment)
is, prima facie, not simple at all (because of bridge rules)! Admittedly, the increase in complexity from the bridge rule is low compared to the cost of specifying the universe state, but once the agent learns the transition rules and bridge rule for the universe it's in, learning the state of the universe in addition doesn't seem to yield any particular unforeseen metaphysical difficulties. Further, the description complexity cost of the bridge rule seems likely to be above the description complexity of the laws of physics.

Hence,
there is some disconnect between the motivation for using a simplicity
prior and its implementation in a cartesian framework. 

Moreover, if the true hypothesis is highly complex, it implies that
the sample complexity of learning it is very high. And, as previously mentioned, the sample complexity issues are worse in practice than Solomonoff suggests. This should make
us suspect that such a learning process is not properly exploiting
Occam's razor. Intuitively, such an agent a-priori considers it equally plausible
to discover itself to be a robot or to discover itself to be a random
clump of dust in outer space, since it's about as hard to specify a bridge rule interface between the computer and observations as it is to specify a bridge rule interface between the dust clump and observations and needs a lot of data to resolve all
those possibilities for how its observations connect to a world. Also, though Solomonoff is extremely effective at slicing through the vast field of junk hypotheses that do not describe the thing being predicted, once it's whittled things down to a small core of hypotheses that do predict things fairly accurately, the data to further distinguish between them may be fairly slow in coming. If there's a simple predictor of events occurring in the world but it's running malign computation, then you don't have the luxury of 500 bits of complexity wiggle room (to quickly knock out this hypothesis), because that's a factor of $2^{500}$ probability difference. Doing worst-case hypothesis testing as in KWIK learning would require a very aggressive threshold indeed, and mispredictions can be rare but important.

Furthermore, some events are simple to describe from the subjective
(cartesian) point of view, but complex to describe from an objective
(physical) point of view. (For example, all the pixels of the camera
becoming black.) Modifying a hypothesis by positing exceptional behavior
following a simple event only increases its complexity by the difficulty of specifying the event and what occurs afterwards, which could be quite low. Hence,
AIXI-like agents would have high uncertainty about the consequences
of observationally simple events. On the other hand, from an objective perspective such
uncertainty seems irrational. (Throwing a towel on the camera should
not break physics.) In other words, cartesian reasoning is biased
to privilege the observer.

Yet another failure of cartesian agents is the inability to reason
about origin theories. When we learn that a particular theory explains
our own existence (e.g. evolution), this serves as a mark in favor
of the theory. We can then exploit this theory to make useful predictions
or plans (e.g. anticipate that using lots of antibiotics will cause
bacteria to develop resistance). However, for a cartesian agent the
question of origins is meaningless. Such an agent perceives its own
existence as axiomatic, hence there is nothing to explain.

Finally, cartesian agents are especially vulnerable to [acausal
attacks](https://ordinaryideas.wordpress.com/2016/11/30/what-does-the-universal-prior-actually-look-like/). Suppose we deploy a superintelligent Cartesian AI called Kappa. And, imagine a superintelligent agent Mu that inhabits some purely hypothetical universe. If Mu is motivated to affect our own (real) universe, it can run simulations of Kappa's environment. Kappa, who doesn't know a priori whether it exists in our universe or in Mu's universe, will have to seriously consider the hypothesis it is inside such a simulation. And, Mu will deploy the simulation in such manner as to make the simulation hypothesis much simpler, *thanks to simpler bridge rules*. This will cause Kappa to become overwhelmingly confident that it *is* in
 a simulation. If this is achieved, Mu can cause the simulation to diverge from our reality in a strategically chosen point such that Kappa is induced to take an irreversible action in Mu's favor (effectively a treacherous turn). Of course, this requires Kappa to predict Mu's motivations in some detail. This is possible if Kappa develops a good enough understanding of [metacosmology](https://www.alignmentforum.org/posts/dPmmuaz9szk26BkmD/shortform?commentId=N8oamtFAhWKEbyCBq). 

An upcoming post by Diffractor will discuss acausal attacks in more detail.

In the following sections, we will develop a "physicalist" formalism
that entirely replaces the cartesian framework, curing the
abovementioned ills, though we have not yet attained the stage of proving improved regret bounds with it, just getting the basic mathematical properties of it nailed down. As an additional benefit, it allows naturally
incorporating utility functions that depend on unobservables, thereby avoiding the problem of ["ontological crises"](https://arxiv.org/abs/1105.3821). At the
same time, it seems to impose some odd constraints on the utility
function. We discuss the possible interpretations of this.

# 1. Formalism

$$\newcommand{\E}{\mathrm{E}}$$

$$\newcommand{\Pr}[1]{\mathrm{pr}_{#1}}$$

$$\newcommand{\N}{\mathbb{N}}$$

$$\newcommand{\R}{\mathbb{R}}$$

$$\newcommand{\Com}{\mathbb{C}}$$

$$\newcommand{\Con}[1]{\Delta^{c}#1}$$

$$\newcommand{\HUC}[1]{\Box^{c}#1}$$

## Notation

It will be more convenient to use ultradistributions rather than infradistributions.
This is a purely notational choice: the decision theory is unaffected,
since we are going to apply these ultradistributions to loss functions
rather than utility functions. As support for this claim, Diffractor originally wrote down most of the proofs in infradistribution form, and then changing their form for this post was rather straightforward to do. In addition, for the sake of simplicity,
we will stick to finite sets: more general spaces will be treated
in a future article. So far, we're up to countable products of finite sets.

We denote $\mathbb{R}_{+}:=[0,\infty)$. Given a finite set $X$, a
*contribution* on $X$ is $\theta:X\rightarrow\mathbb{R}_{+}$
s.t. $\sum_{x}\theta(x)\leq1$ (it's best to regard it as a *measure*
on $X$). The space of contributions is denoted $\Con X$. Given $f:X\rightarrow\mathbb{R}$
and $\theta\in\Con X$, we denote $\theta(f):=\sum_{x}\theta(x)f(x)$.
There is a natural partial order on contributions: $\theta_{1}\leq\theta_{2}$
when $\forall x\in X:\theta_{1}(x)\leq\theta_{2}(x)$. Naturally,
any distribution is in particular a contribution, so $\Delta X\subseteq\Con X$.
A *homogenous ultracontribution* (HUC) on $X$ is non-empty closed
convex $\Theta\subseteq\Con X$ which is downward closed w.r.t. the
partial order on $\Con X$. The space of HUCs on $X$ is denoted $\HUC X$.
A *homogenous ultradistribution* (HUD) on $X$ is a HUC $\Theta$
s.t. $\Theta\cap\Delta X\ne\varnothing$. The space of HUDs on $X$
is denoted $\Box X$. Given $f:X\rightarrow\mathbb{R}$ and $\Theta\in\Box^{c}X$,
we denote $\Theta(f):=\max_{\theta\in\Theta}\theta(f)$. 

Given $s:X\rightarrow Y$, $s_*:\HUC X\rightarrow\HUC Y$ is the
pushforward by $s$:

$$
s_*\Theta:=\{s_*\theta\mid\theta\in\Theta\}
$$

$$
(s_*\theta)(y):=\sum_{x\in s^{-1}(y)}\theta(x)
$$

Given $\Xi:X\rightarrow\HUC Y$, $\Xi_*:\HUC X\rightarrow\HUC Y$
is the pushforward by $\Xi$:

$$
\Xi_*\Theta:=\{\kappa_*\theta\mid\theta\in\Theta,\:\kappa:X\rightarrow\Con Y,\:\forall x\in X:\kappa(x)\in\Xi(x)\}
$$

$$
(\kappa_*\theta)(y):=\sum_{x\in X}\theta(x)\kappa(x;y)
$$

$\Pr X:X\times Y\rightarrow X$ is the projection mapping and $\Pr{-Y}:=\Pr X$.
We slightly abuse notation by omitting the asterisk in pushforwards
by these. 

Given $\Theta\in\HUC X$ and $\Xi:X\rightarrow\HUC Y$, $\Theta\ltimes\Xi\in\HUC{(X\times Y)}$
is the semidirect product:

$$
\Theta\ltimes\Xi:=\{\kappa\ltimes\theta\mid\theta\in\Theta,\:\kappa:X\rightarrow\Con Y,\:\forall x\in X:\kappa(x)\in\Xi(x)\}
$$

$$
(\kappa\ltimes\theta)(x,y):=\theta(x)\kappa(x;y)
$$

We will also use the notation $\Xi\rtimes\Theta\in\HUC{(Y\times X)}$
for the same HUC with $X$ and $Y$ flipped. And, for $\Lambda\in\HUC Y$, $\Theta\ltimes\Lambda\in\HUC{(X\times}Y)$
is the semidirect product of $\Theta$ with the *constant* ultrakernel
whose value is $\Lambda$[^3]. 

For more discussion of HUDs, see [previous article](https://www.alignmentforum.org/s/CmrW8fCmSLK7E25sa/p/idP5E5XhJGh9T5Yq9),
where we used the equivalent concept of "cohomogenous infradistribution".


## Notation Reference
If you got lost somewhere and wanted to scroll back to see some definition, or see how the dual form of this with infradistributions works, that's what this section is for.


$\theta$ is a contribution, a measure with 1 or less measure in total. The dual concept is an a-measure $(\lambda\mu,b)$ with $\lambda+b=1$.


$\Theta$ is a HUC (homogenous ultracontribution) or HUD (homogenous ultradistribution), a closed convex downwards closed set of contributions. The dual concepts are cohomogenous inframeasure and cohomogenous infradistribution, respectively.


$\Con{X},\HUC{X},\Box X$ are the spaces of contributions, homogenous ultracontributions, and homogenous ultradistributions respectively.


$\theta(f),\Theta(f)$ are the expectations of functions $f:X\to[0,1]$, defined in the usual way. For $\theta$, it's just the expectation of a function w.r.t. a measure, and for $\Theta$, it's $\Theta(f):=\max_{\theta\in\Theta}\theta(f)$, to perfectly parallel a-measures evaluating functions by just taking expectation, and inframeasures evaluating functions via $\min_{(m,b)\in\Psi}m(f)+b$.


$\le$ is the ordering on contributions/HUC's/HUD's, which is the function ordering, where $\theta\le\theta'$ iff for all $f:X\to[0,1]$, $\theta(f)\le\theta'(f)$, and similar for the HUC's. Inframeasures are equipped with the opposite ordering.


$s_*\Theta$ is the pushforward along the function $s:X\to Y$. This is a standard probability theory concept which generalizes to all the infra and ultra stuff.


$\Xi_*\Theta$ is the pushforward of $\Theta$ along the ultrakernel $\Xi:X\to\HUC{Y}$. This is just the generalization to infra and ultra stuff of the ability to push a probability distribution on $X$ through a probabilistic function $X\to\Delta Y$ to get a probability distribution on $Y$.


$\Theta\ltimes\Xi$ is the semidirect product of $\Theta$ and $\Xi$, an element of $\HUC{(X\times Y)}$. This is the generalization of the ability to take a probability distribution on $X$, and a probabilistic kernel $X\to\Delta Y$, and get a joint distribution on $X\times Y$.


$\mathcal{A},\mathcal{O}$ are the set of actions and observations.


$N$ is the time horizon.


$\mathcal{R}$ is the space of programs, $\Sigma$ is the space of outputs.


$\Gamma=\Sigma^{\mathcal{R}}$ is the space of functions from programs to the results they output.  An element of $\Gamma$ can be thought of as the state of the "computational universe", for it specifies what all the programs do.


$\mathcal{H}$ is the space of histories, action-observation sequences that can end at any point, ending with an observation.


$\mathcal{D}$ is the space of destinies, action-observation sequences that are as long as possible, going up to the time horizon.


$\mathcal{C}$ is a relation on $\Gamma\times\mathcal{D}$ that says whether a computational universe is consistent with a given destiny. A very important note is that this is *not* the same thing as "the policy is consistent with the destiny" (the policy's actions are the same as what the destiny advises). This is saying something more like "if the destiny has an observation that the computer spat out result 1 when run on computation A, then that is only consistent with mathematical universes which have computation A outputting result 1". Except we don't want to commit to the exact implementation details of it, so we're leaving it undefined besides just "it's a relation"


$\Phi$ is the space of "physics outcomes", it can freely vary depending on the hypothesis. It's not a particular fixed space.


$\Theta$ is the variable typically used for physicalist hypotheses, elements of $\Box(\Gamma\times\Phi)$. Your uncertainty over the joint distribution over the computational universe and the physical universe.


$G$ is the code of the program-which-is-the-agent. So, $G(h)$ would be the computation that runs what the agent does on history $h$, and returns its resulting action.


$\text{el}^{\Gamma}$ is the subset of the space $\Gamma\times 2^{\Gamma}$ consisting of $(y,\alpha)$ pairs s.t. $y\in\alpha$. The $y$ can be thought of as the mathematical universe, and $\alpha$ can be thought of as the set of mathematical universes that are observationally indistinguishable from it.


$\chi_{A}$ is the indicator function that's 1 on the set $A$, and 0 everywhere else. It can be multiplied by a measure, to get the restriction of a measure to a particular set.


$\text{Br}(\Theta)$ is the bridge transform of $\Theta$, defined in definition 1.1, an ultracontribution over the space $\text{el}^{\Gamma}\times\Phi$.


$\mathcal{H}_{y\alpha}$ is the set of instantiated histories, relative to mathematical universe $y$, and set of observationally indistinguishable universes $\alpha$. It's the set of histories $h$ where all the "math universes" in $\alpha$ agree on how the agent's source code reacts to all the prefixes of $h$, and where the history $h$ can be extended to some destiny that's consistent with math universe $y$. Ie, for a history to be in here, all the prefixes have to be instantiated, and it must be consistent with the selected math universe.


## Setting

$$\newcommand{\A}{\mathcal{A}}$$

$$\newcommand{\O}{\mathcal{O}}$$

$$\newcommand{\Pro}{\mathcal{R}}$$

$$\newcommand{\C}{\mathcal{C}}$$

$$\newcommand{\H}{\mathcal{H}}$$

$$\newcommand{\D}{\mathcal{D}}$$

As in the cartesian framework, we fix a finite set $\A$ of actions
and a finite set $\O$ of observations. We assume everything happens
within a fixed finite[^4] time horizon $N\in\mathbb{N}.$
We assume that our agent has access to a *computer*[^5]
on which it can execute some finite[^4] set of
programs $\Pro$ with outputs in a finite alphabet $\Sigma$. Let
$\Gamma:=\Sigma^{\Pro}$ be the set of "possible computational universes"[^6].
We denote $\H:=(\A\times\O)^{<N}$ (the set of histories) and $\D:=(\A\times\O)^{N-1}\times\A$
(the set of "destinies").

To abstract over the details of how the computer is operated, we assume
a relation $\C\subseteq\D\times\Gamma$ whose semantics is, $d\C y$
(our notation for $(d,y)\in\C$) if and only if destiny $d$ is a
priori consistent with computational universe $y$. For example, suppose
some $a\in\A$ implies a command to execute program $r\in\Pro,$and
if $o\in\O$ follows $a$, it implies observing the computer return
output $i\in\Sigma$ for $r$. Then, if $d$ contains the substring
$ao$ and $d\C y$, it must be the case that $y(r)=i$.

A *physicalist hypothesis* is a pair $(\Phi,\Theta$), where $\Phi$
is a finite[^4] set representing the physical states
of the universe and $\Theta\in\Box(\Gamma\times\Phi)$ represents
a joint belief about computations and physics. By slight abuse of
notation we will refer to such $\Theta$ as a physicalist hypothesis,
understanding $\Phi$ to be implicitly specified. Our agent will have
a prior over such hypotheses, ranging over different $\Phi$.

Two questions stand out to us at this point. The first is, what is
the domain over which our loss function should be defined? The second
is, how do we define the counterfactuals corresponding to different
policies $\pi:\H\rightarrow\A$? The answers to both questions turn
out to require the same mathematical building block.

For the first question, we might be tempted to identify $\Phi$ as
our domain. However, prima facie this doesn't make sense, since $\Phi$
is hypothesis-dependent. This is the [ontological crisis](https://www.lesswrong.com/tag/ontological-crisis)
problem: we expect the agent's values to be defined within a certain
ontology which is not the best ontology for formulating the laws of
the physical universe. For example, a paperclip maximizer might benefit
from modeling the universe in terms of quantum fields rather than
paperclips. In principle, we can circumvent this problem by requiring
our $\Phi$ to be equipped with a mapping $\nu:\Phi\rightarrow\Phi_{0},$where
$\Phi_{0}$ is the "axiological" ontology. However, this $\nu$
is essentially a bridge rule, carrying with it all the problems of
bridge rules: acausal attack is performed by the adversarial hypothesis imprinting the "axiological rendering" of the target universe on the microscopic degrees of freedom of the source universe in order to have a low-complexity $\Phi\to\Phi_0$ function; the analogue of the towel-on-camera issue is that, once you've already coded up your uncertainty over math and physics, along with how to translate from physics to the ontology of value, it doesn't take too much extra complexity to tie "axiologically-simple" results (the analogue of low-complexity observations) to physics-simple results (the analogue of a low-complexity change in what happens), like "if all the paperclips are red, the fine-structure constant doubles in value".

Instead, we will take a *computationalist* stance: value is a
not property of physical states or processes, but of the *computations*
realized by physical processes. For example, if our agent is "selfish"
in the sense that, rewards/losses are associated purely with subjective
histories, the relevant computation is the *agent's own source code*.
Notice that, for the program-which-is-the-agent $G$, histories are
input. Hence, given loss function $l:\H\rightarrow\mathbb{R}$ we
can associate the loss $l(h)$ with the computation $G(h)$. Admittedly, there is an implicit assumption that the agent has access to its own source code, but modal decision theory made the same assumption. For another
example, if our agent is a diamond maximizer, then the relevant computations
are simulations of the physics used to define "diamonds". A more concrete analogue of this is worked out in detail in section 3, regarding Conway's Game of Life.


For the second question, we might be tempted to follow [updateless
decision theory](https://www.lesswrong.com/tag/updateless-decision-theory):
counterfactuals correspond to conditioning on $G=\pi$. Remember, G is the code of the agent. However, this
is not "fair" since it requires the agent to be "responsible"
for copies of itself instantiated with fake memories. Such a setting
admits no learning-theoretic guarantees, since learning requires trusting
your own memory. (Moreover, the agent also has to be able to trust
the computer.) Therefore our counterfactuals should only impose $G(h)=\pi(h)$
when $h$ is a "real memory", which we against interpret through
computationalism: $h$ is real if and only if, $G(h')$ is physically
realized for any prefix $h'$ of $h$. 

$$\newcommand{\P}[1]{\mathrm{el}^{#1}}$$

$$\newcommand{\Sup}{\mathrm{supp\,}}$$

Both of our answers requires a formalization of the notion "assuming
hypothesis $\Theta$, this computation is physically realized".
More precisely, we should allow for computations to be realized with
certain *probabilities*, and more generally allow for ultradistributions
over which computations are realized. We will now accomplish this
formalization.

## Bridge transform

Given any set $A$, we denote $\P A=\{(a,B)\in A\times2^{A}\mid a\in B\}.$
$\Sup$ stands for "support" and $\chi_{A}$ is the characteristic
function of $A$.

$$\newcommand{\B}{\mathrm{Br}}$$

**Definition 1.1:** *Let $\Gamma,\Phi$ be finite sets and
$\Theta\in\HUC{(\Gamma\times\Phi)}$. The *bridge transform* of
$\Theta$ is $\B_{\Gamma}(\Theta)\in\HUC{(\Gamma\times2^{\Gamma}\times\Phi)}$
s.t. $\theta\in\B_{\Gamma}(\Theta)$ if and only if*

* $\Sup\theta\subseteq\P{\Gamma}\times\Phi$

* *for any $s:\Gamma\rightarrow\Gamma$, $\Pr{\Gamma\times\Phi}\chi_{\P{\Gamma} \times \Phi}(s \times \mathrm{id}_{2^\Gamma \times \Phi})_*\theta\in\Theta$.*


*We will use the notation $\B(\Theta)$ when $\Gamma$ is obvious
from the context.*

Notice that we are multiplying by $\chi_{\P{\Gamma} \times \Phi}$, not pushforwarding.

The $2^{\Gamma}$ variable of the bridge transform denotes the "facts
about computations realized by physics". In particular, if this
$\alpha\in2^{\Gamma}$ takes the form $\{y\in\Gamma\mid\forall r\in\Pro_{0}:y(r)=y_{0}(r)\}$
for some $\Pro_{0}\subseteq\Pro$ and $y_{0}\in\Sigma^{\Pro_{0}}$,
then we may say that the computations $\Pro_{0}$ are "realized"
and the computations $\Pro\setminus\Pro_{0}$ are "not realized".
More generally, talking only about which computations are realized
is imprecise since $\alpha$ might involve "partial realization"
and/or "entanglement" between computations (i.e. not be of the
form above).

Intuitively, this definition expresses that the "computational universe"
can be freely modified as long as the "facts known by physics"
are preserved. However, that isn't what originally motivated the definition. The bulk of its justification comes from its
pleasing mathematical properties, discussed in the next section.

A physicalist agent should be equipped with a prior over physicalist
hypotheses. For simplicity, suppose it's a discrete Bayesian prior
(it is straightforward to generalize beyond this): hypothesis $\Theta_{i}$
is assigned probability $\zeta(i)$ and $\sum_{i}\zeta(i)=1$. Then,
we can consider the *total* bridge transform of the prior.  It can't be given by mixing the hypotheses together and applying the bridge transform, because every hypothesis has its own choice of $\Phi$, its own ontology, so you can't mix them before applying the bridge transform. You have to apply the bridge transform to each component first, forget about the choice of $\Phi$ via projecting it out, and then mix them afterwards. This receives further support from Proposition 2.13 which takes an alternate possible way of defining the bridge transform for mixtures (extend all the hypotheses from $\Gamma\times\Phi_i$ to $\Gamma\times\bigsqcup_{i}\Phi_i$ in the obvious way so you can mix them first) and shows that it produces the same result.

**Definition 1.2:** $$\B(\zeta\ltimes\Theta):=\sum_{i}\zeta(i)\Pr{\P{\Gamma}}\B(\Theta_{i})\in\Box\P{\Gamma}$$

## Evaluating policies

Given $A\subseteq X$, $\top_{A}\in\Box X$ is defined as $\{\theta\in\Delta^{c}X\mid\Sup\theta\subseteq A\}$. I.e., it's total uncertainty over which point in $A$ will be selected.

$$\newcommand{\Dec}{\mathrm{dec}}$$

$$\newcommand{\Co}[2]{\mathrm{C_{#2}^{#1}}}$$

We need to assume that $\Pro$ contains programs representing the
agent itself. That is, there is some $M\in\mathbb{N}$, $\mathrm{\Dec:\Sigma^{M}\rightarrow\A}$
and for each $h\in\H$, $\ulcorner G(h)\urcorner\in\Pro^{M}$. Pretty much, if you have large numbers of actions available, and a limited number of symbols in your language, actions (and the outputs of other programs with a rich space of outputs) can be represented be m-tuples of programs, like "what's the first bit of this action choice" and "what's the second bit of this action choice" and so on. So, $M$ is just how many bits you need, $\Dec$ is the mapping from program outputs to the actual action, and $\ulcorner G(h)\urcorner$ is the m-tuple of of programs which implements the computation "what does my source code do on input history $h$". The behavior of the agent in a particular mathematical universe is given by taking each program in $\ulcorner G(h)\urcorner$, using the mathematical universe to figure out what each of the programs outputs, and then using $\Dec$ to convert that bitstring to an action.

**Definition 1.3:** *Given $y\in\Gamma$ and $h\in\H$,
we define $G^{y}(h)\in\A$ to be $\Dec(a)$ for $a\in\Sigma^{M}$
given by $a_{i}:=y(\ulcorner G(h)\urcorner_{i})$.*

We can now extract counterfactuals from any $\Lambda\in\Box\P{\Gamma}.$
Specifically, given any policy $\pi$ we define some $\Co{\pi}{}\subseteq\P{\Gamma}$ (the set of stuff consistent with behaving according to $\pi$, for a yet-to-be-defined notion of consistency)
and define the counterfactual as $\Lambda\cap\top_{\Co{\pi}{}}$.
We could use the "naive" definition:

**Definition 1.4:** *$\Co{\pi}{naive}$ is the set of all
$(y,\alpha)\in\P{\Gamma}$ s.t. for any $h\in\H$, $G^{y}(h)=\pi(h)$.*

Per discussion above, it seems better to use a different definition.
We will use the notation $h\sqsubset g$ to mean "$h$ is a proper
prefix of $g$" and $h\sqsubseteq g$ to mean "$h$ is a prefix
of $g$",

**Definition 1.5:** *Given $(y,\alpha)\in\P{\Gamma}$, let
$\H_{y\alpha}$ be the set of all $h\in\H$ s.t. the following two
conditions hold:*

1. *For all $ga\in\H\times\A$ and $y'\in\Gamma$ where $ga\sqsubset h$ and $y'\in\alpha$, $G^{y'}(g)=a$.*

2. *$h\sqsubset d$ for some $d\in\D$ s.t. $d\C y$.*

*$\Co{\pi}{fair}$ is the set of all $(y,\alpha)\in\P{\Gamma}$
s.t. for any $h\in\H_{y\alpha}$, $G^{y}(h)=\pi(h)$.*

Here, condition 1 says that we only "take responsibility" for
the action on a particular history when the history was actually observed
(all preceding evaluations of the agent are realized computations).
Condition 2 says that we only "take responsibility" when the computer
is working correctly.

At this stage, Definition 1.5 should be regarded as tentative, since
we only have one result so far that validates this definition, namely that the set of $(y,\alpha)$ in $C^{\pi}_{\text{fair}}$ only depends on what the policy $\pi$ does on possible inputs, instead of having the set depending on what the policy does on impossible inputs where one of the past memorized actions is not what the policy would actually do. We hope to rectify this in future
articles.

$$\newcommand{\LPol}[1]{#1^{\text{pol}}}$$

Putting everything together: given a loss function $L:\P{\Gamma}\rightarrow\mathbb{R}$, which depends on how the mathematical universe is, and which computations are realized or not-realized,
the loss of policy $\pi$ is given by just applying the bridge transform to coax the hypotheses into the appropriate form, intersecting with the set of possibilities consistent with the agent behaving according to a policy, and evaluating the expectation of the loss function, as detailed below.

$$
\LPol L(\pi,\zeta):=(\B(\zeta\ltimes\Theta)\cap\top_{\Co{\pi}{fair}})(L)
$$

## Evaluating agents

So far we regarded the agent's source code $G$ and the agent's policy
$\pi$ as independent variables, and explained how to evaluate different
policies given fixed $G$. However, in reality the policy is determined
by the source code. Therefore, it is desirable to have a way to evaluate
different *codes*. We achieve this using an algorithmic [information](https://www.lesswrong.com/posts/Q4hLMDrFd8fbteeZ8/measuring-optimization-power)
[theory](https://www.alignmentforum.org/posts/dPmmuaz9szk26BkmD/vanessa-kosoy-s-shortform?commentId=ovBmi2QFikE6CRWtj#comments)
approach. 

We also want to allow the loss function to depend on $G$. That is,
we postulate $L:(\Pro^{M})^{\H}\times\P{\Gamma}\rightarrow\R$. Specifically, since $\Pro^{M}$ can be thought of as the space of actions, $(\Pro^{M})^{\H}$ is basically the space of policies. In
section 3 we will see why in some detail, but for now think of the
difference between "maximizing my own happiness" and "maximizing
Alice's happiness": the first is defined *relative* to the agent
(depends on $G$) whereas the second is absolute (doesn't depend on
$G$). In particular, for a selfish agent that just cares about its own observations, its loss function must reference its own source code.

$$\newcommand{\G}{\mathrm{g}}$$

**Definition 1.6:** *Denote $G^*:\H\rightarrow\A$ the
policy actually implemented by $G$. Fix $\xi\in\Delta(\A^{\H}$).
The *physicalist intelligence of $G$ relative to the baseline policy mixture $\xi$,
prior $\zeta$ and loss function $L$* is defined by:*

$$
\mathrm{\G}(G\mid\xi;\zeta,L):=-\log\mathrm{Pr}_{\pi\sim\xi}[\LPol L(\ulcorner G\urcorner,\pi,\zeta)\leq\LPol L(\ulcorner G\urcorner,G^*,\zeta)]
$$

Notice that $\LPol L$ depends on $\ulcorner G\urcorner$ in two ways:
through the direct dependence of $L$ on $\ulcorner G\urcorner$ and
through $\Co{\pi}{fair}$.

In particular, it makes sense to choose $\zeta$ and $\xi$ as simplicity
priors.

There is no obvious way to define "physicalist AIXI", since we
cannot have $\mathrm{\G}=\infty$. For one thing, $\G$ is not even
defined for uncomputable agents. In principle we could define it for
[non-uniform](https://en.wikipedia.org/wiki/Advice\_(complexity))
agents, but then we get a fixpoint problem that doesn't obviously
have a solution: finding a non-uniform agent $G$ s.t. $\LPol L(\ulcorner G\urcorner,G^*,\zeta)=\min_{\pi}\LPol L(\ulcorner G\urcorner,\pi,\zeta)$. On the other hand, once we spell out the infinitary
($N=\infty$) version of the formalism, it should be possible to prove
the existence of agents with arbitrarily high finite $\G$. That's
because our agent can use quining to access its own source code $\ulcorner G\urcorner$,
and then brute force search a policy $\pi_{\epsilon}^*$ with $\LPol L(\ulcorner G\urcorner,\pi_{\epsilon}^*,\zeta)<\inf_{\pi:\H\rightarrow\A}\LPol L(\ulcorner G\urcorner,\pi,\zeta)+\epsilon$.

# 2. Properties of the bridge transform

In this section we will *not* need to assume $\Gamma$ is of the
form $\Sigma^{\Pro}$: unless stated otherwise, it will be any finite
set.

## Sanity test

**Proposition 2.1:** *For any $\Gamma$, $\Phi$ and $\Theta\in\HUC{(\Gamma\times\Phi)}$,
$\B(\Theta)$ exists and satisfies $\Pr{\Gamma\times\Phi}\B(\Theta)=\Theta$.
In particular, if $\Theta\in\Box(\Gamma\times\Phi)$ then $\B(\Theta)\in\Box(\P{\Gamma}\times\Phi)$.*

## Downwards closure

Roughly speaking, the bridge transform tell us which computations
are physically realized. But actually it only bounds it from one side:
some computations are definitely realized but any computation *might*
be realized. One explanation for why it must be so is: if you looked at the world in more detail, you might realize that there are small-scale, previously invisible, features of the world which depend on novel computations. There is a direct tension between bounding both sides (i.e. being able to say definitively that a computation isn't instantiated) and having the desirable property that learning more about the small-scale structure of the universe narrows down the uncertainty. To formalize this, we require the following definitions:

**Definition 2.1:** *Let $X$ be a partially ordered set
(poset). Then the *induced partial order* on $\Delta^{c}X$ is
defined as follows. Given $\theta,\eta\in\Con X$, $\theta\preceq\eta$
if and only if for any monotonically non-decreasing function $f:X\rightarrow\R_{+}$,
$\theta(f)\leq\eta(f)$.*

This is also called the stochastic order (which is standard mathematical terminology). Intuitively, $\theta\preceq\eta$ means that $\eta$ has its measure further up in the poset than $\theta$ does. To make that intuition formal, we can also characterize the induced order as follows:

**Proposition 2.2:** *Let $X$ be a poset, $\theta,\eta\in\Con X$.
Then, $\theta\preceq\eta$ if and only if there exists $\kappa:X\rightarrow\Delta X$
s.t.:*

* *For all $x\in X$, $y\in\Sup\kappa(x)$: $x\leq y$.*

* $\kappa_*\theta\leq\eta$

**Proposition 2.3:** *Let $X$ be a poset, $\theta,\eta\in\Con X$.
Then, $\theta\preceq\eta$ if and only if there exists $\kappa:X\rightarrow\Delta X$
s.t.:*

* *For all $x\in X$, $y\in\Sup\kappa(x)$: $x\geq y$.*

* $\theta\leq\kappa_*\eta$

Or, in words, you can always go from $\theta$ to $\eta$ by moving probability mass upwards from where it was, since $\kappa(x)$ is always supported on the set of points at-or-above $x$.

We can now state the formalization of only bounding one side of the bridge transform. Let $\P{\Gamma}\times\Phi$ be
equipped with the following order. $(y,\alpha,x)\leq(y',\alpha',x')$
if and only if $y=y'$, $x=x'$ and $\alpha\subseteq\alpha'$. Then:

**Proposition 2.4:** *For any $\Gamma$, $\Phi$ and $\Theta\in\HUC{(\Gamma\times\Phi)}$, $\B(\Theta)$
is downwards closed w.r.t. the induced order on $\Con{(\P{\Gamma}\times\Phi)}$.
That is, if $\theta\in\B(\Theta)$ and $\eta\preceq\theta$ then $\eta\in\B(\Theta)$.*

$$\newcommand{\Dtv}[2]{\mathrm{d_{\text{TV}}(#1,#2)}}$$

## Simple special case

Let's consider the special case where there's only one program, which can produce two possible outputs, a 0 and a 1. And these two outputs map to two different distributions over physics outcomes in $\Phi$. Intuitively, if the computation isn't realized/instantiated, the distribution over physics outcome should be identical, while if the computation is realized/instantiated, it should be possible to look at the physics results to figure out how the computation behaves. The two probability distributions may overlap some intermediate amount, in which case it should be possible to write the two probability distributions as a mixture between a probability distribution that behaves identically regardless of the program output (the "overlap" of the two distributions), and a pair of probability distributions corresponding to the two different program outputs which are disjoint. And the total variation distance ($\Dtv{\phi(0)}{\phi(1)}$) between the two probability distributions is connected to the size of the distribution overlap. Proposition 2.5 makes this formal.

**Proposition 2.5:** *Consider a finite set $X$, $\phi_{?}\in\Delta X$,
$\phi_{!}:\{0,1\}\rightarrow\Delta X$, $p\in[0,1]$ and $\phi:=(1-p)\phi_{?}+p\phi_{!}$.
Then, $p\geq\Dtv{\phi(0)}{\phi(1)}$. Conversely, consider any $\phi:\{0,1\}\rightarrow\Delta X$.
Then, there exist $\phi_{?}\in\Delta X$ and $\phi_{!}:\{0,1\}\rightarrow\Delta X$
s.t. $\phi=(1-p)\phi_{?}+p\phi_{!}$ for $p:=\Dtv{\phi(0)}{\phi(1)}$.*

The bridge transform should replicate this same sort of analysis. We can interpret the case of "total uncertainty over math, but knowing how physics turns out conditional on knowing how math turns out" by $\top_{\Gamma}\ltimes\phi$ for some $\phi:\Gamma\rightarrow\Delta\Phi$. Taking the special case where $\Gamma=\{0,1\}$ for one program that can output two possible answers, it should return the same sort of result, where the two probability distributions can be decomposed into their common overlap, and non-overlapping pieces, and the overlapping chunk of probability measure should be allocated to the event "the computation is not instantiated", while the disjoint ones should be allocated to the event "the computation is instantiated". As it turns out, this does indeed happen, with the same total-variation-distance-based upper bound on the probability that the computation is unrealized (namely, $1-\Dtv{\phi(0)}{\phi(1)}$)

**Proposition 2.6:** *Consider any $\Phi$ and $\phi:\{0,1\}\rightarrow\Delta\Phi$.
Denote $U:=\{0,1\}\times\{\{0,1\}\}\times\Phi$ (the event "program
is unrealized"). Let $\Lambda:=\B(\top_{\{0,1\}}\ltimes\phi)$.
Then,*

$$
\Lambda(\chi_{U})=1-\Dtv{\phi(0)}{\phi(1)}
$$


## Bound in terms of support

If the physical state is $x\in\Phi$, then it is intuitively obvious
that the physically manifest facts about the computational universe
$y\in\Gamma$ must include the fact that $(y,x)\in\Sup\Theta$. Formally:

$$\newcommand{\Sus}[1]{\mathrm{sus}^{#1}}$$

**Proposition 2.7:** *If $X$ is a poset and $\Theta\in\HUC X$,
then $\Theta^{\downarrow}$ will denote downward closure of $\Theta$.
For any $\Gamma$, $\Phi$ and $\Theta\in\HUC{(\Gamma\times\Phi)}$
if $(y,\alpha,x)\in\Sup\B(\Theta)$ and $y'\in\alpha$, then $(y',x)\in\Sup\Theta$.
Moreover, define $\Sus{\Theta}:\Phi\rightarrow2^{\Gamma}$ by $\Sus{\Theta}(x):=\{y\in\Gamma\mid(y,x)\in\Sup\Theta\}$.
Then, $\B(\Theta)\subseteq(\Theta\ltimes\Sus{\Theta})^{\downarrow}$
(we slightly abuse notation by treating $\Sus{\Theta}$ as a mapping
$\Gamma\times\Phi\rightarrow2^{\Gamma}$ that doesn't depend on the
first argument, and also playing loose with the order of factors in
the set on which our HUCs live).*

Putting that into words, all the $(\alpha,x)$ that are in the support of the bridge transform have $\alpha$ being a subset of the support of $\Theta$ restricted to $x$. The bridge transform knows not to provide $\alpha$ sets which are too large. Further, the bridge transform is smaller/has more information than the transform that just knows about what the support of $\Theta$ is.

## Idempotence

$$\newcommand{\Id}[1]{\mathrm{id}_{#1}}$$

$$\newcommand{\Di}[1]{\mathrm{diag}_{#1}}$$

Imagine replacing the physical state space $\Phi$ by the space of
manifest facts $2^{\Gamma}$. Intuitively, the latter should contain
the same information about computations as the former. Therefore,
if we apply the bridge transform *again* after making such a replacement,
we should get the same thing. Given any set $X$, we denote $\Id X:X\rightarrow X$
the identity mapping and $\Di X:X\rightarrow X\times X$ the diagonal
mapping.

**Proposition 2.8:** *For any $\Gamma$, $\Phi$ and $\Theta\in\HUC{(\Gamma\times\Phi)}$*

$$
\B(\Pr{\Gamma\times2^{\Gamma}}\B(\Theta))=[(\Id{\Gamma}\times\Di{2^{\Gamma}})_*\Pr{\Gamma\times2^{\Gamma}}\B(\Theta)]^{\downarrow}
$$

## Refinement

If we refine a hypothesis (i.e. make it more informative) in the ultradistribution
sense, the bridge transform should also be more refined:

**Proposition 2.9:** *For any $\Gamma$, $\Phi$ and $\Theta_{1},\Theta_{2}\in\HUC{(\Gamma\times\Phi)}$,
if $\Theta_{1}\subseteq\Theta_{2}$ then $\B(\Theta_{1})\subseteq\B(\Theta_{2})$.*

Another way of refining a hypothesis is refining the ontology, i.e.
moving to a richer state space. For example, we can imagine one hypothesis
that describes the world in terms of macroscopic objects and another
hypothesis that describes the worlds in term of atoms which are otherwise
perfectly consistent with each other. In this case, we also expect
the bridge transform to become more refined. This desiderata is where the downwards closure comes from, because it's always possible to pass to a more detailed view of the world and new computations could manifest then. It's also worth noting that, if you reduce your uncertainty about math, or physics, or move to a more detailed state space, that means more computations are instantiated, and the loss goes down as a result.

**Proposition 2.10:** *For any $\Gamma$, $\Phi_{1}$, $\Phi_{2}$,
$t:\Phi_{2}\rightarrow\Phi_{1}$ and $\Theta\in\HUC{(\Gamma\times\Phi_{2})}$*

$$
(\Id{\P{\Gamma}}\times t)_*\B(\Theta)\subseteq\B((\Id{\Gamma}\times t)_*\Theta)
$$

In general, when we refine the ontology, the manifest facts can become
strictly refined: the richer state "knowns more" about computations
than the poorer state. However, there is a special case when we don't
expect this to happen, namely when the rich state depends on the computational
universe only through the poor state. Roughly speaking, in this case
once we have the poor state we can sample the rich state without evaluating
any more programs. To formalize this, it is convenient to introduce
pullback of HUCs, which is effectively the probabilistic version of taking a preimage.

**Definition 2.2:** *Let $X,Y$ be finite sets and $t:X\rightarrow Y$
a mapping. We define the pullback[^7] operator
$t^*:\HUC{Y\rightarrow\HUC X}$ by*

$$
t^*\Theta:=\{\theta\in\Con X\mid t_*\theta\in\Theta\}
$$

**Proposition 2.11:** *Consider any $\Gamma$, $\Phi_{1}$,
$\Phi_{2}$, $t:\Phi_{2}\rightarrow\Phi_{1}$, $\Xi:\Phi_{1}\rightarrow\HUC{\Phi_{2}}$
and $\Theta\in\HUC{(\Gamma\times\Phi_{1})}$ s.t. $t\Xi=\Id{\Phi_{1}}$.
Then,*

$$
(\Id{\P{\Gamma}}\times\Xi)_*\B(\Theta)\subseteq\B((\Id{\Gamma}\times\Xi)_*\Theta)\subseteq(\Id{\P{\Gamma}}\times t)^*\B(\Theta)
$$

*In particular,*

$$
\Pr{\P{\Gamma}}\B(\Theta)=\Pr{\P{\Gamma}}\B((\Id{\Gamma}\times\Xi)_*\Theta)
$$

In other words, we can think of $\Phi_1$ as the coarse-grained version, and $\Phi_2$ as the fine-grained version, and $t$ as the coarse-graining function, and $\Xi$ as the function mapping a coarse grained state to some uncertainty over what the corresponding fine-grained state is. $\Theta$ is your starting uncertainty over computations and the coarse-grained state. Then, in order from most to least informative, you could apply $\Xi$ after the bridge transform, apply $\Xi$ before the bridge transform, or pull back along $t$ after the bridge transform (because the pullback is the most uninformative inverse, in a sense). But all this stuff only affects the information about the fine-grained state $\Phi_2$. If you were to forget about that, the data about computations and how they're instantiated is the same no matter whether you do the bridge transform with the coarse state, or try your best to fill in what the fine state is first, because the "fill in the fine state" function doesn't depend on your uncertainty about computations, so it doesn't contain any more information about how math is connected to physics than the original coarse-grained view had.

## Mixing hypotheses

In general, we cannot expect the bridge transform to commute with
taking probabilistic mixtures. For example, let $\Gamma=\{0,1\}$,
$\Phi=\{0,1\}$, $\Theta_{1}=\top_{\{00,11\}}$, $\Theta_{2}=\top_{\{01,10\}}$.
Clearly either for $\Theta_{1}$ or for $\Theta_{2}$, we expect the
program to be realized (since the physical state encodes its output).
On the other hand, within $\frac{1}{2}\Theta_{1}+\frac{1}{2}\Theta_{2}$
we have both the distribution "50% on 00, 50% on 01" and the
distribution "50% on 10, 50% on 11". The physical state no longer
necessarily "knows" the output, since the mixing "washes away"
information. However, there's no reason why mixing would *create*
information, so we can expect the mixture of bridge transforms to
be a *refinement* of the bridge transform of the mixture.

**Proposition 2.12:** *For any $\Gamma$, $\Phi$, $\Theta_{1},\Theta_{2}\in\HUC{(\Gamma\times\Phi)}$
and $p\in[0,1]$*

$$
p\B(\Theta_{1})+(1-p)\B(\Theta_{2})\subseteq\B(p\Theta_{1}+(1-p)\Theta_{2})
$$

Now suppose that we are we are mixing hypotheses whose state spaces
are *disjoint*. In this case, there can be no washing away since
the state "remembers" which hypothesis it belongs to.

**Proposition 2.13:** *Consider some $\Gamma$, $\Phi_{1}$,
$\Phi_{2}$, $\Theta_{1}\in\HUC{(\Gamma\times\Phi_{1})}$, $\Theta_{1}\in\HUC{(\Gamma\times\Phi_{2})}$
and $p\in[0,1]$. Regard $\Phi_{1,2}$ as subsets of $\Phi_{1}\sqcup\Phi_{2}$,
so that $p\Theta_{1}+(1-p)\Theta_{2}\in\HUC{(\Gamma\times(\Phi_{1}\sqcup\Phi_{2}))}$.
Then,*

$$
p\B(\Theta_{1})+(1-p)\B(\Theta_{2})=\B(p\Theta_{1}+(1-p)\Theta_{2})
$$

In particular, if we consider the hypotheses within a prior as having
disjoint state spaces because of using different ontologies, then the total bridge transform (Definition 1.2)
is just the ordinary bridge transform, justifying that definition.

## Conjunction

Given a hypothesis and a subset of $\Gamma$, we can form a new hypothesis
by performing conjunction with the subset. Intuitively, the same conjunction
should apply to the manifest facts.

**Proposition 2.14:** *Consider some $\Gamma$, $\Phi$,
$\Theta\in\HUC{(\Gamma\times\Phi)}$ and $F\subseteq\Gamma$. Let
$\cap_{F}:2^{\Gamma}\rightarrow2^{\Gamma}$ be defined by $\cap_{F}(\alpha):=F\cap\alpha$.
Then,*

$$
\B_{\Gamma}(\Theta\cap\top_{F\times\Phi})=(\Id{\Gamma}\times\cap_{F}\times\Id{\Phi})_*(\B_{\Gamma}(\Theta)\cap\top_{F\times2^{\Gamma}\times\Phi})
$$

*Moreover, let $\iota:F\rightarrow\Gamma$ and $\P{\iota}:\P F\rightarrow\P{\Gamma}$
be the natural injections. Then,*

$$
\B_{F}((\iota\times\Id{\Phi})^*(\Theta\cap\top_{F\times\Phi}))=(\P{\iota}\times\Id{\Phi})^*\B_{\Gamma}(\Theta\cap\top_{F\times\Phi})
$$

That first identity says that it doesn't matter whether you update on a fact about math first and apply the bridge transform later, or whether you do the bridge transform first and then "update" (actually it's slightly more complicated than that since you have to narrow down the sets $\alpha$) the hypothesis afterwards.
The second identity shows that, post conjunction with $F$, taking
the bridge transform with respect to $F$ is essentially the same
as taking it with respect to $\Gamma$. If a batch of math universes is certain not to occur, then dropping them entirely from the domain $\Gamma$ produces the same result.

As a corollary, we can eliminate dependent variables. That is, suppose
some programs can be expressed as functions of other programs (using
$\Theta$). Intuitively, such programs can be ignored. Throwing in the dependent variables and doing the bridge transform on that augmented space of "math universes" produces the same results as trying to compute the dependent variables *after* the bridge transform is applied. So, if you know how to generate the dependent variables, you can just snip them off your hypothesis, and compute them as-needed at the end. Formally:

**Proposition 2.15:** *Consider some $\Gamma_{0}$, $\Gamma_{1}$,
$r:\Gamma_{0}\rightarrow\Gamma_{1}$, $\Phi$, $\Theta\in\HUC{(\Gamma_{0}\times\Phi)}$.
Let $\iota:\Gamma_{0}\rightarrow\Gamma_{0}\times\Gamma_{1}$ be given
by $\iota(y):=(y,r(y))$. Then,*

$$
\B((\iota\times\Id{\Phi})_*\Theta)=(\P{\iota}\times\Id{\Phi})_*\B(\Theta)
$$

## Factoring $\Gamma$

When $\Gamma$ is of the form $\Gamma_{0}\times\Gamma_{1}$, we can
interpret the same $\Theta\in\Box(\Gamma\times\Phi)$ in two ways:
treat $\Gamma_{1}$ as part of the computational universe, or treat
it as part of the physical state. We can also "marginalize" over
it altogether staying with a HUD on $\Gamma_{0}\times\Phi$. The bridge
transforms of these objects satisfy a simple relationship:

**Proposition 2.16:** *Consider some $\Gamma_{0}$, $\Gamma_{1}$,
$\Phi$ and $\Theta\in\HUC{(\Gamma_{0}\times\Gamma_{1}\times\Phi)}$.
Define $\pi:\P{\Gamma_{0}\times\Gamma_{1}}\rightarrow\P{\Gamma_{0}}$
by $\pi(y,z,\alpha):=(y,\{y'\in\Gamma_{0}\mid(y',z)\in\alpha\})$.
Then,*

$$
(\pi\times\Id{\Phi})_*\B_{\Gamma_{0}\times\Gamma_{1}}(\Theta)\subseteq\Pr{\P{\Gamma_0}\times\Phi}\B_{\Gamma_{0}}(\Theta)\subseteq\B_{\Gamma_{0}}(\Pr{\Gamma_{0}\times\Phi}\Theta)
$$

Intuitively, the least informative result is given by completely ignoring the $\Gamma_1$ part of math about how a particular batch of computations turns out. A more informative result is given by treating $\Gamma_1$ as an aspect of physics. And the most informative result is given by treating $\Gamma_1$ as an aspect of math, and then pruning the resulting $\alpha:2^{\Gamma_0\times\Gamma_1}$ down to be over $2^{\Gamma_0}$ by plugging in the particular $\Gamma_1$ value.

## Continuity

**Proposition 2.17:** *$\B(\Theta)$ is a continuous function
of $\Theta$.*

Lamentably, this only holds when all the involved sets are finite. However, a generalization of continuity, scott-continuity, still appears to hold in the infinite case. Intuitively, the reason why is, if $\Phi$ is a continuous space, like the interval $[0,1]$, you could have a convergent sequence of hypotheses $\Theta$ where the events "program outputs 0" and "program outputs 1" are believed to result in increasingly similar outputs in physics, and then in the limit, where they produce the exact same physics outcome, the two possible outputs of the program suddenly go from "clearly distinguishable by looking at physics" to "indistinguishable by looking at physics" and there's a sharp change in whether the program is instantiated. This is still a Scott-continuous operation, though.

# 3. Constructing loss functions

In section 1 we allowed the loss function $L:\P{\Gamma}\rightarrow\mathbb{R}$
to be arbitrary. However, in light of Proposition 2.4, we might as
well require it to be monotonically non-decreasing. Indeed, we have
the following:

**Proposition 3.1:** *Let $X$ be a finite poset, $f:X\rightarrow\R$
and $\Theta\in\HUC X$ downward closed. Define $f^{\max}:X\rightarrow\R$
by $f^{\max}(x):=\max_{y\leq x}f(y)$. Observe that $f^{\max}$ is
always non-decreasing. Then, $\Theta(f)=\Theta(f^{\max})$.*

This essentially means that downwards-closed HUC's are entirely pinned down by their expectation value on monotone functions, and all their other expectation values are determined by the expectation value of the "nearest monotone function", the $f^{\max}$.

This requirement on $L$ (we will call it the "monotonicity principle")
has strange consequences that we will return to below.

The informal discussion in section 1 hinted at how the computationalist
stance allows describing preferences via a loss function of the type
$\P{\Gamma}\rightarrow\mathbb{R}$. We will now proceed to make it
more formal.

## Selfish agents

A selfish agent naturally comes with a cartesian loss function $L:\D\rightarrow\mathbb{\R}.$ 

$$\newcommand{\X}[1]{\mathcal{X}_{#1}}$$

$$\newcommand{\Ph}[1]{#1^{\text{phys}}}$$

**Definition 3.1:** *For any $(y,\alpha)\in\P{\Gamma}$,
the set of *experienced histories* $\X{y\alpha}$ is the set of
all $ha\in\H_{y\alpha}\times\A$ (see Definition 1.5) s.t. for any
$y'\in\alpha$, $G^{y'}(h)=a$. The *physicalized loss function*
is*

$$
\Ph L(y,\alpha):=\min_{ha\in\X{y\alpha}}\max_{d\in\D:ha\sqsubseteq d}L(d)
$$

Notice that $\Ph L$ implicitly depends on $\ulcorner G\urcorner$
through $\X{y\alpha}$.

Intuitively, if $\alpha$ is a large set, there's different mathematical universes that result in the agent's policy being different, but which are empirically indistinguishable (ie, they're in the same set $\alpha$), so the agent's policy isn't actually being computed on those histories where the contents of $\alpha$ disagree on what the agent does. The agent takes the loss of the best history that occurs, but if there's uncertainty over what happens afterwards/later computations aren't instantiated, it assumes the worst-case extension of that history.

This definition has the following desirable properties:

* If $g\in\X{y\alpha}$, $ha\in\H\times\A$ and $ha\sqsubseteq g$,
then $h\in\X{y\alpha}.$

* $\Ph L$ is monotone. If the set of indistinguishable mathematical universes gets larger, fewer histories are instantiated, so the loss is larger because of the initial minimization.

* Fix $(y,\alpha)\in\P{\Gamma}$. Suppose there is $d\in\D$ s.t.
$\X{y\alpha}=\{ha\in\H\times\A\mid ha\sqsubseteq d\}$. Then $\Ph L(y,\alpha)=L(d)$. If the set of indistinguishable mathematical universes all agree on what the destiny of the agent is and disagree elsewhere, then the loss is just the loss of that destiny.

We can think of inconsistent histories within $\X{y\alpha}$ as associated
with different *copies* of the agent. If $ha\in\X{y\alpha}$ is
s.t. for any $g\sqsupset ha$, $g\not\in\X{y\alpha}$, then $h$ represents
the history of a copy which reached the end of its life. If there
is only one copy and its lifespan is maximal, then $\Ph L$ produces
the cartesian loss of this copy. If the lifespan is less than maximal,
$\Ph L$ produces the loss of the *worst-case* continuation of
the history which was experienced. In other words, death (i.e., the future behavior of the agents policy not being a computation that has any effect on the state of the universe) is always
the worst-case outcome. If there are multiple copies, the *best-off*
copy determines the loss.

Definition 3.1 is somewhat arbitrary, but the monotonicity principle
is not. Hence, for a selfish agent (i.e. if $L$ only depends on $y,\alpha$
through $\X{y\alpha}$) death must be the worst-case outcome (for
fixed $y$). Note that this weighs against getting the agent to shut down if it's selfish. And creating additional copies must always be beneficial.
For non-selfish agents, similar observations hold if we replace "death"
by "destruction of everything that matters" and "copies of the
agent" by "copies of the (important sector of the) world".

This is an odd state of affairs. Intuitively, it seems coherent to
have preferences that contradict the monotonicity principle. Arguably,
humans have such preferences, since we can imagine worlds that we
would describe as net negative. At present, we're not sure how to
think about it, but here are some possibilities, roughly in order
of plausibility:

1. Perfect physicalists must obey the monotonicity principle, but
humans are not perfect physicalists. Indeed, why would they be? The
ancestral environment didn't require us to be physicalists, and it
took us many centuries to stop thinking of human minds as "metaphysically
privileged". This might mean that we need of theory of "cartesian
agents in a physicalist universe". For example, perhaps from a cartesian
perspective, a physicalist hypothesis needs to be augmented by some
"theory of epiphenomena" which predicts e.g. which of several
copies will "I" experience.

2. Something is simply wrong about the formalism. One cheap way to
escape monotonicity is by taking the *hull of maximal points*
of the bridge transform. However, this spoils compatibility with refinements
(Propositions 2.9 and 2.10). We expect this to spoil ability to learn
in some way.

3. Death is inherently an irreversible event. Every death is the agent's
first death, therefore the agent must always have uncertainty about
the outcome. Given a non-dogmatic prior, there will be some probability
on outcomes that are not actually death. These potential "afterlives"
will cause the agent to rank death as better than the absolute worst-case.
So, "non-survivalist" preferences can be reinterpreted as a prior
biased towards not believing in death. Similar reasoning applies to
non-selfish agents.

4. We should just bite the bullet and prescriptively endorse the odd consequences
of the monotonicity principle.

In either case, we believe that continuing the investigation of monotonic physicalist agents is a promising path forward. If a natural extension or modification of the theory exists which admits non-monotonic agents, we might discover it along the way. If we don't, at least we will have a better lower bound for the desiderata such a theory should satisfy.

## Unobservable states

$$\newcommand{\S}{\mathcal{S}}$$

$$\newcommand{\AB}{\mathcal{B}}$$

What if the loss depends on an unobservable state from some state
space $\S$? In order to encode states, let's assume some $M_{\S}\in N$
and $\Dec_{\S}:\Sigma^{M_{\S}}\rightarrow\S$. That is, just as the action of the agent can be thought of as a tuple of programs returning the bits of the computation of the agent's action, a  state can be thought of as a tuple of programs returning the bits of the computation of what the next state is. Let $\AB$ be a finite
set ("nature's actions"). Just as observations, in the selfish case, occur based on what computations are instantiated, nature's choice of action $\AB$ depends on what computations are instantiated. Denote $\H_{\S}:=(\A\times\AB\times\O\times\S)^{<N}$,
$\D_{\S}:=(\A\times\AB\times\O\times\S)^{N-1}\times\A$. Let $\ulcorner T\urcorner:\S\times\A\times\AB\rightarrow\Pro^{M_{\S}}$
be the "ontological"[^8] transition rule
program, $E:\S\rightarrow\O$ the observation rule and $L:\D_{\S}\rightarrow\R$
the loss function.

**Definition 3.2:** *For any $(y,\alpha)\in\P{\Gamma}$,
the set of *admissible histories* $\mathcal{Z}_{y\alpha}$ is
the set of all $ha\in\H_{\S}\times\A$ s.t. the following conditions
hold:*

1. $\Pr{\H\times\A}ha\in\X{y\alpha}$

2. *If $g\in\H_{\S}\times\A\times\AB$, $o\in\O$ and $s\in\S$
are s.t. $gos\sqsubseteq h$ then $o=E(s)$.*

3. *If $g\in\H_{\S}\times\A\times\AB\times\O$, $s_{1},s_{2}\in\S$,
$a\in\A$, $b\in\AB$ and $o\in\O$ are s.t. $gs_{1}abos_{2}\sqsubseteq h$
and $y'\in\alpha$ then $T^{y'}(s_{1},a,b)=s_{2}$ (the notation $T^{y}$
is analogical to Definition 1.3).*

Pretty much, condition 1 says that the visible parts of the history are consistent with how math behaves in the same sense as we usually enforce with $\mathcal{H}_{y\alpha}$. Condition 2 says that the history should have the observations arising from the states in the way that $E$ demands. And condition 3 says that the history should have the states arising from your action and nature's action and the previous state in the way the transition program dictates (in that batch of mathematical universes).

*The *physicalized loss function* is*

$$
L^{\text{s-phys}}(y,\alpha):=\min_{ha\in\mathcal{Z}_{y\alpha}}\max_{g\in\D_{\S}:ha\sqsubseteq g}L(g)
$$

This arises in a nearly identical way to the loss function from Definition 3.1, it's precisely analogous.

That is, for a history to be admissible we require not only the agent
to be realized on all relevant inputs but also the transition rule.

We can also define a version with $\ulcorner T\urcorner:\S\times\AB\rightarrow\Pro^{M_{\S}}$
in which the agent is *not* an explicit part of the ontology.
In this case, the agent's influence is solely through the "entanglement"
of $G$ with $T$. I.e., the agent's beliefs over math have entanglements between how the agent's policy turns out and how the physics transition program behaves, s.t. when conditioning on different policies, the computations corresponding to different histories become instantiated.

## Cellular automata

$$\newcommand{\Z}{\mathbb{Z}}$$

What if our agent maximizes the number of gliders in the game of life?
More generally, consider a cellular automaton with cell state space
$\S$ and dimension $d\in\N$. For any $t\in\N$, let $N_{t}\subseteq\Z^{d}$
be the set of cells whose states are necessary to compute the state
of the origin cell at timestep $t$. Let $\ulcorner T_{t}\urcorner:\S^{N_{t}}\rightarrow\Pro^{M_{\S}}$
be the program which computes the time evolution of the cellular automaton. I.e., it's mapping the state of a chunk of space to the computation which determines the value of a cell $t$ steps in the future.
Denote $\D_{\S}^{d}:=\S^{\N\times\Z^{d}}$. This is the space of all possible histories, specifying everything that occurs from the start of time. Let $L:\D_{\S}^{d}\rightarrow\R$
be the loss function (the $\N$ coordinate is time). 

**Definition 3.3:** *For any $\alpha\in2^{\Gamma}$, the
set of *admissible histories* $\mathcal{Z}_{\alpha}$ is the set
of all pairs $(C,h)$ where $C\subset\N\times\Z^{d}$ is finite and
$h:C\rightarrow\S$ s.t. the following conditions hold:*

1. *If $(t_{2},c)\in C$, $t_{1}<t_{2}$ and $v\in N_{t_{2}-t_{1}}$
then $(t_{1},c+v)\in C$.*

2. *If $(t_{2},c)\in C$, $t_{1}<t_{2}$ and $y\in\alpha$ then
$T_{t_{2}-t_{1}}^{y}(\lambda v.h(t_{1},c+v))=h(t_{2},c)$.*

Basically, the $C$ is a chunk of spacetime, and condition 1 says that said chunk must be closed under taking the past lightcone, and condition 2 says the state of that spacetime chunk must conform with the behavior of the transition computation.

*The *physicalized loss function* is* defined very much as it used to be, except that for what used to be "past histories" now it has states of past-closed chunks of spacetime.

$$
L^{\text{c-phys}}(y,\alpha):=\min_{(C,h)\in\mathcal{Z}_{\alpha}}\max_{g\in\D_{\S}^{d}:h=g|_{C}}L(g)
$$

Notice that, since the dynamics of the cellular automaton is fixed,
the only variables "nature" can control are the initial conditions.
However, it is straightforward to generalize this definition to cellular
automata whose time evolution is underspecified (similarly to the
$\AB$ we had before).

## Diamond maximizer

What if our agent maximizes the amount of diamond (or some other physical
property)? In order to define "diamond" we need a theory of physics.
For example, we can start with macroscopic/classical physics and define
diamond by its physical and/or chemical properties. Or, we can start
with nonrelativistic quantum mechanics and define diamond in terms
of its molecular structure. These definitions might be equivalent
in our universe, but different in other hypothetical universes (this
is similar to [twin Earth](https://en.wikipedia.org/wiki/Twin\_Earth\_thought\_experiment)).
In any case, there is no single "right" choice, it is simply a
matter of definition.

A theory of physics is mathematically quite similar to a cellular
automaton. This theory will usually be incomplete, something that
we can represent in infra-Bayesianism by Knightian uncertainty. So,
the "cellular automaton" has underspecified time evolution. Using
this parallel, the same method as in the previous subsection should
allow physicalizing the loss function. Roughly speaking, a possible
history is "real" when the computation simulating the time evolution
of the theory of physics with the corresponding initial condition
is realized. However, we are not sure what do if the theory of physics
is *stochastic*. We leave it as an open problem for now.

To summarize, you can just take a hypothesis about how computations are entangled with some theory of physics, some computations simulating the time evolution of the universe (in which the notion of "diamond" is actually well-defined letting the amount of diamond in a history be computed), and a utility function over these states. The bridge transform extracts which computations are relevant for the behavior of physics in the hypothesis (which may or may not include the diamond-relevant computations), and then you just check whether the computations for a particular diamond-containing history are instantiated, take the best-case one, and that gives you your loss. 


Human preferences seem not substantially different from maximizing
diamonds. Most of the things we value are defined in terms of people
and social interactions. So, we can imagine a formal loss function
defined in terms of some "theory of people" implicit in our brain.

# 4. Translating Cartesian to Physicalist

Physicalism allows us to do away with the cartesian boundary between
the agent and its environment. But what if there *is* a cartesian
boundary? An important sanity test is demonstrating that our theory
can account for purely cartesian hypotheses as a *special case*.

## Ordinary laws

We will say "law" to mean what was [previously](https://www.alignmentforum.org/s/CmrW8fCmSLK7E25sa/p/GS5P7LLLbSSExb3Sk)
called "belief function". We start from examining a causal law
$W:\A^{\H}\rightarrow\Box\D$. I.e., it maps a policy to an ultradistribution over destinies, in a way that looks like a closed convex set of environments. Let $\Pro:=\H$ and $\Sigma=\A$
so that $\Gamma$ becomes the set of policies (we identify the history $h\in\H$
with the computation $\ulcorner G(h)\urcorner\in\Pro$ given by what the agent's source does on input $h$.). In particular, for now there
is no computer, or the computer can only simulate the agent itself.
Let $\Phi:=\D$. This is our set of "physics outcomes", complete action-observation sequences. Then, $W$ induces the physicalist hypothesis $\Theta_{W}\in\Box(\Gamma\times\Phi)$
by $\Theta_{W}:=\top_{\Gamma}\ltimes W$. $\Theta_{W}$ is what we
called the ["first-person static view"](https://www.lesswrong.com/s/CmrW8fCmSLK7E25sa/p/GS5P7LLLbSSExb3Sk) of the law. Total uncertainty over the choice of policy, interacting with the kernel which maps a policy to the resulting uncertainty over the environment.

We want to find the bridge transform of $\Theta_{W}$ and evaluate
the corresponding counterfactuals, in order to show that the resulting
$\LPol L$ is the same as in the cartesian setting. In order to do
this, we will use some new definitions.

**Definition 4.1:** *Let $X,Y$ be finite sets and $Q\subseteq Y\times X$
a relation. A *polycontribution on $Q$* is $\varpi:X\rightarrow\R_{+}$
s.t. for any $y\in Y$:*

$$
\sum_{x\in Q(y)}\varpi(x)\leq1
$$

*$\varpi$ is a *polydistribution* when for any $y\in Y$*

$$
\sum_{x\in Q(y)}\varpi(x)=1
$$

$$\newcommand{\PCon}[2]{\Delta_{#1}^{c}#2}$$

*We denote the space of polycontributions by $\PCon QX$ and the
space of polydistributions by $\Delta_{Q}X$.*

Here, $Q(y)$ stands for $\{x\in X\mid yQx\}$. Polycontributions and polydistributions are essentially a measure on a space, along with a family of subsets indexed by the $y\in Y$, s.t. restricting to any particular subset makes a contribution/distribution, respectively.

**Definition 4.2:** *Let $X$ and $Y$ be finite sets and
$Q\subseteq Y\times X$ a relation. A *polycontribution kernel*
(PoCK) over $Q$ is $\kappa:Y\rightarrow\Con X$ s.t. there is $\varpi\in\PCon QX$
s.t. for any $x\in X$ and $y\in Y$*

$$
\kappa(y;x)=\chi_{Q(y)}(x)\varpi(x)
$$

Or, in other words, a PoCK is the kernel that you get by taking a polycontribution and using the input point $y$ to decide which $Y$-indexed subset to restrict the measure to.
Let $Q_{0}\subseteq\A^{\H}\times\D$ be defined by $Q_{0}:=\{(\pi,a_{0}o_{0}\ldots a_{N-1})\mid\forall i<N:a_{i}=\pi(a_{0}\ldots o_{i-1})\}$. This is the "destiny is consistent with the policy" relation where the policy produces the actions that the destiny says it does, when placed in that situation.
Let $\phi:\A^{\H}\rightarrow\Delta\D$ be a Bayesian causal law (an
environment). Then, $\phi$ is a PoCK over $Q_{0}$! Conversely, any $\phi:\A^{\H}\rightarrow\Delta\D$ which is a PoCK
is an environment. 

Essentially, for an environment, each choice of policy results in a different probability distribution over outcomes. However, these resulting probability distributions over outcomes are all coherent with each other where the different policies agree on what to do, so an alternate view of an environment is one where there's one big measure over destinies (where the measure on a particular destiny is the probability of getting that destiny if the policy plays along by picking the appropriate actions) and the initial choice of policy restricts to a particular subset of the space of destinies, yielding a  probability distributions over destinies compatible with a given policy.

Clearly, the next step is to generalize this beyond probability distributions.

**Definition 4.3:** *Let $X$ and $Y$ be finite sets $Q\subseteq Y\times X$
a relation. A *polyultracontribution kernel* (PUCK) over $Q$
is $\Xi:Y\rightarrow\HUC X$ s.t. there is $\Pi\subseteq Y\rightarrow\Con X$
a set of PoCKs s.t. for any $y\in Y$, $\Xi(y)$ is the closure of
the convex hull of $\bigcup_{\kappa\in\Pi}\kappa(y)$.*

In particular, $W$ is a PUCK over $Q_{0}$. Conversely, any $W:\A^{\H}\rightarrow\Box\D$
which is a PUCK is a causal law. 

**Proposition 4.1:** *Consider some $\Gamma$, $\Phi$,
a relation $Q\subseteq\Gamma\times\Phi$ and a PUCK $\Xi$ over $Q$.
Let $\Theta:=\top_{\Gamma}\ltimes\Xi$. Then,*

$$
\B(\Theta)=[\top_{\Gamma}\ltimes(\Sus{\Theta}\rtimes\Xi)]^{\downarrow}=[\top_{\Gamma}\ltimes(Q^{-1}\rtimes\Xi)]^{\downarrow}
$$

Here, $Q^{-1}(x):=\{y\in Y\mid yQx\}$. The first identity says that
the inequality of Proposition 2.7 is saturated.

Specifically, as shown in the next proposition, for our case of cartesian causal laws, the computations realized according to
$\B(\Theta_{W})$ are exactly the histories that occur.

**Corollary 4.3:** *Suppose that for any $h\in\D$ and $\pi:\H\rightarrow\A$
s.t. $h\in\Sup W(\pi)$, it holds that $h\C\pi$. That is, the observations
$W$ predicts to receive from the computer are consistent with the
chosen policy. Let $L:\D\rightarrow\R$ be a cartesian loss function
and $\pi:\H\rightarrow\A$ a policy. Then,*

$$
(\Pr{\P{\Gamma}}\B(\Theta_{W})\cap\Co{\pi}{\text{fair}})(\Ph L)=W(\pi;L)
$$

Notice that the assumption $W$ is causal combined with the assumption
$W$ is consistent with $\C$ imply that it's not possible to use
the computer to simulate the agent on any history other than factual
past histories. For pseudocausal $W$ the bridge transform no longer
agrees with the cartesian setting: if a selfish agent is simulated in some
counterfactual, then the computationalist stance implies that it
can incur the loss for that counterfactual. This artifact should be avoidable for non-selfish agents, but we leave this for future articles.

## Turing laws

Now let's examine a cartesian setting which involves a general computer.
Without further assumptions, we again cannot expect the physicalist
loss to agree with the cartesian, since the programs running on the
computer might be entangled with the agent. If the agent runs simulations
of itself then it should expect to experience those simulations (as
opposed to the cartesian view). Therefore, we will need a "no entanglement"
assumption.

Let $\Gamma:=\Gamma_{1}\times\A^{\H}$ for some $\Gamma_{1}$ (the
computations that don't involve the agent) and $\Phi$ is still $\D$.
We want a notion of "joint belief about the environment and $\Gamma_{1}$"
in the cartesian setting, which generalizes causal laws. We will call
it a "Turing law" or "Turing environment" in the Bayesian case.

For our particular application (though the theorems are more general), $Z$ will be generally taken to be the space of computations that aren't about what you personally do, $Y$ is the space of computations about what you personally do, and $X$ is the space of destinies/physics outcomes, if the reader wants to try their hand at interpreting some of the more abstract theorems.

**Definition 4.4:** *Let $X$, $Y$ and $Z$ be finite sets
and $Q\subseteq Y\times X$ a relation. A *$Z$-polycontribution
on $Q$* is $\varpi:X\times Z\rightarrow\R_{+}$ s.t.*

$$
\sum_{z\in Z}\max_{y\in Y}\sum_{x\in Q(y)}\varpi(x,z)\leq1
$$

*$\varpi$ is a *$Z$-polydistribution on $Q$* when it is both
a $Z$-polycontribution on $Q$ and a polydistribution on $Q\times Z$. Here, we regard $Q\times Z$ as a relation between $Y$ and $X\times Z$.*

*We denote the space of $Z$-polycontributions by $\PCon Q{XZ}$
and the space of $Z$-polydistributions by $\Delta_{Q}XZ$.*

Notice that a $Z$-polycontribution on $Q$ is in particular a polycontribution
on $Q\times Z$: $\PCon Q{XZ}\subseteq\PCon{Q\times Z}{(X\times Z)}$.

**Definition 4.5:** *Let $X$, $Y$, $Z$ be finite sets
and $Q\subseteq Y\times X$ a relation. A *$Z$-polycontribution
kernel* ($Z$-PoCK) over $Q$ is $\kappa:Y\rightarrow\Con{(X\times Z)}$
s.t. there is $\varpi\in\PCon Q{XZ}$ s.t. for any $x\in X$, $y\in Y$
and $z\in Z$*

$$
\kappa(y;x,z)=\chi_{Q(y)}(x)\varpi(x,z)
$$

In the context of the identification we'll be using where $Z$ is the rest of math, $Y$ is the parts of math about what you personally do, and $X$ is the space of physics outcomes/destinies, this definition is saying that the Z-PoCK is mapping a policy to the joint distribution over math and destinies you'd get from restricting the measure to "I'm guaranteed to take this policy, regardless of how the rest of math behaves".

A Turing environment can be formalized as a $\Gamma_{1}$-PoCK $\kappa:\A^{\H}\rightarrow\Delta(\D\times\Gamma_{1})$
over $Q_{0}$. Indeed, in this case Definition 4.4 essentially says
that the environment can depend on a $\Gamma_{1}$ variable (Remember, $\Gamma_1$ is the rest of math), and also
we have a belief about this variable that doesn't depend on the policy.
Now, how to transform that into a physicalist hypothesis:

**Proposition 4.2:** *Let $X$, $Y$ and $Z$ be finite
sets, $Q\subseteq Y\times X$ a relation, $\kappa:Y\rightarrow\Con{(X\times Z)}$
a $Z$-PoCK over $Q$ and $\Theta\in\HUC Y$. Then, there exist $\mu\in\Con Z$
and $\phi:Z\times Y\rightarrow\Con X$ s.t. for all $z\in Z$, $\lambda y.\phi(z,y)$ is a PoCK over $Q$, and for all $y\in Y$,
$\kappa(y)=(\lambda z.\phi(z,y))\rtimes\mu$. Moreover, suppose that $(\mu_{1},\phi_{1})$
and $(\mu_{2},\phi_{2})$ are both as above[^9].
Then,*

$$
\mu_{1}\ltimes\Theta\ltimes\phi_{1}=\mu_{2}\ltimes\Theta\ltimes\phi_{2}
$$

Or, for a more concrete example, this is saying that it's possible to decompose the function mapping a policy to a distribution over the rest of math and the destiny, into two parts. One part is just a probability distribution over the rest of math, the second part is a function mapping a policy and the behavior of the rest of math to a probability distribution over destinies. Further, this decomposition is essentially unique, in that for any two ways you do it, if you take the probability distribution over the rest of math, have it interact with some arbitrary way of mapping the rest of math to policies, and then have that interact with the way to get a distribution over destinies, that resulting joint distribution could also be produced by any alternate way of doing the decomposition.

In the setting of Proposition 4.2, we define $\Theta*\kappa\in\HUC{(Z\times Y\times X)}$
by $\mu\ltimes\Theta\ltimes\phi$. The physicalist hypothesis corresponding
to Turing environment $\kappa$ turns out be $\top_{\A^{\H}}*\kappa$.
The no-entanglement assumption is evident in how the $\A^{\H}$ and
$\Gamma_{1}$ variables are "independent". Well, the actual choice of policy can depend on how the rest of math turns out, but the set of available choices of policy *doesn't* depend on how the rest of math turns out, and that's the sense in which there's independence.

To go from Turing environments to Turing laws, we will need the following generalization to sets of contributions.

**Definition 4.6:** *Let $X$, $Y$ and $Z$ be finite sets
$Q\subseteq Y\times X$ a relation. A *$Z$-polyultracontribution
kernel* ($Z$-PUCK) over $Q$ is $\Xi:Y\rightarrow\HUC{(X\times Z)}$
s.t. there is $\Pi\subseteq(\Con{(X\times Z))}^{Y}$ a set of $Z$-PoCKs
s.t. for any $y\in Y$, $\Xi(y)$ is the closure of the convex hull
of $\bigcup_{\kappa\in\Pi}\kappa(y)$.*

A Turing law is a $\Gamma_{1}$-PUCK $W:\A^{\H}\rightarrow\Box(\D\times\Gamma_{1})$
over $Q_{0}$.

**Definition 4.7:** *Let $X$, $Y$ and $Z$ be finite sets,
$Q\subseteq Y\times X$ a relation, $\Xi:Y\rightarrow\HUC{(X\times Z)}$
a $Z$-PUCK over $Q$ and $\Theta\in\HUC Y$. Let $\Pi\subseteq(\Con{(X\times Z))}^{Y}$
be the set of $Z$-PoCKs $\kappa$ s.t. for any $y\in Y$, $\kappa(y)\in\Xi(y)$.
Then, $\Theta*\Xi\in\HUC{(Z\times Y\times X)}$ is the closure of
the convex hull of $\bigcup_{\kappa\in\Pi}(\Theta*\kappa)$.*

As a sanity test for the sensibility of Definition 4.7, we have the following proposition. It demonstrates that instead of taking all compatible $Z$-PoCKs, we could take any subset whose convex hull is all of them. If that was not the case, it would undermine our justification for using convex sets (namely, that taking convex hull doesn't affect anything).

**Proposition 4.3:** *Let $X$, $Y$ and $Z$ be finite
sets, $Q\subseteq Y\times X$ a relation, $\kappa_{1},\kappa_{2}:Y\rightarrow\Con{(X\times Z)}$
$Z$-PoCKs over $Q$, $\Theta\in\HUC Y$ and $p\in[0,1]$. Then, $p\kappa_{1}+(1-p)\kappa_{2}$
is also a $Z$-PoCK over $Q$, and*

$$
\Theta*(p\kappa_{1}+(1-p)\kappa_{2})\subseteq p\Theta*\kappa_{1}+(1-p)\Theta*\kappa_{2}
$$

The physicalist hypothesis correspond to Turing law $W$ is $\Theta_{W}:=\top_{\A^{\H}}*W$. Total uncertainty over the available choices of policy interacting with the correspondence between the policy, the rest of math, and the resulting destiny.
Now, we need to study its bridge transform with a necessary intermediate result to proceed further, bounding the bridge transform result between an upper and lower bound:

**Proposition 4.4:** *Let $X$, $Y$ and $Z$ be finite
sets, $Q\subseteq Y\times X$ a relation and $\Xi:Y\rightarrow\HUC{(X\times Z)}$
a $Z$-PUCK over $Q$. Denote $\Theta:=\top_{Z\times Y}*\Xi$. Define
$\beta_{0},\beta_{1}:Z\times Y\times X\rightarrow2^{Z\times Y}$ by
$\beta_{0}(z,y,x):=\{z\}\times Q^{-1}(x)$, $\beta_{1}(z,y,x):=Z\times Q^{-1}(x)$.
Then,*

$$
(\Theta\ltimes\beta_{0})^{\downarrow}\subseteq\B(\Theta)\subseteq(\Theta\ltimes\beta_{1})^{\downarrow}
$$

Here, we slightly abuse notation by implicitly changing the order
of factors in $Z\times Y\times2^{Z\times Y}\times X$.

The application of this is that the resulting uncertainty-over-math/info about what computations are realized (i.e., the $\alpha$ set) attained when you do the bridge transform is bounded between two possible results. One bound is perfect knowledge of what the rest of math besides your policy is (everything's realized), paired with exactly as much policy uncertainty as is compatible with the destiny of interest. The other bound is no knowledge of the rest of math besides your policy (possibly because the policy was chosen without looking at other computations, and the rest of reality doesn't depend on those computations either), combined with exactly as much policy uncertainty as is compatible with the destiny of interest.

**Corollary 4.5:** *Suppose that for any $h\in\D$, $z\in\Gamma_{1}$
and $\pi:\H\rightarrow\A$ s.t. $(h,z)\in\Sup W(\pi)$, it holds that
$h\C z\pi$. That is, the observations $W$ predicts to receive from
the computer are consistent with the chosen policy and $W$'s beliefs
about computations. Let $L:\D\rightarrow\R$ be a cartesian loss function
and $\pi:\H\rightarrow\A$ a policy. Define $\tilde{L}:\D\times\Gamma_{1}\rightarrow\R$
by $\tilde{L}(h,z):=L(h)$. Then,*

$$
(\Pr{\P{\Gamma}}\B(\Theta_{W})\cap\Co{\pi}{fair})(\Ph L)=W(\pi;\tilde{L})
$$

And so, the Turing law case manages to add up to normality in the cartesian setting (with computer access) as well.

# 5. Discussion

## Summary

Let's recap how our formalism solves the problems stated in section
0:

* Physicalist hypotheses require no bridge rules, since they are
formulated in terms of a physical state space (that be chosen to be
natural/impartial) rather than agent's actions and observations. Therefore,
they accrue no associated description complexity penalty, and should
allow for a superior learning rate. Intuitively, a physicalist agent
knows it is an manifestation of the program $G$ rather than just
any object in the universe.

* A physicalist agent doesn't develop uncertainty after subjectively-simple
but objectively-complex events. It is easy to modify a cartesian hypothesis
such that the camera going black leads to completely new behavior.
However, it would require an extremely contrived (complex) modification
of *physics*, since the camera is not a simple object and has
no special fundamental significance. Basically, the simplest way that computations and reality correspond which instantiates the computations of the agent seeing black forever afterward gets a complexity boost by about the complexity of the agent's own source code.

* A physicalist agent doesn't regard its existence as axiomatic.
A given physicalist hypothesis might contain 0 copies of the agent,
1 copy or multiple copies (more generally, some ultradistribution
over the number of copies). Therefore, hypotheses that predict the
agent's existence with relatively high probability gain an advantage (i.e. influence the choice of policy more strongly),
as they intuitively should. At the same time, hypotheses that predict
too many copies of the agent to have predictive power are typically[^10]
also suppressed, since policy selection doesn't affect their loss
by much (whatever you do, everything happens anyway).

* Acausal attacks against physicalist agents are still *possible*,
but the lack of bridge rule penalty means the malign hypotheses are
not overwhelmingly likely compared to the true hypothesis. This should
allow defending using algorithms of "[consensus](https://arxiv.org/abs/1907.08461)"
[type](https://www.lesswrong.com/posts/CnruhwFGQBThvgJiX/formal-solution-to-the-inner-alignment-problem).

## What about Solomonoff's universality?

For selfish agents the physicalist framework can be reinterpreted
as a cartesian framework with a peculiar prior. Given that the Solomonoff
prior is *universal* (dominates every computable prior), does
it mean AIXI is roughly as good as a selfish physicalist agent? Not
really. The specification of that "physicalist prior" would involve
the *source code of the agent*. (And, if we want to handle hypotheses
that predict multiple copies, it would also involve the loss function.)
For AIXI, there is no such source code. More generally, the intelligence
of an agent w.r.t. the Solomonoff prior is upper bounded by its Kolmogorov
complexity (see [this](https://www.alignmentforum.org/posts/dPmmuaz9szk26BkmD/vanessa-kosoy-s-shortform?commentId=Tg7A7rSYQSZPASm9s)).
So, highly intelligent cartesian agents place small prior probability
on physicalism, and it would take such an agent a lot of data to learn
it.

Curiously, the converse is also true: the translation of section 4
also involves the source code of the agent, so intelligent physicalist
agents place small prior probability on cartesian dualism. We could
maybe use description complexity *relative* to the agent's own
source code, in order to define a prior[^11]
which makes dualism and physicalism roughly equivalent (for selfish
agents). This might be useful as a way to make cartesian algorithms
do physicalist reasoning. But in any case, you still need physicalism
for the *analysis*.

It would be interesting to do a similar analysis for bounded Solomonoff-like
priors, which might require understanding the computational complexity
of the bridge transform. In the bounded case, the Kolmogorov complexity of intelligent agents need not be high, but a similar penalty might arise from the size of the environment state space required for the physicalist-to-cartesian translation.

## Are manifest facts objective?

Physicalism is supposed to be "objective", i.e. avoid privileging
the observer. But, is the computationalist stance truly objective?
Is there an objective fact of the matter about which facts about computations
are physically manifest / which computations are realized?

At first glance it might seem there isn't. By Proposition 2.7, if
the physicalist hypothesis $\Theta\in\Box(\Gamma\times\Phi)$ is s.t.
$\Sup\Theta\subseteq F\times\Phi$ for some $F\subseteq\Gamma$, then
$\Sup\B(\Theta)\subseteq\P F\times\Phi$. So, if the hypothesis asserts
a fact $F$ then $F$ is always physically manifest. But, isn't $\Theta$
merely the *subjective* knowledge of the agent? And if so, doesn't
it follow that physical manifesting is also subjective?

At second glance things get more interesting. Suppose that agent $G$
learned hypothesis $\Theta$. Since $G$ is part of the physical universe,
it is now indeed manifest in the state of the physical universe that
hypothesis $\Theta$ (and in particular $F$) is true. Looking at
it from a different angle, suppose that a different agent $G'$ knows
that $G$ is a rational physicalist. Then it must agree that *if*
$G$ will learn $\Theta$ then $F$ is manifest. So, possibly there
is a sense in which manifesting is objective after all.

It would be interesting to try and build more formal theory around
this question. In particular, the ability of the user and the AI to agree on manifesting (and agree in general) seems important for alignment in general (since we want them to be minimizing the same loss) and for ruling out acausal attacks in particular (because we don't want the AI to believe in a hypothesis s.t. it wouldn't be rational for the user to believe in it).

## Physicalism and alignment

The domain of the physicalist loss function doesn't depend on the action and observation sets, which is an advantage for alignment since it can make it easier to "transfer" the loss function from the user to the AI when they have different action or observation sets.

On the other hand, the most objectionable feature of physicalism is the monotonicity
principle (section 3). The default conclusion is, we need to at least
generalize the formalism to allow for alignment. But, suppose we stick
with the monotonicity principle. How bad is the resulting misalignment?

Consider an alignment protocol in which the AI somehow learns the loss
function of the user. As a toy model, assume the user has a cellular
automaton loss function $L_{\text{user}}$, like in section 3. Assume
the AI acts according to the loss function $L_{\text{user}}^{\text{c-phys}}$
of Definition 3.3. What will result?

Arguably, in all probable scenarios the result will be pretty close
to optimal from the user's perspective. One type of scenario where
the AI makes the wrong choice is, when choosing between destruction
of everything and a future which is even *worse* than destruction
of everything. But, if these are the only available options then probably
something has gone terribly wrong at a previous point. Another type
of scenario where the AI possibly makes the wrong choice is, if it
creates multiple disconnected worlds each of which has valence from
the perspective of the user, at least one of which is good but some
of the which are bad. However, there usually seems to be no incentive
to do that[^12].

In addition, the problems might be milder in approaches that don't
require learning the user's loss function, such as IDA of imitation.

So, the monotonicity principle might not introduce egregious misalignment
in practice. However, this requires more careful analysis: there can
easily be failure modes we have overlooked.

## Future research directions

Here's a list of possible research directions, not necessarily comprehensive.
Some of them were already mentioned in the body of the article.

Directions which are already more or less formalized:

* Generalizing everything to infinite spaces (actually, we already
have considerable progress on this and hopefully will publish another
article soon). [**EDIT:** See also [profinite field spaces](https://www.lesswrong.com/posts/dPmmuaz9szk26BkmD/vanessa-kosoy-s-shortform#ETKbrbs9HiGosC3ho).]

* Generalizing section 4 to certain pseudocausal laws. Here, we
can make use of the monotonicity principle to argue that if Omega
discontinues all of its simulations, they correspond to copies which
are not the best-off and therefore don't matter in the loss calculus. Alternatively, we can design a non-selfish loss function that only counts the "baseline reality" copies.
Moreover, it would be interesting to understand the connection between
the pseudocausality condition and the "fairness" of Definition
1.5.

* Define a natural candidate for the Solomonoff prior for physicalist
hypotheses.

* Prove that the physicalist intelligence $\G$ is truly unbounded,
and study its further properties.

Directions which are mostly clear but require more formalization:

* Prove learning-theoretic results (regret bounds). For example,
if we assume that there is a low-cost experiment that reveals the
true hypothesis, then low regret should be achievable. In particular,
such theorems would allow validating Definition 1.5.

* Generalize section 3 to stochastic ontologies.

* Allow uncertainty about the loss function, and/or about the source
code. This would introduce another variable into our physicalist hypothesis,
and we need to understand how to bridge transform should act on it. [**EDIT**: Maybe we don't need uncertainty about source code, see [quining in IBP](https://www.lesswrong.com/posts/dPmmuaz9szk26BkmD/vanessa-kosoy-s-shortform?commentId=CxAqCuxDGt5JvtrBP).]

* Analyze the relationship between cartesian and physicalist agents
with bounded simplicity priors.

* For (cartesian) reinforcement learning, MDPs and POMDPs are classical
examples of environment for which various theoretical analysis is
possible. It would be interesting to come up with analogical examples
of physicalist hypotheses and study them. For example, we can consider
infra-Markov chains where the transition kernel depends on $\Gamma$.

Directions where it's not necessarily clear how to approach the problem:

* Understand the significance of the monotonicity principle better,
and whether there are interesting ways to avoid it. [**EDIT:** Looks like there [are](https://www.lesswrong.com/posts/DobZ62XMdiPigii9H/non-monotonic-infra-bayesian-physicalism).]

* Formally study the objectivity of manifesting, perhaps by deriving
some kind of agreement theorems for physicalist agents.

* Define and analyze physicalist alignment protocols. [**EDIT:** one important candidate is [Physicalist Superimitation](https://www.lesswrong.com/posts/ZwshvqiqCvXPsZEct/the-learning-theoretic-agenda-status-2023#Physicalist_Superimitation).]

* It is easy to imagine how a physicalist hypothesis can describe
a universe with *classical* physics. But, the real universe is
quantum. So, it doesn't really have a distribution over some state
space $\Phi$ but rather a density matrix on some Hilbert space. Is
there a natural way to deal with this in the present formalism? If
so, that would seem like a good solution to the interpretation of
quantum mechanics. [**EDIT**: Indeed, there is a [solution](https://www.alignmentforum.org/posts/GctPwQhrvwBpzyt9q/interpreting-quantum-mechanics-in-infra-bayesian-physicalism).]

[^1]: If we ignore ideas such as the von Neumann-Wigner
interpretation of quantum mechanics.

[^2]: This other direction also raises issues
with counterfactuals. These issues are also naturally resolved in
our formalism.

[^3]: The notation $\Theta\times\Lambda$ is
reserved for a different, commutative, operation which we will not
use here.

[^4]: A simplifying assumption we are planning
to drop in future articles.

[^5]: We have considered this type of setting
[before](https://www.alignmentforum.org/posts/3qXE6fK47JhSfkpnB/do-sufficiently-advanced-agents-use-logic#fEKc88NbDWZavkW9o)
with somewhat different motivation.

[^6]: The careful reader will observe that programs
sometimes don't halt which means that the "true" computational
universe is ill-defined. This turns out not to matter much.

[^7]: [Previously](https://www.alignmentforum.org/s/CmrW8fCmSLK7E25sa/p/idP5E5XhJGh9T5Yq9)
we defined pullback s.t. it can only be applied to particular infra/ultradistributions.
Here we avoid this limitation by using infra/ultracontributions as
the codomain.

[^8]: Meaning that, this rule is part of the
*definition* of the state rather than a claim about physical reality.

[^9]: Disintegrating a distribution into a semidirect
product yields a *unique* result, but for contributions that's
no longer true, since it's possible to move scalars between the two
factors. 

[^10]: One can engineer loss functions for which
they are not suppressed, for example if the loss only depends on the
actions and not on the observations. But such examples seem contrived.

[^11]: It's a self-referential definition, but
we can probably resolve the self-reference by quining.

[^12]: One setting in which there *is* an
incentive: Suppose there are multiple users and the AI is trying to
find a compromise between their preferences. Then, it might decide
to create disconnected worlds optimized for different users. But,
this solution might be much worse than the AI thinks, if Alice's world
contains Bob!atrocities.