# Proof Section to Formalizing Newcombian Problems with Fuzzy Infra-Bayesianism

This proof section accompanies [Formalizing Newcombian problems with fuzzy infra-Bayesianism](https://www.lesswrong.com/posts/E6HwiG6TaZ338ub3t/formalizing-newcombian-problems-with-fuzzy-infra-bayesianism). We prove the following result. 

> **Theorem** \[Alexander Appel ([@Diffractor](https://www.lesswrong.com/users/diffractor?mention=user)), Vanessa Kosoy ([@Vanessa Kosoy](https://www.lesswrong.com/users/vanessa-kosoy?mention=user))\]:
> 
> *Let *$\nu: \Pi_H \times O^{<H}\to \Delta O$* be a Newcombian problem of horizon *$H \in \mathbb{N}$ *that satisfies pseudocausality. Let *$M_\nu = (S, \Theta_0, A, O, T, B)$* denote the associated supra-POMDP with infinite time horizon and time discount *$\gamma \in [0, 1).$* Then *
> 
> $$
> \lim_{\gamma \to 1} |\min_{\pi \in \Pi} \mathbb{E}_{M_\nu^\pi}[L^\gamma] - \min_{\pi \in \Pi} \mathbb{E}_{\nu^\pi}[L^\gamma]| = 0.
> $$
> 
> $$*Furthermore, if *$\{\pi_\gamma\}_{\gamma \in [0, 1)}$ *is a family of policies such that  *$\lim_{\gamma \to 1} \mathbb{E}_{M_\nu^{\pi_\gamma}}[L^\gamma] - \min_{\pi \in \Pi} \mathbb{E}_{M_\nu^\pi}[L^\gamma] = 0,$* then*
> 
> $$
> \lim_{\gamma \to 1} \mathbb{E}_{\nu^{\pi_\gamma}}[L^\gamma] - \min_{\pi \in \Pi} \mathbb{E}_{\nu^{\pi}}[L^\gamma] = 0.
> $$

*Proof:* Let $\lambda$ denote the empty history. Given a supracontribution $\Theta$, let $\max(\Theta)$ denote the set of maximal extreme points of $\Theta.$ First we remark that for any supra-POMDP,  without loss of generality, a set of copolicies $$$\Xi$ can always be replaced by

$$
\{\tau \in \Xi  |  \tau(\lambda) \in \max(\Theta_0), \forall h \neq \lambda \in (S \times A)^*, \tau(h) \in \max(T(s, a))\}.
$$

Given an episode policy $\pi \in \Pi_H,$ let $\tau_\pi$ denote the episode copolicy that initializes the state to $(\pi, \lambda),$ i.e. $\tau_\pi(\lambda) = \delta_{\pi \times \lambda}.$ Let $\tau_\pi^\pi \in \Delta (A \times O)^H$ denote the distribution over outcomes determined by the interaction of $\pi$ and $\tau_\pi.$ Note that the expected loss with respect to $\tau_\pi^\pi$ is equal to the expected loss for the Newcombian problem, i.e. 

$$
\mathbb{E}_{\tau_\pi^\pi}[L] = \mathbb{E}_{\nu^\pi}[L].
$$

Recall that throughout this sequence, we assume that $O$ is finite. By the remark at the beginning of the proof, the expected loss in one episode for the corresponding supra-POMDP can be written as a maximum expected loss over a finite set of $M_\nu$-copolicies $\tau.$ Namely, 

$$
\mathbb{E}_{M_\nu^\pi}[L] = \max_{\tau} \mathbb{E}_{\tau^\pi}[L].
$$

Then

$$
\max_{\tau} \mathbb{E}_{\tau^\pi}[L] \geq \mathbb{E}_{\tau_\pi^\pi}[L] = \mathbb{E}_{\nu^\pi}[L],
$$

and thus for any episode policy $\pi \in \Pi_H,$

$$
\mathbb{E}_{M_\nu^\pi}[L] \geq \mathbb{E}_{\nu^\pi}[L].
$$

We now extend this analysis to the optimal loss over $n$ episodes for $n \in \mathbb{N}.$[^7ts4xpkm5qh] Let $L^*:= \min_{\pi \in \Pi_H} \mathbb{E}_{\nu^\pi}[L]$ denote the episode optimal loss for $\nu.$ Let $\pi_n \in \Pi$ be an arbitrary policy for $n$ episodes of $M_\nu.$ Then as before,

$$
\mathbb{E}_{M_\nu^{\pi_n}}[L_n^\gamma] = \max_{\tau_n} \mathbb{E}_{{\tau_n}^{\pi_n}} [L_n^\gamma]
$$

where the maximum is over a finite set of $n$-episode copolicies $\tau_n.$ By the single episode case,

$$
\max_{\tau_n} \mathbb{E}_{{\tau_n}^{\pi_n}} [L_n^\gamma] \geq \left(1-\gamma \right) \sum_{k = 0}^{n -1} \gamma^k L^* = L^*,
$$

and thus $\min_{\pi_n \in \Pi}\mathbb{E}_{M_\nu^{\pi_n}}[L_n^\gamma] \geq L^*.$

It remains to show that the opposite inequality holds in the many-episode and $\gamma \to 1$ limit.

Recall that given $\pi \in \Pi_H,$ we define

$$
C_\pi:=\{h \in (A \times O)^H  |  \forall \text{ prefixes } h'a \in (A \times O)^* \times A \text{ of } h, a = \pi(h') \}.
$$

Recall that since $\nu$ satisfies pseudocausality, there exists a $\nu$-optimal policy $\pi^* \in \Pi_H$ such that for all $\pi \in \Pi_H,$ if $\text{supp}(\nu^{\pi}) \subseteq C_{\pi^*},$ then $\pi$ is also optimal for $\nu.$ Consequently, for any episode copolicy $\tau,$ either $\mathbb{E}_{\tau^{\pi^*}}[L] \leq L^*$ or $\mathbb{E}_{\tau^{\pi^*}}[1] < 1.$ To see this, suppose there exists an episode copolicy $\tau$ such that $\mathbb{E}_{\tau^{\pi^*}}[1] = 1.$ Then there exists a policy $\pi^\dagger$ such that $\text{supp}(\nu^{\pi^\dagger}) \subseteq C_{\pi^*}$ and $\mathbb{E}_{\tau^{\pi^*}}[L] = \mathbb{E}_{\nu^{\pi^\dagger}}[L]$. By pseudocausality, $\mathbb{E}_{\nu^{\pi^\dagger}}[L] = L^*.$ Thus $\mathbb{E}_{\tau^{\pi^*}}[L] \leq L^*.$

Define

$$
\alpha:=\max_{\tau: \mathbb{E}_{\tau^{\pi^*}}[L]> L^*} \mathbb{E}_{\tau^{\pi^*}}[1].
$$

$$By the remark at the beginning of the proof, the relevant set of copolicies in the definition of $\alpha$ is finite, and thus $\alpha$ is well-defined. If $\mathbb{E}_{\tau^{\pi^*}}[L] > L^*$ then $\mathbb{E}_{\tau^{\pi^*}}[1] < 1.$ Thus $\alpha \in [0, 1).$

Consider the iterated Newcombian problem over $n$ episodes. Let $\pi_n^*$ denote the multi-episode policy such that $\pi_n^*$ restricted to every episode is $\pi^*.$ Let $\tau_n$ denote an arbitrary copolicy that interacts with $\pi_n^*.$ Furthermore, let $m \leq n$ denote the number of episodes for which the episode-restriction $\tau$ of $\tau_n$ interacting with $\pi_n^*$ satisfies $\mathbb{E}_{\tau^{\pi^*}}[L] > L^*.$[^vx4si012l5d]

We have

$$
\mathbb{E}_{\tau_n^{\pi_n^*}}[L_n^\gamma] \leq \mathbb{E}_{\tau_n^{\pi_n^*}}[1] \leq \alpha^m.
$$

Furthermore,

$$
\begin{align} \mathbb{E}_{\tau_n^{\pi_n^*}}[L_n^\gamma] & \leq (1-\gamma)\left(\sum_{t = 0}^{m-1} \gamma^t \cdot 1 + \sum_{t = m}^{n - 1}\gamma^t \cdot L^*\right)\ & = (1-\gamma) \left( \frac{1-\gamma^m}{1-\gamma} + L^*\gamma^m \cdot \frac{1-\gamma^{n-m}}{1-\gamma} \right) \ &= 1 - \gamma^m + L^*(\gamma^m - \gamma^{n}). \end{align}
$$

We leave it to the reader to verify that

$$
\lim_{\gamma \to 1}\lim_{n \to \infty} \mathbb{E}_{\tau_n^{\pi_n^*}}[L_n^\gamma] \leq \lim_{\gamma \to 1} \lim_{n \to \infty}\max_{m \leq n} \min(\alpha^m, 1 - \gamma^m + L^*(\gamma^m - \gamma^{n})) = L^*. \square
$$

[^7ts4xpkm5qh]: Recall that if \(n \in \mathbb{N}\) and \(h \in (A \times O)^{nH}\) is given by \(h = a_0 o_0 \dots a_{nH - 1}o_{nH-1},\) then the loss over \(n\) episodes with geometric time discount \(\gamma \in [0, 1)\) is defined by\[L_n^\gamma(h) = (1 - \gamma) \sum_{k = 0}^{n-1}\gamma^kL(a_{kH} o_{kH} \cdots a_{kH + H - 1} o_{kH + H - 1}).\] 

[^vx4si012l5d]: A copolicy can depend on the past, meaning it can depend on the policy. Thus \(m\) can depend on \(\pi_n^*\).