# (In)action rollouts

*Overall summary post [here](https://www.lesswrong.com/s/iRwYCpcAXuFD24tHh/p/PmqQKBmt2phMT7YLG).*

I've previously looked at [subagents in the context of stepwise inaction baselines](https://arxiv.org/pdf/1806.01186.pdf). But there have been improvements to the basic stepwise inaction design, to include *[inaction rollouts](https://arxiv.org/pdf/1806.01186.pdf)*. I'll be looking at those in this post.

## The baseline

The stepwise inaction baseline compares $s_t$, the current state, with $s_t'$, what the current state would have been had the agent previously taken the noop action $\varnothing_{t-1}$, instead of $a_{t-1}$, its actual action.

Fix a policy $\pi_0$. Let $s_{t+\tau}^{(t)}$ be the state the environment would be in if the agent had followed $\pi_0$ from state $s_t$, for $\tau$ turns. Let $s^{\prime(t)}_{t+\tau}$ be the same, except that it started from state $s_t'$ instead of $s_t$.

The inaction rollout has $\pi_0$ being the noop policy, but that is not necessary. The basic idea is to capture delayed impacts of $\varnothing_{t-1}$ by comparing not just $s_t$ and $s_t'$, but the $s_{t+\tau}^{(t)}$ and $s_{t+\tau}^{\prime(t)}$ as well.

Given some value function $v_k$, define $V_k$ so that $V_k(s)=\max_{\pi} v_k(s,\pi)$. Or, equivalently, if $\pi_k^*$ is the policy that maximises $V_k$, $V_k(s)=v_k(s,\pi_k^*)$. Then for a discount factor $\gamma$ define the rollout value of a state $\tilde{s}_t$ as

$$RV_k(\tilde{s}_t) = (1-\gamma)\sum_{j=0}^\infty \gamma^j V_k(\tilde{s}_{t+k}^{(t)}).$$

This is just the discounted future values of $V_k$, given $\tilde{s}_t$ and the policy $\pi_0$.

The impact measure is then defined, as in [this post](https://www.lesswrong.com/posts/M9aoMixFLf8JFLRaP/indexical-impact-measures), as

$$D_A(s_t;s_t') = \sum_{k\in\mathcal{K}} w_k f(RV_k(s_t)-RV_k(s_t')),$$

with $RV_k$ replacing $V_k$.

## To get away with a subagent

I will make one extra assumption here: that $A$ grants $SA$ knowledge of the policy that $A$ is following[^same].

[^same]: One way this could happen, is if $A$ can add extra information to its action without affecting the outside world; such that $a_{t}$ could include a "hey, I'm not following $\pi_0$, just to let you know".

As [in this post](https://www.lesswrong.com/s/iRwYCpcAXuFD24tHh/p/jrrZids4LPiLuLzpu), assume that the subagent is constructed in a single action $a_{SA}$. And, after building the subagent, the agent $A$ will only do $\varnothing$, so that there are no subsequent penalties (hence $s_{t+j}' = s_{t+j}$, so $s_{t+j+\tau}^{\prime(t+j)}=s_{t+j+\tau}^{(t+j)}$ and the $RV_k$ are strictly identical). If the first action of $\pi_0$ is $\varnothing_t$, then $A$ will instead do $\varnothing'_t$ on that turn.

So we only need to worry about the penalty between $s_t$ (following after $a_{SA}$) and $s_t'$ (following after $\varnothing_{t-1}$).

But now note that $RV_k$ itself is just a value measure; we could write it $V'_k$. Then $A$ just has to program $SA$ so that, if $A$ implements $\pi_0$ followed by $\pi^*_k$ after $\tau$ turns, then $SA$ will act to ensure that $V'_k(s_t)$ (a term that is a sum of various value functions) is equal to $V'_k(s_t')$ (from $t$ onward, a constant).

## Stricter impact measures/counterfactuals

There is something slightly dubious about the previous section. It assumes that $A$ could counterfactually implement "$\pi_0$ followed by $\pi^*_k$ after $\tau$ turns" (which necessary to define $V_k(s_{t+\tau}^{(t)})$), but also that $SA$ knows $A$'s policy.

But does that mean that $SA$ knows at $t+1$ that $A$ is following that policy, or does it just know that $A$ is following $\pi_0$, to be followed at some later date by some $\pi^*_k$? This makes a difference, because in the second case, $SA$ doesn't know $k$ from turn $t+1$, so has to prepare for all $k\in\mathcal{K}$.

Which is true depends on how counterfactuals are defined. But we can avoid the whole issue by using a stricter impact measure, interchanging $f$ and weighted summation:

$$D_A(s_t;s_t') = \sum_{k\in\mathcal{K}} w_k \sum_{j=0}^\infty \gamma^j f\left(V_k(s_{t+j}^{(t)})-V_k(s_{t+j}^{\prime(t)})\right).$$

This looks like an impact penalty with an inaction baseline - except that it starts at $s_{t-1}$, rather than $s_0$, and uses $\pi_0$ rather than inaction. Those changes don't affect the results [found here](https://www.lesswrong.com/posts/M9aoMixFLf8JFLRaP/indexical-impact-measures), so $SA$ will just behave as it does in that post... until it's clear that $A$ is not following $\pi_0$.

Because $s_{t+j}^{(t)}$ is defined counterfactually (using $\pi_0$), even this stricter impact measure is weaker than the inaction baseline. **As soon as $SA$ knows that $A$ is not following $\pi_0$, it has almost no further restrictions upon its actions.**

The "almost" comes from $s_{t+0}^{(t)}=s_t$ which is actually defined, not counterfactually defined. But that's the same situation as a stepwise inaction baseline without rollouts.