# An environment for studying counterfactuals

_Summary: I introduce a decision theory framework where successful agents are those with good counterfactuals._

Motivation
==========

The problem of logical counterfactuals is how to define probabilities < refresh to render LaTeX > when < refresh to render LaTeX > is known to be false. (I'll ignore more general counterfactuals < refresh to render LaTeX > in this post.)

The theory of logical induction provides a joint distribution over sentences, so the problem becomes: How do you condition on < refresh to render LaTeX > when < refresh to render LaTeX > has negligible probability?

Exploration tries to solve this by making sure that < refresh to render LaTeX > never has negligible probability. But it doesn't work in problems like Agent Simulates Predictor that contain predictors who can't tell when the agent explores.

A better solution is [early exploration](https://agentfoundations.org/item?id=1683), which uses an early stage of the logical inductor to do exploration. But then the later stages of the inductor know that < refresh to render LaTeX > is false, and we're back where we started.

I'm going to describe an environment that captures these features of the problem — it's got reflection, early exploration, counterfactuals, and a Bayesian update that stands in for the evolution of a logical inductor.

Informal definition
===================

The agent outputs counterfactual distributions < refresh to render LaTeX > . This determines an expected utility for each action. Most of the time, an action is chosen for the agent that maximizes this expected utility. But a small fraction of the time, an exploration action is chosen instead.

The agent receives an observation < refresh to render LaTeX > as input, from which it can infer whether exploration will occur. The agent also receives a prior < refresh to render LaTeX > as input, and this prior accurately reflects the behavior of the agent as a function of < refresh to render LaTeX > and < refresh to render LaTeX > . (This uses a fixed-point theorem.)

If action < refresh to render LaTeX > is chosen, then the counterfactual < refresh to render LaTeX > is factual; the rest are counterfactual. We judge an agent according to how accurate its factual counterfactual is, in addition to how much utility it gets.

Here's an agent that does okay in this environment: It adopts < refresh to render LaTeX > as its epistemic state and ignores < refresh to render LaTeX > . Because of exploration, it can compute counterfactuals by conditioning. This agent does okay but not great, since it ignores < refresh to render LaTeX > .

You could try to make a better agent as follows: Adopt < refresh to render LaTeX > as a prior and then do a Bayesian update on < refresh to render LaTeX > . But now you've inferred whether exploration occurs, so some actions have probability zero, and it's not clear how to compute counterfactuals.

If you find a good agent for this environment, you'll probably have learned something about making good counterfactuals.

Formal definition
=================

  

![](https://i.imgur.com/2k0iIMU.png)

  

A decision problem consists of a tuple of random variables:

*   < refresh to render LaTeX > is a finite set of actions.
*   < refresh to render LaTeX > determines whether the agent explores. If < refresh to render LaTeX > , no exploration takes place.
*   < refresh to render LaTeX > is a finite set of observations.
*   < refresh to render LaTeX > is the space of utilities.
*   < refresh to render LaTeX > is the space of distributions over the above variables.
*   < refresh to render LaTeX > represents an infinite source of random bits that the agent can use.
*   < refresh to render LaTeX > is the agent's output, representing a counterfactual distribution < refresh to render LaTeX > for each action.

along with some likelihoods:

*   < refresh to render LaTeX > for all < refresh to render LaTeX > , and < refresh to render LaTeX > , for some choice of < refresh to render LaTeX > .
*   Likelihoods < refresh to render LaTeX > , depending on the problem.
*   i.i.d. uniform distributions on each bit of < refresh to render LaTeX > .
*   < refresh to render LaTeX > is mostly determined by < refresh to render LaTeX > and < refresh to render LaTeX > as follows: If < refresh to render LaTeX > , then < refresh to render LaTeX > . (If there is a tie, < refresh to render LaTeX > is undetermined.) Otherwise, < refresh to render LaTeX > .
*   A distribution over < refresh to render LaTeX > conditional on each value of < refresh to render LaTeX > and < refresh to render LaTeX > , depending on the problem.

An agent is a function < refresh to render LaTeX > . A decision and an agent together _almost_ determine a joint distribution over all the variables. What's missing is < refresh to render LaTeX > and tiebreakers for < refresh to render LaTeX > . These are determined by finding a fixed point satisfying:

*   If < refresh to render LaTeX > is the resulting marginal distribution over < refresh to render LaTeX > , then < refresh to render LaTeX > .
*   For each < refresh to render LaTeX > and < refresh to render LaTeX > , there is a distribution over the set < refresh to render LaTeX > such that < refresh to render LaTeX > is sampled from that distribution.

(I might prove the existence of a fixed point in a comment.)

We'll informally say that an agent does well on a decision problem if, for every fixed point, the following are true:

*   < refresh to render LaTeX > is high.
*   The factual counterfactual is accurate — say, it's close to the marginal over < refresh to render LaTeX > conditional on the true action in total variation distance: < refresh to render LaTeX > .

Future work
===========

I have an idea for defining an optimal agent for every decision problem in this family; I'll explore that in another post.

Once we find a general solution, we'd ideally transfer it to the setting of logical induction, and then we'd have logical counterfactuals.