# Incorporating Mechanism Design Into Decision Theory

In the [previous post](https://www.lesswrong.com/posts/pxuvAMFbdGLmSL2oG/reframing-acausal-trolling-as-acausal-patronage), we looked at one way of handling externalities: letting other agents pay you to shift your decision. And we also considered the technique of aggregating those offers into an auction. This technique of "implementing a [mechanism](https://en.wikipedia.org/wiki/Mechanism_design) to handle an incentive misalignment" is extremely useful and seems like a promising avenue for future improvements to decision theory.

I want to frame mechanisms as "things that reshape incentives." [Auctions](https://en.wikipedia.org/wiki/Auction), [markets](https://en.wikipedia.org/wiki/Market_design), and [voting systems](https://en.wikipedia.org/wiki/Electoral_system) are all mechanisms; social technologies that can be invented by working backwards from a social welfare measure (a [social choice theory](https://en.wikipedia.org/wiki/Social_choice_theory)) and designing a [game](https://www.lesswrong.com/tag/game-theory) such that players following their individual incentives will find themselves in socially high-ranking [Nash equilibria](https://en.wikipedia.org/wiki/Nash_equilibrium).

I suspect that incorporating mechanism design more fully into decision theory will be extremely fruitful. Yudkowsky's probabilistic rejection algorithm[^2eff59ialjb] first identifies the socially optimal outcome (a fair Pareto optimum), and works backwards to identify a decision procedure which:

*   If [universalized](https://www.lesswrong.com/posts/R3pWxhWEDdr3YHZYs/a-benchmark-for-decision-theories) leads to the socially optimal outcome
*   If [best-responded](https://en.wikipedia.org/wiki/Best_response) to still leads to the socially optimal outcome (stabilizing it as a Nash equilibrium)

The [Robust Cooperation](https://www.lesswrong.com/posts/iQWk5jYeDg5ACCmpx/robust-cooperation-in-the-prisoner-s-dilemma) paper does the same thing for the Prisoners' Dilemma. Probabilistic rejection also only uses [appropriate-threats](https://www.lesswrong.com/posts/6dAK3sdRabNYNy5zJ/using-threats-to-achieve-socially-optimal-outcomes), like sometimes rejecting unfair offers even at cost to oneself, leading it to degrade gracefully when negotiators [disagree](https://www.lesswrong.com/tag/metaethics/) about what the socially optimal outcome is. The term "[non-credible threat](https://en.wikipedia.org/wiki/Non-credible_threat)" is named that way because a [classically-rational agent](https://en.wikipedia.org/wiki/Rational_agent) would never actually pay a cost "merely" for its incentive-reshaping effect on the other players. Not all non-credible threats are appropriate, but there are times when it's appropriate to pay costs to reshape the incentives of others.

Policy Counterfactuals
----------------------

There is a representation theorem by Joyce, which I understand to be something like "a Joyce-rational agent will choose an action $a$ that maximizes an expected utility expression that looks like $\mathcal{EU}(a) := \sum_{j=1}^{N} P(a \hookrightarrow o_j; \: x) \cdot \mathcal{U}(o_j)$." The [Functional Decision Theory (FDT)](https://www.lesswrong.com/tag/functional-decision-theory) [paper](https://arxiv.org/pdf/1710.05060.pdf) frames the differences between major decision theories like [CDT](https://www.lesswrong.com/tag/causal-decision-theory), [EDT](https://www.lesswrong.com/tag/evidential-decision-theory), and [FDT](https://www.lesswrong.com/tag/functional-decision-theory) as using different ways to compute the free parameter $P(a \hookrightarrow o_j; \: x)$. This parameter is an action-counterfactual, whose interpretation is something like "if I were to take this action $a$, what is the probability that outcome $o_j$ would occur, given some background knowledge $x$?"[^5lqqqbgohhf]

So we can look at "reshaping an agent's incentives" through the lens of "reshaping an agent's counterfactual expectations." We can think of other agents performing this scan over policies they could implement, looking for what will elicit the most desirable response from our decision theory. And the structure of our decision theory determines their counterfactual expectations about what those responses are.

Hardening Your Decision Theory
------------------------------

From this perspective, the assumption that other agents have [legible](https://en.wikipedia.org/wiki/Formal_verification) [open-source](https://www.lesswrong.com/tag/open-source-game-theory) access to our decision theory forces us to [harden](https://en.wikipedia.org/wiki/Hardening_(computing)) the [attack surface](https://en.wikipedia.org/wiki/Attack_surface) of our decision theory, rather than relying on [security through obscurity](https://en.wikipedia.org/wiki/Security_through_obscurity). 

Giving in to inappropriate-threats like [blackmail](https://www.lesswrong.com/posts/Ljy3CSwTFPEpnGLLJ/the-blackmail-equation) is a [software vulnerability](https://en.wikipedia.org/wiki/Vulnerability_(computing)) which another decision theory might [exploit](https://en.wikipedia.org/wiki/Exploit_(computer_security)). Similarly for [failing to make appropriate-threats](https://www.lesswrong.com/posts/6dAK3sdRabNYNy5zJ/using-threats-to-achieve-socially-optimal-outcomes), like accepting any positive offer in the Ultimatum game or unconditionally Cooperating in every round of an [infinitely-iterated Prisoners' Dilemma](https://en.wikipedia.org/wiki/Prisoner's_dilemma#The_iterated_prisoner's_dilemma). (We want our decision theory to implement a strategy more like [tit-for-tat](https://en.wikipedia.org/wiki/Tit_for_tat), which Cooperates conditional on reciprocal Cooperation, or some other compensation.)$$

We want to design our decision theory so that, to the greatest extent possible, other agents find that best-responding to our decision theory leads to high-ranking outcomes according to our [social choice theory](https://en.wikipedia.org/wiki/Social_choice_theory).

Mechanism Counterfactuals
-------------------------

Software systems can simulate any computable mechanism given enough computing power. This is a sufficient condition for such a mechanism to influence the behavior of software agents; such agents can perform a [logical handshake](https://www.lesswrong.com/posts/6hgmEvEdrf6SYBiqK/social-choice-theory-and-logical-handshakes) to act as if a mechanism "exists", even if it is not "really" implemented.

How do you act "as if" a voting system exists? By imagining how everyone would vote, if such a voting system existed, and then acting in accordance with the results. The same works for auctions, markets, negotiations, anything that reshapes incentives compared with the underlying strategic context. And these can be composed together into networks, like first imagining "as if" [private property](https://en.wikipedia.org/wiki/Private_property) ownership-tags exist, and then imagining "as if" there were a market for the goods on the underlying consensus-imaginary property rights layer.

### State Channels

One specific architecture for this sort of thing is [state channels](https://ethereum.org/en/developers/docs/scaling/state-channels): a relatively simple [smart contract](https://en.wikipedia.org/wiki/Smart_contract) serves as a dispute resolution mechanism for this whole scheme. Alice and Bob each deposit some funds with this smart contract, and then conduct most of their activity together without involving the [blockchain](https://en.wikipedia.org/wiki/Blockchain) at all. Alice and Bob exchange signed messages with each other, which enables them to prove the [authenticity](https://en.wikipedia.org/wiki/Message_authentication) of these messages to the dispute resolution smart contract.

One of the most valuable features of a state channel is that it shapes the counterfactual expectations of each participant, so that each can safely treat the interaction "as if" it was happening with all of the security guarantees of the underlying blockchain, without all of the [overhead](https://ethereum.org/en/developers/docs/gas). This includes being able to act "as if" further smart contracts had been deployed to the block chain, and these virtual smart contracts can be composed together into networks within the state channel.

At the end of their interaction, Alice and Bob can inform the smart contract of the result and withdraw whatever funds they're entitled to. Neither has an incentive to distort the report in their favor, because each has enough cryptographic information to prove the actual result in the event of a dispute.

### Logical Commitments

Software systems with legible access to each other's source code don't even need the overhead of a blockchain to hold a consensus model of other software systems in their heads. The [legibility makes this logical line-of-sight transitive](https://www.lesswrong.com/posts/M7jg8FiXHt5YpcW4D/legibility-makes-logical-line-of-sight-transitive); any software system that **AliceBot** reasons about, **BobBot** can also reason about.

A general-purpose technology we'll want for open-source game theory is a logical commitment. (Or just commitment when it's clear from context.) When **AliceBot** can implement any policy $\pi \in \Pi$, a logical commitment $\Phi \subseteq \Pi$ is the legible fact that **AliceBot** will only implement a policy $\phi \in \Phi$ from this subset. When $\Phi = \Pi$, this corresponds to the null commitment "I will implement a policy $\pi \in \Pi$." When $\Phi = \{\phi\}$, this corresponds to the very specific commitment "I will implement exactly the policy $\phi$."

We'll also want conditional commitments, which apply if some condition is true. **FairBot** offers the conditional commitment "if I can prove that you'll Cooperate with me, I'll Cooperate with you." It also offers the complementary commitment to cover the case where such a proof search fails: "In that case, I'll Defect." For any [domain](https://en.wikipedia.org/wiki/Domain_of_a_function) where a decision theory has a defined output, it is implicitly making commitments and conditional commitments.

Finally, a joint commitment $\mathbb \Phi \subseteq \mathbb \Pi$ is subset of a joint policy space $\mathbb \Pi$, and represents a commitment for each corresponding player. These can also be conditional.

In the [next post](https://www.lesswrong.com/posts/nb48zxPErAFWTaZFm/to-boldly-code) we'll see an example of how networks of counterfactual mechanisms can be used to produce useful logical commitments.

[^2eff59ialjb]: The algorithm is described in Project Lawful so spoilers but it's here, and it's discussed without spoilers earlier in this sequence. 

[^5lqqqbgohhf]: I assume the theorem still makes sense if you think of agents as optimizing their global policy rather than their local action after Bayesian updating, but I haven't been able to look at the original paper. It looks to be available behind a paywall here. But policy optimization currently seems like the obvious way to go so I'll go back to talking about policy counterfactuals.EDIT: gwern has put up a copy of the relevant chapter, see the discussion here for more details.