# To Boldly Code

In the [previous post](https://www.lesswrong.com/posts/bia3YMojTHKFGgep9/incorporating-mechanism-design-into-decision-theory), we looked at how networks of counterfactual mechanisms can be used to make logical commitments in [open source games](https://www.lesswrong.com/tag/open-source-game-theory). And this post goes through an example!

I Reject Your Incentives and Substitute My Own
----------------------------------------------

Alice and Bob are designing open-source programs to play the game $G$ on their behalf. Alice decides that a negotiation process is a useful mechanism to handle an incentive misalignment in $G$, and designs **AliceBot** to direct its [logical crystal ball](https://www.lesswrong.com/posts/DzfBgxoAC3kPir4PX/in-strategic-time-open-source-games-are-loopy) to a different game $N(G)$, which is a negotiation over [conditional commitments](https://www.lesswrong.com/posts/bia3YMojTHKFGgep9/incorporating-mechanism-design-into-decision-theory) for $G$.

Since **AliceBot** has **BobBot**'s source code, it simulates how a negotiation between them would go. This introduces new instances of **AliceBot** and **BobBot**, so let's call the instances that will actually implement any policy Implementers, and the instances which negotiate conditional commitments over those policies Negotiators.

Implementer **AliceBot** supplies both Negotiators with all the relevant information they need for this negotiation. Conveniently, this negotiation game includes a communication channel where Negotiators can send each other messages. This happens to make it a lot easier to coordinate than in the underlying game $G$.

Negotiator **AliceBot** sends over a preliminary message offering to negotiate over of conditional joint commitments $\mathbb \Phi \subseteq \mathbb \Pi$, and sends over the conditions that Implementer **AliceBot** will be verifying before implementing its part of $\mathbb \Phi$. In this negotiation protocol, it is now Negotiator **BobBot's** turn to reply with Implementer **BobBot's** or else terminate negotiations.

Transitive Logical Line-Of-Sight
--------------------------------

To understand how that instance of **BobBot** responds, we need to switch over to Bob's POV. Inconveniently, the [strong superrationality postulate](https://www.lesswrong.com/posts/R3pWxhWEDdr3YHZYs/a-benchmark-for-decision-theories#Superrationality) failed to hold: Bob reasoned differently than Alice about how to solve their incentive misalignment and designed **BobBot** to look first at some completely different mechanism-counterfactual game. That instance of **AliceBot** has enough information to know that its Implementer instance isn't going to condition its behavior on the output of this hypothetical game, and so helpfully terminates with an error message rather than make conditional commitments that its Implementer instance won't uphold.

Conveniently, **BobBot** was also designed with a fallback in this case, and launches a powerful [formal verification](https://en.wikipedia.org/wiki/Formal_verification) tool that quickly builds a model of **AliceBot** and the information flow between modules. This allows both instances of **BobBot** to work backwards and notice that Implementer **AliceBot** is conditioning its behavior on the output of some other counterfactual game.

Joint Policy Optimization
-------------------------

Negotiator **BobBot**, finding itself in that counterfactual, begins to study the structure of the negotiation protocol Alice designed and how this will supposedly influence the joint policy implemented in $G$. Negotiator **BobBot** can't discount the possibility that it was given incomplete or inaccurate information, but this negotiation protocol and **AliceBot** were both designed to be legible to analysis tools.

The joint policy space of $G$ is too large to [search exhaustively](https://en.wikipedia.org/wiki/Brute-force_search), but Alice used some clever insights into the structure of this space to [ignore large sub-spaces](https://www.lesswrong.com/s/sZyz9daQRoCLnxeoK/p/cLoxazqoGAPLjAhPL#Speeding_Up_Argmax) full of knowably-suboptimal joint policies. **AliceBot** and **BobBot** have common knowledge of their computational resources, and if they could communicate they could pass messages back and forth to [divide up the computational load](https://www.lesswrong.com/posts/pBkoHBYpvfouaMKEY/optimization-markets).

This protocol is instead being performed entirely acausally, which gives them a computational budget that looks like $min(A, B)$ rather than $A + B$. Which is a pain, but it lets Alice and Bob coordinate despite not being able to communicate.

Alice and Bob each supplied their delegates with their respective social choice theories, which score joint policies. **AliceBot** and **BobBot** negotiate at the level of social-utilities rather than individual-utilities, which doesn't eliminate all disagreements but tends to get closer to agreement.

**BobBot** responds with the three conditions that Implementer **BobBot** will verify before upholding its part of any joint commitment $\mathbb \Phi$:

*   Implementer **AliceBot** did indeed supply this instance of **BobBot** with all [material](https://en.wikipedia.org/wiki/Material) information
*   Implementer **AliceBot** will uphold its part of $\mathbb \Phi$
*   $\mathbb \Phi$ is fair according to $F_B$

$F_B$ is a fairness criterion that Bob thought of using a different perspective than Alice, and **BobBot** was designed without knowing what Alice's social choice theory would be. Perhaps Alice and Bob might agree at the [meta-level](https://www.lesswrong.com/tag/metaethics/) about the morally correct way to define a social choice theory, if given the chance to discuss the matter. But Alice and Bob aren't participants in this protocol, and they didn't design their delegates with any pathway towards this kind of convergence. Like thermostats or network routers, **AliceBot** and **BobBot** implement their decision theory without reflecting on how it might be improved.

Of course, Alice designed this negotiation protocol such that no participant could do better for themselves by adding more constraints. $F_B$ ends up being somewhat skewed in Bob's favor, from **AliceBot's** perspective. $G$ isn't an ultimatum game, but it would be the equivalent of saying that Alice should only get $48 when splitting $100.

Negotiator **AliceBot** has already sent over her own conditions, including how they [interact](https://www.lesswrong.com/posts/5bd75cc58225bf0670374f20/modal-bargaining-agents) with conditions like these. If Negotiator **BobBot** had capped Alice's gains at the equivalent of $50, his own gains would have likewise been capped at $50. Instead, each find their gains capped at the equivalent of $47.98. (The $48 that was effectively offered to Alice, minus a small penalty that each impose to give the other an incentive to converge at the meta-level.)

**AliceBot** and **BobBot** are programmed to accept losses like these as the price of incentivizing what Alice and Bob respectively see as the socially optimal outcome. They feel no temptation to [best-respond](https://www.lesswrong.com/posts/uguxceuv7ERp26kn9/best-responding-is-not-always-the-best-response) and abandon the [Schelling boundary](https://en.wikipedia.org/wiki/Focal_point_(game_theory)) that their creators tasked them with enforcing. The negotiation protocol returns a conditional joint commitment $\mathbb \Phi$ that Alice and Bob will each separately think achieves about 96% of the achievable social utility, had their decision theory been [universalized](https://www.lesswrong.com/posts/R3pWxhWEDdr3YHZYs/a-benchmark-for-decision-theories).

One level of simulation up, each Implementer checks the conditions that their Negotiator instances foresaw. All checks pass, and Implementer **AliceBot** and **BobBot** implement a joint policy from $\mathbb \Phi$ rather than $\mathbb \Pi$.

Robust Delegation
-----------------

Alice and Bob each see a message like the following, written using words and concepts native to each: 

> Logical Handshake Successful!
> 
> Enacting commitment.
> 
> Economic surplus generated: $959,608,188.31.
> 
> Warning: Economic efficiency at 96%. Click here for more details.

The first acausal treaty to be negotiated between their civilizations, but not the last.

[^ld3x6jny4gn]: Thanks to ChatGPT 3.5 for the title!