# Agent membranes/boundaries and formalizing “safety”

It might be possible to formalize what it means for an agent/moral patient to be **safe** via membranes/boundaries. This post tells one (*just one*)story for how the membranes idea could be useful for thinking about existential risk and AI safety.

Formalizing “safety” using agent membranes
==========================================

A few examples:

*   A bacterium uses its membrane to protect its internal processes from external influences.

![AI generated image: microscope view of a amoeba](https://lh7-us.googleusercontent.com/bdNIR08f1aDynLLLXFtS11LhYDqBfgDItx1ZfLd7SqjkMf1K_GGSM8jAWvzLg1WHdXkbDVxf6w5FUIKGwrbu47SZLyGuC-Jit2iZstIWyzsCGg0T9cFZCUrfwxUdvdEz92vcFo8wyQQe1UmHDjX69Ys)

*   A nation maintains its sovereignty by defending its borders.

![](https://lh7-us.googleusercontent.com/YNJKKupWLMkoAn_13od0FQ6WFAkmdfpc3LV0kzlNCFXJ8O2USMujcKJRGRfBJYr7FZg_LGBRoeZO9DjwM4-2IOhLqLazQvyKiAitBu3Ak9gPe41RNwYYu1F18FOZkbYARfg4vxDOnyS6d0Xy0e_z7us)

*   A human protects their mental integrity by selectively filtering the information that comes in and out of their mind.

![AI generated image: a brain surrounded by a fence](https://lh7-us.googleusercontent.com/d1pGJfqJcXsc_lJ52IOKL0ekNPxIqXsyZxpCFOI_Q9xI28FgOiLiTHj-X3ebMLYEoAoG-GOuwUF6sT7cxZKluNNspvnec7fUDjdIRysJuPkMk8kNLsZ-Xek_18mTRDFSCddI3s9gcz9e5EskqxQGTcc)

A natural abstraction for agent safety?
=======================================

Agent boundaries/membranes seem to be a natural abstraction representing the **safety and autonomy** of agents.

*   A bacterium survives only if its membrane is preserved.
*   A nation maintains its sovereignty only if its borders aren’t invaded.
*   A human mind maintains mental integrity only if it can hold off informational manipulation.

**Maybe the safety of agents could be largely formalized as** ***the preservation of their membranes**.*

Distinct from preferences!
==========================

Boundaries are also cool because they show **a way to** ***respect*** **agents without needing to talking about their preferences or utility functions**. Andrew Critch has said the following about this idea:

> my goal is to treat boundaries as more fundamental than preferences, rather than as merely a feature of them.  In other words, I think boundaries are probably better able to carve reality at the joints than either preferences or utility functions, for the purpose of creating a good working relationship between humanity and AI technology ([«Boundaries» Sequence, Part 3b](https://www.lesswrong.com/s/LWJsgNYE8wzv49yEc/p/SajYfrsoTHxiXPNtf))

For instance, respecting the boundary of bacterium would probably mean “preserving or not disrupting its membrane” (as opposed to knowing its preferences and satisfying them).

Protecting agents and infrastructure
====================================

By formalizing and preserving the important boundaries in the world, we could be in a better position to protect humanity from AI threats. Examples:

*   Critical computing infrastructure could be secured by creating strong boundaries around them. This can be enforced by cryptography and formal methods such that only the subprocesses that *need* to have read and/or write access to a particular resource (like memory) have the encryption keys to do so. 
    *   Related: [Object-capability model](https://en.wikipedia.org/wiki/Object-capability_model), [Principle of least privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege#:~:text=In%20information%20security%2C%20computer%20science,as%20a%20process%2C%20a%20user%2C), Evan Miyazono’s [Atlas Computing](https://atlascomputing.org/), Davidad’s [Open Agency Architecture](https://www.lesswrong.com/tag/open-agency-architecture).

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/97ad9288c35ee79e72eb116d4d8d15e56c44b7da2d7dfbd4.png)

*   It may be possible to specify or agree on a minimal “membrane” for each agent/moral patients humanity values, such that when each membrane is preserved, that agent largely stays safe and maintains its autonomy over the inside of its membrane. 
    *   If your physical boundary isn't violated, you don't die. If your mental boundary isn't violated aren't manipulated. Etc…
    *   Note: if your membrane is preserved, this *just* means that you stay safe and that you maintain autonomy over everything with your membrane. It does *not* necessarily mean that you get actively positive outcomes to occur in the outside world. This is all about *bare minimum safety*.
    *   See [this thread below and davidad's comment](https://www.lesswrong.com/posts/SuktPEmBGxPrnZRHd/agent-membranes-and-formalizing-safety?commentId=EKf8nKigmWtDumEEx)

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/SuktPEmBGxPrnZRHd/eaca0tuvng9dgbrvp6te)

*   Similarly, it may be possible to formalize and enforce the boundaries of physical property rights.

This is for safety, not full alignment
--------------------------------------

Note that this is only about specifying *safety*, not full alignment.

See: $$[Safety First: safety before full alignment. The deontic sufficiency hypothesis.](https://www.lesswrong.com/posts/dLbBiAkJFdWK3uu3u/safety-first-safety-before-full-alignment-the-deontic)

Caveats
-------

I don't think the absence of membrane piercing formalizes *all* of safety, but I think it gets at a good chunk of what "safety" should mean. Future thinking will have to determine what more is required. 

**What are examples of violations of agent safety that do not involve membrane piercing?** $$

Markov blankets
===============

How might membranes/boundaries be formalized mathematically? **Markov blankets** seem to be a fitting abstraction. 

Diagram:

![](https://lh7-us.googleusercontent.com/rhQlLwsitY6mStPDxeVyav1dRZnUmJWMSUS2L0bcaAOZorHeFWABEx4DiyOYPWMT7pIg4S4MLhJd1ZA9aJI3mF3bwhAT5Rh_nBHw6sMuYI3vy5ZeN1P0jNlzCXq1qbBN3buS7T_c-YR4N5XJJfUQASY)

Notice that there are no arrows directly between the agent and its environment. Ideally, all influence from one to the other flows through the boundary/membrane (e.g.: your skin).

In which case,

*   Infiltration of information across this Markov blanket measures membrane piercing, and low infiltration indicates the absence of such piercing.
*   (And it may also be useful to keep track of exfiltration across the Markov blanket?[^3messiaiwuq])

**For more details, see distillation**[** Formalizing**  **«Boundaries» with Markov blankets**](https://www.lesswrong.com/posts/z4o4iAFgnmaBmksN2/formalizing-boundaries-with-markov-blankets#Explaining_Markov_blankets)**. **

Also, there are probably other information-theoretic measures that are useful for formalizing membranes/boundaries.

Protecting agent membranes/boundaries
=====================================

**See:** [**Protecting agent boundaries**](https://www.lesswrong.com/posts/GZap3EfpdftKSanHK/protecting-agent-boundaries)**.**

* * *

**Subscribe** **to the**[** boundaries/membranes LessWrong tag**](https://www.lesswrong.com/tag/boundaries-technical) **to get notified of new developments.**

*Thanks to Jonathan Ng, Alexander Gietelink Oldenziel, Alex Zhu, and Evan Miyazono for reviewing a draft of this post.*

[^3messiaiwuq]: exfiltration, i.e.: privacy and the absence of mind-reading. But I need to think more about this. Related section: “Maintaining Boundaries is about Maintaining Free Will and Privacy” by Scott Garrabrant.