# An Aphoristic Overview of Technical AI Alignment proposals

Alignment Aphorisms
===================

Most alignment overviews are too long, but what if we rewrote one as a series of aphorisms?

I like Epictetus's confronting style: abrasive, clarifying. See [my fuller post](https://www.lesswrong.com/posts/EjsceYeeKEMoAohMs/wassname-s-shortform?commentId=FB6nH6h9t6woPeodR) for links and nuance.

* * *

I.
--

Some problems can be solved by being smarter.   
Some problems can only be solved by having help.   
Aligning something smarter than you is the second kind.

So many proposals collapse to:   
*use AI to help supervise AI.*   
It sounds too simple. It's the only thing that scales.

* * *

II. On using AI to supervise AI
-------------------------------

A teacher can grade a student slightly smarter than herself.   
But not much smarter - unless she has tricks.

How does weak supervise strong?   
Hard problems decompose.   
Supervise each step; align the whole.   
This is [Iterated Amplification](https://www.lesswrong.com/tag/iterated-amplification).

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/831dc41d6a2bd5b667636483e6cc8130d31706662fa21241.png)

If two models argue, a weaker one can judge:   
judging is easier than creating.   
This is [Debate](https://www.lesswrong.com/tag/debate-ai-safety-technique-1).

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/1a373d86cebdfa56d2338462c057b8df31c1ce2eaddf3ea8.jpg)

Give the model principles and let it judge its own actions:   
principles generalize where labels cannot.  
This is [Constitutional AI](https://www.lesswrong.com/tag/constitutional-ai). 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/07eca807046d6979cafd18a9000846fd87618d8bb42086bb.jpg)

They already know what's good.   
Weak teaching can unlock it.   
This is [Weak-to-Strong Generalization](https://openai.com/index/weak-to-strong-generalization/).

* * *

III. On not building one big AI
-------------------------------

A corporation can be smarter than any employee.   
Yet no employee optimizes the world.

Many bounded tools, no unified goal.   
No unified agent; no unified misalignment.   
This is [CAIS](https://www.alignmentforum.org/posts/5hApNw5f7uG8RXxGS/the-open-agency-model).

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/3aff010329763ff27087897f1c5384b9da17f2162b8b7f79.png)

* * *

IV. On making the objective less wrong
--------------------------------------

Make the AI uncertain about what we want.   
Then it must ask.   
Uncertainty makes cooperation optimal.   
This is [CIRL](https://www.lesswrong.com/posts/qdcfTWbeSvcBRgCjs/is-cirl-a-promising-agenda).

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/142c830cbf27ab785185aa4f53c9ec8f477cb7b976301af2.jpg)

If it can't be satisfied, it won't stop.   
Give it a notion of "enough", and it can rest.   
A satisficer has no incentive to rewrite itself.   
This is [satisficing](https://www.lesswrong.com/tag/mild-optimization).

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/cca7eaced297272ec3a234e893b6a0c5a16bd24e8de07794.jpg)

* * *

V. On control tools
-------------------

Align them to want what we want.   
Control will catch when they don't.   
Both are needed. The weaker model needs eyes inside.

If the model lies, look past the output.   
A weaker model can supervise a stronger one - if it can read its mind.   
This is [ELK](https://www.lesswrong.com/tag/eliciting-latent-knowledge).

Like us, models say one thing but value another.   
Prompting shapes words. Steering shapes thoughts.   
This is [steering](https://www.lesswrong.com/tag/activation-engineering).

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/a46c2facb166db1ef012544aa0c831d9516c01e5710b507f.jpg)

When models outsmart the weaker one,   
their words can deceive.   
See the reasons, not the results. Even the hidden thoughts.   
This is [interpretability](https://www.lesswrong.com/tag/interpretability-ml-and-ai).

* * *

VI. On understanding what we build
----------------------------------

Tricks that work now might break at scale.   
Understanding doesn't.

What is an agent? What is optimization?   
If you can't define it, you can't align it.   
This is [agent foundations](https://www.lesswrong.com/tag/agent-foundations).

* * *

VII. On older proposals
-----------------------

Keep it in a box. ([AI boxing](https://www.lesswrong.com/w/boxed-ai))   
But boxes leak.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/a14424a5aa6fb02d7a7e32d7f2839403b034d6cd55f277f7.jpg)

Let it answer, not act. ([Oracle AI](https://www.lesswrong.com/tag/oracle-ai))   
But answers shape action.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/145e4d3b7ce01d0fc43262e27ced8351600826fe4eaecffe.jpg)

Do what we'd want if we were wiser. ([CEV](https://www.lesswrong.com/tag/coherent-extrapolated-volition))   
But wiser toward what?

* * *

VIII. Beyond alignment *(bonus)*
--------------------------------

The optimist case: AIs are white boxes. Brains are not.   
Values are simple. They're learned early.   
This is the [optimist case](https://optimists.ai/2023/11/28/ai-is-easy-to-control/).

* * *

IX. On who aligns the aligners *(bonus)*
----------------------------------------

Suppose we solve alignment perfectly.   
Aligned to whom?

A safe AI in the wrong hands is still a problem.   
This is [governance risk](https://arxiv.org/abs/2306.12001).

* * *

*Rewritten from my original draft with Claude. Compressed from* [*Shallow review of live agendas*](https://www.lesswrong.com/posts/zaaGsFBeDTpCsYHef/shallow-review-of-live-agendas-in-alignment-and-safety) *and others.*