# Optimization Amplifies

I talk here about how a mathematician mindset can be useful for AI alignment. But first, a puzzle:

Given < refresh to render LaTeX > , what is the least number < refresh to render LaTeX > such that for < refresh to render LaTeX > , the base < refresh to render LaTeX > representation of < refresh to render LaTeX > consists entirely of 0s and 1s?

If you want to think about it yourself, stop reading.

For < refresh to render LaTeX > =2, < refresh to render LaTeX > =2.

For < refresh to render LaTeX > =3, < refresh to render LaTeX > =3.

For < refresh to render LaTeX > =4, < refresh to render LaTeX > =4.

For < refresh to render LaTeX > =5, < refresh to render LaTeX > =82,000.

Indeed, 82,000 is 10100000001010000 in binary, 11011111001 in ternary, 110001100 in base 4, and 10111000 in base 5.

What about when < refresh to render LaTeX > =6?

So, a mathematician might tell you that this is an open problem. It is not known if there is any < refresh to render LaTeX > which consists of 0s and 1s in bases 2 through 6.

A scientist, on the other hand, might just tell you that clearly no such number exists. There are < refresh to render LaTeX > numbers that consist of < refresh to render LaTeX > 0s and 1s in base 6. Each of these has roughly < refresh to render LaTeX > digits in base 5, and assuming things are roughly evenly distributed, each of these digits is a 0 or a 1 with "probability" < refresh to render LaTeX > . The "probability" that there is any number of length < refresh to render LaTeX > that has the property is thus less than < refresh to render LaTeX > . This means that as you increase < refresh to render LaTeX > , the "probability" that you find a number with the property drops off exponentially, and this is not even considering bases 3 and 4. Also, we have checked all numbers up to 2000 digits. No number with this property exists.

Who is right?

Well, they are both right. If you want to have fun playing games with proofs, you can consider it an open problem and try to prove it. If you want to get the right answer, just listen to the scientist. If you have to choose between destroying the world with a 1% probability and destroying the world if a number greater than 2 which consists of 0s and 1s in bases 2 through 6 exists, go with the latter.

It is tempting to say that we might be in a situation similar to this. We need to figure out how to make safe AI, and we maybe don't have that much time. Maybe we need to run experiments, and figure out what is true about what we should do and not waste our time with math. Then why are the folks at MIRI doing all this pure math stuff, and why does CHAI talk about "proofs" of desired AI properties? It would seem that if the end of the world is at stake, we need scientists, not mathematicians.

I would agree with the above sentiment if we were averting an astroid, or a plague, or global warming, but I think it fails to apply to AI alignment. This is because optimization amplifies things.

As a simple example of optimization, let < refresh to render LaTeX > for < refresh to render LaTeX > be i.i.d. random numbers which are normally distributed with mean 0 and standard deviation 1. If I choose an < refresh to render LaTeX > at random, the probability that < refresh to render LaTeX > is greater than 4 is like 0.006%. However, if I optimize, and choose the greatest < refresh to render LaTeX > , the probability that it is greater that 4 is very close to 100%. This is the kind of thing that optimization does. It searches through a bunch of options, and takes extreme ones. This has the effect of making things that would be very small probabilities much larger.

Optimization also leads to very steep phase shifts, because it can send something on one side of a threshold to one extreme, and send things on the other side of a threshold to another extreme. Let < refresh to render LaTeX > for < refresh to render LaTeX > be i.i.d. random numbers that are uniform in the unit interval. If you look at the first 10 numbers and take the one that is furthest away from .499, the distribution over numbers will be bimodal peaks near 0 and 1. If you take the one that is furthest away from .501, you will get a very similar distribution. Now instead consider what happens if you look at all < refresh to render LaTeX > numbers and take the one that is furthest from .499. You will get a distribution that is almost certainly 1. On the other hand, the one that is furthest from .501 will be almost certainly 0. As you slightly change the optimization target, the result of a weak optimization might not change much, but the result of a strong one can change things drastically.

As a very rough approximation, a scientist is good at telling the difference between probability 0.01% and probability 99.99%, while the mathematician is good at telling the difference between 99.99% and 100%. Similarly, the scientist is good at telling if < refresh to render LaTeX > , while the mathematician is good at telling if < refresh to render LaTeX > when you already know that < refresh to render LaTeX > .

If you only want to get an approximately correct answer almost surely, the absence of strong optimization pressure makes the mathematician skills much less useful. However strong optimization pressure amplifies and creates discontinuities, which creates the necessity for a mathematician level of precision even to achieve approximate correctness in practice.

Notes:

1) I am not just saying that adversarial optimization makes small probabilities of failure large. I am saying that in general any optimization at all messes with small probabilities and errors drastically.

2) I am not saying that we don't need scientists. I am saying that we don't just need scientists, and I am saying that scientists should pay some attention to the mathematician mindset. There is a lot to be gained from getting your hands dirty in experiments.

3) I am not saying that we should only be satisfied if we achieve certainty that an AI system will be safe. That's an impossibly high standard. I am saying that we should aim for a deep formal understanding of what is going on, more like the "fully reduced" understanding we have of steam engines or rockets.