# Probability and radical uncertainty

In the [previous article](/lw/igv/probability_knowledge_and_metaprobability/) in this sequence, I conducted a thought experiment in which simple probability was not sufficient to choose how to act. Rationality required reasoning about _meta-probabilities_, the probabilities of probabilities.

Relatedly, lukeprog has [a brief post](/lw/h78/estimate_stability/) that explains how this matters; [a long article](/lw/745/why_we_cant_take_expected_value_estimates/) by HoldenKarnofsky makes meta-probability  central to utilitarian estimates of the effectiveness of charitable giving; and Jonathan_Lee, in [a reply](/lw/hnf/model_stability_in_intervention_assessment/) to that, has used the same framework I presented.

In my previous article, I ran thought experiments that presented you with various colored boxes you could put coins in, gambling with uncertain odds.

The last box I showed you was blue. I explained that it had a fixed but unknown probability of a twofold payout, uniformly distributed between 0 and 0.9. The overall probability of a payout was 0.45, so the expectation value for gambling was 0.9—a bad bet. Yet your optimal strategy was to gamble a bit to figure out whether the odds were good or bad.

Let’s continue the experiment. I hand you a black box, shaped rather differently from the others. Its sealed faceplate is carved with runic inscriptions and eldritch figures. “I find this one _particularly_ interesting,” I say.

What is the payout probability? What is your optimal strategy?

In the framework of the previous article, you have no knowledge about the insides of the box. So, as with the “sportsball” case I analyzed there, your meta-probability curve is flat from 0 to 1.

The blue box also has a flat meta-probability curve; but these two cases are very different. For the blue box, you know that the curve _really is_ flat. For the black box, you have no clue what the shape of even the meta-probability curve is.

The relationship between the blue and black boxes is the same as that between the coin flip and sportsball—except at the meta level!

So if we’re going on in this style, we need to look at the distribution of _probabilities of probabilities of probabilities_. The blue box has a sharp peak in its meta-meta-probability (around flatness), whereas the black box has a flat meta-meta-probability.

You ought now to be a little uneasy. We are [putting epicycles on epicycles](http://en.wikipedia.org/wiki/Epicycle#Epicycles). An infinite regress threatens.

Maybe at this point you suddenly reconsider the blue box… I _told_ you that its meta-probability was uniform. But perhaps I was lying! How reliable do you think I am?

Let’s say you think there’s a 0.8 probability that I told the truth. That’s the meta-meta-probability of a flat meta-probability. In the _worst_ case, the actual payout probability is 0, so the average _just plain probability_ is 0.8 x 0.45 = 0.36. You can feed that worst case into your decision analysis. It won’t drastically change the optimal policy; you’ll just quit a bit earlier than if you were entirely confident that the meta-probability distribution was uniform.

To get this really right, you ought to make a best guess at the meta-meta-probability _curve_. It’s not just 0.8 of a uniform probability distribution, and 0.2 of zero payout. That’s the _worst_ case. Even if I’m lying, I might give you better than zero odds. How much better? What’s your confidence in your meta-meta-probability curve? Ought you to draw a meta-meta-meta-probability curve? Yikes!

Meanwhile… that black box is _rather sinister_. Seeing it makes you wonder. What if I rigged the blue box so there is a small probability that when you put a coin in, it jabs you with a poison dart, and you die horribly?

Apparently a zero payout is _not_ the worst case, after all! On the other hand, this seems paranoid. [I’m odd](http://buddhism-for-vampires.com/dark-culture), but probably not _that_ evil.

Still, what about the black box? You realize now that it could do _anything_.

*   It might spring open to reveal a collection of fossil trilobites.
*   It might play Corvus Corax’s _[Vitium in Opere](http://www.youtube.com/watch?v=jgBvEVy__qs)_ at ear-splitting volume.
*   It might analyze the trace DNA you left on the coin and use it to write you a _personalized_ love poem.
*   It might emit a strip of paper with a recipe for dundun noodles written in Chinese.
*   It might sprout six mechanical legs and jump into your lap.

What is the probability of its giving you $2?

That no longer seems quite so relevant. In fact… it might be utterly meaningless! This is now a situation of **radical uncertainty**.

What is your optimal strategy?

I’ll answer that later in this sequence. You might like to figure it out for yourself now, though.

Further reading
---------------

The black box is an instance of [Knightian uncertainty](http://en.wikipedia.org/wiki/Knightian_uncertainty). That’s a catch-all category for any type of uncertainty that can’t usefully be modeled in terms of probability (_or_ meta-probability!), because you can’t make meaningful probability estimates. Calling it “Knightian” doesn’t help solve the problem, because there’s lots of sources of non-probabilistic uncertainty. However, it’s useful to know that there’s a literature on this.

The blue box is closely related to [Ellsberg’s paradox](http://en.wikipedia.org/wiki/Ellsberg_paradox), which combines probability with Knightian uncertainty. Interestingly, it was invented by the same Daniel Ellsberg who released the Pentagon Papers in 1971. I wonder how his work in decision theory might have affected his decision to leak the Papers?