Likelihood functions, p-values, and the replication crisis

__Or:  Switching From Reporting p-values to Reporting Likelihood Functions Might Help Fix the Replication Crisis:  A personal view by Eliezer Yudkowsky.__

_Disclaimers:_

- _This dialogue was written by a [Bayesian](https://arbital.com/p/1r8).  The voice of the Scientist in the dialogue below may fail to pass the [Ideological Turing Test](https://en.wikipedia.org/wiki/Ideological_Turing_Test) for frequentism, that is, it may fail to do justice to frequentist arguments and counterarguments._
- _It does not seem sociologically realistic, to the author, that the proposal below could be adopted by the scientific community at large within the next 10 years. It seemed worth writing down nevertheless._

_If you don't already know Bayes' rule, check out Arbital's [Guide to Bayes' Rule](https://arbital.com/p/1zq) if confused._

----------

**Moderator:**  Hello, everyone.  I'm here today with the **Scientist,** a working experimentalist in... chemical psychology, or something; with the **Bayesian,** who's going to explain why, on their view, we can make progress on the replication crisis by replacing p-values with some sort of Bayesian thing--

**Undergrad:**  Sorry, can you repeat that?

**Moderator:**  And finally, the **Confused Undergrad** on my right.  **Bayesian,** would you care to start by explaining the rough idea?

**Bayesian:**  Well, the rough idea is something like this.  Suppose we flip a possibly-unfair coin six times, and observe the sequence HHHHHT.  Should we be suspicious that the coin is biased?

**Scientist:**  No.

**Bayesian:**  This isn't a literal coin.  Let's say we present a series of experimental subjects with two cookies on a plate, one with green sprinkles and one with red sprinkles.  The first five people took cookies with green sprinkles and the sixth person took a cookie with red sprinkles. %note: And they all saw separate plates, on a table in the waiting room marked "please take only one" so nobody knew what was being tested, and none of them saw the others' cookie choices.%  Do we think most people prefer green-sprinkled cookies or do we think it was just random?

**Undergrad:**  I think I would be *suspicious* that maybe people liked green sprinkles better.  Or at least that the sort of people who go to the university and get used as test subjects like green sprinkles better.  Yes, even if I just saw that happen in the first six cases.  But I'm guessing I'm going to get dumped-on for that.

**Scientist:**  I think I would be genuinely not-yet-suspicious.  There's just too much stuff that looks good after N=6 that doesn't pan out with N=60.

**Bayesian:**  I'd at least strongly suspect that people in the test population *don't* mostly prefer red sprinkles.  But the reason I introduced this example is as an oversimplified example of how current scientific statistics calculate so-called "p-values", and what a Bayesian sees as the central problem with that.

**Scientist:**  And we can't use a more realistic example with 30 subjects?

**Bayesian:**  That would not be nice to the Confused Undergrad.

**Undergrad:**  *Seconded.*

**Bayesian:**  So:  Heads, heads, heads, heads, heads, tails.  I ask: is this "statistically significant", as current conventional statisticians would have the phrase?

**Scientist:**  I reply: no.  On the null hypothesis that the coin is fair, or analogously that people have no strong preference between green and red sprinkles, we should expect to see a result as extreme as this in 14 out of 64 cases.

**Undergrad:**  Okay, just to make sure I have this straight:  That's because we're considering results like HHHTHH or TTTTTT to be equally or more extreme, and there are 14 total possible cases like that, and we flipped the coin 6 times which gives us $2^6 = 64$ possible results.  14/64 = 22%, which is not less than 5%, so this is not statistically significant at the $p<0.05$ level.

**Scientist:**  That's right.  However, I'd also like to observe as a matter of practice that even if you get HHHHHH on your first six flips, I don't advise stopping there and sending in a paper where you claim that the coin is biased towards heads.

**Bayesian:**  Because if you can decide to *stop* flipping the coin at a time of your choice, then we have to ask "How likely it is that you can find some place to stop flipping the coin where it looks like there's a significant number of heads?"  That's a whole different kettle of fish according to the p-value concept.

**Scientist:**  I was just thinking that N=6 is not a good number of experimental subjects when it comes to testing cookie preferences.  But yes, that too.

**Undergrad:**  Uh... why does it make a difference if I can decide when to stop flipping the coin?

**Bayesian:**  What an excellent question.

**Scientist:**  Well, this is where the concept of p-values is less straightforward than plugging the numbers into a statistics package and believing whatever the stats package says.  If you previously decided to flip exactly six coins, and then stop, regardless of what results you got, then you would get a result as extreme as "HHHHHH" or "TTTTTT" 2/64 of the time, or 3.1%, so p<0.05.  However, suppose that instead you are a bad fraudulent scientist, or maybe just an ignorant undergraduate who doesn't realize what they're doing wrong.  Instead of picking the number of flips in advance, you keep flipping the coin until the statistics software package tells you that you got a result that *would have been* statistically significant *if,* contrary to the actual facts of the case, you'd decided *in advance* to flip the coin exactly that many times.  But you didn't decide in advance to flip the coin that many times.  You decided it after checking the actual results.  Which you are not allowed to do.

**Undergrad:**  I've heard that before, but I'm not sure I understand on a gut level why it's bad for me to decide when I've collected enough data.

**Scientist:**  What we're trying to do here is set up a test that the null hypothesis cannot pass--to make sure that where there's no fire, there's unlikely to be smoke.  We want a complete experimental process which is unlikely to generate a "statistically significant" discovery if there's no real phenomenon being investigated.  If you flip the coin exactly six times, if you decide that in advance, then you are less than 5% likely to get a result as extreme as "six heads" or "six tails".  If you flip the coin repeatedly, and check *repeatedly* for a result that *would* have had p<0.05 if you'd decided in advance to flip the coin exactly that many times, your chance of getting a nod from the statistics package is *much greater* than 5%.  You're carrying out a process which is much more than 5% likely to yell "smoke" in the absence of fire.

**Bayesian:**  The way I like to explain the problem is like this:  Suppose you flip a coin six times and get HHHHHT.  If you, in the secret depths of your mind, in your inner heart that no other mortal knows, decided to flip the coin exactly six times and then stop, this result is not statistically significant; p=0.22.  If you decided in the secret depths of your heart, that you were going to *keep flipping the coin until it came up tails,* then the result HHHHHT *is* statistically significant with p=0.03, because the chance of a fair coin requiring you to wait six or more flips to get one tail is 1/32.

**Undergrad:**  What.

**Scientist:**  It's a bit of a parody, obviously--nobody would really decide to flip a coin until they got one tail, then stop--but the Bayesian is technically correct about how the rules for p-values work.  What we're asking is how rare our outcome is, within the class of outcomes we *could* have gotten.  The person who keeps flipping the coin until they get one tail has possible outcomes {T, HT, HHT, HHHT, HHHHT, HHHHHT, HHHHHHT...} and so on.  The class of outcomes where you get to the sixth round or later is the class of outcomes {HHHHHT, HHHHHHT...} and so on, a set of outcomes which collectively have a probability of 1/64 + 1/128 + 1/256... = 1/32.  Whereas if you flip the coin exactly six times, your class of possible outcomes is {TTTTTT, TTTTTH, TTTTHT, TTTTHH...}, a set of 64 possibilities within which the outcome HHHHHT is something we could lump in with {HHHHTH, HHHTHH, THTTTT...} and so on.  So although it's counterintuitive, if we really had decided to run the first experiment, HHHHHT would be a statistically significant result that a fair coin would be unlikely to give us.  And if we'd decided to run the second experiment, HHHHHT would not be a statistically significant result because fair coins sometimes do something *like* that.

**Bayesian:**  And it doesn't bother you that the meaning of your experiment depends on your private state of mind?

**Scientist:**  It's an honor system.  Just like the process doesn't work if you lie about which coinflips you actually saw, the process doesn't work--is not a fair test in which non-fires are unlikely to generate smoke--if you lie about *which experiment you performed.*  You must honestly say the rules you followed to flip the coin the way you did.  Unfortunately, since what you were thinking about the experiment is less clearly visible than the actual coinflips, people are much more likely to twist *how they selected* their number of experimental subjects, or how they selected which tests to run on the data, than they are to tell a blatant lie about what the data said.  That's p-hacking.  There are, unfortunately, much subtler and less obvious ways of generating smoke without fire than claiming post facto to have followed the rule of flipping the coin until it came up tails.  It's a serious problem, and underpins some large part of the great replication crisis, nobody's sure exactly how much.

**Undergrad:**  That... sorta makes sense, maybe?  I'm guessing this is one of those cases where I have to work through a lot of example problems before it becomes obvious.

**Bayesian:**  No.

**Undergrad:**  No?

**Bayesian:**  You were right the first time, Undergrad.  If what the experimentalist is *thinking* has no causal impact on the coin, then the experimentalist's thoughts cannot possibly make any difference to what the coin is saying to us about Nature.  My dear Undergrad, you are being taught weird, ad-hoc, overcomplicated rules that aren't even internally consistent--rules that theoretically output *different* wrong answers depending on your private state of mind!  And *that* is a problem that runs far deeper into the replication crisis than people misreporting their inner thoughts.

**Scientist:**  A bold claim to say the least.  But don't be coy; tell us what you think we should be doing instead?

**Bayesian:**  I analyze as follows:  The exact result HHHHHT has a 1/64 or roughly 1.6% probability of being produced by a fair coin flipped six times.  To simplify matters, suppose we for some reason were already pondering the hypothesis that the coin was biased to produce 5/6 heads--again, this is an unrealistic example we can de-simplify later.  Then this hypothetical biased coin would have a $(5/6)^5 \cdot (1/6)^1 \approx 6.7\%$ probability of producing HHHHHT.  So between our two hypotheses "The coin is fair" and "The coin is biased to produce 5/6ths heads", our exact experimental result is *4.3 times more likely* to be observed in the second case.  HHHHHT is also 0.01% likely to be produced if the coin is biased to produce 1/6th heads and 5/6ths tails, so we've already seen some quite strong evidence *against* that particular hypothesis, if anyone was considering it.  The exact experimental outcome HHHHHT is 146 times as likely to be produced by a fair coin as by a coin biased to produce 1/6th heads. %note: Recall the Bayesian's earlier thought that, after seeing five subjects select green cookies followed by one subject selecting a red cookie, we'd already picked up strong evidence *against* the proposition: "Subjects in this experimental population lopsidedly prefer red cookies over green cookies."%

**Undergrad:**  Well, I think I can follow the calculation you just did, but I'm not clear on what that calculation means.

**Bayesian:**  I'll try to explain the meaning shortly, but first, note this.  That calculation we just did has *no dependency whatsoever* on *why* you flipped the coin six times.  You could have stopped after six because you thought you'd seen enough coinflips.  You could have done an extra coinflip after the first five because Namagiri Thayar spoke to you in a dream.  The coin doesn't care.  The coin isn't affected.  It remains true that the exact result HHHHHT is 23% as likely to be produced by a fair coin as by a biased coin that comes up heads 5/6ths of the time.

**Scientist:**  I agree that this is an interesting property of the calculation that you just did.  And then what?

**Bayesian:**  You report the results in a journal.  Preferably including the raw data so that others can calculate the likelihoods for any other hypotheses of interest.  Say somebody else suddenly becomes interested in the hypothesis "The coin is biased to produce 9/10ths heads."  Seeing HHHHHT is 5.9% likely in that case, so 88% as likely than if the coin is biased to produce 5/6ths heads (making the data 6.7% likely), or 3.7 times as likely than if the coin is fair (making the data 1.6% likely).  But you shouldn't have to think of all possible hypotheses in advance.  Just report the raw data so that others can calculate whatever likelihoods they need.  Since this calculation deals with the *exact* results we got, rather than summarizing it into some class or set of supposedly similar results, it puts a greater emphasis on reporting your exact experimental data to others.

**Scientist:**  Reporting raw data seems an important leg of a good strategy for fighting the replication crisis, on this we agree.  I nonetheless don't understand what experimentalists are supposed to *do* with this "X is Q times as likely as Y" stuff.

**Undergrad:**  Seconded.

**Bayesian:**  Okay, so... this isn't trivial to describe without making you run through [a whole introduction to Bayes' rule](https://arbital.com/p/1zq)--

**Undergrad:**  Great.  Just what I need, another weird complicated 4-credit course on statistics.

**Bayesian:**  It's literally [a 1-hour read if you're good at math](https://arbital.com/p/693). It just isn't literally *trivial* to understand with no prior introduction.  Well, even with no introduction whatsoever, I may be able to fake it with statements that will *sound* like they might be reasonable--and the reasoning *is* valid, it just might not be obvious that it is.  Anyway.  It is a theorem of probability that the following is valid reasoning:

*(the Bayesian takes a breath)*

**Bayesian:**  Suppose that Professor Plum and Miss Scarlet are two suspects in a murder.  Based on their prior criminal convictions, we start out thinking that Professor Plum is twice as likely to have committed the murder as Miss Scarlet.  We then discover that the victim was poisoned.  We think that, assuming he committed a murder, Professor Plum would be 10% likely to use poison; assuming Miss Scarlet committed a murder, she would be 60% likely to use poison.  So Professor Plum is around *one-sixth as likely* to use poison as Miss Scarlet.  Then after observing the victim was poisoned, we should update to think Plum is around one-third as likely to have committed the murder as Scarlet: $2 \times \frac{1}{6} = \frac{1}{3}.$

**Undergrad:**  Just to check, what do you mean by saying that "Professor Plum is one-third as likely to have committed the murder as Miss Scarlet"?

**Bayesian:**  I mean that if these two people are our only suspects, we think Professor Plum has a 1/4 probability of having committed the murder and Miss Scarlet has a 3/4 probability of being guilty.  So Professor Plum's probability of guilt is one-third that of Miss Scarlet's.

**Scientist:**  Now *I'd* like to know what you mean by saying that Professor Plum had a 1/4 probability of committing the murder.  Either Plum committed the murder or he didn't; we can't observe the murder be committed multiple times and Professor Plum doing it 1/4th of the time.

**Bayesian:**  Are we going there?  I guess we're going there.  My good Scientist, I mean that if you offered me either side of an even-money bet on whether Plum committed the murder, I'd bet that he didn't do it.  But if you offered me a gamble that costs \$1 if Professor Plum is innocent and pays out \$5 if he's guilty, I'd cheerfully accept that gamble.  We only ran the 2012 US Presidential Election one time, but that doesn't mean that on November 7th you should've refused a \$10 bet that paid out \$1000 if Obama won.  In general when prediction markets and large liquid betting pools put 60% betting odds on somebody winning the presidency, that outcome tends to happen 60% of the time; they are *well-calibrated* for probabilities in that range.  If they were systematically uncalibrated--if in general things happened 80% of the time when prediction markets said 60%--you could use that fact to pump money out of prediction markets.  And your pumping out that money would adjust the prediction-market prices until they were well-calibrated.  If things to which prediction markets assign 70% probability happen around 7 times out of 10, why insist for reasons of ideological purity that the probability statement is meaningless?

**Undergrad:**  I admit, that *sounds* to me like it makes sense, if it's not just the illusion of understanding due to my failing to grasp some far deeper debate.

**Bayesian:**  There is indeed a [deeper debate](https://arbital.com/p/4y9), but what the deeper debate works out to is that your illusion of understanding is pretty much accurate as illusions go.

**Scientist:**  Yeah, I'm going to want to come back to that issue later.  What if there are two agents who both seem 'well-calibrated' as you put it, but one agent says 60% and the other agent says 70%?

**Bayesian:**  If I flip a coin and don't look at it, so that I don't know yet if it came up heads or tails, then my ignorance about the coin isn't a fact about the coin, it's a fact about me.  Ignorance exists in the mind, not in the environment.  A blank map does not correspond to a blank territory.  If you peek at the coin and I don't, it's perfectly reasonable for the two of us to occupy different states of uncertainty about the coin.  And given that I'm not absolutely certain, I can and should quantify my uncertainty using probabilities.  There's like [300 different theorems](https://arbital.com/p/7ry) showing that I'll get into trouble if my state of subjective uncertainty *cannot* be viewed as a coherent probability distribution.  You kinda pick up on the trend after just the fourth time you see a slightly different clever proof that any violation of the standard probability axioms will cause the graves to vomit forth their dead, the seas to turn red as blood, and the skies to rain down dominated strategies and combinations of bets that produce certain losses--

**Scientist:**  Sorry, I shouldn't have said anything just then.  Let's come back to this later?  I'd rather hear first what you think we should do with the likelihoods once we have them.

**Bayesian:**  On the laws of probability theory, those likelihood functions *are* the evidence.  They are the objects that send our prior odds of 2 : 1 for Plum vs. Scarlet to posterior odds of 1 : 3 for Plum vs. Scarlet.  For any two hypotheses you care to name, if you tell me the relative likelihoods of the data given those hypotheses, I know how to update my beliefs.  If you change your beliefs in any other fashion, the skies shall rain dominated strategies etcetera.  Bayes' theorem:  It's not just a statistical method, it's the LAW.

**Undergrad:**  I'm sorry, I still don't understand.  Let's say we do an experiment and find data that's 6 times as likely if Professor Plum killed Mr. Boddy than if Miss Scarlet did.  Do we arrest Professor Plum?

**Scientist:**  My guess is that you're supposed to make up a 'prior probability' that sounds vaguely plausible, like 'a priori, I think Professor Plum is 20% likely to have killed Mr. Boddy'.  Then you combine that with your 6 : 1 likelihood ratio to get 3 : 2 posterior odds that Plum killed Mr. Boddy.  So your paper reports that you've established a 60% posterior probability that Professor Plum is guilty, and the legal process does whatever it does with that.

**Bayesian:**  *No.*  Dear God, no!  Is that really what people think Bayesianism is?

**Scientist:**  It's not?  I did always hear that the strength of Bayesianism is that it gives us posterior probabilities, which p-values don't actually do, and that the big weakness was that it got there by making up prior probabilities more or less out of thin air, which means that nobody will ever be able to agree on what the posteriors are.

**Bayesian:**  Science papers should report *likelihoods.*  Or rather, they should report the raw data and helpfully calculate some likelihood functions on it.  Not posteriors, never posteriors.

**Undergrad:**  What's a posterior?  I'm trusting both of you to avoid the obvious joke here.

**Bayesian:**  A [posterior probability](https://arbital.com/p/1rp) is when you say, "There's a 60% probability that Professor Plum killed Mr. Boddy."  Which, as the Scientist points out, is something you never get from p-values.  It's also something that, in my own opinion, should never be reported in an experimental paper, because it's not *the result of an experiment.*

**Undergrad:**  But... okay, Scientist, I'm asking you this one.  Suppose we see data statistically significant at p<0.01, something we're less than 1% probable to see if the null hypothesis "Professor Plum didn't kill Mr. Boddy" is true. Do we arrest him?

**Scientist:**  First of all, that's not a realistic null hypothesis.  A null hypothesis is something like "Nobody killed Mr. Boddy" or "All suspects are equally guilty."  But even if what you just said made sense, even if we could reject Professor Plum's innocence at p<0.01, you still can't say anything like, "It is 99% probable that Professor Plum is guilty."  That is just not what p-values mean.

**Undergrad:**  Then what *do* p-values mean?

**Scientist:**  They mean we saw data inside a preselected class of possible results, which class, as a whole, is less than 1% likely to be produced if the null hypothesis is true.  That's *all* that it means.  You can't go from there to "Professor Plum is 99% likely to be guilty," for reasons the Bayesian is probably better at explaining.  You can't go from there to anywhere that's someplace else.  What you heard is what there is.

**Undergrad:**  Now I'm doubly confused.  I don't understand what we're supposed to do with p-values *or* likelihood ratios.  What kind of experiment does it take to throw Professor Plum in prison?

**Scientist:**  Well, realistically, if you get a couple more experiments at different labs also saying p<0.01, Professor Plum *is* probably guilty.

**Bayesian:**  And the 'replication crisis' is that it turns out he's *not* guilty.

**Scientist:**  Pretty much.

**Undergrad:**  That's not exactly reassuring.

**Scientist:**  Experimental science is not for the weak of nerve.

**Undergrad:**  So... Bayesian, are you about to say similarly that once you get an extreme enough likelihood ratio, say, anything over 100 to 1, or something, you can probably take something as true?

**Bayesian:**  No, it's a bit more complicated than that.  Let's say I flip a coin 20 times and get HHHTHHHTHTHTTHHHTHHHTTHT.  Well, the hypothesis "This coin was rigged to produce exactly HHHTHHHTHTHTTHHHTHHHTTHT" has a likelihood advantage of roughly a million-to-one over the hypothesis "this is a fair coin".  On any reasonable system, unless you wrote down that single hypothesis in advance and handed it to me in an envelope and didn't write down any other hypotheses or hand out any other envelopes, we'd say the hypothesis "This coin was rigged to produce HHHTHHHTHTHTTHHHTHHHTTHT" has a complexity penalty of at *least* $2^{20} : 1$ because it takes 20 bits just to describe what the coin is rigged to do.  In other words, the penalty to prior plausibility more than cancels out a million-to-one likelihood advantage.  And that's just the start of the issues.  But, *with* that said, I think there's a pretty good chance you could do okay out of just winging it, once you understood in an intuitive and common-sense way how Bayes' rule worked.  If there's evidence pointing to Professor Plum with a likelihood of 1,000 : 1 over any other suspects you can think of, in a field that probably only contained six suspects to begin with, you can figure that the prior odds against Plum weren't much more extreme than 10 : 1 and that you can legitimately be at least 99% sure now.

**Scientist:**  But you say that this is *not* something you should report in the paper.

**Bayesian:**  That's right.  How can I put this... one of the great commandments of Bayesianism is that you ought to take into account *all* the relevant evidence you have available; you can't exclude some evidence from your calculation just because you don't like it.  Besides sounding like common sense, this is also a rule you have to follow to prevent your calculations from coming up with paradoxical results, and there are various particular problems where there's a seemingly crazy conclusion and the answer is, "Well, you *also* need to condition on blah blah blah."  My point being, how do I, as an experimentalist, know what *all the relevant evidence* is?  Who am I to calculate a posterior?  Maybe somebody else published a paper that includes more evidence, with more likelihoods to be taken into account, and I haven't heard about it yet, but somebody else has.  I just contribute my own data and its likelihood function--that's all!  It's not my place to claim that I've collected *all* the relevant evidence and can now calculate posterior odds, and even if I could, somebody else could publish another paper a week later and the posterior odds would change again.

**Undergrad:**  So, roughly your answer is, "An experimentalist just publishes the paper and calculates the likelihood thingies for that dataset, and then somebody outside has to figure out what to do with the likelihood thingies."

**Bayesian:**  Somebody outside has to set up priors--probably just reasonable-sounding ignorance priors, maximum entropy stuff or complexity-based penalties or whatever--then try to make sure they've collected all the evidence, apply the likelihood functions, check to see if the result [makes sense](https://arbital.com/p/227), etcetera.  And then they might have to revise that estimate if somebody publishes a new paper a week later--

**Undergrad:**  That sounds *awful.*

**Bayesian:**  It would be awful if we were doing meta-analyses of p-values.  Bayesian updates are a *hell* of a lot simpler!  Like, you literally just [multiply](https://arbital.com/p/1zg) the old posterior by the new likelihood [function](https://arbital.com/p/1zj) and normalize.  If experiment 1 has a likelihood ratio of 4 for hypothesis A over hypothesis B, and experiment 2 has a likelihood ratio of 9 for A over B, the two experiments together have a likelihood ratio of 36.

**Undergrad:**  And you can't do that with p-values, I mean, a p-value of 0.05 and a p-value of 0.01 don't multiply out to p<0.0005--

**Scientist:**  *No.*

**Bayesian:**  I should like to take this moment to call attention to my superior smile.

**Scientist:**  I am still worried about the part of this process where somebody gets to make up prior probabilities.

**Bayesian:**  Look, that just corresponds to the part of the process where somebody decides that, having seen 1 discovery and 2 replications with p<0.01, they are willing to buy the new pill or whatever.

**Scientist:**  So your reply there is, "It's subjective, but so is what you do when you make decisions based on having seen some experiments with p-values."  Hm.  I was going to say something like, "If I set up a rule that says I want data with p<0.001, there's no further objectivity beyond that," but I guess you'd say that my asking for p<0.001 instead of p<0.0001 corresponds to my pulling a prior out of my butt?

**Bayesian**:  Well, except that asking for a particular p-value is not actually as good as pulling a prior out of your butt.  One of the first of those 300 theorems proving *doom* if you violate probability axioms, was Abraham Wald's "complete class theorem" in 1947.  Wald set out to investigate all the possible *admissible strategies,* where a strategy is a way of acting differently based on whatever observations you make, and different actions get different payoffs in different possible worlds.  Wald termed an *admissible strategy* a strategy which was not dominated by some other strategy across all possible measures you could put on the possible worlds.  Wald found that the class of admissible strategies was simply the class that corresponded to having a probability distribution, doing Bayesian updating on observations, and maximizing expected payoff.

**Undergrad:**  Can you perhaps repeat that in slightly smaller words?

**Bayesian:**  If you want to do different things depending on what you observe, and get different payoffs depending on what the real facts are, *either* your strategy can be seen as having a probability distribution and doing Bayesian updating, *or* there's another strategy that does better given at least some possible measures on the worlds and never does worse.  So if you say anything as wild as "I'm waiting to see data with p<0.0001 to ban smoking," in principle there must be some way of saying something along the lines of, "I have a prior probability of 0.01% that smoking causes cancer, let's see those likelihood functions" which does at least as well or better no matter what anyone else would say as their own prior probabilities over the background facts.

**Scientist:** Huh.

**Bayesian:**  Indeed.  And that was when the Bayesian revolution very slowly started; it's sort of been gathering steam since then.  It's worth noting that Wald only proved his theorem a couple of decades after "p-values" were invented, which, from my perspective, helps explain how science got wedged into its peculiar current system.

**Scientist:**  So you think we should burn all p-values and switch to reporting all likelihood ratios all the time.

**Bayesian:**  In a word... yes.

**Scientist:**  I'm suspicious, in general, of one-size-fits-all solutions like that.  I suspect you--I hope this is not too horribly offensive--I suspect you of idealism.  In my experience, different people need different tools from the toolbox at different times, and it's not wise to throw out all the tools in your toolbox except one.

**Bayesian:**  Well, let's be clear where I am and amn't idealistic, then.  Likelihood functions cannot solve the entire replication crisis.  There are aspects of this that can't be solved by using better statistics.  Open access journals aren't something that hinge on p-values versus likelihood functions.  The broken system of peer commentary, presently in the form of peer review, is not something likelihood functions can solve.

**Scientist:**  But likelihood functions will solve everything else?

**Bayesian:**  No, but they'll at least *help* on a surprising amount.  Let me count the ways:

**Bayesian:**  One.  Likelihood functions don't distinguish between 'statistically significant' results and 'failed' replications.  There are no 'positive' and 'negative' results.  What used to be called the null hypothesis is now just another hypothesis, with nothing special about it.  If you flip a coin and get HHTHTTTHHH, you have not "failed to reject the null hypothesis with p<0.05" or "failed to replicate".  You have found experimental data that favors the fair-coin hypothesis over the 5/6ths-heads hypothesis with a likelihood ratio of 3.78 : 1.  This may help to fight the file-drawer effect--not entirely, because there is a mindset in the journals of 'positive' results and biased coins being more exciting than fair coins, and we need to tackle that mindset directly.  But the p-value system *encourages* that bad mindset.  That's why p-hacking even exists.  So switching to likelihoods won't fix everything right away, but it *sure will help.*

**Bayesian:**  Two.  The system of likelihoods makes the importance of raw data clearer and will encourage a system of publishing the raw data whenever possible, because Bayesian analyses center around the probability of the *exact* data we saw, given our various hypotheses.  The p-value system encourages you to think in terms of the data as being just one member of a class of 'equally extreme' results.  There's a mindset here of people hoarding their precious data, which is not purely a matter of statistics.  But the p-value system *encourages* that mindset by encouraging people to think of their result as part of some undistinguished class of 'equally or more extreme' values or whatever, and that its meaning is entirely contained in it being a 'positive' result that is 'statistically significant'.

**Bayesian:**  Three.  The probability-theoretic view, or Bayesian view, makes it clear that different effect sizes are different hypotheses, as they must be, because they assign different probabilities to the exact observations we see.  If one experiment finds a 'statistically significant' effect size of 0.4 and another experiment finds a 'statistically significant' effect size of 0.1 on whatever scale we're working in, the experiment *has not replicated* and we do not yet know what real state of affairs is generating our observations.  This directly fights and negates the 'amazing shrinking effect size' phenomenon that is part of the replication crisis.

**Bayesian:**  Four.  Working in likelihood functions makes it far easier to aggregate our data.  It even helps to [point up](https://arbital.com/p/227) when our data is being produced under inconsistent conditions or when the true hypothesis is not being considered, because in this case we will find likelihood functions that end up being nearly zero everywhere, or where the best available hypothesis is achieving a much lower likelihood on the combined data than that hypothesis [expects itself to achieve](https://arbital.com/p/227).  It is a stricter concept of replication that helps quickly point up when different experiments are being performed under different conditions and yielding results incompatible with a single consistent phenomenon.

**Bayesian:**  Five.  Likelihood functions are objective facts about the data which do not depend on your state of mind.  You cannot deceive somebody by reporting likelihood functions unless you are literally lying about the data or omitting data.  There's no equivalent of 'p-hacking'.

**Scientist:**  Okay, that last claim in particular strikes me as *very* suspicious.  What happens if I want to persuade you that a coin is biased towards heads, so I keep flipping it until I randomly get to a point where there's a predominance of heads, and then choose to stop?

**Bayesian:**  "Shrug," I say.  You can't mislead me by telling me what a real coin actually did.

**Scientist:**  I'm asking you what happens if I keep flipping the coin, checking the likelihood each time, until I see that the current statistics favor my pet theory, and then I stop.

**Bayesian:**  As a pure idealist seduced by the seductively pure idealism of probability theory, I say that so long as you present me with the true data, all I can and should do is update in the way Bayes' theorem says I should.

**Scientist:**  Seriously.

**Bayesian:**  I am serious.

**Scientist:**  So it doesn't bother you if I keep checking the likelihood ratio and continuing to flip the coin until I can convince you of anything I want.

**Bayesian:**  Go ahead and try it.

**Scientist:**  What I'm actually going to do is write a Python program which simulates flipping a fair coin *up to* 300 times, and I'm going to see how many times I can get a 20:1 likelihood ratio falsely indicating that the coin is biased to come up heads 55% of the time... why are you smiling?

**Bayesian:**  I wrote pretty much the same Python program when I was first converting to Bayesianism and finding out about likelihood ratios and feeling skeptical about the system maybe being abusable in some way, and then a friend of mine found out about likelihood ratios and *he* wrote [essentially the same program, also in Python](https://gist.github.com/Soares/941bdb13233fd0838f1882d148c9ac14).  And lo, he found that false evidence of 20:1 for the coin being 55% biased was found at least once, somewhere along the way... 1.4% of the time.  If you asked for more extreme likelihood ratios, the chances of finding them dropped off even faster.

**Scientist:**  Okay, that's not bad by the p-value way of looking at things.  But what if there's some more clever way of biasing it?

**Bayesian:**  When I was... I must have been five years old, or maybe even younger, and first learning about addition, one of the earliest childhood memories I have at all, is of adding 3 to 5 by counting 5, 6, 7 and believing that you could get different results from adding numbers depending on exactly how you did it.  Which is cute, yes, and also indicates a kind of exploring, of probing, that was no doubt important in my starting to understand addition.  But you still look back and find it humorous, because now you're a big grownup and you know you can't do that.  My writing Python programs to try to find clever ways to fool myself by repeatedly checking the likelihood ratios was the same, in the sense that after I matured a bit more as a Bayesian, I realized that the feat I'd written those programs to try to do was *obviously* impossible.  In the same way that trying to find a clever way to break apart the 3 into 2 and 1, and trying to add them separately to 5, and then trying to add the 1 and then the 2, in hopes you can get to 7 or 9 instead of 8, is just never ever going to work.  The results in arithmetic are *theorems,* and it doesn't matter in what clever order you switch things up, you are never going to get anything except 8 when you carry out an operation that is validly equivalent to adding 3 plus 5.  The theorems of probability theory are also theorems.  If your Python program had actually worked, it would have produced a contradiction in probability theory, and thereby a contradiction in Peano Arithmetic, which provides a model for probability theory carried out using rational numbers.  The thing you tried to do is *exactly* as hard as adding 3 and 5 using the standard arithmetic axioms and getting 7.

**Undergrad:**  Uh, why?

**Scientist:**  Seconded.

**Bayesian:**  Because letting $e$ denote the evidence, $H$ denote the hypothesis, $\neg$ denote the negation of a proposition, $\mathbb P(X)$ denote the probability of proposition $X$, and $\mathbb P(X \mid Y)$ denote the [https://arbital.com/p/-1rj](https://arbital.com/p/-1rj) of $X$ assuming $Y$ to be true, it is a theorem of probability that $$\mathbb P(H) = \left(P(H \mid e) \cdot P(e)\right) + \left(P(H\mid \neg e) \cdot P(\neg e)\right).$$ Therefore likelihood functions can *never* be p-hacked by *any possible* clever setup without you outright lying, because you can't have any possible procedure that a Bayesian knows in advance will make them update in a predictable net direction.  For every update that we expect to be produced by a piece of evidence $e,$ there's an equal and opposite update that we expect to probably occur from seeing $\neg e.$

**Undergrad:**  What?

**Scientist:**  Seconded.

**Bayesian:**  Look... let me try to zoom out a bit, and yes, look at the ongoing replication crisis.  The Scientist proclaimed suspicion of grand new sweeping ideals.  Okay, but the shift to likelihood functions is the kind of thing that *ought* to be able to solve a lot of problems at once.  Let's say... I'm trying to think of a good analogy here.  Let's say there's a corporation which is having a big crisis because their accountants are using floating-point numbers, only there's three different parts of the firm using three different representations of floating-point numbers to do numerically unstable calculations.  Somebody starts with 1.0 and adds 0.0001 a thousand times and then subtracts 0.1 and gets 0.999999999999989.  Or you can go to the other side of the building and use a different floating-point represenatation and get a different result.  And nobody has any conception that there's anything wrong with this.  Suppose there are BIG errors in the floating-point numbers, they're using the floating-point-number equivalent of crude ideograms and Roman numerals, you can get big pragmatic differences depending on what representation you use.  And naturally, people 'division-hack' to get whatever financial results they want.  So all the spreadsheets are failing to replicate, and people are starting to worry the 'cognitive priming' subdivision has actually been bankrupt for 20 years.  And then one day you come in and you say, "Hey.  Everyone.  Suppose that instead of these competing floating-point representations, we use my new representation instead.  It can't be fooled the same way, which will solve a surprising number of your problems."

*(The **Bayesian** now imitates the **Scientist's** voice:)*  "I'm suspicious," says the Senior Auditor.  "I suspect you of idealism.  In my experience, people need to use different floating-point representations for different financial problems, and it's good to have a lot of different numerical representations of fractions in your toolbox."

**Bayesian:**  "Well," I reply, "it may sound idealistic, but in point of fact, this thing I'm about to show you is *the* representation of fractions, in which you *cannot* get different results depending on which way you add things or what order you do the operations in.  It might be slightly more computationally expensive, but it is now no longer 1920 like when you first adopted the old system, and seriously, you can afford the computing power in a very large fraction of cases where you're only working with 30,000,000 bank accounts or some trivial number like that.  Yes, if you want to do something like take square roots, it gets a bit more complicated, but very few of you are actually taking the square root of bank account balances.  For the vast majority of things you are trying to do on a day-to-day basis, this system is unhackable without actually misreporting the numbers."  And then I show them how to represent arbitrary-magnitude finite integers precisely, and how to represent a rational number as the ratio of two integers.  What we would, nowadays, consider to be a direct, precise, computational representation of *the* system of rational numbers.  The one unique axiomatized mathematical system of rational numbers, to which floating-point numbers are a mere approximation.  And if you're just working with 30,000,000 bank account balances and your crude approximate floating-point numbers are *in practice* blowing up and failing to replicate and being exploited by people to get whatever results they want, and it is no longer 1920 and you can afford real computers now, it is an obvious step to have all the accountants switch to using *the* rational numbers.  Just as Bayesian updates are *the* rational updates, in the unique mathematical axiomatized system of probabilities.  And that's why you can't p-hack them.

**Scientist:**  That is a rather... audacious claim.  And I confess, even if everything you said about the math were true, I would still be skeptical of the pragmatics.  The current system of scientific statistics is something that's grown up over time and matured.  Has this bright Bayesian way actually been tried?

**Bayesian:**  It hasn't been tried very much in science.  In machine learning, where, uh, not to put too fine a point on it, we can actually see where the models are breaking because our AI doesn't work, it's been ten years since I've read a paper that tries to go at things from a frequentist angle and I can't *ever* recall seeing an AI algorithm calculate the p-value of anything.  If you're doing anything principled at all from a probability-theoretic stance, it's probably Bayesian, and pretty much never frequentist.  If you're classifying data using n-hot encodings, your loss function is the cross-entropy, not... I'm not even sure *what* the equivalent of trying to use 1920s-style p-values in AI would be like.  I would frankly attribute this to people in machine learning having to use statistical tools that visibly succeed or fail; rather than needing to get published by going through a particular traditional ritual of p-value reporting, and failure to replicate not being all that bad for your career.

**Scientist:**  So you're actually more of a computer science guy than an experimentalist yourself.  Why does this not surprise me?  It's not impossible that some better statistical system than p-values could exist, but I'd advise you to respect the wisdom of experience.  The fact that we know what p-hacking is, and are currently fighting it, is because we've had time to see where the edges of the system have problems, and we're figuring out how to fight those problems.  This shiny new system will also have problems; you just have no idea what they'll be.  Perhaps they'll be worse.

**Bayesian:**  It's not impossible that the accountants would figure out new shenanigans to pull with rational numbers, especially if they were doing some things computationally intensive enough that they could no longer afford to use *the* rational numbers and had to use some approximation instead.  But I stand by my statement that if your financial spreadsheets are *right now* blowing up in a giant replication crisis in ways that seem clearly linked to using p-values, and the p-values are, frankly, bloody ad-hoc inconsistent nonsense, an obvious first step is to *try* using the rational updates instead.  Although, it's possible we don't disagree too much in practice.  I'd also pragmatically favor trying to roll things out one step at a time, like, maybe just switch over the psychological sciences and see how that goes.

**Scientist:**  How would you persuade them to do that?

**Bayesian:**  I have no goddamn idea. Honestly, I'm not expecting anyone to actually fix anything.  People will just go on using p-values until the end of the world, probably.  It's just one more Nice Thing We Can't Have.  But there's a *chance* the idea will catch on.  I was pleasantly surprised when open access caught on as quickly as it did.  I was pleasantly surprised when people, like, actually noticed the replication crisis and it became a big issue that people cared about.  Maybe I'll be pleasantly surprised again and people will actually take up the crusade to bury the p-value at a crossroads at midnight and put a stake through its heart.  If so, I'll have done my part by making an understanding of [Bayes' rule](https://arbital.com/p/1lz) and likelihoods [more accessible](https://arbital.com/p/1zq) to everyone.

**Scientist:**  Or it could turn out that people don't *like* likelihoods, and that part of the wisdom of experience is the lesson that p-values are a kind of thing that experimentalists actually find useful and easy to use.

**Bayesian:**  If the experience of learning traditional statistics traumatized them so heavily that the thought of needing to learn a new system sends them screaming into the night, then yes, change might need to be imposed from outside.  I'm hoping though that the Undergrad will read a [short, cheerful introduction to Bayesian probability](https://arbital.com/p/1zq), compare this with his ominous heavy traditional statistics textbook, and come back going "Please let me use likelihoods please let me use likelihoods oh god please let me use likelihoods."

**Undergrad:**  I'll guess I'll look into it and see?

**Bayesian:**  Weigh your decision carefully, Undergrad.  Some changes in science depend upon students growing up familiar with multiple ideas and choosing the right one.  Max Planck said so in a famous aphorism, so it must be true.  Ergo, the entire ability of science to distinguish good and bad ideas within that class must rest upon the cognitive capacities of undergrads.

**Scientist:**  Oh, now that is just--

**Moderator:**  And we're out of time.  Thanks for joining us, everyone!

Eric Bruylant
Grigoriy Beziuk
Eric Rogstad
Patrick Stevens
Nate Soares
Drake Thomas
Adnll
Eliezer Yudkowsky

Or, __Why Switching From Reporting p-values to Reporting Likelihood Functions Might Help Fix the Replication Crisis:  A personal view by Eliezer Yudkowsky.__

Short version: Report [likelihoods](https://arbital.com/p/56t), not p-values.


C-Class

C-Class pages have substantial content and will often be useful to readers, but they may have significant prose and style problems, may require cleanup, may not cover all significant areas of the topic, or may not explain the concept in a way the target audience will reliably understand.

Contrast:

- [Start-Class](https://arbital.com/p/3rk) pages, which are sufficiently unfinished that they should be viewed by editors rather than readers.
- [B-Class](https://arbital.com/p/4yd) pages, which are reasonably comprehensive, have no major stylistic issues, and explain the concept in a way which seems likely to be often enlightening to the target audience.

**[Quality scale](https://arbital.com/p/4yg)**

* [https://arbital.com/p/4ym](https://arbital.com/p/4ym)
* [https://arbital.com/p/4gs](https://arbital.com/p/4gs)
* [https://arbital.com/p/5xq](https://arbital.com/p/5xq)
* [https://arbital.com/p/72](https://arbital.com/p/72)
* [https://arbital.com/p/3rk](https://arbital.com/p/3rk)
* [https://arbital.com/p/4y7](https://arbital.com/p/4y7)
* [https://arbital.com/p/4yd](https://arbital.com/p/4yd)
* [https://arbital.com/p/4yf](https://arbital.com/p/4yf)
* [https://arbital.com/p/4yl](https://arbital.com/p/4yl)

Alexei Andreev
Eric Rogstad
Eric Bruylant
Mark Chimes
Eliezer Yudkowsky

Meta tags which request an edit to the page


B-Class

A B-Class page is fairly comprehensive, with good prose and structure; but needs further work to reach the A-Class requirement for engaging, professional, intuitive explanations, checked by asking for detailed feedback from both the target audience and [reviewers](https://arbital.com/p/5ft).

B-Class pages are:

1. Comprehensive, covering all major areas of the topic without significant omissions or inaccuracies.
2. Fairly well-written and in line with the [https://arbital.com/p/-16t](https://arbital.com/p/-16t).
3. Reasonably well-organized, with similar concepts grouped together and reasonable section ordering.
4. Aimed at an audience, with a level of technicality, jargon, and notation appropriate for them.
5. Good for people wishing to learn the topic (well-explained, using examples, links, and visual aids as appropriate).
6. Well-[summarized](https://arbital.com/p/1kl), in ways appropriate for the most likely audiences.

This tag should only be added by [reviewers](https://arbital.com/p/5ft). To request a review ask on [https://arbital.com/p/4ph](https://arbital.com/p/4ph) or [propose a page for B-Class](https://arbital.com/p/5g3).

**[Quality scale](https://arbital.com/p/4yg)**

* [https://arbital.com/p/4ym](https://arbital.com/p/4ym)
* [https://arbital.com/p/4gs](https://arbital.com/p/4gs)
* [https://arbital.com/p/5xq](https://arbital.com/p/5xq)
* [https://arbital.com/p/72](https://arbital.com/p/72)
* [https://arbital.com/p/3rk](https://arbital.com/p/3rk)
* [https://arbital.com/p/4y7](https://arbital.com/p/4y7)
* [https://arbital.com/p/4yd](https://arbital.com/p/4yd)
* [https://arbital.com/p/4yf](https://arbital.com/p/4yf)
* [https://arbital.com/p/4yl](https://arbital.com/p/4yl)

Eric Rogstad
Eric Bruylant
Eliezer Yudkowsky
Alexei Andreev


Value

### Introduction

In the context of [value alignment](https://arbital.com/p/2v) as a subject, the word 'value' is a speaker-dependent variable that indicates our ultimate goal - the property or meta-property that the speaker wants or 'should want' to see in the final outcome of Earth-originating intelligent life.  E.g: [human flourishing](https://arbital.com/p/), [fun](https://arbital.com/p/), [coherent extrapolated volition](https://arbital.com/p/3c5), [normativity](https://arbital.com/p/).

Different viewpoints are still being debated on this topic; people [sometimes change their minds about their views](https://arbital.com/p/).  We don't yet have full knowledge of which views are 'reasonable' in the sense that people with good cognitive skills might retain them [even in the limit of ongoing discussion](https://arbital.com/p/313).  Some subtypes of potentially internally coherent views may not be sufficiently [interpersonalizable](https://arbital.com/p/) for even very small AI projects to cooperate on them; if e.g. Alice wants to own the whole world and will go on believing that in the limit of continuing contemplation, this is not a desideratum on which Alice, Bob, and Carol can all cooperate.  Thus, using 'value' as a potentially speaker-dependent variable isn't meant to imply that everyone has their own 'value' and that no further debate or cooperation is possible; people can and do talk each other out of positions which are then regarded as having been mistaken, and completely incommunicable stances seem unlikely to be reified even into a very small AI project.  But since this debate is ongoing, there is not yet any one definition of 'value' that can be regarded as settled.

Nonetheless, on many of the current views being advocated, it seems like very similar technical problems of value alignment seem to arise in many of them.  We would need to figure out how to [identify](https://arbital.com/p/6c) the objects of value to the AI, robustly assure that the AI's preferences are [stable](https://arbital.com/p/1fx) as the AI self-modifies, or create [corrigible](https://arbital.com/p/45) ways of recovering from errors in the way we tried to identify and specify the objects of value.

To centralize the very similar discussions of these technical problems while the outer debate about reasonable end goals is ongoing, the word 'value' acts as a metasyntactic placeholder for different views about the target of value alignment.

Similarly, in the larger [value achievement dilemma](https://arbital.com/p/2z), the question of what the end goals should be, and policy difficulties of getting 'good' goals to be adopted in name by the builders or creators of AI, are factored out as the [value selection problem](https://arbital.com/p/value_selection).  The output of this process is taken to be an input into the value loading problem, and 'value' is a name referring to this output.

'Value' is *not* assumed to be what the AI is given as its utility function or [preference framework](https://arbital.com/p/5f).  On many views implying that [value is complex](https://arbital.com/p/5l) or otherwise difficult to convey to an AI, the AI may be, e.g., a [Genie](https://arbital.com/p/6w) where some stress is taken off the proposition that the AI exactly understands value and put onto human ability to use the Genie well.

Consider a Genie with an explicit preference framework targeted on a [Do What I Know I Mean system](https://arbital.com/p/) for making [checked wishes](https://arbital.com/p/).  The word 'value' in any discussion thereof should still only be used to refer to whatever the AI creators are targeting for real-world outcomes.  We would say the 'value alignment problem' had been successfully solved to the extent that running the Genie produced high-value outcomes in the sense of the humans' viewpoint on 'value', not to the extent that the outcome matched the Genie's preference framework for how to follow orders.

### Specific views on value

Obviously, a listing like this will only summarize long debates.  But that summary at least lets us point to some examples of views that have been advocated, and not indefinitely defer the question of what 'value' could possibly refer to.

Again, keep in mind that by technical definition, 'value' is what we are using or should use to rate the ultimate real-world consequences of running the AI, *not* the explicit goals we are giving the AI.

Some of the major views that have been advocated by more than one person are as follows:

- **Reflective equilibrium.**  We can talk about 'what I *should* want' as a concept distinct from 'what I want right now' by construing some limit of how our present desires would directionally change given more factual knowledge, time to consider more knowledge, better self-awareness, and better self-control.  Modeling this process is **extrapolation**, a reserved term to mean this process in the context of discussing preferences.  Value would consist in, e.g., whatever properties a supermajority of humans would agree, in the limit of reflective equilibrium, are desirable.  See also [coherent extrapolated volition](https://arbital.com/p/).
- **Standard desires.**  An object-level view that identifies value with qualities that we currently find very desirable, enjoyable, fun, and preferable, such as [Frankena's list of desiderata](https://arbital.com/p/) (including truth, happiness, aesthetics, love, challenge and achievement, etc.) On the closely related view of **Fun Theory**, such desires may be further extrapolated, without changing their essential character, into forms suitable for transhuman minds.  Advocates may agree that these object-level desires will be subject to unknown normative corrections by reflective-equilibrium-type considerations, but still believe that some form of Fun or standardly desirable outcome is a likely result.  Therefore (on this view) it is reasonable to speak of value as probably mostly consisting in turning most of the reachable universe into superintelligent life enjoying itself, creating transhuman forms of art, etcetera.
- **[Immediate goods](https://arbital.com/p/ImmediateGoods).**  E.g., "Cure cancer" or "Don't transform the world into paperclips."  Such replies arguably have problems as ultimate criteria of value from a human standpoint (see linked discussion), but for obvious reasons, lists of immediate goods are a common early thought when first considering the subject.
- **Deflationary moral error theory.**  There is no good way to construe a normative concept apart from what particular people want.  AI programmers are just doing what they want, and confused talk of 'fairness' or 'rightness' cannot be rescued.  The speaker would nonetheless personally prefer not to be turned into paperclips.  (This mostly ends up at an 'immediate goods' theory in practice, plus some beliefs relevant to the [value selection](https://arbital.com/p/value_selection) debate.)
- **Simple purpose.**  Value can easily be identified with X, for some X.  X is the main thing we should be concerned about passing on to AIs.  Seemingly desirable things besides X are either (a) improper to care about, (b) relatively unimportant, or (c) instrumentally implied by pursuing X, qua X.

The following versions of desiderata for AI outcomes would tend to imply that the value alignment / value loading problem is an entirely wrong way of looking at the issue, which might make it disingenuous to claim that 'value' in 'value alignment' can cover them as a metasyntactic variable as well:

- **Moral internalist value.**  The normative is inherently compelling to all, or almost all cognitively powerful agents.  Whatever is not thus compelling cannot be normative or a proper object of human desire.
- **AI rights.**  The primary thing is to ensure that the AI's natural and intrinsic desires are respected.  The ideal is to end up in a diverse civilization that respects the rights of all sentient beings, including AIs.  (Generally linked are the views that no special selection of AI design is required to achieve this, or that special selection of AI design to shape particular motivations would itself violate AI rights.)

## Modularity of 'value'

### Alignable values

Many issues in value alignment seem to generalize very well across the Reflective Equilibrium, Fun Theory, Intuitive Desiderata, and Deflationary Error Theory viewpoints.  In all cases we would have to consider stability of self-modification, the [Edge Instantiation](https://arbital.com/p/2w) problem in [value identification](https://arbital.com/p/6c), and most of the rest of 'standard' value alignment theory.  This seemingly good generalization of the resulting technical problems across such wide-ranging viewpoints, and especially that it (arguably) covers the case of intuitive desiderata, is what justifies treating 'value' as a metasyntactic variable in 'value loading problem'.

A neutral term for referring to all the values in this class might be 'alignable values'.

### Simple purpose

In the [simple purpose](https://arbital.com/p/) case, the key difference from an Immediate Goods scenario is that the desideratum is usually advocated to be simple enough to negate [Complexity of Value](https://arbital.com/p/5l) and make [value identification](https://arbital.com/p/6c) easy.

E.g., Juergen Schmidhuber stated at the 20XX Singularity Summit that he thought the only proper and normative goal of any agent was to increase compression of sensory information .  Conditioned on this being the sum of all normativity, 'value' is algorithmically simple.  Then the problems of [Edge Instantiation](https://arbital.com/p/2w), [Unforeseen Maximums](https://arbital.com/p/47), and Nearest Unblocked Neighbor are all moot.  (Except perhaps as there is an Ontology Identification problem for defining exactly what constitutes 'sensory information' for a [self-modifying agent](https://arbital.com/p/).)

Even in the [simple purpose](https://arbital.com/p/) case, the [value loading problem](https://arbital.com/p/) would still exist (it would still be necessary to make an AI that cared about the simple purpose rather than paperclips) along with associated problems of [reflective stability](https://arbital.com/p/71) (it would be necessary to make an AI that went on caring about X through self-modification).  Nonetheless, the overall problem difficulty and immediate technical priorities would be different enough that the Simple Purpose case seems importantly distinct from e.g. Fun Theory on a policy level.

### Moral internalism

Some viewpoints on 'value' deliberately reject [Orthogonality](https://arbital.com/p/1y).  Strong versions of the [moral internalist position in metaethics](https://arbital.com/p/) claim as an empirical prediction that every sufficiently powerful cognitive agent will come to pursue the same end, which end is to be identified with normativity, and is the only proper object of human desire.  If true, this would imply that the entire value alignment problem is moot for advanced agents.

Many people who advocate 'simple purposes' also claim these purposes are universally compelling.  In a policy sense, this seems functionally similar to the Moral Internalist case regardless of the simplicity or complexity of the universally compelling value.  Hence an alleged simple universally compelling purpose is categorized for these purposes as Moral Internalist rather than Simple Purpose.

The special case of a Simple Purpose claimed to be universally [instrumentally convergent](https://arbital.com/p/10g) also seems functionally identical to Moral Internalism from a policy standpoint.)

### AI Rights

Someone might believe as a proposition of fact that all (accessible) AI designs would have 'innate' desires, believe as a proposition of fact that no AI would gain enough advantage to wipe out humanity or prevent the creation of other AIs, and assert as a matter of morality that a good outcome consists of everyone being free to pursue their own value and trade.  In this case the value alignment problem is implied to be an entirely wrong way to look at the problem, with all associated technical issues moot.  Thus, it again might be disingenuous to have 'value' as a metasyntactic variable try to cover this case.

Eric Bruylant
Eliezer Yudkowsky
Alexei Andreev

Different people advocate different views on what we should want for the outcome of an '[aligned](https://arbital.com/p/value)' AI (desiderata like human flourishing, or a [fun-theoretic eudaimonia](https://arbital.com/p/), or [coherent extrapolated volition](https://arbital.com/p/3c5), or an AI that mostly leaves us alone but protects us from other AIs).  These differences might not be [irreconcilable](https://arbital.com/p/); people are sometimes persuaded to change their views of what we should want.  Either way, there's (arguably) a tremendous overlap in the technical issues for aligning an AI with any of these goals.  So in the technical discussion, 'value' is really a metasyntactic variable that stands in for the speaker's current view, or for what an AI project might later adopt as a reasonable target after further discussion.


Definition

When substantial controversy exists about how to define a term, good epistemic policy is for both sides to adopt new, more specific terms whose definitions are not further disputed.  To whatever extent possible, definitions should not be phrased in a way that tries to pre-emptively settle an argument or 'bake in' one answer to a factual or policy disagreement.  See [A Human's Guide to Words](http://wiki.lesswrong.com/wiki/A_Human%27s_Guide_to_Words).

Kevin Clancy
Alexei Andreev
Eric Rogstad
Patrick Stevens
Nate Soares
Eric Bruylant
Jaime Sevilla Molina
Eliezer Yudkowsky


Goodness estimate biaser

A "[goodness](https://arbital.com/p/55) estimate [biaser](https://arbital.com/p/statistical_bias)" is a system setup or phenomenon that seems [foreseeably](https://arbital.com/p/6r) likely to cause the actual goodness of some AI plan to be systematically lower than the AI's estimate of that plan's goodness.  We want the AI's estimate to be [unbiased](https://arbital.com/p/statistically_unbiased).

## Ordinary examples

Subtle and unsubtle [estimate-biasing](https://arbital.com/p/statistical_bias) issues in machine learning are well-known and appear far short of [advanced agency](https://arbital.com/p/2c):

●  A machine learning algorithm's performance on the training data is not an unbiased estimate of its performance on the test data.  Some of what the algorithm seems to learn may be particular to noise in the training data.  This fitted noise will not be fitted within the test data.  So test performance is not just unequal to, but *systematically lower than,* training performance; if we were treating the training performance as an estimate of test performance, it would not be an [unbiased](https://arbital.com/p/statistically_unbiased) estimate.

●  The [Winner's Curse](https://en.wikipedia.org/wiki/Winner%27s_curse) from auction theory observes that if bidders have noise in their unbiased estimates of the auctioned item's value, then the *highest* bidder, who receives the item, is more likely to have upward noise in their individually unbiased estimate, [conditional](https://arbital.com/p/1ly) on their having won.  (E.g., three bidders with Gaussian noise in their value estimates submit bids on an item whose true value to them is 1.0; the winning bidder is likely to have valued the item at more than 1.0.)

The analogous [Optimizer's Curse](https://faculty.fuqua.duke.edu/~jes9/bio/The_Optimizers_Curse.pdf) observes that if we make locally unbiased but noisy estimates of the [subjective expected utility](https://arbital.com/p/subjective_expected_utility) of several plans, then selecting the plan with 'highest expected utility' is likely to select an estimate with upward noise.  Barring compensatory adjustments, this means that actual utility will be systematically lower than expected utility, even if all expected utility estimates are individually unbiased.  Worse, if we have 10 plans whose expected utility can be unbiasedly estimated with low noise, plus 10 plans whose expected utility can be unbiasedly estimated with high noise, then selecting the plan with apparently highest expected utility favors the noisiest estimates!

## In AI alignment

We can see many of the alleged [foreseeable difficulties](https://arbital.com/p/6r) in [AI alignment](https://arbital.com/p/2v) as involving similar processes that allegedly produce systematic downward biases in what we see as actual [goodness](https://arbital.com/p/55), compared to an AI's estimate of goodness:

●  [https://arbital.com/p/2w](https://arbital.com/p/2w) suggests that if we take an imperfectly or incompletely learned value function, then looking for the *maximum* or *extreme* of that value function is much more likely than usual to magnify what we see as the gaps or imperfections (because of [fragility of value](https://arbital.com/p/fragile_value), plus the Optimizer's Curse); or destroy whatever aspects of value the AI didn't learn about (because optimizing a subset of properties is liable to set all other properties to extreme values).

We can see this as implying both "The AI's apparent goodness in non-extreme cases is an upward-biased estimate of its goodness in extreme cases" and "If the AI learns its goodness estimator less than [perfectly](https://arbital.com/p/41k), the AI's estimates of the goodness of its best plans will systematically overestimate what we see as the actual goodness."

●  [https://arbital.com/p/42](https://arbital.com/p/42) generally, and especially over [instrumentally convergent incorrigibility](https://arbital.com/p/instrumental_incorrigibility), suggests that if there are naturally-arising AI behaviors we see as bad (e.g. routing around [shutdown](https://arbital.com/p/2xd)), there may emerge a pseudo-adversarial selection of strategies that route around our attempted [patches](https://arbital.com/p/48) to those problems.  E.g., the AI constructs an environmental subagent to continue carrying on its goals, while cheerfully obeying 'the letter of the law' by allowing its current hardware to be shut down.  This pseudo-adversarial selection (though the AI does not have an explicit goal of thwarting us or selecting low-goodness strategies per se) again implies that actual [goodness](https://arbital.com/p/55) is likely to be systematically lower than the AI's estimate of what it's learned as 'goodness'; again to an [increasing degree](https://arbital.com/p/6q) as the AI becomes [smarter](https://arbital.com/p/9f) and [searches a wider policy space](https://arbital.com/p/47).

[Mild optimization](https://arbital.com/p/2r8) and [conservative strategies](https://arbital.com/p/2qp) can be seen as proposals to 'regularize' powerful optimization in a way that *decreases* the degree to which goodness in training is a biased (over)estimate of goodness in execution.


Linguistic conventions in value alignment

A central page to list the language conventions in [value alignment theory](https://arbital.com/p/2v).  See also [https://arbital.com/p/9m](https://arbital.com/p/9m).

# Language dealing with wants, desires, utility, preference, and value.

We need a language rich enough to distinguish at least the following as different [intensional concepts](https://arbital.com/p/10b), even if their [extensions](https://arbital.com/p/10b) end up being identical:

- A.  What the programmers explicitly, verbally said they wanted to achieve by building the AI.
- B.  What the programmers wordlessly, intuitively meant; the actual criterion they would use for rating the desirability of outcomes, if they could actually look at those outcomes and assign ratings.
- C.  What programmers *should* want from the AI (from within some view on normativity, shouldness, or rightness).
- D.  The AI's explicitly represented cognitive preferences, if any.
- E.  The property that running the AI tends to produce in the world; the property that the AI behaves in such fashion as to bring about.

So far, the following reserved terms have been advocated for the subject of value alignment:

- [**Value**](https://arbital.com/p/55) and **valuable** to refer to C.  On views which identify C with B, it thereby refers to B.
- **Optimization target** to mean only E.  We can also say, e.g., that natural selection has an 'optimization target' of inclusive genetic fitness.  'Optimization target' is meant to be an exceedingly general term that can talk about irrational agents and nonagents.
- [**Utility**](https://arbital.com/p/109) to mean a Von Neumann-Morgenstern utility function, reserved to talk about agents that behave like some bounded analogue of expected utility optimizers.  Utility is explicitly not assumed to be normative.  E.g., if speaking of a paperclip maximizer, we will say that an outcome has higher utility iff it contains more paperclips.  Thus 'utility' is reserved to refer to D or E.
- **Desire** to mean anthropomorphic human-style desires, referring to A or B rather than C, D, or E.  ('Wants' are general over humans and AIs.)
- **Preference** and **prefer** to be general terms that can be used for both humans and AIs.  'Preference' refers to B or D rather than A, C, or E.  It means 'what the agent explicitly and cognitively wants' rather than 'what the agent should want' or 'what the agent mistakenly thinks it wants' or 'what the agent's behavior tends to optimize'.  Someone can be said to prefer their extrapolated volition to be implemented rather than their current desires, but if so they must explicitly, cognitively prefer that, or accept it in an explicit choice between options.
- [**Preference framework**](https://arbital.com/p/5f) to be an even more general term that can refer to e.g. meta-utility functions that change based on observations, or to meta-preferences about how one's own preferences should be extrapolated.  A 'preference framework' should refer to constructs more coherent than the human mass of desires and ad-hoc reflections, but not as strictly restricted as a VNM utility function.  Stuart Armstrong's [utility indifference](https://arbital.com/p/) framework for [value learning](https://arbital.com/p/) is an example of a preference framework that is not a vanilla/ordinary utility function.
- **Goal** remains a generic, unreserved term that could refer to any of A-E, and also particular things an agent wants to get done for [instrumental](https://arbital.com/p/) reasons.
- **[Intended goal](https://arbital.com/p/6h)** to refer to B only.
- **Want** remains a generic, unreserved term that could refer to humans or other agents, or terminal or instrumental goals.

'Terminal' and 'instrumental' have their standard contrasting meanings.


Limited AGI

One of the reasons why a [Task AGI](https://arbital.com/p/6w) can potentially be safer than an [Autonomous AGI](https://arbital.com/p/1g3), is that since Task AGIs only need to carry out activities of limited scope, they [may only need limited material and cognitive powers](https://arbital.com/p/7tf) to carry out those tasks.  The [nonadversarial principle](https://arbital.com/p/7g0) still applies, but takes the form of "[don't run the search](https://arbital.com/p/7fx)" rather than "make sure the search returns the correct answer".

# Obstacles

• Increasing your material and cognitive efficacy is [instrumentally convergent](https://arbital.com/p/10g) in all sorts of places and would presumably need to be [averted](https://arbital.com/p/2vk) all over the place.

• Good limitation proposals are [not as easy as they look](https://arbital.com/p/deceptive_ease) because [particular domain capabilities can often be derived from more general architectures](https://arbital.com/p/7vh).  An Artificial *General* Intelligence doesn't have a handcrafted 'thinking about cars' module and a handcrafted 'thinking about planes' module, so you [can't just handcraft the two modules at different levels of ability](https://arbital.com/p/7vk).

E.g. many have suggested that 'drive' or 'emotion' is something that can be selectively removed from AGIs to 'limit' their ambitions; [presumably](https://arbital.com/p/43h) these people are using a mental model that is not the standard [expected utility agent](https://arbital.com/p/18r) model.  To know which kind of limitations are easy, you need a sufficiently good background picture of the AGI's subprocesses that you understand which kind of system capabilities will naturally carve at the joints.

# Related ideas

The research avenue of [Mild optimization](https://arbital.com/p/2r8) can be viewed as pursuing a kind of very general Limitation.

[Behaviorism](https://arbital.com/p/102) asks to Limit the AGI's ability to model other minds in non-whitelisted detail.

[Taskishness](https://arbital.com/p/4mn) can be seen as an Alignment/Limitation hybrid in the sense that it asks for the AI to only *want* or *try* to do a bounded amount at every level of internal organization.

[https://arbital.com/p/2pf](https://arbital.com/p/2pf) can be seen as an Alignment/Limitation hybrid in the sense that a [successful impact penalty](https://arbital.com/p/4l) would make the AI not *want* to implement larger-scale plans.

Limitation may be viewed as yet another subproblem of the [https://arbital.com/p/3ps](https://arbital.com/p/3ps), since it seems like a type of precaution that a generic agent would desire to construct into a generic imperfectly-aligned subagent.

Limitation can be seen as motivated by both the [https://arbital.com/p/7g0](https://arbital.com/p/7g0) and the [https://arbital.com/p/7tf](https://arbital.com/p/7tf).