Harley-ml/lesswrong huggingface dataset:
# Formalising continuous info cascades? [Info-cascade series]

_This is a question in_ _[the info-cascade question series](https://www.lesswrong.com/posts/2uDBJWCksvzhDzHGf/understanding-info-cascades). There is a prize pool of up to $800 for answers to these questions. See the link above for full background on the problem (including a bibliography) as well as examples of responses we’d be especially excited to see._

* * *

Mathematically formalising info-cascades would be great.

Fortunately, it's already been done in the simple case. See this excellent [LW post](https://www.lesswrong.com/posts/DNQw596nPCX4x7xT9/information-cascades) by Johnicholas, where he uses upvotes/downvotes as an example, and shows that after the second person has voted, all future voters are adding zero new information to the system. His explanation using likelihood ratios is the most intuitive I've found.

The [Wikipedia entry](https://en.wikipedia.org/wiki/Information_cascade) on the subject is also quite good.

However, these two entries primarily explain how information cascades when people have to make a binary choice - good or bad, left or right, etc. The question I want to understand is how to think of the problem in a continuous case - do the problems go away? Or more likely, what variables determine the speed at which people update to one extreme? And how far toward that extreme do people go before they realise their error?

Examples of continuous variables include things like project time estimates, stocks, and probabilistic forecasts. I imagine it's very likely that significant quantitative work has been done on the case of market bubbles, and anyone can write an answer summarising that work and explaining how to apply it to other domains like forecasting, that would be excellent.
# Question: MIRI Corrigbility Agenda

MIRI's reading list on corrigbility seems out dated, and I can't find a centralised list Does anyone have, or know of, one?

As a side note, has MIRI stopped updating their reading list? It seems like that's the case.

  

EDIT:

Links given in the comment section to do with corrigibility. I'll try and update this with some summaries as I read them.

[https://www.greaterwrong.com/posts/5bd75cc58225bf0670375041/a-first-look-at-the-hard-problem-of-corrigibility](https://www.greaterwrong.com/posts/5bd75cc58225bf0670375041/a-first-look-at-the-hard-problem-of-corrigibility)

[https://arbital.com/p/corrigibility/](https://arbital.com/p/corrigibility/)

[https://arbital.com/p/updated_deference/](https://arbital.com/p/updated_deference/)

[https://arxiv.org/pdf/1611.08219.pdf](https://arxiv.org/pdf/1611.08219.pdf)
# AI Safety Prerequisites Course: Basic abstract representations of computation

Followup to [AI Safety Prerequisites Course: Revamp and New Lessons](https://www.lesswrong.com/posts/QnZjZvh4SxTP3bHS8/ai-safety-prerequisites-course-revamp-and-new-lessons). [First post](https://www.lesswrong.com/posts/kK67yXhmDYwXLqXoQ/fundamentals-of-formalisation-level-1-basic-logic).

These are three new lessons of our [online course](https://play.ihatestatistics.com/#/course/141?access_token=3HCK4oRipeFY2ghyYMqDJKYX57KUnzNL) on math formalizations required for AI safety research:

* Level 10: Recursive Functions
* Level 11: Set Theoretic Recursion
* Level 12: The Equivalence of Different Notions of Computability

With these lessons, the student now should:

* Understand the basic abstract representations of computation.
* Know some of what we can expect from computers and also what we can’t expect from them.
* Know a lot more set theoretic tools like equivalence relations and orderings.
* Have seen the construction of the natural numbers from the perspective of set theory.
* Know about mathematical induction, and have used it!
* Know about recursion, and have used it!

If you study using our course, please give us feedback. Leave a comment here or email us at [raise@aisafety.info](mailto:raise@aisafety.info), or through the [contact form](https://www.aisafety.info/contact-us). Do you have an idea about what prerequisites are most important for AI Safety research? Do you know an optimal way to learn them? Tell us using the same methods or collaborate with us.

Can you check if a mathematical proof is correct? Do you know how to make proofs understandable and easy to remember? Would you like to help to create the prerequisites course? If yes, consider [volunteering](https://docs.google.com/forms/d/1NbXKjFFQMA3Z5SwaFT722Sh0zq24EYOQz9ipV16T3YM/edit?fbclid=IwAR2pw5UpI38nscbeoCi24eDi6hz-eHb3r_pgZKvuPVl4cd4savW1B1qPYYo).
# Mysteries, identity, and preferences over non-rewards

*Note: working on a research agenda, hence the large amount of small individual posts, to have things to link to in the main documents*.

I've [given examples](https://www.lesswrong.com/posts/kgQCpiuHwDWNgz7E4/preferences-over-non-rewards) of preferences over non-rewards.

But those examples are easy to dismiss as irrational or imposing on others, as they involved religion, or things like preferences over other people knowing the truth.

But if we think in terms of [preferences over personal identity](https://www.lesswrong.com/posts/MxLK2fvEuijAYgsc2/smoothmin-and-personal-identity), we have far more widespread and natural examples of this.

One small example: in most murder-mysteries, I'd prefer not to know the identity of the killer at the beginning of the show. If I'm taking part in a murder mystery dinner, I'd definitely not want too. If I'm playing games of hidden information for fun, then I would not want to know all the hidden information - or else that makes this pointless. Lots of people enjoy watching sport events live; much less people re-watch random past sporting events whose result they already know. Eliezer talks about the [joy in scientific discovery](https://www.lesswrong.com/posts/KfMNFB3G7XNviHBPN/joy-in-discovery).

So preferences over non-rewards are quite natural and common.

## Hedonism solution?

Now, you could say you don't have a "preferences over knowing/not knowing something", but that you "enjoy the experience of learning", and thus reduce the whole thing to hedonism. Eliezer [argues against this](https://www.lesswrong.com/posts/synsRtBKDeAFuo7e3/not-for-the-sake-of-happiness-alone), claiming to have preferences that don't reduce to hedonism (preferences I share, incidentally).

But even fully giving in to hedonism only partially solves the problem; yes, we can model someone's faith as saying they enjoy the experience of belief, and murder-mystery watchers enjoy the experience of seeing intricate puzzles solved. But this doesn't solve the problem that, to maximise hedonic experience, we still have to control what knowledge the human has.
# Combining individual preference utility functions

*Note: working on a research agenda, hence the large amount of small individual posts, to have things to link to in the main documents*.

I've been [working on a way of synthesising the preferences of a single given human](https://www.lesswrong.com/posts/qezBTig6p6p5xtL6G/a-theory-of-human-values).

After this is done, there is the business of combining these utilities into a single utility function for the whole human population. One obvious way of doing this is to weigh the different utilities by some individual measure of intensity, then add them together, and maximise the sum.

But there are some arguments against this. For example, a lot of people's preferences is over the behaviour or experiences of other people. This includes things like promoting religious or cultural beliefs (and this can cover preferences like wanting others to have basic human rights or to be free from oppressive situations).

Apart from simple summing, which may result in an excessive strength to a majority, there are two other obvious solutions: the first is to remove preferences over other people entirely. Note that this would also remove most preferences over social and cultural systems. The other one is to remove any anti-altruistic components to the system: preferences over the suffering of others are no longer valid (note this would allow punishment-as-a-deterrence, or punishment-as-retribution-to-a-victim, but not punishment-as-abstract-justice). This may be a bit tricky to define - what's the clear difference between wanting to be of higher status and wanting others to be of lower status? - but might be a desirable compromise. It might, indeed, be the kind of compromise that different people would "negotiate to" in some sort of "moral parliament" example, since people often tend to prefer their own individual desires over desires to control other people, so this might be a solution that would have majority support.
# Boeing 737 MAX MCAS as an agent corrigibility failure

The Boeing Maneuvering Characteristics Augmentation System (MCAS) can be thought of, if reaching a bit, as a specialized AI: it performs a function normally reserved for a human pilot: [pitching the nose down when it deems the angle of attack to be dangerously high](http://www.b737.org.uk/mcas.htm). This is not, by itself, a problem. There are pilots in the cockpit who can take control when needed.

Only in this case they couldn't. Simply manually pitching the nose up when the MCAS pitching it down too much would not disengage the system, it would activate again, and again. One has to manually disengage the autopilot (this information was not in the pilot training). For comparison, think of the cruise control system in a car: the moment you press the brake pedal, it disengages; if you push the gas pedal, then release, it return to the preset speed. At no time it tries to override your actions. Unlike MCAS.

MCAS disregards critical human input and even fights the human for control in order to reach its goal of "nominal flight parameters". From the [Corrigibility](https://intelligence.org/files/Corrigibility.pdf) paper:

> We say that an agent is “corrigible” if it tolerates or assists many forms of outside correction, including atleast the following: (1) A corrigible reasoner must at least tolerate and preferably assist the programmers in their attempts to alter or turn off the system...

In this case the "agent" actively fought its human handlers instead of assisting them. Granted, the definition above is about programmers, not pilots, and the existing MCAS probably would not fight a software update, being a dumb specialized agent.

But we are not that far off: a lot of systems include built-in security checks for the remote updates. If one of those checks were to examine the algorithm the updated code uses and reject it when it deems it unacceptable because it fails its internal checks, the corrigibility failure would be complete! In a life-critical always-on system this would produce a mini-Skynet. I don't know whether something like that has happened yet, but I would not be surprised if it has, and resulted in catastrophic consequences.
# How large is the fallout area of the biggest cobalt bomb we can build?

Cobalt bombs are the currently most effective way I know of to make large areas of land unlivable for a long period of time. I don't actually know the fallout area of such a bomb. [The Wikipedia article](https://en.m.wikipedia.org/wiki/Cobalt_bomb) only has the following mildly helpful section:

> Theoretically, a device containing 510 [metric tons](https://en.m.wikipedia.org/wiki/Metric_ton) of Co-59 can spread 1 g of the material to each square km of the Earth's surface (510,000,000 km2). If one assumes that all of the material is converted to Co-60 at 100 percent efficiency and if it is spread evenly across the Earth's surface, it is possible for a single bomb to kill every person on Earth. However, in fact, complete 100% conversion into Co-60 is unlikely; a 1957 British experiment at Maralinga showed that Co-59's neutron absorption ability was much lower than predicted, resulting in a very limited formation of Co-60 isotope in practice.

I don't know whether a single bomb that efficiently distributes 510 tons of material is remotely feasible, and the section about Co-60 conversion also suggests that the actual effectiveness of such a bomb would be far below this theoretical maximum.

Does there exist any existing literature that estimates the practical fallout area of a cobalt bomb? Or can we just do a simple Fermi to estimate things ourselves?

I am interested in this question because it seems relevant to estimate the worst case scenario of a nuclear war, and the likelihood of civilization recovering after such an event.

I will award $50 to the best answer by the end of next week, based on my gut sense, and I might pay out more if someone gives a particularly exceptional answer.
# What societies have ever had legal or accepted blackmail?

Robin Hanson's post [History of Blackmail](https://www.overcomingbias.com/2011/08/history-of-blackmail.html) only mentions cases where blackmail was illicit or illegal. Have there ever been any societies, large or small, where blackmail was widely accepted?
# Ask LW: Have you read Yudkowsky's AI to Zombie book?

How many times have you read it? Did you start with the book and move to the website? (Anecdotally: I enjoyed until the half of it reading it while commuting but lately I've found that you can read them online (for free?!) And also have comments to every essay in part.(which I found insightful) should one start again but read it through LessWrong and the comments?
# Is there a difference between uncertainty over your utility function and uncertainty over outcomes?

I was discussing UDT yesterday and the question came up of how to treat uncertainty over your utility function. I suggested that this could be transformed into a question of uncertainty over outcomes. The intuition is that if you were to discover that apples were twice as valuable, you could simply pretend that you instead received twice as many apples. Is this approach correct? In particular, is it transformation compatible with UDT-style reasoning?
# Some thoughts after reading Artificial Intelligence: A Modern Approach

Intro
-----

I have been trying to learn what I need to know to contribute to AI safety research. To that end, I recently finished working through _Artificial Intelligence: A Modern Approach_. Before reading this textbook, I knew a bit about machine learning, but did not know much about search algorithms, knowledge representation, Bayesian networks, or the many many other topic AIMA covers. I found this textbook a useful overview of its field. Below I will talk about a few concepts that I found particularly interesting.

Generating heuristics
---------------------

If an algorithm is searching through an action space, it needs some way of deciding how close the possible next states it can reach are to its goal. One way of generating a useful heuristic is to relax some of the limitations on actions. Computing “how hard is it to get from state A to the goal?” takes a lot of work, so instead the algorithm could calculate “how hard would it be to get from state A to the goal if I could do things I cannot?” For example, if you are trying to plan how to coordinate planes, fuel, and cargo to fly various packages where they need to go, you could create a heuristic by ignoring fuel limitations of planes, forming a problem that is easier to solve but still gives information about the true problem. I would expect it to be hard to find relaxed problems that are still close enough to the full problem to give useful heuristics, but in the examples the book gives it seems to work pretty well.

Reasons for failure to learn
----------------------------

There are a few different reasons a learning algorithm can fail to find the right function: the correct hypothesis might not be in the hypothesis space, the data can be noisy so that perfect prediction from the information that the algorithm has access to is impossible, sample variance can makes the data the algorithm sees unrepresentative of the true distribution, and complexity of the hypothesis space can prevents the algorithm from finding the correct hypothesis. I found this a useful summary of possible barriers to learning.

Boosting
--------

In boosting, you train an initial classifier, see which data it gets wrong, and then train another classifier by weighting the training samples that the first classifier got wrong more heavily. You do this a few times and then combine the classifiers to make your actual predictions. This gives surprisingly (to me, at least) good performance. If the first classifier gets examples wrong because of noise, adding more classifiers doesn’t seem like it would help, because the new classifiers don’t have any new information. I guess it would help if the first classifier predicted based on the more common samples, while the later classifiers can help improve performance on the more uncommon data points. It could also help if one of the classifiers gets stuck in a bad local minimum.
# What's wrong with these analogies for understanding Informed Oversight and IDA?

In [Can HCH epistemically dominate Ramanujan?](https://www.lesswrong.com/posts/4qY9zEHLa2su4PkQ4/can-hch-epistemically-dominate-ramanujan) Alex Zhu wrote:

>If HCH is ascription universal, then it should be able to epistemically dominate an AI theorem-prover that reasons similarly to how Ramanujan reasoned. But I don’t currently have any intuitions as to why explicit verbal breakdowns of reasoning should be able to replicate the intuitions that generated Ramanujan’s results (or any style of reasoning employed by any mathematician since Ramanujan, for that matter).

And I [answered](https://www.lesswrong.com/posts/4qY9zEHLa2su4PkQ4/can-hch-epistemically-dominate-ramanujan#9esxLBJSNtmiiwvvG):

>My guess is that HCH has to reverse engineer the theorem prover, figure out how/why it works, and then reproduce the same kind of reasoning.

And then I followed up my own comment with:

>It occurs to me that if the overseer understands everything that the ML model (that it’s training) is doing, and the training is via some kind of local optimization algorithm like gradient descent, the overseer is essentially manually programming the ML model by gradually nudging it from some initial (e.g., random) point in configuration space. 

No one answered my comments with either a confirmation or denial, as to whether these guesses of how to understand Universality / Informed Oversight and IDA are correct. I'm surfacing this question as a top-level post because if "Informed Oversight = reverse engineering" and "IDA = programming by nudging" are good analogies for understanding Informed Oversight and IDA, it seems to have pretty significant implications. 

In particular it seems to imply that there's not much hope for IDA to be competitive with ML-in-general, because if IDA is analogous to a highly constrained method of "manual" programming, that seems unlikely to be competitive with less constrained methods of "manual" programming (i.e., AIs designing and programming more advanced AIs in more general ways, similar to how humans do most programming today), which itself is presumably not competitive with general (unconstrained-by-safety) ML (otherwise ML would not be the competitive benchmark).

If these are not good ways to understand IO and IDA, can someone please point out why?
# Criticism catalyzes analytical thinking in groups


Brainstorming, a technique long thought to enhance group creativity. Although the research shows little evidence of effectiveness, this technique (Alex Faickney Osborn, 1957) has persisted in practice. More specifically, brainstorming instructions increase the number of ideas in a group but are usually less than the total number of ideas generated by the same number of brainstorming individuals alone (Brown and Paulus, 1996).

The instructions for brainstorming are fairly precise:
- Quantity: come up with as many ideas as you can.
- Do not criticise others’ ideas.
- Build on others’ ideas.
- Freewheeling is welcome.

Not to criticise each others’ idea is an admonition as it may cause evaluation apprehension: people will be reluctant to express creative ideas; they won’t _freewheel_ for fear of evaluation and risk.

We’re traditionally taught that the criticism inherent to analysis is a no-go in the process of _ideation_: the second rule of brainstorming is _don’t criticise_. The absence of criticism is supposed to set free creativity but as it happens, the human mind is not a creative volcano waiting for an excuse to erupt.

Charlan Nemeth, a professor of psychology at the University of California, demonstrated that people are not very creative when they are given complete freedom. As part of an experiment on word association, she asked people to associate freely on the word _blue,_ the vast majority said first _green,_ then _sky_ or _ocean._ Her research shows that people are able to generate both a larger number of associations and more creative associations when faced with dissent and criticism.

As part of the aforementioned experiment, Nemeth placed her test subjects alongside actors posing as test subjects who would disagree on established truths like what colour an object is. It turned out that the test subjects confronted with the _dissenting_ actors had much more creative associations than the test subjects in the group without actors. Rather than associating _blue_ with _green, sky,_ or _ocean,_ they associated _blue_ with words like _berry pie._ Instead of killing creativity, disagreement seems to foster it.

In dire straits, analysis becomes an essential tool to understand what is going on in the world and help evaluate existing ideas—it is also a catalyst of the creativity needed to come up with solutions. 

Analysis injects dissent and criticism into the ideation process, and contrary to what the rules of brainstorming tell us, this makes people more, not less, creative.
# What should we expect from GPT-3?

When it will appear? (My guess is 2020).

Will it be created by OpenAI and will it be advertised? (My guess is that it will not be publicly known until 2021, but other companies may create open versions before it.)

How much data will be used for its training and what type of data? (My guess is 400 GB of text plus illustrating pictures, but not audio and video.)

What it will be able to do? (My guess: translation, picture generation based on text, text generation based on pictures – with 70 per cent of human performance.)

How many parameters will be in the model? (My guess is 100 billion to trillion.)

How much compute will be used for training? (No idea.)
# Willing to share some words that changed your beliefs/behavior?

I'm collecting data on powerfully persuasive speech acts; it's part of a dangling thread of curiosity after GPT-2 (a new and fairly powerful text generation algorithm). I'm skeptical of the danger of mind-warping sentences as sometimes presented in fiction, or AI scenarios, and trying to get a sense of what the territory is like.

I've made a form to collect personal examples of things-someone-said that caused you to seriously change some belief or behavior. An easy example would be if someone declared that they love you, and this caused you to suddenly devote a lot more (or a lot less!) time and attention to them as a person.

If you have five minutes, my goal for this form is 1000+ responses and your own response(s) will help with that. All replies are anonymous, and there's a place for you to restrict how the information is used/state confidentiality desires. You can also fill it out more than once if you want.

[https://goo.gl/forms/39x3vJqNomAome382](https://goo.gl/forms/39x3vJqNomAome382) is the link to the form, if you want to share with anyone else; I'm happy to have this spread around wherever.
# 800 scientist call out against statistical significance 

LessWrong has [many](https://www.lesswrong.com/posts/bXuAXCbzw9hsJSuEN/the-control-group-is-out-of-control) [criticisms](https://www.lesswrong.com/posts/ETe2SZacmLvvr8H9n/how-to-fix-science) [of](https://www.lesswrong.com/posts/XrxsFR2WLXWzgLEoy/case-study-abuse-of-frequentist-statistics) [P-values](https://www.lesswrong.com/posts/ttvnPRTxFyru9Hh2H/against-nhst)/[Statistical significance](https://www.lesswrong.com/posts/9qCN6tRBtksSyXfHu/frequentist-statistics-are-frequently-subjective), And is often given as a coordination problem (or an inadequate equilibria).

In addition to the American Statistical Association - which, In 2016, 'released a [statement in _The American Statistician_](https://amstat.tandfonline.com/doi/abs/10.1080/00031305.2016.1154108#.XInvWyPMzyW) warning against the misuse of statistical significance and _P_ values' - these authors now join the call too, with an article in nature signed by over 800 scientists. they write:

> We are far from alone. When we invited others to read a draft of this comment and sign their names if they concurred with our message, 250 did so within the first 24 hours. A week later, we had more than 800 signatories.

  

What do you think will be the impact of it?
# Did the recent blackmail discussion change your beliefs?

Various rationalist blogs and Less Wrong have recently posted on and discussed blackmail, and specifically legality and acceptability of such. I found the discussion unsatisfying, and I'm trying to understand why that is, and whether I'm alone in that.

As it was happening, it didn't feel like a particularly political topic - nobody seemed personally invested in the outcome. But it did seem like everyone (including myself, sometimes) was presenting examples or (over)generalizing to support their beliefs, and very few were seeking counterexamples or cruxes or lines of demarcation between different intuitions.

So - was this politics in disguise? Was some other bias interfering with the discussion? Was it useful and I just missed it? Did any sort of consensus emerge?
# Do you like bullet points?

I think more naturally in bullet points, and I (sometimes) like reading posts that are written in bullet style. ([This](http://bewelltuned.com/) [website](http://bewelltuned.com/) is one of my favorites, and is written entirely in bullets).

(Disclaimer, although I wrote this post in bullet points because it was cute, I don't think it's the best exemplar of them. Or rather, it's an example of using bullet points to do rough thinking, rather than an example of using them to illustrate a complex argument)

I like bullet points because:

*   _It's easier to skim, and build up a high level understanding of a post's structure._ If you understand a concept you can skip it and move on, if you want to drill down and understand it better you can do so.

*   Relatedly, it _exposes your cruxes_ more readily. You can pick out and refute points, in a way that can be harder with meandering prose.

*   _It's easier to hash out early stage ideas._  When I'm first thinking about something, my brain is jumping around and forming connections, developing a model at multiple levels of resolution. Bullet lists make this easier to keep track of.

*   I like this for _other_ _people's_ posts as well, since it feels more playful, like I can be part of their early generation process. I think LessWrong would be better if more people wrote more unpolished things to get early feedback on them, and bullet lists are a nice way to signal that something is still in development.

*   _Prose often adds unnecessary cruft._  In the transition from bullets-to-prose, posts can go 2x-3x as long (or, when I go to write a short bullet summary of something I wrote in prose, it turns out to be much shorter, and the prose mostly unnecessary)

I had assumed this was a common experience, and that it was in fact a weakness of humanity that we didn't have better, more comprehensive bullet-point tools.

But, alas, _Typical Mind Fallacy._ It turned out a couple people on the LessWrong team reacted very negatively to bullet points. Concerns include:

*   It's easy to think you've communicated more clearly than you have, because you didn't bother writing the connecting words between paragraphs.
*   They're _harder_ to read straight through. If you **include bold words,** readers might not bother reading the non-bold words, and miss nuance.
*   "I like numbered arguments, since that makes it easier to respond to individual points. But unnumbered bullet lists are just hard to parse."

*   _\[Alas, the LessWrong website currently doesn't enable this very well because our Rich Editor's implementation of numbered lists was annoying\]_

*   "I dunno man it's just really hard to read. My brain keeps trying to collapse the bullets like they're code."

I asked a couple more people, and they said "I dunno, bullet points seem fine. Depends on the situation?"

So...

I am curious what the LessWrong userbase thinks about them overall. Raise your hand if you think bullet points are fine? Terrible? Great? Any particular types of posts you prefer reading bullet-style, and types of posts you think fare poorly if not written in prose?
# "Moral" as a preference label

*Note: working on a research agenda, hence the large amount of small individual posts, to have things to link to in the main documents.*

In my [quest to synthesise human preferences](https://www.lesswrong.com/posts/qezBTig6p6p5xtL6G/a-theory-of-human-values), I've occasionally been asked whether I distinguish moral preferences from other types of preferences - for example, whether preferences for Abba or Beethoven, or avocado or sausages, should rank as high as human rights or freedom of speech.

The answer is, of course not. But these are not the sort of things that should be built into the system by hand. This should be reflected in the meta-preferences. We label certain preferences "moral", and we often have the belief that these should have priority, to some extent, over merely "selfish" preferences (the extent of this belief varies from person to person, of course).

I deliberately wrote the wrong word there for this formalism - we don't have the "belief" that moral preferences are more important, we have the meta-preference that a certain class of beliefs, labelled "moral", [whatever that turns out to mean](https://www.lesswrong.com/posts/EEPdbtvW8ei9Yi2e8/bridging-syntax-and-semantics-empirically), should be given greater weight. This is especially the case as there are a lot of cases where it is very unclear if a preference is moral or not (many people have strong moral-ish preferences over mainstream cultural and entertainment choices).

This is an example of the sort of challenges that a preference synthesis process should be able to figure out on its own. If the method needs to be constantly tweaked to get over every small problem of definition, then it cannot work. As always, however, it need not get everything exactly right; indeed, it needs to be robust enough that it doesn't change much if a borderline meta-preference such as "everyone should know their own history" gets labelled as moral or not.
# Unsolved research problems vs. real-world threat models

I've given a lightning talk twice now about "why should we care about adversarial examples?", highlighting the distinction (which ML experts are usually aware of, but media and policy folks often miss) between an unsolved research problem and a real-world threat model.

I've now published a version of this as a general-audience Medium post.

For a LessWrong / Alignment Forum audience, I'll highlight the fact that I'm directly and consciously drawing on the "intellectual puzzle" vs. "instrumental strategy" framing from Abram Demski's [Embedded](https://intelligence.org/2018/11/08/embedded-curiosities/) [Curiosities](https://intelligence.org/2018/11/08/embedded-curiosities/) post (at the conclusion of the [Embedded Agency](https://intelligence.org/embedded-agency/) series).

I'm mostly making an observation at the level of "how a research field works", and find it interesting that agent foundations and adversarial examples research are both hitting similar confusions when justifying the importance of their work.

I also think a lot more could be said and explored on the object-level about which angles on "adversarial examples" are likely to yield valuable insights in the long run, and which are dead ends. But this post isn't really about that.
# What are CAIS' boldest near/medium-term predictions?

**Background and questions**

Since Eric Drexler publicly released his “[Comprehensive AI services model](https://www.fhi.ox.ac.uk/wp-content/uploads/Reframing_Superintelligence_FHI-TR-2019-1.1-1.pdf)” (CAIS) there has been a series of analyses on LW, from [rohinmshah](https://www.alignmentforum.org/posts/x3fNwSe5aWZb5yXEG/reframing-superintelligence-comprehensive-ai-services-as#sXHXAfSKWPyEMhtcu), [ricraz](https://www.lesswrong.com/posts/HvNAmkXPTSoA4dvzv/comments-on-cais), [PeterMcCluskey](https://www.lesswrong.com/posts/bXYtDfMTNbjCXFQPh/drexler-on-ai-risk), and others.

Much of this discussion focuses on the implications of this model for safety strategy and resource allocation. In this question I want to focus on the empirical part of the model.

*   What are the boldest predictions the CAIS model makes about what the world will look in <=10 years?

“Boldest” might be interpreted as those predictions which CAIS gives a decent chance, but which have the lowest probability under other “worldviews” such as the Bostrom/Yudkowsky paradigm.

A prediction which _all_ these worldviews agree on, but which is nonetheless quite bold, is less interesting for present purposes (possibly something like that we will see faster progress than places like mainstream academia expect).

Some other related questions:

*   If you disagree with Drexler, but expect there to be empirical evidence within the next 1-10 years that would change your mind, what is it?
*   If you expect there to be events in that timeframe causing you to go “I told you so, the world sure doesn’t look like CAIS”, what are they?

  

**Clarifications and suggestions**

I should clarify that answers can be about things that would change your mind about whether CAIS is safer than other approaches (see e.g. the Wei_Dai comment linked below).

But I suggest avoiding discussion of cruxes which are more theoretical than empirical (e.g. how decomposable high-level tasks are) unless you have a neat operationalisation for making them empirical (e.g. whether there will be evidence of large economies-of-scope of the most profitable automation services).

Also, it might be _really_ hard to get this down to a single prediction, so it might be useful to pose a cluster of predictions and different operationalisations, and/or using conditional predictions.
# Announcing predictions

Today I unveiled `predictions`, a command-line program to score predictions you’ve written down in a YAML file. In my estimation, the program and its supporting documentation is alpha quality. You’ll need to build it yourself and the documentation isn’t all there and well-organized yet.

If you’re familiar with Go toolchains, have a look. I’d be happy to take feedback here in addition to on GitHub.

<https://github.com/adiabatic/predictions>
# Parfit's Escape (Filk)

_To the tune of "Escape" (The Piña Colada Song), with apologies to Rupert Holmes._

I was lost in the desert, hopelessly dying of thirst  
And I thought to myself, this can't get any worse  
I heard the roar of an engine, surely it was my hearse  
But then a tall shadow cooled me, it spoke and I heard

"If you like living not dying, I got an offer for you  
If you give me your money, I'll give you water and food  
I'll take you away to salvation, I'll get you out of this scrape  
So just promise you'll pay me, to make Parfit's escape"

I hadn't solved decision theory, boy I sure wish that I had  
'Cause my savior was Omega, and if I lied I'd be had  
But CDT said don't pay, just take the ride for free  
And though it seemed kind of foolish, I went ahead and agreed

"Yes, I like living not dying, so I'll make your offer good  
When we get back to town, I've got money for you  
I've got to get out of this desert now, I'm so tired of this place  
Yes I two box on Newcomb, and I take Parfit's escape"

Omega dragged me to the car, put a canteen to my lips  
We drove into the sunset, I felt nothing but bliss  
I caught a glimpse of his face, and to my wondering eyes  
My driver wasn't Omega, but Singer in disguise

"You don't owe me any money, you're the life I can save  
And you should know I would have helped you, if you had lied to my face  
I really hope you've learned a lesson though, about the perils of this place  
Decision problems are dangerous, it's moral luck you escaped"

I said "Oh thank you Mr. Singer, let me buy you a drink  
At a bar called O'Malley's, they make their Coladas pink  
'Cause I'm eternally grateful, for all that you've done  
Now let's get out of this desert, and be done with this song"
# Please use real names, especially for Alignment Forum?

As the number of AI alignment researchers increases over time, it's getting hard for me to keep track of everyone's names. (I'm probably worse than average in this regard.) It seems the fact that some people don't use their real names as their LW/AF usernames makes it harder than it needs to be. So I wonder if we could officially encourage people to use their real firstname and lastname as their username, especially if they regularly participate on AF, unless they're deliberately trying to keep their physical identities secret? (Alternatively, at least put their real firstname and lastname in their user profile/description?)
# Relative exchange rate between preferences

*Note: working on a research agenda, hence the large amount of small individual posts, to have things to link to in the main documents.*

In my [initial post](https://www.lesswrong.com/posts/Y2LhX3925RodndwpC/resolving-human-values-completely-and-adequately) on synthesising human preferences, I imagined a linear combination of [partial preferences](https://www.lesswrong.com/posts/CiB3myyeEhFRgmKPL/partial-preferences-and-models). Later, when talking about [identity preferences](https://www.lesswrong.com/posts/MxLK2fvEuijAYgsc2/smoothmin-and-personal-identity), I proposed a smoothmin instead (which can also be seen as strong diminishing returns to any one partial preference).

I was trying to formalise how humans seem to trade-off their various preferences in different circumstances. However the ideal is not to *a priori* decide how humans are trading off the preferences, but instead to copy how humans actually do trade off the preferences.

To do that, we need to imagine the human in situations quite distant from their current ones - situations where some of their partial preferences are more or less fulfilled. This brings up the problem of [modelling preferences in distant situations](https://www.lesswrong.com/posts/WAmpz8Z4FZ4FbCNtR/models-of-preferences-in-distant-situations). Assuming some acceptable resolution to that problem, the AI would have an exchange rate between different preferences, in different situations and with different levels of preference fulfilment.

Meta-preferences may constrain the preferences in very distant situations. Most paradoxes of  population ethics are in situations very different from today, so population ethics preferences can be seen in this way. Universal moral principles act in a similar way, giving limits on what can happen in all situations, including extreme ones - though note there are arguments to [avoid some unusual situations entirely](https://www.lesswrong.com/posts/WAmpz8Z4FZ4FbCNtR/models-of-preferences-in-distant-situations).

Then the AI's task is to come up with a general formula for the exchange rate between different preferences, that extends to all situations and respects the constraints of the meta-preferences. It will probably smooth out quite a bit of "noise" in the exchange rates between different preferences, while respecting the general trends.
# What LessWrong/Rationality/EA chat-servers exist that newcomers can join?

A few times over the past few years newcomers to the community have asked me how they can start engaging with the community, and one of the lowest-cost ways to engage has historically been chatrooms.

I know that there has been a pretty significant proliferation of LW/Rationality/EA-related chat-servers in the past few years, and it seems like a good idea to collate all of them into a central repository.

I know about [Elo's LW-Slack](https://wiki.lesswrong.com/wiki/Less_Wrong_Slack), and I know of the abstract existence of a bunch of EA-related Discord channels, but haven't been super active in this space, so I would appreciate other people's contributions.

If we get a good list, I think it would make sense to add that list to the community tab to make it easier to discover (or at least link this question from the community section).
# What are effective strategies for mitigating the impact of acute sleep deprivation on cognition?

I've recently been finding that I struggle much more with intellectual work (math, hard programming, writing, etc.) when I sleep less 6.5-7 hours. While I'm at peace with the fact that I seem to generally require >7 hours a sleep, it's frustrating that even though I set aside enough time for adequate sleep, I'll often wake up after only ~6 hours of sleep and not be able to fall back asleep. 

My cognitive ability seems to be impacted by a single night of bad sleep even when I've been sleeping well in the recent past. Concretely, if I've slept 8 hours every night for two weeks, a single night of poor sleep can still result in a ~50% less productive day. 

In addition to impacting productivity, acute sleep deprivation also leaves me much less capable of entertaining myself by thinking, so I become much more inclined to seek out distracting forms of entertainment like scrolling through the internet. It also seems to increase my cravings for generally "unhealthy" foods (I've seen references to this in literature, but won't bother linking them since it's not the focus of my question).

Other useful notes about my general sleep habits/history include:
* I'm not sure if I've always been this sensitive to sleep deprivation and just notice it more due to a combination of more introspective and spending more time on certain activities or if something's changed and I've become more sensitive.
* I generally have 1 cup of coffee in the morning around when I wake up. More cups of coffee do not seem to offset sleep deprivation's impact on my cognitive ability, and in fact have at times exacerbated it.
* I've tried napping when it's fit with my schedule and each time ended up lying awake for the 20-40 minutes during which I intended to nap.

I'd love to hear others' strategies for mitigating the impact of acute sleep deprivation on cognitive ability. I've done some preliminary searching for papers, articles, etc., but those that I've found focus on reducing tiredness rather than on returning cognitive ability to baseline. I'm open to trying strategies including but not limited to diet changes, supplements, medication, and habit changes.
# Open Thread April 2019

If it’s worth saying, but not worth its own post, you can put it here.

Also, if you are new to LessWrong and want to introduce yourself, this is the place to do it. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are welcome. If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), and [seeing if there are any meetups in your area](https://www.lesswrong.com/community).

The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# Experimental Open Thread April 2019: Socratic method

[This](https://www.lesswrong.com/posts/L4TKbfYsvxMJwNGTQ/experimental-open-threads) post was popular, but the idea never got picked up. Let's have an experimental open thread this month!

The rules:

> Top level comments would be claims. Second level comments would be discouraged from directly saying that someone is wrong and instead encouraged to ask them questions instead to get them to think

Let top level comments be debatable claims, first tier responses be questions, second tier answers, responses, answers, etc. Try to go as deep as possible, I'd expect an actual update to be increasingly likely to happen as you continue the conversation.
# Learning "known" information when the information is not actually known

Methods like [cooperative inverse reinforcement learning](https://arxiv.org/abs/1606.03137) assume that the human knows their "true" reward function $R(\theta)$, and then that the human and the robot cooperate to figure out and maximise this reward.

This is fine as far as the model goes, and can allow us to design many useful systems. But it has a problem: the assumption is not true, and, moreover, its falsity can have major detrimental effects.

Contrast two situations:

1. The human knows the true $R(\theta)$.
2. The human has a collection of [partial models](https://www.lesswrong.com/posts/CiB3myyeEhFRgmKPL/partial-preferences-and-models) in which they have clearly defined preferences. As a bounded, limited agent whose internal symbols are only [well-grounded](https://www.lesswrong.com/posts/EEPdbtvW8ei9Yi2e8/bridging-syntax-and-semantics-empirically) in [standard situations](https://www.lesswrong.com/posts/WAmpz8Z4FZ4FbCNtR/models-of-preferences-in-distant-situations), their stated preferences will be a simplification of their mental model at the time. The true $R(\theta)$ is constructed from some [process of synthesis](https://www.lesswrong.com/posts/Y2LhX3925RodndwpC/resolving-human-values-completely-and-adequately).

Now imagine the following conversation:

* **AI**: What do you really want?
* **Human**: Money.
* **AI**: Are you sure?
* **Human**: Yes.

Under most versions of hypothesis 1., this will be in a disaster. The human has expressed their preferences, and, when offered the opportunity for clarification, didn't give any. The AI will become a money-maximiser, and things go pear shaped.

Under hypothesis 2., however, the AI will attempt to get more details out of the human, suggesting hypothetical scenarios, checking what happens when money and other things in money's [web of connotations](https://www.lesswrong.com/posts/ix3KdfJxjo9GQFkCo/web-of-connotations-bleggs-rubes-thermostats-and-beliefs) come apart - eg "What if you had a lot of money, but couldn't buy anything, and everyone despised you?" The synthesis may fail, but, at the very least, the AI will investigate more.

Thus assuming the AI will be learning a truth that humans already know, is harmless assumption in many circumstances, but will result in disasters if pushed to the extreme.
# Prompts for eliciting blind spots/bucket errors/bugs

_This post is to make publicly available a few prompts/questions I came up with aiming to uncover blind spots around identity/self-concepts._

*   Select a trait X that you believe you have, and where you like that you have it (e.g. rational, kind, patient...)
*   Try to imagine a character that is a caricature of someone with trait X. Or another way to think about this: The way Spock is a Straw Man version of a rational character, what would a Straw Man version of a character with trait X look like? (referred to in the following as X-Spock)

*   What are blind spots an X-Spock is likely to have?
*   In what sorts of situations is an X-Spock especially likely to fail?
*   What would an X-Spock have a lot of trouble admitting to? (e.g. someone who considers themselves courageous may be unable to admit they are afraid)
*   What are traits that seem like opposites of X?

*   Could the opposite traits actually be beneficial?
*   Is what seems like an opposite trait in actuality orthogonal? (e.g. rational and emotional)