Harley-ml/lesswrong huggingface dataset:
# Two Small Experiments on GPT-2

I did two small experiments on the GPT-2 small model. First experiment: can GPT-2-small answer sentiment analysis questions? (It can't.) Second experiment: When GPT-2 writes continuations of Howl, is it picking up the "Moloch in X!" template from its priming, or from a copy of Howl in its original training set? (It's from the training set.)

Sentiment analysis experiment:

I downloaded the MPQA Subjectivity Lexicon, which is a dictionary in which words are marked as positive or negative. For example hopelessness=>negative, humour=>positive, grace=>positive, corruption=>negative. I primed GPT-2 with a list of 20 questions like "Is a <noun> good? Yes. Is a <noun> good? No." followed by an unanswered question of the same form, and had it continue for one more word. In its priming, half the answers were yes and the other half were no. It answered "No" 37/40 times, and neither its answers nor its yes answers were better than chance.

Howl experiment:

When given some lines from Ginsberg's Howl as priming, it writes a good continuation (similar to the one Chelsea Voss and Qiaochu Yuan got from it). In particular, it uses the "Moloch in X!" template repeatedly.

If I take its continuation of Howl and feed it back in as a prompt, I get more Howl (Moloch in X!). If I take Howl and replace "Moloch" with "Lomoch", I get more Howl. But if I take its continuation of Howl from the first step and replace Moloch with Lomoch \*there\*, I get unrelated text which does not use the "Moloch in X!" template.

So, it isn't inferring the template from its priming; rather, it learned the template from its training set (which probably included Howl), and it produces Howl-like text iff it's given a cue strong enough to remind it of the source.
# What's your favorite LessWrong post?

This was last asked in 2014 on the [annual survey](https://www.lesswrong.com/posts/oGezscrQvPDgGvrbt/2014-less-wrong-census-survey). Since we have strong upvotes, you may vote for 2 posts. Each answer should link to a single post, and I will delete repeats. Please use a strong and weak upvote each for at most one answer (and posting an answer is a vote).

Basically any LessWrong URL is ok - links, events, questions, and archive links to pages only on LessWrong 1.0 are fine. I only expect regular posts and links to be chosen though.

Feel free to comment on answers if you're dying to explain your reasoning, or comment on this post if you're dying to mention your third favorite and lower posts.
# If a "Kickstarter for Inadequate Equlibria" was built, do you have a concrete inadequate equilibrium to fix?

[Yoav Ravid asks](https://www.lesswrong.com/posts/98HihPxpoRFja2JNF/is-there-an-assurance-contract-website-in-work): "Is there an assurance-contract website in work?"

i.e. a site where, if there's a locally bad equilibrium that would be better if _everyone_ changed strategies at once, but which requires a critical mass of people in order to be worthwhile, you can all say "I'll put the effort if other people put in the effort", and then if X people agree, you all go into work the next day and demand a policy change, or a go to a political rally, or change a social norm, or whatever.

Some attempts have been made at such a system. It's not that technically hard to build. But I think it'd need a couple major "flagship" Coordinated Actions in order to rally people's attention and turn it into a more frequently used tool.

So, _if_ a good website existed to coordinate action, do you have a well operationalized action you'd want to coordinate? ("Everyone leaves Facebook at once" doesn't work IMO, because it doesn't say where people are moving _to_, or otherwise replacing FB's tools with.

"Everyone on one platform switches to another platform" seems viable.

"Everyone at my office signs a letter demanding change for a particular policy" seems viable (although in cases like this, where you maybe _don't_ want your boss to know you're planning a revolution, and I'm not sure how to best achieve common knowledge without risk)

(For further reading, see "T[he Costly Coordination Mechanism of Common Knowledge](https://www.lesswrong.com/posts/9QxnfMYccz9QRgZ5z/the-costly-coordination-mechanism-of-common-knowledge)" and "[Inadequate Equilibria](https://www.lesswrong.com/sequences/oLGCcbnvabyibnG9d)")
# How could "Kickstarter for Inadequate Equilibria" be used for evil or turn out to be net-negative?

Following up to "[If a "Kickstarter for Inadequate Equlibria" was built, do you have a concrete inadequate equilibrium to fix?](https://www.lesswrong.com/posts/fQkzGfRoL82XX4cPr/if-a-kickstarter-for-inadequate-equlibria-was-built-do-you)"

I _think_ a kickstarter for coordinated action would be net positive, but it's the sort of general purpose powerful tool that might turn out bad in ways I can't easily predict. It might give too much power to mobs of people who don't know what they're doing, or have weird/bad goals.

How bad might it be if misused? What equilibrias might be we end up in in the world where everyone freely has access to such a tool?
# Rationalist Vipassana Meditation Retreat

Vipassana: Silent Meditation with group exchange for and with rationalists! Details see flyer: http://www.das-buddhistische-haus.de/pages/images/stories/dokumente-deutsch/Vipassana_retreat_Julia_Harfensteller_BH_12062019.pdf (the flyer has different dates, it's for a different retreat with the same content)

Full 10 day version: Friday, 26.04.2019 16:00 – Sunday, 05.05.2019 15:00
Or 3 day version: Friday, 26.04.2019 16:00 – Monday, 29.04.2019 15:00

Location: Edelhofdamm 54, 13465 Berlin (http://www.das-buddhistische-haus.de)

There are 14 spots available for on-site accomodation, first come first served. There's nearby housing if these spots are taken, the course has a limit of 40 people.

For any questions join: https://mattermost.gitlab.rlp.net/signup_user_complete/?id=iacjkp5yppfftqr3k738y1k7kc – I am organizing this event alone, so my goal is to minimize the organization overhead for me. This means that I would highly appreciate your help. Please post all questions and communication in the mattermost channel, and help each other answering questions. All further communication of the event will be at the mattermost channel, please make sure to regularly check and/or turn on email notifications for channel mentions for announcements.

Course schedule:
# Friday, 26.04.2019
16:00 – 17:00 Arrival
17:00 – 18:00 Welcome
18:00 – 19:00 Dinner
19:00 – 20:15 Introduction, evening meditation

# Saturday, 27.05.2019 – Saturday, 04.05.2019
05:30 – 07:00 Mindful movement and meditation, optional
07:00 – 08:30 Breakfast
08:30 – 09:30 Mindful activity
10:00 – 12:30 Meditation and lecture
12:30 – 14:30 Lunch
14:30 – 16:45 Mindful movement and meditation
17:00 – 18:00 Dinner
18:00 – 20:00 Meditation, review of the day

# Sunday, 05.05.2019 (or Monday, 29.04.2019 with the 3 day version)
05:30 – 07:00 Mindful movement and meditation, optional
07:00 – 08:30 Breakfast
08:30 – 09:30 Mindful activity
10:00 – 12:30 Meditation and review
12:30 – 15:00 Lunch, cleaning, packing, leaving
# Disclosing the unsaid

My colleagues often mock me for my faith in human sciences, "Studying cultures and people in the age of algorithms, you just need data and machine learning model" they said. I've been an engineer by profession for a decade, but to develop new perspectives on the characteristics of our product, I interpret data points and anthropological input across information systems.

I try not to see people as numbers, but as individuals. Choosing a group and then understanding them should be the goal. All great companies are based on an insight into how to serve a group of people in a meaningful way. Looking at and listening to people is better than asking them.

My job is to listen, not just to people but also to the world they make for themselves, their tone, intentions, and moments of silence. Listening requires more than just registering sounds. It means decoding them, from the most distinct, to the quiet and the obscure. I stay in the unfamiliar for as long as something is meaningful. I’ve found that when exploring the unknown, learning to decode its sounds pays off.

Like learning a new language it involves building sensitivity towards the words, interactions, and the gestures which give rhythm to a culture. In my work listening is not passive. It takes risks to get real answers. The key for me is learning how to maintain an insider-outsider role - keeping up the rhythm of a conversion while at the same time being able to take a step back to interpret, probe, and hear what’s not being said.

The most exciting part of the work is its dynamic element. I’ve travelled around the world and seen my hypotheses turned on their heads. The critical factor - the linking piece often ends up being a surprise - is something I identify only when running back through the data and seeing it in the context of the whole. This whole is the entire experience of the culture, the people I’ve met, and the odd things that were mentioned in passing. Sometimes, one phrase can make everything else make sense.

Studying people is not a linear process. It is often blurry, ambiguous, and yields responses slowly. But I always leave the field feeling like I’ve seen something true. The reality of the place or a phenomenon dons on me later; not in a flash of insight but through the rigorous process of creating meaning.
# How good is a human's gut judgement at guessing someone's IQ?

In her latest post, Sarah Constantin writes:

> Whatever ability IQ tests and math tests measure, I believe that lacking that ability doesn’t have _any _effect on one’s ability to make a good social impression or even to “seem smart” in conversation.

I remember that I made the opposite prediction a few months ago, while I was studying psychometrics more actively. In particular, I had the sense that because of the positive manifold (i.e. almost all positive mental attributes quite strongly correlate), it seems like judging someone's IQ should be relatively easy, and also quite valuable in assessing a large number of other positive qualities, which should make it quite evolutionarily advantageous to be able to assess it.

Two concrete experiments I've thought about:

1\. Take a group of people whose IQ you now, then just take pictures of their faces (or short videos of them saying something), and then have a group of participants rate them on their intelligence (probably in order). See how strong they correlate.

2\. Throw a bunch of participants into a group, have them talk to each other, then afterwards ask them about the relative judgement of the intelligence of the other members of the group, see how predictive they are.

These seem like very straightforward experiments, that I can imagine someone having already run, and where I would be pretty interested in the results. If only so that I can be better calibrated on how much to trust my gut judgement of someone's intelligence.

Does anyone know of any similar experiments that have been run?
# New versions of posts in "Map and Territory" and "How To Actually Change Your Mind" are up (also, new revision system)

A few weeks ago [MIRI released](https://www.lesswrong.com/posts/NjFgqv8bzjhXFaELP/new-edition-of-rationality-from-ai-to-zombies) new editions of the "Map and Territory" and "How To Actually Change Your Mind" books, and those changes are now finally also reflected in [the posts on LessWrong](https://www.lesswrong.com/rationality).

To make that happen smoothly, I've also implemented a revision and versioning system, allowing you to view the past revisions of all the posts in those books, and marking comments made against older versions of the post. I did this because a significant fraction of the discussion in the comments was referencing problems in the posts that are addressed in the latest versions, as well as wording that has been changed.

A post having past revisions is indicated by a small history-icon next to its date, which when clicked will allow you to select from all past versions of the post after it has been published. This is currently only available on desktop, the mobile version will follow soon.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1551151437/Screen_Shot_2019-02-25_at_7.23.28_PM_hacncy.png)

We are planning to make the revision system available to everyone soon, which will allow you to uniquely link to any article in its current version, without having to worry that it will ever change. I hope to also make it so that you can get notifications to new major versions of posts that you are subscribed to. This is part of a broader effort to make posts on LessWrong longer-lived and have more of a wiki-feeling to them, and also to make it harder to accidentally delete your whole post with a wrong submission.
# Where to find Base Rates?

As we are all aware from _Thinking, Fast and Slow,_ one of the largest decision-making errors we make is the [Base Rate Fallacy](https://en.wikipedia.org/wiki/Base_rate_fallacy). Just to remind you - the error is that we don't correctly apply probability reference classes to ourselves.

Two examples:

*   The likelihood of winning the lottery is _x,_ but I am more likely to win than _x_.
*   1 in _x_ people will get lung cancer from smoking, but I won't.

This is all well and good, but my question is where _you_ go about actually finding _x_. In other words - where do you find your base rates? Any particular repositories of knowledge that we should know about? Any processes that we should follow?

Let's say you've decided you want to reduce your risk of getting Coronary Heart Disease. Is your first stop to go to WHO and gather their list of recommendations, or something else?
# RAISE is launching their MVP

After over a year of [building the organisation](https://www.lesswrong.com/posts/ATQ23FREp9S4hpiHc/raising-funds-to-establish-a-new-ai-safety-charity) and working and reworking the product, I’m glad to announce that [RAISE](https://aisafety.info/) will begin publishing lessons on a regular basis starting next Monday.

These lessons will include:

*   An Inverse Reinforcement Learning (IRL) track, sourced from a set of papers that we have distilled by our own efforts.
*   A track on Iterated Distillation and Amplification (IDA), largely adapted from Paul Christiano’s LessWrong sequence on the matter.

A lesson on the former will be published [on our platform](https://app.grasple.com/#/course/141?access_token=3HCK4oRipeFY2ghyYMqDJKYX57KUnzNL) every Monday, and a batch of lessons on the latter will be published every Thursday. The former will be finished in approximately 10 weeks, the latter in 4. If you plan to take these lessons, we recommend scheduling one full day per week for each track you’re following. Once published, the lessons will be available indefinitely for self-paced study.

In the meantime, we continue to semi-regularly publish textbook guides for our prerequisites track. These guides should eventually cover all the prerequisite material one should understand in order to properly follow work in AI Safety.

In the coming months, our focus will be on testing and refining our MVP. As long as we operate, we hope to make learning AI Safety less and less costly, creating common knowledge and paying our [research debt](https://distill.pub/2017/research-debt/).

To do this testing properly, we are seeking individuals that would like to join our testing panel. This means that, as you go through our lessons, we will guide you through the lessons and interview you in-depth about your experience, most importantly about where you’re most confused. If you would like to join us, please fill in [this form](https://goo.gl/forms/rUAEWU6cSjQGnzEw1).  
  
This work has been sponsored, among others, by the [EA Hotel](http://eahotel.org). If you like our work, consider [donating](http://eahotel.org/fundraiser/) there.
# Is LessWrong a "classic style intellectual world"?

Robin Hanson just published a post titled "[Classic Style Intellectual World](http://www.overcomingbias.com/2019/02/classic-style-intellectual-worlds.html)". It's short, so I recommend reading it, as it sets up this question, but in short it suggests that many intellectual activities are organized in a way similar to the way paintings on the walls of Renaissance churches are. That is, small spaces are allocated to lower-status artists where they are rewarded mainly for blending in well with architecture and surrounding context and only already high-status artists are asked to be more creative by giving them large spaces where they might do something less conforming (but not too non-conforming!). The parallel he draws is that lower-status intellectuals are expected to talk within the frame society and other intellectuals have already set up for them within a frame sized to match their status, while higher-status intellectuals are allowed a larger frame in which they may be more creative, the point being that status drives permitted creativity (that is, creativity that will not be ignored in obscurity) rather than creativity and innovation (or whatever traits are ostensibly sought in intellectuals) being what confers high status.

So the question I pose here is, how much does LW look like this picture Robin has painted? Specifically, is it that high readership, karma, post scores, etc. accumulates to authors who are the best at whatever it is you think the values of LW are or do those things go to those who best work "within the system" with LW value satisfaction as secondary concern?
# Have a camel!

1.

The East is rich and her tales abundant, and here is but one of them.*  
  
  
Once upon a time, a man lost his camel and went about looking for it. He met three boys whom he asked if they had seen the animal.  
  
"Is your camel pregnant?" asked the first boy.  
  
"Why, so she is!" said the owner, surprised.  
  
"Does she have only one eye?" asked the second boy then.  
  
"Indeed she does!" said the owner, now hopeful.  
  
"And does she have a sore on her back?" asked the third boy at last.  
  
"A big one!" cried the owner happily. "Where has she gone?"  
  
"Sorry," said the children. "But we haven't seen her."  
  
To make an Eastern story short... er, let us turn to the end where the man demanded from a third party to make the boys tell him where to find the camel.  
  
This is how it went:  
  
Boy 1: "\[O ... emir,\] I saw the camel's tracks and where it urinated, and so knew it was pregnant."  
  
Boy 2: "\[O ... emir,\] I saw that the camel only ate grass from one side of the road, and so knew it was one-eyed."  
  
Boy 3: "\[O ... emir,\] I saw some of the grass that the camel had rubbed against and knew there was a sore on its back."  
  
Emir: "Thanks everybody, you may go. And you may keep looking for your animal."  
  
  
2.  
  
Compare this to the infamous tale of blind men groping an elephant.  
  
The observers had a single common object to describe. They did not exactly see it. Every one of them had a single piece of intelligence to add to the pile, inferred from a single kind of study apiece. Their answers did not depend on the order or completeness of other reports, even when it could be done.  
  
Yet in case of the elephant, which was known to be the one and only thing to study from the outset till the end, there was much dissent and no synthesis.  
  
In case of the camel, which was only confirmed to be the same animal by the outsider who had lost it, the researchers never argued.  
  
So why do some people lose elephants when other people only lose camels?
# What would a zetetic explanation be for the rationality community?

Zetetic explanations as described by benquo ([blogpost](http://benjaminrosshoffman.com/zetetic-explanation/), [lw linkpost](https://www.lesswrong.com/posts/i2Dnu9n7T3ZCcQPxm/zetetic-explanation)).

*   Interdisciplinary

*   covers a mixture of social and natural factors leading up to the isolation of the thing
*   historical in the original, investigative sense of the term "history."
*   risks being wrong

*   Empowering

*   integrates concrete and model-based thinking that is checkable on multiple levels
*   affirms basic competence of humans to explore our world
*   centers process of discovery rather than a finished product

  

The rationality sequences created a community with shared language for talking about mind and optimization space. If we have that history understood on a technical level, I think we'll be better at seeing through founder effects to recognize ally and sister movements forming in other contexts.
# Karma-Change Notifications

Starting this weekend, LessWrong will be displaying karma notifications in the top-right corner, telling you about when you've been upvoted or downvoted. You can click the star icon to see which of your posts and comments have been voted on, and how much their score has changed.

This works a little differently from how most web sites do it. I've noticed a tendency, in myself and others, to sometimes obsessively refresh pages hoping to have gotten Likes. The rest of the LessWrong team has noticed this too. That's why on LessWrong, by default, this won't work; karma-change notifications are grouped into daily batches, so after you've checked it, you won't be notified of any additional votes until the next day.

While I hope that this prevents people from using LessWrong in ways that they don't endorse, daily batching might not be a strong enough safeguard for everyone. If you find yourself tempted to check LessWrong more often than you think is ideal, you can change the karma-notifier batches to weekly, or disable them entirely. On the other hand, if checking and posting on LessWrong is something you'd rather do more of (because you'd otherwise do something less valuable), you can set it to real-time.

As a nonprofit with no advertising, LessWrong is under a different set of incentives than most websites. We worry a lot about the trade-off between [engagement](https://www.lesswrong.com/posts/2E3fpnikKu6237AF6/the-case-for-a-bigger-audience) and [addictiveness](https://www.lesswrong.com/s/HXkpm9b8o964jbQ89/p/ENBzEkoyvdakz4w5d). I want people to use the site in ways they reflectively endorse, even if that means using it less. I think LessWrong should leave that under user control, as much as feasible.

(As a reminder: Voting on LessWrong is a way of saying "I want to see more/less things like this". Upvotes are a way to tell people that their post or comment was worth their time to write, and worth your time to read. It does not necessarily indicate agreement; sometimes a good write-up of an ultimately incorrect idea is what a conversation needs. Conversely, downvotes do not necessarily indicate disagreement; sometimes a correct points is written in a way that's confusing, inflammatory or otherwise detracts from the conversation.)

As always, bug reports are welcome here, via the Intercom widget, or on the [GitHub issue tracker](https://github.com/LessWrong2/Lesswrong2/issues).
# LW Update 2019-01-03 – New All-Posts Page, Author hover-previews and new post-item

Daily page replaced with new "All-posts" page
---------------------------------------------

The old `/daily` page (accessible via clicking "Daily" on the frontpage, or clicking on "All Posts" in the hamburger menu in the top-left) has been replaced with a [new `/allPosts` page](http://lesswrong.com/allPosts) that is showcasing a bunch of new designs that we are planning to roll out across the site over the next few weeks.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1551550617/Screen_Shot_2019-03-02_at_10.16.16_AM_ktnrii.png)

Note in particular the changes to the post-preview, which is now reduced to one line and puts all the key information into easily skimmable columns. We also got rid of our beloved little title-column things on the left side, and moved everything into a central column in preparation for some upcoming changes to the overall site structure.

By default the page will function just like the old `/daily` page, but with optional additional functionality in the form of new filters and sorting options (this now also finally allows you to easily sort by "total karma" again).

Author hover-preview
--------------------

If you hover over the author names of comments and posts, you can now see a small preview of their bio, the time they signed up and the number of posts and comments that they've written. The goal is to make it easier to figure out whether an author has written other content you might want to read, and whether the user you are looking at is new, which helps you set expectations and hopefully encourages people to help new users get onboarded.

Karma notifications
-------------------

Jim talked a bit more about karma notifications over [in another meta post](https://www.lesswrong.com/posts/yb3Js7ArenCiiHxKF/karma-change-notifications). I think this is a pretty important feature, and am interested in people's thoughts on the new notifications.
# To understand, study edge cases

This seems obvious, but is often forgotten. When a system functions efficiently, its individual parts are hard to notice. If you want to understand how a system functions, what it is made of, how each part works, but you cannot easily disassemble it and have a look, you have to get out of the nominal regime and let each part function independently, as much as possible.

Some examples:

*   To understand subatomic particles in a nucleus, or to figure out the structure of atom, hit them hard with other particles and see what happens.
*   To understand free will, focus on the situations where the notion fails.
*   To understand consciousness, study altered states.
*   To understand mental health, study mental illness.
*   To understand anatomy, dissect cadavers.
*   To understand logical reasoning, study biases.
*   To understand evolution, study artificial selection.
*   To understand economics, study market failures.
# How much funding and researchers were in AI, and AI Safety, in 2018?

I'm trying to build up a picture of how "much" research is going into general AI capabilities, and how much is going into AI safety.

The _ideal_ question I'd be asking is "how much progress \[measured in "important thoughts/ideas/tools" was being made that plausibly could lead to AGI in 2018, and how much progress was made that could plausibly lead to safe/aligned AI\].

I assume that question is nigh impossible, so instead asking the approximation:

a) how much money went into AI capabilities research in 2018

b) how much money went into AI alignment research in 2018

c) how many researchers (ideally "research hours" but I'll take what I can get) were focused on capabilities research in 2018

d) how many researchers were focused on AI safety in 2018?
# IRL 1/8: Inverse Reinforcement Learning and the problem of degeneracy

Every Monday for 8 weeks, we will be posting lessons about Inverse Reinforcement Learning. This is lesson 1.

Note that access to the lessons requires creating an account [here](https://app.grasple.com/#/course/141?access_token=3HCK4oRipeFY2ghyYMqDJKYX57KUnzNL).

To help students get up to speed, we have also published 2 supplementary lessons that cover prerequisite material:

*   [The POMDP](https://app.grasple.com/#/level/1326)
*   [Bellman equations](https://app.grasple.com/#/level/1492)

Have a nice day!
# Asking for help teaching a critical thinking class.

I will soon be teaching a "critical thinking" class for undergraduates. Feel free to mentally replace "critical thinking" with "epistemic virtue". I would appreciate answers to any of these questions:

What would you do if you only had one three hour class period to teach a group of 15-30 undergrads "critical thinking"?

If you know me (Ronny Fernandez) what one educational objective would you give me, knowing it will not be my only one, to come up with a plan to achieve?

How would you given three hours or less make it so that 15-30 students are all absolutely sure of something, and then realize they were all wrong, without feeling like them feeling like you had cheated or done anything shady?

What are the most transferable rationality skills that are abundant in our community and rare elsewhere and how do you transfer them in a classroom setting?

How do you teach the things in the vicinity of scout mindset vs warrior mindset, arguments as soldiers, politics is the mind killer, without making your student temporarily dumber by giving them a fully general excuse to not engage with any disagreement they feel like ignoring?

How do you teach the difference between the kind of cognition you use to figure out how to get to your friends house and the kind of cognition Malfoy used in hpmor to think about the heritability of magic?

How do you teach the virtue of lightness, the virtue of curiosity?

How do you teach fallacies and cognitive biases without making your students temporarily dumber by giving them a fully general excuse to disregard any position or thinker they disagree with?
# Open Thread March 2019

If it’s worth saying, but not worth its own post, you can put it here.

Also, if you are new to LessWrong and want to introduce yourself, this is the place to do it. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are welcome. If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), and [seeing if there are any meetups in your area](https://www.lesswrong.com/community).

The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# Smoothmin and personal identity

I value existential risk reduction; I also value poverty reduction. These two things trade off against each other.

I value being generous; I also value reading interesting books. These two things trade off against each other.

But the way the two tradeoffs work do not seem to be the same. For the first one, I feel comfortable having a utility $U_{\neg X}$ (for no existential risks) and $U_{\neg P}$ (for no poverty), and then weighting them and maximising the sum:

* $\lambda_{\neg X} U_{\neg X} + \lambda_{\neg P} U_{\neg P}$.

If this ends up with me only reducing existential risks, or only reducing poverty, then that's fine, I'm working on the option with the most marginal impact.

For the second one, I would not want to maximise some sum $\lambda_G U_{G} + \lambda_R U_{R}$, and would certainly complain if I ended up never reading again, or never being generous again. I'd prefer to maximise something like the [smooth minimum](https://en.wikipedia.org/wiki/Smooth_maximum) of $U_G$ and $U_R$, something like:

* $\left(\frac{\lambda_G e^{-U_G}}{e^{-U_G}+e^{-U_R}}\right) U_G + \left(\frac{\lambda_R e^{-U_R}}{e^{-U_G}+e^{-U_R}}\right) U_R$.

And I'd want the weights to be chosen so that I am very likely to both be generous and read, to some extent, over longer periods of time.

# World preferences vs identity preferences

Some time ago, I [wrote a post](https://www.lesswrong.com/posts/kgQCpiuHwDWNgz7E4/preferences-over-non-rewards) about "preferences over non-rewards". I'm planning to collect most of these preferences into the category of "personal identity": the sort of being you want to be.

The "You're not the boss of me!" preference from that post - meaning you change your preferred action because you were told to/told not to - is very similar to the "4 Problems with self-referential $\Theta$" from [this post](https://www.lesswrong.com/posts/Y2LhX3925RodndwpC/resolving-human-values-completely-and-adequately), and will be both be grouped under "personal identity".

It's my hope that all human preferences and meta-preferences can be synthesised into one of "world preferences" or "personal identity preferences". As this post suggests, the methods of aggregation may be different for the two categories.
# IRL 2/8: Mitigating degeneracy: multiple experimentation

  

Every Monday for 8 weeks, we will be posting lessons about Inverse Reinforcement Learning. This is lesson 2.

Note that access to the lessons requires creating an account [here](https://app.grasple.com/#/course/141?access_token=3HCK4oRipeFY2ghyYMqDJKYX57KUnzNL).

Have a nice day!
# Blegg Mode

Fanfiction for the [blegg/rube parable](https://www.lesswrong.com/posts/4FcxgdvdQP45D6Skg/disguised-queries) in ["A Human's Guide to Words"](https://www.lesswrong.com/posts/FaJaCgqBKphrDzDSj/37-ways-that-words-can-be-wrong), ~800 words. (_Content notice_: in addition to making a point about epistemology (which is why it may have been worth sharing here), this piece is also an obvious allegory about a potentially [mindkilling](https://www.lesswrong.com/posts/9weLK2AJ9JEt2Tt8f/politics-is-the-mind-killer) topic; read with caution, as always.)
# Example population ethics: ordered discounted utility

*This article is a stub. Alas, you can't help Wikipedia (or LessWrong) by expanding it. Except through good comments.*

Here I'll present an old idea for a theory of population ethics. This post exists mainly so that I can have something to point to when I need this example.

Given a total population $\Omega$, each with total individual utility $u_i$ over the whole of their lives, order them from lowest utility to the highest so that $i<j$ implies $u_i \leq u_j$. These utilities are assumed to have a natural zero point (the "life worth living" standard, or similar).

Then pick some discount factor $0<\gamma<1$, and define the total utility of the world $W$ with population $\Omega(W)$ (which is the total population of the world across all time) as

* $U(W) = \sum_{i=1}^{||\Omega(W)||} \gamma^i u_i$.

This is a [prioritarian](https://en.wikipedia.org/wiki/Prioritarianism) utility that gives greater weight to those least well off. It is not average utilitarianism, and would advocate creating a human with utility larger than than all other humans (as long as it was positive), and would advocate against creating a human with negative utility (for a utility in between, it depends on the details). In the limit $\gamma\to 1$, it's total utilitarianism. Increasing someone's individual utility always improves the score. It (sometimes) accepts the ["sadistic conclusion"](https://www.cambridge.org/core/services/aop-cambridge-core/content/view/S0266267100000249), but I've argued that that [conclusion is misnamed](http://blog.practicalethics.ox.ac.uk/2014/02/embracing-the-sadistic-conclusion/) (the conclusion is a choice between two negative outcomes, meaning that calling it "sadistic" is a poor choice - the preferred outcome is not a good one, just a less bad one). Killing people won't help, unless they will have future lifetime utility that is negative (as everyone that ever lived is included in the sum). Note that this sets up a minor asymmetry between not-creating people and killing them.

Do I endorse this? No; I think a genuine population ethics will be more complicated, and needs a greater asymmetry between life and death. But it's good enough for an example in many situations that come up.
# If you wrote a letter to your future self every day, what would you put in it?

Several days ago, I wrote an email to myself.

That email will now be sent to me every day.

All it is is a single draft in my Gmail drafts folder, with the [Mail Conductor](https://www.mailconductor.com/) extension sending it out at 10:00 am. I can modify the draft whenever I want, each time [improving](https://jamesclear.com/continuous-improvement) it.

Consider, with the fervent munchkinry of a final exam... What would _you_ send yourselves?

(Helpful anchor point: What would you share with a guaranteed audience of thousands of cooperative strangers who thought _very much_ \- but not _quite_ totally - like you?)
# The tech left behind

Hello, I am asking for some insights for a research I am doing. Can you cite examples of technologies that have been forgotten? What I mean by "forgotten" is not things we don't know how to do but used to (I suspect there aren't that many), nor things that are no longer in use but used to (mechanical television), but things that were decently developed (either in theory or in practice) but never "saw the light of day" anyway.

It's my first time posting, so I won't do much policing on the answers, thanks in advance.
# You Get About Five Words

_Cross posted from the [EA Forum](https://forum.effectivealtruism.org/posts/BSJbPZP9AAyivpmBE/you-have-four-words)._

_Epistemic Status: all numbers are made up and/or sketchily sourced. Post errs on the side of simplistic poetry – take seriously but not literally._

* * *

If you want to coordinate with one person on a thing about something nuanced, you can spend as much time as you want talking to them – answering questions in realtime, addressing confusions as you notice them. You can trust them to go off and attempt complex tasks without as much oversight, and you can decide to change your collective plans quickly and nimbly.

You probably speak at around [100 words per minute](https://wordcounter.net/blog/2016/06/02/101702_how-fast-average-person-speaks.html). That's 6,000 words per hour. If you talk for 3 hours a day, every workday for a year, you can communicate 4.3 million words worth of nuance.

You can have a [real conversation with up to 4 people](https://www.psychologytoday.com/us/blog/out-the-ooze/201707/what-is-the-right-size-group-conversation).

(Last year the small organization I work at considered hiring a 5th person. It turned out to be very costly and we decided to wait, and I think the reasons were related to this phenomenon)

* * *

If you want to coordinate on something nuanced with, say, 10 people, you realistically can ask them to read a couple books worth of words. A book is maybe 50,000 words, so you have maybe 200,000 words worth of nuance.

Alternately, you can monologue at people, scaling a conversation past the point where people realistically can ask questions. Either way, you need to hope that your books or your monologues happen to address the particular confusions your 10 teammates have.

* * *

If you want to coordinate with 100 people, you can _ask_ them to read a few books, but chances are they won't. They might all read a few books worth of stuff, but they won't all have read the same books. The information that they can be coordinated around is more like "several blogposts." If you're trying to coordinate _nerds_, maybe those blogposts add up to one book because nerds like to read.

* * *

If you want to coordinate 1,000 people... you realistically get one blogpost, or maybe one blogpost worth of jargon that's hopefully self-explanatory enough to be useful.

* * *

If you want to coordinate thousands of people...

You have about five words.

This has ramifications on how complicated a coordinated effort you can attempt.

What if you _need_ all that nuance _and_ to coordinate thousands of people? What would it look like if the world was filled with complicated problems that required lots of people to solve?

I guess it'd look like this one.
# LW Update 2019-03-12 -- Bugfixes, small features

We've deployed an updated version of LessWrong, with a bunch of bug fixes and small features added since last time (March 1).

Features and Changes

 * Auto-linkify URLs in posts and comments
 * Coauthored posts appear on your user profile page
 * Drafts that are shared with you appear in the Drafts section of the user profile page
 * Negative-karma change notifications don't use red color
 * Posts on the All Posts page have a menu on the right

Bug Fixes

 * Address some compatibility issues with Opera Mini, Edge, and IE11
 * Remove broken New Users Notification and New Posts Notification settings
 * Table of Contents: Empty blocks can't be headings
 * Remove spurious All Posts View setting from user profile page
 * Fix a visual glitch during loading (SSR) of the all-posts page
 * Fix a bug where the RSS synchronization cron job would sometimes stop running
 * Don't redundantly show the AF icon on alignmentforum.org
 * Remove a duplicate Load More button on the user profile page
 * Fix a crash in the Sunshine Sidebar with reports of already-deleted comments
 * Various fixes to search indexing
 * Fix a security vulnerability that could reveal unpublished drafts

API Changes

 * Add localStartTime and localEndTime resolvers to posts, which return the time of a meetup in the time zone of the meetup location (null if no location specified)

Technical

 * Clean up sequences
 * Infrastructure for validating database contents against schemas
 * Optimize some slow mongodb queries
 * Migration infrastructure for dealing with denormalized fields
 * robots.txt now blocks crawlers from our test servers (no change on the main site)
# [Link] OpenAI LP

https://openai.com/blog/openai-lp/

> We’ll need to invest billions of dollars in upcoming years into large-scale cloud compute, attracting and retaining talented people, and building AI supercomputers.

> We want to increase our ability to raise capital while still serving our mission, and no pre-existing legal structure we know of strikes the right balance. Our solution is to create OpenAI LP as a hybrid of a for-profit and nonprofit—which we are calling a “capped-profit” company.

> Going forward (in this post and elsewhere), “OpenAI” refers to OpenAI LP (which now employs most of our staff), and the original entity is referred to as “OpenAI Nonprofit.”
# Verifying vNM-rationality requires an ontology

**Result**  
It is impossible to verify that an agent is [vNM-rational](https://wiki.lesswrong.com/wiki/Expected_utility) by observing its actions without access to the domain of its utility function.

  
**Motivation**  
Alphonso and Beatriz both go the market to buy fruit.  
Alphonso prefers grapes to oranges.  
He fills his basket with grapes and pays for them.  
  
Beatriz carefully picks through the fruit and purchases some oranges and some grapes.  
Callisto arrives with a package of grapes.  
  
"Say, Beatriz, would you like to trade some of your oranges for this package of grapes?" Callisto offers.  
  
"Gladly." Beatriz replies, exchanging some of her oranges for the grapes.  
  
A few moments later, Alphonso notices Beatriz giving Deion some grapes in exchange for some oranges.  
  
"You are acting irrationally, Beatriz!" Alphonso exclaims. "Your unstable preference between oranges and grapes makes it possible for a malicious agent to exploit you and exhaust your entire grocery budget!  
  
"Ah, but I am acting rationally." Beatriz replied with a smile. "I prefer fruit that is fresh enough to last more than seven days. Thus, I trade away fruit that will spoil before that time."  
  
**Explanation**  
Consider an agent A.  
We are interested in verifying whether or not A is vNM-rational.  
However, we are only able to observe A's decisions without any access to the domain of A's utility function.  
  
Without this access, it is impossible to distinguish between vNM-irrational choices (i.e. choices that violate one of the axioms of vNM-rationality) and choices that are vNM-rational but made under an unexpected ontology.

  
In other words, we need to know how A perceives outcomes of the world before we can verify that A's preferences over those outcomes are vNM-rational.
# Distribution of info-cascades across fields? [Info-cascade series]

_This is a question in_ _[the info-cascade question series](https://www.lesswrong.com/posts/2uDBJWCksvzhDzHGf/understanding-info-cascades). There is a prize pool of up to $800 for answers to these questions. See the link above for full background on the problem (including a bibliography) as well as examples of responses we’d be especially excited to see._

\_\_\_

How common, and how large, are info-cascades in communities that seek to make intellectual progress, such as academia? This distribution is presumably very heavy-tailed as we are dealing with network phenomena. But what is its actual values? How can we estimate this number?

A good starting point for thinking about this might be the paper “How citation distortions create unfounded authority: analysis of a citation network” ([Greenberg, 2009](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2714656/)), which uses social network theory and graph theory to trace how an at best very uncertain claim in biomedicine cascading into established knowledge. We won’t attempt to summarise the paper here.
# How can we respond to info-cascades? [Info-cascade series]

_This is a question in_ _[the info-cascade question series](https://www.lesswrong.com/posts/2uDBJWCksvzhDzHGf/understanding-info-cascades). There is a prize pool of up to $800 for answers to these questions. See the link above for full background on the problem (including a bibliography) as well as examples of responses we’d be especially excited to see._

\_\_\_

In my (Jacob's) work at [Metaculus AI](https://goo.gl/forms/gMGGGtGYHztYJJ2s2), I'm trying to build a centralised space for both finding forecasts as well as the reasoning underlying those forecasts. Having such a space might serve as a simple way for the AI community to avoid runway info-cascades.

However, we are also concerned with situations where new forecasters overweight the current crowd opinion in their forecasts, compared to the underlying evidence, and see this as major risk for the trustworthiness of forecasts to those working in AI safety and policy.

With this question, I am interested in previous attempts to tackle this problem, and how successful they have been. In particular:

* What existing infrastructure has been historically effective for avoiding info-cascades in communities? (Examples could include short-selling to prevent bubbles in asset markets, or norms to share the causes rather than outputs of one’s beliefs)

* What problems are not adequately addressed by such infrastructure?
# How large is the harm from info-cascades? [Info-cascade series]

_This is a question in_ _[the info-cascade question series](https://www.lesswrong.com/posts/2uDBJWCksvzhDzHGf/understanding-info-cascades). There is a prize pool of up to $800 for answers to these questions. See the link above for full background on the problem (including a bibliography) as well as examples of responses we’d be especially excited to see._

\_\_\_

How can we quantify the impact (harm) of info-cascades?

There are many ways in which info-cascades are harmful. Insofar as people base their decisions on the cascaded info, this can result in bad career choices, mistaken research directions, misallocation of grants, a culture that is easier to hijack by cleverly signalling outsiders (by simply “joining the bubble”), and more.

But in order to properly allocate resources to work on info-cascades we need a better model of how large the effects are, and how they compare with other problems. How can we think about info-cascades from a cost-effectiveness perspective?

We are especially interested in answers to this question that ultimately bear on the effective altruism/rationality communities, or analyses of other institutions with insights that transfer to these communities.

As an example step in this direction, we built [a Guesstimate model](https://www.getguesstimate.com/models/12929), which is described in an answer below.