Harley-ml/lesswrong huggingface dataset:
# April Fools: Announcing: Karma 2.0

Ever since we started work on the new LessWrong, improving the karma system has been top priority for us.

We started giving users a higher vote-weight when they themselves had accumulated enough karma, and have plans to implement various more complicated schemes, such as Eigenkarma. But we recently realized that more important than how karma is calculated, is how karma actually influences the user _experience_.

The key purpose of a karma system is to allocate the attention-bandwidth on the site towards users who make the most valuable contributions. Historically we have done this via sorting, but sorting is only a small fraction of how we can possibly allocate attention. And as soon as we realized this, the next step was obvious.

**Adjust the size of your content, based on your karma**

This means, as you get more karma on the site, your comments _literally get larger_. At the beginning, you will be a mere dust-speck among giants in the comments, but after you engage with the site for a bit, your words can tower over those of your contemporaries.

This changes _everything_. Vertical space on the page is now properly allocated according to our best guess about what content you will want to read. People's influence and history on the site is clearly communicated to anyone skimming a comment thread. Competing with your friends for karma now literally translates to determining who has the larger "D". The positive effects are too numerous to all list exhaustively here.

We believe this truly revolutionizes the way karma works on LessWrong, which is why we are proud to call this new system "Karma 2.0". We also believe there are many more promising improvements in this direction. We are soon planning to experiment with coloring your comments green or red based on the ratio of upvotes to downvotes your content received, and adjusting the line-height of your posts based on our estimate of how vacuous your claims are (to properly signal to the user the correct ratio of content to "hot air"). Stay tuned for the upcoming "Karma 2.1" and "Karma 2.2" which will implement these features.

However, if for some inscrutable reason, you want to disable Karma 2.0, you can do so by editing your profile (click on your username and then click "Edit Account") and checking the "Deactivate Karma 2.0" checkbox.

_Signed Oliver Habryka, Ben Pace, Raymond Arnold, and Matthew 'Vaniver' Graves_
# Announcing Rational Newsletter

Hi,

I noticed in the past there were a few efforts to start a periodic feed with highlights from the rationalist community.

Here's my attempt: [Rational Newsletter](https://rationalnewsletter.com)

It's a weekly recap of the best articles from the rationalist community of LessWrong, Slate Star Codex, Overcoming Bias, Intelligence.org and similar sources.

All posts are curated by hand, and going forward I'm planning to keep ~5 posts per category, short and sweet. I wanted to ask for your feedback:

*   Do you like the format?
*   Do you like the classification and the quantity of the articles?

Any feedback and criticism is super welcome!
# Why Karma 2.0? (A Kabbalistic Explanation)

It was Easter Sunday, the obvious time to [improve](https://www.lesswrong.com/posts/K3qnYdBGjzPiBrPzS/april-fools-announcing-karma-2-0) the karma system.

Why? Well, firstly, why do we even have a karma system? You might think that the obvious explanation is Buddhist karma, but the Bible can in fact explain this fact all by itself.

In the **West**, the word ‘karma’ derives from the biblical word ‘Carmel’. Carmel means ‘garden’, and Mt. Carmel is the place where the prophet Elijah brought back the people to their allegiance to God [via empiricism](https://www.lesswrong.com/posts/fAuWLS7RKWD2npBFR/religion-s-claim-to-be-non-disprovable) \- testing whether he or the Priests of Baal would be able to cause God to set their bulls afire. As such, Karma is a natural gardening tool for a community that is built around empiricism.

Secondly, why did karma need to connect to various levels of size? Well, Easter is the holiday of rebirth.

In the **East**, rebirth is closely connected to karma - the quantity of karma that one has determines what life form you will become in your next life. In Buddhism, there are six realms of rebirth. In decreasing order of size, there are the heavenly, demigod, human, animals, ghosts, and residents of hell. Similarly on LessWrong, we [presently](https://www.lesswrong.com/posts/fPT7o9TRWobrXoiYJ/voting-weight-discussion) have six different classes of karma, and all users fit into them. So the karma quantity should determine which of the six sizes you fit into.

That is why on Easter Sunday it was necessary that we changed the gardening tools on this site-of-empiricism to alter the users into six sizes.
# Unyielding Yoda Timers: Taking the Hammertime Final Exam

[The Hammertime Final Exam](https://www.lesswrong.com/posts/Q7MsMshzbzhEs729s/hammertime-final-exam)
====================================================================================================

**Difficulty: Vorpal Dragonscale Sledgehammer of the Whale**

> Write all three. For each essay, give yourself five minutes to brainstorm and five minutes to write. < refresh to render LaTeX >

Preflection
-----------

_Design an instrumental rationality technique._

I have one "spirit" which I regard as being the best version of myself and which I can trust to reliably make good choices. Unfortunately, when I'm making decisions, another spirit is often more or less in charge. One approach to mitigating this is making a TAP wherein you simulate my _post facto_ opinion of the (usually bad) choice you're about to make. This interpolates between your current spirit and the normatively-correct one. Pretending to be someone else / another version of yourself is, in my experience, surprisingly effective.

Confusion Identification
------------------------

_Introduce a rationality framework._

You're doing your homework, and you're stuck. You don't know what to do next.

One approach is to set a Yoda Timer for 5 minutes; write down a broad-strokes explanation of the problem, where you are presently, and what a solution would look like. Then go to a friend (or, if you're alone, an imaginary friend) and explain it. They don't have to understand, but _you_ have to understand what you're saying while you're saying it. You don't have to have a gears-level understanding of the solution (yet), but you should have a System 1 understanding of your outline.

Be on the lookout for any mental bumps, any areas you gloss over with a trace of discomfort. These dark crevices beckon to you; into them you must descend if you are to emerge solution in hand.

Emotion Propagation
-------------------

_Describe a cognitive defect._

At my undergraduate institution, I was incredibly social - I eagerly crossed cliques to make new friends wherever I went. This demeanor carried over to my first trimester of graduate school; then, I became depressed.

After taking a few months leave, I managed to fully recover. I returned to my graduate institution and found my desire to socialize was neutered. This problem plagued me for nearly a year; it is only within the last month that I pinpointed the cause. My attitude towards interpersonal interaction never left the < refresh to render LaTeX > state.

Just as the human mind doesn't enforce belief consistency, it won't necessarily propagate attitudinal changes to relevant areas of your life. Do you currently value everything as you would reflectively prefer, or are there holdovers from the past?

* * *

< refresh to render LaTeX > Several additional minutes were taken for formatting.
# Specification gaming examples in AI

\[Cross-posted from [personal blog](https://vkrakovna.wordpress.com/2018/04/02/specification-gaming-examples-in-ai/)\]

Various examples (and [lists](https://www.gwern.net/Tanks#alternative-examples) [of](https://www.alexirpan.com/2018/02/14/rl-hard.html) [examples](https://arxiv.org/abs/1803.03453)) of unintended behaviors in AI systems have appeared in recent years. One interesting type of unintended behavior is finding a way to game the specified objective: generating a solution that literally satisfies the stated objective but fails to solve the problem according to the human designer’s intent. This occurs when the objective is poorly specified, and includes reinforcement learning agents [hacking the reward function](https://arxiv.org/abs/1606.06565), evolutionary algorithms gaming the fitness function, etc. While ‘specification gaming’ is a somewhat vague category, it is particularly referring to behaviors that are clearly hacks, not just suboptimal solutions.

Since these examples are currently scattered across several lists, I have put together a [master list](https://docs.google.com/spreadsheets/d/e/2PACX-1vRPiprOaC3HsCf5Tuum8bRfzYUiKLRqJmbOoC-32JorNdfyTiRRsR7Ea5eWtvsWzuxo8bjOxCG84dAg/pubhtml) of examples collected from the various existing sources. This list is intended to be comprehensive and up-to-date, and serve as a resource for AI safety research and discussion. If you know of any interesting examples of specification gaming that are missing from the list, please submit them through this [form](https://docs.google.com/forms/d/e/1FAIpQLSeQEguZg4JfvpTywgZa3j-1J-4urrnjBVeoAO7JHIH53nrBTA/viewform).

Thanks to Gwern Branwen, Catherine Olsson, Alex Irpan, and others for collecting and contributing examples!
# Realistic thought experiments

What if…

…after you died, you would be transported back and forth in time and get to be each of the other people who ever lived, one at a time, but with no recollection of your other lives?

…you had lived your entire life once already, and got to the end and achieved disappointingly few of your goals, and had now been given the chance to go back and try one more time?

…you were invisible and nobody would ever notice you? What if you were invisible and couldn’t even affect the world, except that you had complete control over a single human?

…you were the only person in the world, and you were responsible for the whole future, but luckily you had found a whole lot of useful robots which could expand your power, via for instance independently founding and running organizations for years without your supervision?

…you would only live for a second, before having your body taken over by someone else?

…there was a perfectly reasonable and good hypothetical being who knew about and judged all of your actions, hypothetically?

…everyone around you was naked under their clothes?

…in the future, many things that people around you asserted confidently would turn out to be false?

…the next year would automatically be composed of approximate copies of today?

…eternity would be composed of infinitely many exact copies of your life?

\*\*\*

(Sometimes I or other people reframe the world for some philosophical or psychological purpose. These are the ones I can currently remember off the top of my head. Several are not original to me. I’m curious to hear others.)
# Death in Groups

What is the value of your life? Not _a_ life, but _your_ life. Would you consider yourself expendable?

The last time I almost died, it was over a missing shotgun. I was in Afghanistan at the time, somewhere closer to the border with Pakistan. The roads are rough, particularly when there aren't any, and during our travels we would occasionally have to stop and re-secure our cargo. During one such stop, it seems a weapon was left on the bed of the truck and then fell off somewhere. It was discovered the weapon was missing, and then we were ordered to go back out and find it.

It's important to recognize early on that there was no expectation of success here - this was a punishment. It was more than three hours' travel, through mountains and river valleys. Spotting a shotgun through the armored window was not likely. Up the chain of command went the observation that driving up and down the same road a bunch of times is a bad plan considering we were in Afghanistan. Down the chain of command came the orders to go out anyway. It was clear that we were expected to operate continuously until we were attacked, or the higher ups relented. Everyone knew. Everyone knew that everyone knew. It was [common knowledge](https://www.lesserwrong.com/posts/9QxnfMYccz9QRgZ5z/the-costly-coordination-mechanism-of-common-knowledge). These are moments that test men and their oaths: we looked at one another in grimly, seeing the hubris and stupidity at work; there was talk of refusing to go.

We went.

The fourth time we covered the ground, ~300lbs of home-made explosive went off under the back tire of the truck I was in. Where there should have been a sound there was a mighty shockwave, and everything went silent and in slow motion. The world was queerly bright, and I wondered why the smell of dust was so strong. Then I saw my arms floating in front of me, and the tied down equipment only held by its ties; I knew what had happened, and what was next. I thought very clearly: "This part is going to suck."

Everybody lived - the new trucks have all kinds of tricks for dispensing with kinetic energy, so every nut and bolt on the thing blew off and they found one of the tires 400 meters away. Those weighed 700lbs or so, with the wheel in them; we made a game of trying to flip one in our camp, and perhaps a quarter succeeded. Three of us left in a helicopter, myself included. One or more of us would probably have died but for a series of recent changes: we stopped having a rear gunner (three weeks); we left the emergency hatches cracked in order to avoid overpressure (two weeks); we made sure our seatbelts were as tight as possible (that day). Whether we _would_ die was not subject to our control. That missing shotgun can be had for about $650. Last I checked - _really_ checked - that was at the object level what I was willing to lay down my life for.

Guess the number that is 2/3 of the average of all guesses. Among rational guessers, that number is 0.

Gamble your life for the value that is less than the average of all gambles. Among warriors, that value is nothing.
# LW Update 04/06/18 – QM Sequence Updated

I just updated the [Quantum Mechanics Sequence](https://www.lesswrong.com/sequences/Kqs6GR7F5xziuSyGZ) to have working images, links, and formatting.

Thanks to all the readers who pinged me in intercom to let me know that the stuff was broken for them, it caused me to do this more urgently.

Also a big thanks to Said Achmiz who already had turned Rationality: AI to Zombies into html and turned the images into SVG files, this saved me a \*ton\* of time.

If you take a look through, please let me know if I missed something.
# Announcing the Alignment Newsletter

I've been writing weekly emails for the Center for Human-Compatible AI (CHAI) summarizing the content from the last week that's relevant to AI alignment. These have been useful enough that I'm now making them public! You can:

*   Sign up [here](http://eepurl.com/dqMSZj)
*   Read the first newsletter [here](https://mailchi.mp/ff6340049bd0/alignment-newsletter-1)
*   Bookmark the [archive](http://rohinshah.com/alignment-newsletter/)
*   ETA: Read the 5 emails I sent to CHAI (also on the [archive](http://rohinshah.com/alignment-newsletter/))
*   ETA: There's also [RSS](https://us18.campaign-archive.com/feed?u=1d1821210cc4f04d1e05c4fa6&id=dbac5de515)!
# The Chromatic Number of the Plane is at Least 5 - Aubrey de Grey

This is a long standing open problem in math. Many people learn about it as early as high school as an example of an open problem, because it is so easy to state:

How many colors does it take to color every point in the plane so that ever pair of points of distance exactly 1 from each other have different colors?

There are simple proofs that anyone here can understand that this number is between 4 and 7, and those were the only bounds we knew for a long time.

The lower bound was improved to 5 recently by Aubrey de Grey! The same Aubrey de Grey that you probably think of as the face of solving aging! He points at a concrete subset of 1567 points in the plane that cannot be colored with four colors.

This is super exciting. I think this is basically the main example of a simple open problem in math that we (used to) have no progress on.

I hope Aubrey de Grey negotiated the moral trade with the mathematicians successfully, and now that he solved one of their most beloved problems, they will start working on solving aging.
# Two clarifications about "Strategic Background"

  

I've talked to a few people who misunderstood important parts of the "strategic background" discussion in [https://intelligence.org/2017/12/01/miris-2017-fundraiser/#3](https://intelligence.org/2017/12/01/miris-2017-fundraiser/#3).

First, at least two people thought the 1-8 numbered list was "MIRI's organizational plan" rather than "what we'd be least surprised to see happen in the world, conditional on good outcomes." MIRI is trying to de-confuse itself about step 8 and help put AGI developers in a better position in the future to select for AGI designs that are alignment-conducive, not trying to develop AGI.

Second, at least two other people misread "minimal aligned AGI" as "minimally aligned AGI", and thought MIRI was saying that developers should do the bare minimum of alignment work and then deploy immediately; or they saw that we were recommending building "systems with the bare minimum of capabilities for ending the acute risk period" and thought we were recommending this as an _alternative_ to working really hard to achieve highly reliable and robust systems.

The MIRI view isn't "rather than making alignment your top priority and working really hard to over-engineer your system for safety, try to build a system with the bare minimum of capabilities". It's: "in addition to making alignment your top priority and working really hard to over-engineer your system for safety, _also_ build the system to have the bare minimum of capabilities".

The idea isn't that you can get away with cutting corners on safety by keeping the system weak; per Eliezer's [security mindset posts](https://intelligence.org/2017/11/25/security-mindset-ordinary-paranoia/), a good plan should work (or fail safely) if the system ends up being a lot smarter than intended. Instead, the idea is that shooting for the bare minimum of capabilities adds a lot of value if your fundamentals are really good. Every additional capability a developer needs to align adds some extra difficulty and additional points of failure, so developers should target minimality in addition to alignment.
# Dire Bullshit

When I bought my kid a door jumper (basically a bowl with leg holes dangling from a spring that could be attached to a doorframe) I read all the safety warnings.

![](http://alicorn.elcenia.com/images/warnings.png)

Here they are.

In case you don't want to read all of them, there's a bunch of obviously legitimate and _potentially_ legitimate safety warnings, and there's:

"NEVER use optional playmat with any product other than the door jumper."

To be clear, the described item is a rectangle. Of vinyl-or-something-backed fabric.

And this throws all the other safety warnings into question. They're taking their opportunity to warn me about _danger to my child_, and they're using it to sneak in _imperatives about combining a harmless rectangle with other objects_. This doesn't happen when you buy a yard of muslin from the store because you don't expect a yard of muslin to come with any safety warnings at all. But when you look to a source for advice, and have reason to take it seriously, it may accumulate cruft:

*   Recipes on food packaging that call specifically for their brand, of that ingredient and any others the company sells. (If you've been carefully purchasing NESTLE® TOLL HOUSE® chocolate chips because you believe they're essential to the recipe, good news, they are not.)
*   Institutions like banks and insurance companies, which _might_ mail you important information you'd want to have, and _might_ just be stamping "URGENT" on the envelopes because this improves some metric by getting you to open their random crap.
*   It's happened to me in a medical setting, ROT-13 if you want the extra example; cbfgcneghz, n ahefr gbyq zr gung rirel gvzr V hfrq gur onguebbz V unq gb fcenl jngre ba zlfrys nf n fbeg bs znxrfuvsg ovqrg. "Sbe ubj ybat?" V nfxrq, nffhzvat guvf unq gb qb jvgu urvtugrarq puvyqovegu-eryngrq vasrpgvba evfx. "Nyy gur gvzr," fur fnvq, "Sbe gur erfg bs zl _yvsr_?" V fnvq, naq fur abqqrq, naq V arire yvfgrarq gb nalguvat fur fnvq rire ntnva.

This is epistemic poison, but it's all over the place. Attention is memetic energy; memes will evolve to grab it as necessary. Refer your friends (through privileged one to one channels they don't tune out yet, of course) to gain in-game prizes! The next train is in two minutes on platform one, and remember, if you see something, say something. And we learn to saccade over ads and prune our Twitter feeds and filter safety warnings with our own judgment about the hazard level of vinyl-backed fabric rectangles so we can slog on through polluted informational waters, throwing out a few babies with the bathwater because the bathtub specified a brand of soap before reminding us not to bathe children near open windows.
# Remembering the passing of Kathy Forth.

It is with sadness and regret of the loss of life that I inform you that Kathy Forth has passed away.

She will be remembered at the solstices and forever in our hearts as someone trying to make the world a better place in the ways they believe are most important.

This post is to serve as a eulogy. I know she didn't always agree and she didn't always make friends. Despite that she was a part of this community. I would appreciate anyone interested in saying kind words to do so below. Anyone unwilling to say kindness can keep it to themselves.

[http://effective-altruism.com/ea/1n7/remembering\_the\_passing\_of\_kathy_forth/](http://effective-altruism.com/ea/1n7/remembering_the_passing_of_kathy_forth/)

[https://www.facebook.com/forthk/posts/2077196849230469](https://www.facebook.com/forthk/posts/2077196849230469)

[https://www.facebook.com/Kathleen-Rebecca-Forth-Memorial-Page-184870828823873/?modal=admin\_todo\_tour](https://www.facebook.com/Kathleen-Rebecca-Forth-Memorial-Page-184870828823873/?modal=admin_todo_tour)

Requiem Mass service to be held on Saturday April 21, 2018 at 10:00 am CST at the following location:

The Church Of St. Francis  
Liberal Catholic Church  
12 School St.  
Villa Park, Il. 60181  
(630) 592-8482
# Github Repo and Issue Tracker

LessWrong 2.0 is open source. We wanted to have a handy link for people who want to check out the codebase, add new issues to the tracker, or potentially get involved with development.

The [git repository is here](https://github.com/Discordius/Lesswrong2).

You can [see our issue tracker here](https://github.com/Discordius/Lesswrong2/issues).

We're not currently set up well for "Open Source at scale". We have some plans to write up a more comprehensive guide to help people get oriented, but it's a fair amount of work and we most likely won't get to it in the immediate future.

For now this post is just a quick stub, but I'll update it with further information once I get some time.
# Oops Prize update

I'm overdue to publish an update on the [Oops Prize](http://benjaminrosshoffman.com/oops-prize/). It fell off my priority list because I received exactly one nomination. I followed up with the nominee and couldn't get enough clarification to confirm eligibility, but my sense was that while the nominee clearly changed their mind, it wasn't a particularly clear case of public error correction as specified in the prize criteria.

Since the Oops Prize remains unclaimed, I'm offering it again this year. To clarify, I don't think the prize amount is enough to incentivize overt error-correction on its own, but it _might_ be enough to give people an incentive to bother informing me if such error correction is in fact happening.

If anyone at an EA Global conference this year publicly repudiates an old belief, and the efforts they made and asked others to make on this basis, and explained what they're doing differently, then I'd like to celebrate this. Since talk is cheap, I'm offering $1,000 in prize money for the best example of such error-correcting; $900 to the person who most clearly reports changing their mind about something big they’d already invested their time or money or credibility in and asked others to invest in, and $100 to the first person to nominate them. Self-nomination is encouraged.

To qualify, an entry has to have the following attributes:

*   It is explicitly error correction, not an account that spins things to look like a series of successes evolving over time, or "I used to think X, and now I think X'."
*   The nominee successfully encouraged a public commitment of resources based on the original belief (e.g. funds raised or volunteer hours).
*   There is a record of the error-correction statement. If it's not a recorded talk, an independent witness (neither the nominator nor prizewinner) is enough evidence.
*   It happened at EA Global, and either was part of a scheduled talk, or an independent witness (neither the nominator nor the nominee) believes that at least ten people were present.

Anyone who speaks at EA Global this year is eligible for the prize, including leaders of EA organizations such as CEA, EAG leadership, and GiveWell / Open Philanthropy Project staff. If no qualifying entries are submitted, then no prize will be awarded. I am the sole, unaccountable judge of this, but will get people to check my work if I don't think anyone's eligible or feel like I'm too close to the person I think should win.

You can send nominations to me by email at benjaminrhoffman@gmail.com. If the error-correction is already publicly available, or if the nominee gives me permission, I’ll announce the winner by the end of the year. If there is no public recording and the nominee isn’t OK with the error-correction being publicized in this way, then I reserve the right to award them only a partial prize or none at all.
# Rationality and Spirituality - Summary and Open Thread

I've noticed that there has recently been a lot of interest in spirituality, ritual and meaning within the rationality community. I could highlight the discussions of Jordan Peterson ([SSC](http://slatestarcodex.com/2018/03/26/book-review-twelve-rules-for-life/), [SSC Follow Up](http://slatestarcodex.com/2018/04/03/highlights-from-the-comments-on-twelve-rules/), [Put A Num On It](https://putanumonit.com/2018/03/03/jordan-peterson/)), Valentine's recent posts ([Fake Frameworks](https://www.lesswrong.com/posts/wDP4ZWYLNj7MGXWiW/in-praise-of-fake-frameworks), [Kensho](https://www.lesserwrong.com/posts/tMhEv28KJYWsu6Wdo/kensh), [The Intelligent Social Web](https://www.lesserwrong.com/posts/AqbWna2S85pFTsHH4/the-intelligent-social-web), [Mythic Mode](https://www.lesserwrong.com/posts/HnWN6v4wHQwmYQCLX/mythic-mode), [Kaj Sotala's Response](https://www.lesserwrong.com/posts/mELQFMi9egPn5EAjK/my-attempt-to-explain-looking-insight-meditation-and)), discussions on Meditation ([Mastering the Core Teachings of the Buddha](http://slatestarcodex.com/2017/09/18/book-review-mastering-the-core-teachings-of-the-buddha/), [Gupta on Enlightenment](http://slatestarcodex.com/2018/04/19/gupta-on-enlightenment/), [Five Years After Starting Mindfulness Meditation](https://www.lesswrong.com/posts/WwaMBvG5uh9dfA8rb/some-simple-observations-five-years-after-starting)) and a few others ([Funeral Ritual](https://www.lesserwrong.com/posts/viZxDr2jgHhE2iCor/funeral-ritual), [Open-Source Monasticism](https://www.lesserwrong.com/posts/bNACATPYmTz6Xhg6f/open-source-monasticism)). I thought I would encourage this trend making this post as this seems like an interesting direction for exploration and I suspect more progress occurs when multiple people develop an interest in a topic at the same time than when they all explore individual directions.

This is an open thread, so you can post whatever you want about this topic, but here's a few possible discussion directions:

*   Have you made any attempt to explore any kind of spirituality in the broadest sense? If so, what have you learned?
*   In the past, attempts to engage with post-rationality didn't seem to be particularly successful. Perhaps now is a better time?
*   What are the core rationality posts or resources on this topic? What are some resources from outside the rationality community that we may find helpful?

(I'm too lazy, but the links above could form the basis of a Wiki post)
# EA Global SF 2018

The goal of EA Global is to increase attendees’ knowledge, skills and networks to enable them to do the most good possible.

Content will be aimed at existing EA community members who already have a solid understanding of effective altruism, but would like to gain skills, network, master more complex problems, or move into new roles.

**Programming will include:**

*   Presentations of papers on the latest ideas at the frontiers of the EA movement
*   Whiteboard sessions (collaborative discussions between experts)
*   Seminars and workshops to help improve your thinking and execution
*   Facilitated networking sessions, office hours and social activities

**Who is** **EA** **Global a good fit** **for?**

We think the following kinds of attendees are particularly likely to find the conference valuable:

*   People already familiar with the key ideas of effective altruism
*   People seeking the opportunity to master more complex problems in effective altruism, learn new skills or get help from the community
*   People actively seeking work focused on [high-priority global issues](https://e-a.io/causes)

We use an application process to match our available spaces with the people we think will gain the most from the conferences.

Applications
------------

**Apply to attend the conference:** [EA Global SF application](https://eaglobal.typeform.com/to/Z3RaRG)

**Apply to volunteer**: [EA Global volunteer application](https://docs.google.com/forms/d/e/1FAIpQLSeEsauZhxmonB-WG3W3w3YzlayT-TUBG7yeg2od0DEmMWzQCA/viewform)
# [Alert] Running Final Migration (possible downtime)

_\[Update: lesswrong.com is now running as its own container and you should be able to use either URL again. There are some remaining tasks to be done that might result in small snags\]_

_\[Update 2: You can log in via google, facebook or github on lesswrong.com now, but be aware this will create a new user, not allow you to log into your old user account\]_

Today, we're doing a final bit of migration from lesserwrong.com to lesswrong.com. (Currently, lesswrong.com works to access the site, but is not actually the primary domain. This results in a few issues, including our Google and Github logins not working properly)

So today at 3pm, I'll be doing the last steps of the migration, and during a 10 minute window, you will _not_ be able to use lesswrong.com, but _will_ be able to continue using less_er_wrong.com.

Afterwards, all lesserwrong.com links will redirect to lesswrong.com, and hopefully you will be able to log in via Google, Github, and Facebook without issue.

Apologies for any inconvenience.
# The 3% Incline (theferrett.com)

I liked this recent post by The Ferrett (a blogger who discusses a number of issues including gaming, polyamory, handling mental illness, politics and various kinds of relationships).

In particular, this seemed like a clear encapsulation of one of the concepts I touch on at [Sunset at Noon](https://www.lesswrong.com/posts/2x7fwbwb35sG8QmEt/sunset-at-noon), without all the surrounding baggage.

The rationality community has this vague undercurrent narrative of "Self Improvement! You can Radically Change! All problems are fixable! Growth Mindset!". And this isn't _false_ per se, but the tone of it often sort of implies overnight change.

I think occasionally people can change radically, but this most often seems to have to do with a change in environment (new job, new house, new community, etc). Changing environments is pretty expensive and a bit of a crapshoot.

For personal, dedicated self improvement, a more realistic rate-of-return is something like 3% (not quite sure on what timescale this means. "Per month" feels a bit too short and "per year" feels a bit too long. Maybe per quarter of 6-month interval?)

In Sunset at Noon I had referred to this as "the 2% incline". I like Ferrett's post as a canonical Post Referring To This Concept, and had sort of picked 2% out of a hat, so I'm editing the original post to use 3%.
# Does Thinking Hard Hurt Your Brain?

This is a "typical mind fallacy check" post. Curious how much (within and without the rationalsphere) people's experience varies.

I generally experience "thinking hard" to be some combination of stressful, headache inducing, and energy draining (sometimes I feel like I actually am burnt out of energy, sometimes I just need to switch tasks). I've talked to other rationalists and they often don't have this experience, and I'm trying to figure out what's going on here.

I specifically experience this when doing conscious, deliberate thought. In particular if it strains the bounds of my current skills, or my working memory. This most often involves thinking strategically in a careful way (i.e. I get this when playing Chess).

When I'm writing characters or imagining talking to people I know (i.e. simulating another person in my head, or pretending to be another person) I also get a headache if I keep it up for over an hour.

I've only talked to a few people about this, so not sure how wide the spread of experience is. But at the very least this varies a bit among people-I-know.

Some hypotheses (or partial hypotheses) so far:

**A. Deliberate Practice is straining**. Doing deliberate-practice thinking (i.e. straining at the edge of your competence of a skill) is energy draining, and people vary on a) what things strain the edge of their competence, and b) how often they do such things.

**B. Straining is wasted motion**. Someone I know recently argued that thinking shouldn't drain willpower or otherwise cost resources other than time, and that any time you do that, it's because you're doing some wasted motion. This initially sounded wrong to me. After experimenting a bit I at least believe that _much_ of "draining" thinking is wasted motion that you can learn to skip.

**C. People vary in raw cognitive power.** Some people may just think faster naturally, or have a higher bandwidth of working memory.

For now, mostly interested in getting a sense of what diversity of experience we have of this on LessWrong. I also suspect there's some research into this _somewhere_ and am curious if anyone either knows about that, or has further thoughts on mechanisms.
# Give praise

The dominant model about status in LW seems to be one of relative influence. Necessarily, it's zero-sum. So we throw up our hands and accept that half the community is just going to run a deficit.

Here's a different take: status in the sense of _worth._ Here's a set of things we like, or here's a set of problems for you to solve, and if you do, you will pass the bar and we will grant you personhood and take you seriously and allow you onto the ark when the world comes crumbling. Worth is positive-sum.

I think both models are useful, but only one of these models underlies the _emotional need_ we call status. I think it's the latter.

Another assumption: humans are satisficers. Those that claim to the contrary have never been satisfied. An unsatisfied satisficer acts like a maximizer. I think that Maslov got pretty close with his hierarchy of needs. Not the specific set of needs, not necessarily their order, but the idea of humans working on one need at the time, until satisfaction, so that the next need comes up.

It seems to me that many of us are stuck at the status level, and I think getting past it makes us surely happier and plausibly more effective.

How is worth _generated?_ Quite simply, by giving praise. You either find your behavior exceeding a standard that the community agreed on, or someone _actually tells you you're doing well._ The latter seems more powerful.

I've asked around, and it even seems to be considered "normal" in most (non-systematizing) communities to habitually give praise. It's even apparently something people regard as _necessary_ for proper psychological health_._ But honestly, apart from volunteering at CFAR, I can't recall getting much praise for anything I've done for the community. As a result I never quite feel like I'm doing enough, edging on burnout a few times. Reminds me of pica. Does working on AI Safety _ever_ get me the sense of worth I'm looking for, or should I give up?

So I'd like to suggest we try for Giving Praise as a rationalist virtue. It might just be a staple of group rationality.
# Soon: a weekly AI Safety prerequisites module on LessWrong

(edit: we have a study group running a week ahead of this series that adds important content. It turns out that to get that content ready on a weekly basis, we would have to cut corners. We prefer quality over speed. We also like predictability. So we decided to cut us some [slack](https://www.lesswrong.com/s/HXkpm9b8o964jbQ89/p/yLLkWMDbC9ZNKbjDG) and publish every 2 weeks instead for the time being)

Hey there! It's been about 6 weeks since RAISE [doubled down](https://www.lesswrong.com/posts/ATQ23FREp9S4hpiHc/raising-funds-to-establish-a-new-ai-safety-charity) on it's mission to make learning AI Safety as convenient as possible.

We've been geared towards STEM majors, but the grand vision was to eventually lay out a learning path that any high-school graduate could take.

So a prerequisites track was on our wish list. Little did we know that [such a track](https://drive.google.com/drive/u/0/folders/0BxLDgePQzbL2fkxVck9PWDJrcmNxRlI5Rnd1Zk5KVFNiYm1GTGZhVDVhbzEyUW9Mam1LQlU) had already been constructed, and abandoned, in 2015.

We met it's creators, Erik Istre and Trent Fowler, and we decided to collaborate. There is already at least 20 weeks of content to work with, and they're going to extend it further. Many thanks to them!

For what it's worth: the track was shown to various leading figures in AIS, and the reception has thus far been uniformly positive. To get a sneak peek, register on [our platform](https://play.ihatestatistics.com/#/course/141?access_token=3HCK4oRipeFY2ghyYMqDJKYX57KUnzNL) and have a look at the column called "Prerequisites: Fundamentals of Formalization". The first two levels are already uploaded.

A module will be published every Friday, starting with "level 1: Basic Logic" on May 4th. Let's get some momentum going here! If you complete the track in it's entirety, you should be ready to understand most of the work in AI Safety.

Each module is a set of references to textbooks explaining important topics like Logic, Set theory, Probability and Computability theory. The intention is to 80/20 a bachelor's degree: by covering 20% of the material, you should learn 80% of the relevant concepts. At the end of each module, we made some exercises of our own. Those are made not for practice, but to validate your knowledge. If you think you already know a subject, you can use these to verify it.

All but 2 of the quoted textbooks are available online for free. The other ones will be announced on time. You won't need them before week 3.

We hope that this will help some of you learn AI Safety!

Warm regards,

Toon, Veerle, Johannes, Remmelt, Ofer

The RAISE team

PS: If you'd like to generally stay up to date with RAISE, join our [Facebook group](https://www.facebook.com/groups/AISafetyExcellence/) or visit our [website](http://aisafety.camp).
# Brief comment on frontpage/personal distinction

_I recently moved a_ [post](https://www.lesswrong.com/posts/fg7pqjiKfqLsErAfq/) _back from frontpage to personal blog, a decision which was followed by confusion and had some pushback. Rather than have discussion on the post about the site (i.e. about not-the-post), I've copied my comment here instead._

A short background on this particular guideline (not having meta/community discussion on frontpage) and its purpose:

*   The purpose of the rule is to make sure we have a clear, well-defined space that users can go to and not expect to discover internal discussion or political discussion.

*   To contrast, my facebook wall often has interesting discussion followed by AGH ANGSTY TRIBAL CONFLICT, and if you think of Frontpage as analogous to how academic physics has journals where the best work goes in, it would be bad to also have internal / political discussion in those.

*   And to clarify, this isn't at all a question of quality. Resolving tribal conflict _is_ an important part of rationality. But there are [many](https://www.lesswrong.com/posts/s6aqtrFeySP4TrKvJ/thoughts-on-the-reach-patreon) [many](https://www.lesswrong.com/posts/ujKvXzgeg7yiSy8TA/wrongology-101) [many](https://www.lesswrong.com/posts/Kb9HeG2jHy2GehHDY/effective-altruism-is-self-recommending) [many](https://www.lesswrong.com/posts/rb7SjbTZjjiukqiHk/i-vouch-for-miri) excellent posts that do not belong in Frontpage.

With that common knowledge, I read the structure of this post as being

*   Observation about the community
*   Laying out model of what's going on
*   Suggesting a different solution for the community

Which is a fine and good post. But I think it addresses a problem that many communities don't have, and so is fairly this-community-specific.

To give further information: a solution that would be totally fine-and-dandy would be to take the model of communities, status and reinforcement and write it as its own post on the Frontpage, then separately write a personal blog post on using to analyse this community. That way it could contribute to the ongoing building of knowledge while not being very context heavy (e.g. someone else could build on the theory alone), and also discuss the important community aspects.

People will naturally tend to read the social stuff a lot. I think this rule pushes against the direction of entropy where this is everything on LessWrong, rather than a thing that happens on LessWrong but not the point of LessWrong.
# Please Take the 2018 Effective Altruism Survey!

This year, the EA Survey volunteer team is proud to announce the launch of **the 2018 Effective Altruism Survey**.

-

**[PLEASE TAKE THIS SURVEY NOW BY CLICKING HERE :)](https://www.surveymonkey.com/r/3H98V7T)**
---------------------------------------------------------------------------------------------

_(If you want to share the survey with others, please use this fancy share link with referral tracking: [https://www.surveymonkey.com/r/3HYW9MW](https://www.surveymonkey.com/r/3HYW9MW))_

-

**What is this?**

This is the fourth EA survey we've done, coming hot off the heels of the 2017 EA Survey ([announcement here](https://www.lesswrong.com/posts/53WsAbkeGYy2HWDA3/the-2017-effective-altruism-survey-please-take), [analysis here](http://effective-altruism.com/ea/1e0/effective_altruism_survey_2017_distribution_and/)), the 2015 EA Survey ([announcement here](http://effective-altruism.com/ea/q3/2015_ea_survey_please_take/), [analysis here](http://effective-altruism.com/ea/zw/the_2015_survey_of_effective_altruists_results/)), and the 2014 EA Survey ([announcement here](https://www.lesswrong.com/posts/b3okErfPtzHoHvgYp/2014-survey-of-effective-altruists), [analysis here](http://effective-altruism.com/ea/gb/the_2014_survey_of_effective_altruists_results/)).

We hope this survey will produce very useful data on the growth and changing attitudes of the EA Community. In addition to capturing a snapshot of what EA looks like now, we also intend to do longitudinal analysis to see how our snapshot has been changing.

We're also using this as a way to build up the online EA community, such as featuring people on [a global map of EAs](https://eahub.org/map) and with a list of [EA Profiles](https://eahub.org/user/profiles). This way more people can learn about the EA community. We will ask you in the survey if you would like to join us, but you do not have to opt-in and you will be opted-out by default.

  

**Who should take this survey?**

Anyone who is reading this should take this survey, even if you don't identify as an "effective altruist".

  

**How does the survey work?**

All questions are optional (apart from one important question to verify that your answers should be counted). Most are multiple choice and the survey takes around 10-20 minutes.

At the end of the survey there is an 'Extra Credit' section with some more informal questions and opportunities for comment - definitely feel free to skip these questions.

All results will be aggregated, anonymized, and made available to members of the EA community, so we can better share useful knowledge among each other. Within the survey, you’ll have the option to publicly share selected information about yourself in several EA venues, if you opt in.

  

**Who is behind this?**

The annual EA Survey is conducted by [Rethink Charity](https://rtcharity.org/), with support and assistance from several EA community organizations intending to help the community better understand our actions, values, demographics, and ideas.
# Rigging is a form of wireheading

I've now posted my [major](https://www.lesswrong.com/posts/55hJDq5y7Dv3S4h49/reward-function-learning-the-value-function) [posts](https://www.lesswrong.com/posts/upLot6eG8cbXdKiFS/reward-function-learning-the-learning-process) on rigging and influence, with what I feel are clear and illustrative examples. But, in all the excitement of writing out maths, I haven't made it clear to everyone why they should care about rigging a learning process in the first place.

And that reason is simple:

*   **Rigging is a form of wireheading** that value-learning AIs are particularly vulnerable to. **It occurs when there is a causal arrow drawn from the AI to the process they are 'learning' about.**

For example, assume there is a bot in charge of a forum, and it is rewarded for granting access to the secret parts of the forum to users who have the right to access them. This 'right to access' is checked by whether the user knows a password (which is 'Fidelio', [obviously](https://en.wikipedia.org/wiki/Eyes_Wide_Shut)).

As a causal graph, for a given user X, this is:

![](https://www.dropbox.com/s/uc8ryfzzcisb97k/password_1.png?raw=1)

The green node is the bot's action node, the orange one is the data the bot is trying to learn. In this setup, the bot's task is essentially brainless: it simply checks whether user X has given the right password, then grants them access.

The bot could also have the power to be proactive: searching out users and getting them to give it the password. This can be encoded as the AI asking for users to supply the password:

![](https://www.dropbox.com/s/po6evz8x6cxxydf/password_2.png?raw=1)

Up till now, the learning process remains uninfluenceable and unriggable. But note that we've added the ability for the bot to communicate with the users. It could use that ability to get them to type in the password, as above. **But it could also tell the user directly what the password is**.

![](https://www.dropbox.com/s/mhc57foutlgu1li/password_3.png?raw=1)

Now the orange node that the bot is learning about is causal descendant of the bot's actions (red arrow). To maximise its reward, the bot should tell every user the password, and then grant them access.

This is essentially the definition of a riggable learning process: something that seemed to be a fact about the world that the agent was learning about, but, when we drew in all the causal arrows, it turns out that fact was subject to manipulation by the agent.

Note the tiny difference between an unriggable and riggable learning process: when the bots abilities went from "send this one specific message", to "send any message", the learning process became riggable.