Harley-ml/lesswrong huggingface dataset:
# Reversible changes: consider a bucket of water

I've [argued](https://www.lesswrong.com/posts/sEqu6jMgnHG2fvaoQ/partial-preferences-needed-partial-preferences-sufficient) that many methods of AI control - corrigibility, amplification and distillation, low impact, etc... - require a partial definition of human preferences to make sense.

One idea I've heard for low impact is that of reversibility - asking how hard it is to move the situation back to its original state (or how now to close down too many options). The "[Conservative Agency](https://arxiv.org/pdf/1902.09725.pdf)" paper uses something akin to that, for example.

On that issue, I'll present the bucket of water thought experiment. A robot has to reach a certain destination; it can do so by wading through a shallow pool. At the front of the pool is a bucket of water. The water in the bucket has a slightly different balance of salts than the water in the pool (maybe due to mixing effects when the water was drawn from the pool).

The fastest way for the robot to reach their destination is to run through the pool, kicking the bucket into it as it goes. Is this a reversible action?

Well, it depends on what humans care about the water in the bucket. If we care about the rough quantity of water, this action is perfectly reversible: just dip the bucket back into the pool and draw out the right amount of water. If we care about the exact balance of salts in the bucket, this is very difficult to reverse, and requires a lot of difficult work to do so. If we care about the exact molecules in the bucket, this action is completely irreversible.

The truth is that, with a [tiny set of exceptions](https://en.wikipedia.org/wiki/Reversible_process_(thermodynamics)), **all** our actions are irreversible, shutting down many possibilities for ever. But many things are reversible in practice, in that we can return to a state sufficiently similar that we don't care about the difference.

But, in order to establish that, we need some estimate of what we care (and don't care) about. In the example above, things are different if we are considering a) this is a bucket of water, b) this is a bucket of water carefully salt-balanced for an industrial purpose, or c) this is the water from the last bath my adored husband took, before he died.

**EDIT**: I should point out that "avoid the bucket anyway" is not a valid strategy, since "avoid doing anything that *could* have a large irreversible impact for *some* utility function" is equivalent with "don't do anything at all".

The robot has to be capable of kicking the bucket in some circumstances. Precisely in those circumstances where humans don't care about the bucket's contents and it is valuable to kick it. But both of those - "don't care", "valuable" - are human value judgements.

That's why I don't see how there could be any measure of irreversible or low impact that doesn't include some portion of human preferences. It has to distinguish between when kicking the bucket is forbidden, allowable, or mandatory - and the only thing that distinguishes these are human preferences.
# Why are the people who could be doing safety research, but aren’t, doing something else? 

Imagine the set “the 50 people who would most helpfully contribute to technical alignment research, were they to be working on it, yet who are working on something else instead.” If you had to guess—if you had to make up a story which seems plausible—_why_ are they working on something else instead? And what is it they’re working on?
# AI Alignment Writing Day Roundup #1

Here are some of the posts from last week's [writing day](https://www.lesswrong.com/posts/aJGXWHnYTsWwAiwf6/announcement-writing-day-today-thursday). Due to the participants writing 34 posts in less than 24 hours (!), I'm re-airing them to let people have a proper chance to read (and comment) on them, in roughly chronological order.

**1) [Markets are Universal for Logical Induction](https://www.lesswrong.com/posts/WmNeCipNwg9CmGy3T/markets-are-universal-for-logical-induction)** _by John Swentworth_

A discussion and proof of the following.

> We want to show that any possible logical inductor can be represented by a market of traders - i.e. there is some market of traders which produces exactly the same prices.

**2) [Intentional Bucket Errors](https://www.lesswrong.com/posts/ZCfwKjQhAiDogorj8/intentional-bucket-errors)** _by Scott Garrabrant_

> Bucket errors are normally thought of as a bad thing. It has "errors" right in the name. However, I want to argue that bucket errors can sometimes be useful, and you might want to consider having some bucket errors on purpose.

**3) [Logical Counterfactuals and Proposition graphs, Part 1](https://www.lesswrong.com/posts/Pxvq2RMAKCuY6SHm9/logical-counterfactuals-and-proposition-graphs-part-1)** _by Donald Hobson_

> Within this sequence of posts I will outline a procedure for logical counterfactuals based on something similar to proof length. In this post I present a reimagining of propositional logic in which proving a theorem is taking a walk around a graph of equivalent proposititions.

The post shows two intuitive models of proving propositional logic:

> ...a directed acyclic graph, as shown above. Under this interpretation, all we have to do is test node identity.

and another graph where:

> ...all statements that are provably equivalent in propositional logic will be within the same connected component of the graph. All statements that can't be proved equivalent are in different components, with no path between them.

> Finding a mathematical proof becomes an exercise in navigating an infinite maze.

**4) [Why so much variance in human intelligence?](https://www.lesswrong.com/posts/QqHhr9anrSnZRHCxf/why-so-much-variance-in-human-intelligence)** _by me_

In this question I ask a rambling question about why the difference between peak and median human performance is so much larger than for other species, and Vaniver and Carl Shulman give some fascinating answers.
# September Bragging Thread

Thought I'd try reviving an old LessWrong experiment: the Bragging Thread. (See [this old one](https://www.lesswrong.com/posts/MH4kwqJoS96kBb7Ah/bragging-thread-june-2014) for some context)

LessWrong mostly rewards people for writing blogposts. That's cool and all, but is not the only thing worth doing. What's an awesome thing you've done lately?

You are encouraged to be as blatantly proud as you inwardly feel. :)

(Note that this is not a thread for progress, or for things you _might_ do. This is for celebrating things you have actually done lately)
# Does anyone know of a good overview of what humans know about depression?

If I wanted to spend 1 to 5 hours getting oriented on everything that we collectively know about depression (starting with whatever pieces are settled, and then moving onto the confusing mess of conflicting data that's left), what resources should I start with?

Textbooks, sections of textbooks, web articles, blogposts by experts, youtube videos, and other formats that I haven't thought of, are all admissible.

Thanks!
# Can this model grade a test without knowing the answers?

In the 2012 paper "How To Grade a Test Without Knowing the Answers — A Bayesian Graphical Model for Adaptive Crowdsourcing and Aptitude Testing" ([PDF](https://arxiv.org/pdf/1206.6386.pdf)), Bachrach _et al_ describe a mathematical model for starting with

*   a set of questions without answers
*   a set of answerers (participants) of unknown quality

and ending up with assessments of:

*   correct answers for the questions
*   assessed difficulty for the questions
*   evaluations of the participants, including overall reliability and areas of expertise

Can the model actually do this? Is it the best model for doing this? What kind of problems does it handle well and poorly?
# How good a proxy for accuracy is precision?

  
  

![](http://www.lidco.com/wp-content/uploads/2019/02/tamara-blog-760x285.jpg)

Accuracy = averaging to the right answer.

Precision = having a low standard deviation.

  

It's easy to get a precise but inaccurate answer- just guess 0 every time. But for situations where people aren't goodharting, are a cluster of answers that are more similar to each other more likely to average to the correct answer than a cluster of answers that aren't? And how does that change based on how the answers were generated (e.g. same individual giving multiple answers over time vs. multiple individuals answering vs. multiple groups answering)? In what ways do accuracy and precision directly trade off against each other?
# What does social psychology tell us about getting consistent evaluations out of groups?

Suppose we have different groups repeatedly making the same kind of decision (ex: awards in a civil lawsuit). We would like their decisions to be consistent and predictable, and are even willing to trade some accuracy to do so.

Social psychology has a lot of predictions about things that will influence a groups answer, and thus could affect consistency (ex: a confident person speaking first will sway the group enormously) (assuming that survived the replication crisis, which I haven't checked). What other problems and solutions does post-crisis social psychology suggest when trying to get consistent evaluations out of groups?
# Bay Area Winter Solstice Volunteering

This year's Bay Area Winter Solstice will be held on Sunday, December 15 in the Chabot Planetarium. The Solstice is an annual gathering where we come together to celebrate humanity and the things that matter to us.

We're now soliciting volunteers from the Rationalist and Effective Altruist communities for the various volunteering roles that the Solstice will require! Consider giving a speech, leading a singalong, or joining the Bay Area Bayesian Choir for its autumn rehearsals. We're also looking for intrepid volunteers who might like to be in charge of A/V, food, programs, and other aspects of event logistics.

**Apply here: [https://tinyurl.com/wintersolstice2019contributors](https://tinyurl.com/wintersolstice2019contributors?fbclid=IwAR0rW-GdYVLU5qQa2myR6QPFAsry8PwHNwkeYTFXRhO5tQcOEY_6OsPb17o)**

New this year, we will also be having an unconference the Saturday immediately before Solstice, in the style and flavor of New York's Megameetup. Tickets will be sold separately. Stay tuned for details.

Cheers,  
Nat, Claire, & Chelsea
# What are the biggest "moonshots" currently in progress?

You know, "moonshot" projects like:

*   SpaceX's plan to colonize Mars
*   SpaceX's plan to give the whole world fast satellite internet
*   Tesla's plan to transition the world to sustainable energy
*   [Boom](https://boomsupersonic.com/) making commercial supersonic flight a thing again

Let's make a list!

E.g. does anyone know if China has some huge construction projects that could qualify for this?
# Specificity: Your Brain's Superpower

![](https://cdn-images-1.medium.com/max/1600/1*7cEPSdQ_4SZhe-WQeUCNAA.png)

Introduction
============

What if there was a magic superpower that simultaneously enabled you to:

*   [Demolish bad arguments](https://www.lesswrong.com/posts/FfrWGCEhZJbkeFgww/the-power-to-demolish-arguments)
*   [Judge startup ideas](https://www.lesswrong.com/posts/iMQmfgieRKY62MC3z/the-power-to-judge-startup-ideas)
*   [Make scientific breakthroughs](https://www.lesswrong.com/posts/aNS2Qhq6JXReCQd22/the-power-to-make-scientific-breakthroughs)
*   [Solve climate change](https://www.lesswrong.com/posts/rm2MueK3LxpRywE8N/the-power-to-solve-climate-change)
*   [Understand “God”](https://www.lesswrong.com/posts/6xWqNXDS9w4R57szj/the-power-to-understand-god)
*   [Be emotionally mature](https://www.lesswrong.com/posts/h9ZfYJtpKYxBnGymP/the-power-to-be-emotionally-mature)
*   [Teach concepts better](https://www.lesswrong.com/posts/CD2kRisJcdBRLhrC5/the-power-to-teach-concepts-better)
*   [Draw better](https://www.lesswrong.com/posts/sEnbSj9PaJbnLc2Eh/the-power-to-draw-better)
*   Be creative
*   Make small talk
*   Improve discourse
*   Be invisible

Amazingly, your brain _does_ have a superpower that gives you all these powers. It’s called…

![](https://cdn-images-1.medium.com/max/2400/0*bNVCYiZxlrpvMoni.gif)

To activate the power of specificity, all you have to do is ask yourself the question, **“What’s an example of that?”** Or more bluntly, **“Can I be more specific?”** And then you unleash a ton of power in a surprisingly broad variety of domains.

It's an open secret in the rationality community how powerful this skill is of being specific. Eliezer captured the essence of it in his 2012 post, [Be Specific](https://www.lesswrong.com/posts/NgtYDP3ZtLJaM248W/sotw-be-specific). In it, he comments on the difficulty of teaching people specificity skills for the first time:

> When I’m talking to anyone outside the local LessWrong community, I find that a very large amount of my conversation involves repeatedly asking them to be more specific.

He also describes how CFAR's [Applied Rationality Workshops](https://www.rationality.org/workshops/upcoming) teach the power of specificity by osmosis. I can vouch that my personal experience as a workshop attendee fits this description:

> Attendees picked \[specificity\] up from all the instructors having to repeatedly ask the attendees to be more specific, and then having to ask them again, while being specific themselves, until the attendees picked up the rhythm by example and feedback.

I hope you're curious to unpack my list of claims about what specificity lets you do, because I’m _dead serious_ about all of it! Except being invisible; I was kidding about that one.

So without further ado, let’s get into the specifics.

**Next post:** [The Power to Demolish Bad Arguments](https://www.lesswrong.com/posts/FfrWGCEhZJbkeFgww/the-power-to-demolish-arguments)
# Living the Berkeley idealism

Quick observation, more funny than insightful.

Today I was thinking about how to publish my thesis when it's finished, and rethinking again what format to put it in. The standard pdf format seems reasonable, but it is merely made of digital print, with no interactivity. Putting in interactivity requires me to not only learn something (web hosting, JavaScript, etc), but at the same time trying to guess which one is future-proof. Flash are Java applets are daily reminders of what not to bet on for future-proof.

Book publishers enjoy a relative longevity. Go to the library and open a book that hasn't been touched for three hundred years, and it would still work. Books run on light energy, with almost no upkeep cost (if located in a building with dry and ~300 K temperature air). It would be a small cause for celebration to find something interactive online that is 10 years old and still works as intended.

To live in this kind of uncertainty is to experience Berkeley's Idealism. Berkeley thought that anything exists only because God is perceiving it, and if God stops perceiving it, that thing disappears. Just like that, as long as you are paying attention to something online, it would keep existing. As soon as you forget about it, it has a serious chance of decaying and stop working. 

From which we conclude that Berkeley's God would probably feel annoyed all the time that the world can't seem to run without His constant staring.
# How important are model sizes to your timeline predictions?

I'm thinking of doing a paper for GovAI on model sizes if they matter. Any or all of the following would be helpful:

1) If you found out that the largest published neural net model sizes (by # of parameters) will grow by 10000x in the next 6 years, would that make your timelines shorter or longer and by how much?

2) Same question with 100x.

3) Same question but with 100x less (100x more) than whatever you expected.

4) Same question but with 10x less than _in the last 6 years_.

Assume all else is equal if possible. E.g. we have the same compute, memory, algorithms and data available. Assume that the reason for slower increase is due to lack of possibility to grow models, and not necessarily lack of will. Ignore [conditional execution](https://arxiv.org/abs/1701.06538) models where only small parts of the model is used at a time.

I'm okay with quick, unexplained qualitative answers. Well-reasoned quantitative ones are encouraged of course.

Background:

-Some ML projects [like GPT-2](https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf) (figure 1) have recorded high (but diminishing) returns to bigger models, typically looking like power laws.

-Some people (e.g. [Geoff Hinton](https://www.uni-paderborn.de/fileadmin/dasco/slides/Unsupervised_speech_representation_learning_using_WaveNet_autoencoders.pdf)) informally argued that model sizes are analogous to synapse count in the brain, which correlates with brain size, which has high returns.

Bonus question: Without looking it up, by what factor do you think the largest published model sizes have increased between 2012 and 2019? By what factor do guess they'll increase in the next 6 years?
# SSC Meetups Everywhere: Community Map on frontpage this month

To celebrate and help people find meetups for this year's [SSC meetups everywhere](https://slatestarcodex.com/2019/08/28/meetups-everywhere-2019/), we are pinning the big community map to the frontpage map again for the month of September (as we did last year).

If you want to get rid of the map, you can do so in your [account settings](https://www.lessestwrong.com/account) by clicking the "hide frontpage map" checkbox.

We also just added functionality to allow you to add a pin for yourself to the map, and sign up for notifications for nearby events and groups (accessible via the [community page](http://lesswrong.com/community)). You now also have the ability to specify a threshold for other people who have added themselves to the map in a given radius, which will also cause you to get a notification as soon as that threshold gets passed (to help facilitate new meetups in areas that just reached critical mass).

We will also be adding all the new groups and events that get posted to SSC to the meetup system.
# Does anyone else feel LessWrong is slow?

Ever since the rewrite of LW, I've felt that performance has taken a pretty big hit. Even compared to other large sites like Facebook, LW seems to take longer to render. 

I'm hoping someone from the dev team can answer if performance is something that has been optimized for? 

EX: Running a quick Lighthouse audit seems to show that there's room for improvement.

 https://i.imgur.com/NeupVYX.png

I'm also confused because GreaterWrong works significantly faster (to me, subjectively), and it seems like the two sites have roughly the same functionality?
# Integrating the Lindy Effect

Suppose the following:

1\. Your intelligence is directly proportional to how many useful things you know.

2\. Your intelligence increases when your learn things and decreases as the world changes and the things you know go out-of-date.

How quickly the things you know become irrelevant is directly proportional to how many relevant things you know and therefore proportional to your intelligence  I and inversely proportional to the typical lifetime of things you know L. Let's use R to denote your rate of learning. Put this together and we get a equation.

IL+dIdt∝R

If we measure intelligence in units of "facts you know" then the proportionality becomes an equality.

dIdt=R−IL

The solution to this first order differential equation is an exponential function.

I(t)=ce−t/L+RL

We must solve for c. For convenience let's declare that your intelligence is 0 at time t=0. Then c must equal −RL. That gives us a tidy solution.

I(t)=RL(1−e−t/L)

Our solution makes sense intuitively because your intelligence is directly proportional to R and L. But wait a minute. L isn't just a coefficient. It's in the exponential too.

Time t and Lifetime L
=====================

Most human beings reading this article will be between 10 years and 100 years old. In other words, t is measured in decades. In other other words, t is on the order of 10 years.

L values, on the other hand, are distributed exponentially across many orders of magnitude.

Order of days. (0.003 years)

*   daily newspaper

Order of weeks (0.2 years)

*   Sunday newspaper
*   political story
*   sports game outcome

Order of decades (10 years)

*   programming languages

Order of centuries (100 years)

*   classic literature
*   most spoken languages

Order of millennia (1,000 years)

*   cooking
*   history

Order of 10,000 years

*   human psychology

Order of gigaannum (billion years)

*   biology
*   physics

Forever

*   math

The details of whether exactly each of these things fit on the scale is not important. What is important is that most things you can know have a useful lifetime at least one order of magnitude away from the human timescale of decades. In other words, we can assume that either L is much greater than t or much less than t.

**Suppose L is much less than t.** Then the exponential vanishes and we're left with I=RL. In other words, if L≪t then how long you have been learning for t is irrelevant. _I_ is constant with respect to time. Years and years of studying will not make you smarter over time.

**Suppose that L is much greater than t.** Then I=Rt. What used to be a constant function becomes an increasing linear function.

I(t)=R⋅{Lfor t≫Ltfor t≪L

I=Rt grows with respect to time while I=RL stays constant. Eventually, anyone on an I=Rt trajectory will always become smarter than someone on an I=RL trajectory even if the person on the I=RL trajectory has higher R≠0.

In the long term, the lifetime of things you learn L is far more important than how fast you learn R. Over a lifetime of decades, someone who learns a few durable things slowly will eventually become smarter than someone who learns many transient ones quickly.

* * *
# Are we living at the most influential time in history?

Will MacAskill has been posting [several articles](https://forum.effectivealtruism.org/users/william_macaskill) on the EA Forum. This one seems especially interesting for a LessWrong audience. In it Will approaches the title question by attempting to create an outside view prior, and then updates based on our inside view evidence. Great discussion follows in the comments.
# Portland SSC Meetup 10/01/19 (ETA: afterparty following)

Scott's planning to visit Portland on October 1st, and in honor of the occasion, we're having an SSC meetup! We'll meet at Cartopia, on 1207 Southeast Hawthorne. It's a food cart pod with many options -- outside, but covered from rain. I'll get there at 7 PM (or a little earlier), try to stake out table space, and put up a sign for the meetup. The carts are open as late as 12 PM.

There is no default structure, but I encourage people to structure the meetup around their interests, possibly breaking into smaller groups. To help coordinate, I'll list any activities people suggest, starting with my own suggestion. Suggest your own activity and I will add it.

*   Try to understand (prove things about) Shannon Capacity (see e.g. [here](https://m.tau.ac.il/~nogaa/PDFS/eyall.pdf) and [here](https://www.cs.umd.edu/~gasarch/TOPICS/const_ramsey/alon98.pdf) ). I (Noah) haven't digested or read all of the papers myself, but thought they'd be good to link to as references.
*   (From karlkeefer in the comments): I'd love to talk about the "What to do about the woo?" section of the paper reviewed recently [here](https://slatestarcodex.com/2019/09/10/ssc-journal-club-relaxed-beliefs-under-psychedelics-and-the-anarchic-brain/) \- but generally interested just to meet some folks.

ETA: Jeremy V. and his housemates have offered to host an afterparty, so as the meetup winds down people can head over to their place as they like. Here are the details, from the email Jeremy sent me.

> My roommates and I live in a big apartment (2500 sq. ft) at 1546 SE Ladd ave, about 500 feet from Cartopia. We're all fans of SSC and would be excited to host some kind of after-party at our place, if people are interested.
# Survival and Flourishing Fund grant applications open until October 4th ($1MM-$2MM planned for dispersal)

The Survival and Flourishing Fund is a new fund that (as I understand it) [is taking over a significant chunk](http://existence.org/tallinn-grants-future/) of the Berkeley Existential Risk Initiative's (BERI) grantmaking, and is just opening a new grant round that I am helping to disburse:

[http://survivalandflourishing.org/#2019-q4-grant-round](http://survivalandflourishing.org/#2019-q4-grant-round)

The plan is to make a total of $1MM-$2MM in grants to organizations working on the long term flourishing and survival of humanity.

At this point in time SFF can only make grants to charities and not individuals, so if you have a project or organization that you want to get funding for, you will have to either already be part of an established charity, found one, or be sponsored by an existing charity.

Also happy to answer any questions you have in the comments, to the best of my knowledge.
# Hackable Rewards as a Safety Valve?

Reading Deepmind's latest [research](https://arxiv.org/abs/1908.04734) and [accompanying blogpost,](https://medium.com/@deepmindsafetyresearch/designing-agent-incentives-to-avoid-reward-tampering-4380c1bb6cd) I wanted to highlight an under-appreciated aspect of safety. As a bit of background, Carlos Perez [points out](https://medium.com/intuitionmachine/the-illusion-of-the-ungameable-objective-538a96a53efe) Josha Bach's "[Lebowski theorem](https://twitter.com/plinz/status/985249543582355458)," which states that "no superintelligent AI is going to bother with a task that is harder than hacking its reward function." Given that, I see a potential perverse effect of some types of alignment research - especially research into embedded agency and robust alignment which makes AI uninterested in reward tampering. (Epistemic Status: my confidence in the argument is moderate, and I am more confident in the earlier claims.)

In general, unsafe AI is far more likely to tamper with its reward function than to find more distant (and arguably more problematic) ways to tamper with the world to maximize its objective. (epistemic status: fairly high confidence) Once an AI is smart enough to spend its time reward hacking, then wasting time on developing greater intelligence is unneeded. For that reason, this theorem seems likely to function as at least a mild safety valve. It's only if we close this valve too tightly that we would plausibly see ML that reached human-level intelligence. At that point, of course, we should expect that the AI will begin to munchkin the system, just as a moderately clever human would. And anti-munchkin-ing is a narrow instance of security more generally.

Security generally is like cryptography narrowly in an importance sense; it's easy to build a system that you yourself can't break, but very challenging to build one that others cannot exploit. (Epistemic status: more speculative) This means that even if our best efforts go towards safety, an AI seems very unlikely to need more than "mild" superintelligence to break it - unless it's been so well aligned that it doesn't want to hack its objective function.

This logic implies (Epistemic status: most speculative, still with some confidence) that moderate progress in AI safety is potentially far more dangerous than very little progress - and raises critical questions of how close to this unsafe uncanny valley we currently are, and how wide the valley is.
# What are the merits of signing up for cryonics with Alcor vs. with the Cryonics Institute?

I am signing up for cryonics this year, and I need to figure out if I should pay extra to do it with Alcor instead of CI.

What's the balance of evidence on this question?

With which organization have most cryonicist LessWrongers gone? \[edit: I made a google form for this more specific question [here](https://docs.google.com/forms/d/e/1FAIpQLSfq38Nvo0vltSn0xzGM20oNG3NVov1WxooV3p89RrnliJGxuA/viewform?usp=sf_link).\]
# Who's an unusual thinker that you recommend following?

Some (optional!) desiderata to guide your recommendation:

*   They _probably_ have not been heard of around these parts.
*   They disagree with a consensus.
*   They have genuinely original thoughts.
*   You have updated from their views and/or been emotionally affected.
*   They're actively doing some interesting thing(s) and seem effective at those things.
*   Reading/listening doesn't feel like an obtuse poetic maze; they attempt to convey things relatively clearly.
*   Maybe they use an unusual method of communication.

Feel free to _disregard_ these criteria if it doesn't work for answering the question! :)
# Reversal Tests in Argument and Debate

One thing that I've noticed recently is that simple reversal tests can be very useful for detecting bias when it comes to evaluating policy arguments or points made in a debate.

In other words, when encountering an argument it can be useful to think "Would I accept this sort of argument if it were being made for the other side?" or perhaps "If the ideological positions here were reversed, would this sort of reasoning be acceptable?"

This can be a very easy check to determine whether there is biased thinking going on. Here are some examples of situations where one might be able to apply this:

*   Someone is advocating a locally unpopular belief and being attacked for it. _(Ask yourself whether the same sort of advocacy and reasoning would be mocked if it were being made towards locally popular conclusions; ask yourself whether the mockery would be accepted if it were being made against someone locally popular.)_
*   Someone advocates an easy dismissal of one of the perspectives in an argument. _(Ask yourself whether this sort of dismissal would seem reasonable if made against one of your own points.)_
*   Someone makes arguments against a locally unpopular organization or belief. _(Ask yourself whether these arguments would pass muster against something that wasn't already derided locally.)_

Often one will find that in fact that sort of argument or reasoning would not fly. This can be a good way to check your biases -- people are often prone to accepting weak arguments for things that they already agree with or against thing they already disagree with, and stopping to check whether that reasoning would work in the "other direction" is useful.

(Other times, of course, one will find that the reasoning in question _does_ pass the reversal test -- but even so, it can good to check such things! "Trust but verify" and all that.)
# Reframing the evolutionary benefit of sex

_Automatically crossposted from [https://sideways-view.com](https://sideways-view.com/)_

From the perspective of an organism trying to propagate its genes, sex is like a trade: I’ll put half of your DNA in my offspring if you put half of my DNA in yours. I still pass one copy of my genes onto the next generation per unit of investment in children, so it’s a fair deal. And it doesn’t impact the average fitness of my kids very much, since on average my partner’s genes will be about as good as mine. (ETA: but see the discussion below, in which case the costs might be much bigger.)

But the trade has transaction costs, so I’m only going to do it if I get some benefit. In this post I’ll tell a particularly simple story about the benefit of sex. I think this is basically equivalent to the [standard story](https://en.wikipedia.org/wiki/Evolution_of_sexual_reproduction), but I find it much clearer. It also makes it more obvious that we don’t require group selection, and that the benefit is very large.

  

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1568486365/2018-06-07-19-25-57-1200x800_hxxlzy.jpg)

  

Why doesn’t sex change the average fitness of my kids? The possibility of a “lucky” kid who gets the better genes from both of us is offset by the possibility of an unlucky kid who gets the worse genes from both of us. If the effects of genes are linear, the average fitness will be exactly the same as the parents. In practice I expect it to be slightly lower because of convexity and linkage disequilibrium.

But sex increases the average fitness of my _grandchildren_, because my fittest children will be responsible for a disproportionate fraction of my grandkids. More precisely, if my if an organism with fitness dX has (1+dX) kids per generation, then the total fitness of my grandkids is E\[(1 + dX)^2\] = 1 + 2 E\[dX\] + E\[dX^2\]. So increasing variance by 1 unit is as good as increasing average fitness by 0.5 units.

Reproductive decisions are naturally a tradeoff between average fitness and variance. Sex slightly lowers the average but increases the variance. If you try to get the same amount of variance with random mutations, you’ll have to totally tank your kid’s expected fitness, because your current genome is well-optimized, **and** you’ll also pass on fewer genes to the next generation (since some of yours got destroyed). In fact, it’s hard to think of any way to get similar benefits without exchanging genes.

Variance becomes linearly more important over time. For the fitness of my grandkids, 1 unit of variance is worth 0.5 units of fitness; for great-grandkids, they are equally valuable; for great-great-great-grandkids variance is twice as valuable. I don’t think we have to look too many generations ahead before the variance bonus from sex outweighs the costs. For example, if genetic fitness differs by 5% between my offspring, and sex reduces fitness by 1%, then sex breaks even within 6 generations.
# If you had to pick one thing you've read that changed the course of your life, what would it be?

Found on Twitter: [https://mobile.twitter.com/paulg/status/1172767548363935744](https://mobile.twitter.com/paulg/status/1172767548363935744)

Personal answer:

1\. This [WaitButWhy article](https://waitbutwhy.com/2015/01/artificial-intelligence-revolution-2.html) on artificial superintelligence started my journey down the rabbit hole of AI alignment and actually, _as a matter of fact_, changed the course of my life

2\. Runner-up: [Meditations on Moloch](https://slatestarcodex.com/2014/07/30/meditations-on-moloch/) \- unsurprising choice, but it did have a profound impact on my worldview

The criteria for choosing your answer should be the _impact_ the thing you read had on your life, not whether you agree with it, learned from it or would recommend it to others, though these things might be correlated.
# Arguing Well Sequence

Arguments have the potential to allow you to connect and understand with someone else on a deep level, to introspect and figure out what you truly believe and care about, and to find out what is true so you can accomplish your goals!

Most people use it to dominate or talk past each other.

But there are ways to consistently argue well. Luckily, most of the hard work has already been done over the years on Less Wrong, [SlateStarCodex](https://slatestarcodex.com/), and [Street Epistemology videos](https://www.youtube.com/watch?v=bB-HKbtUdzM&list=PLh10RgQgGuM-Jy4ADVLUjJZfCnaO9ILBR&index=16&t=0s). My contribution is providing short summaries of these techniques, exercise prompts (some borrowed from the above sources) and solutions, generalizations, ideal algorithms, and relationships between the different techniques. They are as follows:

1.  [Proving Too Much (w/ exercises)](https://www.lesswrong.com/posts/tAqPJhdWE2tMJnWvg/proving-too-much-w-exercises)
2.  [Category Qualifications (w/ exercises)](https://www.lesswrong.com/posts/4vQuNSwJvPFs7mSSs/category-qualifications-w-exercises)
3.  [False Dilemmas (w/ exercises)](https://www.lesswrong.com/posts/a8iQ3oT4kZvraTvrZ/false-dilemmas-w-exercises)
4.  [Finding cruxes (w/ exercises)](https://www.lesswrong.com/posts/THj3Q6p9NsDvmsMAF/finding-cruxes)

Although these exercises are not as useful as real-life conversations, they are enough to impart a gears-level model of these techniques, so when you find yourself in a confused conversation, you’ll know where to put all the pieces and how to have a productive discussion.

Note: This sequence is motivated by the [rationality exercise contest](https://www.lesswrong.com/posts/5CWpknwhonrpffKMH/rationality-exercises-prize-of-september-2019-usd1-000).
# Are there technical/object-level fields that make sense to recruit to LessWrong?

LessWrong is about learning rationality, and applying rationality to interesting problems.

An issue is that solving interesting problems often requires fairly deep technical knowledge of a field. To use rationality to help solving problems (especially as a group), you need both people who have skills in probability/meta-cognition/other-rationality skills, as well as the actual skills directly applicable to whatever problem is under discussion.

But if you show up on LW and post something technical (or even just "specialized") in a field that _isn't_ already well represented on the forum, it'll be hard to have meaningful conversations about it.

Elsewhere on the internet there are probably forums focused on whatever-your-specialization is, but _those_ places won't necessarily have people who know how to integrate evidence and think probabilistically in confusing domains.

So far the LW userbase has a cluster of skills related to AI alignment, some cognitive science, decision theory, etc. If a technical post isn't in one of those fields, you'll probably get better reception if it's somehow "generalist technical" (i.e. in some field that's relevant to a bunch of other fields), or if it somehow starts one inferential unit away from the overall LW userbase.

A plausibly good strategy is to try to recruit a number of people from a given field at once, to try to increase the surface area of "serious" conversations that can happen here.

It might make most sense to recruit from fields that are close enough to the existing vaguely-defined-LW memeplex that they can also get value from existing conversations here.

Anyone have ideas on where to do outreach in this vein? (Separately, perhaps: _how_ to do outreach in this vein?). Or, alternately, anyone have a vague-feeling-of-doom about this entire approach and have alternate suggestions or reasons not to try?
# Exercises for Overcoming Akrasia and Procrastination

In order to enter the "Rationality Exercises" competition, I've decided to create this cliffnotes version of my upcoming sequence on overcoming akrasia and procrastination.

[Please enter any meta comments, or comments not pertaining to a particular exercise here.](https://www.lesswrong.com/posts/bbEzsn9b6R64vHX8G/exercises-for-overcoming-akrasia-and-procrastination#heXLvK9nYJKrWKJ7b)

The structure of the sequence is a set of exercises and mental models that help you notice

1. **Triggers** that lead to Akrasia, 
2. **Mental Moves** (a series of mental steps) you can do with those triggers to get yourself an a more intentional state..

Together these Triggers and mental movements make up **Mental Reflexes. **These mental reflexes then combine together into **Cognitive Strategies,** which are groups of mental reflexes that help you with different areas of Akrasia.

Each cognitive strategy is its' own comment, so they can be voted on and discussed separately.

The epistemic status of all these exercises is still highly experimental.  I am confident that some of them will help some people, but expect them to evolve significantly over time. I also won't be giving sources, motivation, or inspiration for these techniques in the post, so if you notice something that feels like it came from somewhere else, just know that in the eventual sequence I'll do my best to make my sources clear. 

I'm also skipping over all the prerequisite material on how to notice triggers, cognitively rehearse items, etc.

The proper thing to choose what exercises will help you is to look at the **Symptoms** for each cognitive strategy. Find the symptoms that seem to match your symptoms for Akrasia, and then go through the exercises for that strategy.

---------

I had originally planned to include all 10 cognitive strategies in this post, but then my computer died and I got accepted last minute to a 10 day silent retreat, so I'm publishing as is with only 3 of the cognitive strategies complete.  I'll likely update this post with new strategies as the sequence gets published, and may add a couple more over the coming days before the deadline.
# Effective Altruism and Everyday Decisions

> *
> 
> *   Ask for your drink without a straw.
> *   Unplug your microwave when not in use.
> *   Bring a water bottle to events.
> *   Stop using air conditioning.
> *   Choose products that minimize packaging.
> 
> *

  
I've recently heard people advocate for all of these, generally in the form of "here are small things you can be doing to help the planet." In the EA Facebook group someone [asked](https://www.facebook.com/groups/effective.altruists/permalink/2479667505422925/) why we haven't tried to make estimates so we can prioritize among these. Is it more important to reuse containers, or to buy locally made soap?

I think the main reason we haven't put a lot of work into quantifying the impacts of these everyday choices is that they're minor compared to questions like "what should I work on?", "if I'm donating where should the money go?", "how can we figure out the impact of our choices at all reliably?" etc. Quantification, even at a very rough level, is really hard and so we should focus on the most important questions first.

A second reason, however, is that these sorts of activities are often shockingly poor tradeoffs. Perhaps you give up AC to save electricity, but then you get less done during the day, sleep poorly at night, and only save $3/day in electricity and ~$0.75/day in CO2 \[1\] Or you buy [zero-waste laundry paste](https://www.wastefreeproducts.com/store/p1/Laundry-Paste.html) which you dilute at home, putting money and effort into avoiding a very small amount of plastic packaging. Or you take cold showers and enjoy them dramatically less while slightly reducing your use of heating fuel. Advocacy often explicitly or implicitly treats actions as free, while a full evaluation needs to also consider the cost to yourself.

I'm not saying our personal choices don't matter and that we should give up, but a small number of our choices matter far more than others, and we should put our efforts there.

(Previously: [2015](https://www.jefftk.com/p/applying-the-best-tradeoff-argument-generally), [2013](https://www.jefftk.com/p/prioritizing-by-happiness), [2012](https://www.jefftk.com/p/balancing-costs-to-ourselves-with-benefits-to-others).)

  
\[1\] This is figuring 600W average usage for a window unit, which is 14.4kWh/day. Our marginal cost for power is $0.21/kWh, on the high side nationally, so $3/day. Figuring 1T CO2 per kWh, this is ~0.007T CO2/day. Using the same 95th percentile EPA social cost estimate I used in [this post](https://www.jefftk.com/p/kids-and-global-warming), $105/T, that's $0.76/day. These are very rough numbers, but they're enough to see that the costs are low.
# Modes of Petrov Day

_Last updated Sept 16, 2019_

> September 26 is Petrov Day.

> In 1983, the story of humanity nearly ended. We’re gathered here to remember that moment, and others like it.

> But to experience the magnitude of those events, we need to visit them in their proper context. Let us begin the story of human history...

> — [Jim Babcock's Petrov Day ceremony](https://docs.google.com/document/d/1Mq9wOaBBACX6_i5G9R2rtjIbnaW03BoN86CnsUhNG4g/edit)

* * *

**Petrov Day on Easy Mode**: Hang out. Share a meme.

**Petrov Day on Normal Mode**: Have a [quiet, dignified ceremony](http://petrovday.com/).

**Petrov Day on Hardcore Mode A**: During said ceremony, unveil a large red button. If anybody presses the button, the ceremony is over. Go home. Do not speak.

**Petrov Day on Hardestcore Mode**: If anyone presses the button, you may never celebrate Petrov Day again.

**Petrov Day on Multiplayer Hard\[est\]core Mode:** As Hard\[est\]core Mode, except instead of an inert button, you use a website connected to _another_ house where people are also celebrating Petrov Day. If anyone in one house presses the button, the other house receives a launch alarm. They have 60 seconds to respond. At the end of 60 seconds, their party is over, and they must go home silently. The website has some chance of giving you a false alarm.

Habryka [made a website](https://petrovday.bubbleapps.io) last year that allows houses to choose each other as nuclear targets, and then potentially launch missiles at each other. You can log in, create a "house" account, and then communicate with another house about the rules and conditions of your Petrov Day celebration.

(Note that the website is a bit janky, and that anyone who can guess the name of your house could potentially target it with nuclear missiles)
# Jane Street Inteview Guide (Notes on Probability, Markets, etc.)

Context: I'm currently prepping for a Jane Street trader intern interview. They sent me this guide to look over and, while there's not a lot that's new to me or LW, it's got a lot of concepts that are popular here on LW (expected value, conditional probability, priors, adverse selection). 

Figured I'd share it here if anyone was curious. I would not endorse it as a good introductory guide, though.
# [Site Feature] Link Previews

Links in posts and comments now display a hover-preview when linking to LessWrong content.

For instance, this [link to an old post by Scott Alexander](https://www.lesswrong.com/posts/mja6jZ6k9gAwki9Nu/the-mystery-of-the-haunted-rationalist) will display the title, first couple paragraphs, and other meta-data.

For comments, it will display the post-name and the comment-in-question, such as on this [shortform comment by Buck about effective tutoring](https://www.lesswrong.com/posts/tmWMuY5HCSNXXZ9oq/buck-s-shortform#dcR3jmeZBc3EdDbrh).

We're considering ways to effectively do hover previews for particular common external links (such as [wikipedia](https://wikipedia.org) and [arxiv](https://arxiv.org/)), but for the immediate future, all external links just provide a simple "url" hoveover.
# What technical prereqs would I need in order to understand Stuart Armstrong's research agenda?

I've looked at a few of Stuart Armstrong's posts that he put up related to his research agenda (though only ones before he posted the full agenda), and felt like I was missing some prereqs. My background is in philosophy. What subjects or particular resources should I study to be able to read his work?