Harley-ml/lesswrong huggingface dataset:
# I'm Voting For Ranked Choice, But I Don't Like It

This fall, [Ranked Choice](https://en.wikipedia.org/wiki/Instant-runoff_voting) / Instant Runoff Voting (IRV) will be [on the ballot](https://ballotpedia.org/Massachusetts_Question_2,_Ranked-Choice_Voting_Initiative_(2020)) in Massachusetts. I'm voting for it, but only because it's better than the status quo, not because I think it's a very good voting system.

Massachusetts currently uses traditional majority ("first past the post") voting: whoever gets the most votes wins. Unfortunately, this only works well when you have two candidates. With more candidates, the candidates tend to [hurt their allies](https://en.wikipedia.org/wiki/Vote_splitting#Spoiler_effect) by competing for the same pool of votes, making it more likely that an opponent wins.

In IRV each voter lists their preferred candidates in order, and if your first choice is eliminated then your vote goes to your next favorite. This mostly fixes the problem of minor spoiler candidates: anyone who is not a serious contender will get eliminated and their votes redistributed.

Unfortunately, IRV has major problems when you have more than two serious candidates. For example, even if there is a candidate that a [majority of voters prefer to every other](https://en.wikipedia.org/wiki/Condorcet_method), they can still lose if their competitors happen to be eliminated in the wrong order. In [Why Ranked Choice Voting Isn't Great](https://www.jefftk.com/p/why-ranked-choice-voting-isnt-great) I give examples of realistic situations in which IRV can give poor results.

While every voting method has cases it handles poorly, some are better than others. One attempt to compare them is called [Voter Satisfaction Efficiency](http://electionscience.github.io/vse-sim/VSEbasic/) ([more details](http://electionscience.github.io/vse-sim/VSE/)). The idea is, you run a large number of simulations and see how different methods perform. It turns out that IRV does very poorly here, and if voters are highly strategic IRV does [even worse](http://electionscience.github.io/vse-sim/vse.html) than traditional plurality voting.

While I wish the voting method for us to consider were [Approval](https://en.wikipedia.org/wiki/Approval_voting) (or maybe [3-2-1](https://electowiki.org/wiki/3-2-1_voting) or [STAR](https://en.wikipedia.org/wiki/STAR_voting)), I do still think IRV is better than what we have today, and I'm planning on voting for it. One specific way in which IRV is an improvement is that it mostly doesn't, in its failings, benefit one type of party. This means that if we switch to IRV, and then as third-party candidates become stronger contenders we start to run into IRV's problems, it should be politically practical to switch to a better system. I do think there is some risk of setting back alternative voting systems in general by implementing an inferior version, but on balance I think the benefit of fixing the "minor spoiler" problem is likely larger.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100184817106872)*
# Are aircraft carriers super vulnerable in a modern war?

It seems like aircraft carriers are a good candidate for something that only exists because it made sense in WWII and it looks the part of “impressive military asset”, i.e. it’s all larping at this point. It seems vulnerable to attack relative to its huge cost because offense has an advantage over defense: I.e. can’t an enemy send tons of cheap drone planes and drone submarines to first hunt for its location and then swarm-attack it?

Note: I don’t know anything about this subject. The ideal answerer is someone with domain knowledge who has good epistemology; that’s why I wasn’t satisfied with a Google search and I’m asking here.
# How often do series C startups fail to exit?

How often do series C startups really fail? By fail I mean never have an acquisition or IPO. Internet says 80% (see https://medium.com/journal-of-empirical-entrepreneurship/dissecting-startup-failure-by-stage-34bb70354a36) but this seems very high to me.  
  
Most Series C companies are worth in the 100-200M range, the one I'm at is worth 270M. How does all the value just evaporate? What happens to the companies that "fail"?  
  
Asking to decide whether to exercise my options. I only need my company to exit at 41M to break even. I am bearish on the company but with around 40M in ARR it is hard to imagine it not exiting.
# Anthropomorphisation vs value learning: type 1 vs type 2 errors

The [Occam's razor paper](https://arxiv.org/abs/1712.05812) showed that one cannot deduce an agent $H$'s reward function ($R_H$ - using the notation from that paper) or their level of rationality ($p_H$) by observing their behaviour or even by knowing their policy ($\pi_H$). Subsequently, in a [LessWrong post](https://www.lesswrong.com/posts/9rjW9rhyhJijHTM92/learning-human-preferences-black-box-white-box-and), it was demonstrated that even knowing the agent's full algorithm (call this $a_H$) would not be enough to deduce either $R_H$ or $p_H$ individually.

In an [online video](https://www.youtube.com/watch?v=1M9CvESSeVc), I argued that the reason humans can do this when assessing other humans, is because we have an empathy module/theory of mind $E_H$, that allows us to model the rationality and motives of other humans. These $E_H$ are, crucially, quite similar from human to human, and when we turn them on ourselves, the results are similar to what happens when others assess us. So, roughly speaking, there is an approximate 'what humans want', at least in typical environments[^atyp], that most humans can agree on.

[^atyp]: See [these](https://www.lesswrong.com/posts/pfmFe5fgEn2weJuer/go-west-young-man-preferences-in-imperfect-maps) [posts](https://www.lesswrong.com/posts/k54rgSg7GcjtXnMHX/model-splintering-moving-from-one-imperfect-model-to-another-1) for how we can and do extend this beyond typical environments.

I struggled to convince people that, without this module, we would fail to deduce the motives of other humans. It is hard to imagine what we would be like if we were fundamentally different.

But there is an opposite error that people know very well: [anthropomorphisation](https://en.wikipedia.org/wiki/Anthropomorphism). In this situation, humans attribute motives to the behaviour of the wind, the weather, the stars, the stock market, cute animals, uncute animals...

![](https://www.dropbox.com/s/sxppdk7lfheceml/the_north_wind_and_the_sun_-_wind_-_project_gutenberg_etext_19994.jpg?raw=1)

So the same module that allows us to, somewhat correctly, deduce the motivations of other humans, also sets us up to fail for many other potential agents. If we started 'weakening' $E_H$, then we would reduce the number of anthropomorphisation errors we made, but we'd start making more errors about actual humans.

So our $E_H$ can radically fail at assessing the motivations of non-humans, and also [sometimes fails](https://www.researchgate.net/publication/16462204_Culture_and_the_Development_of_Everyday_Social_Explanation) at assessing the motivations of humans. Therefore I'm relatively confident in arguing that $E_H$ is not some "a priori" object, coming from pure logic, but is contingent and dependent on human evolution. If we met an alien race, they we would likely assess their motives in ways they would find incorrect - and they'd assess our motives in ways we would find incorrect, no matter how much information either of us had.
# The Haters Gonna Hate Fallacy

Occasionally I see people doing what I think of as the “Haters Gonna Hate Fallacy”.

The HGHF says something like: “People are going to misinterpret you no matter how carefully you word things. Therefore, there’s no point wasting time wording things carefully.”

An example:

“I think \[term X\] in your post is going to cause misunderstandings, I’d suggest phrasing it differently.”  
“Oh, haters are gonna hate, there’s no amount of rephrasing that’s going to prevent this from being misinterpreted if people want to.”

Now there’s obviously a grain of truth in this. It _is_ impossible to phrase something in a way that would always be interpreted correctly, and for pretty much any message there _are_ people who are hostile to it and who will twist it in the most uncharitable possible way.

The fallacy is in assuming that if you cannot avoid _all_ misunderstandings, there is no point in avoiding _any_ misunderstandings. Maybe 5% of your audience will dismiss the message no matter what, but 30% will dismiss the old phrasing while being receptive to the new phrasing.

This is most obvious if you take it to an extreme:

“Hey maybe you shouldn’t start your essay by saying that all of your readers are idiots who deserve to be shot.”  
“Eh, if that upsets them then they wouldn’t like me explaining the theory of general relativity anyway.”

Communication is hard and – importantly – contextual. Most of your readers will be reasonable people and assume you to use words to mean things they’re used to them meaning. If they’re used to word X being used differently than how you mean it, that doesn’t make them haters.

When I’ve fallen into something like the fallacy myself, it has often been motivated by an unwillingness to put in work. Other people should just understand me right away! “It’s beneath me to waste my time on doing other people’s interpretative work for them!” It’s dangerous to psychoanalyze others, but I have seen at least one person communicate unclearly, have that pointed out to them, then argue for why it was _right_ for them to be unclear… only to later on admit that they were _enjoying_ the frustration of being misunderstood.

Now avoiding misunderstandings _is_ a lot of work, and it’s totally valid not to bother! It’s alright to just focus on a particular target audience who understands you. I’m not saying that you _should_ always put in maximal effort into being understood – I certainly don’t.

But I do suggest owning up to it if you are _choosing_ to write something in a way that is going to cause misunderstandings that could have been avoided.

_Cross-posts: [Twitter](https://twitter.com/xuenay/status/1308376151732215810), [Facebook](https://www.facebook.com/Xuenay/posts/10160240681193662)._
# Sunday September 27, 12:00PM (PT) — talks by Alex Flint, Alex Zhu and more

This Sunday at 12pm (PT), we're running another session of "lightning talks" by curated LessWrong authors (see [here](https://www.lesswrong.com/tag/lesswrong-events) for previous weeks' transcripts).

*   For the first hour, we will have a series of lightning talks each lasting about 5 minutes followed by discussion. The talks will be short and focus on presenting one core idea well, rather than rushing through a lot of content.
*   From 1PM to 2PM, we will hangout in Gather.town. Link will be sent out closer to the event.
*   We want to give top LessWrong writers an interesting space to discuss their ideas, and have more fruitful collaboration between users. Think of it like a cross between an academic colloquium and some friends chatting by a whiteboard.

*If you're a curated author and interested in giving a 5-min talk at a future event, which will then be transcribed and edited, sign up* [*here*](https://forms.gle/iwFatbhys9muPmQA7)*.*

Speakers
--------

*   **Alex Flint:** "Why go on retreat?"
    *   **Curated posts:** [**The Ground of Optimization**](https://www.lesswrong.com/posts/znfkdCoHMANwqc2WE/the-ground-of-optimization-1), [**Reply to Paul Christiano on Inaccessible Information**](https://www.lesswrong.com/posts/A9vvxguZMytsN3ze9/reply-to-paul-christiano-on-inaccessible-information), [**Our take on CHAI's research agenda in under 1500 words**](https://www.lesswrong.com/posts/qPoaA5ZSedivA4xJa/our-take-on-chai-s-research-agenda-in-under-1500-words)
*   **Other speakers: TBA**

Details
-------

**When?** Sunday September 27, 12:00PM (PT)

**Where?** [https://us02web.zoom.us/j/87259855821](https://us02web.zoom.us/j/87259855821)
# The new Editor

| **Look** | **at** | **this** | **glorious** | **table** |
| --- | --- | --- | --- | --- |

**Celebrations! The new editor is finally here! **
--------------------------------------------------

Starting from today, all desktop users will by-default use the new visual editor that we've been testing for a while. While the primary goal of this is to have a better foundation on which to build future editor features, here are a number of things you can do starting from today: 

*   Insert tables! A heavily requested feature.
*   Copy-paste LaTeX without everything breaking!
*   Nest bullet lists and other block elements in blockquotes! (still no nested blockquotes though, though if enough people want that, it would be easy to change)
*   Image Uploads! (Just drag-and-drop images into the editor, and things should work out naturally. You can also copy-paste, though beware that copy-pasting from other websites means we link to the copies of those images from other websites, and won't reupload them.)
*   Much less jankyness and brokenness!

Let us know what you think about the new editor. We've been testing it for a while and have been pretty happy with it (and users who had opted into beta features also had predominantly positive feedback). You can also use the old editor if you run into any problems by checking the "Restore the previous WYSIWYG editor" checkbox in your user settings.
# Dehumanisation *errors*

In response to my post [contrasting value learning with anthropomorphisation](https://www.lesswrong.com/posts/LkytHQSKbQFf6toW5/anthropomorphisation-vs-value-learning-type-1-vs-type-2), steve2152 brought up the fact that [dehumanisation can be seen as the opposite of anthropomorphisation](https://www.lesswrong.com/posts/LkytHQSKbQFf6toW5/anthropomorphisation-vs-value-learning-type-1-vs-type-2?commentId=4PcxAHE9Cm82wvkoW).

I agree with this insight, but only when dehumanisation causes *errors* of interpretation. I was using empathy in the sense of "insight into the other agent", rather than "sympathy with the other agent".

In practice, dehumanisation does tend to cause errors. We see outgroups as [more homogeneous](https://en.wikipedia.org/wiki/Out-group_homogeneity), coherent, and organised than they actually are. Despite the suave psychopaths depicted in movies, psychopaths tend to be less effective at achieving their goals (as evidenced by the large number of psychopaths in prison). Torturers are [less effective](https://en.wikipedia.org/wiki/Effectiveness_of_torture_for_interrogation) at extracting true information than classical interrogators.

Now, it's not a universal law by any means, but it does seem that dehumanisation can often lead to errors, and from that perspective can be seen as a failure of value learning.

# The meaning of errors

* "Objection! Hold on just a minute!" screams the convenient strawman I have just constructed.

![](https://www.dropbox.com/s/q8r2v496nl82d0k/xbox_ban_objection-thumb-550x404-28829.jpg?raw=1)

* "You've claimed that 'agent's goals' are [interpretations by the outside observer](https://youtu.be/1M9CvESSeVc?t=810); that you can model a human as perfectly rational, [without being wrong](https://arxiv.org/abs/1712.05812). You've [claimed](https://www.lesswrong.com/posts/9rjW9rhyhJijHTM92/learning-human-preferences-black-box-white-box-and) that this is 'structured white box knowledge', which can't be deduced from the agent's policy or its algorithm."

* "Given that, how can you claim that anyone 'fails' at interpreting the goals of others, or that they make 'errors'?"

This a very valid point, strawman, but I've also [pointed out](https://www.lesswrong.com/posts/LkytHQSKbQFf6toW5/anthropomorphisation-vs-value-learning-type-1-vs-type-2) that human theory of mind/empathy is very similar from human to human, and tends to agree with how we interpret our own goals. Because of this, there is a rough "universal human theory of mind", ie a universal way of going from human policy to human preferences.

When I'm talking about errors, I'm talking about deviations from this ideal[^irr].

[^irr]: Because human theories of mind do *not* agree perfectly, there will always be an irreducible level of uncertainty in this ideal, but there is agreement on the broad strokes of it.
# Should it be a research paper or a blog post?

*The answer to this question may seem obvious to some, but let's see.*

My impression is that some people write long blog posts about things that they perceive as innovative new ideas in philosophy, AI research, or whatever. So how do you decide that your idea should be a blogpost (here or on some other site) instead of submitting it to a journal?

Is the university research system just too closed? Or do you think it is superfluous?

Or do you think that you can do it faster for a blog? (This may not be unambiguously good. My impression is that there is less literature research for some blog articles than I would like to see, which often creates noise.)

Or do you want to have feedback for a blogpost first and submit later?

(Note that the question can be generalized to other media and forms, e.g. magazine essay vs. journal article.)
# Spoiler-Free Review: The Stanley Parable

At Ben Pace’s recommendation, I recently played The Stanley Parable. Ben considers the game Tier 1.5. On reflection I consider this Tier 2. If you can spare the $15 for a short game, it’s Worth It to play.

If there’s one game I’ve played that needs to be played blind, this is it, so that’s all I’ll say.
# Petrov Event Roundup 2020

Petrov Day is this weekend. Various LessWrong folk have taken the opportunity to reflect on the [Day that Stanislav Petrov Didn't Destroy the World](https://www.lesswrong.com/posts/QtyKq4BDyuJ3tysoK/9-26-is-petrov-day). Some people have employed [ritual](https://www.lesswrong.com/posts/fr8MEigHzJeFQkctL/petrov-day-is-september-26), and others parties. Some employ [Big Red Buttons](https://www.lesswrong.com/posts/XJxwFMSL5TPN2usC6/modes-of-petrov-day) of some sort.

This year, many of the traditional manners people have commemorated the event are a bit different. 

**East Coast Petrov Megameetup**

The New York Rationality community will be holding an [Online Petrov Megameetup](https://www.lesswrong.com/posts/bh76S7YhiHYufjpk9/petrov-day-2020-virtual-celebration), which includes:

*   3:00pm - 3:15am Welcome
*   3:15pm - 4:15pm Ice Breakers
*   4:15pm - 5:30pm Lightning Talks & Activities
*   5:30pm - 6:00pm Snack/Water Break
*   6:00pm - 7:00pm Ritual

It's recommended that you get 8 candles/candleholders for the ritual if you can, but if not you can listen along, and maybe download a Candle App for your phone that you can use for a few key moments.

**Austin Petrov Day**

The folks in Austin are holding an [in person, masked, outdoor Petrov Day](https://www.lesswrong.com/events/7TMap3isj36wY9Rdz/austin-petrov-day-6-30pm-9-26).

> We have [modified](https://www.lesswrong.com/posts/jPSJTh5ctffyzFNB8/socially-distanced-outdoor-petrov-day-ceremonial-manual) the ceremonial manual to accommodate the outdoor setting. If possible, please bring a printed copy of the [double-sided booklet version](https://github.com/GeneralAntilles/PetrovDay/releases/download/1.3.COVID-19/PetrovDay-1.3-COVID-19-DoubleSidedBooklet.pdf) (or a mobile device on which you can read the [mobile-friendly version](https://github.com/GeneralAntilles/PetrovDay/releases/download/1.3.COVID-19/PetrovDay-1.3-COVID-19-MobileFriendly.pdf)), and a pen/pencil.

**Littleton, CO Petrov Day**

Littleton is having an [outdoor potluck](https://www.lesswrong.com/events/oYbztGevs9iT6LLMW/petrov-day-celebration-1):

> Come join us as we celebrate the world [not having been destroyed](https://www.lesswrong.com/posts/QtyKq4BDyuJ3tysoK/9-26-is-petrov-day), and raise a glass of Vodka to Stanislav Petrov.
> 
> We have chosen an outdoor venue for pandemic concerns, and will have a potluck for anyone willing to break bread during these troubled times. Masks are encouraged if you are not eating or drinking, and social distancing is recommended for anyone not already spending time together (some attendees have created quarantine circles, so don't be surprised if you see people ignoring typical precautions).

If you're hosting a Petrov Day celebration of some kind that people are welcome to join, please list them in the comments below!
# Up close and personal with the world

Explicitly building models of the world is often done through understanding the mechanics of how things work, the relations between them, causal effects and more. But I don't put enough emphasis on _quantifying_. I'm not talking about doing Fermi estimates because I do them and my rationalist(-adjacent) friends do them. There is a difference between Fermi-estimating quantities and knowing them from heart. I think there is value in the latter.

It is good to know, on an intuitive level, how a force of 100 newtons _feels_ like. To have an intuitive grasp of how much a volt is, and how some number of amperes feels like. Take acceleration and velocity, for example. You have a very intuitive grasp of them because of the experience of driving and looking at the speedometer. The idea is to expand the same intuition to other units of measurement. You want to _feel_ what the numbers mean. When you do this, you get very up close and personal to your model of the world. Things are not abstract, black boxes anymore.  

Tech-savvy dads were right all along when they were impressed by some specific number related to fuel consumption, or torque, or clamping pressure, or whatever, and I failed for not being interested in what seemed like random numbers and units. I understood the mechanism and the underlying idea, wasn't that enough? But it wasn't, because I was not fluent, I did not _own_ the model in my head. It didn't yet get that [quality of ordinariness](https://www.lesswrong.com/posts/Sdx6A6yLByRRs8iLY/fact-posts-how-and-why) that I now seek.
# Petrov Day is not about unilateral action

It's about the dangers of unrestrained escalation. It's a failure of humanity that we even got to the point where a few people's small and difficult decisions decided the fate of so much. This post is a friendly reminder to...

*   Please avoid escalating socio-economic disagreements to nuclear levels.
*   Please avoid escalating market competitions to Moloch levels.
*   Please avoid escalating paperclippers to superintelligent levels.

Learn to recognize signs of unrestrained escalation. You may be participating in unrestrained escalation...

*   If your information landscape becomes increasingly narrowed by a centralized source or by recommender algorithms.
*   If you attempt to fight over-escalated issues in a way that encourages opposing parties to fight back. For example, if you try to overtly starve the object of escalation of its resources and ability to act.
*   If your actions become increasingly coordinated with large and powerful groups.
*   If your judgment becomes increasingly dictated by stories told from a single point of view or single optimization criterion.

When you do recognize signs of unrestrained escalation, please take actions to de-escalate. You may do this...

*   By learning and teaching others to recognize unrestrained escalations.
*   By refusing to participate in group actions that encourage further escalation.
*   By starving the participants of their motivation (rather than of their resources and ability to act).
*   By using every escalation as a teaching opportunity for how to recognize and reduce future escalations.

And, of course, when considering whether a contentious situation warrants escalation, please do not divorce your opinions from the nuances of the current situation. In high-energy, high-pressure scenarios, small-scale effects often have large-scale consequences, and dually large-scale consequences are often driven by small-scale effects.

Thank you for your attention. Escalate responsibly.
# Notebook for generating forecasting bets

We made a [colab notebook](https://colab.research.google.com/drive/1YfCIhOJ6v0BcX-RB47ziJAvXj6lkr2Ay?usp=sharing&utm_source=lw&utm_medium=colab&utm_campaign=lwbettingcolab) that lets you generate a bet from two people's [Elicit](https://elicit.ought.org/) distributions. You can edit the notebook to generate your bet (the changes won't be saved). Here's an example of a suggested bet between Ben Pace and SDM on AI timelines:

Comparison of predictions:
--------------------------

[Snapshot link](https://elicit.ought.org/builder/xul7srb37)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8ea00123107ad2f6e1cbc99bff7f18fc43f9e9e12b4d5e65.png)

What the notebook outputs:
--------------------------

**If the event occurs between 24 Oct 2051 and 17 Sep 2059, Ben Pace should pay SDM $87. Otherwise, SDM should pay Ben Pace $13. **

**Disagreement:** You disagree most between 24 Oct 2051 and 17 Sep 2059. 

**Probabilities:** Ben Pace thinks the probability the event occurs in this range is 8%, while SDM thinks it's 18%. 

**Odds:** You should make a bet with 87:13 odds. 

**Expected Value:** Given your beliefs, you each win $5 in expectation

What this is actually doing:
----------------------------

*   We search across the question range to find the interval (with a width of 10% of the total range) where your probabilities differ the most, including the probabilities outside of the bounds
*   We determine odds that create equal and positive expected value for each person using the method outlined in [this blog post](https://statmodeling.stat.columbia.edu/2010/07/10/creating_a_good/)
# Blog posts as epistemic trust builders

I've really been enjoying Zvi's weekly posts on the coronavirus. Keeping up with what's going on is something I want to do, but not badly enough to put in the time myself. I'm not even sure how capable I would be even if I wanted to put in the time myself; it seems difficult to sift through all of the information out there.

Reading Zvi's posts works out perfectly for me though. 20-30 minutes a week and I get just what I need.

But all of this only works because I trust Zvi.

* * *

Like most people nowadays, I spend a lot of time online. In particular, reading blog posts. LessWrong, Overcoming Bias, Slate Star Codex, Hacker News, FiveThirtyEight, etc. When I need a break, I have my little routine of websites that I click through.

Sometimes I reflect on how much value I get out of reading all of these blog posts. Nothing against the authors, but when I finish an article, I usually am not left with the feeling that I've gained much. I see it as a numbers game: most of the time I don't gain much, but once in a while I come across something that really influences me.

But even when I'm not left off feeling particularly inspired by a post, I think that there is something more subtle that I gain by reading it: epistemic trust.

By reading the same authors over and over again, I start to get a feel for how much I can trust their reasoning ability. The more I trust them, the more I update in response to what they say. And when I reflect on the updates I perform, a surprisingly large proportion of them are of the (rough) form "I'll take your word for it".

* * *

The ultimate example of this is probably with respect to AI safety. I think AI safety is a huge deal, but the reason why I think so largely comes from me saying "I'll take your word for it". I have a very amateurish understanding of it all and wouldn't really be able to come to the conclusion "this is by far the most important thing in the world" via gears level reasoning.

But fortunately, I have a very high level of epistemic trust for the rationalist community. I've been able to cash in on this trust and update my beliefs about something that is very, very important.

* * *

I want to be careful about what I'm implying here. I'm not trying to imply that epistemic trust building is the *main* purpose of blog posts. I'm not even trying to be at all precise about how important I think that function is. My point is just that I think it's a function important enough to take note of.
# A long reply to Ben Garfinkel on Scrutinizing Classic AI Risk Arguments

In July, Ben Garfinkel scrutinized the classic AI Risk arguments in a 158 minute-long [interview](https://80000hours.org/podcast/episodes/ben-garfinkel-classic-ai-risk-arguments/) with 80000 hours, which I strongly recommend. 

I have formulated a reply, and recorded 80 minutes of video, as part of two presentations in the AISafety.com Reading Group:

[196. Ben Garfinkel on Scrutinizing Classic AI Risk Arguments](https://youtu.be/_kNvExbheNA )  

[197. Ben Garfinkel on Scrutinizing Classic AI Risk Arguments 2](https://youtu.be/j-_FvJ-XbWA )

I strongly recommend turning subtitles on. Also consider increasing the playback speed.

---

_"I have made this longer than usual because I have not had time to make it shorter."_  
-Blaise Pascal

The Podcast/Interview format is less well suited for critical text analysis, compared to a formal article or a LessWrong post, for 3 reasons:  

1. **Lack of precision.** It is a difficult skill to place each qualifier carefully and deliberately when speaking, and at several points I was uncertain if I was parsing Ben's sentences correctly.

2. **Lack of references.** The "Classic AI Risk Arguments" are expansive, and critical text analysis require clear pointers to the specific arguments that are being criticized.  

3. **Expansiveness.** There are a lot of arguments presented, and many of them deserve formal answers. Unfortunately, this is a large task, and I hope you will forgive me for replying in the form of a video.  

tl;dw: A number of the arguments Ben Garfinkel criticize are in fact not present in "Superintelligence" and "The AI Foom Debate". (This summary is incomplete.)
# What hard science fiction stories also got the social sciences right?

Robin Hanson says that those stories are rare.
> Most discussion of the non-immediate future seems to take place in science fiction, and the small subset of science fiction where authors try hard to remain realistic is called hard science fiction. Loosely associated with hard science fiction is an intellectual community of people who try to make projections which are true to our best understanding of the world. They work out the broad science and engineering of plausible future space colonies, starships, virtual reality, computer networks, survellience, software assistants, genetically engineered people, tiny machines made to atomic accuracy, and much more.
> 
> Unfortunately, few if any people of these people know much social science. So their projections often combine reasonable physics or computers with laughable economic assumptions. This often seriously compromises their ability to make useful projections.
(source: [The Economics of Science Fiction](http://mason.gmu.edu/~rhanson/econofsf.html))

This is different from [social science fiction](https://en.wikipedia.org/wiki/Social_science_fiction), which is deliberate exploration of other possible forms of sociological organisation.
# What are good rationality exercises?

I want to know what are good rationality exercises.

I was just on a call with Liron and PhilH, hanging out after the weekly LessWrong weekend event, and we discussed exercises that could happen on LessWrong.

Here is the list we generated:

*   Thinking Physics
*   Fermi Estimates
*   Project Euler
*   Calibration Training
*   Basic probabilistic reasoning
*   Basic have-you-read-the-sequences knowledge test (e.g. "Which of the following is an example of 'belief as attire'?")

Another user on the call (whose name I forget) suggested it could be fun to have a daily Fermi Estimate on LessWrong, where everyone submits their number and the model they used to reach the number. I think this would be quite exciting.

Please write answers with other exercises that you think are or might be great for rationality training, some explanation of why you think it could be good, and a suggestion of how it could be incorporated into LessWrong. I'll probably add some of the above myself.
# What to do with imitation humans, other than asking them what the right thing to do is?

This question is about whether you have clever ideas about how to use AI imitations of humans for AI safety. The two main ideas I'm familiar with only seem to interface with these imitations as if they're humans.

*   The most obvious thing one might do with a good predictor of a human is just to write software that queries the imitation human about what the right thing to do is, and then does it.
*   The less obvious thing to do is to try and amplify it - e.g. use teams of them working together to try to choose good actions. Or maybe even an IDA loop - use your learner that learned to imitate a human, and train it to imitate the teams working together. Then make teams of teams, etc.

But can we use human imitations to increase the effectiveness of value learning in a way other than amplification/distillation? For example, is there some way of leveraging queries to human imitations to train a non-human AI that has a human-understandable way of thinking about the world?

Keep in mind the challenge that these are only imitation humans, not oracles for the best thing to do, and not even actual humans. So we can't give them problems that are too weird, or heavily optimized by interaction with the imitation humans, because they'll go off-distribution.

Another possible avenue is ways to "look inside" the imitation humans. One analogy would be how if you have an image-generating GAN, you can increase the number of trees in your image by finding the parameters associated with trees and then turning them up. Can you do the same thing with human-imitating GAN, but turning up "act morally" or "be smart?"
# What are examples of Rationalist fable-like stories?

I will post some of the collection I've acquired. I have ideas for more. And would be interested in seeing more.

If there's enough interest, and with the permission of the authors, I would like to make a book of Rationalist fables / parables (but with a more generic name). Maybe it could be adapted for children. Thoughts?

Let me know if you're interested in helping me with this project, including but not limited to: finding stories, selecting stories, adapting them, reaching out to authors, writing more stories, writing their morales (?), making a book cover, (self?) publishing it, marketing it, choosing a title, donating money to contract external help, etc.
# What Decision Theory is Implied By Predictive Processing?

At a fairly abstract/stylized level, [predictive processing](https://slatestarcodex.com/2017/09/05/book-review-surfing-uncertainty/) models human cognition and behavior as always minimizing predictive error. Sometimes, the environment is "fixed" and our internal models are updated to match it - e.g. when I see my untied shoelace, my internal model updates to include an untied shoelace. Other times, our internal model is "fixed", and we act on the environment to make it better match the model - e.g. "wanting food" is internally implemented as a strong expectation that I'm going to eat soon, which in turn makes me seek out food in order to make that expectation true. Rather than having a utility function that values food or anything like that, the decision theory implied by predictive processing just has a model in which we obtain food, and we try to make the model match reality.

Abstracting out the key idea: we pack all of the complicated stuff into our world-model, hardcode some things into our world-model which we *want* to be true, then generally try to make the model match reality.

While making the model match reality, there will be knobs we can turn both "in the model" (i.e. updates) and "in reality" (i.e. actions); there's no hard separation between the two. There will be things in both map and reality which we can change, and there will be things in both map and reality which we can't change. It's all treated the same. At first glance, that looks potentially quite useful for [embedded agency](https://www.lesswrong.com/posts/p7x32SEt43ZMC9r7r/embedded-agents).

(My own interest in this was piqued partly because a predictive-processing-like decision theory seems likely to produce [abstraction boundaries which look like Cartesian boundaries](https://www.lesswrong.com/s/ehnG4mseKF6xALmQy/p/JasCkaPtZEJsYDX8H). As in that post, it seems like some of the intuitive arguments we make around decision theories would naturally drop out of a predictive-processing-like decision theory.)

What problems does such a decision theory run into? What sort of things can we hardcode into our world-model without breaking it altogether? What things must be treated as "fixed" when making the model match reality? Does such an approach have any "invariant" implications, i.e. implications independent of *which* model we're trying to match? What further requirements are there on the target model in order for a predictive-processing-style agent to have "good" behavior, in the ways characterized by other decision theories?

This is intended to be an open-ended research question, but off-the-cuff thoughts and links to relevant work are welcome.
# Examples of self-governance to reduce technology risk?

*Cross-posted from the [EA Forum](https://forum.effectivealtruism.org/posts/KJw6RDm4M6gAfqW6X/examples-of-self-governance-to-reduce-technology-risk)*

I'm searching for examples of self-governance efforts to reduce technology risk. Do people have cases to suggest?

The more similar to AI development the better. That is, efforts by companies or academic communities to address risks that affect third parties, with minimal involvement from governments beyond basic law and order.

Examples from academia:

*   Leo Szilard and other physicists coordinating to prevent Germany from obtaining atomic bomb data, 1939-1940
*   Various efforts in biotechnology:
    *   Asilomar conference on recombinant DNA, 1975
    *   Mutations Database Initiative, 1999-2001
    *   Synthetic biology conferences SB1.0 and SB2.0, 2006
    *   Biology journals discussing publication restrictions, 2001-2011

Examples from the commercial sector:

*   DNA synthesis companies screening orders and buyers, 2004-2012
*   Efforts by the nanotechnology companies in the US, UK and Europe, 2004-2007
# Three car seats?

I recently was linked to [Car Seats as Contraception](https://privpapers.ssrn.com/sol3/papers.cfm?abstract_id=3665046), which argues that car seat laws have led to parents choosing to have fewer children because "physical limitations of many cars precludes the use of three car seats in the back seat." Now, this is a correlational study, so I'm [skeptical that this is a real effect](https://www.jefftk.com/p/be-skeptical-of-correlational-studies), but the premise here is also strange. If the only way to have three children in your car was to buy a larger car, I see why that could be a major issue for a lot of people. On the other hand, buying a narrow car seat is pretty straightforward, and three narrow seats fit across the back of almost anything.

We have several [Cosco seats](https://www.walmart.com/ip/Cosco-Scenera-Next-Convertible-Car-Seat-Ocean-Breeze/406706457) that are 17" wide. They are one of the cheapest seats out there ($50), and they fit three-across in almost anything. Manufacturers publish "rear shoulder room" measurements for their cars, so roughly we're talking about cars with at least 51". Alternatively, if you're willing to spend 4x as much ($200, still small compared to the cost of a car or of raising a child) there are [16"](https://smile.amazon.com/dp/B07DWB2Q3C/) seats which would only need 48" across the back. Small cars I checked:

*   Toyota Corolla: [52](https://www.toyota.com/corolla/features/dimensions/1882/1856/1866)"
*   Honda Fit: [53](https://automobiles.honda.com/fit/specs-features-trim-comparison)"
*   Chevy Sonic: [53](https://www.caranddriver.com/chevrolet/sonic/specs)"
*   Ford Focus: [54](https://www.caranddriver.com/ford/focus/specs)"
*   Honda Civic: [55](https://owners.honda.com/vehicles/information/2020/Civic%20Sedan/specs)"
*   Kia Forte: [55](https://www.kia.com/us/en/forte/specs)"
*   Nissan Sentra: [55](https://www.caranddriver.com/nissan/sentra/specs)"
*   Hyundai Elantra: [55](https://www.caranddriver.com/hyundai/elantra/specs)"
*   Subaru Impreza: [56](https://www.caranddriver.com/subaru/impreza/specs)"

Of these, I think we've put our seats three-across in both the Toyota Corolla and Honda Fit, the two smallest on the list, without issue. It's more annoying than just putting in two, but if you own your car ([unlike us](https://www.jefftk.com/p/transportation-tracking)) you can leave them installed. [There](https://carseatblog.com/49714/3across/) [are](https://thecarseatlady.com/vehicles/3-across) [tons](https://www.babycenter.com/baby-products/car-seats/3-across-car-seats_40007326) [of](https://momlovesbest.com/narrow-car-seats) [blog](https://www.thecarcrashdetective.com/3-across-car-seat-guide-html/) [posts](https://www.elitecarseats.com/blog/fit-seats-3-across/) [on](https://www.consumerreports.org/car-seats/how-to-fit-child-car-seats-three-across/) [this](https://saferide4kids.com/blog/car-seats-fitting-3-across/). This mild hassle doesn't seem like it could be large enough to cause a substantial shift in whether or when people choose to have a third child?

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100186528307612)*
# Open & Welcome Thread – October 2020

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited.

If you want to explore the community more, I recommend [reading the Library,](https://www.lesswrong.com/library) [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/posts/2rWKkWuPrgTMpLRbp/lesswrong-faq-1#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/posts/2rWKkWuPrgTMpLRbp/lesswrong-faq-1). If you want to orient to the content on the site, you can also check out the new [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/tag/open-threads?sortedBy=new).
# Sunday Meetup: Workshop on Online Surveys with Spencer Greenberg 

Alternate title: "Geez LessWrong peeps do more empiricism please".

At this Sunday's LW meetup, special guest Spencer Greenberg will be running a workshop on how to run a psychology study using online tools. 

The LessWrong community raises a lot of interesting questions. I've long wished we did more empirical answering of those questions. In the past year I've become excited about several of the tools Spencer has created, which help make it easier to run surveys and studies over mechanical turk. I think it'd be valuable if a number of LessWrong users developed a basic affordance doing that. 

So this Sunday at noon PT, we'll be hosting a 2 hour workshop, with two phases:

**In the first half,** Spencer will walk us through how to use [Positly](https://www.positly.com/) (a wrapper for mechanical turk that gives you lots of demographic data and streamlines the process), and [GuidedTrack](https://www.guidedtrack.com/), which can be used to create flexible surveys and online experiments. He'll also talk about practices for good study design.

**In the second half,** people will work to create a simple project in Positly and Guided Track. Spencer will answer questions that come up. At the end, people are welcome to briefly share what they ended up working on.

The meetup will be held in Gather Town – we'll be experimenting with using their Presenter Mode functionality so that everyone can hear Spencer give his talk, and then chat in small groups as they work on their separate projects. Users will be able to "walk up to the microphone" to ask questions that the whole room can hear.

Gather Town instance:

https://gather.town/app/Dgohwk4BUg5HjGZE/lesswrong-clubhouse
# Muting on Group Calls

Ben recently wrote a post with a lot of advice on how to have [better video calls](https://www.benkuhn.net/vc/), and reading the [HN discussion](https://news.ycombinator.com/item?id=24610166) one of the [most](https://news.ycombinator.com/item?id=24633001) [controversial](https://news.ycombinator.com/item?id=24630177) suggestions was that you should figure out a mic setup where you can stay unmuted.

I see where this is coming from. We've all been on calls with people who had all sorts of extraneous background noise. It's distracting and frustrating, especially in large groups or when combined with call software that focuses whichever participant is the loudest. If you're just using your laptop's speakers and microphone, then yes, muting is good.

On the other hand, video calls are famously draining. They don't feel natural, and conversation doesn't quite work properly. A lot of this is latency ([No Bluetooth!](https://www.jefftk.com/p/wired-headsets-for-video-calls) [Run Ethernet!](https://www.jefftk.com/p/ethernet-is-worth-it-for-video-calls) [Wireless is a Trap!](https://www.benkuhn.net/wireless/)) but every barrier to conversational flow makes things worse. If you have to choose, muting yourself when you're not speaking is generally better than everyone hearing your background noise, but we have the technology so that you don't have to choose: point a directional microphone at your mouth.

[![](https://www.jefftk.com/headset-zoomed.jpg)](https://www.jefftk.com/headset-zoomed-big.jpg)

This works even in relatively noisy environments: my kids can be playing loudly, even shouting, right outside my door and my coworkers don't hear anything. [Compare](https://www.jefftk.com/p/wired-headsets-for-video-calls) my voice at the mic([mp3](https://www.jefftk.com/headset-test-spoken-wearing.mp3)) to 5ft away ([mp3](https://www.jefftk.com/headset-test-spoken-5ft.mp3)). I have a Jabra Evolve 40 (per the [Wirecutter](https://www.nytimes.com/wirecutter/reviews/best-usb-office-headset/)) and it works well. My guess is that anything else (wired!) with a boom mic should be pretty good, but I haven't tested.

Since it doesn't look like remote work will be ending anytime soon, if you have a job where are you do a lot of calls, making them feel natural is worth solid investment.
# One hub, or many?

I was reading the comments to [The Rationalist Community's Location Problem](https://www.lesswrong.com/posts/FghubkDy6Dp6mnxk7/the-rationalist-community-s-location-problem) and especially this comment by [Damien Tatum](https://www.lesswrong.com/posts/FghubkDy6Dp6mnxk7/the-rationalist-community-s-location-problem?commentId=nTZG7kv4WWaQP7DvF), and thinking about some classic questions of LessWrong/EA/fully-general physical community.

I think a guiding idea for years has been that if we get as many good people and organizations to have close proximity in the bay area, good things will happen. But I don't know whether that's been positive or negative overall.

For years I've heard about brain-drain problems in other would-be hubs. The best people move to the bay area, and the local hub either gets much worse or disappears. (I'm talking especially about meetup groups, but in the hypothetical where this sort of think didn't happen so much, we might also see more serious organizations in those other hubs.)

In a significant sense, this makes for a higher bar to getting involved. If what you have to do in order to get good in-person discussions with rationalists/EAs is *move to the bay area* rather than get to the closest decent meetup in your city/state/country, well, that puts a big damper on things.

If there have been discussions/modeling about this trade-off before, I'd especially appreciate links.

Edit: [The Berkeley Community & The Rest Of Us](https://www.lesswrong.com/posts/zAqoj79A7QuhJKKvi/the-berkeley-community-and-the-rest-of-us-a-response-to-zvi) discusses the brain-drain dynamic I'm discussing.