Harley-ml/lesswrong huggingface dataset:
# AI Research Considerations for Human Existential Safety (ARCHES)

Andrew Critch's ([Academian](https://www.lessestwrong.com/users/academian)) and David Krueger's review of 29 AI (existential) safety research directions, each with an illustrative analogy, examples of current work and potential synergies between research directions, and discussion of ways the research approach might lower (or raise) existential risk.
# [Resource Request] What are good resources for best practice for creating bureaucracy?

Especially, after just having read The Utopia of Rules: On Technology, Stupidity, and the Secret Joys of Bureaucracy by David Graeber, I feel that most bureaucracy we have is very bad and might create more harm then benefits. 

At the same time there are needs for bureaucracy. At Wikidata I care for it's functioning and it seems that some bureaucracy is needed to make norms legible for new users and also do reduce the necessity to waste time by having certain conflicts not resolved and refought over and over. 

While Wikimedia projects are in some sense different then other organizations, it's likely that studies of other bureaucratic structures might provide valuable knowledge. 

Is there any reading you can recommend?
# Featured Tags: July 2020

The LessWrong team is continuing to develop and validate the Tagging feature we introduced a few months ago. And we just [opened up tag creation to everyone!](https://www.lesswrong.com/posts/Kr46cTCoo7nkdRhdN/the-new-frontpage-design-and-opening-tag-creation) (at least for now)

As inspiration and because they're great, I wanted to share some of the best tags we've got so far. 

I've selected these for already having a pretty good collection of content on their respective topics.

*   [Scholarship & Learning](https://www.lesswrong.com/tag/scholarship-and-learning)
*   [Postmortems & Retrospectives](https://www.lesswrong.com/tag/postmortems-and-retrospectives)
*   [Decision Theory](https://www.lesswrong.com/tag/decision-theory)
*   [Financial Investing](https://www.lesswrong.com/tag/financial-investing)
*   [Progress Studies](https://www.lesswrong.com/tag/progress-studies)
*   [Conversation](https://www.lesswrong.com/tag/conversation-topic)
*   [Book Reviews](https://www.lesswrong.com/tag/book-reviews)
*   [Fiction](https://www.lesswrong.com/tag/fiction)

*Note that you can change the ordering of posts with the sorting selector.*

**If you want to get involved with tagging (please do):**

1.  You can: 1) *add tags to a post* from the posts page OR 2) *add posts to a tag* *page* at the bottom right of the tag page. 
2.  You can create tags anywhere you see an "add tag" button, from [www.lesswrong/tags/all](https://www.lesswrong/tags/all) or [www.lesswrong.com/tag/create](https://www.lessestwrong.com/tag/create).
3.  Any feedback is very welcome! We're doing a bunch of development know, so the more we hear from users the more we can incorporate that.

See the announcement for more details: [The New Frontpage Design & Opening Tag Creation!](https://www.lesswrong.com/posts/Kr46cTCoo7nkdRhdN/the-new-frontpage-design-and-opening-tag-creation)
# Talk: Key Issues In Near-Term AI Safety Research

I gave a [talk](https://www.youtube.com/watch?v=LHEE_iqzv-8) for the Foresight Institute yesterday, followed by a talk from Dan Elton (NIH) on explainable AI and a panel discussion that included Robert Kirk and Richard Mallah.

While AI Safety researchers are often pretty well aware of related work in the computer science and machine learning fields, I keep finding that many people are not aware of a lot of very related work that is taking place in an entirely different research community. This other community is variously referred to as Assured Autonomy, Testing Evaluation Verification & Validation (TEV&V), or Safety Engineering (as they relate to AI-enabled systems, of course). As I discuss in the talk, this is a much larger and more established research community than AI Safety, but unfortunately until recently there was very little acknowledgement by the Assured Autonomy community of closely associated work by the AI Safety community, and vice versa.

Recently organizations such as CSER and FLI have been doing a lot of great work helping to connect these two communities with jointly-sponsored workshops at major AI conferences - some of you may have attended those. But I still think it would be useful if more people in both communities were a bit more aware of the work of the other community. This talk represents my attempt at a short intro to that.

[Video](https://www.youtube.com/watch?v=LHEE_iqzv-8) (my presentation is from 2:28 to 19:00)

[Short version of slide deck](https://drive.google.com/file/d/193TNgNdyx5YYM7f3s0ySQM51FW3BJyPu/view?usp=sharing) (the one I used in the presentation)

[Longer version](https://drive.google.com/file/d/1-Mb2_h5UTp0wxObdXn-NZndrfEBa8b3j/view?usp=sharing)
# Mod Note: Tagging Activity is Not Private

We don’t yet have UI for it, so this isn’t obvious (hence the announcement), but it seemed good to clearly communicate that if you tag a post or vote on the tag relevance of a tag-post combination, this information may be viewed by other users and/or the mod team (after we built the relevant UI).

Tagging activity is unlike normal (karma-related) voting, for which the LessWrong team is fully committed to maintaining privacy for. 

Karma voting expresses a judgment of quality and/or approval, and it makes sense that for users to vote honestly, they need to feel that their votes will not be scrutinized or judged by others. In contrast, tagging activity – both tagging and tag voting – is an act of content creation for the community. In the same way we want to know the authors of posts and comments, we want to know the “authors” of the tags and tag pages that arise from tagging activity.

Knowledge of the content creators for tagging helps us both in positive ways, i.e. giving credit to those who make valuable contributions, and also in “negative” ways by making it easy to catch abuse/vandalism of the system. And by making it clear who is tagging in what way, it also becomes easier to have discussions about which tags apply where, which are good tags, etc.

A step beyond all this is that possibly the tagging feature will be extended into a full-fledged wiki feature. Obviously, wiki activity (creating and modifying wiki pages) needs to be linked to whoever is making changes. That makes it naturally the case that the tagging system (almost a proto-wiki) needs to link activity to user accounts.

All of the above makes it seem correct for tagging activity to not be private, unlike regular karma-related voting. We will soon launch the expanded UI for tagging that contains a history for each tag which will more clearly display what information is getting shared and how.

Sorry for not having been clearer about this earlier. If you have any concerns about this, don’t hesitate to reach out. Also if you made any activity in the last few weeks that you really don't want to be public, feel free to ping us and we can make sure that those stay private (DM, Intercom, or team@lesswrong.com are all good for reaching out about this.)
# How well can the GPT architecture solve the parity task?

Suppose I give it pairs of strings and ask it to output 1 if the the number of 1s in the string is even and zero if it's odd.

e. g. 

0 -> 0

1 -> 1

11 -> 0

101 -> 0

1101-> 1 

10101001 -> 0

111000101110 -> 1

How well does it do on this task? What if we finetune it on sample data?
# In praise of contributing examples, analogies and lingo

Examples, analogies and lingo all make posts easier to understand. So then, I want to encourage people to write more comments like:

*   X is an example of what you're describing.
*   Y might be a good analogy for this.
*   Z could be a good term for that.

I see comments like this sometimes, but much less frequently than I would like. And I suspect that readers often have thoughts like "Z could be a good term for that", but don't think it's worth saying. I think these comments are almost always worth saying.

Crowdsourcing
=============

Examples, analogies and lingo seem like such great things to crowdsource. Even if the post author is intimately familiar with the post's content, I would expect 100 readers who are vaguely familiar with the content to do a better job of coming up with examples, analogies and lingo than the single post author. They're just the sorts of things that benefit a lot from a diversity of viewpoints, it seems to me.

Non-expert explanation
======================

Let's say I'm wrong in the above section and the reality is that the post author is better than the crowd at coming up with good examples, analogies and phrases. Even so, sometimes the "good" example from the post author doesn't make much sense to a particular reader whereas a "mediocre" example from a commenter is what makes it click.

Scott Alexander does a great job of talking about this concept in his post [Non-Expert Explanation](https://web.archive.org/web/20200223103151/https://slatestarcodex.com/2017/11/02/non-expert-explanation/).

Post authors don't always do a good job
=======================================

Coming up with good examples, analogies and lingo is *hard*! There's a lot of times when I have an idea in my mind that I want to write about but I get stuck because I have trouble coming up with good examples, analogies or lingo. There are also times when I post it anyway, knowing that I haven't done a good job with those things.

I doubt I'm alone. I doubt that every time someone clicks "Submit" on a post that they think they've done as good a job as possible in the examples, analogies and lingo they've used.
# 2020 LessWrong Demographics Survey Results

Thank you to everyone who participated! You can see the [original post here](https://www.lesswrong.com/posts/mf2XKPfT2Brj8mho7/2020-lesswrong-demographics-survey), but the survey is now closed. Here are the results:

Google forms results: [https://docs.google.com/forms/d/e/1FAIpQLSfHJtg9ju6cCDl7T1PZvuBC9aIYIEN4Qb_MoQFPa9iMkUIFLQ/viewanalytics](https://docs.google.com/forms/d/e/1FAIpQLSfHJtg9ju6cCDl7T1PZvuBC9aIYIEN4Qb_MoQFPa9iMkUIFLQ/viewanalytics)

Copy of the original survey (to view the questions in context): [https://docs.google.com/forms/d/e/1FAIpQLSfBt1l3XBCZHx8DTJhWWTSK_bH7Tw6W4PdYYLKmMyplt1k0Ag/viewform](https://docs.google.com/forms/d/e/1FAIpQLSfBt1l3XBCZHx8DTJhWWTSK_bH7Tw6W4PdYYLKmMyplt1k0Ag/viewform)

  

_\[Anyone wanting to revive the old (larger) annual LW-surveys can contact me if they need help/funding.\]_
# Calibration Practice: Retrodictions on Metaculus

*Edit: there is now a cool app for this over here* [*https://www.quantifiedintuitions.org/pastcasting*](https://www.quantifiedintuitions.org/pastcasting)

I've been working on improving my calibration. I practiced on things like the [Credence Game](http://www.acritch.com/credence-game) and OpenPhil's [Calibration Game](openphilanthropy.org/calibration), and both were fine. But I felt like they didn't quite capture the felt senses from making real-world predictions about topics that I really cared about. 

After [exploring the options,](/posts/HzRLuTPZYbqPau6eu/what-are-the-best-tools-for-recording-predictions) I started making predictions on [PredictionBook](predictionbook.com) – it was simple, lightweight, and let me enter dates for prediction-resolution using intuitive English phrases like "a week from now" or "August 1st" instead of having to go hunting through the date picker. 

I've found it generally useful to notice when I've formed an implicit prediction (for example: "My coworker isn't going to get their project done today", or "Covid cases will be lower-per-capita at this particular town that I'm planning to move to"), then actually enter it into PredictionBook. 

If I notice an implied prediction while I'm not near my computer, I sometimes make the prediction as *if* I were at PredictionBook, and then see it resolve, and then enter everything into PredictionBook after the fact to see how it affects my overall calibration. (One example, when I had taken a walk and gotten lost: "At this next road I'm supposed to turn *right",* which turned out to be correct)

**Retrodictions on Metaculus**

My prediction-habits still didn't add up to *that* many real-world predictions. But, I found a neat way to practice calibration on a bunch of real-world questions in a row is to go to [Metaculus](https://www.metaculus.com), pick a [category](https://www.metaculus.com/questions/categories/) with a lot of questions, and then filter by *resolved* questions. 

By default this will show you questions along with the resolution, but you can hide the resolutions by creating a custom style using the Stylus browser extension ([chrome](https://chrome.google.com/webstore/detail/stylus/clngdbkpkpeebahjckkjfobafhncgmne?hl=en) / [firefox](https://addons.mozilla.org/en-US/firefox/addon/styl-us/)). The relevant CSS is:

    .question-table__header > :nth-child(2), .question-table__row > :nth-child(2) {
        display:none
    }
    
    .question-table__header > :nth-child(3), .question-table__row > :nth-child(3) {
        display:none
    }

Some questions were ones I already knew the answers to, but for many of them there was still some uncertainty about exactly how some event had played out, enough such that practicing calibration on it was useful.

Overall, the number of questions turned out to be not be as large as I'd originally thought, but still enough to make it a good exercise.

Happy Calibrating.
# Dremeling

In high school, I was on a robotics team (FRC) who, in honesty, were basically incompetent. (Me included.) We were bad at building a robot, bad at programming it, bad at design, but most relevantly for this post bad at _planning ahead_.

The particular instance which added a term to my internal monologue was this: We needed to cut slots in some aluminum in order to do...something, I don't remember what. It needed to be about a foot long, so it clearly was worth getting a dedicated tool - I forget what that was, probably a jigsaw. But we didn't have one and it was going to take a week or so to arrive.

What we did have, was a dremel tool. Dremels are small, portable grinders, usable for many things but rarely the best tool for the job. Using a dremel as a slot cutter is workable, but inefficient; it quickly ruins the bit and takes a long time. Nonetheless, despite knowing that the job could be done much more quickly and efficiently if they waited a week, people on our team repeatedly returned to dremeling the slots.

I think this is a common pattern. It's similar to the politician's syllogism ("Something must be done. This is something. Therefore, this must be done."), and to [Lost Purposes](https://www.lesswrong.com/posts/sP2Hg6uPwpfp3jZJN/lost-purposes), but distinct. "Dremeling" is the act of doing something you know is a bad way to attack the problem, and probably a waste of time, because it moves you incrementally towards the goal, and you can't do anything more helpful _right now_.

There are better ways; you know there are better ways and have at least a pretty good idea of what they are and how you could switch to one of those ways. But, _in the moment_, the better ways are unavailable and the bad way is available. And so you do the (dumb, wasteful, barely-helpful) thing you can do _right now_, because it will technically make progress.

Why does this happen? Well, not doing anything is uncomfortable. It often feels like failure. If others see you not doing anything, it may open you up to shame; at the national scale this is the politician's syllogism, but the politics of a peer group are just as terrifying - and probably more terrifying - than the party politics of a large polity.

So, as we learn from (Brooks, Reiner, 1975), [it stems from fear](https://www.youtube.com/watch?v=2aHxL0gTbmQ&t=470).
# What do we now know about long-term consequences of a COVID-19 infection?

At the beginning there were concerns about permanently reduced lung-function and CFS after some COVID-19 infections. I haven't kept up with the discourse in the last two months. What's the current state of what we now about the long-term effects?
# How big of an issue are patent trolls to the average startup?

There are many individual stories about patent trolls being problematic, but I have a hard time estimating how big of an effect their activity happens to be overall. Can someone point me to guide analysis of the problem overall that estimates their impact?
# Sunday July 19, 1pm (PDT) — talks by Raemon, ricraz, mr-hire, Jameson Quinn

This Sunday at 1pm (PDT), we're running another session of "lightning talks" by curated LessWrong authors (see [here](https://www.lesswrong.com/tag/lesswrong-events) for previous weeks' transcripts).

*   Each talk will be 3-5 minutes followed by discussion. Afterwards, we'll have a hangout in breakout rooms. The talks will be short and focus on presenting one core idea well, rather than rushing through a lot of content.
*   We want to give top LessWrong writers an interesting space to discuss their ideas, and have more fruitful collaboration between users. Think of it like a cross between an academic colloquium and some friends chatting by a whiteboard.

*If you're a curated author and interested in giving a 5-min talk at a future event, which will then be transcribed and edited, sign up* [*here*](https://forms.gle/iwFatbhys9muPmQA7)*.*

Speakers
--------

*   **Raemon** — "Intentional Privacy Practices"
    *   Curated posts: [The Schelling Choice is "Rabbit", not "Stag"](https://www.lesswrong.com/posts/zp5AEENssb8ZDnoZR/the-schelling-choice-is-rabbit-not-stag), [Noticing Frame Differences](https://www.lesswrong.com/posts/f886riNJcArmpFahm/noticing-frame-differences), [Propagating Facts into Aesthetics](https://www.lesswrong.com/posts/YN6daWakNnkXEeznB/propagating-facts-into-aesthetics) ...
*   **ricraz **
    *   Curated posts: [Book review: The Sleepwalkers by Arthur Koestler](https://www.lesswrong.com/posts/3Ts4GrXGQEqTinPD2/book-review-the-sleepwalkers-by-arthur-koestler), [Realism about rationality](https://www.lesswrong.com/posts/suxvE2ddnYMPJN9HD/realism-about-rationality), [Some cruxes on impactful alternatives to AI policy work](https://www.lesswrong.com/posts/DJB82jKwgJE5NsWgT/some-cruxes-on-impactful-alternatives-to-ai-policy-work)
*   **mr-hire**
    *   Curated posts:  [The 3 Books Technique for Learning a New Skill](https://www.lesswrong.com/s/myahjP2SYWEtSEPrH), [S-Curves for Trend Forecasting](https://www.lesswrong.com/posts/oaqKjHbgsoqEXBMZ2/s-curves-for-trend-forecasting)
*   **Jameson Quinn**
    *   Curated post: [A voting theory primer for rationalists](https://www.lesswrong.com/posts/D6trAzh6DApKPhbv4/a-voting-theory-primer-for-rationalists)

Details
-------

**When?** Sunday July 19, 1pm (PDT)

**Where?** [Zoom]( https://us02web.zoom.us/j/86790479361).
# Anthropomorphizing Humans

Sometimes, we assign desires and emotions to cars. We say that the car wants to drive, or we say that a strange vibration is the car expressing its displeasure at a too-long gap between oil changes. That’s anthropomorphization: we imagine that non-human objects have desires and emotions driving their behavior. We model the car as having desires (i.e. oil changes), emotions in response to those desires being met or not met, and observable behavior corresponding to those emotions. Of course, the actual cause of the car’s behavior is much more mechanical - low oil or coolant, built-up sludge, etc.

Now, consider [hangriness](https://www.merriam-webster.com/words-at-play/hangry-meaning).

I don’t notice that I’m hungry. I notice that the trash is overflowing and hasn’t been taken out. I feel angry about the trash, so I model myself as angry *because* of the trash. If someone says “why are you angry?”, I talk about how I want a clean house, and how annoying it is that the trash has not been taken out. But the actual cause is simply low blood sugar, or something like that.

This is anthropomorphization of myself: I imagine that my behavior is driven by some desire (i.e. I want a clean house) and the frustration of not having that desire met. Yet the actual cause is much more mechanical, and unrelated to the supposed desire.

Likewise, we often anthropomorphize other humans. If someone else is hangry, I might notice their anger and ask them what they’re angry about, without realizing that they just haven’t eaten in a while. In general, if I ask someone why they think X or why they decided Y, they’ll come up with a whole explanation for why X or Y makes sense, which may or may not have anything at all to do with the actual causes of their belief/decision - i.e. they rationalize post-hoc. Mistaking that post-hoc justification for the actual cause of the belief/action would be anthropomorphization.

Empathic Reasoning
------------------

Empathic reasoning is especially prone to the anthropomorphization failure mode in general, and to anthropomorphization of humans in particular.

Empathic reasoning is all about putting yourself in someone else’ shoes, asking “What do I want? What do I feel?”, and explaining behavior in terms of those wants and feelings. Essentially, empathic reasoning *assumes* the anthropomorphic hypothesis - it assumes that behavior is a result of desires and emotions - and tries to back out those desires and emotions by simulating oneself in the same situation.

In cases like hangriness, where the real cause diverges heavily from the first-person experience, that’s going to be highly misleading. Empathy may yield a good idea of what the situation feels like/looks like to another person, but the other person’s experience includes a wildly inaccurate model of the underlying causes. If we’re going to leverage empathic reasoning successfully, we need to be very careful about separating what the person perceives from reality, and in particular separating what the person perceives as causing their beliefs/behavior from what actually causes their beliefs/behavior.
# "Should Blackmail Be Legal" Hanson/Zvi Debate (Sun July 26th, 3pm PDT)

This weekend's LessWrong event is a debate! Robin Hanson and Zvi Mowshowitz will be debating blackmail this coming Sunday.

The Zoom link is here: [https://us02web.zoom.us/j/85013937335](https://us02web.zoom.us/j/85013937335). ([Facebook event.](https://www.facebook.com/events/2114445585359306))

We will release a recording and a transcript afterwards. There'll be a Q&A (live on Zoom).

See you there :)

**Background**

Robin's initial post on the subject is [Checkmate on Blackmail](http://www.overcomingbias.com/2019/02/checkmate-on-blackmail.html).

Zvi's initial post on the subject is [Blackmail](https://www.lesswrong.com/posts/EYd63hYSzadcNnZTD/blackmail) (epistemic status: surprisingly controversial).

Paul Christiano and Tyler Cowen both wrote posts too, [Checkmate on Blackmail?](https://sideways-view.com/2019/06/02/checkmate-on-blackmail/) and [Why should blackmail be illegal?](https://marginalrevolution.com/marginalrevolution/2019/02/why-should-blackmail-be-illegal.html) respectively.

There's also a [blackmail tag](https://www.lesswrong.com/tag/blackmail) for various related posts (someone should write a description and add any missing posts).
# If you are signed up for cryonics with life insurance, how much life insurance did you get and over what term? 

Hey all,

I'm been cryo-crastinating, but I'm trying to push through and actually get signed up in the next two months.

A next step is that I need to buy life insurance. I'm a little bit at a loss, because I'm not sure how much I should get. It seems like I want _some_ margin of error above the minimum amount required by the cryonics org that I'm signing up with, but I'm not sure how much is a reasonable margin of error.

If you have life insurance for cryonics, how much did you get, and why? What is the term of your policy, and why did you select that length of time?

Also, for those of you who paid / are planning to pay out of pocket, why did you opt to do that instead?

Thanks!
# Online Meetup: Balaji Srinivasan on "Nation of Emigrants"

Balaji Srinivasan will speak on **"A Nation of Emigrants: How Technology Will Enable Collective Bargaining with Governments"**. See abstract below.

After the talk and Q&A, we will also allow plenty of time for online mingling and discussion.

[Click here to register,](https://forms.gle/PHD9nhYfrfbhN2wc9) up to an hour before the talk, and we'll send you an invitation to the Google Meet.

**Abstract**:

Technology is the driving force of history. It lies upstream of culture, and thus upstream of politics. And it determines which fringe ideas have become newly feasible, and what elements of official consensus have become suddenly obsolete.

Our thesis has five stages. First, we describe how recent technological developments have dramatically shifted the balance of power away from states and towards individuals. Next, we diagnose the problems with existing states - particularly related to immigration and regulation - that would drive individuals to seek alternatives. Third and fourth, we give a concrete set of recommendations enabled by modern technology that companies and individuals alike can act upon to gain financial independence.

Finally, we describe how the combination of technological driving forces, regulatory repulsion, and newly independent individuals leads to a novel phenomenon that we term crowdchoice: internet-coordinated mass migrations that enable collective bargaining with governments.
# Online Meetup: Balaji Srinivasan on "Nation of Emigrants"

In our ongoing SlateStarCodex online series, Balaji Srinivasan will speak on **"A Nation of Emigrants: How Technology Will Enable Collective Bargaining with Governments"**. See abstract below.

After the talk and Q&A, we will also allow plenty of time for online mingling and discussion.

[Click here to register,](https://forms.gle/PHD9nhYfrfbhN2wc9) up to an hour before the talk, and we'll send you an invitation to the Google Meet.

The talk is August 2 at at 10:30 PDT, 17:30 GMT, 20:30 IDT. [This link shows your local time](https://www.thetimezoneconverter.com/?t=10%3A30&tz=San%20Francisco).

**Abstract**:

Technology is the driving force of history. It lies upstream of culture, and thus upstream of politics. And it determines which fringe ideas have become newly feasible, and what elements of official consensus have become suddenly obsolete.

Our thesis has five stages. First, we describe how recent technological developments have dramatically shifted the balance of power away from states and towards individuals. Next, we diagnose the problems with existing states - particularly related to immigration and regulation - that would drive individuals to seek alternatives. Third and fourth, we give a concrete set of recommendations enabled by modern technology that companies and individuals alike can act upon to gain financial independence.

Finally, we describe how the combination of technological driving forces, regulatory repulsion, and newly independent individuals leads to a novel phenomenon that we term crowdchoice: internet-coordinated mass migrations that enable collective bargaining with governments.
# Replicating the replication crisis with GPT-3?

I am getting worried that people are having so much fun doing interesting stuff with GPT-3 and AI Dungeon that they're forgetting how easy it is to fool yourself. Maybe we should think about how many different cognitive biases are in play here? Here are some features that make it particularly easy during casual exploration.

First, it works much like autocomplete, which makes it the most natural thing in the world to "correct" the transcript to be more interesting. You can undo and retry, or trim off extra text if it generates more than you want.

Randomness is turned on by default, so if you try multiple times then you will get multiple replies and keep going until you get a good one. It would be better science but less fun to keep the entire distribution rather than stopping at a good one. Randomness also makes a lot of gamblers' fallacies more likely.

Suppose you don't do that. Then you have to decide whether to share the transcript. You will probably share the interesting transcripts and not the boring failures, resulting in a "file drawer" bias.

And even if you don't do that, "interesting" transcripts will be linked to and upvoted and reshared, for another kind of survivor bias.

What other biases do you think will be a problem?
# Restocking

The pandemic has illustrated the resiliency benefits of keeping extras on hand. During the grocery panic, when everyone was trying to stock up on weeks' worth of groceries at the same time, people who had extra set aside didn't need to go out to a potentially dangerous environment. Similarly, by avoiding shopping at this time when there was sudden massive demand, these people were able to help blunt the shock.

It's far from over, but at least around here the supply chain seems to mostly have recovered. You still can't get N95 masks, but there's no trouble getting flour, rice, beans, toilet paper, etc. If you've let your supply run down over the last few months, or didn't have extra set aside to begin with, now would be a good time to think about fixing that. Not only could the pandemic get worse in a way that starts to threaten the supply chain again, but many more conventional disasters, like earthquakes or hurricanes, could be much worse with the pandemic as a background.

[![](https://www.jefftk.com/beans-and-pasta-basement.jpg)](https://www.jefftk.com/beans-and-pasta-basement-big.jpg)

As I wrote [just before the pandemic](https://www.jefftk.com/p/disasters), it's worth putting time into thinking about potential disasters and getting at least somewhat prepared for them. Lots of links in that post if you're looking for advice on how to approach this.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100175566150862)*
# Photos Before Drywall

Houses are strange: customarily, there's no documentation. Want to know where the plumbing is? Where the wires are? Where the structure is sturdy enough to support something heavy? Sorry, the walls are in the way. You can't see it, and no one wrote down where to find it.

Sure, we have workarounds. Stud finders can tell you what the structure looks like, unless your walls are thick (most of mine are two layers of drywall over lath and plaster...). Non-contact voltage probes can detect live wires, unless they're shielded, or not near the surface of the wall. You can make decent guesses about where it would've made sense to run things, but past people who have worked on the house have likely made all sorts of strange decisions for reasons that are now lost. All of which is to say, when you cut or screw into a wall, you're always taking a bit of a risk.

With that in mind, any time you have the finished surface of your house opened up, please take lots of pictures! Take pictures of the walls, the ceiling, the floors, anything. Later on, if you think you might need to open one of those places up for something, you will be glad you did!

[![](https://www.jefftk.com/bathroom-old-and-new-stack.jpg)](https://www.jefftk.com/bathroom-old-and-new-stack-big.jpg)

Even though I've had this approach for years, I still often find myself wishing I had taken more pictures.

I wish there were a culture of this kind of picture taking, and including them in the sales of our houses. One where a house would be worth a bit more if there was documentation for the parts you can't see, and where people took absence of documentation as a sign that a person might be trying to hide something. Perhaps now that most everyone carries a good camera on them at all times this will change, but I expect not. Among other things, providing pictures means likely opening yourself up to some kind of liability, in case anything wasn't done perfectly. If only people would [consume transparency responsibly](https://www.jefftk.com/p/responsible-transparency-consumption)!

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100175826783552)*
# Using books to prime behavior

My purpose for reading has shifted from "I want to learn this" to "I want to use this to prime my behavior".

After years of consuming psychology, business management and generally "how-to" books(as opposed to "what" books), I've learnt that I will only have gotten something tangible out of the book if it changes my behavior. If some of my mental framework(with regards to eg social interaction, persevering in hard workouts or challenging problem sets etc) shifts and I'm acting in a different way than before.

This works especially for biographies. When reading about someone I emulate, I tend to step into his/her shoes and adopt his/her mental disposition for a trait I'm weak in, or a similar problem I'm trying to solve in my daily life.

Problem is, this priming doesn't last for long. After a week or two the effects tend to be diminished. All the accumulated knowledge of "stand up confidently straight as if you were being pulled at the crown of your head by a string from the ceiling", or "the mind is like a muscle too, and can be trained by effortful focus" tend to fall prey to the forgetting curve.

This might be a sign that the knowledge has become unconscious, implicit. It would however be nice if I can consciously _feel that I'm using it_ that so as to motivate further efforts.

One can used spaced repetition techniques to shift hard facts to long-term memory. My question is, how can a similar thing be done for soft skills and mental frameworks?
# Construct a portfolio to profit from AI progress.

It is probably impossible to hedge against a FOOM scenario. But it seems like there are plausible 'slow' but still transformative AI scenarios we might want to hedge against.

Here is an example of a portfolio you could construct:

40% VUG (covers most larger us tech companies, has some stuff you don't specifically want but gives you some hedging)

40% IGV (smaller us tech companies)

20% CQQQ (china, really high expenses but oh well)

A related question is how you could add exposure to an existing more balanced portfolio. (example answer: Buy CQQQ and IGV)

I am fully aware that trying to get lots of exposure to AI progress is the opposite of diversification for most lesswrong users. But if you are expecting rapid progress soon it might still be worth doing.

Some sub-questions that might be useful:

\-\- Which US Tech ETfs are most useful

\-\- Should people buy specific stocks instead of ETFs? Which ones?

\-\- How should we get Exposure to China? Do we want/need China tech exposure?

\-\- How do we get exposure to areas besides the USA and China? Do we need to?

\-\- Should we invest in land if we expect AI progress?

\-\- Should we directly invest in hardware (Nvidia has already gone up a lot)?
# Jam is obsolete

Jam is a very natural way to preserve fruit. The fruit is ripe only a small fraction of the year, you'd like to be able to eat it all year long, so you cook it with enough sugar that it won't go bad for a long time. But now that we have freezers, jam is obsolete.

When you make jam, you change the taste and texture of the fruit in several ways:

*   Squishing the fruit makes it into a more consistent texture, suitable for spreading.
*   Adding sugar makes the jam sweeter, and also much less healthy.
*   Heat changes the flavor of the fruit, With more volatile flavors evaporating.
*   Heat evaporates some of the water, thickening it (reducing).
*   Heat releases pectin, or you add additional pectin, which gels the jam.

Some people make freezer jam, which skips the heated steps, and gives you a flavor much closer to fresh fruit. Because it hasn't been cooked thoroughly, you need to store it in the freezer, but since freezers are a thing now, that's not a problem.

We can go a step further, though, and just use frozen fruit. Take frozen raspberries, defrost them, and squish them up with a fork:

[![](https://www.jefftk.com/raspberry-non-jam-bowl.jpg)](https://www.jefftk.com/raspberry-non-jam-bowl-big.jpg)

[![](https://www.jefftk.com/raspberry-non-jam.jpg)](https://www.jefftk.com/raspberry-non-jam-big.jpg)  
*on [eggy waffles](https://www.jefftk.com/p/eggy-waffles), with nutella*

This takes about a minute for defrosting, 10s for squishing, and is far superior to jam. Not so sweet, better flavor, and it's cheaper \[1\] and healthier as well!

Being able to store traditional jam in the refrigerator once you've opened it is an advantage, but I think it's a temporary one. Over time, I expect households will shift towards more freezer space relative to refrigerator space, as the quality of frozen food continues to rise. We got a chest freezer soon after we bought our house, and I'm glad we did!

  
\[1\] Our grocery store has raspberry jam for $2.75/lb, frozen raspberries for $4.39/lb, and jam is ~40% fruit. Though since we have the freezer space like raspberries a lot I've been buying them online, [30lb](https://www.webstaurantstore.com/30-lb-iqf-raspberries-pieces-and-whole/877RSPPC30.html) at $3.08/lb shipped. A shipment fills 6-8 one-gallon ziplocs depending on whether it ends up mostly whole or mostly pieces.

[![](https://www.jefftk.com/30lb-of-bagged-raspberries.jpg)](https://www.jefftk.com/30lb-of-bagged-raspberries-big.jpg)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100176016678002)*
# What specific dangers arise when asking GPT-N to write an Alignment Forum post?

Last year Stuart Armstrong [announced a contest](https://www.alignmentforum.org/posts/cSzaxcmeYW6z7cgtc/contest-usd1-000-for-good-questions-to-ask-to-an-oracle-ai) to come up with the best questions to ask an Oracle AI. Wei Dai [wrote](https://www.alignmentforum.org/posts/cSzaxcmeYW6z7cgtc/contest-usd1-000-for-good-questions-to-ask-to-an-oracle-ai?commentId=JMABP4HCXFvAX8JXw),

> Submission. For the counterfactual Oracle, ask the Oracle to predict the n best posts on AF during some future time period (counterfactually if we didn’t see the Oracle’s answer).

He later [related](https://www.alignmentforum.org/posts/cSzaxcmeYW6z7cgtc/contest-usd1-000-for-good-questions-to-ask-to-an-oracle-ai?commentId=xccNqxt3eas3LCBPv) his answer to Paul Christiano's posts on [Human-in-the-counterfactual-loop](https://ai-alignment.com/counterfactual-human-in-the-loop-a7822e36f399#.vvtw3tjqh) and [Elaborations on apprenticeship learning](https://ai-alignment.com/elaborations-of-apprenticeship-learning-eb93a53ae3ca#.5ubczdqf0). Here I'm interested in _concrete_ things that can be expected to go wrong in the near future if we gave GPT-N this task.

To provide a specific example, suppose we provided the prompt,

> _This is the first post in an Alignment Forum sequence explaining the approaches both MIRI and OpenAI staff believe are the most promising means of auditing the cognition of very complex machine learning models._

If by assumption, GPT-N is at least as good as a human expert team at generating blog posts, we could presumably expect this GPT-N to produce a very high quality post explaining how to inspect machine learning models. We would therefore have a way of to automate alignment research at a high level. But a number of important questions remain, such as,

*   How large would GPT-N need to be before it started producing answers comparable to a human expert team, and
*   Given the size of the model, what high-level incentives should we expect to guide the training of the model? In other words, what [mesa optimization](https://www.lesswrong.com/s/r9tYkB2a8Fp4DN8yB)-like instantiations can we expect to result from training, exactly, and
*   Is there a clear and unambiguous danger that the model would be manipulative? If so, why?
*   Is the threat model more that we don't know what we don't know, or that we have a specific reason to believe the model would be manipulative in a particular direction?
# Billionaire Economics

I have progressive friends in my circles and so I see a lot of stuff like this:

![](https://pbs.twimg.com/media/Ednp23aWkAA8AqF.jpg)

As well as somewhat more eloquent takes [like this one](https://mkorostoff.github.io/1-pixel-wealth/?fbclid=IwAR0keGKk-wnjNBwpzcQMrQzQTjMmy2WbSqvmg92vN5CTYtTH8lUsuzw9eIg), and a number of similar things I haven't bothered digging up.

But I notice that I'm still confused. The entry-level mistake (which some progressives do make) is thinking that having a net worth of $48 billion means that you have $48 billion sitting in a bank account somewhere. It's clear that none of them have that kind of liquidity. I've even seen, very occasionally, [back and forth takes on these issues](https://github.com/MKorostoff/1-pixel-wealth/blob/master/THE_PAPER_BILLIONAIRE.md). So not everyone is making this mistake.

Ok, I'm still confused. What is going on?

*   Possibly estimates like this are just way off on how much money it takes to solve problems. Is $20 billion really enough to end homelessness in the US? End for how long? [The NYT suggests that we already spend about $12 billion](https://www.nytimes.com/2020/05/15/opinion/sunday/homeless-crisis-affordable-housing-cities.html) (though perhaps not very well), and that's **per year**! $300 billion does seem awfully optimistic for halting climate change. Maybe no one's personal fortune is actually enough to do _any_ of these things.
*   Still, with several billion dollars, you could probably do _something_. Let's assume that there is at least one major, important thing you could do with a one-time investment of $20 billion (seems reasonable). Billionaires are well-informed, they probably know about whatever that thing is. There are several who could apparently afford it. Surely one of them would "defect" and burn half of their fortune on it? This suggests that either 1) billionaires really are universally evil just as progressives think, 2) no opportunity like this exists and billionaires are not, in practical terms, all that rich, or 3) billionaires can't access all of their wealth for some reason.
*   (Or 4) billionaires do this already and I don't know about it? IDK)
*   One angle on that last point is like this: If Mark Zuckerberg wanted to liquidate all $54.7 billion of his wealth to give it away to some cause (give every American... wow only $182.33, can that be right?), how quickly / easily could he do that? Would he be able to do it without any trouble? What are the barriers in his way? Does he simply not have access to that kind of liquidity?

I know that's not especially clear but I hope that y'all can help me out. Any thoughts or discussion on these points would be welcome and appreciated. Thank you!
# Asset Prices Consistently Violate Efficient Market Hypothesis

Lyall Taylor points out two reasons why asset prices consistently violate the EMH:

(1) Liquidity flywheels make the market inefficient from a long-term value perspective, because shorter-term momentum dynamics exist for structural reasons decoupled from underlying asset values

(2) The cost of capital in various assets is determined by a coarse-grained bucketing into leaky abstractions like “listed equities”, and there aren’t many participants capable of arbitraging when the specifics of an asset defy their category bucket, due to structural forces of short time horizons and fund management principal-agent problems
# Probability that other architectures will scale as well as Transformers?

GPT-1, 2, and 3 have shown impressive scaling properties. How likely is it that, in the next five years, many other architectures will also be shown to get substantially better as they get bigger? EDIT I am open to discussion of better definitions of the scaling hypothesis. For example, maybe Gwern means something different [here](https://www.lesswrong.com/posts/N6vZEnCn6A95Xn39p/are-we-in-an-ai-overhang?commentId=jbD8siv7GMWxRro43) in which case I'm also interested in that.
# Does the lottery ticket hypothesis suggest the scaling hypothesis?

The [lottery ticket hypothesis](https://arxiv.org/abs/1803.03635), as I (vaguely) understand it, is that artificial neural networks tend to work in the following way: When the network is randomly initialized, there is a sub-network that is already decent at the task. Then, when training happens, that sub-network is reinforced and all other sub-networks are dampened so as to not interfere.

\[EDIT: This understanding goes beyond what the original paper proved, it draws from things proved (or allegedly proved) in later papers. See [thread below](http://Machine learning theory suggests that the amount of data required to find a model that is “close to optimal” is a linear function of parameter count and the variance of the loss function.). EDIT EDIT: Daniel Filan has now convinced me that my understanding of the LTH as expressed above was importantly wrong, or at least importantly goes-beyond-the-evidence.\]

By the scaling hypothesis I mean that in the next five years, many other architectures besides the transformer will also be shown to get substantially better as they get bigger. I'm also interested in defining it differently, as whatever Gwern is [talking about](https://www.lesswrong.com/posts/N6vZEnCn6A95Xn39p/are-we-in-an-ai-overhang?commentId=jbD8siv7GMWxRro43).
# To what extent are the scaling properties of Transformer networks exceptional?

Part of the point of GPT3 is that bigger continues to be better. ([Computerphile discussion](https://www.youtube.com/watch?v=_8yVOC4ciXc&t=392s).) [A recent question](https://www.lesswrong.com/posts/kpK6854ArgwySuv7D/probability-that-other-architectures-will-scale-as-well-as) asked whether this would turn out to be true for other architectures as well. But the question seemed to take for granted that we haven't seen this phenomenon in other cases yet. To what extent is this scaling phenomenon special to GPT? To what extent is it special to Transformer networks? To what extent is it special to unsupervised NLP?

My impression:

*   By 2011, the "bigger is better" trend was already well-established in deep learning. (See ["Big Data" on Google Trends](https://trends.google.com/trends/explore?date=all&geo=US&q=big%20data).) Major breakthroughs in what neural networks can do (in terms of performance on tasks such as image recognition) have generally been facilitated by bigger models, more data, and more training time, even in cases where there are also technical breakthroughs (such as convolutional neural networks). So, to an extent, there is nothing special about Transformers or GPT.
*   However, the data-hungry nature of deep learning has meant that *labelled datasets* are a major bottleneck to scaling. GPT, like other unsupervised learning methods, does not face this problem. In this sense, it does have a special scaling advantage.
*   Furthermore, for the particular task of NLP, we continue to see quantitative and qualitative improvements that we care about (at least intellectually) as we pour more money into this. In other words, NLP has a looooong and gradual learning curve (at least if you look at it a certain way). This means the task is difficult enough to see the benefits of throwing more at it, while easy enough to feel like you're getting something out of doing so.
# How will internet forums like LW be able to defend against GPT-style spam?

GPT-3 seems to be skilled enough to write forum comments that aren't easy to identify as spam. While OpenAI reduces the access to it's API it will likely don't take that long till other companies develop similar API's that are more freely available. While this isn't the traditional AI safety question, it does seem like it starts to become a significant safety question.