Harley-ml/lesswrong huggingface dataset:
# Auctioning Off the Top Slot in Your Reading List

Or, A Nicer Way to Commodify Attention  
 
------------------------------------------

**Observation 1:** I read/listen to a lot of public intellectuals--podcasters, authors, bloggers, and so on--and I frequently find myself thinking:

> Ugh, I hate it when this guy spouts ignorant nonsense about \[some domain\]. And he's been doing it for *years*. I wish he would just read \[relevant book\]. Then he’d at least be *aware* of  the strongest counterarguments.

**Observation 2:** Sometimes podcasters/bloggers will poll their followers with questions like, “Hey guys, who should I interview next?” or “what book should I review next?”

Having observed these things I wonder if they could be improved by monetary transactions.

For a specific example, it would be cool for David Deutsch to let the highest bidder choose a book for him to read and review. If we're lucky (or spendthrift), we get to see him finally give a considered response to the particular claims in *Human Compatible* or *Superintelligence* or *Life 3.0*.

Less specifically, there are a lot of cognoscenti who command substantial influence while holding themselves to disappointingly low epistemic standards. For example, Sean Carroll is a science communicator who dabbles in a little bit of everything on the side; and although I consider his epistemic standards to be above-average, I can tell he has not read the best of Slate Star Codex. I think if he did read a post such as "Asymmetrical Weapons", there’s a decent chance he would feel compelled to raise the bar for himself.

I have some close friends who sometimes spew (what I perceive to be) ignorant drivel. For some of them, I might be willing to pay a surprisingly high price to see them write a review of a book that cogently challenges their stupid priors. I would pay the highest price for friends that I already know can update on sound arguments, I would pay a lower price to *find out* if a friend has that ability, and for the attention of the hopelessly obstinate I would not want to pay anything.

Why This Wouldn’t Work
----------------------

*   It's easy to underestimate how pervasive and sticky those signaling/tribal motivations are. The gains from trade available here might not be enough to overcome the pressure to protect narratives and maintain appearances.
*   There might be perverse incentives. Maybe the cognoscenti want to charge more, so they reduce supply (that is, they read less). Maybe they spout more ignorant nonsense in order to increase the bids.

Those are just off the top of my head. If you can think of more/better reasons please share.
# What weird beliefs do you have? 

We know that mainstream thinking gets a lot of things wrong. Many of us have experienced being mocked because of our concern for AI extinction-risk. There are plenty of other examples of times where now well-evidenced beliefs were seen as crazy in some way. This post was prompted by my reading around meditation and mindfulness - twenty years ago if you said that meditation had a number of mental and even physical health benefits and was worth practicing for non-religious reasons, then you would be laughed at as a New Age type who probably believed in crystal healing and astrology too. Now there's stacks of scientific evidence supporting that view.

I would like to keep an open mind and not dismiss fact-claims just because they pattern-match to weird people or because they don't pass the absurdity heuristic. On the other hand, there are a lot of crazy people out there and I don't really want to wade through dumb stuff by flat-earth types. So I figured posting this question here is a good way to find some interesting ideas. Fellow Rationalists, what beliefs do you have that would cause the average member of society to laugh at you or call you weird? 

I have at least one such belief, but I'll post it as an answer to this question, because I want the focus to be on the question and not on my specific belief.

Edited to add: please include a summary of why you believe what you do - what evidence or chain of reasoning led you to this belief?
# Place-Based Programming - Part 1 - Places

When I write machine learning software, I tend to use the Place-Based Programming (PBP) paradigm. PBP caches your computations so you rarely have to perform the same computation twice.

The fundamental unit of data is a **place**, which refers to a location on disk. Consider the hard-coded string `"I am Satoshi Nakamoto."`. You can complete the place of a string by hashing it.

```
;; This code is written in Hy.
;; See https://docs.hylang.org/en/stable/ for documentation of the language.
(import [hashlib [md5]] os)
(setv +place-dir+ ".places/")

(defn place-of [expression]
  "Returns the place of an expression"
  (os.path.join
    +place-dir+
    "str/"
    (+ (.hexdigest (md5 (.encode (str expression))))
       ".pickle")))

;; prints ".places/<class 'hy.models.HyString'>/17f36dc3403a328572adcea3fd631f55.pickle"
(print (place-of '"I am Satoshi Nakamoto."))
```

In Lisp, the `'` tag means "do not evaluate the following expression". Note how we did not compute the place of the string's value directly. We evaluated the place of the source code which defines the string. We can replace our function with a macro so the user does not have to quote his or her code.

```
(import [hashlib [md5]] os)
(setv +place-dir+ ".places/")

(defmacro place-of [expression]
  "Returns the place of an expression"
  `(os.path.join
    +place-dir+
    (str (type '~data))
    (+ (.hexdigest (md5 (.encode (str '~expression))))
       ".pickle")))

;; prints ".places/<class 'hy.models.HyString'>/17f36dc3403a328572adcea3fd631f55.pickle"
(print (place-of "I am Satoshi Nakamoto."))
```

Whenever a function returns a place, it implicitly guarantees that the place is populated. The `place-of` macro is not allowed to just compute where a place would be if it existed. The macro must also save our data to the place if the place is not already populated.

```
(defmacro/g! place-of [expression]
  "Returns the place of an expression"
  `(do
     (setv ~g!place
           (os.path.join
             +place-dir+
             (str (type '~code))
             (+ (.hexdigest (md5 (.encode (str '~expression))))
                ".pickle")))
     (if-not (os.path.exists ~g!place)
             (with [f (open ~g!place "wb")]
               (pickle.dump (eval '~expression) f)))
     ~g!place))

;; prints ".places/<class 'hy.models.HyString'>/17f36dc3403a328572adcea3fd631f55.pickle"
(print (place-of "I am Satoshi Nakamoto."))
```

Reading from a place is easier.

```
(defn value-of [place]
  (with [f (open place "rb")]
    (pickle.load f)))

;; prints  "I am Satoshi Nakamoto."
(print (value-of (place-of "I am Satoshi Nakamoto.")))
```

This constitutes a persistent memoization system where code is evaluated no more than once.

```
(import [time [sleep]])
(print (value-of (place-of (do (sleep 5) "This computation takes 5 seconds"))))
```

The first time you call the above code it will take 5 seconds to execute. On all subsequent runs the code will return instantly.
# Training the YouTube Algorithm

Listening to music and watching dance lessons on YouTube makes me happy. Everything else on YouTube makes me unhappy. It doesn't matter what the video is, [watching YouTube](https://www.lesswrong.com/posts/koa48Qio5LP48xMM2/mediums-overpower-messages) makes me unhappy.

If all of YouTube made me unhappy then I should just not use YouTube. But I like listening to music and watching dance lessons on YouTube. I dislike getting recommendations for non-music non-dance videos. Fortunately, YouTube personalizes its recommendations. I wondered if I could train YouTube to only recommend music and dance lessons.

I created a brand new Google account just for YouTube. I carefully refrained from clicking on anything other than music and dance lessons. I quickly learned that "Don't recommend channel" works better than "Not interested". If you press "Don't recommend channel" then YouTube will stop recommending the channel.

I have trained my YouTube algorithm for about two years now. When I wanted to watch a non-music non-dancing video I did it incognito or on another account. Here are what my homepage recommendations look like right now.

![](https://s3-us-west-2.amazonaws.com/www.lsusr.com/youtube-home.png)

The first 12 recommendations are all music. The 13th recommendation is [a clip](https://www.youtube.com/watch?v=1dpm-mfY2Cw) related to Doctor Who. I think the non-music Doctor Who recommendation came from clicking on [The 10th Doctor Epic Suite](https://www.youtube.com/watch?v=BqOw2L-nhLY) which got recommended because I was watching music from *Sherlock*. From there, the next 72 recommendations are all music.

If I just keep scrolling and scrolling until I get non-music non-dance recommendations, they tend to be silly stuff like [this](https://www.youtube.com/watch?v=Z9TbbMEJb2c), [this](https://www.youtube.com/watch?v=HpPm7ULW7wU) or [this](https://www.youtube.com/watch?v=LI_oGrgfVLc). I think the *Lord of the Rings* recommendations come from listening to fantasy-themed music.

If you want to use YouTube for a specific purpose without distractions then it is entirely possible to train YouTube to do this. All you have to do is create an account, watch videos on a specific topic and press "Don't recommend channel" whenever you encounter an off-topic recommendation.
# The Scout Mindset - read-along

I just started reading Julia Galef's new book "[The Scout Mindset: Why Some People See Things Clearly and Others Don't](https://www.amazon.com/Scout-Mindset-People-Things-Clearly-ebook/dp/B07L2HQ26K)".  Here's a description:  
 

> When it comes to what we believe, humans see what they want to see. In other words, we have what Julia Galef calls a "soldier" mindset. From tribalism and wishful thinking, to rationalizing in our personal lives and everything in between, we are driven to defend the ideas we most want to believe—and shoot down those we don't.
> 
> But if we want to get things right more often, argues Galef, we should train ourselves to have a "scout" mindset. Unlike the soldier, a scout's goal isn't to defend one side over the other. It's to go out, survey the territory, and come back with as accurate a map as possible. Regardless of what they hope to be the case, above all, the scout wants to know what's actually true.
> 
> In *The Scout Mindset*, Galef shows that what makes scouts better at getting things right isn't that they're smarter or more knowledgeable than everyone else. It's a handful of emotional skills, habits, and ways of looking at the world—which anyone can learn. With fascinating examples ranging from how to survive being stranded in the middle of the ocean, to how Jeff Bezos avoids overconfidence, to how superforecasters outperform CIA operatives, to Reddit threads and modern partisan politics, Galef explores why our brains deceive us and what we can do to change the way we think.

  
This seems like a good book to do a read-along to, since there are probably a decent number of people reading it at the same time.   
  
If possible put your comments under the correct chapter parent comment as you go. If you get further ahead than me, feel free to create a new chapter parent comment.
# Could degoogling be a practice run for something more important?

Andrew Critch's [recent threat model ends](https://www.alignmentforum.org/posts/LpM3EAakwYdS6aRKf/what-multipolar-failure-looks-like-and-robust-agent-agnostic) with the following: 
> We humans eventually realize with collective certainty that the companies have been trading and optimizing according to objectives misaligned with preserving our long-term well-being and existence, but by then their facilities are so pervasive, well-defended, and intertwined with our basic needs that we are unable to stop them from operating.  With no further need for the companies to appease humans in pursuing their production objectives, less and less of their activities end up benefiting humanity.  
> Eventually, resources critical to human survival but non-critical to machines (e.g., arable land, drinking water, atmospheric oxygen…) gradually become depleted or destroyed, until humans can no longer survive.

I occasionally see posts by people who believe that surveillance advertising is bad, and we should try to write google out of our lives. Regardless of the merit of this argument, I admire the discipline it takes to degoogle. Gmail and gdocs are really high quality, add a ton of value, which is why they've become so entrenched. I can scarcely imagine _actually_ doing without google docs, at this point! 

It occurs to me: should we be practicing the skill of doing without something that's pervasively adding a lot of value, just in case that skill helps to keep us from being enfeebled by an aligned AI system, or destroyed by a misaligned AI system? Is it also practice coordinating, which would payoff more generally than AI problems?
# Alex Flint on "A software engineer's perspective on logical induction"

This Sunday at noon PT, [Alex Flint](https://www.lesswrong.com/users/alexflint) will be giving a short talk on how he thinks about [Logical Induction](https://www.lesswrong.com/tag/logical-induction) from a software engineer's perspective. He describes the talk as "My attempt to understand and re-explain logical induction as a practical alternative to probability."

We'll be meeting in the Walled Garden at noon, on Sunday April 18th.

[http://garden.lesswrong.com?code=qCM0&event=alex-flint-on-logical-induction-from-a-software-engineer-s](http://garden.lesswrong.com?code=qCM0&event=alex-flint-on-logical-induction-from-a-software-engineer-s)
# Anger

I don't get angry. I haven't gotten angry in ten years. A Buddhist would say I have "pulled up anger by the roots".

Anger is predicated on the desire to hurt someone. The desire to hurt someone as a terminal value is pure evil.

I have been on the receiving end of a lot of anger. Ostensibly, it is because of things I did or I didn't do. None of that matters. How often a particular person gets angry at me has little to do with me and everything to do with that person.

Though I no longer get angry, I do feel stress. My baseline stress is a function of physical exercise, social interaction, food, meditation and art. But that's not goes through my head. When I feel stress, I go looking for something I'm doing wrong like an appointment I've forgotten about, a chore I've been putting off or some other personal inadequacy. I misattribute the stress to a [confabulation](https://www.lesswrong.com/posts/ZvLPnJrgHxhkEyXKu/confabulation) and then go about solving the wrong problem.

I think angry people do the same thing. They feel anger before identifying a target. This would explain why observation that whether someone gets angry at me in a particular interaction correlates with how frequently that person has gotten angry at me in the past and cannot be predicted by my proximate influence.

With particularly angry people, I can predict when they will get angry *before they do* but I cannot predict what they will get angry at. Except that, if I am alone with the person, the target will eventually become me *no matter what I do*.

> 57. There are two red flags to avoid almost all dangerous people: 1. The perpetually aggrieved ; 2. The angry. 
>
> ―[*100 Tips for a Better Life*](https://www.lesswrong.com/posts/7hFeMWC6Y5eaSixbD/100-tips-for-a-better-life) by Ideopunk

When I was a kid I had an adult berate me for not shoveling dirt into a hole and then, an hour later, berate me for shoveling dirt into the hole. Neither beratement had anything to do with the task. He just wanted to make me suffer. But he didn't understand his own motivation. He confabulated that I was a lazy kid who should have known better than than to do (or not do) whatever I was doing (or not doing).

The most undeserving victims of anger are the victims of genocides and such. But in my personal experience, the people for whom anger causes the most suffering are angry people themselves. Anger is incompatible with happiness. If you are frequently angry then you are frequently unhappy.

Pulling up anger by the roots makes you a gentler, happier person. The catch is that anger is a natural defense against other angry people. To safely remove a natural defense you must replace it with an artificial one.

If you are not an angry person then the best way to deal with angry people is to avoid them. If you cannot avoid angry people then distance yourself from them. If you bump into angry people then diffuse the situation. If you cannot diffuse the situation then fight as a last resort.
# Are there good classes (or just articles) on blog writing?

I've been writing [a blog](https://www.brendanlong.com/) for a long time, and I think the articles that actually get posted tend to be decent. Unfortunately, I have a lot of articles that never make it past the draft (or even outlining) stages. It seems like once I get past a certain level of complexity I just get lost. I have trouble deciding how to structure posts (or whether I should split them into a series), and I'm terrible at writing introductions and conclusions. I'm also not particularly good at headlines, although I consider this a less-important problem.

Are there good articles / books / classes on how to do this, ideally with well-chosen practice ("Outline articles on these topics", "Write an article on this topic", "Write an introduction for this post?", etc)? I would be willing to pay money for this if it's good enough, especially if I had access to someone who could review my work and give suggestions.

I assume one method of doing this is "practice until you get good at it". I've occasionally just forced myself to write something and then post it, but I almost always end up deleting these posts because they're not up to my standards, and I feel like they don't count as "deliberate practice" because I can tell that something is wrong with them but I don't have the experience to know what it is.

I've been looking through SkillShare and other things I've found via searches, but they also seem to skip over the "actually writing an article" step and focus on things I don't need help with, like:
- I don't need to setup my blog (I already have one)
- I don't need help with SEO
- I don't care about monetization right now
- I already have ideas for articles
# Parameter count of ML systems through time?

[Pablo Villalobos](https://www.lesswrong.com/users/pvs) and I have been working to compile a rough dataset of parameter counts for some notable ML systems through history.

This is hardly the most important metric about the systems (other interesting metrics we would like to understand better are training and inference compute , and dataset size), but it is nonetheless [an important one](https://arxiv.org/abs/2001.08361) and particularly easy to estimate.

So far we have compiled what it is (to our knowledge) [the biggest dataset so far of parameter counts](https://docs.google.com/spreadsheets/d/1AAIebjNsnJj_uKALHbXNfn3_YsT6sHXtCU0q7OIPuc4/edit#gid=0), with over a 100 entries.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8db2058e595c26a84748b0b256cb0862a15dbfcc04c1ed54.png)

  
But we could use some help to advance the project:

1.  Is there any previous relevant work?  We are aware of the [AI and compute](https://openai.com/blog/ai-and-compute/) post by OpenAI, and there are some papers with some small [tables of parameter counts](https://arxiv.org/pdf/1910.01108.pdf).
2.  If you want to contribute with an entry, please do! The key information for an entry is a reference (citation and link), domain (language, vision, games, etc), main task the system was designed to solve, parameter count (explained with references so its easy to double check), and date of publication. The criteria for inclusion is not very well defined at this stage in the process; we have been focusing on notable papers (>1000 citations), significant SOTA improvements (>10% improvement on a metric over previous system) and historical relevance (subjective). We mostly have ML/DL/RL papers, and some statistical learning papers. To submit an entry either leave an answer here, send me a PM, email `jaimesevillamolina at gmail dot com` or leave a comment in the spreadsheet.
3.  If you'd be interested in joining the project, shoot me an email. The main commitment is to spend 1h per week curating dataset entries. Our current goal is compiling parameter counts of one system per year between 2000 and 2020 and per main domain. If you can compute the number of parameters of a CNN from its architecture you are qualified. I expect participating will be most useful to people who would enjoy having an excuse to skim through old AI papers.

Thank you to Girish Sastry and Max Daniel for help and discussion so far!
# Wanted: Research Assistant for The Roots of Progress

***UPDATE: I’ve filled this role. Thanks to all who applied!***

I’m hiring a part-time research assistant to support work on my [essays](https://rootsofprogress.org/posts), [talks](https://rootsofprogress.org/interviews), and the [book I’m writing](https://rootsofprogress.org/discussion-series-the-story-of-industrial-civilization) on the history of industrial civilization.

You must have the ability to orient yourself in unfamiliar mental territory; to penetrate the fog of confusing, incomplete, and contradictory information; to sniff out reliable sources of key facts and to corroborate them; and to quickly sketch out a new intellectual landscape.

You will handle queries such as:

*   What happened to the price of cotton and the wages of textile laborers before, during, and after textile mechanization in the 18th/19th centuries? Find data and analysis on this, including relevant statistics on labor productivity, and produce a list of sources.
*   What startups or other commercial projects are pursuing advanced nuclear reactor designs? Make a list, and fill out details of each in a spreadsheet, such as type of reactor, amount and sources of funding, etc.
*   Find first-person accounts of agricultural life and work before the 19th century, including descriptions of regular planting and harvesting seasons, and also times of crop failure or even famine.
*   What is the difference between a bloomery, a blast furnace, and a Catalan forge? Make a list of sources that address this question.

The deliverable will typically be a list of sources, with brief notes on what each one contains, ranked roughly in order of relevance to the original query. You don’t have to answer the questions I pose, but you need to find sources that help me answer them.

The only real requirements are writing skills and attention to detail. However, the ideal candidate would be:

*   A graduate student in history, economics, or a related field (ideally with access to scholarly sources)
*   Familiar with and interested the progress community in general, and my work in particular
*   Able to put in part-time work with fairly quick turnaround (24 hours for small queries would be excellent)

If you lack experience and credentials, apply anyway: you can make up for it by being dedicated, diligent, and willing/able to be trained.

The work will be variable, up to roughly 10–15 hours/week. We’ll mostly communicate by email/messaging, so you can be in any time zone. Pay: $25–30/hour, depending on qualifications.

To apply, send a CV/resume and writing sample to me at [jason@rootsofprogress.org](mailto:jason@rootsofprogress.org).
# Quick examination of miles per micromort for US drivers, with adjustments for safety-increasing behavior

This post links [to a Google Sheet containing a quick investigation into the accuracy of Wikipedia's figure for miles per micromort (230) for US drivers](https://docs.google.com/spreadsheets/d/1F0CmumTbASFwki665tuN5YFriXeqi23WjY5RqjYbczA/edit?usp=sharing), when accounting for preventative behaviors.

The following are the main outcome estimates:

| Miles per micromort, no adjustments, in US (2019) | 91  |
| --- | --- |
| \\-\\- If excluding motorcycles | 105 |
| \\-\\- If excluding motorcycles and pedestrians, pedalcyclists, and other nonoccupants | 137 |
| \\-\\- Amongst passenger vehicle occupants only | 132 |
| \\-\\- Amongst passenger vehicle occupants only, if setting single-car crashes to 0 | 235 |
| \\-\\- Amongst passenger vehicle occupants only, if approximating the seatbelt-wearing only rate | 245 |
| \\-\\- Amongst passenger vehicle occupants only, if setting single-car crashes to 0 and approximating the seatbelt-wearing only rate | 442 |
| \\-\\- Amongst passenger vehicle occupants only, if setting single-car crashes to 0 and approximating the seatbelt-wearing only rate and if setting alcohol-impaired, drowsiness-associated, and distraction-associated deaths to 50% of current level (as an approximation of controlling one driver's behavior in two driver crashes) | 548 |

This rapid (~1.5 hrs including documentation) investigation was [funded by Ruby Bloom via the Bountied Rationality](https://www.facebook.com/groups/bountiedrationality/permalink/2921507771426600/) FB group.
# Does an app/group for personal forecasting exist?

I'm interested in personal forecasting - predicting my own future behavior on a range of timescales. I see it as a more useful skill than forecasting on world events. Formulating personally useful forecasts seems like an important and neglected skill in the rationalist community. And it would be nice to have some company, and tools to make it more convenient. Right now, I'm just using a spreadsheet. Does anybody know if there are groups doing this sort of thing? Is there a good app to manage the process?
# Why “put down your phone and go outside” is Bad Advice. 

If you’re like me in 2017, your phone is your drug of choice. My insurance didn’t provide me with any internet abuse rehab options. (Though at least one place exists.) I was sick of being a NEET and spending all my time online, so I gave my phone and laptop to a friend because a lot of people on Reddit had talked about this “cold turkey” approach. Cut to me asking for my devices back a day later. My friends were all too nice to actually hold me accountable. So I descended back into the murky ocean of online lurking, my life experience reduced to an endless scroll, searching for novel information and parasocial fantasies. 

The cold turkey approach and similar “digital detox” methods are very popular in the “no surf” community. I see the appeal, it’s tempting to blame the internet on all of your problems and assume you’re better off without it. For hardcore internet users, which I believe is becoming the norm for young and old alike, it’s just probably not realistic or even all that helpful. 

Cut to 2021- I’m in school and employed. I’m still an “internet person” but my phone screen time is around 2.5 hours per day when I used to hover around 15. I use my laptop for school and only fall into endless scrolling hell when I’m particularly depressed. 

“Put down your phone and go outside.” didn’t work for me. I went to therapy. At some point I gained the ability to actually experience life directly, to be present in the current moment. I realized that the reason I had no life is because I was constantly escaping it. I had to start feeling my feelings. Running away wasn’t working.

That's the tldr; of how I escaped NEETdom. The recovery process is a bit more complex than I've alluded to, maybe I'll make a part 2 in the future.
# Why don't we vaccinate people against smallpox any more?

The eradication of smallpox in 1979 represented one of the greatest achievements of modern civilization. However, since then most countries have elected to stop vaccinating their populations against the disease. This seems like a very concerning vulnerability to me, with waning herd immunity due to more and more of the world's population being replaced by unvaccinated young people. 

What if there was an unintentional release from one of the labs around the world that still hold on to samples of the virus? What about an intentional release by terrorists/rogue nations? What gives scientists the confidence that there are no undiscovered animal reservoirs or uncontacted tribes in remote places where smallpox is still circulating? Smallpox is highly contagious and hundreds of times more deadly the SARS-Cov-2.

How would the world respond to such a release? Is there enough capacity to rapidly produce and deploy billions of doses of smallpox vaccines? (Right now we're at an all-time high in terms of pandemic preparedness; I'm thinking decades down the road when the lessons from Covid-19 has been all but forgotten)
# Faerie Rings

This post describes "Faerie rings", a copiable form of community that forms a building block for capturing the upsides:  learning from each other, deeper relationships, and support to solve problems that are beyond one's abilities. I believe it counts as a nascent prototype for an [Iterated Trust Kickstarter](https://www.lesswrong.com/posts/5AoLQsRL8eShLApN9/iterated-trust-kickstarters-1).

I copied most of this description outright from @mixmix on the scuttlebutt platform. Citation details:

> @ye+QM09iPcDJD6YvQYjoQc7sLF/IFhmNbEqgdzQo3lQ=.ed25519  
> %YrKbs+z2CmLt2+VtaEvUCDY+DUQAqQgB7vuUh3kNsdk=.sha256

The form
--------

*   A group of up to 8 people
*   A call once every 2 weeks, for 2-3 months. The schelling days for this are Full Moon and New Moon.

Everyone has a turn checking in

*   how are you
*   how is it going

There's a host (who changes)

*   They open & close the circle
*   They make sure everyone gets to speak

The rest of the format grows to fit the group. It might include things like

*   a focus question
*   asks / offers
    *   do you have something you’d like to get help with / to offer ?
*   freestyle

(I have some ideas of my own, but this post is for getting the main concept across.)

Every 4th meeting, one person from a faerie ring temporarily swaps with a person from another ring. This encourages networks to graft together in [microsolidarity](https://www.microsolidarity.cc/) and cross-pollinate good format ideas.

Check out [this doc](https://docs.google.com/document/d/1h3cQLWkFl8kJduGtRXurzCMaT5VTCmF2-lsDofDizA8/edit#) for coordinating a ring.
# Have an idea for an interactive learning module? Consider applying for a ClearerThinking.org micro grant!

If you have an idea for an interactive learning module (on topics such as rationality, psychology, self-improvement, statistics, probability, effective altruism, philosophy economics, math, science, behavior change, etc.) that you think would be valuable to a wide audience, we hope you'll consider applying to our second annual ClearerThinking.org micro grants program! Our program last year was a huge success, resulting in [15 new interactive modules](https://www.clearerthinking.org/post/we-re-launching-15-new-tools-made-by-clearer-thinking-users-as-part-of-our-micro-grants-program), so we decided to do it again.

Applying to stage 1 is very fast and simple. Using the step-by-step process we provide, and feedback from our team plus a study we run on your work, winners will end up producing full-blown interactive learning modules like those on [ClearerThinking.org](https://www.clearerthinking.org/). The top programs will also be featured on our website and sent out to our 130,000 email list subscribers!

Applications are due by May 17, 2021. All of the details of the program [can be found here.](https://docs.google.com/document/d/1TqDG30MuBD6pkQ4_ahUZZGfNg9A3AGY_3Hju5lEV-9Y/edit)

Some benefits you may get from applying to the program:

*   Helping other people improve their lives by creating a fun, educational tool that can be used by anyone, anywhere in the world (who speaks English and has internet access).
*   Gaining experience developing an idea into a complete, high-quality, interactive learning module with help from experienced professionals.
*   Receiving a monetary prize for each stage of the program you advance to (as long as you’re successful in the first stage).
*   The possibility of having your work published on [ClearerThinking.org](http://clearerthinking.org/), and shared with our mailing list of 130,000+ people.
*   An easy application process - all you have to do for stage 1 is describe your idea for a beneficial learning module in around 150 to 400 words.

[Go here](https://docs.google.com/document/d/1TqDG30MuBD6pkQ4_ahUZZGfNg9A3AGY_3Hju5lEV-9Y/edit) to learn more and apply.
# Estimating COVID cases & deaths in India over the coming months

Does anyone have a good method to estimate the number of COVID cases India is likely to experience in the next couple of months? I realize this is a hard problem but any method I can use to put bounds on how good or how bad it could be would be helpful.

I'm trying to convince some people who work in global health to send the US stockpile of unused and unapproved Astra Zeneca vaccines to the hardest hit parts of India. My naive calculation indicates sending 25,000,000 doses (basically our whole stockpile) would save at least 17,000 lives using fairly conservative assumptions. But that was based on an estimate that 20% of the total population would be infected and a third of those would be infected after the vaccines we send are injected and become effective.

The 20% is based on the rate of US infections and the 1/3rd is basically a random guess about how many infections there would be left between vaccination and the end of the pandemic.
# Best empirical evidence on better than SP500 investment returns?

I am not smart enough to be a real rationalist, and I am really lazy. But I am smart enough to understand and implement rationalist advice and I suspect y'all are a better source of advice than normal investors. What is the highest level of returns I can achieve with a buy and hold strategy of some kind, across any asset class, and what is the empirical evidence for this?  Also, I am self-employed, in a field where the 99th percentile earners make 7 figures per year and I make only low six figures. So how should I assess investment in an asset vs trying to invest in my career? I am not quite risk-neutral but pretty close to it. My expenses are well below my income; I'm done saving for retirement; further investing is just a game at this point. But I don't know how to play it.
# FAQ: Advice for AI Alignment Researchers

To quote Andrew Critch:

> I get a lot of emails from folks with strong math backgrounds (mostly, PhD students in math at top schools) who are looking to transition to working on AI alignment / AI x-risk. There are now too many people “considering” transitioning into this field, and not enough people actually working in it, for me, or most of my colleagues at Stuart Russell’s [Center for Human Compatible AI (CHAI)](http://humancompatible.ai/), to offer personalized mentorship to everyone who contacts us with these qualifications.
> 
> [*From math grad school to AI alignment*](http://acritch.com/from-math-grad-school-to-ai-alignment/)*, Andrew Critch*

I’m pretty sure he wrote that at least 4 years ago (2016 or earlier). The field has grown enormously since then, but so has the number of people considering it as a research area. So far, I’ve tried to give at least 10 minutes of my time to anyone who emails me with questions; that probably won’t be sustainable for much longer. So now I’m answering the questions I get most frequently.  I hope to keep this up to date, but no promises.

Usually, I write a blog post when I think I have something important and novel to say, that I am relatively confident in. That’s not the case for this post. This time, I’m taking all the questions that I frequently get and writing down what I’d say in response. Often, this is (a) not that different from what other people would say, and (b) not something I’m very confident in. Take this with more grains of salt than usual.

Thanks to Neel Nanda, Nandi Schoots, and others who wish to remain anonymous for contributing summaries of conversations.

*See the linked post for the FAQ; which will hopefully be kept up to date over time.*
# [Linkpost] Treacherous turns in the wild

Mesa-optimizers potentially escape behavioral adversarial training by realizing they're being tested, gaming the test, then defecting when they're deployed. Luke Muelhauser recounts an example of such a treacherous turn in the wild. 

Most relevant quote:
> As part of a project studying the evolution of (simulated) organisms, computer scientist Charles Ofria wanted to limit the replication rate of a digital organism. So, he programmed the system to pause after each mutation, measure the mutant’s replication rate in an isolated test environment, and delete the mutant if it replicated faster than its parent. However, the organisms evolved to recognize when they were in the test environment and “play dead” (pause replication) so they would not be eliminated and instead be kept in the population where they could continue to replicate outside the test environment. Once he discovered this, Ofria randomized the inputs of the test environment so that it couldn’t be so easily detected, but the organisms evolved a new strategy, to probabilistically perform tasks that would accelerate their replication, thus slipping through the test environment some percentage of the time and continuing to accelerate their replication thereafter.

The full story is given in the original post.
# Announcing the Alignment Research Center

(Cross-post from [ai-alignment.com](https://ai-alignment.com/announcing-the-alignment-research-center-a9b07f77431b))

I’m now working full-time on the [Alignment Research Center](http://alignmentresearchcenter.org) (ARC), a new non-profit focused on intent alignment research.

I left OpenAI at the end of January and I’ve spent the last few months planning, doing some theoretical research, doing some logistical set-up, and taking time off.

For now it’s just me, focusing on theoretical research. I’m currently feeling pretty optimistic about this work: I think there’s a good chance that it will yield big alignment improvements within the next few years, and a good chance that those improvements will be integrated into practice at leading ML labs.

My current goal is to build a small team working productively on theory. I’m not yet sure how we’ll approach hiring, but if you’re potentially interested in joining you can fill out [this tiny form](https://forms.gle/RnpZwU9Ld4qBGPja9) to get notified when we’re ready.

Over the medium term (and maybe starting quite soon) I also expect to implement and study techniques that emerge from theoretical work, to help ML labs adopt alignment techniques, and to work on alignment forecasting and strategy.
# What topics are on Dath Ilan's civics exam?

[Dath Ilan](https://www.lesswrong.com/posts/gvA4j8pGYG4xtaTkw/i-m-from-a-parallel-earth-with-much-higher-coordination-ama) is a parallel Earth on which human civilization has its act together, in ways that actual-Earth does not. Like actual-Earth, citizens of Dath Ilan sometimes take standardized tests, both to figure out what sort of jobs they'd be suited for, to make sure that its educational institutions are functioning, and to give people guidance about what they might want to study. Unlike Earth's, Dath Ilan's tests have had a lot of thought put into the choice of topics: rather a lot more economics, rather a lot less trigonometry and literature. Topics are selected based on cost/benefit; something that takes a long time to learn would need to be a lot more useful, or have major positive externalities to more people knowing it.

I want to create a test, that will tell people what topics they ought to learn, and enable people to make their knowledgeability legible.

What topics belong on it?
# Jaan Tallinn's 2020 Philanthropy Overview

to follow up my [philantropic pledge](/posts/ZMRCpfGD4iXs5Mi5Q/jaan-tallinn-s-philanthropic-pledge) from last year, i've updated my [philanthropy page](https://jaan.online/philanthropy/) with 2020 results.

TL;DR: i made $4.3MM worth of endpoint grants in 2020, mostly via SFF's grant rounds, beating my minimum commitment by about 2x.
# Covid-19 in India: Why didn't it happen earlier?

When I look at the numbers of Covid-19 cases and Covid-19 deaths among G20 countries, India still has a relatively low number per capita. The arguments for why the current wave is particularly catastrophical in India (like population density or poverty) applied all the time, however, except for the mutation. How did India manage to have relatively low case rates up to now?
# Rationality Cardinality

[Rationality Cardinality](https://www.rationalitycardinality.com) is a humorous card game, built out of rationality concepts and vocabulary and designed to teach them. I have recently finished an online-game implementation, which can be embedded into gather.town worlds. Try it out by joining the [Rationality Cardinality gather.town server](https://gather.town/app/uMF8dIMms1EUhMDl/Rationality%20Cardinality)! It helps to have other people to play with, so I suggest  Saturday May 1, 11am PST as a Schelling meetup time.

Rationality Cardinality is a two-for one combined bet on dick jokes and the Sapir-Whorf hypothesis. Sapir-Whorf is the hypothesis that many people are bottlenecked on what thoughts they can think, on having vocabulary for the relevant concepts. I've been curating a set of cards (concepts, explanations, and jokes) over the years, and I'm quite proud of it. (Except for the cards about concepts that fell to the replication crisis, of course.)
# Faerie Ring meet #2: At Gather.Town

This is the second meeting of a Faerie Ring, an instance of a copyable ritual for community. [Here's slightly more explanation](https://www.lesswrong.com/posts/eNDB2F8CdktderNyR/faerie-rings) of the format.

*   A group of up to 8 people
*   A call once every 2 weeks, for 2-3 months.
*   Checking in with each other and seeing how we can help one another

My ring still has room for more peeps, and I'm looking to fill predominantly rationalists so I'm advertising here.

We will meet on [Gather.town](https://gather.town/app/ggpbEQp6icnzxkgN/My%20Home%20Space).

Please RSVP with your email address in a DM, or on [this google doc](https://docs.google.com/document/d/1h3cQLWkFl8kJduGtRXurzCMaT5VTCmF2-lsDofDizA8/edit#)
# Better air is the easiest way not to die

> It’s hard to prioritize health advice. I’m told I should limit salt and eat cruciferous vegetables and do cardio and sleep well and limit alcohol and reduce stress and go for regular checkups. But how much do each of these matter? If you’re a fallible hairless ape, what should you do first?

> You should put the stuff with the highest return on effort first, and that’s air.
# Is sitting in the sun much better than sitting in the shade? 

How much worse is sitting outside in the shade compared to the sun, for the purpose of experiencing enough lux to feel alert?

Making the difference between shade and sun a bit more extreme than it probably is in practice, we can take "sun" to mean "10,000 lux" and "shade" to mean "1,000 lux". Data to back this up: 

> [Wikipedia](https://en.wikipedia.org/wiki/Daylight) gives these lux values for different conditions:
> 
> *   20,000 lux "Shade illuminated by entire clear blue sky, midday"
> *   1,000–2,000 lux "Typical overcast day, midday"
> *   <200 lux "Extreme of thickest storm clouds, midday"
> 
> [NOAO](https://www.noao.edu/education/QLTkit/ACTIVITY_Documents/Safety/LightLevels_outdoor+indoor.pdf) (National Optical Astronomy Observatory) says:
> 
> *   10,000 lux "Full Daylight"
> *   1,000 lux "Overcast Day"
> *   100 lux "Very Dark Day"
> 
> \[Side question: What does Wikipedia exactly mean with "Shade illuminated by entire clear blue sky"? Is this a strange way of describing sunlight, or are they referring to the lighting experienced standing in the shade of an object during an otherwise sunny day? The words sound like the latter to me, but the former fits better with NOAO's description. I don't think this distinction breaks my question either way.\]
> 
> Same orders of magnitude, just seems like NOAO is averaging throughout the day and Wikipedia is looking at midday. 

Meanwhile, [this LW post](https://www.lesswrong.com/posts/7izSBpNJSEXSAbaFh/why-indoor-lighting-is-hard-to-get-right-and-how-to-fix-it) discusses how to install indoor lighting to get about 1,000 lux inside. If 1,000 lux is enough to keep a human alert and non-SAD, then it seems like shade is no worse than sun. Or, maybe people installing lumenators in their homes just stop at 1,000 lux because it's impractical to get an order of magnitude more lux than that. 

One way of phrasing the argument in favor of lumenators is that going from 100 lux to 1,000 lux provides huge benefits. Can the same thing be said about going from 1,000 lux to 10,000 lux, or are there diminishing returns? In other words, should I sit in the sun instead of the shade?
# AMA: Paul Christiano, alignment researcher

I'll be running an Ask Me Anything on this post from Friday (April 30) to Saturday (May 1).

If you want to ask something just post a top-level comment; I'll spend at least a day answering questions.

You can find some background about me [here](http://paulfchristiano.com).
# Draft report on existential risk from power-seeking AI

I’ve written a draft report evaluating a version of the overall case for existential risk from misaligned AI, and taking an initial stab at quantifying the risk from this version of the threat. I’ve made the draft viewable as a public google doc [here](https://docs.google.com/document/d/1smaI1lagHHcrhoi6ohdq3TYIZv0eNWWZMPEy8C8byYg/edit#) (Edit: arXiv version [here](https://arxiv.org/abs/2206.13353), video presentation [here](https://harvard.zoom.us/rec/play/kZc6rR2Ynx6z_J4rmmJUaq12-AcVDZLlY9eniU5ZKioodJm_yJORyy9UiLI8zFRuQvfLDAYO3j5TPciD.WSBvvCvXIto0mF4w?continueMode=true&_x_zm_rtaid=4hUT6sI8R4GPtHZS-tZBXg.1651957013327.8c9781863a967470f7270aa15ef4c1f4&_x_zm_rhtaid=917), human-narrated audio version [here](https://joecarlsmithaudio.buzzsprout.com/2034731/12113681-is-power-seeking-ai-an-existential-risk)). Feedback would be welcome. 

This work is part of Open Philanthropy’s “[Worldview Investigations](https://www.openphilanthropy.org/blog/our-progress-2019-and-plans-2020#Worldview_investigations)” project. However, the draft reflects my personal (rough, unstable) views, not the “institutional views” of Open Philanthropy.
# 25 Min Talk on MetaEthical.AI with Questions from Stuart Armstrong

*Previous Posts*:   
[Formal Metaethics and Metasemantics for AI Alignment](https://www.lesswrong.com/posts/85vp2kgFZoycFqr5G/formal-metaethics-and-metasemantics-for-ai-alignment-1)  
[New MetaEthical.AI Summary and Q&A at UC Berkeley](https://www.lesswrong.com/posts/YAd4NieT4iq6uY8AS/new-metaethical-ai-summary-and-q-and-a-at-uc-berkeley) 

This time I tried to focus less on the technical details and more on providing the intuition behind the principles guiding the project. I'm grateful for questions and comments from Stuart Armstrong and the [AI Safety Reading Group](https://aisafety.com/reading-group/). I've posted the [slides on Twitter](https://twitter.com/metaethical_ai/status/1383513411313569799).

*Abstract*: We construct a fully technical ethical goal function for AI by directly tackling the philosophical problems of metaethics and mental content. To simplify our reduction of these philosophical challenges into "merely" engineering ones, we suppose that unlimited computation and a complete low-level causal model of the world and the adult human brains in it are available.

Given such a model, the AI attributes beliefs and values to a brain in two stages. First, it identifies the syntax of a brain's mental content by selecting a decision algorithm which is i) isomorphic to the brain's causal processes and ii) best compresses its behavior while iii) maximizing charity. The semantics of that content then consists first in sense data that primitively refer to their own occurrence and then in logical and causal structural combinations of such content.

The resulting decision algorithm can capture how we decide what to do, but it can also identify the ethical factors that we seek to determine when we decide what to value or even how to decide. Unfolding the implications of those factors, we arrive at what we should do. All together, this allows us to imbue the AI with the necessary concepts to determine and do what we should program it to do.