Harley-ml/lesswrong huggingface dataset:
# How to get past Haidt's elephant and listen

First, some examples. 

In my beginning programming classes, I tell students (and show them): write comments first, before coding. Some won't. I tell them: write the algorithm first. I prove this will save them time. There's a whole chapter on why and how to do this. Many won't. (Except twice, they really got it. IDK why.) As time goes by -- I hope! -- they learn that this is the way to go. How can I help them hear at the start?  
  
I was with my 6-ish nephew, and he wanted to show me a computer game. Computer wouldn't start. He called in his mom for help. She said, Sure, I'll try, but I think you should know your uncle knows everything I do about computers and more. I could see on his face: he was waiting for her to stop talking nonsense and fix the damn thing. Why couldn't he hear her?  
  
I was at an informal lecture; the presenter used the story of Cain and Abel to comment on a supposed conflict between ranchers and farmers in ancient times. I went to speak to him. He said, "I'm sorry if you were offended." I said, "No, it was great! I really liked \[blah blah\]." He said, "OK. I'm sorry if you were offended." I said, "I wasn't offended at all. \[More details\]." He said, "OK. I have to go. I'm sorry you were offended." Why couldn't he hear?

And then there's the kid with the terrible twos. "It's time to eat. We have pizza, your favorite!"  Response: "NO!" (Didn't he hear you say "pizza"?)  
  
Haidt ([*The Righteous Mind*](https://www.goodreads.com/book/show/11324722-the-righteous-mind)) and probably others think it's the elephant, the part of the brain that's emotion and intuition and gut, that's charging on, heedless of the babblings of the rider, the intellectual part we think it in charge but actually functions to justify what the elephant already decided to do. OK. But it sure would be helpful if we could get to that elephant before it's made up its stubborn mind. Haidt suggests: convince the elephant, not the rider (however that would work). I'll add: if it's me, I can decide not to care so much, so the elephant will be more cooperative.

In the case of the beginning programmers, I already use self-interest (think of the time you'll save!) and availability bias (every program they see starts with commenting). It worked better this semester. All I can think I did different was show them the grading scheme for a program, that 60% of it was commenting and algorithm. :) Maybe that was it; IDK. 

This being LessWrong, of course we should consider how *we* have trouble hearing.  This may be a challenge. If I didn't hear it, I'll have a hard time reporting that here. :)

What examples do you have? Why can't you/the other person hear? What can be done?
# Furry Rationalists & Effective Anthropomorphism both exist

(cross-posted on [EA Forum](https://forum.effectivealtruism.org/posts/Sj9RMx3gn8oxi3ycX/furry-rationalists-and-effective-anthropomorphism-both-exist) and [my website](https://agentydragon.com/posts/2022-10-24-furry-rationalists.html))

Hi!

I’m Rai and I’m a furry (specifically, dragon). The last couple years, I’ve been running a *Furry Rationalists* Telegram group. It looks like we exist, and not everyone who should know we exist does yet, so I wanted to just write this to advertise that this furry+EA/rationality corner exists, and if you’re furry-adjacent & rationality-adjacent and nice, you’re invited to join us :)

Here’s the invite link for the Furry Rationalists group: [https://adgn.link/furry-rationalists-telegram](https://adgn.link/furry-rationalists-telegram)

There’s ~50 of us and we’re chill - we have self-improvement, science, and cute animal GIFs. If you’d like a preview, here’s the [guidelines + meta doc](https://docs.google.com/document/d/1xqNgRYzdVFO736oOIHK2oTIN3F8gty-lyYMYFyz1dLQ/edit#heading=h.tzj16eykkjpy). We’re 18+, but we’re not adult-oriented - we’re 18+ just so that we can talk about adult stuff if it does come up. If you happen to be <18 and wanna join, let me know, we might update this.

If you’re reading this a while later, and the link expired, contact me (via some method on [my website, agentydragon.com](http://agentydragon.com)), or look us up on [https://www.furry-telegram-groups.net/](https://www.furry-telegram-groups.net/), a search for “rationality” should find us.

There’s also a smaller Effective Anthropomorphism Discord server, run by [bird](https://twitter.com/birdpathy): [https://adgn.link/effective-anthropomorphism-discord](https://adgn.link/effective-anthropomorphism-discord)

Come say hi, and feel free to share if you know anyone who’d be interested!
# What will the scaled up GATO look like? (Updated with questions)

Demis Hassabis mentioned a few months ago that Deepmind is in the middle of scaling up its generalist agent GATO. Based on this, I would except it to come out either by the end of the year or in the beginning of the next year. The original model had just 1 billion parameters and was able to play atari games, operate a robotic arm and be used as a (relativel small) language model, so scaling it up to several hundred billion(trillions?) parameters seems to have a great potential in capability progress.

There were similar discussions about what would the most impressive thing that GPT-4 would be capable of, and similarly what it won't be able to achieve. However, since GPT-4 is likely going to be limited to text, it is probable that scaled up GATO would exhibit an entirely new/different set of capabilities. 

So let's discuss the following questions: What will the scaled up GATO model/data/training look like? What will it be capable of and what would be the most impressive capabilities? On the other hand, what are the things it won't achieve?

**Update: **

To get more engagement with the post I provide some more specific questions to make predictions on (Thanks Daniel Kokotajlo for the idea):

How many parameters will the model have? (Current Gato has size was 1 billion)

How large will the context window be?

Will we see some more addvanced algorithmic improvements? E.g. new type of long-term memory, actual RL and similar

What type of data would it be trained on? 

Will we observe transfer learning? e.g. seeing improvements in language model after training on audio

Will GATO see bigger improvements from Chain-of-thought style prompting compared to a LLM of similar size?

Will it be able to play new Atari games without being trained on them?

Feel free to suggest more questions and I will add them.
# Consider Applying to the Future Fellowship at MIT

**Update 2022-11-11**: given the [collapse of FTX](https://forum.effectivealtruism.org/posts/yjGye7Q2jRG3jNfi2/ftx-will-probably-be-sold-at-a-steep-discount-this-is-bad) and the [resignation of the FTX Future Fund team](https://forum.effectivealtruism.org/posts/xafpj3on76uRDoBja/the-ftx-future-fund-team-has-resigned) this fellowship does not look like it's happening.

* * *

Are you interested in doing a PhD or Masters on something altruistic? The MIT Media Lab has a [new fellowship](https://www.media.mit.edu/posts/mit-media-lab-ftx-future-fellowship/) funded by the [FTX Future Fund](https://ftxfuturefund.org/) for "ambitiously benevolent projects that could not be accomplished anywhere else". They give some examples:

> *   creating works that expand our moral circle,
> *   designing to cultivate wisdom,
> *   developing new technologies to promote or safeguard wellbeing, or
> *   directly improving the lives of current beings in ways anticipated to benefit future generations

If you're interested you would apply for admission to the MIT Media Lab for Fall 2023 ([details](https://www.media.mit.edu/posts/academics-faq/)) and also to the fellowship ([details](https://www.media.mit.edu/posts/mit-media-lab-ftx-future-fellowship/)), deadline 2022-12-01. If accepted to both, the fellowship would cover your tuition and health insurance, pay you a stipend ([~$45k/y](https://oge.mit.edu/graduate-admissions/costs-funding/stipend-rates/)), and cover $25k/y in additional research costs.

While the application requires you to say which research groups you're interested in joining, what it doesn't make clear is that this is, unusually for the US, a direct-admission system: each faculty member *chooses individually* who they would like to admit. This means it's definitely worth reaching out specifically to someone whose group you're interested in joining (though they may not respond—they get tons of email) and also thinking carefully about how your research interests intersect.

Unlike many programs, you can work on a [very wide range](https://en.wikipedia.org/wiki/MIT_Media_Lab) of things: art, science, engineering, law, governance, economics, etc, and research that would be hard to do in a more traditional graduate program is especially welcome. If you have questions about what the Media Lab is like I'm happy to try to answer, though I've only been here a few months.

(Disclosure: I [work](https://www.jefftk.com/p/leaving-google-joining-the-nucleic-acid-observatory) at the [NAO](https://www.naobservatory.org/) which is part of Kevin Esvelt's [Sculpting Evolution](https://www.sculptingevolution.org/) group at the Media Lab, and Kevin asked if I might want to write something about this opportunity.)
# A Walkthrough of A Mathematical Framework for Transformer Circuits

A Mathematical Framework for Transformer Circuits is, in my opinion, the coolest paper I've ever had the privilege of working on. But it's also very long and dense and at times confusing, and this makes me sad! So I've run an experiment, where I recorded myself reading through the paper and narrated a stream of conscious as I go - which bits are particularly cool but under-appreciated, which bits are a bit of a waste of time, which bits do I think do or do not replicate, attempting to explain the parts I think are particularly confusing, etc. You can watch it [here](https://www.youtube.com/watch?v=KV5gbOmHbjU). Sadly, it turns out I have a lot of things to say about Transformer Circuits and this turned into a 3 hour monologue, but I hope it's still useful! This is an experimental format for me for good research communication, and I'd love to hear feedback on how well it works for you! This was much easier to make than writing an entire paper, but could easily be a total waste of time if it's not clear enough to be useful!

*Disclaimer: The views in this video are entirely my personal takes - the paper was a team effort from everyone at Anthropic, especially Chris Olah, Nelson Elhage and Catherine Olsson, and I am no longer employed by Anthropic. I do not necessarily expect that any of the other authors would agree with any specific thing that I've said, but hope an unfiltered series of takes is useful!*
# How to become more articulate? 

"Being articulate" is a skill I feel like I do not have, and I would like to practice this skill. Here are some concrete aspects of "being articulate": 

*   Speaking clearly
    *   Not using filler words "um", "uh", "like", "I guess", "So", ...
    *   Not starting a sentence, getting lost halfway through, and having to restart that sentence 
    *   Using more professional vocab
    *   "Sound bite"-ability: subsets of my speech make sense on their own 
        *   e.g. not "Um, so this is why that aspect is super crucial."
*   Asking clear questions
    *   Often I feel like the question I want to ask is, "You know a lot about X, and I don't. Please talk about X for the next 2 minutes so I can absorb your knowledge." Unfortunately this is usually not socially acceptable. How to ask questions like these? 

**What concrete actions can I take to become more articulate? **
# Paper: In-context Reinforcement Learning with Algorithm Distillation [Deepmind]

Authors train transformers to imitate the trajectory of reinforcement learning (RL) algorithms. Find that the transformers learn to do *in-context RL* (that is, the transformers implement an RL algorithm)---the authors check this by having the transformers solve new RL tasks. Indeed, the transformers can sometimes do better than the RL algorithms they're trained to imitate.

Seems like more evidence for the "a generative model contain agents" point.  
  
Abstract:

> We propose Algorithm Distillation (AD), a method for distilling reinforcement learning (RL) algorithms into neural networks by modeling their training histories with a causal sequence model. Algorithm Distillation treats learning to reinforcement learn as an across-episode sequential prediction problem. A dataset of learning histories is generated by a source RL algorithm, and then a causal transformer is trained by autoregressively predicting actions given their preceding learning histories as context. Unlike sequential policy prediction architectures that distill post-learning or expert sequences, AD is able to improve its policy entirely in-context without updating its network parameters. We demonstrate that AD can reinforcement learn in-context in a variety of environments with sparse rewards, combinatorial task structure, and pixel-based observations, and find that AD learns a more data-efficient RL algorithm than the one that generated the source data.
# Reslab Request for Information: EA hardware projects

TL;DR
=====

We want to hear ideas for physical engineering projects - that is, hardware prototypes moving around atoms (not just bits) - from the EA and/or longtermist community. [Submit here](https://docs.google.com/forms/d/e/1FAIpQLScFChKQ6uZtXlNTcKlvHgB0zvi6dHjhaPV6DywYnuqaftSYhg/viewform?usp=sf_link)!

Introducing Reslab
==================

There has been a lot of talk about EA needing to take more large-scale action recently (e.g. [Longtermist EA needs more Phase 2 work](https://forum.effectivealtruism.org/posts/TruJuwtdfszFJgzwB/longtermist-ea-needs-more-phase-2-work)). And there likely exist cases where it would be helpful for action plans to start with small-scale hardware prototypes/MVPs. We came across such a case recently: as participants on [SHELTER Weekend](https://forum.effectivealtruism.org/posts/ZkkeLBwRGgxmsiqrh/apply-to-join-shelter-weekend-this-august), we realised that the smallest scale MVPs for civilizational shelters often required technical implementation and tests.

We are starting Resilience Lab (or ‘Reslab’) to build capacity for prototyping hardware ideas with relevance to areas identified as important to the long-term future.

At this stage, we would love to hear from people with more project ideas, or an interest in working on someone else’s submitted idea. We will award $200, $150, and $50 prizes to the first-, second-, and third-best ideas respectively!

Long-term goal
==============

Our intent is to develop a long-term resource to support EA and/or longtermist hardware projects. If successful, this pilot project would lead to a dedicated space with more full time staff and projects spinning out.

Which projects are suitable?
============================

Any projects that:

1.  Are primarily focused on EA and/or longtermist cause areas, and
2.  Could make substantial progress with materials funding of up to $100k and project ownership by a junior mechanical engineer and intern for six months.

To give you a better sense of what we’re interested in, here are some ideas we have for possible projects:

1.  Scaling of mushroom manufacturing techniques.
2.  Positive pressure air filtration system kits.
3.  Disinfection treatment testing.
4.  PPE development and testing.
5.  ALLFED technical development projects (e.g. open source hardware designs for greenhouse construction, seaweed farming, or food processing).

Submit your hardware project ideas [here](https://docs.google.com/forms/d/e/1FAIpQLScFChKQ6uZtXlNTcKlvHgB0zvi6dHjhaPV6DywYnuqaftSYhg/viewform?usp=sf_link)!
===========================================================================================================================================================

The deadline to be considered for prizes is January 31st, 2023.
# Quantum Suicide and Aumann's Agreement Theorem

Alice and Bob want to know which interpretation of quantum mechanics is true.  They devise an experiment: Bob will enter a box, where a quantum random number generator kills him with 99.99% probability.

Conditional on Many-Worlds being true, Bob expects to survive with 100% probability. Conditional on Copenhagen being true, Bob expects to survive with only 0.01% probability. So if Bob exits the box alive, this is strong evidence for him in favor of Many-Worlds.

Alice on the other hand, calculates a 0.01% probability of seeing Bob exit the box alive, no matter which interpretation is true. Moreover, she knows that in the world where she sees Bob come out alive, Bob will be convinced that Many-Worlds is true, and this is again independent of which theory is really true.

As such, if Bob exits the box alive, Bob should update strongly in favor of Many-Worlds being true, and Alice should leave her prior probability unchanged, as she has gained no evidence in either direction.

But if Alice and Bob are fully rational Bayesian agents with the same priors, Aumann's Agreement Theorem says that they should land on the same posterior probability for the truth of Many-Worlds after Bob exits the box. Any evidence that's valid for Bob should be equally valid for Alice, and they shouldn't be able to "agree to disagree" on what probability to assign to the Many-Worlds interpretation.

What resolves this seeming contradiction?
# Is the Orthogonality Thesis true for humans?

Basically, the Orthogonality Thesis is arguing that intelligence and goals do not correlate, at least in the limit of infinite compute and data. That is, minds and goals are very separate.

Now the question is, is the Orthogonality Thesis at least approximately true in humans?

Specifically, are behaviors and goals systematically correlated with intelligence, or is there very little correlation between goals/behavior and intelligence?

Also, what implications does the Orthogonality Thesis being true or false in humans imply about X-risk and technology?
# Prizes for ML Safety Benchmark Ideas

*“If you cannot measure it, you cannot improve it.” – Lord Kelvin (paraphrased)*

Website: [benchmarking.mlsafety.org](https://benchmarking.mlsafety.org/) – receiving submissions until August 2023.

ML Safety lacks good benchmarks, so the [Center for AI Safety](https://safe.ai/) is offering $50,000 - $100,000 prizes for benchmark ideas (or full research papers). We will award at least $100,000 total and up to $500,000 depending on the quality of submissions.

###   
**What kinds of ideas are you looking for?**

Ultimately, we will are looking for benchmark ideas that motivate or advance research that reduces [existential risks from AI](https://benchmarking.mlsafety.org/about). To provide more guidance, we’ve outlined four research categories along with [example ideas](http://benchmarking.mlsafety.org/ideas).

*   **Alignment**: building models that represent and safely optimize difficult-to-specify human values.
*   **Monitoring**: discovering unintended model functionality.
*   **Robustness**: designing systems to be reliable in the face of adversaries and highly unusual situations.
*   **Safety Applications**: using ML to address broader risks related to how ML systems are handled (e.g. for cybersecurity or forecasting).

See [Open Problems in AI X-Risk \[PAIS #5\]](https://forum.effectivealtruism.org/s/8EqNwueP6iw2BQpNo/p/hNPCo4kScxccK9Ham) for example research directions in these categories and their relation to existential risk.

### **What are the requirements for submissions?**

Datasets or implementations are not necessary, though empirical testing can make it easier for the judges to evaluate your idea. All that is required is a brief write-up ([guidelines here](https://benchmarking.mlsafety.org/guidelines)). How the write-up is formatted isn’t very important as long as it effectively pitches the benchmark and concretely explains how it would be implemented. If you don’t have prior experience designing benchmarks, we recommend reading [this document](https://docs.google.com/document/d/1Pesm4eDQKK96ZMsmzbLqfb-VvmisikJrZ7NhygS-QTU/edit?usp=sharing) for generic tips.

### **Who are the judges?**

[Dan Hendrycks](http://danhendrycks.com/), [Paul Christiano](https://paulfchristiano.com/), and [Collin Burns](http://collinpburns.com/).

  
If you have questions, they might be answered on [the website](https://benchmarking.mlsafety.org/index.html), or you can post them here. We would also greatly appreciate it if you helped to spread the word about this opportunity.

*Thanks to Sidney Hough and Kevin Liu for helping to make this happen and to Collin Burns and Akash Wasil for feedback on the website.* *This project is supported by the* [*Future Fund regranting program*](https://ftxfuturefund.org/announcing-our-regranting-program/)*.*
# New book on s-risks

I have just published my new book on s-risks, titled *Avoiding the Worst: How to Prevent a Moral Catastrophe*. You can find it on [Amazon](https://www.amazon.com/dp/B0BK59W7ZW), read the [PDF version](https://centerforreducingsuffering.org/wp-content/uploads/2022/10/Avoiding_The_Worst_final.pdf), or listen to the [audio v](https://www.youtube.com/watch?v=mjJpZNsEqTg&ab_channel=AaronBergman)[ersion](https://www.youtube.com/watch?v=xvP_dJM1X4k).

![](https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/XyCLLYkBCPw44jpmQ/g4db7qy5rqbprw346hgf)

The book is primarily aimed at longtermist effective altruists. I wrote it because I feel that s-risk prevention is a somewhat neglected priority area in the community, and because a single, comprehensive introduction to s-risks did not yet exist. My hope is that a coherent introduction will help to strengthen interest in the topic and spark further work.

Here’s a short description of the book:

> From *Nineteen Eighty-Four* to *Black Mirror*, we are all familiar with the tropes of dystopian science fiction. But what if worst-case scenarios could actually become reality? And what if we could do something now to put the world on a better path?
> 
> In *Avoiding the Worst*, Tobias Baumann lays out the concept of risks of future suffering (**s-risks**). With a focus on s-risks that are both realistic and avoidable, he argues that we have strong reasons to consider their reduction a top priority. Finally, he turns to the question of what we can do to help steer the world away from s-risks and towards a brighter future.

For a rough overview, here’s the book's table of contents:

> **Part I: What are s-risks?**
> 
> Chapter 1: Technology and astronomical stakes
> 
> Chapter 2: Types of s-risks
> 
> **Part II: Should we focus on s-risks?**
> 
> Chapter 3: Should we focus on the long-term future?
> 
> Chapter 4: Should we focus on reducing suffering?
> 
> Chapter 5: Should we focus on worst-case outcomes?
> 
> Chapter 6: Cognitive biases
> 
> **Part III: How can we best reduce s-risks?**
> 
> Chapter 7: Risk factors for s-risks
> 
> Chapter 8: Moral advocacy
> 
> Chapter 9: Better Politics
> 
> Chapter 10: Emerging technologies
> 
> Chapter 11: Long-term impact

And finally, some blurbs for the book:

> “One of the most important, original, and disturbing books I have read. Tobias Baumann provides a comprehensive introduction to the field of s-risk reduction. Most importantly, he outlines sensible steps towards preventing future atrocities. Highly recommended.”
> 
> **— David Pearce, author of *****The Hedonistic Imperative*** **and*****Can Biotechnology Abolish Suffering?***
> 
> “This book is a groundbreaking contribution on a topic that has been severely neglected to date. Tobias Baumann presents a powerful case for averting worst-case scenarios that could involve vast amounts of suffering. A much needed read for our time.”
> 
> **— Oscar Horta, co-founder of Animal Ethics and author of *****Making a Stand for Animals***
# Syncretism

### The act or system of blending, combining or reconciling inharmonious elements.

![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F2ed854f0-a690-4229-a8ae-5e110f0ed60b_4032x2268.jpeg)

Kalpa, Sutlej Valley, India

I learned the word *syncretism* when I was reading about Kalpa, a village in the Indian Himalaya that I visited on October 19, 2022. In ancient times, Kalpa was at the crossroads of Tibet and India, and received constant influence from both cultures. What I found truly fascinating about Kalpa is that part of its population practices syncretism. Instead of choosing between Hinduism or Buddhism, they decided to practice both, merging its rituals and customs. Furthermore, the religions not just only coexist harmoniously in Kalpa: They share temples and festivals.

Syncretism between Hinduism and Buddhism is not only practiced in Kalpa; it is common in the rest of the Kinnaur District of Himachal Pradesh as well as other areas of India. As far as I am aware, in modern times there aren’t many places in the world where two major religions are blended together like Hinduism and Buddhism are in Kalpa. This got me thinking: Why doesn’t this happen more often in modern times?

![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F0aad932d-4126-436d-a3bb-6dbfe40f1ea9_4032x2268.jpeg)

Rakchham, Baspa Valley, India

I believe more syncretism between faiths does not happen because modern society would dismiss it: It would likely mock it and call it *pseudo-faith*. I believe this is a shame: to me it makes sense that if religious syncretism was practiced and accepted more in society, the world would be more harmonious. The Kinnauris were too isolated to be told that merging religious practices was wrong, and by the time their lands were well connected to the modern world, syncretism was very ingrained in their culture.

Syncretism does not only apply to theology: it also applies to ideology. In the past, people would blend schools of thought, mainly in the name of compromise. As political polarization continues to thrive in the world, my experience spending time in a place where religious syncretism got me thinking: How can we apply political syncretism in our communities so that we can live in more harmoniously?

![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2F7a0eade6-0c21-4075-985f-e8df3d8e53bd_4032x2268.jpeg)

Key Monastery, Spiti Valley, India
# Rapid Test Throat Swabbing?

Back in January there was a bunch of buzz around using standard rapid tests but swabbing the throat. Ex:

> On one side are well-regarded experts who argue that swabbing your throat in addition to your nose may increase the chances that a home test can detect omicron. On the other side are similarly well-regarded experts, including those at the Food and Drug Administration, who urge Americans to avoid experimenting with kits that were developed and tested using only nasal swabs.  
> —[Washington Post, 2022-01-06](https://www.washingtonpost.com/wellness/2022/01/06/adding-throat-swab-covid-test/)

or:

> if you want to know if you have infectious Covid-19, you need to use throat swabs. It's still not 100% sensitivity or anything, but it's a lot better, whereas nose swabs are useful but in no way all that reassuring.  
> —[Zvi, 2022-01-13](https://thezvi.wordpress.com/2022/01/13/omicron-post-14/)

Then the buzz died down, and it seems like everyone just forgot about it? No clear recommendation either way, just a lot of Tweets, a few blog posts, news stories about the controversy, and then not much else. Google Trends shows the spike and drop-off:

[![](https://www.jefftk.com/google-trends-throat-swab.png)](https://www.jefftk.com/google-trends-throat-swab-big.png)

\[EDIT: I wouldn't read too much into the chart: it turns out that most covid-related keywords followed as similar trend, as did covid itself\]

What happened? Why did we all stop talking about it, without what seems like any firm conclusions? Some of this is maybe that there were conflicting studies: Schrom et. al. posted a [preprint](https://www.medrxiv.org/content/10.1101/2022.01.08.22268954v4) on 2022-01-19 ([published version](https://www.acpjournals.org/doi/10.7326/m22-0202)) where it looked like it worked poorly, but Goodall et. al. posted [a different preprint](https://www.medrxiv.org/content/10.1101/2022.01.18.22269426v1) on 2022-01-21 ([published version](https://journals.asm.org/doi/10.1128/spectrum.00217-22)) where it did look good. I also saw speculation that some of the cases where people got a negative with nasal swabbing but a positive with throat swabbing may have been false positives—rapid tests are [sensitive to pH](https://journals.asm.org/doi/10.1128/Spectrum.00683-21), and especially if you've been eating or drinking recently your throat might be pretty far from neutral.

Ten months later, why hasn't this settled out into recommendations that either say to swab throat and then nose, or that seem to have considered the new evidence and say still don't? We're doing a lot of testing, and if this is a better way to be using the tests we already have it would be great to know! Does anyone whose been following this have a better understanding of what happened?

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02vUpm6YJhXcbSwe3Tzabm2Ds3zJMC6Barnt5aYuHUn6UjFdLgYpbQaefLnuiNuxbPl)*
# aisafety.community - A living document of AI safety communities

*Thanks to* [*plex*](https://forum.effectivealtruism.org/users/plex-1) *for co-authoring the post (co-authors are currently not reflected in EA Forum when crossposted from LessWrong).*

![](https://lh3.googleusercontent.com/r3y0mGBWw6QEHG8klHNoIwj-Ny_c2_B2F72RCiNl9v3Risux-SjuZMSEwkphqSYm7WXXuB6yswn7xMK5cuLLOuA9_oXImCPG7LCEb-k4WZN90_i3fqZmAn1hx1f5pFi5r7vi8stOyy5sK7vVcRTckpMCeRMlGKX7rFL5aAKbdDH0JGmzFu_fdEV8JA)

*The AI Safety Communities logo, by DALL-E 2*

The AI safety field has been rapidly growing over the last few years, and more and more communities have been sprouting up all over the world, both physically and online.

The Alignment Ecosystem Development (AED) team identified having a living document listing them all as a low-hanging fruit, and volunteers are maintaining this [live database of AI safety communities](https://aisafety.community/).

Here’s what you can do now:

1.  **Join communities** you’re interested in and participate.
2.  **Link this to your friends.**
3.  **Add communities** to the document if they’re missing (account creation with coda.io is required)
4.  **Improve the descriptions** of any communities which you’re a part of. Many of the current descriptions are incomplete or outdated, jump right in and share your knowledge.
5.  **Volunteer as a maintainer**, or join the Alignment Ecosystem Development [Discord server](https://discord.gg/gBEJvahKRz), where they host [monthly calls to pitch or join shovel ready volunteering opportunities](https://calendar.google.com/event?action=TEMPLATE&tmeid=NDIzMG1sMDZqa2g4cHAyNXMycXZyb2JsaGxfMjAyMjExMTVUMTcwMDAwWiA4OTllZDdjMDc0NGRkZmNmYWM2NzA1NTJjZTcyNzZmOGJkMTgwMDVmZTlmZGZkM2NiYTE0YmQ2YzFmNWM1NDIzQGc&tmsrc=899ed7c0744ddfcfac670552ce7276f8bd18005fe9fdfd3cba14bd6c1f5c5423%40group.calendar.google.com&scp=ALL).

  
*Thanks to JJ Hepburn from *[*AI Safety Support*](https://www.aisafetysupport.org/) *for providing the domain through*[*ea.domains*](https://ea.domains/)*, another soon-to-be-launched AED project!*
# Electric Kettle vs Stove

There was a discussion at lunch yesterday: what's faster to boil water, an (American) electric kettle or a pot on the stove? The kettle is more efficient but plugs into a 120V 15A receptacle on a shared circuit, while the stove has to transfer heat to a pot but gets exclusive use of a 240V 50A circuit (though not all of that goes into a single burner). I thought the stove, a coworker thought the kettle. Let's find out!

The contestants:

[![](https://www.jefftk.com/electric-kettle.jpg)](https://www.jefftk.com/electric-kettle-big.jpg)

A Homigrand electric kettle, rated as drawing 1000W.

[![](https://www.jefftk.com/pot-on-stove.jpg)](https://www.jefftk.com/pot-on-stove-big.jpg)

A Faberware stainless steel pot on a GE Spectra glass-top resistive electric stove ([JBP66BBWH](https://www.geappliances.com/appliance/GE-Spectra-30-Free-Standing-QuickClean-Electric-Range-JBP66BBWH)). The 6" burner I used is rated as drawing 1500W.

I filled them both with a quart (about a liter) of water, which was near the capacity of the kettle, and verified that both were starting at the same temperature (62F):

[![](https://www.jefftk.com/kettle-water-62f.jpg)](https://www.jefftk.com/kettle-water-62f-big.jpg)

[![](https://www.jefftk.com/pot-water-62f.jpg)](https://www.jefftk.com/pot-water-62f-big.jpg)

The thermometer is wet in the second picture because I accidentally dropped it into the pot, but it's rated [IP66](https://www.thermoworks.com/thermopop/) and didn't mind.

I started them simultaneously, and waited. The kettle took 6:32 to reach its automatic shutoff, and I measured at at 212F. The pot took 7:20 until I guessed it was boiling from the sound, and it also measured 212F. So the pot is about 12% slower despite using 50% more power, so 68% more energy. Still, it's only 0.18kWh vs 0.11kWh which at [$0.21/kWh](https://www.jefftk.com/p/rough-utility-costs) is $0.04 vs $0.02 so I'm more interested in the time difference. Even that is small enough, however, that if what I want is boiling water in a pot to cook something I think I'll still use the pot.

One exception, which I was already doing, is that when I'm making pasta in a hurry, typically for the kids, I'll put half the water in each and then pour the kettle into the pot when it boils. (If I had been dividing the water exactly I'd have already known the answer to this question!) The main gains here are from parallelism, however, not efficiency.

This does make me wonder whether Americans (or British expats) ever put dedicated 240V outlets in their kitchens so they can use ~2-3x more powerful European kettles?
# Me (Steve Byrnes) on the “Brain Inspired” podcast

[***Official episode page***](https://braininspired.co/podcast/151/) ***— Listen on your favorite podcasting app —*** [***Watch on YouTube***](https://www.youtube.com/watch?v=w7k3o5uIBjs)

The discussion centered around my blog post series from earlier this year—[“Intro to Brain-Like AGI Safety”](https://www.lesswrong.com/s/HzcM2dkCq7fwXBej8)—as well as AGI and safety and alignment more generally.

As context, the [Brain Inspired podcast](https://braininspired.co/episodes/) (which I have long listened to myself!) focuses on the intersection between neuroscience and AI. **My assumption was that most listeners would be either unfamiliar with or skeptical about AGI in general and AGI x-risk in particular, so I tried my best to keep things simple and relatable, avoid jargon, and so on.**

As is inevitable in a real-time format, there are lots of places where I misspoke[^epo1vsc1ytd] or could have said things better. But I’m generally pretty happy with how it turned out.  :)

Thanks Paul Middlebrooks for having me on!

[^epo1vsc1ytd]: Next time I hear someone on a podcast say something that betrays a complete lack of understanding of some topic X, I will be more open to the possibility that they understand X perfectly well, and were just tripping over their words. Talking is hard, man!!
# Average utilitarianism is non-local

This is a pretty obvious point, which is sort of made [here](https://slatestarcodex.com/2017/08/09/the-lizard-people-of-alpha-draconis-1-decided-to-build-an-ansible/) but I haven't seen anyone lay out explicitly.

Under average utilitarianism the morality of having a child depends on whether a billion light years away there's any sentient aliens, how many there are, and whether their average happiness is greater or less than your child's would be. This despite your actions having zero impact on them, and vice versa.

This, far more than the sadistic conclusion, seems to me to sound the death knell of average utilitarianism. If your formalisation of utilitarianism both completely diverges from intuition in a straightforward situation (should I have kids), and is incalculable not just practically, but even in theory (it requires knowing the contents of the whole universe), what's the point?
# Why Aren't There More Schelling Holidays?

A common pattern when working in teams: when one person is out on vacation, there’s a disproportionate drop in productivity for the team as a whole. Lots of things end up blocked on the person who’s out. For instance, in a small software team, maybe the developer who owns a particular API is out, and nobody else knows that API well enough to confidently make changes, so anything involving changes in that API ends up blocked until the owner is back. Or, even if someone else steps in to handle changes to the API, they’re much more likely to introduce bugs.

To some extent, we can structure teams to mitigate that kind of problem. “Everyone does everything” is very costly, but underrated. Weaker versions like “At least three people can cover any given thing” are also costly, but underutilized.

But there’s a much less costly strategy which gains a decent chunk of the same benefits: coordinate vacation. If everyone goes on vacation 3 weeks per year *on the same 3 weeks*, then that’s only 3 weeks of nonproductivity for the team; the other 49 weeks are full steam. If everyone on a 10-person team goes on vacation 3 weeks per year at different times, then that’s 30 weeks per year of disproportionate productivity-loss; the team is at full steam less than half the time.

Thus the case for national holidays: it’s not like everyone is required to get the day off (at least in the US), but it’s a Schelling point for lots of people to take a vacation at the same time. People get their much-needed recovery time simultaneously, so teams can spend more time working at full capacity.

But in practice, most people take a lot more vacation days than there are national holidays - which suggests that the current number of holidays is too small.

It seems like companies (or nonprofit orgs) could profit by filling that gap at the company level: declare official “company holiday” weeks, and offer incentives for employees to take their vacations during those weeks. Unlike national holidays, the company holidays won’t necessarily line up neatly with good vacation times for spouses/children/parents/friends. But on the upside, since a company holiday need not be at the same time as the rest of the country, traveling should be easier.

Expanding from there, one could imagine a few companies/orgs which need to interface with each other a lot coordinating their Schelling holidays. For instance, when I worked at a trading company, the usual rule was “if the markets aren’t open today, we’re not working today”.
# Open & Welcome Thread - November 2022

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the [Highlights from the Sequences](https://lesswrong.com/highlights), a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/faq#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/faq). If you want to orient to the content on the site, you can also check out the [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/tag/open-threads?sortedBy=new). The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# Real-Time Research Recording: Can a Transformer Re-Derive Positional Info?

**New experiment:** Recording myself real-time as I do mechanistic interpretability research! I try to answer the question of what happens if you train a toy transformer without positional embeddings on the task of "predict the previous token" - turns out that a two layer model can rederive them! You can watch me do it [here](https://youtu.be/yo4QvDn-vsU), and you can follow along with my code [here](https://colab.research.google.com/github/neelnanda-io/Easy-Transformer/blob/no-position-experiment/No_Position_Experiment.ipynb). This uses a transformer mechanistic interpretability library I'm writing called EasyTransformer, and this was a good excuse to test it out and create a demo!

This is an experiment in recording and publishing myself doing "warts and all" research - figuring out how to train the model and operationalising an experiment (including 15 mins debugging loss spikes...), real-time coding and tensor fuckery, and using my go-to toolkit. My hope is to give a flavour of what actual research can look like - how long do things actually take, how often do things go wrong, what is my thought process and what am I keeping in my head as I go, what being confused looks like, and how I try to make progress. I'd love to hear whether you found this useful, and whether I should bother making a second half!

Though I don't want to overstate this - this was still a small, self-contained toy question that I chose for being a good example task to record (and I wouldn't have published it if it was TOO much of a mess).
# Sequence Reread: Fake Beliefs [plus sequence spotlight meta]

A couple months ago the LessWrong team started putting Sequence Spotlights at the top of the home page, to give people reasonably bite-sized chunks of canonical LessWrong content to read (or reread). i.e, this thing here:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/acf341561a675dbbe75bc7ca3112688382d17b776a5898b2.png)

I thought I'd experiment with making a post where people could discuss the posts, whether you're reading them for the first time or coming back to them.

The [Fake Beliefs sequence](https://www.lesswrong.com/s/7gRSERQZbqTuLX5re ) introduces a bunch of ways that "things we call beliefs" can, well, not actually be beliefs. Noticing you don't actually believe a thing is a foundational rationalist skill.

**Object level discussion prompts:**

*   What does it feel like to have a "fake belief"? 
*   What notable fake beliefs have you found since getting into "sequences-style-rationality?". Were there any subtler fake beliefs that took longer to notice?
*   If you first read this awhile ago, has your perspective on fake beliefs changed?
*   If you think someone's belief is fake, and they don't share a bunch of rationality culture, what's the best way to engage them?

**Meta level discussion prompts:**

*   How are you generally feeling about Sequence Spotlights? 
    *   What is your experience of them like? Have you clicked on anything? 
    *   Do you get value from seeing art or canonical reading on the top of the page, independent from reading it?
*   I've considered giving Sequence Spotlights a "Create Discussion Meetup" button, such that Sequence Spotlights more easily serve as default reading material for local communities. Would anyone be interested in that?
# Is there a good way to award a fixed prize in a prediction contest?

Let's say we're having a prediction contest where we want people to pre-register their probability estimates for the outcomes of a set of discrete events (e.g. the candidates for various races in an upcoming election). We have a fixed sum of, say, $100 that we must award to the winner(s) in some way. We are not allowed to award any more or less than that. How can we determine the award in such a way as to incentivize people to honestly report their estimates?

The naïve approach would be to do the following:

1. Score everyone's predictions according to some [proper scoring rule](https://en.wikipedia.org/wiki/Scoring_rule#Examples_of_proper_scoring_rules)
2. Award the $100 to whoever has the highest score.

However, I'm pretty sure this doesn't work. A proper scoring rule incentivizes honest estimates if the goal is to maximize your total score; but in this case, maximizing your score is not the goal. Under this system, I'm incentivized to make estimates that may have a suboptimal expected score as long as the expectation is high-variance, because that will increase the probability that I get the top spot.

Can you figure out a system that avoids this problem?

(Distributing the prize randomly or in partial shares is allowed. However, destroying money or creating extra money is not.)
# Lazy Python Argument Parsing

When I'm writing real Python programs I use [argparse](https://docs.python.org/3/library/argparse.html). When I'm writing quick scripts, I destructure `sys.argv[1:]`. For example:

width, depth, height = sys.argv\[1:\]
...

Or, if I need to `import` the file:

def start(width, depth, height):
  ...

if \_\_name\_\_ == '\_\_main\_\_':
  start(*sys.argv\[1:\])

This is a very simple way of handling positional arguments. It ignores `sys.argv[0]` which is likely the name of the script, and then assigns the remaining arguments to the variables. If I provide the wrong number of arguments it's a bit shouty, but it's clear enough for a quick script and does remind me of the intended arguments:

Traceback (most recent call last):
  File "example.py", line 2, in width, depth, height = sys.argv\[1:\]
ValueError: too many values to unpack (expected 3) 

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02fDMMMxPjBkCe3QKLHyQFBsJUZ8QtaD9KF4AX33Tef3moU65QKGkVwe1g5fAkcTtal)*
# Open Letter Against Reckless Nuclear Escalation and Use

There's been ample interest in the probability that the Ukraine conflict escalates into a nuclear way, ranging from predictions on [Metaculus](https://www.metaculus.com/questions/12591/nuclear-detonation-in-ukraine-by-2023/), [Manifold](https://manifold.markets/AndyMartin/will-a-nuclear-weapon-be-launched-i-015e44ed91f5)  and [Polymarket](https://polymarket.com/market/will-russia-use-a-nuclear-weapon-before-2023) to estimates by [David Orr](https://www.lesswrong.com/posts/WXRQwuapvGP4FesnW/why-i-think-nuclear-war-triggered-by-russian-tactical-nukes), [myself](https://www.lesswrong.com/posts/Dod9AWz8Rp4Svdpof/why-i-think-there-s-a-one-in-six-chance-of-an-imminent) and others. Aside from *estimating* this probability, what can we do to *reduce* it?

Current discourse is often framed as if there were only two options:  

1.  Be forceful
2.  Deescalate by appeasement, effectively capitulating to nuclear blackmail

This is misleading, because there's more than one way to stand strong (including going on the offense militarily):  (a) pursuing or (b) eschewing escalation that has negligible military value. Suppose a bully punches you. If you choose to stand strong, I'd encourage you not to combine it with spitting in his face, poking fun of his appearance and goading him to pull out that gun you see he has holstered. Unfortunately, the Ukraine conflict has seen many instances of analogous escalation without significant military benefit, and arguably not only from one side. Stigmatizing such reckless behavior can in my opinion reduce the risk of nuclear war without any appeasement or concessions. I'm therefore inviting you to join me and many others as a signatory on an [open letter](https://futureoflife.org/open-letter/open-letter-against-reckless-nuclear-escalation-and-use/) that aims both to de-normalize nuclear threats and to re-mainstream non-appeasing de-escalation strategies that reduce the risk of nuclear war without giving into blackmail.

**Would you like to join as a signatory? **This would be wonderful, because it would help reduce the probability of the greatest catastrophe in human history. To read and potentially sign the open letter, please click [here](https://futureoflife.org/open-letter/open-letter-against-reckless-nuclear-escalation-and-use/).