Harley-ml/lesswrong huggingface dataset:
# Apply to the Cooperative AI PhD Fellowship by October 14th!

Applications are now open for the [2025 Cooperative AI PhD Fellowship](https://www.cooperativeai.com/phd-fellowship/2025)! Fellows will receive up to **$40,000** per year plus tuition fees, alongside many other benefits. Applicants should be enrolled or are about to be enrolled in a PhD programme at an accredited university (anywhere in the world), with a specific interest in **multi-agent/cooperation problems** involving AI systems. Your research proposal should aim to contribute to societally beneficial AI development. The [Cooperative AI Foundation](http://www.cooperativeai.org/) is committed to the growth of a diverse and inclusive research community, and we especially welcome applications from under-represented backgrounds. Find out more and apply [here](https://www.cooperativeai.com/phd-fellowship/2025) before the deadline: **October 14th**.
# Seeking AI Alignment Tutor/Advisor: $100–150/hr

I am actively looking for a tutor/advisor with expertise in AI x-risk, with the primary goal of collaboratively determining the most effective ways I can contribute to reducing AI existential risks (X-risk).

### Tutoring Goals

I suspect that I misunderstand key components of the mental models that lead some highly rational and intelligent individuals to assign a greater than 50% probability of AI-related existential catastrophe ("p-doom"). By gaining a clearer understanding of these models, I aim to refine my thinking and make better-informed decisions about how to meaningfully reduce AI X-risk.

Specifically, I want to delve deeper into why and how misaligned AGI might be developed, and why it wouldn’t be straightforward to solve alignment before it becomes a critical issue.

To clarify, I do NOT believe we could contain or control a misaligned AGI with current safety practices. What I do find likely is that we will be able to avoid a situation altogether.

In addition to improving my understanding of AI X-risks, I also seek to explore strategies that I could aid in implementing in order to reduce AI X-risk.

### About Me

\- My primary motivation is effective altruism, and I believe that mitigating AI X-risk is the most important cause to work on.  
\- I have 7 years of experience working with machine learning, with a focus on large language models (LLMs), and possess strong technical knowledge of the field.  
\- My current p-doom estimate is 25%, derived from my own model, which gives about 5%, but I adjust upward in since some highly rational thinkers predicts significantly higher p-doom. Even if my p-doom were 1%, I would still view AI X-risk as the most pressing issue and dedicate my time to it.  
   
Why Become My Tutor?

\- You will be directly contributing to AI safety/alignment efforts, working with someone highly committed to making an impact.  
\- Opportunity for \*\*highly technical 1-on-1 discussions\*\* about the cutting-edge in AI alignment and X-risk reduction strategies.  
\- Compensation: $100–150 per hour (negotiable depending on your experience).

### Ideal Qualifications

\- Deep familiarity with AI existential risks and contemporary discussions surrounding AGI misalignment.  
\- A genuine interest in refining mental models related to AI X-risk and collaborating on solutions.  
\- p-doom estimate above 25%, since I aim to understand high p-doom perspectives.  
\- Strong interpersonal compatibility: It’s crucial that we both find these discussions rewarding and intellectually stimulating.

### Structure & Logistics

\- Weekly one-hour meetings focused on deep discussions of AI X-risk, strategic interventions, and mental model refinement.  
\- Flexible arrangement: you can invoice my company for the tutoring services.

### How to Apply

If this opportunity sounds appealing to you, or if you know someone who may be a good fit, please DM me here on LessWrong.
# Open Thread Fall 2024

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the [Highlights from the Sequences](https://lesswrong.com/highlights), a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/faq#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/faq). If you want to orient to the content on the site, you can also check out the [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/tag/open-threads?sortedBy=new). The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# An argument that consequentialism is incomplete

I think consequentialism describes only a subset of my wishes. For example, maximizing money is well modeled by it. But when I'm playing with something, it's mostly about the process, not the end result. Or when I want to respect the wishes of other people, I don't really know what end result I'm aiming for, but I can say what I'm willing or unwilling to do.

If I try to shoehorn everything into consequentialism, then I end up looking for "consequentialist permission" to do stuff. Like climbing a mountain: consequentialism says "I can put you on top of the mountain! Oh, that's not what you want? Then I can give you the feeling of having climbed it! You don't want that either? Then this is tricky..." This seems a lot of work, just to do something I already want to do. There are many reasons to do things - not everything has to be justified by consequences.

There are of course objections. Objection one is that non-consequentialist wishes can make you go in circles, like that Greg Egan character who spent thousands of hours carving table legs, making himself forget the last time so he could enjoy the next. But when pushed to such extremes, a consequentialist goal like maximizing happiness can also lead to weird results (vats of happiness goo...) And if we don't push quite so hard, then I can imagine utopia containing both consequentialist and non-consequentialist stuff, doing things for their own sake and such. So there's no difference here.

Objection two is that our wishes come from evolution, which wants us to actually achieve things, not go in circles. But our wishes aren't all perfectly aligned with with evolution's wish (procreate more). They are a bunch of heuristics that evolution came up with, and a bunch of culturally determined stuff on top of that. So there's no difference here either - both our consequentialist and non-consequentialist wishes come from an equally messy process, so they're equally legitimate.
# Metaculus Is Open Source

We completely rewrote the platform to make it easier for open-source contributors to build on our code. Maybe that's you? If it is, you can find the Metaculus code repo and can learn more [here](https://github.com/Metaculus/metaculus?tab=readme-ov-file#contributing). 

We've included instructions for setting up a local development environment and on creating an issue. We've also selected some easier [issues](https://github.com/Metaculus/metaculus/issues?q=is%3Aissue+is%3Aopen+label%3ASimple), perfect for getting started.

Make a significant contribution, and we'll mail you a Metaculus hoodie to show our gratitude. 

Also, in rewriting the platform, we introduced a variety of new features and quality-of-life improvements, including improvements to user profiles, notifications, the ability to zoom in on a time period in a forecast plot, and much more. You can find all the details [here](https://www.metaculus.com/notebooks/28734/).
# There is a globe in your LLM

[Gurnee & Tegmark (2023)](https://arxiv.org/abs/2310.02207) trained linear probes to take an LLM's internal activation on a landmark's name (e.g. "The London Eye"), and predict the landmark's longitude and latitude. The results look like this:[^6wr9vzyw3j5]

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d6d183f22e900998bca8f78868b58396dee6a66a21c57528.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/7f042e1a20568cc825c74a891dc1f458d02962e9be755b48.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/16a1d2684c5402c4a0f2171ee3325aa3c6f1065513ad17af.png)

Two angles of true world atlas, with predicted atlas hovering above. True locations are red points; predicted locations are blue, in a slightly raised plane, linked to the corresponding true location by a grey line.

So LLMs (or at least, Llama 2, which they used for this experiment) contain a pretty good linear representation of an atlas.

Sometimes, like when thinking about distances, a globe is more useful than an atlas. Do models use the globe representation? To find out, we can train probes to predict the (x,y,z) coordinates of landmarks, viewed as living in 3D space. Here are the results:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e2b68fe9604d046442adc834c3a01a52b8ae370198a62a75.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/c333c528df917e18376ae545249c2c2858d253ef40ce5898.png)

Left: Europe faces us. Middle: The Pacific faces us. Right: India faces us. Predicted points are scaled up by 1.5x for visualization purposes.

You can rotate the plot yourself (and see the code used to generate it) in [this notebook](https://colab.research.google.com/drive/11tAVSiLa_8KeNq0F3NJgfrW1Jo1s2DV5?usp=sharing).

The average Euclidean (i.e. "through-the-Earth") distance between the true and predicted locations is 535 miles: roughly the distance from Paris to Berlin, from LA to Portland, or from Mumbai to New Delhi.

So LLMs contain a pretty good linear representation of a globe!

I don't know if there's any useful takeaway from this. I just thought it was cool.

[^6wr9vzyw3j5]: These plots are my reproduction. I used Llama-3-8B-Instruct, and Gurnee & Tegmark's dataset. For ~7,000 of the 40,000 landmarks, the model did not correctly answer the question "which country is {place} in?", so I removed these from the dataset. I fed landmark names into the model with no prefix, and cached final-token residual-stream activations 70% of the way through the model. All plots and \(R^2\) values in this post were generated using a held-out 20% test split.
# ACX Ballot Meetup: Boston

This is the local instance of [https://www.astralcodexten.com/.../ballots-everywhere...](https://www.astralcodexten.com/p/ballots-everywhere-times-and-places?fbclid=IwZXh0bgNhZW0CMTAAAR3mRE7jS-55291-skWlYgICFA0QfP-sr3aL-cQxoHyKyzmHqTc6QZFqcvQ_aem_7iGh7I6QBsG7yhHi7JY3qQ). We'll debate recommendations for candidates and ballot initiatives, and produce a voting guide.  
We're at apartment 2 of the above address.

You might be interested in the [LessWrong Political Prereqs](https://www.lesswrong.com/events/CL8rAfpHa4RazPuQe/political-prereqs) meetup the week before.

Note: Skyler created the event here, but isn't running it. Taymon B. is.
# Demis Hassabis and Geoffrey Hinton Awarded Nobel Prizes

Geoffrey Hinton received the [Nobel Prize in Physics](https://www.nobelprize.org/prizes/physics/2024/summary/) for his role in creating the modern field of deep learning. This will strengthen his reputation as the "Godfather of AI" which was already used to amplify his [public statements about AI risk](https://www.lesswrong.com/posts/bLvc7XkSSnoqSukgy/a-brief-collection-of-hinton-s-recent-comments-on-agi-risk).[^36idbawnci]

> The Nobel Prize in Physics 2024 was awarded to John J. Hopfield and Geoffrey E. Hinton “for foundational discoveries and inventions that enable machine learning with artificial neural networks.”

Demis Hassabis, Deepmind's co-founder and CEO, received the [Nobel Prize in Chemistry](https://www.nobelprize.org/prizes/chemistry/2024/summary/) for his role in creating AlphaFold.

> The Nobel Prize in Chemistry 2024 was divided, one half awarded to David Baker "for computational protein design", the other half jointly to Demis Hassabis and John M. Jumper "for protein structure prediction".

AlphaFold's extraordinary contributions to the field of computational biology are relatively undisputed, but the relevance of neural networks to physics has led to some discussions. Nevertheless, it can be said that AI-related research scored two Nobel Prizes, which corresponds to the credit and attention AI currently receives in the public eye.

Nobel Prize in Medicine was awarded for the discovery of miRNA. This year's Nobel Prizes in Literature and Peace are yet to be awarded.[^ygdfymmzr7] 

[^36idbawnci]: The media attention also amplifies many other remarks too (e.g. takes on Trump and AI or UBI). I also remember being disappointed by the lack of depth behind his public claims about AI, though it might have been caused by the reporting filter. I haven't spent much time on this but it's not clear to me what his concrete recommendations are or which policy proposals would he would support. 

[^ygdfymmzr7]: Will ChatGPT complete the trio? Sam Altman would be a very funny Literature laureate : )
# Do you want to do a debate on youtube? I'm looking for polite, truth-seeking participants.

I am testing a new debate format and am looking for people to debate any of the following (or make a case for other things):

*   Policy decisions
*   EA Cause prioritisation
*   What should/will happen in Taiwan/Ukraine/Gaza
*   US elections
*   Economics
*   Animal consciousness

The debate would happen via a video call and then be put on my youtube (and I guess other places if you want). 

Why am I asking for volunteers?
-------------------------------

Most debates are bad, at least in part because the people who have them just want to win. Go on youtube and you can find 10s of debates with bad faith arguments and a descent into insults. These debates are usually about either US politics, trans issues or religion. 

I think a debate will go much better if it starts with a pair of people who both have some moderate care for the truth and have reputation locked into being open and polite. The work of having good discussions is hard enough, without making it harder.

What is the format?
-------------------

I am testing a new format, which I ran at Manifest (and which many seemed to enjoy[^rskf86lwmyk]):

*   Poll the two people debating
*   Find things they agree and disagree on
*   Introduce the debaters and the things they agree on
*   Take a specific area they disagree on
*   Each debater gets \[2 x 5\] minutes where they are in control. They can speak or ask questions
*   As the moderator, I uphold their boundaries during this time - they can't be interrupted.
*   Sometimes I will write key cruxes on a shared whiteboard
*   After the time is up for that topic, we move onto another specific disagreement
*   There is no mechanism to figure out who has "won"

Optiona additions I might try later.

*   [@Katja Grace](https://www.lesswrong.com/users/katja-grace?mention=user) might sometimes help and try and disentangle cruxes. I am interested in having active moderation[^r4xnyj3owu]
*   Give both candidates M10,000 on manifold and require that they bet it during the debate on markets relating to the debate which I will create
*   Some kind of live LLM based fact checking
*   Some kind of "throw to the audience" fact checking if I do a livestream

So do you have a view on these topics and want to discuss it?
-------------------------------------------------------------

The format aims to be relaxed and not require you to manage your own boundaries. If I do my job well, you don't have to worry about someone being rude, taking more time than allotted or shifting the question around. I hope this might allow participants more space to focus on the discussion.

[^rskf86lwmyk]: Including @Eliezer Yudkowsky who asked if I could run one with him. He seems pretty hard to please. 

[^r4xnyj3owu]: I don't intuitively think I'd be very good at this. I think Katja would probably be very good.
# Rationality Quotes - Fall 2024

[Once upon a time](https://www.lesswrong.com/tag/rationality-quotes?sortedBy=old), there were posts where people would comment with quotes. The last one was [a while ago](https://www.lesswrong.com/posts/EdsGjj3aM9osyDHqa/rationality-quotes-august-2016), and they'd been getting less and less traffic. 

I have a theory that there's a quote overhang. Firstly, I think LessWrong has recovered a lot since the 2015~2017 decline, and so there might just be more users around. Second, it's been half a decade and so many new quotable lines may have been written. I have a another theory that the way LessWrong 2.0 splits Frontpage and Personal is rougher on a quotes thread than 1.0's Main and Discussion. The best thing to do with a theory is to test it.

The rules are as follows:

*   Please post all quotes separately, so that they can be upvoted or downvoted separately. (If they are strongly related, reply to your own comments. If strongly ordered, then go ahead and post them together.)
    
*   Do not quote yourself.
    
*   Do not quote from LessWrong itself, HPMoR, Eliezer Yudkowsky, or Robin Hanson. If you'd like to revive an old quote from one of those sources, please do so [here](https://www.lesswrong.com/r/discussion/lw/i6h/rationality_quotes_from_people_associated_with/). **I'm going to declare an exception:** Planecrash quotes are fine to post this round. Planecrash was half written by Eliezer, but wasn't written on LessWrong and was written entirely since the last quotes post, so there's close to no chance someone already put it in a quotes thread.
    
*   No more than 5 quotes per person per monthly thread, please.
    
*   Provide sufficient information (URL, title, date, page number, etc.) to enable a reader to find the place where you read the quote, or its original source if available. Do not quote with only a name.
# Lighthaven Sequences Reading Group #6 (Tuesday 10/15)

Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then split off into randomized groups for some sequences discussion. Please do the reading beforehand - it should be no more than 20 minutes of reading.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This meetup will also have dinner provided! We'll be ordering pizza-of-the-day from Sliver (including 2 vegan pizzas). Please RSVP to this event so we know how many people to have food for.

We're roughly working through [the sequences highlights](https://www.lesswrong.com/highlights). The mandatory readings this week are

1.  [How An Algorithm Feels From The Inside](https://www.lesswrong.com/s/paoDwasxFpSpzwA2f/p/yA4gF5KrboK2m2Xu7)
2.  [Cluster Structure of Thing Space](https://www.lesswrong.com/posts/WBw8dDkAWohFjWQSk/the-cluster-structure-of-thingspace)
3.  [Superexponential Conceptspace, And Simple Words](https://www.lesswrong.com/posts/82eMd5KLiJ5Z6rTrr/superexponential-conceptspace-and-simple-words)
4.  Optional: [The Categories Were Made For Man, Not Man For The Categories](https://slatestarcodex.com/2014/11/21/the-categories-were-made-for-man-not-man-for-the-categories/)

The posts should take around 15-20 minutes to read. Our optional post has also been pretty influential on the rationalistsphere, and somewhat disagrees with the others.

Doors open 6pm (yes okay we'll let you in early if you get here). The event starts at 6:15 when we'll welcome people, give a few announcements, and then split into groups. Dinner will be served at about 7:30pm, after which point we'll hangout around the fireside as late as we feel like.

You *can* come without having read the essays from the sequences, we do want you to get to join, but then you will have to do a **punishment**. (And then you still have to catch up on reading the essays during the meetup.)

**Some questions to ask yourself about the essays as you read them**

*   What's the most important point in the essay?
*   What's the weakest point in the essay? Or what is the essay wrong about?
*   Can you think of a way to apply the ideas in this essay to your own life?

**For the future**

This is a weekly meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.
# A Triple Decker for Elfland

In 2021 one of the kids in Somerville [started Elfland](https://www.bostonglobe.com/2021/11/24/metro/welcome-elfland-an-8-year-old-somerville-boy-started-building-miniature-village-vacant-lot-then-strangers-secretly-pitched/), a miniature community, in a vacant lot. There had been a gas station there which was [demolished](https://www.reddit.com/r/Somerville/comments/ntjza0/former_somerville_gas_73_summer_st_something/) to [build housing](https://traxboston.com/projects/1-strathmore-2/), but with construction delays there it was open for a while. When construction resumed there were calls to " [defend Elfland](https://orionmagazine.org/article/defend-elfland-somerville-massachusetts/)", and while the original location [closed](https://www.reddit.com/r/Somerville/comments/umqgh0/rip_elfland/) it's now on the Somerville Community Path just west of Willow Ave.

My kids like it a lot, and Anna and I decided to build something for it. Anna wanted to make a house, and I sold her on making a [triple decker](https://en.wikipedia.org/wiki/Three-decker_(house)). These are three-unit buildings, one on each of three floors, that are common in Somerville and other older Boston-area neighborhoods.

We cut some 2x4s to size and glued them up:

[![](https://www.jefftk.com/triple-decker-glued.jpg)](https://www.jefftk.com/triple-decker-glued-big.jpg)

I did the rough sanding with the belt sander, and then Anna did the finish sanding:

[![](https://www.jefftk.com/triple-decker-sanding.jpg)](https://www.jefftk.com/triple-decker-sanding-big.jpg)

We primed and painted together, and Anna put on the doors and windows:

[![](https://www.jefftk.com/triple-decker-painted.jpg)](https://www.jefftk.com/triple-decker-painted-big.jpg)

We brought it over today:

[![](https://www.jefftk.com/triple-decker-with-elfland-mural.jpg)](https://www.jefftk.com/triple-decker-with-elfland-mural-big.jpg)

Here it is, with Somerville's best grocery store and the Elfland Museum of Hopeful Art:

[![](https://www.jefftk.com/triple-decker-with-market-basket.jpg)](https://www.jefftk.com/triple-decker-with-market-basket-big.jpg)

I'm strongly in favor of constructing more housing in Somerville, though we'll need a lot more than this!
# An AI crash is our best bet for restricting AI

If there is a fast take-off, or corporations start earning billions on large models in the next years, we’ll get locked into a trajectory toward extinction.

Now, I don’t think either will happen. I think the AI market will crash in the next few years. But my credence here is beside the point.

My point is that the period after an AI crash would be super high leverage to finally get robust enforceable restrictions in place. Besides some ‘warning shot’ where a badly designed AI system happens to cause or come close to causing a catastrophe, I can’t think of a better window of opportunity.

So even if you think it is _highly unlikely_, preparing for the possibility is worth doing.

As funding dries up and the media turns against AI corporations, their executives get distracted and lose political sway. For a short period, AI Safety and other concerned communities can pack some real punch.

It is the time we can start enforcing all the laws on the books (and put more on the books!) to prevent corporations from recklessly developing and releasing AI models.

Compute limits! Anti-data-scraping! Worker protections! Product liability!

It will be harder to put in place regulations against risks of AGI specifically, because the public will have turned skeptical that AGI could be a thing.

But that’s okay. Enough people are sick of AI corporations and just want to restrict the heck out of them. Environmentalists, the creative industry, exploited workers and whistleblowers, experts fighting deepfakes and disinformation – each has a bone to pick.

There are plenty of robust restrictions we can build consensus around that will make it hard for multi-domain-processing models to get commercially developed.

I’m preparing for this moment.

If you are a funder, keep the possibility of an AI crash in mind. When the time comes, talk with me. Happy to share the information and funding leads I have.
# Open letter to young EAs

In March I published a critique of EA in *Wired*. The responses I got, especially from young EAs, prompted me to write an open letter to my young friends in EA. 

[https://docs.google.com/document/d/1ytfQOfmjWTDiGdjuyNBfWJ\_g-644y\_Pd/edit?usp=sharing&rtpof=true&sd=true](https://docs.google.com/document/d/1ytfQOfmjWTDiGdjuyNBfWJ_g-644y_Pd/edit?usp=sharing&rtpof=true&sd=true)

Thanks for reading.
# Dario Amodei — Machines of Loving Grace

> I think and talk a lot about the risks of powerful AI. The company I’m the CEO of, Anthropic, does a lot of research on how to reduce these risks. Because of this, people sometimes draw the conclusion that I’m a pessimist or “doomer” who thinks AI will be mostly bad or dangerous. I don’t think that at all. In fact, one of my main reasons for focusing on risks is that they’re the only thing standing between us and what I see as a fundamentally positive future. **I think that most people are underestimating just how radical the upside of AI could be**, just as I think most people are underestimating how bad the risks could be.
> 
> In this essay I try to sketch out what that upside might look like—what a world with powerful AI might look like if everything goes *right*. Of course no one can know the future with any certainty or precision, and the effects of powerful AI are likely to be even more unpredictable than past technological changes, so all of this is unavoidably going to consist of guesses. But I am aiming for at least educated and useful guesses, which capture the flavor of what will happen even if most details end up being wrong. I’m including lots of details mainly because I think a concrete vision does more to advance discussion than a highly hedged and abstract one.
# How Should We Use Limited Time to Maximize Long-Term Impact?

I've been reflecting on how researchers—particularly those with limited time or resources—can best contribute to influencing the long-term future.

**Assumption**: Individually, we may not have the resources to fund large projects, but we still want to make contributions that move the needle, however slightly.

I approach this question with epistemic humility—recognizing that predicting long-term impact is incredibly difficult—and with humility about *our own limitations*. I am personally guided by two key principles:

*   **Asymmetry of suffering and joy**: I believe that one conscious being enduring astronomical suffering (assuming that is a coherent concept) can potentially outweigh arbitrary numbers of others experiencing joy. This idea shapes where I think our focus should be.
*   **Alignment challenge**: Aligning AGI/ASI with any prescribed value system will be incredibly difficult. I do not subscribe to the view that human values are a reasonable target, given that our current systems already perpetuate (needlessly) enormous suffering (e.g., factory farming, extreme inequality).

Much has already been impressively written about these topics, and I’m particularly interested in mitigating *s-risks*, since reducing the chance of such suffering seems more urgent than pursuing joy-maximizing goals. However, many of us—whether due to health, time, or other constraints—are left wondering how to allocate our remaining time wisely.

**My personal context**: I have stage 4 cancer, so statistically, my time may be more limited than most. I’m a PhD student in Computer Science with a strong background in math (Masters). Given my situation, I’m especially interested in how those of us with limited time or resources can still make *non-negligible* contributions to reducing *s-risks* and positively shaping the long-term future.

**What I’m asking for**: How should we, as researchers or contributors with limited time, focus our efforts to maximize our positive impact? Are there specific areas of research or actions that seem particularly high-leverage for reducing future suffering on an astronomical scale?

Finally, I’m aware that this topic has been discussed extensively, and I’d be grateful for any *links to relevant resources* or existing discussions that cover this ground. My goal is not to waste anyone's time, as I’m not deeply well-read on the full body of literature. Any guidance, new or old, would be greatly appreciated.
# Binary encoding as a simple explicit construction for superposition

Superposition is the possibility of storing more than $n$ features in an $n$-dimensional vector, by letting the features be slightly correlated with each other. It turns out that one can store exponentially many features in a given vector. The ability to store that many features in a single vector space is sometimes explained using the [Johnson–Lindenstrauss lemma](https://en.wikipedia.org/wiki/Johnson%E2%80%93Lindenstrauss_lemma), but the lemma seems counterintuitive, so I came up with an alternative approach that I found simpler:

Suppose you have a set $F$ with $2^d$ elements and you want to embed it in a $d$-dimensional vector space. We label each element $x_i$ with integers $i$ such that $F = \{x_0, x_1, \dots, x_{2^d - 1}\}$. You can write each integer $i$ as a string of $d$ bits, $b_{d-1} \dots b_2 b_1 b_0$. To improve symmetry, we translate a bit $b$ from being $0$ or $1$ to being $-1$ or $1$ by taking $2b-1$. Join all the digits into a vector and normalize to get the embedding:

$e(x_i) = \frac{1}{\sqrt{d}} [2 b_{d-1} - 1, \dots , 2 b_2 - 1, 2 b_1 - 1, 2 b_0 -1]^\top$

That is, we map each bit to a separate dimension, with a $b=1$ bit mapping to a positive value and a $b=0$ bit mapping to a negative value, and scale the embedding by $\frac{1}{\sqrt{d}}$ to keep the embedding vector of unit length.

If we pick two random elements $x_a$ and $x_b$ of $F$, then an elementary argument shows that their dot product is well-approximated as following a normal distribution $N(0, \frac{1}{\sqrt{d}})$.

In some ways this isn't quite as perfect as the Johnson-Lindenstrauss lemma since you could in principle be unlucky and get two elements that accidentally have a high similarity. After all, for a given element $x_i$, there will be $d$ elements $x_{i \oplus 2^k}$ whose numbers merely differ from $x_i$ by a bitflip. However, it is straightforward to reduce the noise: just concatenate multiple embeddings based on different labels. If instead of using $d$ dimensions, you use $Rd$ dimensions, then you can pump down the noise to $N(0, \frac{1}{\sqrt{Rd}})$.
# Parental Writing Selection Bias

In general I'd like to see a lot more of people writing about their failures in addition to their successes. If a bunch of people all try a thing and have mixed results, and only the people with good results write about it, people who don't know about this selection bias or don't realize its extent are going to end up with overly positive views. I've [written](https://www.jefftk.com/p/failed-vegan-choux-experiments) [about](https://www.jefftk.com/p/loft-bed-fan-guard) [some](https://www.jefftk.com/p/a-strange-ach-corner-case) [of](https://www.jefftk.com/p/failed-adventures-in-delay) [my](https://www.jefftk.com/p/when-i-left-google) [mistakes](https://www.jefftk.com/p/disappointing-table-refinishing), and I think it would be good if this were a higher fraction of my posts.

On the other hand, once other people are involved this isn't entirely up to me. One place this comes up a lot is [parenting](https://www.jefftk.com/news/kids): I don't want to write things about my kids that they don't (or won't) want public. This is especially tricky if I write a post about something we've tried which worked well in part because the kids did a good job with it, and then later they stop doing a good job.

I don't have a good solution here. I don't want to go all the way to "if this had come out with my kids looking bad I wouldn't write about it, so I also won't write about it if they look good" because this would exclude a huge fraction of things involving the kids (there are a tremendous number of possible ways kids could do something that would be embarrassing). Sometimes I can handle it with plausible deniability (one of our kids did embarrassing thing X) but often it would be clear to some people which kid actually did it, or it's bad enough that even being in a pool of three is mortifying. Other times I'm able to include some minor negative information, if it's about them when they were enough younger and it's combined with positive information. But mostly I think this will just need to be something people keep in mind when reading my posts, and posts by other parents.

I asked one of my kids what they thought about this issue and they suggested: "only write about things where \[sibling\] looks bad and I look good, such as \[redacted\]".

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02yZqvgCUv2FQ1JwEKwE5XEFQ5Lauh6TmX456GS4iRr395Xim5rSimbcZFsZu17w1l), [mastodon](https://mastodon.mit.edu/@jefftk/113300444974052819)*
# What are your favorite books or blogs that are out of print, or whose domains have expired (especially if they also aren't on LibGen/Wayback/etc, or on Amazon)? 

Also [posted the question](https://www.reddit.com/r/slatestarcodex/comments/1g2yo2c/what_are_your_favorite_books_or_blogs_that_are/?) to r/slatestarcodex.
# LW resources on childhood experiences?

Does LW have a curated or tagged set of resources on childhood experiences including "trauma"? Personally I'm interested in perspectives from developing world countries (I belong to one), but I'm totally interested in different people coming at this topic from different angles.

I know Yudkowsky has written about his experience for example, so have a couple of other people, but I'm curious if there's a well-organised list or tag or similar.

Reasons to study this on lesswrong should be obvious but I'll list some of the main ones out:

1.  [Social dark matter](https://homosabiens.substack.com/p/social-dark-matter) \- rest of society will study it poorly by default. LW members are sometimes privileged enough and brave enough to discuss such things on the internet, which has vast audience.
2.  Good decision-making, managing motivation and executive function, etc - requires discussing pyschology, which often requires discussing childhood experiences.
3.  Deconfusing your core values -  requires discussing psychology, which often requires discussing childhood experiences.
4.  Coordination - improving human coordination requires understanding each other's core psychological differences and being able to find ways to bridge past them.
5.  Fans - some people love the idea of solving childhood trauma. An idea that has  people who love rather than just like it, is more likely to be worth discussing. Examples: \[[1](https://wildtruth.net/the-briefest-nutshell-of-my-entire-point-of-view/)\] \[[2](https://www.lesswrong.com/posts/vNdYBWhgLsnwd8oZb/what-is-it-like-to-be-psychologically-healthy-podcast-ft)\]
# When is reward ever the optimization target?

Alright, I have a question stemming from TurnTrout's post on Reward is not the optimization target, where he argues that the premises that are required to get to the conclusion of reward being the optimization target are so narrowly applicable as to not apply to future RL AIs as they gain more and more power:

[https://www.lesswrong.com/posts/pdaGN6pQyQarFHXF4/reward-is-not-the-optimization-target#When\_is\_reward\_the\_optimization\_target\_of\_the\_agent_](https://www.lesswrong.com/posts/pdaGN6pQyQarFHXF4/reward-is-not-the-optimization-target#When_is_reward_the_optimization_target_of_the_agent_)

But [@gwern](https://www.lesswrong.com/users/gwern?mention=user) argued with Turntrout that reward is in fact the optimization target for a broad range of RL algorithms:

[https://www.lesswrong.com/posts/ttmmKDTkzuum3fftG/#sdCdLw3ggRxYik385](https://www.lesswrong.com/posts/ttmmKDTkzuum3fftG/#sdCdLw3ggRxYik385)

[https://www.lesswrong.com/posts/nmxzr2zsjNtjaHh7x/actually-othello-gpt-has-a-linear-emergent-world#Tdo7S62iaYwfBCFxL](https://www.lesswrong.com/posts/nmxzr2zsjNtjaHh7x/actually-othello-gpt-has-a-linear-emergent-world#Tdo7S62iaYwfBCFxL)

So my question is are there known results, ideally proofs, but I can accept empirical studies if necessary that show when RL algorithms treat the reward function as an optimization target?

And how narrow is the space of RL algorithms that don't optimize for the reward function?

A good answer will link to results known in the RL literature that are relevant to the question, and give conditions under which a RL agent does or doesn't optimize the reward function.

The best answers will present either finite-time results on RL algorithms optimizing the reward function, or argue that the infinite limit abstraction is a reasonable approximation to the actual reality of RL algorithms.

I'd like to know which RL algorithms optimize the reward, and which do not.
# Rashomon - A newsbetting site

Hello Rationalists!  
  
A while ago I created a concept to evaluate political biases that I call "newsbetting" and made a demo site at [rashomonnews.com](https://www.rashomonnews.com).  Essentially, one receives a news article stripped of identifying information (so stuff like title, author, news source, etc) so that they only get the text of the news article. The reader then places a bet on what they believe the political bias of the news source is - far left, left, center, right or far right. The news source evaluations were taken from [allsides.com](https://www.allsides.com), but the plan was to eventually have users vote on what they believed the bias was for each news source. 

I kind of consider the project closed for the time being, but I wanted to share my results with the community [here](https://nassharaf.github.io/ideasthete/projects/Rashomon.html). There's more data and analysis based off of an initial demo of the site with ~70 users, mostly friends and family. 

I understand that "politics is the mindkiller" but I'm hoping it's obvious why this idea is so suited towards rationalists. This idea was directly based on Robin Hanson's famous quote "We vote on values, we bet on beliefs" and combines or potentially could have combined many mainstay ideas found in rationalist communities - betting markets, cryptocurrencies, bayesian statistics, a preoccupation with quantification and bias. I'm hoping to hear feedback from the community that inspired the idea, nothing more. It's a free to use demo site, there's no actual money involved, it's just a way to quantify you biases. Obviously, it's possible to cheat, but I'm hoping this community approaches it in the spirit it was meant to be used in.   
  
Thanks, hope to hear more from the Bayesian perspective regarding this idea.
# Lighthaven Sequences Reading Group #7 (Tuesday 10/22)

Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then split off into randomized groups for some sequences discussion. Please do the reading beforehand - it should be no more than 20 minutes of reading.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This meetup will also have dinner provided! We'll be ordering pizza-of-the-day from Sliver (including 2 vegan pizzas). Please RSVP to this event so we know how many people to have food for.

We're roughly working through [the sequences highlights](https://www.lesswrong.com/highlights). The mandatory readings this week are

1.  [The Fallacy of Gray](https://www.lesswrong.com/posts/dLJv2CoRCgeC2mPgj/the-fallacy-of-gray)
2.  [Lonely Dissent](https://www.lesswrong.com/posts/CEGnJBHmkcwPTysb7/lonely-dissent)
3.  [Positive Bias: Look Into the Dark](https://www.lesswrong.com/posts/rmAbiEKQDpDnZzcRf/positive-bias-look-into-the-dark)
4.  [Knowing About Biases Can Hurt People](https://www.lesswrong.com/posts/AdYdLP2sRqPMoe8fb/knowing-about-biases-can-hurt-people)

The posts should take around 15-20 minutes to read if you've read them before, and 30-60 mins if it's your first time.

Doors open 6pm (yes we'll let you in early if you get here). The event starts at 6:15 when we'll welcome people, give a few announcements, and then split into groups. Dinner will be served at about 7:30pm, after which point we'll hangout around the fireside as late as we feel like. If you'd like to help us keep dinner going, [you can donate here](https://www.lesswrong.com/donate) (dinner costs about $200 each time).

You *can* come without having read the essays from the sequences, we do want you to get to join, but then you will have to do a **punishment**. (And then you still have to catch up on reading the essays during the meetup.)

**Some questions to ask yourself about the essays as you read them**

*   What's the most important point in the essay?
*   What's the weakest point in the essay? Or what is the essay wrong about?
*   Can you think of a way to apply the ideas in this essay to your own life?

**For the future**

This is a weekly meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.
# Request for advice: Research for Conversational Game Theory for LLMs

Greetings and thank you for your attention. I am an independent researcher and creator of [Conversational Game Theory](https://aikiwiki.com), a novel computational, cognitive, and psychological consensus building game for human to human, AI to human, and AI to AI collaboration. Its notable feature is its ability to reach consensus without voting, simply through conversation, something not previously thought possible. Conflict resolution is a mechanism design of the system itself, and the only possible outcome is a "win-win" of some kind.

This year we engineered and piloted our computational system, and recently we were able to train AI agents on different perspectives to play CGT, build consensus, publish a collaborative article and demonstrated that AI agents trained on GPT score much higher benchmark testing scores that GPT without.

This gives us a capability to create a large Global Library of Consensus Articles, train AI agents on any possible perspective in any possible conflict in the world, publish consensus resolution articles to those conflicts, filter bad faith actors and disinformation/misinformation, and allow humans to join in at anytime.

This global library would serve a purpose as a training ground for LLMs. CGT brings remarkable alignment properties as collaboration is deeply embedded into the mechanism design.

We were not expecting such a profound capability to emerge so quickly, so we are readjusting our focus and seeking advice, advisors, perhaps even some co-founders. We want to continue to roll out research and testing on this capability and any advice anyone has is deeply appreciated.
# The Computational Complexity of Circuit Discovery for Inner Interpretability

Authors: [Federico Adolfi](https://arxiv.org/search/cs?searchtype=author&query=Adolfi,+F), [Martina G. Vilas](https://arxiv.org/search/cs?searchtype=author&query=Vilas,+M+G), [Todd Wareham](https://arxiv.org/search/cs?searchtype=author&query=Wareham,+T).

Abstract:

> Many proposed applications of neural networks in machine learning, cognitive/brain science, and society hinge on the feasibility of inner interpretability via circuit discovery. This calls for empirical and theoretical explorations of viable algorithmic options. Despite advances in the design and testing of heuristics, there are concerns about their scalability and faithfulness at a time when we lack understanding of the complexity properties of the problems they are deployed to solve. To address this, we study circuit discovery with classical and parameterized computational complexity theory: (1) we describe a conceptual scaffolding to reason about circuit finding queries in terms of affordances for description, explanation, prediction and control; (2) we formalize a comprehensive set of queries that capture mechanistic explanation, and propose a formal framework for their analysis; (3) we use it to settle the complexity of many query variants and relaxations of practical interest on multi-layer perceptrons (part of, e.g., transformers). Our findings reveal a challenging complexity landscape. Many queries are intractable (NP-hard, \\sigma\_2^p hard), remain fixed-parameter intractable (W\[1\]-hard) when constraining model/circuit features (e.g., depth), and are inapproximable under additive, multiplicative, and probabilistic approximation schemes. To navigate this landscape, we prove there exist transformations to tackle some of these hard problems (NP- vs. \\sigma\_2^p-complete) with better-understood heuristics, and prove the tractability (PTIME) or fixed-parameter tractability (FPT) of more modest queries which retain useful affordances. This framework allows us to understand the scope and limits of interpretability queries, explore viable options, and compare their resource demands among existing and future architectures.

Seems like bad news for enumerative interp agendas.