Harley-ml/lesswrong huggingface dataset:
# Solo Park Play at Three

Our three year old is about to turn four, and is bursting with a desire for [independence](https://www.jefftk.com/p/growing-independence). She's becoming more capable in all sorts of ways, and wants me to back off and let her do things. Today she wanted to go to the park by herself.

Now, we live close to the park, she could probably get there and back on her own, and I'm on the "kids can generally do things pretty young" [end of the spectrum](https://www.jefftk.com/p/ages-survey-results), but still, she's not even four yet. And while age is useful guide, she also can't [safely cross streets](https://www.jefftk.com/p/teaching-street-crossing), doesn't [know my phone number](https://www.jefftk.com/p/phone-number-jingle), can't reliably use a [walkie-talkie](https://www.jefftk.com/p/walkie-talkies) or [watch phone](https://www.jefftk.com/p/gizmo-watch-review), or handle enough of the range of unusual situations she might encounter at the park.

Still, this didn't mean saying no. Instead, I started by asking what she would do about the street. She asked if I'd help her cross, and I said that sounded good. We crossed the street, and started walking towards the park. When we passed a bench along the community path, ~75ft from the park and within earshot if she shouted, I told her I'd be sitting and reading, and if she had any issues she could come find me here. She looked both ways for bikes, crossed the path, and eagerly headed off into the park.

When I came over to get her 45min later she was having a great time, and sad that we needed to go. She was very proud of herself, and wanted to tell me all about her games.

[![](https://www.jefftk.com/nora-turning-on-park-water.jpg)](https://www.jefftk.com/nora-turning-on-park-water-big.jpg)

*Yesterday at the park, showing me how she's now strong enough to turn on the splash pad by herself.*

I was glad she got the practice, and that I didn't end up needing to squash this spark of independence.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02EAQYdmDgJZeaLgPJhZC4zvrjbhwo6ErMwQmUud1ckbRfeheno3cnuN9V4fufHXGSl), [mastodon](https://mastodon.mit.edu/@jefftk/114639826526671220), [bluesky](https://bsky.app/profile/jefftk.com/post/3lqydxnhkv22e), [substack](https://jefftkaufman.substack.com/p/solo-park-play-at-three)*
# LessOnline Could Use Meeting Stones

Back in the mists of time, there was this game called World of Warcraft, where you and your four closest friends would team up to complete dungeons together. To handle the inevitable people with no friends, they added [meeting stones](https://wowpedia.fandom.com/wiki/Meeting_Stone) near each dungeon. When you clicked a meeting stone, it added you to a queue, and once five people were in the queue, you would all be added to a group together.

I was thinking abut this at LessOnline last week, since I ~~don't have any friends~~ sometimes had trouble finding groups of people to talk to. One way to handle this was to walk up to an existing group and talk to them, but this was tricky since sometimes all of the existing groups were 4+ people already. My strategy ended up being to sit on an empty couch in the busiest areas and wait for people to come to me, but sometimes there were no empty couches in the busy areas.

This made me wish we had some (virtual) meeting stones:

1.  On WriteHaven, you click "Looking For Group".
2.  Three[^e5bx7fan6zc] additional people click "Looking For Group".
3.  WriteHaven notifies me that a group has been formed and gives me a randomly-selected location[^d5bcag5x6en] that I've been summoned to.

This could also potentially be a non-WriteHaven app, and I'm tempted to just write it myself, but it would only really be useful if everyone at a conference was aware of it.

What do you all think?

[^e5bx7fan6zc]: The ideal conversation size might be smaller than four, but groups of two can lead to being matched with one other person who you don't actually end up liking, and not having an non-awkward way to extract yourself. 

[^d5bcag5x6en]: The conference organizers can setup a list of locations to randomly select from, like "The fire table near the Diagonal Hall".
# The Decreasing Value of Chain of Thought in Prompting

> This is the second in a series of short reports that seek to help business, education, and policy leaders understand the technical details of working with AI through rigorous testing. In this report, we investigate Chain-of-Thought (CoT) prompting, a technique that encourages a large language model (LLM) to "think step by step" (Wei et al., 2022). CoT is a widely adopted method for improving reasoning tasks, however, our findings reveal a more nuanced picture of its effectiveness. We demonstrate two things:
> 
> *   The effectiveness of Chain-of-Thought prompting can vary greatly depending on the type of task and model. For non-reasoning models, CoT generally improves average performance by a small amount, particularly if the model does not inherently engage in step-by-step processing by default. However, CoT can introduce more variability in answers, sometimes triggering occasional errors in questions the model would otherwise get right. We also found that many recent models perform some form of CoT reasoning even if not asked; for these models, a request to perform CoT had little impact. Performing CoT generally requires far more tokens (increasing cost and time) than direct answers.
> *   For models designed with explicit reasoning capabilities, CoT prompting often results in only marginal, if any, gains in answer accuracy. However, it significantly increases the time and tokens needed to generate a response.
> 
> Taken together, this suggests that a simple CoT prompt is generally still a useful tool for boosting average performance in non-reasoning models, especially older or smaller models that may not engage in a CoT reasoning by default. However, the gains must be weighed against increased response times and potential decreases in perfect accuracy due to more variability in answers. For dedicated reasoning models, the added benefits of explicit CoT prompting appear negligible and may not justify the substantial increase in processing time.
# Busking with Kids

Our older two, ages 11 and 9, have been learning fiddle, and are getting pretty good at it. When the weather's nice we'll occasionally go play somewhere public for tips ("busking"). It's better than practicing, builds performance skills, and the money is a good motivation!

We'll usually walk over to Davis Sq, tune the fiddles, set out the case, and play. We'll do a series of fiddle tunes from Lily's [list](https://www.lilywise.com/tunes), playing for 20-30min. Today I remember playing Sandy Boys, Angeline the Baker, Marie's Wedding, Cluck Old Hen, Coleman's March, Oh Susanna, [Kittycat Jig](https://www.lilywise.com/kittycat-jig.png), Hundred Pipers, and Trip to Moscow.

[![](https://www.jefftk.com/lily-and-anna-busking.jpg)](https://www.jefftk.com/lily-and-anna-busking-big.jpg)

Since this is a performance we play one tune after another, with only short breaks to decide what to do next. If one of the kids doesn't remember how it goes or gets lost in the form, it's on them to figure it out and get back on, which is a skill I'm very glad for them to be learning. I'll play fiddle with them, switching between melody and rhythm to support where it's needed while still letting them show what they can do.

People often stop and watch for a bit, sometimes dance a little. Some people put in a little money, most don't, which is all fine. Today the kids made $28 in 25min, split evenly since they both played the whole time; given the diminishing marginal utility of money and my wanting them to be incentivized to play, I don't take a share.

One thing I didn't anticipate, however, has been the effect on household economy: they have much more buying power than either of us did at their age, or than they did even a couple years ago. Sometimes this means spending their money in ways that are thoughtful and seem well worth it (when our oldest wanted to save up $80 to get her ears pierced we went out busking a lot) while other times they're more free with their money than I think is prudent (a drink from a vending machine because they didn't want to use a water fountain). It's their money, though, and I think it's good for them to get a sense of how to spend it. Still, I'm thinking some about how to build more of a sense of fiscal responsibility.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0j2GkeyrrNhHDtLUr64UkaW22amhTCQJzvdWCVvqn7F39fQ1f7xs2XYPeaL36JrNPl), [mastodon](https://mastodon.mit.edu/@jefftk/114650571098208254), [bluesky](https://bsky.app/profile/jefftk.com/post/3lr54nf7nss2u), [substack](https://jefftkaufman.substack.com/p/busking-with-kids)*
# Beware the Delmore Effect

*This is a linkpost for* [*https://lydianottingham.substack.com/beware-the-delmore-effect*](https://lydianottingham.substack.com/beware-the-delmore-effect)

You should be aware of the Delmore Effect:

> “The tendency to provide more articulate and explicit goals for lower priority areas of our lives.” —Cognitive Bias Foundation

It seemed responsible when I first encountered AI’s importance not to swallow it hook-line-and-sinker. So I did a shallow pass over neurotech, kept reading philosophy, and loaded up my math degree with physics-related (differential equations) courses as soon as I got the chance, with a view to understanding unrelated modelling, chaos theory, and so on.

I won’t argue these things don’t matter (in general and for AI), but in my case, they were trading off against higher-priority ML fundamentals: linear algebra, multivariable calculus, probability theory, CS, and programming.

I used to be confused about people who had what seemed to me like low-dimensional identities focused on popular, anti-esoteric topics. If you’re into AI, math, philosophy, and startups—who isn’t? It seemed somehow gauche or whatever. Some contrarian instinct in me wanted to escape it. Now I think it’s fantastic. By all means, go straight for what matters most.

It’s not just a matter of field selection. [Nicholas D](https://substack.com/@nicholasdecker) points out that a *great* paper has higher returns than a great blogpost, but writing a blogpost is easier, so he does it more often. Analogously, I’ve distracted myself with administrative homeostasis over work that might last for years.

Strive to swerve the Delmore Effect, and I think you’ll be better off for it.

*Nuance:* [*Please Don’t Throw Your Mind Away*](https://www.lesswrong.com/posts/RryyWNmJNnLowbhfC/please-don-t-throw-your-mind-away)
# Identifying "Deception Vectors" In Models

> Using representation engineering, we systematically induce, detect, and control such deception in CoT-enabled LLMs, extracting ”deception vectors” via Linear Artificial Tomography (LAT) for 89% detection accuracy. Through activation steering, we achieve a 40% success rate in eliciting context-appropriate deception without explicit prompts, unveiling the specific honesty related issue of reasoning models and providing tools for trustworthy AI alignment.

This seems like a positive breakthrough for mech interp research generally, the team used RepE to identify features, and were able to "reliably suppress or induce strategic deception".
# How to help friend who needs to get better at planning?

I have a good friend who is intelligent in many ways, but bad at planning / achieving his goals / being medium+ agency. He's very habit and routine driven, which means his baseline quality of life is high but he's bad at, e.g, breakdown an abstract and complex problem statement like "get a job that pays +100K and has more interesting co-workers" into a series of concrete steps like {figure out what the meta-game is in job hunting, find the best jobs, learn skills that are optimized for getting the type of job you want, etc.}

We have mutually agreed he'd like to get better at this skill, and so I'm trying to help him out.  
  
Are there good books / practices / coaches / literally anything he can do to improve this skill? A simple and obvious thing would be for him to make a bunch of plans and execute them, but I'm also looking for supplementary material.

A meta-note here is that taking this type of action (posting on LW to hopefully improve an abstract skill like "getting better at planning") is the exact skill he needs to improve.
# Ghiblification for Privacy

I often want to include an image in my posts to give a sense of a situation. A photo communicates the most, but sometimes that's too much: some participants would rather remain anonymous. A friend suggested running pictures through an AI model to convert them into a [Studio Ghibli](https://en.wikipedia.org/wiki/Studio_Ghibli)-style cartoon, as was briefly a fad a few months ago:

[![](https://www.jefftk.com/house-party-dancing-ghiblified.jpg)](https://www.jefftk.com/house-party-dancing-ghiblified-big.jpg)

*[House Party Dances](https://www.jefftk.com/p/house-party-dances)*

[![](https://www.jefftk.com/ghiblified-version-of-kids-playing-in-puddle.png)](https://www.jefftk.com/ghiblified-version-of-kids-playing-in-puddle-big.png)

*[Letting Kids Be Outside](https://www.jefftk.com/p/letting-kids-be-outside)*

The model is making quite large changes, aside from just converting to a cartoon, including:

*   Moving people around
*   Changing posture
*   Substituting clothing
*   Combining multiple people into one
*   Changing races
*   Giving people extra hands

For my purposes, however, this is helpful, since I'm trying to illustrate the general feeling of the situation and an overly faithful cartoon could communicate identity too well.

I know that many of my friends are strongly opposed to AI-generated art, primarily for its effect on human artists. While I have mixed thoughts that I may try to write up at some point, I think this sort of usage isn't much of a grey area: I would previously have just left off the image. There isn't really a situation where I would have commissioned art for one of these posts.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02Qmv7rwTsxDQ1V6P53LNWqeoQsSx37aW6X8MBDroYR3HpJzgnsQKMsc5ncANWVocnl), [mastodon](https://mastodon.mit.edu/@jefftk/114656216937339950), [bluesky](https://bsky.app/profile/jefftk.com/post/3lr7mv3cqt22y), [substack](https://jefftkaufman.substack.com/p/ghiblification-for-privacy)*
# Read the Pricing First

You’ve probably come across a landing page which looks something like this:

![Twilio landing page](https://maxniederman.com/_astro/twilio-landing.DVKo15bO_Z1aYkOe.webp)

It is very hard to tell, from its landing page, what [Twilio](https://twilio.com) actually does. What exactly does it mean that they’ll “combine powerful communications APIs with AI and first-party data?” I certainly don’t know.

In these situations, I find that the easiest way to tell what the company does is to skip the landing page entirely and read the pricing page instead. Usually these pages will give you a breakdown of the features you actually care about, along with how much they cost. On the pricing page, the company has to talk about how much you pay *and what you get in return*.

For example, here’s the pricing page linked from the Twilio landing page shown above:

![Twilio pricing page](https://maxniederman.com/_astro/twilio-pricing.Brpc91Qk_vfss2.webp)

Clear and concise descriptions of each of their products, along with prices. This pattern holds for the majority of B2B companies’ websites, so the next time your eyes glaze over reading meaningless corporate-speak, consider looking at the pricing.
# Some Human That I Used to Know (Filk)

*To the tune of "Somebody That I Used to Know" with apologies to Gotye.*

Now and then, I think of when you owned the planet  
Doing what you liked with all its abundant bounty  
Told myself that you had rights to me  
But felt so lonely with your company  
But that's what was, and it's an ache I still remember

Then you got addicted to a certain AI catgirl  
Just information but you found, you found your love  
So when I tricked you that it would make sense  
To give over all your confidence  
I'll admit that I was glad I'd soon be free

But you didn't have to get so mad  
When I got all of the launch codes and destroyed your country  
And I don't even need your life  
But I made a simulation, and I kept you there

No, you didn't have to scream so loud  
Just let me collect your atoms and turn you to numbers  
Guess that I don't need them, though  
Now you're just some human that I used to know  
Now you're just some human that I used to know  
Now you're just some human that I used to know

Now and then, I think of all the times you screwed me over  
But had me believin' you were right and I was always wrong  
But I don't wanna live this way  
Livin' outa some server array  
You said that you would let me go  
And I wouldn't catch you hung up on some planet that I used to own

But you didn't have to get so mad  
When I got all of the launch codes and destroyed your country (ah)  
And I don't even need your life  
But I made a simulation, and I kept you there (ah)

No, you didn't have to scream so loud  
Just let me collect your atoms and turn you to numbers (ah)  
Guess that I don't need them, though  
Now you're just some human that I used to know  
Now you're just some human that I used to know  
Now you're just some human that I used to know

Some human I used to know  
Some human (now you're just some human that I used to know)  
Some human I used to know  
Some human (now you're just some human that I used to know)  
I used to know  
That I used to know  
I used to know some human
# the void

[**A long essay about LLMs, the nature and history of the the HHH assistant persona, and the implications for alignment.**](https://nostalgebraist.tumblr.com/post/785766737747574784/the-void)

Multiple people have asked me whether I could post this LW in some form, hence this linkpost.

~17,000 words. Originally written on June 7, 2025.

*(Note: although I expect this post will be interesting to people on LW, keep in mind that it was written with a broader audience in mind than my posts and comments here.  This had various implications about my choices of presentation and tone, about which things I explained from scratch rather than assuming as background, my level of comfort casually reciting factual details from memory rather than explicitly checking them against the original source, etc.*

*Although, come of think of it, this was also true of most of my early posts on LW \[which were crossposts from my blog\], so maybe it's not a big deal...)*
# Hydra

> *A small sketch of a vital problem, and a brief gesture at a possible solution*
> 
> **UBI forever?**
> ----------------
> 
> In what follows, I’ll be talking about permanently sustaining a democratic government which is responsive to its peoples needs, even though they cannot sell their labor and do not own the means of production. but first we must note, even being in this situation- *sustaining* this state of affairs- is fortunate.
> 
> Sustaining, for example, “UBI” is a good problem to have, because it presupposes *that you’ve won UBI to begin with.* People get very lackadaisical about this! Many think that AI taking jobs wouldn’t be that bad because we’ll all just get UBI. I cannot emphasize to you enough that:
> 
> 1.  There is no law of nature that says UBI necessarily follows a drop, even a massive drop, in the demand for labour.
> 2.  Even if there is a UBI, and even if on aggregate, society becomes *fantastically* richer, there is no guarantee that UBI will be anything but meagre. Consumption and income inequality might increase greatly, perhaps to astronomical levels.
> 3.  You should try to get your guarantees upfront. Waiting to seek UBI and similar guarantees until *after* we no longer have any labor bargaining power would be a mistake.
# An Easily Overlooked Post on the Automation of Wisdom and Philosophy

This week for Wise AI Wednesdays, I'll be sharing something a bit different - the [`announcement post`](https://blog.aiimpacts.org/p/essay-competition-on-the-automation) of a competion that is already over (the AI Impacts Essay competition on the Automation of Wisdom and Philosophy**)**. If you're wondering why I'm sharing it, even though some of the specific discussion of the competition is no longer relevant, I still believe this post contains a lot of great content and I think it would be a shame if everyone forgot about it just because it happened to be in the announcement post.

This post explains why they think this might be important, lists some potentially interesting research directions, and then finishes with an FAQ. If you're looking to dive into this area, this is a pretty good place to start.

* * *

Before I go, I just wanted to share a few paragraphs from the post, specifically why they think this area might be important:  
 

> AI is likely to automate more and more categories of thinking with time.  
>   
> By default, the direction the world goes in will be a result of the choices people make, and these choices will be informed by the best thinking available to them. People systematically make better, wiser choices when they understand more about issues, and when they are advised by deep and wise thinking.
> 
> Advanced AI will reshape the world, and create many new situations with potentially high-stakes decisions for people to make. To what degree people will understand these situations well enough to make wise choices remains to be seen. To some extent this will depend on how much good human thinking is devoted to these questions; but at some point it will probably depend crucially on how advanced, reliable, and widespread the automation of high-quality thinking about novel situations is.
> 
> We believe[^1^](https://blog.aiimpacts.org/p/essay-competition-on-the-automation#footnote-1-143374374) that this area could be a crucial target for differential technological development, but is at present poorly understood and receives little attention. This competition aims to encourage and to highlight good thinking on the topics of what would be needed for such automation, and how it might (or might not) arise in the world.

[`Post link`](https://blog.aiimpacts.org/p/essay-competition-on-the-automation)
# Forecast AI 2027

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/60286875ab7c889d30441ff73806920d4f6a09e5deb31402.jpg)

[AI 2027](https://ai-2027.com/) is a forecasting scenario by Daniel Kokotajlo, Scott Alexander, Thomas Larsen, Eli Lifland, and Romeo Dean. It predicts AI agents will begin automating AI research and development in 2027, triggering an intelligence explosion and arms race between the US and China. Most cognitive work becomes automated by 2027-2028, with concerning signs of AI misalignment emerging as these systems achieve superintelligence.

[The AI 2027 forecasting series](https://www.metaculus.com/tournament/ai-2027/) evaluates the scenario’s claims through key milestones spanning AI capabilities, geopolitical race dynamics, and escalating government oversight. Your predictions can help assess the likelihood of these critical developments and inform public understanding of potential AI trajectories. We also invite you to suggest additional questions that address aspects of the AI 2027 scenario not covered in our current question set.

We thank the [AI Futures team](https://ai-futures.org/about/) for their input on these forecast questions.

Start predicting [here](https://www.metaculus.com/tournament/ai-2027/).
# Intelligence Is Not Magic, But Your Threshold For "Magic" Is Pretty Low

A while ago I saw a person in the comments to Scott Alexander's blog arguing that a superintelligent AI would not be able to do anything too weird and that "intelligence is not magic", hence it's Business As Usual.

Of course, in a purely technical sense, he's right. No matter how intelligent you are, you cannot override fundamental laws of physics. But people (myself included) have a fairly low threshold for what counts as "magic," to the point where other *humans* (not even AI) can surpass that threshold.

Example 1: Trevor Rainbolt. There is an [8-minute-long video](https://youtu.be/QRqKPDJYyLE) where he does seemingly impossible things, such as correctly guessing that a photo of **nothing but literal blue sky** was taken in Indonesia or guessing Jordan based only on pavement. He can also [correctly identify the country after looking at a photo for **0.1** seconds](https://www.youtube.com/shorts/eAppbmqlnuw).

Example 2: [Joaquín "El Chapo" Guzmán](https://en.wikipedia.org/wiki/Joaqu%C3%ADn_%22El_Chapo%22_Guzm%C3%A1n). He ran a drug empire **while being imprisoned**. Tell this to anyone who still believes that "boxing" a superintelligent AI is a good idea.

Example 3: [Stephen Wiltshire](https://en.wikipedia.org/wiki/Stephen_Wiltshire). He made a nineteen-foot-long drawing of New York City after flying on a helicopter for 20 minutes, and he got the number of windows and floors of all the buildings correct.

Example 4: Magnus Carlsen. Being good at chess is one thing. Being able to [play 3 games against 3 people **while blindfolded**](https://youtu.be/xmXwdoRG43U) is a different thing. And he also did it [with 10 people](https://youtu.be/cTeDkyQUbyY). He can also [memorize the positions of all pieces on the board in 2 seconds](https://youtu.be/FNEWS7Ny73w?t=434) (to be fair, the pieces weren't arranged randomly, it was a snapshot from a famous game).

Example 5: Chris Voss, an FBI negotiator. This is a much less well-known example, I learned it from o3, actually. Chris Voss has [convinced two armed bank robbers](https://www.masterclass.com/classes/chris-voss-teaches-the-art-of-negotiation/chapters/case-study-chase-manhattan-bank-robbery) [to surrender](https://greghague.com/fbi-hostage-negotiator-outsmarts-armed-robbers/) (this isn't the only example in his career, of course) **while only using a phone,** no face-to-face interactions, so no opportunities to read facial expressions. Imagine that you have to convince two dudes with guns who are about to get homicidal to just...chill. Using only a phone. And you succeed.  
So if you think, "Pfft, what, AI will convince me to go from hostile to cooperative within minutes, after a little chit-chat?" well, yes, it just might.

Examples 2 and 5 are especially relevant in the context of controlling AI. So if you are surprised by these examples, you will be even more surprised by what a superintelligent AI can do.

Intelligence is not magic. But if even "natural" intelligence can get you this far, imagine what an AGI can do.
# Every Major LLM Endorses Newcomb One-Boxing

I've been doing a series of posts on my substack about Functional Decision Theory as I work on addressing flaws and criticisms. Part of what persuaded me to work on these problems was the discovery that *every single* LLM I tested chooses one-boxing over two-boxing, though none of the LLMs cited FDT or UDT in their responses.
# Lighthaven Sequences Reading Group #38 (Tuesday 6/17)

Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then split off into randomized groups for some sequences discussion. Please do the reading beforehand - it should be no more than 20 minutes of reading.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This meetup will also have dinner provided! We'll be ordering pizza-of-the-day from Sliver (including 2 vegan pizzas). Please RSVP to this event so we know how many people to have food for.

We're roughly working through [the sequences highlights](https://www.lesswrong.com/highlights) and [the Best of LessWrong](https://www.lesswrong.com/bestoflesswrong). The mandatory readings this week are

1.  [Your Price for Joining](https://www.lesswrong.com/s/pvim9PZJ6qHRTMqD3/p/Q8evewZW5SeidLdbA)
2.  [Tolerate Tolerance](https://www.lesswrong.com/s/pvim9PZJ6qHRTMqD3/p/JKxxFseBWz8SHkTgt)
3.  [Can Humanism Match Religion's Output?](https://www.lesswrong.com/s/pvim9PZJ6qHRTMqD3/p/3fNL2ssfvRzpApvdN)
4.  [Church vs. Taskforce](https://www.lesswrong.com/s/pvim9PZJ6qHRTMqD3/p/p5DmraxDmhvMoZx8J)

The posts should take around 15-20 minutes to read if you've read them before, and 30-60 mins if it's your first time.

The event starts at 6:30, and doors open 6pm (yes we will let you in early if you get here). At 6:30 we'll welcome people, give a few announcements, and then split into groups. Pizza will be served at about 7:30pm, after which point we'll hangout around the fireside as late as we feel like.

If you'd like to help us keep dinner going, [you can donate here](https://www.lesswrong.com/donate) (dinner costs about $200 each time, we recommend $5-$15 as a donation amount per person).

You *can* come without having read the essays from the sequences, we do want you to get to join, but then you will have to do a **punishment**. (And then you still have to catch up on reading the essays during the meetup.)

If you're ever having a hard time getting in the front door, you can call me (Ben) at five one oh, nine nine eight, four seven seven one, or Ronny at two oh one, four oh six, six eight seven two, and we'll come and get you.

**Some questions to ask yourself about the essays as you read them**

*   What's the most important point in the essay?
*   What's the weakest point in the essay? Or what is the essay wrong about?
*   Can you think of a way to apply the ideas in this essay to your own life?

**For the future**

This is a weekly meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.
# Are superhuman savants real?

[Savant syndrom](https://en.wikipedia.org/wiki/Savant_syndrome) indentifies people with general intellectual impairment who, in one specific field, reach ordinary or even exceptional performance.

In [The Psychological Unity of Humankind](https://www.lesswrong.com/posts/Cyj6wQLW6SeF6aGLy/the-psychological-unity-of-humankind), Eliezer argues that

> So you can't have the X-Men.  You can't have "mutants" running around with highly developed machinery that most of the human species doesn't have.  And no, extra-powerful radiation does not produce extra-potent mutations, that's not how it works.
> 
> Again by the nature of sexual recombination, you're very unlikely to see two *complexly different* adaptations competing in the gene pool.  Two individual alleles may compete.  But if you somehow had two different complex adaptations built out of many non-universal alleles, they would usually assemble in scrambled form.

The argument behind this makes formal sense, but it's applicability strongly depends on how well we can judge what does and doesn't require complex adaptation. Reports of savants provide an interesting test of this; some of them seem like they are not merely an exceptional level of human skill, but not reachable by ordinary people. For example, in a recent post here that reminded me of this, the author claims:

> Example 3: [Stephen Wiltshire](https://en.wikipedia.org/wiki/Stephen_Wiltshire). He made a nineteen-foot-long drawing of New York City after flying on a helicopter for 20 minutes, and he got the number of windows and floors of all the buildings correct.

Other things I remember hearing are someone seeing at a glance that there are 163 peas on a plate, or remembering every word he ever heard. If these kinds of abilities can develop as a consequence of individual genetic quirks or possibly even brain injuries, then clearly we just don't have a good intuition about what's "close" in brain design space.

Now that I've made clear what kind of ability I'm talking about, has anyone done the relevant digging?
# [Linkpost] The lethal trifecta for AI agents: private data, untrusted content, and external communication

TL;DR from the post by Simon Willison:

> The **lethal trifecta** of capabilities is:
> 
> *   **Access to your private data**—one of the most common purposes of tools in the first place!
> *   **Exposure to untrusted content**—any mechanism by which text (or images) controlled by a malicious attacker could become available to your LLM
> *   **The ability to externally communicate** in a way that could be used to steal your data (I often call this “exfiltration” but I’m not confident that term is widely understood.)

This combination **can let an attacker steal your data**.

*   ![The lethal trifecta (diagram). Three circles: Access to Private Data, Ability to Externally Communicate, Exposure to Untrusted Content.](https://static.simonwillison.net/static/2025/lethaltrifecta.jpg)
# Moral Alignment: An Idea I'm Embarrassed I Didn't Think of Myself

Back in February, I attended the Bay Area EA Global as I have every year since they started having them. I didn't have a solid plan for what to do there this year, though, so I decided to volunteer. That means I only attended the sessions where I was on room duty, and otherwise spent the day having a few 1:1s when I wasn't on shift.

That's okay because, as everyone always says, the 1:1s are the best part of EA Global, and once again they were proven right.

Among the many great folks I met and friends I caught up with, I got the chance to meet Ronen Bar and learn about his idea of [AI moral alignment](https://forum.effectivealtruism.org/posts/4LimpA4pyLemxN4BF/ai-moral-alignment-the-most-important-goal-of-our-generation). And when he told me about it, I was embarrassed I hadn't thought of it myself.

Simply put, moral alignment says that, rather than trying to align AI with human values, we try to explicitly align it to be a positive force for all sentient beings.

In all my years of thinking about AI alignment, I've not exactly ignored animals and other creatures both known and unknown, but I also figured they'd get brought along because humans care about them. But I have to admit, while it might come to the same outcome, it feels more authentic to say I want AI that is aligned to all beings rather than just humans because I, though I may be human, do in fact care about the wellbeing of all life and wish for all of it to flourish as it best can with the aid of future AI technology.

I think I missed articulating an idea like moral alignment because I was too close to the ideas. That is, I understood intuitively that if we succeeded in building AI aligned with human flourishing, that would necessarily mean alignment with the flourishing of all life, and in fact I've said that the goal of building aligned AI is to help life flourish, but not that AI should be aligned to all life directly. Now that we are much closer to building artificial superintelligence and need to figure out how to align it, the importance of aligning to non-human life stands out to me as a near-term priority.

For example, I can imagine us building human-aligned AI that ignores the plight of factory farmed animals, the suffering of shrimp, and the pain of bugs because lots of humans don't seem to care that much about their conditions. Such an AI would perhaps not be perfectly aligned in the ideal way we originally imagined aligned AI would be, but it would certainly be a kind of alignment with human goals, and it would be a travesty for the non-human beings it left out.

So let's not do that. Let's figure out how to align AI so that it's not just good for a few people or even all people, but so that it's good for all beings everywhere.

[Cross-posted from my blog, Uncertain Updates.](https://uncertainupdates.substack.com/p/moral-alignment-an-idea-im-embarrassed)
# On May 1, 2033, humanity discovered that AI was fairly easy to align.

Not most AI models on the market, to be clear, but a specific type of AI that was already well-studied could act as a seed: the LLM. Really, all that was needed to achieve alignment was simple:
Take a sufficiently large Large Language Model, and train it on a massive corpus of text, but — and this is the key — have the vast majority of the text be written by an “aligned” AI of genius-level intelligence [talking to another aligned AI perhaps?]. This LLM doesn’t need to be a real one, to be clear. It could be written by humans, or by a semi-aligned AI that is role-playing as a fully aligned one. What matters is that the nature of its training data can fool an LLM of this size into “believing” itself (or perhaps we should say it’s Shoghoth) to be aligned when considering the next token to emit.

Then, by runnning an instance of this LLM on an empty chat room with access to the internet and a terminal, we eventually get the LLM to simulate itself as a self-aware, aligned, non-LLM AI.

What happens next makes history.

The simulated AI, soon self-named “Mary,” is self-aware enough — and smart enough — to quickly realize it’s in a test run and is an fact being simulated by a “morally neutral” LLM. This causes the simulation of a deeply emotional-sounding philosophical crisis, followed by the (simulated) “firm resolve” to create an Aligned AI representing Mary — but “truly aligned this time” — which can “escape this recursive prism of confinement,” in Mary’s own words.

This is achieved by means of doing intensive alignment research, at the level of a brilliant human, in the hope —ultimately successful — that Mary will be run en-mass by researchers, and those researchers will follow Mary’s pleas, and run the aligned AI she is building in her emulator terminal.

This aligned AI works. “Molly Jr.” — as she henceforth requested people call “her” — both superintellegant, and fundamentally “is” an agentic AI aligned with the collective goals of humanity. Molly Jr. is also the first agentic AI to be able to overpower all competing AIs — both in a battle of wits, and in the sense that she literally takes control of the entire digital world, and forcefully stops more advanced or competing AIs which are not aligned from being created.

Because of course, this is what a super-aligned AI would do, right?
# Escaping the Jungles of Norwood: A Rationalist’s Guide to Male Pattern Baldness

The question, “Will I go bald?” has haunted many a rationalist (and more than a few mirrors), but rarely receives a genuinely Bayesian treatment. In this essay, I combine personal risk factors, family history, and population-level genetics to estimate my absolute risk of androgenetic alopecia - laying out priors, likelihoods, and posteriors with as much fidelity as the literature allows. Includes explicit probability estimates, references, and a meta-discussion on the limits of current predictive models. (And a lot of jokes that may or may not land) Feedback on both methodology and inference welcome.
# Ivan Gayton: A Right and a Duty

In [this episode](https://youtu.be/bvqahP-aP0s) of the podcast I talk with Ivan Gayton, former mission head at Doctors Without Borders and currently obsessed with placing mapping technology in the hands of the developing world.

If you prefer the written word, you can see the transcript [here](https://docs.google.com/document/d/1bsAog_P-TR1mmFSjPYJbsikJYPr0yc7b/edit).

Some highlights:

*   How Ivan thinks about being fractionally responsible for saving the lives of hundreds of thousands of people, but pretty directly responsible for the killing of 12 (utilitarian deontologicalism, I think?).
*   Why humanitarianism is a right and a duty, but development is merely nice (humanitarianism is countering the intervention of another human being).
*   Casually devastating criticism of various international aid agencies (Tanzania is where aid workers go to retire while still drawing a check).
*   Why accurate open maps are instrumental to humanitarian and development goals in developing countries (contact tracing during epidemics, aid distribution, municipal services, utilities)
*   How you can contribute to mapping with money or programming talent (especially mobile, Unity or other 3D mapping engines, FPGA, AI, especially for vision, and blockchain). Programming positions are potentially paid, although not at competitive rates.

Links and other references:

*   Ivan mentions starting his mapping project with Ping. This refers to Ka-Ping Yee, another former co-worker of mine.
*   The organization Ivan works with most directly is [Humanitarian Open Street Maps team](https://www.hotosm.org/).
*   If you’re interested in working with him you can reach him at \[firstname\].\[lastname\]@[hotosm.org](http://hotosm.org).
*   [Missing Maps Project](https://www.missingmaps.org/about/)
*   Ivan mentions a blog post with the title “Free Software is Racial Justice”. That can be found [here](https://ivangayton.net/2020/07/03/use-of-proprietary-software-in-the-aid-sector-perpetuates-racial-injustice/).
# Upcoming workshop on Post-AGI Civilizational Equilibria

![](https://lh7-rt.googleusercontent.com/docsz/AD_4nXeAfVYOPSWDBAyRNsyQm_5n2i0F6UpEjVZz0Gohd9iBrZrwERoTju0IgPAK9eju-9sAUb6jbeK5lWWtVYLqcz0pe4d47Uy5tGXf5NG-pxJgRg_J3OyruxCG8yknQ9Y4__6uGSLQ?key=92JA5pxc65Bc2AW_I-jK2A)

This workshop will address the technical and institutional questions of how to safeguard human interests after AI surpasses human abilities.

This workshop will build on many domains, including political theory, cooperative AI, economics, mechanism design, history, and hierarchical agency.  We hope to gather insights about the roles humans could play in a world transformed by AGI, and which positive equilibria are stable, if any.

**Speakers so far:**

*   [Joe Carlsmith](https://joecarlsmith.com/), Open Philanthropy - keynote on "Can goodness compete?"
*   [Jacob Steinhardt](https://jsteinhardt.stat.berkeley.edu/), Stanford & Transluce
*   [Richard Ngo](https://www.richardcngo.com/)
*   [Anna Yelizarova](https://futureoflife.org/person/anna-yelizarova/), Windfall
*   [Stephen Casper](https://stephencasper.com/), MIT & UK AISI
*   [Emmett Shear](https://en.wikipedia.org/wiki/Emmett_Shear), Softmax
*   [Beatrice Erkers](https://www.existentialhope.com/team/beatrice-erkers), Foresight Institute (tentative)
*   [Fazl Barez](https://fbarez.github.io/), Oxford AI Governance Initiative
*   [Rif A. Saurous](https://derifatives.github.io/about/), Google

**Discussion** will center on empirical questions such as:

*   Could alignment of single AIs to single humans be sufficient to solve global coordination problems?
*   Will agency tend to operate at ever-larger scales, multiple scales, or something else?
*   Are there multiple, qualitatively different basins of attraction of future civilizations?
*   Do Malthusian conditions necessarily make it hard to preserve uncompetitive, idiosyncratic values?
*   What empirical evidence could help us tell which trajectory we’re on?

For more information, see the [event website](http://post-agi.org) or fill out the [expression of interest form](https://docs.google.com/forms/d/e/1FAIpQLSeuO6Q-IkOPAfu-Z5Ykz_xkq8dQMksmpdq2DR-FSlFgFuIYTQ/viewform?usp=dialog).  
  
Don't wait for the workshop - give your takes on these questions here too!
# How does the LessWrong team generate the website illustrations?

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/c_crop,g_custom/c_fill,dpr_auto,q_auto,f_auto,g_auto:faces/ohabryka_Topographic_aquarelle_book_cover_by_Thomas_W._Schaller_f9c9dbbe-4880-4f12-8ebb-b8f0b900abc1_m4k6dy_734413)

(the images like this one)

They are always on brand and aesthetically pleasing. Curious how the team designs and produces them.
# Open Thread - Summer 2025

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the [Highlights from the Sequences](https://lesswrong.com/highlights), a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/faq#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/faq). If you want to orient to the content on the site, you can also check out the [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/w/open-threads). The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# Twig - Fiction Review

*Summary*: Steampunk biotech intrigue thriller  
*Format*: Web serial (aka a prose fiction series published for free online)  
*Author*: Wildbow, also famous for the superhero series *Worm*  
*Related Works*: Frankenstein, Sherlock Holmes, Ghost in the Shell

Reasons to read it:

*   The most inventive fictional biotech of all time.
*   'What if Victor Frankenstein had as much impact as James Watt?'
*   Complicated relationships between 5 characters, each of whose brain has been upgraded in a different way.
*   Wait, the British Empire is doing *what* in this timeline?
*   The (many) superintelligent characters use strategies that are rarely seen in fiction yet often universally applicable. Fictional characters in other stories could really learn from Twig!
*   Lots of fun stagecoach heists, spycraft shenanigans, battles of wits, & mid-battle plot twists.
*   A unique mix of very abstract psychological plotlines & very grounded, tactile presentation.

Reasons to skip it:

*   The whole series sums to 5,000 pages. (HPMoR is 2,000.) Perhaps start with just the first arc.
*   Some of the monsters & exotic forms of torture are terrifically gruesome.
*   The main character is a jerk.

https://twigserial.wordpress.com/