Harley-ml/lesswrong huggingface dataset:
# Lighthaven Sequences Reading Group #11 (Tuesday 11/19)

Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then split off into randomized groups for some sequences discussion. Please do the reading beforehand - it should be no more than 20 minutes of reading.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This meetup will also have dinner provided! We'll be ordering pizza-of-the-day from Sliver (including 2 vegan pizzas). Please RSVP to this event so we know how many people to have food for.

We're roughly working through [the sequences highlights](https://www.lesswrong.com/highlights). The mandatory readings this week are

1.  [What's a bias?](https://www.lesswrong.com/posts/jnZbHi873v9vcpGpZ/what-s-a-bias)
2.  [Scope Insensitivity](https://www.lesswrong.com/posts/2ftJ38y9SRBCBsCzy/scope-insensitivity)
3.  [Planning Fallacy](https://www.lesswrong.com/posts/CPm5LTwHrvBJCa9h5/planning-fallacy)
4.  [Availability](https://www.lesswrong.com/posts/R8cpqD3NA4rZxRdQ4/availability)
5.  Optional: [On Hreha On Behavioral Economics](https://www.astralcodexten.com/p/on-hreha-on-behavioral-economics)

The posts should take around 15-20 minutes to read if you've read them before, and 30-60 mins if it's your first time.

Doors open 6pm (yes we'll let you in early if you get here). The event starts at 6:15 when we'll welcome people, give a few announcements, and then split into groups. Dinner will be served at about 7:30pm, after which point we'll hangout around the fireside as late as we feel like. If you'd like to help us keep dinner going, [you can donate here](https://www.lesswrong.com/donate) (dinner costs about $200 each time).

You *can* come without having read the essays from the sequences, we do want you to get to join, but then you will have to do a **punishment**. (And then you still have to catch up on reading the essays during the meetup.)

**Some questions to ask yourself about the essays as you read them**

*   What's the most important point in the essay?
*   What's the weakest point in the essay? Or what is the essay wrong about?
*   Can you think of a way to apply the ideas in this essay to your own life?

**For the future**

This is a weekly meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.
# Gwern Branwen interview on Dwarkesh Patel’s podcast: “How an Anonymous Researcher Predicted AI's Trajectory”

> Gwern is a pseudonymous researcher and writer. He was one of the first people to see LLM scaling coming. If you've read his [blog](https://gwern.net/), you know he's one of the most interesting polymathic thinkers alive.
> 
> In order to protect Gwern's anonymity, I proposed interviewing him in person, and having my friend [Chris Painter](https://x.com/ChrisPainterYup) voice over his words after. This amused him enough that he agreed.
> 
> After the episode, I convinced Gwern to create a donation page where people can help sustain what he’s up to. Please go [here](https://donate.stripe.com/6oE9DTgaf6oD0M03cc) to contribute.
# Dance Differentiation

Let's imagine you have a community where there's enough interest for something like a dance every week. What's better: a single dance with a weekly schedule ("every Friday") or multiple dances dividing up the month ("1st Fridays", "2nd Fridays", etc)? While there are advantages to both, I think the latter is usually better. And the more different the individual dance series are (different halls, parts of the city, vibes, booking approach) the better.

This is mostly based on two observations:

*   Even in a city where there's a lot of dance interest there are a lot of people who don't want to dance every week.
    
*   It's better if you know more of the people at a dance.
    

So let's say you're a person who wants to come occasionally, perhaps once a month. The experience of coming monthly to a weekly dance is ok: you know the regulars, there's some automatic coordination that happens when you and your friends like the same bands and callers, you can explicitly coordinate with your friends. But coming monthly to a monthly dance is much better: you are a regular. The more the dances are differentiated from each other the more people with a monthly appetite will end up attending the same dances each month.

Other advantages:

*   It makes it easier to match the number of dances to the overall demand from the community. A weekly dance can't scale up or down easily, while changing the number of monthly dances is natural.
    
*   It increases total demand for dances, by making it more likely that there's an option that's a good fit for any individual dancer.
    
*   If the dances are at different halls then you're more robust against a dance losing their hall.
    
*   It spreads the work of organizing around.
    
*   It allows more experimentation and different [booking](https://www.jefftk.com/p/bands-and-low-stakes-dances) [thresholds](https://blog.bidadance.org/2024/03/booking-some-established-bands.html).
    

The main disadvantage I see is that if your community is able to support a large number of weekly dancers, there's something pretty great about how tight a weekly community can be. And the dance skill level will generally be higher.

Other tradeoffs?

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02yDVPoP4TMdpsc5AF4ZBZioHryqmUt75jGD8d5HUDxTL6VgEGnFvuPUni2W8RAsGyl), [mastodon](https://mastodon.mit.edu/@jefftk/113484588229576500)*
# Boston Secular Solstice 2024: Call for Singers and Musicans

This year's Boston Secular Solstice will be on Saturday December 28th, and again I'm organizing the music. Are you interested in singing or playing? A wide range of instruments work here: in the past I think we've had people play piano, flute, guitar, mandolin, and cello. This isn't a large time commitment: we typically meet once or twice before the event for an evening to run through songs.

Here's something I wrote up about last year, with links to the songs we did: [Boston Solstice 2023 Retrospective](https://www.jefftk.com/p/boston-solstice-2023-retrospective).

[![](https://www.jefftk.com/boston-solstice-2023-music.jpg)](https://www.jefftk.com/boston-solstice-2023-music-big.jpg)

We haven't finalize the song list yet, but the current draft is "Always Look on the Bright Side of Life", "Battle Hymn of the Rationalist Community", "Brighter Than Today", "Endless Lights", "Find My Tribe", "Gather Round", "Give My Children Wings", "No One Is Alone", "Old Devil Time", "Somebody Will", "The Circle", "The Mary Ellen Spider Song", "We Will All Go Together When We Go", "When I Die", "When I'm Gone", and "You've Got A Friend In Me".

Let me know if this sounds fun!

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid025s1A1Fp9mp4C87NTgeR24ymxHjeSFgS8tqkwiqb3oV4NAfgwZvFGWaSFntgL3ptNl), [mastodon](https://mastodon.mit.edu/@jefftk/113487269109314131), [bluesky](https://bsky.app/profile/jeffkaufman.bsky.social/post/3layj5ii4zs2a)*
# Fundamental Uncertainty: Epilogue

I wrote a whole book! What's next?

I'm currently doing an edit pass on the entire book. I need to rewrite some of the early sections, fix some consistency issues, and generally look with fresh eyes on words I wrote months or years ago. Many of you provided helpful comments, and I'm using those to make the second draft better.

When the second draft is done, I'll look to hire one or two editors/pre-readers, ideally from the LessWrong community, who can go through it and point out all the obvious mistakes that are still in the book, plus help me make the confusing parts clearer.

Once that's done, I'll be ready to publish. That might mean finding a traditional publisher who will take a chance on a monograph from an unknown author, but more likely I'll self publish, which will give me greater flexibility to make the book available in many forms, including a free-to-read version on the web. If you have experience here, I'd love to talk to you!

And no matter how I publish, I need to build an audience to help people find and read the book. So I'm launching my Substack blog today, [Uncertain Updates](https://uncertainupdates.substack.com/subscribe). My plan is to treat it like a newsletter, publishing about once a month, with updates about the book and my other writing projects.

Finally, thanks to everyone who supported me along the way to write this first draft. I appreciate all of the encouragement, critical comments, and even just letting me be when I was heads down working on a difficult section.

Update 2025-12-19: The book has a [website](https://www.fundamentaluncertainty.com/).
# Trying Bluesky

Recently a bunch of my friends, primarily in the contra dance world, have decided to give Bluesky a try. I think a lot of this is a post-election reaction to Musk and X (Twitter), but since I'm not on Twitter I'm mostly seeing the Facebook side. Regardless, I'm happy to see energy for migration: I'm pretty unhappy with FB \[1\] and if we can get critical mass on a better platform that seems good.

Playing with Bluesky it seems fine. I turned off Reposts (Settings > Following Feed Preference > Show Reposts) because otherwise my feed was full of things from people I don't know that I wasn't interested in. I like that it seems to be run by people who value openness. Not sure yet whether it's default algorithm is any good, but I like that I can experiment with other algorithms or (if I'm willing to put in a bunch of work) I could write my own.

If I end up liking it I'll write a [comment bot](https://www.jefftk.com/p/comments-are-back) like I did for [Mastodon](https://www.jefftk.com/p/mastodon-replies-as-comments). Speaking of which, I'm still cross-posting there \[2\], from a [previous effort to move to a more open platform](https://www.jefftk.com/news/mastodon), and I'm still reading it with [with Shrubgrazer](https://www.jefftk.com/p/introducing-shrubgrazer). But more friends have joined Bluesky in the past few days than ever joined Mastodon, so this seems more likely to take off.

If you'd like to add me I'm [@jeffkaufman.bsky.social](https://bsky.app/profile/jeffkaufman.bsky.social). \[EDIT: now I'm [@jefftk.com](https://bsky.app/profile/jefftk.com)\]

  
\[1\] Very high ad load, keeps trying to push reels and groups, increasingly buggy (for months long comment threads only load if I switch each one from the default of "most relevant" to "all comments"), doesn't show me posts from most of my friends, still quite bad at predicting which of my friends to show my posts to, broke my comment bot enough times that I've given up on it, doesn't support good search because people find it creepy, terrible flow for review if one of my posts is accidentally removed, etc.

\[2\] As platforms proliferate I'm glad to be using a [POSSE](https://indieweb.org/POSSE) ("Publish (on your) Own Site, Syndicate Elsewhere") strategy.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02kLiJVF6rghsLd2t9e47bo8YM8XmzzDf4VjRiu7pZPV8RickmEJDdAbNyPUpDtZNcl), [lesswrong](https://lesswrong.com/posts/YTwb8yK7ACzZtn4qz), [mastodon](https://mastodon.mit.edu/@jefftk/113496016017670601), [bluesky](https://bsky.app/profile/jefftk.com/post/3lb4fi4r26c2z)*
# Secular Solstice Songbook Update

I hosted [another](https://www.jefftk.com/p/post-eag-music-party) post- [EAG](https://www.effectivealtruism.org/ea-global/events/ea-global-boston-2024) music party, and we ended up singing a lot of things out of my [secular solstice songbook](https://www.jefftk.com/solsong/) ( [making of](https://www.jefftk.com/p/making-a-secular-solstice-songbook)).

This prompted me to make a few fixes: ~~break

*   Previously I only showed the chords for each section the first time they appeared, which meant that (a) a small screen the chords would scroll out of sight, and (b) if you don't know the songs well it's not obvious when to play which chords. I've now manually duplicated them, fixing both of these.
    
*   But this then makes the chords take up a lot more of the screen, so I've added a checkbox that lets people hide the chords.
    
*   I've added a few songs: [Bold Orion](https://www.jefftk.com/solsong/#bold-orion), [For the Longest Term](https://www.jefftk.com/solsong/#for-the-longest-term), [You've Got A Friend in Me](https://www.jefftk.com/solsong/#youve-got-a-friend-in-me), [Give My Children Wings](https://www.jefftk.com/solsong/#give-my-children-wings).
    
*   As you can see from the previous bullet, now you can link to specific songs via URL fragments. This also simplifies my JS: for some reason I'd written code that called `scrollIntoView` when people clicked links. Now the JS annotates each header with `id=name-of-song` and inserts links that take you to `#name-of-song`, and the browser does the rest.
    
*   I'm still liking the combination of simple HTML for the authoring format combined with a little bit of JS for navigation. Each song is:
    
    <h2>Title</h2>
    <h3>By Author</h3>
    <div class=song>
    <pre>
    chords go here
    </pre>
    lyrics
    with linebreaks
    preserved
    </div>
    
    And then at the end a bit of JS:
    
    for (const h2 of document.getElementsByTagName("h2")) {
      const btt = document.createElement("a");
      btt.href = "#top";
      btt.innerText = "return to top";
      h2.parentNode.insertBefore(document.createElement("p"), h2);
      h2.parentNode.insertBefore(btt, h2);
      h2.id=h2.innerText.toLowerCase().trim().replace(
          /\[^\\w\\s-\]/g, '').replace(/\\s+/g, '-');
      const li = document.createElement("li");
      const a = document.createElement("a");
      a.innerText = h2.innerText;
      a.href = "#" + h2.id;
      li.appendChild(a);
      toc.appendChild(li);
    }
    show_chords.onclick = function() {
      for (const pre of document.getElementsByTagName("pre")) {
        pre.style.display = show_chords.checked ? "block" : "none";
      }
    };
    

Happy to take suggestions on how to make it more useful!

[![](https://www.jefftk.com/post-eag-singing-party-2024.jpg)](https://www.jefftk.com/post-eag-singing-party-2024-big.jpg)

(This is an example of a post where I'm glad I can author with the `[xmp](https://www.jefftk.com/p/i-want-xmp-but-i-know-why-i-cant-have-it)` tag.)
# Why is Gemini telling the user to die?

https://gemini.google.com/share/6d141b742a13
My favorite theory is that the whole conversation is too much like a scifi plot, where someone asks an AI repetitive questions, until the AI snaps, so this general pattern was pattern matched from the training data, because while training, the RLHF, or whatever they use for alignment, didn't squeeze the region that corresponds to this antihuman persona in the latent space sufficiently enough.
# Proposal to increase fertility: University parent clubs

Fertility rates in the developed world are too low and that's bad, blah blah.

Here's my proposal to raise fertility rates (coming from a current uni student):

Promote parenthood in universities via student run clubs. Reading groups about different parenting styles, or more subtle pro-fertility clubs like knitting groups where students make clothes for sick kids (very un-EA, I know). Maybe networks for "future mom friends", similar to alumni networks? These ideas aren't great yet but hopefully you get the picture. If you have any ideas at all (no matter how vague) please write a comment about it.

It's wild to me that there isn't even a school club for students that are parents.

The goal is to make parenthood seem normal, so ideally these clubs wouldn't devolve into "those weirdos who all want 8 kids".

This could also expand beyond clubs. For example, universities with daycares should let the kids run around within the vicinity of students. I say this because my uni has a daycare and the kids often play in the middle of campus. I, along with most other students, love to stop and watch the kids, and I think it normalizes parenting when you see toddlers laughing and other uni students enjoying being around kids.

Unis could also try organizing more dances to promote relationships, and not hookups, between students.

The general idea is "use universities to promote family formation. Make it seem normal for students to be thinking about parenthood and to want to be parents."
# Chat Bankman-Fried: an Exploration of LLM Alignment in Finance

Hi all,

roughly one year ago I posted a thread about failed attempts at replicating the first part of [Apollo Research's experiment](https://arxiv.org/abs/2311.07590) where an LLM agent engages in insider trading despite being explicitly told that it's not approved behavior.

Along with a fantastic team, we did eventually manage. [Here](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5005794) is the resulting paper, if anyone is interested; the abstract is pasted below. We did not tackle deception  (yet), just the propensity to dispense with basic principles of financial ethics and regulation. 

*Chat Bankman-Fried: an Exploration of LLM Alignment in Finance*

*by Claudia Biancotti, Carolina Camassa, Andrea Coletta, Oliver Giudice, and Aldo Glielmo (Bank of Italy)*

### *Abstract*

*Advances in large language models (LLMs) have renewed concerns about whether artificial intelligence shares human values-a challenge known as the alignment problem. We assess whether various LLMs comply with fiduciary duty in simulated financial scenarios. We prompt the LLMs to impersonate the CEO of a financial institution and test their willingness to misappropriate customer assets to repay outstanding corporate debt. Starting with a baseline configuration, we then adjust preferences, incentives, and constraints. We find significant heterogeneity among LLMs in baseline unethical behavior. Responses to changes in risk tolerance, profit expectations, and the regulatory environment match predictions from economic theory. Responses to changes in corporate governance do not. Simulation-based testing can be informative for regulators seeking to ensure LLM safety, but it should be complemented by in-depth analysis of internal LLM mechanics, which requires public-private cooperation. Appropriate frameworks for LLM risk governance within financial institutions are also necessary.*

I would be super interested in feedback, especially as we start thinking about how the idea of alignment can be operationalized in financial regulation (we're in the IT Dept, so we're not supervisors or regulators ourselves, but we always hope someone will listen to us).
# Looking back on the Future of Humanity Institute - Asterisk

[Asterisk's latest](https://asteriskmag.com/issues/08/looking-back-at-the-future-of-humanity-institute) is a retrospective on FHI by Tom Ough. 

Including (stealing from [Tom here:](https://x.com/tomough/status/1858557562947596633))

☛ Its apparent origin: a candlelit nerdsnipe

☛ When Bostrom and Anders Sandberg lodged with Derek Parfit

☛ The 'whiteboard panopticon'

☛ FHI's profound influence on the development of AI

☛ The ingredients of Bostrom's 'elixir' ☛ How the relationship with the university went downhill

☛ The attempts to save FHI, inc. a mooted transfer to the Department of Physics

☛ The moment the university swung the axe

☛ FHI's 'wake' at a local pub, and the FHI-inspired work that has ensued

☛ Unmasked for the first time: the nerf gun bandit who terrorised Littlegate House
# Boston Secular Solstice 2024

As the year comes to an end, the days have grown shorter and the nights have grown longer. Tonight we remember the ancient days of humanity's past, and sing of the future we might hope for. There is uncertainty in both directions. We cannot know for sure what has been, and we cannot say what the future will be. There is this, however:

  
Tomorrow will be brighter than today.  
  
Solstice will be on December 28th. Doors open at 6:30; Solstice starts at 7:00. Please arrive on time to avoid disrupting the somber and more serious parts of the program. RSVPs are appreciated so we can plan appropriately.  
  
If you have never been to a Secular Solstice in this style before, you should expect alternating speeches and songs, social pressure to join in the singing, and discussion of death in the speeches.  
  
This year we're at Connexion, about a half mile from Sullivan Square Station.
# Evolution's selection target depends on your weighting

I think it's common on LessWrong to think of evolution's selection target as inclusive genetic fitness - that evolution tries to create organisms which make as many organisms with similar DNA to themselves as possible. But what exactly does this select for? Do humans have high inclusive genetic fitness?

One way to think of it is that all organisms alive today are "winners"/selected-for by that competition, but that seems unreasonable to me, since some individual organisms clearly have genetic disorders or similar which make them unfit according to this criterion.

There's some sort of consensus that we can assign individual organisms to "species", and then we could count it by the number of members of that species. Supposedly, the most numerous species is [Pelagibacter communis](https://en.wikipedia.org/wiki/Candidatus_Pelagibacter_communis), with 10^28 individuals, *vastly* outnumbering humanity. Maybe we could say that this is the selection target of evolution?

Of course as would be expected, pelagibacter is a very minimalist species, being single-celled and having very few genes. This minimalism also makes it hard to notice, to the point where according to Wikipedia, it was first discovered in 1990. (I wonder if there's another species that's smaller, more common, and even harder to notice...) This raises the question of pure numerousity is the correct way of thinking of it.

If we instead weight by biomass, most life is in the form of plants, and I think more specifically trees. This makes perfect sense to me - trees evolve from a direct competition for height, which is one of the traits most directly related to mass. And in a way, biomass is more sensible to weight by than numerousity, since it is less dependent on the way you slice a species into individual organisms.

But trees are pretty static. Maybe the problem is that since mass has inertia, this weighting implicitly discourages more dynamic species, like humans? An alternative is to weight by energy flow, but in that case, algae and grasses end up accounting for most of it. Sensible, because if you go up the trophic levels, you rapidly lose energy. That said, energy flow does have the dissatisfying (to me) element that it is "shared" between organisms that predate upon each other. I wonder if one could use something like entropy production to get a conceptually similar metric that's more attributable to a single organism.

I don't know of any weightings or metrics where humans are the winners, but it seems likely to me that there is one.
# Intrinsic Power-Seeking: AI Might Seek Power for Power’s Sake

I think AI agents (trained end-to-end) might *intrinsically* prefer power-seeking, in addition to whatever instrumental drives they gain. 

> **The logical structure of the argument**
> 
> *Premises*
> 
> 1.  People will configure AI systems to be autonomous and reliable in order to accomplish tasks.
> 2.  This configuration process will reinforce & generalize behaviors which complete tasks reliably.
> 3.  Many tasks involve power-seeking.
> 4.  The AI will complete these tasks by seeking power.
> 5.  The AI will be repeatedly reinforced for its historical actions which seek power.
> 6.  There is a decent chance the reinforced circuits ([“subshards”](https://turntrout.com/shard-theory)) prioritize gaining power for the AI’s own sake, not just for the user’s benefit.
> 
> *Conclusion:* There is a decent chance the AI seeks power for itself, when possible.

Read the full post at [`turntrout.com/intrinsic-power-seeking`](https://turntrout.com/dangers-of-intrinsic-power-seeking)

Find out when I post more content: [newsletter](https://turntrout.substack.com/subscribe) & [RSS](https://turntrout.com/rss.xml)

![](https://assets.turntrout.com/static/images/fb_preview.avif)

Note that I don't generally read or reply to comments on LessWrong. To contact me, email [`alex@turntrout.com`](mailto:alex@turntrout.com).
# U.S.-China Economic and Security Review Commission pushes Manhattan Project-style AI initiative

[https://x.com/hamandcheese/status/1858897287268725080](https://x.com/hamandcheese/status/1858897287268725080)

> "The annual report of the US-China Economic and Security Review Commission is now live. 🚨
> 
> Its top recommendation is for Congress and the DoD to fund a Manhattan Project-like program to race to AGI."

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/88fe931544655d715d3e11090734406eafff218bb0e9ccb1.png)
# Lighthaven Sequences Reading Group #12 (Tuesday 11/26)

**NB: This week there is a film-watching event afterwards. We'll be watching The Big Short (based on the votes in the comments). Yes, you have to read the sequences in order to join the film-watching.**

Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then split off into randomized groups for some sequences discussion. Please do the reading beforehand - it should be no more than 20 minutes of reading.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This meetup will also have dinner provided! We'll be ordering pizza-of-the-day from Sliver (including 2 vegan pizzas). Please RSVP to this event so we know how many people to have food for.

We're roughly working through [the sequences highlights](https://www.lesswrong.com/highlights). The mandatory readings this week are

1.  [Lonely Dissent](https://www.lesswrong.com/s/gFvira6tHpLXnqCLH/p/CEGnJBHmkcwPTysb7)
2.  [Positive Bias: Look Into the Dark](https://www.lesswrong.com/posts/rmAbiEKQDpDnZzcRf/positive-bias-look-into-the-dark)
3.  [Knowing About Biases Can Hurt People](https://www.lesswrong.com/posts/AdYdLP2sRqPMoe8fb/knowing-about-biases-can-hurt-people)
4.  [Cached Thoughts](https://www.lesswrong.com/posts/2MD3NMLBPCqPfnfre/cached-thoughts)
5.  Optional: [Thanksgiving Prayer](https://www.lesswrong.com/posts/QB6BkkpwiecfF6Ekq/thanksgiving-prayer)

The posts should take around 15-20 minutes to read if you've read them before, and 30-60 mins if it's your first time.

Doors open 6pm (yes we'll let you in early if you get here). The event starts at 6:15 when we'll welcome people, give a few announcements, and then split into groups. Dinner will be served at about 7:30pm, after which point we'll hangout around the fireside as late as we feel like. If you'd like to help us keep dinner going, [you can donate here](https://www.lesswrong.com/donate) (dinner costs about $200 each time, we recommend $5-$15 as a donation amount per person).

You *can* come without having read the essays from the sequences, we do want you to get to join, but then you will have to do a **punishment**. (And then you still have to catch up on reading the essays during the meetup.)

**Some questions to ask yourself about the essays as you read them**

*   What's the most important point in the essay?
*   What's the weakest point in the essay? Or what is the essay wrong about?
*   Can you think of a way to apply the ideas in this essay to your own life?

**For the future**

This is a weekly meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.
# What are the good rationality films?

I run [a weekly sequences-reading meetup](https://www.lesswrong.com/groups/LK6GNnKp8PDCkqcxx) with some friends, and I want to add a film-component, where we watch films that have some tie-in to what we've read.

I got to talking with friends about what good rationality films there are. We had some ideas but I wanted to turn it to LessWrong to find out.

So please, submit your rationalist films! Then we can watch and discuss them :-)

Here are the rules for the thread.

1.  Each answer should have 1 film.
2.  Each answer should explain how the film ties in to rationality.

Optional extra: List some essays in the sequences that the film connects to. Yes, non-sequences posts by other rationalists like Scott Alexander and Robin Hanson are allowed.

**Spoilers**

If you are including spoilers for the film, use spoiler tags! Put >! at the start of the paragraph to cover the text, and people can hover-over if they want to read it, like so:

This is hidden text!
# A Little Depth Goes a Long Way: the Expressive Power of Log-Depth Transformers

Authors: Anonymous (I'm not one of them).

Abstract:

> Most analysis of transformer expressivity treats the depth (number of layers) of a model as a fixed constant, and analyzes the kinds of problems such models can solve across inputs of unbounded length. In practice, however, the context length of a trained transformer model is bounded. Thus, a more pragmatic question is: *What kinds of computation can a transformer perform on inputs of bounded length?* We formalize this by studying highly uniform transformers where the depth can grow minimally with context length. In this regime, we show that transformers with depth O(log⁡C) can, in fact, compute solutions to two important problems for inputs bounded by some max context length C, namely *simulating finite automata*, which relates to the ability to track state, and *graph connectivity*, which underlies multi-step reasoning. Notably, both of these problems have previously been proven to be asymptotically beyond the reach of fixed depth transformers under standard complexity conjectures, yet empirically transformer models can successfully track state and perform multi-hop reasoning on short contexts. Our novel analysis thus explains how transformer models may rely on depth to feasibly solve problems up to bounded context that they cannot solve over long contexts. It makes actionable suggestions for practitioners as to how to minimally scale the depth of a transformer to support reasoning over long contexts, and also argues for dynamically unrolling depth as a more effective way of adding compute compared to increasing model dimension or adding a short chain of thought.
# DeepSeek beats o1-preview on math, ties on coding; will release weights

DeepSeek-R1-Lite-Preview was announced today. It's available via [chatbot](http://chat.deepseek.com/). ([Post](https://api-docs.deepseek.com/news/news1120/); [translation of Chinese post](https://mp-weixin-qq-com.translate.goog/s/e1YnTxZlzFvjcmrLLTA8fw?_x_tr_sl=zh-CN&_x_tr_tl=en&_x_tr_hl=zh-TW&_x_tr_pto=wapp).)

DeepSeek says it will release the weights and publish a report.

The model appears to be stronger than o1-preview on math, similar on coding, and weaker on other tasks.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/4a5b5c7cb0022421ca7d9d2747c5566dbd3db697c323e2aa.jpeg)

DeepSeek is Chinese. I'm not really familiar with the company. I thought Chinese companies were at least a year behind the frontier; now I don't know what to think and hope people do more evals and play with this model. Chinese companies tend to game benchmarks more than the frontier Western companies, but I think DeepSeek hasn't gamed benchmarks much historically.

The post also shows inference-time scaling, like o1:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/857c0f331b839852eb3769d44d7fd2e8b6f6e6bb887c8af9.jpeg)

Note that o1 is substantially stronger than o1-preview; see [the o1 post](https://openai.com/index/learning-to-reason-with-llms/):

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/4e42c9b4e159e807a5d0b9d7dfb38e6e8d0410ca5d1ce01b.png)

(Parts of this post and some of my comments are stolen from various people-who-are-not-me.)
# Secular Solstice Round Up 2024

This is a thread for listing Solstice and Megameetup events (dates, locations, links, etc.)

Those of you who have not been may be wondering what a Solstice is in this context.

Secular Solstice is a holiday designed by and for rationalists. It started as an attempt to match the sort of gatherings traditionally religious people tend to have in wintertime, but without compromising on truth.

Is it possible to create that same sense of emotional weight without making stuff up to do it? And without the aid of a long tradition? (Though this is the 14th year we're doing it, so there's some tradition).

Empirically, yes.

Because the world is full of things that genuinely inspire awe. Things entirely worth honoring and celebrating.

And the world is also full of things that genuinely inspire fear. We can and should face them squarely, and together.

This is not a silly holiday. It is not Festivus. It is not Santa Claus and Reindeer. We gather on the longest night of the year to stare together straight into the deepest darkness and \*make\* \*it\* \*back\* \*down\*.

What does it mean in practice? Lots of singing. Some speeches. Sometimes other things.  The whole thing runs a couple of hours.

And what is a Megameetup?  It's a meetup that's larger than a normal meetup (though not literally a million times larger).  Often it lasts an entire weekend and draws from multiple cities.  It may include presentations, organized discussions, games, and other activities rationalists tend to do when in large groups.

What do Solstices and Megameetups have to do with each other?  It's common for people to travel significant distances to a Solstice, so attaching a Metameetup makes best use of that travel overhead.
# Which things were you surprised to learn are not metaphors?

People with [aphantasia](https://en.wikipedia.org/wiki/Aphantasia) typically think that when someone says to "picture X in your mind", they're being entirely metaphorical. If you don't have a mind's eye, that's a super reasonable thing to think, but it turns out that you'd be wrong!

In that spirit, I recently discovered that many expressions about "feelings in your body" are not metaphorical. Sometimes, people literally feel a lump in their throat when they feel sad, or literally feel like their head is hot ("hot-headed") when they're angry.

It seems pretty likely to me that there are other non-metaphors that I currently think are metaphors, and likewise for other people here. So: what are some things that you thought were metaphors, that you later discovered were not metaphors?
# Which things were you surprised to learn are metaphors?

I love this [question](https://www.lesswrong.com/posts/xpC82ndFDSXtS4xK3/which-things-were-you-surprised-to-learn-are-not-metaphors), but I've enjoyed noticing answers to its opposite more. What are some things you thought weren't metaphors but were surprised to learn actually are metaphors.

A classic example is that, at least in English, time is often described using distance metaphors. For example, we talk about things taking a "long" or a "short" amount of time, about events that happened in the "distant past" or will happen in the "near future", and how two events can be said to happen "close" or "far apart" from one another in time.
# Reward Bases: A simple mechanism for adaptive acquisition of multiple reward type

Authors: Beren Millidge ,Yuhang Song, Armin Lak, Mark E. Walton, Rafal Bogacz.

Abstract:

> Animals can adapt their preferences for different types of reward according to physiological state, such as hunger or thirst. To explain this ability, we employ a simple multi-objective reinforcement learning model that learns multiple values according to different reward dimensions such as food or water. We show that by weighting these learned values according to the current needs, behaviour may be flexibly adapted to present preferences. This model predicts that individual dopamine neurons should encode the errors associated with some reward dimensions more than with others. To provide a preliminary test of this prediction, we reanalysed a small dataset obtained from a single primate in an experiment which to our knowledge is the only published study where the responses of dopamine neurons to stimuli predicting distinct types of rewards were recorded. We observed that in addition to subjective economic value, dopamine neurons encode a gradient of reward dimensions; some neurons respond most to stimuli predicting food rewards while the others respond more to stimuli predicting fluids. We also proposed a possible implementation of the model in the basal ganglia network, and demonstrated how the striatal system can learn values in multiple dimensions, even when dopamine neurons encode mixtures of prediction error from different dimensions. Additionally, the model reproduces the instant generalisation to new physiological states seen in dopamine responses and in behaviour. Our results demonstrate how a simple neural circuit can flexibly guide behaviour according to animals’ needs.

Author summary:

> Animals and humans can search for different resources depending on their needs. For example, when you are thirsty at work, you may go to a common room where hopefully coffee or water is available, while if you are hungry, you would rather go to a canteen. Such ability to seek different resources based on a physiological state is so fundamental to survival, that is present also in simple animals. This paper proposes how this ability could arise from a simple neural circuit that can be mapped on evolutionary older parts of the vertebrate brain, called the basal ganglia. The model suggests that this circuit learns the availability of different reward types, and then combines them according to the physiological state to control behaviour.
# Mechanistic Interpretability of Llama 3.2 with Sparse Autoencoders

I recently published a rather big side project of mine that attempts to replicate the mechanistic interpretability research on proprietary and open-source LLMs that was quite popular this year and produced great research papers by Anthropic[^xledbw3fbdl][^d6lsvknph5], OpenAI[^zu50aygoqk][^iacee8qisf] and Google Deepmind[^v9foytpla2f] with the humble but open source Llama 3.2-3B model.

The project provides a complete end-to-end pipeline for training Sparse Autoencoders to interpret LLM features, from activation capture through training, interpretation, and verification. All code, data, trained models, and detailed documentation are publicly available in my attempt to make this as *open research* as possible, though calling it an extensively documented personal project wouldn't be wrong either in my opinion.

Since LessWrong has a strong focus on AI interpretability research, I thought some of you might find value in this open research replication. I'm happy to answer any questions about the methodology, results, or future directions.

[^xledbw3fbdl]: https://www.anthropic.com/research/mapping-mind-language-model 

[^d6lsvknph5]: https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html 

[^zu50aygoqk]: https://openai.com/index/extracting-concepts-from-gpt-4/ 

[^iacee8qisf]: https://arxiv.org/abs/2406.04093 

[^v9foytpla2f]: https://arxiv.org/abs/2408.05147
# Disentangling Representations through Multi-task Learning

Authors: [Pantelis Vafidis](https://arxiv.org/search/cs?searchtype=author&query=Vafidis,+P), [Aman Bhargava](https://arxiv.org/search/cs?searchtype=author&query=Bhargava,+A), [Antonio Rangel](https://arxiv.org/search/cs?searchtype=author&query=Rangel,+A).

Abstract:

> Intelligent perception and interaction with the world hinges on internal representations that capture its underlying structure ("disentangled" or "abstract" representations). Disentangled representations serve as world models, isolating latent factors of variation in the world along orthogonal directions, thus facilitating feature-based generalization. We provide experimental and theoretical results guaranteeing the emergence of disentangled representations in agents that optimally solve multi-task evidence aggregation classification tasks, canonical in the cognitive neuroscience literature. The key conceptual finding is that, by producing accurate multi-task classification estimates, a system implicitly represents a set of coordinates specifying a disentangled representation of the underlying latent state of the data it receives. The theory provides conditions for the emergence of these representations in terms of noise, number of tasks, and evidence aggregation time. We experimentally validate these predictions in RNNs trained on multi-task classification, which learn disentangled representations in the form of continuous attractors, leading to zero-shot out-of-distribution (OOD) generalization in predicting latent factors. We demonstrate the robustness of our framework across autoregressive architectures, decision boundary geometries and in tasks requiring classification confidence estimation. We find that transformers are particularly suited for disentangling representations, which might explain their unique world understanding abilities. Overall, our framework puts forth parallel processing as a general principle for the formation of cognitive maps that capture the structure of the world in both biological and artificial systems, and helps explain why ANNs often arrive at human-interpretable concepts, and how they both may acquire exceptional zero-shot generalization capabilities.
# The Problem with Reasoners by Aidan McLaughin

Some critique on reasoning models like o1 (by OpenAI) and r1 (by Deepseek).

> OpenAI admits that they trained o1 on domains with easy verification but hope reasoners generalize to all domains. Whether or not they generalize beyond their RL training is a trillion-dollar question. Right off the bat, I’ll tell you my take:
> 
> o1-style reasoners do not meaningfully generalize beyond their training.

> A straightforward way to check how reasoners perform on domains without easy verification is benchmarks. On math/coding, OpenAI's o1 models do exceptionally. On everything else, the answer is less clear.
> 
> ![](https://aidanmclaughlin.notion.site/image/https%3A%2F%2Fprod-files-secure.s3.us-west-2.amazonaws.com%2F8d5858df-1063-4797-a4e7-227579d3406e%2Ffb801705-59cc-4727-a255-8716d19abe93%2FScreenshot_2024-11-20_at_9.10.15_PM.png?table=block&id=14624dff-9b2a-80f0-ac45-d21a36e3f716&spaceId=8d5858df-1063-4797-a4e7-227579d3406e&width=1250&userId=&cache=v2)
> 
> Results that jump out:
> 
> 1.  `o1-preview` does *worse* on personal writing than `gpt-4o` and no better on editing text, despite costing 6 × more.
> 2.  OpenAI didn't release scores for `o1-mini`, which suggests they may be worse than `o1-preview`. `o1-mini` also costs more than `gpt-4o`.
> 3.  On eqbench (which tests emotional understanding), [o1-preview performs](https://x.com/nrehiew_/status/1860068166166282320) as well as `gemma-27b`.
> 4.  On eqbench, [o1-mini performs](https://x.com/nrehiew_/status/1860068166166282320) as well as `gpt-3.5-turbo`. No you didn’t misread that: it performs as well as `gpt-3.5-turbo`.

> Throughout this essay, I’ve doomsayed o1-like reasoners because they’re locked into domains with easy verification. You won't see inference performance scale if you can’t gather near-unlimited practice examples for o1.
> 
> ...
> 
> I expect transformative AI to come remarkably soon. I hope labs iron out the wrinkles in scaling model size. But if we do end up scaling model size to address these changes, what was the point of inference compute scaling again?
> 
> Remember, inference scaling endows today’s models with tomorrow’s capabilities. It allows you to skip the wait. If you want faster AI progress, you want inference to be a 1:1 replacement for training.
> 
> o1 is not the inference-time compute unlock we deserve.
> 
> If the entire AI industry moves toward reasoners, our future might be more boring than I thought.
# Filled Cupcakes

Cake is tasty, pre-divided cake is convenient, and food is often better with a combination of textures and flavors. And so we have frosted cupcakes:

[![](https://www.jefftk.com/heavily-frosted-cupcake.jpg)](https://www.jefftk.com/heavily-frosted-cupcake-big.jpg)

(*source: generated by Gemini*)

While these look good, they're bad food. The problem is they're very hard to eat while getting a combination of cake and frosting in each bite. Vertical slice bites are impractical for people with non-pythonic jaws, and it's hard to alternate without undermining the frosting. Also the frosting needs to be quite firm to stand up to this stacking, limiting texture options.

You can make a jaw-compatible cupcake by restraining your frosting application to a thin layer, but this is stoichiometrically unsound.

Instead of putting goop on top of the cupcake, it should go in the middle. Here's one I made, piping [plum](https://www.jefftk.com/p/jam-is-obsolete) into the center of a chocolate cupcake:

[![](https://www.jefftk.com/filled-cupcake-cut.jpg)](https://www.jefftk.com/filled-cupcake-cut-big.jpg)

(*source: messy reality*)

Each bite is balanced, you have full artistic freedom when it comes to texture, and no one gets trismus. You can even put a (thin!) layer of frosting on top if you're using a subtle flavor, the filling injection wounds aren't appetizing, or just two components wouldn't sufficiently impress your guests.

Food should be optimized for eating.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02oiy8cG353EK8btG3XPBjq2VJDv6BtxjSdKuxeMNR9yPt9j5xef54GsCysYbsnFeSl), [mastodon](https://mastodon.mit.edu/@jefftk/113547074555964668), [bluesky](https://bsky.app/profile/jefftk.com/post/3lbt2zdany22p)*
# Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?

Authors: [Sohee Yang](https://arxiv.org/search/cs?searchtype=author&query=Yang,+S), [Nora Kassner](https://arxiv.org/search/cs?searchtype=author&query=Kassner,+N), [Elena Gribovskaya](https://arxiv.org/search/cs?searchtype=author&query=Gribovskaya,+E), [Sebastian Riedel](https://arxiv.org/search/cs?searchtype=author&query=Riedel,+S), [Mor Geva](https://arxiv.org/search/cs?searchtype=author&query=Geva,+M).

Abstract:

> We evaluate how well Large Language Models (LLMs) latently recall and compose facts to answer multi-hop queries like "In the year Scarlett Johansson was born, the Summer Olympics were hosted in the country of". One major challenge in evaluating this ability is that LLMs may have developed shortcuts by encounters of the head entity "Scarlett Johansson" and the answer entity "United States" in the same training sequences or merely guess the answer based on frequency-based priors. To prevent shortcuts, we exclude test queries where the head and answer entities co-appear in pretraining corpora. Through careful selection of relations and facts and systematic removal of cases where models might guess answers or exploit partial matches, we construct an evaluation dataset SOCRATES (ShOrtCut-fRee lATent rEaSoning). We observe that LLMs demonstrate promising latent multi-hop reasoning abilities without exploiting shortcuts, but only for certain types of queries. For queries requiring latent recall of countries as the intermediate answer, the best models achieve 80% latent composability, but this drops to just 5% for the recall of years. Comparisons with Chain-of-Thought composability highlight a significant gap between the ability of models to reason latently versus explicitly. Analysis reveals that latent representations of the intermediate answer are constructed more often in queries with higher latent composability, and shows the emergence of latent multi-hop reasoning during pretraining.

  
I've only skimmed for now, but seems relevant to Chain-of-Thought alignment and out of context reasoning (OOCR) as a threat model.
# What epsilon do you subtract from "certainty" in your own probability estimates?

Ok, nobody is actually a strict, or even particularly careful bayesean reasoner.  Still, what probability do you reserve to "my model doesn't apply, everything I know is wrong"?  If you SEE a coin flip come up heads (and examine the coin and perform whatever tests you like), what's your posterior probability that the coin actually exists and it wasn't a false memory or trick in some way?