Harley-ml/lesswrong huggingface dataset:
# 10 Aphorisms from 𝘛𝘩𝘦 𝘉𝘦𝘥 𝘰𝘧 𝘗𝘳𝘰𝘤𝘳𝘶𝘴𝘵𝘦𝘴

> 1. Never explain why something important is important.
>
> 2. If you must explain it, don’t.
>
> 3. Knowing stuff others don’t know is most effective when others don’t know you know stuff they don’t know.
>
> 4. Language is largely made to show off, gossip, confuse people, delude them, charm them, seduce them, scare them, and exploit them. And, as a side effect, convey information. Just a side effect, you know.
>
> 5. The characteristic feature of the loser is to bemoan, in general terms, mankind’s flaws, biases, contradictions, and irrationality—without exploiting them for fun and profit
>
> 6. When you say something, you think you are just saying something, but you are largely communicating *why* you had to say it.
>
> 7. Small injustice dresses up as vice; large injustice dresses up as virtue.
>
> 8. The worst damage has been caused by competent people trying to do good; the best improvements have been brought by incompetent ones *not* trying to do good.
>
> 9. An idea starts to be interesting when you get scared of taking it to its logical conclusion.
>
> 10. The problem of knowledge is that there are many more books on birds written by ornithologists than books on birds written by birds and books on ornithologists written by birds.
# Announcing: OpenAI's Alignment Research Blog

The OpenAI Alignment Research Blog launched today at 11 am PT! With 1 introductory post, and 2 technical posts.

**Blog:** [https://alignment.openai.com/](https://alignment.openai.com/)

**Thread on X:** [https://x.com/j_asminewang/status/1995569301714325935](https://x.com/j_asminewang/status/1995569301714325935?t=O5FvxDVP3OqicF-Y4sCtxw&s=19)

* * *

[Speaking purely personally](https://x.com/NaomiBashkansky/status/1995573051216593024?t=tHQa-aFp6_AsBihJWE7icg&s=19): when I joined the Alignment team at OpenAI in January, I saw there was more safety research than I'd expected. Not to mention interesting thinking on the future of alignment. But that research & thinking didn't really have a place to go, considering it's often too short or informal for the main OpenAI blog, and most OpenAI researchers aren't on LessWrong. I'm hoping the blog is a more informal, lower-friction home than the main blog, and this new avenue of publishing encourages sharing and transparency.
# Future Proofing Solstice

Bay Solstice is this weekend (Dec 6th at 7pm, with a Megameetup at Lighthaven on Friday and Saturday).

I wanted to give people a bit more idea of what to expect. 

I created Solstice in 2011. 

Since 2022, I've been worried that the Solstice isn't really set up to handle "actually looking at human extinction in nearmode" in a psychologically healthy way. (I tried to set this up in the beginning, but once my p(Doom) crept over 50% I started feeling like Solstice wasn't really helping the way I wanted).

People 'round here disagree a lot on how AI will play out. But, [Yes Requires the Possibility of No](https://www.lesswrong.com/posts/G5TwJ9BGxcgh5DsmQ/yes-requires-the-possibility-of-no), and as the guy who made rationalist solstice, it seemed like I should either:

1.  say explicitly: "sorry guys, I don't know how to have 500 people look at 'are we gonna make it?' in a way that would be healthy if the answer was 'no'. So, we're just not actually going to look that closely at the question."
2.  or, figure how to do a good job at that.

This Solstice is me attempting to navigate option #2, while handling the fact that we have lots of people with lots of different worldviews who want pretty different things out of solstice, many of whom don't care about the AI question at all. 

It has been pretty challenging, but I've been thinking about it over 3 years. I'm feeling pretty good about how I've thread the needle of making a solid experience for everyone.

Somewhat more broadly: Solstice is about looking at the deep past, the present, and the far future. When I created it in 2011, the future was sort of comfortably "over there." Now it feels like The Future is just... already happening, sort of. And some of the framing felt a bit in need of an update.

(Meanwhile there is a separate subthread about making Solstice do a better job of helping people singalong, both with a smoother singing difficulty curve, and picking songs that sort of naturally introduce new required skills in a way that helps you learn, mostly without feeling like you were "learning").

It's sort of a capstone project for the first half of my adult life. It's felt a bit to me like the season finale for Solstice ~~Season 4~~ Season 5, if Solstice was a TV show.

It's still a bit of a weird thing that's not for everyone, but, if the above sounds worthwhile to you, hope to see you there.

[Tickets to Solstice and the Megameetup are here.](https://waypoint.lighthaven.space/)
# Everyone Can Be High Status In Utopia

In [Rubber Souls](https://tomasbjartur.bearblog.dev/rubber-souls/), Bjartus Tomas argues that we can have cruely-free status games by creating underpeople without moral worth, perhaps because they are non-conscious, worth to serve as our permanent underclass. This removes the current problem where some poor bugger has to be at the bottom of the barrel, or the bottom quarter or half or so forth. 

Needless to say, I approve. 

But I think it's worth fleshing out a bit why this is possible, and why you won't wind up with everyone associating humans as a high status source of esteem as underpeople as a low status source. 

We build our sense of self-worth not from some abstract global ranking, such as percentile ranking of wealth or h-index, but through comparing ourselves to people in our social circles. So in the glorious transhumanist future where the labs somehow avoid flubbing it, Dario Amodei may be God-Emperor of the universe, but as long as he's far from your social graph, and your social graph has nary a whisper of him, then you're not likely to compare yourself with him and feel low self-esteem. 

More generally, I expect the far future to have less global status rankings because I expect everything to run at much higher speeds, making fixed travel times feel proportionally longer. If a mind ran 1,000,000,00x faster than us, for them light would only a foot/subjective second, or a measly 1km in a subjective hour. Which means it's harder to communicate and co-ordinate across the total span of human civilization, resulting in smaller, disjoint cultures with their own local hierarchies. 

Secondly, humans have very coarse personhood detectors. Let alone AIs like GPT-4o, we've even treated animals or inanimate objects as people in the past. It's just quite easy to convince our brains that a non-human entity is a person. This makes sense; how could evolution have encoded something as complex as a human detector when building our social reward circuitry? No, it had to make do with mere proxies instead. 

And that means there are bound to be very strange entities we can construct that would count as valid sources of status in the future. Yes, stranger even than LLMs. And probably more effective than humans, to boot. 

So if we build these super-stimuli status sirens, these underpeople forming our permanent underclass, who might I add could well be pleased to be permanent yes-man for mind design space is wide, could we truly say we wouldn't gladly partake of them? 

Yes, perhaps we would view associating with them as abhorrent at first. But for those who are stuck at the bottom of the status hierarchies of utopia, their need for social esteem would compel them to at least try it out. Then, deep in the midst of the underpeople's social scene, would it truly seem so bad? I think not.
# Metric-haven (quick stats on how Inkhaven impacted LessWrong)

*The data here only reflects posting activity **on LessWrong** itself.*

In 2021, the admins of LessWrong had the idea that we'd [pay people to write book reviews](https://www.lesswrong.com/posts/eHnupDgggBqDqT5eg/lesswrong-is-paying-usd500-for-book-reviews). In 2025, we had a much better idea: [people would pay *us* to write all kinds of posts](https://www.lesswrong.com/posts/CA6XfmzYoGFWNhH8e/the-inkhaven-residency)!

I think this went pretty well, final determination pending, but in the meantime I can say the numbers have been impacted. That'll be no surprise to those regularly checking the site.

The number of posts increased by 57% (477 → 749) and number of words by 45% (1.0M → 1.46M). The increases were driven by 21 people officially involved in Inkhaven (residents, coaches, contributing writers) and 3 copycats[^t8oxz4uhzh] I identified by the numbers and their [written](https://www.lesswrong.com/posts/wELmghiTfjWSGyfvN/november-retrospective) [intention](https://www.lesswrong.com/posts/uch7FMWn2EFwpXkkc/ink-without-haven) to [participate](https://www.lesswrong.com/posts/9q3p2XJdFEB6sMyBZ/ftl-travel-and-scientific-realism).

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/0dae70918a4636205833b3855eb92ca567fe49af0cef0b1f.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/ebaa28a8679f0fbd564f6875902a21c732da32cafe0aaad8.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/c958aa609133691776cd40f475d30e662b37e78ed375a2c7.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/22b7a94ed1c53f9ca369409aa554bc110f56f3e8d204c0d3.png)

Curiously, the large boost to LessWrong was affected with only a handful of writers posting ~daily posts to the site. Per the [Inkhaven blogroll](https://www.inkhaven.blog/), most writers published on Substack.

I believe beyond the three copycats on LessWrong, others expressed an intention to blog daily but did so on blogs elsewhere. Lorxus participated in [Halfhaven](https://www.lesswrong.com/posts/7axYBeo7ai4YozbGa/halfhaven-virtual-blogger-camp) and posted [weekly roundups](https://www.lesswrong.com/posts/JCYSfPkEcva9WCXdh/lorxus-does-halfhaven-11-22-11-28) of their posts on LessWrong, but those don't count towards the totals here.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/a26480a19fcc0bd02fcc17826795b83f8a7830f5e880f985.png)

21% of the Inkhaven wordcount on LessWrong came from the LessWrong team. 79% came from others!

Ok, but what of quality? Karma ("baseScore") is the perfect measure of that. The good news is non-Inkhaven participant karma only declined 16 → 13 at the median.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/6b5a8d2fc7a44e7da45de2cff067fd0f85eceacdec6723ab.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/93cd296038aaa92f324dbe7cfc6cc2d952b80ca0a6b22c74.png)

There's more interpretation to be done here but I'm out of time. Such is the Inkhaven way. (This post began as an attempt from me to submit another Inkhaven post myself, but also, it's all graphs and not words!)

[^t8oxz4uhzh]: I use the term affectionately.
# Open Thread Winter 2025/26

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the [Highlights from the Sequences](https://lesswrong.com/highlights), a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/faq#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/faq). If you want to orient to the content on the site, you can also check out the [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/w/open-threads?sortedBy=new). The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# Embedded Universal Predictive Intelligence

A team at Google has substantially advanced the theory of embedded agency with a grain of truth (GOT), including new developments on reflective oracles and an interesting alternative construction (the "Reflective Universal Inductor" or RUI).

(I was not involved in this work)

Abstract:

> The standard theory of model-free reinforcement learning assumes that the environment dynamics are stationary and that agents are decoupled from their environment, such that policies are treated as being separate from the world they inhabit. This leads to theoretical challenges in the multi-agent setting where the non-stationarity induced by the learning of other agents demands prospective learning based on prediction models. To accurately model other agents, an agent must account for the fact that those other agents are, in turn, forming beliefs about it to predict its future behavior, motivating agents to model themselves as part of the environment. Here, building upon foundational work on universal artificial intelligence (AIXI), we introduce a mathematical framework for prospective learning and embedded agency centered on self-prediction, where Bayesian RL agents predict both future perceptual inputs and their own actions, and must therefore resolve epistemic uncertainty about themselves as part of the universe they inhabit. We show that in multi-agent settings, self-prediction enables agents to reason about others running similar algorithms, leading to new game-theoretic solution concepts and novel forms of cooperation unattainable by classical decoupled agents. Moreover, we extend the theory of AIXI, and study universally intelligent embedded agents which start from a Solomonoff prior. We show that these idealized agents can form consistent mutual predictions and achieve infinite-order theory of mind, potentially setting a gold standard for embedded multi-agent learning.
# Blog post: how important is the model spec if alignment fails?

> A model spec is a document that describes the intended behavior of an LLM, including rules that the model will follow, default behaviors, and guidance on how to navigate different trade-offs between high-level objectives for the model. Most thinking on model specs that I’m aware of focuses on specifying the desired behavior for a model that is mostly [intent-aligned](https://www.alignmentforum.org/posts/ZeE7EKHTFMBs8eMxn/clarifying-ai-alignment) to the model spec. In this post, I discuss how a model spec might be important even if the developer fails to produce a system that is fully aligned with the model spec.

*This post was created by* [*Forethought*](https://www.forethought.org/about)*.*
# Front-Load Giving Because of Anthropic Donors?

Summary: Anthropic has many employees with an EA-ish outlook, who may soon have a lot of money. If you also have that kind of outlook, money donated sooner will likely be much higher impact.

It's December, and I'm trying to figure out how much to [donate](https://www.jefftk.com/donations). This is usually a straightforward question: [give 50%](https://www.jefftk.com/p/giving-half). But this year I'm considering dipping into savings.

There are many EAs and EA-informed employees at Anthropic, which has been very successful and is [reportedly considering](https://www.reuters.com/business/retail-consumer/anthropic-plans-an-ipo-early-2026-ft-reports-2025-12-03/) [an IPO](https://www.ft.com/content/3254fa30-5bdb-4c30-8560-7cd7ebbefc5f). The [Manifold market estimates](https://manifold.markets/JonasVollmer/when-will-anthropic-ipo) a median IPO date of June 2027:

At a floated $300B valuation and many EAs among their early employees, the amount of additional funding could be in the billions. Efforts I'd most want to support may become less constrained by money than capacity: as I've experienced in running the [NAO](https://naobservatory.org/), scaling programs takes time. This means donations now seem more valuable; ones that help organizations get into a position to productively apply further funding especially so.

One way to get a sense of the impact of donating sooner is to imagine that others will donate $1M to my preferred charity this year, and $10M next year. If I have $200k, I expect giving it all this year, for a total of $1.2M this year and $10M the next, would be more valuable than splitting it evenly, for $1.1M this year and $10.1M the next. The $100k in question would be a 9% increase in funding this year, but only a 1% increase next year.

In retrospect I wish I'd been able to support [80,000 Hours](https://80000hours.org/) more substantially before ~~Open Philanthropy~~ [Coefficient Giving](https://forum.effectivealtruism.org/posts/vkvtu6xbvfkHPhJkC/open-philanthropy-is-now-coefficient-giving) began funding them; this time, with more ability to see what's likely coming, I'd like to avoid that mistake.

Now, Anthropic could fail, the IPO could take a long time with minimal opportunity for employees to take money off the table before then, or the employees could end up primarily interested in funding different things than I want to see funded. Still, it seems to me that EA-influenced funding likely goes a lot farther in the next few months than it will in a few years, and I think I should probably donate more this year.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0uVMyXzrekYTFh4xF2WEhYf9zjb7QZnfmC18UWR42hgs4UiAfST8SFn8Fs3tgxuqQl), [lesswrong](https://lesswrong.com/posts/oJM6TmFLwsHrBfRpk), [the EA Forum](https://forum.effectivealtruism.org/posts/rRBaP7YbXfZibSn3C), [mastodon](https://mastodon.mit.edu/@jefftk/115658934214610533), [bluesky](https://bsky.app/profile/jefftk.com/post/3m74wduimos27), [substack](https://jefftkaufman.substack.com/p/front-load-giving-because-of-anthropic)*
# Livestream for Bay Secular Solstice

The Livestream for the [Bay Secular Solstice](https://www.lesswrong.com/events/FjHG3XcrhXkGWTDwf/berkeley-secular-solstice-weekend) will be here:

[https://www.youtube.com/live/wzFKAxT5Uyc](https://www.youtube.com/live/wzFKAxT5Uyc)

Should start roughly 7:30pm Saturday, Dec 6th. You can sign up for notifications.

If you wanna come to the in-person Solstice, you can get tickets at [waypoint.lighthaven.space](https://www.waypoint.lighthaven.space/). 

![No photo description available.](https://scontent-sjc6-1.xx.fbcdn.net/v/t39.30808-6/595293370_10237531232554201_3975596945114511309_n.jpg?stp=cp6_dst-jpg_tt6&_nc_cat=111&ccb=1-7&_nc_sid=127cfc&_nc_ohc=HJ3W5Od-lfoQ7kNvwFCy4Zj&_nc_oc=Adm2oWTZgATtauhmdMrTeTRooNCBVtpKz5ZarajkKrOJHt_hOgM8FnnBrHIOyM6c50o&_nc_zt=23&_nc_ht=scontent-sjc6-1.xx&_nc_gid=vhXzRj9GB8xUId-bxAUwxQ&oh=00_AfkkglVT32jPml8BwheLNw9zUiIoETiV3EhCgMfYS8NjOQ&oe=69379565)
# Announcing: Agent Foundations 2026 at CMU

[Iliad](https://www.iliad.ac/) is now opening up [applications](https://forms.gle/y96z33LGoeFUfaJz9) to attend [Agent Foundations 2026 at CMU](https://agentfoundations.net/)!  
  
**Agent Foundations 2026** will be a 5-day conference (of ~35 attendees) on fundamental, mathematical research into agency. It will take place **March 2–6, 2026 at Carnegie Mellon University** in Pittsburgh, Pennsylvania, and will be the [third](https://agentfoundations.net/2023/) [conference](https://www.agentfoundations2025atcmu.org/) in the Agent Foundations conference series.

Topics covered will include:

*   Decision Theory
*   Learning Theory
*   Abstractions and World Models
*   Causality
*   Logical Induction
*   Bounded Optimality

**Apply:** [**Here**](https://forms.gle/y96z33LGoeFUfaJz9) by January 12, 2026 at 11:59 pm AoE.

*Please reach out to us at* [*contact@iliad.ac*](mailto:contact@iliad.ac) *(or below) with any questions.*
# Existential despair, with hope

I have drafted thousands of words of essays on the topic of art and why it sustains my soul in times of despair, but none of it comes close to saying what I want to say, yet.

Therefore, without explanation, on the occasion of Berkeley’s Winter Solstice gathering,  I will simply offer a link to a little-known work of art which has been for me a touchstone in my times of greatest grief and existential despair:

https://www.viselaya.org/sculptures/chandala/chandala.html

May it bring some of you something you need today.
# Ordering Pizza Ahead While Driving

On a road trip there are a few common options for food:

*   Bring food
*   Grocery stores
*   Drive throughs
*   Places that take significant time to prepare food

Bringing food or going to a grocery store are the cheapest (my preference!) but the kids are hard enough to feed that we often buy prepared food when we're traveling. \[1\] And they often prefer food that takes a while to make (usually pizza) over what you can get in a drive through. A couple years ago I realized there's another option: calling in an order for pickup to where you'll be soon.

We'll use Google Maps "search along route" to identify a place ~30min out, and phone in an order. \[2\] By the time we arrive, the food is ready. We can combine the speed (and [buffer maximization](https://www.jefftk.com/p/travel-buffer)) benefits of drive throughs, with the variety of options from the wide range of restaurants that offer pickup.

  
\[1\] I'm also working on getting them to do better with brought food, but I'm focusing on lunch at school here because that's a much larger portion of their food away from home.

\[2\] It kind of amazes me that pizza places will take the costly action of preparing a pizza to my specifications based on a simple phone call, with no contact information beyond me giving a first name. I mean, it's great, but like so many things in our society it only works because there are extremely few people want to cause havoc and are willing to put any effort into doing so.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0XDdJ2YmfWvryd62gni8ENPyJNq6MNxWvMqWHHT38gNEpTdJ4nuuHNnJ6pKKfE7Nnl), [mastodon](https://mastodon.mit.edu/@jefftk/115675799330669317), [bluesky](https://bsky.app/profile/jefftk.com/post/3m7efz6aqxk2n)*
# 2025 Unofficial LessWrong Census/Survey

~~**The Less Wrong General Census is unofficially here! You can take it at**~~ [~~**this link.**~~](https://docs.google.com/forms/d/e/1FAIpQLSfIr9OmrOcpqGmVbHnRUU8q6uwSTT3S2vE6e3ElJLBMViQ2dw/viewform?usp=dialog)

(Update, the census has closed! Thank you for taking it. An analysis is planned to be out in late January.)

The kinda-sorta-annual-if-you-really-squint tradition of the Less Wrong Census is once more upon us!

I want to pitch you on taking the survey. First, the demographics data is our best view into who's using the site and who is in the community. I use it when trying to figure out how to help meetups, I add a bunch of questions the LessWrong team asks which I assume they use to improve the site. Second, because I am madly, ravenously curious what works and what doesn't for raising the sanity waterline and making people more rational. I'm even curious what folks think that would mean! Third, if you're reading this and you're thinking "eh, I'm not really a rationalist, more just rationalist adjacent" then you are also of interest; I include questions from several adjacent communities as well. Also, you make a fantastic active control group.

If you want to just spend five minutes answering the basic demographics questions before leaving the rest blank and hitting submit, that's totally appreciated. The survey is structured so the fastest and most generally applicable questions are generally speaking towards the start. (I did add some intermissions that mix this up. If you're impatient, feel free to skip any or all intermissions.) At any point you can scroll to the bottom and hit Submit, though you won't be able to add more answers once you do. 

The survey shall remain open from now until at least January 7th, 2026. I plan to close it sometime on Jan 8th.

I don't work for LessWrong, but I do work on improving rationalist meetups around the world. Once the survey is closed, I plan to play around with the data and write up an analysis post [like this one](https://www.lesswrong.com/s/MKsuezdHmrHfbQCqT/p/gpZBWNFxymsqnPB92) sometime in late January. 

~~**Please take the survey at**~~ [~~**this link**~~](https://docs.google.com/forms/d/e/1FAIpQLSfIr9OmrOcpqGmVbHnRUU8q6uwSTT3S2vE6e3ElJLBMViQ2dw/viewform?usp=dialog)~~**. **~~

(Update, the census has closed! Thank you for taking it. An analysis is planned to be out in late January.)

Ancient tradition is that if you take the survey you can comment here saying you took the survey, and people upvote you for karma.
# I said hello and greeted 1,000 people at 5am this morning

At the ass crack of dawn, in the dark and foggy mist, thousands of people converged on my location, some wearing short shorts, others wearing an elf costume and green tights.

I was volunteering at a marathon. The race director told me the day before, “these people have trained for the last 6-12 months for this moment. They’ll be waking up at 3am. For many of them, this is the first marathon they’ve ever run. When they get off the bus at 5am, in the freezing cold, you’ll be the first face they see. Smile, welcome them, make them feel excited, and help them forget the next 26.2 miles of pain they’re about to endure.”

Even though I normally have [RBF](https://en.wikipedia.org/wiki/Resting_bitch_face) and consider it a chore to acknowledge people, I slapped a big fat smile on my face and excitedly greeted runners like I was a golden retriever who hasn’t seen his military dad in over a year.

“HELLO!” 

“GOOD MORNING!”

“YOU’RE HERE!”

^That, on repeat for two hours straight. It was actually pretty fun. I calculated the optimal distance to stand from the bus was eight feet away. Stand too close, and the runners were still descending the bus steps and didn’t have time to process my presence. Stand too far away, and they assumed I was just some overly enthusiastic volunteer having a borderline religious experience.

Once I optimized the greeting distance, I observed people’s reactions to me fell into four different categories:

1.  **Genuine smilers**
    1.  All of the runners were cold, sleep-deprived, and nervous. But when they saw me (metaphorically wagging my tail), they reacted like I was a happy dog: their faces lit up; they said hello and gave me a genuine smile.
2.  **Thankers**
    1.  Some runners thanked me for volunteering. I responded with, “If you weren’t here, I’d be excitedly greeting no one and look like a crazy person. So thanks for coming out!” That made them chuckle.
3.  **Thin-lipped smile**
    1.  These runners didn’t say hi back. They acknowledged me with a thin-lipped, polite smile and crow’s feet. 
4.  **Other golden retrievers**
    1.  “GOOD MORNING! YOU’RE HERE!”  
        “I AM HERE, YEAH! LET’S GO!!!”  
        Like the Kool-Aid man, “OHHH YEAHHHH!!!”  
        “YEAHHH!!!”  
        “AGHHHHHHHHHHH.”  
        “AGHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH.”
    2.  This was fun. It made other people laugh.

After the marathon for the rest of the day, I found myself smiling and greeting other people I encountered: at a restaurant for lunch, and at the coffee shop where I’m writing this. I smiled at every stranger I made eye contact with, and every one of them smiled back (with one of the four reactions I listed). It just happened naturally, and it felt amazing. I think it made them feel good, too. That’s why we love golden retrievers—they love and greet us unconditionally.
# I have hope

This is in response to [Zvi Mowshowitz's "Little Echo"](https://www.lesswrong.com/posts/YPLmHhNtjJ6ybFHXT/little-echo) in which he discusses the secular solstice and the pessimistic situation with AI GCR (artificial intelligence global catastrophic risk). He discusses life and working to help make the situation better. He says "I believe that we will win". I am very grateful for the sentiment and ended up typing the following in response:

I have hope. I don't think "hope" is "thinking winning is likely". I think "hope" is "thinking it's possible and worth working to make more likely than it otherwise would be".

I don't think my believing is equipped to deal with things like this. I believe it's raining outside. I believe I should go jog and have my shower and get started with my day. I believe in my immediate surroundings. Everything beyond that feels like a trick. Work? Economies? Billions of people in many countries around the world? Data centers churning datasets into pattern generalization engines? These are things I *think* are real. I *rationalize* that they are real. I *logically deduce* that they are real... but I don't really *believe* they are real.

I don't believe we will win because I struggle to really believe we are even fighting. I imagine a billion bodies dropping dead. I imagine the process of a system comparing possibilities and actions and targeting outcome that have been encoded by our collective inattentiveness and imprecision. Actions and reactions leading the world irreparably off the path of anything I would consider good. I imagine these things and think... yes that is a logical conclusion of everything I know... but only vary rarely do I believe in it in the fully embodied way that I believe in the rain jacket I should be putting on. I believe it as much as I believe in a career I am trying to have in which I help solve the unsolvable problem... but all of that is just a ritual I go through as a result of my rationalization.

I don't know if I believe in believing anymore. What does the word "believe" even mean? I've focused on what could be meant by "AI" and "minds" and "decision processes" and "outcome influencing systems" so much that "intelligence" and "belief" seem like illusions that stop making sense when you look at them carefully. I can't see them anymore... I see embodied sensory. I see abstract models in symbolic structures. I see expectation of constraints on the sets of possible outcomes.

So I don't "believe" we will win. I don't think it is likely we win. But I think winning is possible and winning is what I focus on. Winning is what I think about and rationalize about and apply logical deduction to. I hope other people will focus on winning. I hope it will be enough.
# A few quick thoughts on measuring disempowerment

People want to measure and track gradual disempowerment.  One issue with a lot of the proposals I've seen is that they don't distinguish between empowering and disempowering uses of AI.  If everyone is using AI to write all of their code, that doesn't necessarily mean they are disempowered (in an important sense).  And many people will look at this as a good thing -- the AI is doing so much valuable work for us!  
  
It generally seems hard to find metrics for AI adoption that clearly track disempowerment; I think we may need to work a bit harder to interpret them.  One idea is to augment such metrics with other sources of evidence, e.g. social science studies, such as interviews, of people using the AI in that way/sector/application/etc.  
  
We can definitely try using formal notions of empowerment/POWER (cf [https://arxiv.org/abs/1912.01683).](https://arxiv.org/abs/1912.01683).)  Note that these notions are not necessarily appropriate as an optimization target for an AI agent.  If an AI hands you a remote control to the universe but doesn't tell you what the buttons do, you aren't particularly empowered.  
  
People could also be considered more disempowered if:

*   They are less able to predict important attributes of a system's behavior (cf "simulatability" in interpretability research)
    *   Example: An artist is empowered if they can realize a particular vision for a piece, whereas generating art by prompting current AIs is much more like a process of curation.
    *   This is similar to empowerment but more focused on prediction rather than control. 
*   They self-report it.
*   They are "rubber-stamping" their approval of AI decisions
    
    *   This can be revealed by red-teaming the human overseer, i.e. sending them examples they should flag and seeeing if they fail to flag them.
# High-level approaches to rigor in interpretability

There are three broad types of approach I see for making interpretability rigorous.  I've put them in ascending order of how much assurance I think they can provide.  I think they all have pros and cons, and am generally in favor of rigor.

1.  (weakest) Practical utility: Does this interpretability technique help solve problems we care about, such as jailbreaks?
    1.  This is the easiest to test for, but also doesn't directly bear on the question of "does the system work the way we think it does" that mechanistic interpretability is supposedly all about.
    2.  I think of Cas as having [pushed this approach](https://www.alignmentforum.org/s/a6ne2ve5uturEEQK7) in the AI safety community.  From talking to [Leo Gao](https://www.lesswrong.com/posts/Hy6PX43HGgmfiTaKu/an-ambitious-vision-for-interpretability), it seems like GDM has recently pivoted in this direction.
2.  (stronger) Simulatability: Can a human equipped with this interpretability techniques make better predictions about a given system's behavior?
    1.  This seems very underexplored from what I can tell. 
    2.  Running experiments like this are costly because of the use of human subjects. 
    3.  Ultimately, this isn't satisfying because a tool could improve a human user's predictions through manipulation rather than by informing them. 
3.  (strongest) Principled theoretical understanding: have we developed rigorous mathematical definitions with satisfying conceptual properties and shown that systems meet them? 
    1.  Causal scrubbing and Atticus Geiger's work are examples of such ideas; neither is satisfactory.
    2.  Strict definitions of interpretability are probably completely intractable to satisfy, but we could hope to characterize conditions under which approximations are "good enough" according to various criteria. 

**Some random extra context:**  
I have a bit of a reputation as a skeptic/hater of mechanistic interpretability in the safety community.  This is not entirely unearned, I'm largely born out of an impression that much of the early work lacked rigor, and was basically a bunch of "just-so stories". Colleagues began telling me that this was clearly no longer the case starting with the circuits thread, and I've definitely noticed an improvement.
# Do you take joy in effective altruism?

I have been interested in effective altruism since before I knew the term. I have long done things like give money to charity, buy products that seem ethical and avoid products that seem unethical (such as factory-farmed meat).

I do not derive much joy from any of this, though. It feels like a chore.

Nor do I particularly enjoy things like giving money to beggars. 

The kinds of easily-available altruism that I *do* enjoy tends to be helping people that are already relatively happy - for example when an old lady asks me to reach something for her on the top shelf.

(I might also also conceivably enjoy helping people with major things, but those opportunities rarely arise, and I do not want to seek them out.)

I have been reading a lot of Buddhist-inspired literature recently, and they talk a lot about "the joy of blamelessness". I... don't get that.

I am interested in hearing from other people who are into effective altruism. Do you derive joy from it? If so, how?
# We don't know what most microbial genes do. Can genomic language models help?

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/96e791ee6d1b9dbc89b3cf63f2ec8747faf5355cb2f85e4f.png)

Youtube: [https://youtu.be/w6L9-ySnxZI?si=7RBusTAyy0Ums6Oh](https://youtu.be/w6L9-ySnxZI?si=7RBusTAyy0Ums6Oh)

Spotify: [https://open.spotify.com/episode/2EgnV9Y1Mm9JV5m9KAY6yL?si=GcZR80aFS26uO88lpmadBQ](https://open.spotify.com/episode/2EgnV9Y1Mm9JV5m9KAY6yL?si=GcZR80aFS26uO88lpmadBQ)

Apple Podcast: [https://apple.co/4pu4TRB](https://apple.co/4pu4TRB)

Transcript: [https://www.owlposting.com/p/we-dont-know-what-most-microbial](https://www.owlposting.com/p/we-dont-know-what-most-microbial)

\*\*\*\*

This is an interview with [Yunha Hwang](https://www.yunhahwang.com/), an assistant professor at MIT (and co-founder of the non-profit [Tatta Bio](https://www.tatta.bio/)). She is working on building and applying genomic language models to help annotate the function of the (mostly unknown) universe of microbial genomes.

There are two reasons I filmed this (and think its worth watching):

One, Yunha is working on an absurdly difficult and interesting problem: microbial genome function annotation. Even for E. coli, one of the most studied organisms on Earth, we don’t know what half to two-thirds of its genes actually do. For a random microbe from soil, that number jumps to 80-90%. Her lab is one of the leading groups working to apply deep learning to solving the problem, and last year, [released a paper that increasingly feels foundational within it](https://www.biorxiv.org/content/10.1101/2024.08.14.607850v1) (with [prior podcast guest Sergey Ovchinnikov](https://www.owlposting.com/p/what-could-alphafold-4-look-like) an author on it!). We talk about that paper, its implications, and where the future of machine learning in metagenomics may go.

And two, I was especially excited to film this so I could help bring some light to a platform that she and her team at Tatta Bio has developed: [SeqHub](https://seqhub.org/). There’s been a lot of discussion online about AI co-scientists in the biology space, but I have increasingly felt a vague suspicion that people are trying to be too broad with them. It feels like the value of these tools are not with general scientific reasoning, but rather from deep integration with how a specific domain of research engages with their open problems. SeqHub feels like one of the few systems that mirrors this viewpoint, and while it isn’t something I can personally use—since its use-case is primarily in annotating and sharing microbial genomes, neither of which I work on!—I would still love for it to succeed. If you’re in the metagenomics space, you should try it out at [seqhub.org](https://seqhub.org/)!
# Apply to ESPR & PAIR 2026, Rationality and AI Camps for Ages 16-21

**Update**: The deadline has been extended to **January 12th**.  
  
*TLDR – Apply now to *[*ESPR*](https://espr.camp/) *and*[*PAIR*](https://pair.camp/)*. ESPR welcomes students between 16-19 years. PAIR is for students between 16-21 years.*

The [FABRIC team](http://fabric.camp/) is once again running two immersive summer workshops for mathematically talented students this year.

The [**European Summer Program on Rationality (ESPR)**](https://espr.camp/) is for students with a desire to understand themselves and the world, and interest in applied rationality.

*   The curriculum covers a wide range of topics, from game theory, cryptography, and mathematical logic, to AI, styles of communication, and cognitive science. See the [content details](https://espr.camp/program).
*   For students who are **16-19 **years old.
*   July 26th - August 5th in Somerset, United Kingdom 

The [**Program on AI and Reasoning (PAIR)**](https://pair.camp/) is for students with an interest in artificial intelligence, cognition, and minds in general. 

*   We will study how current AI systems work, mathematical theories about human minds, and how the two relate. Alumni of previous PAIRs described the content as a blend of AI, mathematics and introspection. See the [curriculum details](https://pair.camp/curriculum).
*   For students who are **16-21 **years old.
*   August 22nd - September 1st in Somerset, United Kingdom.

The above lists of topics don't quite cover what the camps are About, though. A lot of the juice comes from being around both peers and older folks who are excited to figure out how the world works and what levers we have within it.

Commonly appreciated themes:

*   Caring about things is good, actually
*   What's stopping you?
*   Other minds are *interesting*
    *   My mind is ***interesting***
*   Why do you believe what you believe, and when did it start?

We encourage all Lesswrong readers interested in these topics who are within the respective age windows to apply! Likewise, if you know someone eligible and potentially interested, please send them the link to the [FABRIC website](http://fabric.camp/).

Both programs are **free **for accepted students including lodgings and meals, and travel scholarships are available. Apply to both camps [here](https://airtable.com/appFhICt0TGITl6ji/pagW57vMcxjzHMYT0/form). The application deadline is Monday January 12th.
# Cognitive Tech from Algorithmic Information Theory

***Epistemic status:** Compressed aphorisms.*

This post contains no algorithmic information theory (AIT) exposition, only the rationality lessons that I (think I've) learned from studying AIT / AIXI for the last few years. Many of these are not direct translations of AIT theorems, but rather frames suggested by AIT. In some cases, they even fall outside of the subject entirely (particularly when the crisp perspective of AIT allows me to see the essentials of related areas).

**Prequential Problem.** The posterior predictive distribution screens off the posterior for sequence prediction, therefore it is easier to build a strong predictive model than to understand its ontology.

**Reward Hypothesis (or Curse).** Simple first-person objectives incentivize sophisticated but not-necessarily-intended intelligent behavior, therefore it is easier to build an agent than it is to align one. 

**Coding Theorem.** A multiplicity of good explanations implies a better (ensemble) explanation.

**Gacs' Separation.** Prediction is close but not identical to compression.

**Limit Computability.** Algorithms for intelligence can always be improved.

**Lower Semicomputability of M.** Thinking longer should make you less surprised. 

**Chaitin's Number of Wisdom.** Knowledge looks like noise from outside. 

**Dovetailing.** Every meta-cognition enthusiast reinvents Levin/Hutter search, usually with added epicycles.

**Grain of Uncertainty (Cromwell's Rule).** Anything with a finite description gets nonzero probability.

**Grain of Truth (Reflective Oracles).** Understanding an opponent perfectly requires greater intelligence or something in common.

**Grain of Ignorance (Semimeasure Loss).** You cannot think long enough to know that you do not need to think for longer.

**Solomonoff Bound.** Bayesian sequence prediction has frequentist guarantees for log loss.

**Information Distance.** There are no opposites.

**Prediction of Selected Bits.** Updating on the unpredictable can damage your beliefs about the predictable.

**Vovk's Trick.** Self-reflection permits partial models.
# The Fantastic Piece of Tinfoil in my Wallet

The gates in the lobby of my workplace annoyed me for years: they would often reject my access card, and I'd need to tap several times. After a while I realized that the reader was getting confused by the other RFID cards in my wallet, and if I pulled the card out of my wallet first it worked every time.

This turned out to be very easy to fix: tape a piece of tinfoil to the back of my access card:

[![](https://www.jefftk.com/tinfoil-on-back-of-card.jpg)](https://www.jefftk.com/tinfoil-on-back-of-card-big.jpg)

I feel a bit silly that after spending months pulling the card out each time the fix ended up taking me a couple minutes. I often decide to put up with minor annoyances instead of thinking about whether there's a way to fix them, and I think overall that has made my life substantially better, but in this case even a little thought would have been well worth it!
# Wages under superintelligence

This is a linkpost to a blogpost I've written about wages under superintelligence, responding to recent discussion among economists. 

TLDR: Under stylized assumptions, I argue that, if there is a superintelligence that generates more output per unit of capital than humans do across all tasks, human wages could decline relative to today, because humans will be priced out of capital markets. At that point, human workers will be reduced to the wage we can get with our bare hands: we won’t be able to afford complementary capital. This result holds even if there is rapid capital accumulation from AI production. To avoid horrible outcomes for labor, we would need redistribution or other political reforms. I also discuss situations where my argument doesn’t go through.
# Permanently Padding a Suitcase

My current setup for playing dances is a [bit excessive](https://www.jefftk.com/p/rhythm-stage-setup-components), and transport my [pedalboard](https://www.jefftk.com/p/sturdier-and-lighter-pedalboard) in an old hardshell suitcase. \[1\] There's no built-in padding, so I use sheets of foam to protect my equipment. Initially I just had these loose, but this was one more piece to deal with, so I decided to permanently attach them to the suitcase.

It's tricky to bond foam to plastic, but after some LLM-assisted searching it seemed like [3M 90 spray adhesive](https://www.amazon.com/dp/B0DVMH31KK) was a good choice. I followed the instructions (outside!) and attached my foam sheets to the case:

[![](https://www.jefftk.com/suitcase-with-padding-attached.jpg)](https://www.jefftk.com/suitcase-with-padding-attached-big.jpg)

I was going to write something up right away, but I was wary of the dynamic where I write when something is new and exciting but [before I know if it will last](https://www.jefftk.com/p/kids-posts-retrospective). Would the sheets just fall off again?

It's now been four months, including six gigs with [Kingfisher](https://www.kingfisherband.com/), three with the [Free Raisins](https://www.freeraisins.com/), and one with [Dandelion](https://www.jefftk.com/dandelion/). \[2\] Four of these involved flights, which with the rough handling and temperature swings are probably the worst abuse the case will get. And... it's great! I haven't had any issues, and they still seem well attached. I guess I could have written the post when I had that just-did-something-new energy after all!

  
\[1\] I looked into [lighter options](https://www.jefftk.com/p/very-light-hardshell-suitcases) but everything was super expensive and I couldn't find good used options.

\[2\] Somehow I ended up playing a very large fraction of my dances for the year this fall. This was more than currently makes sense given work and family, and I'm going to try to spread things out a lot more.