Harley-ml/lesswrong huggingface dataset:
# Human-AI Complementarity: A Goal for Amplified Oversight

*By Sophie Bridgers, Rishub Jain, Rory Greig, and Rohin Shah*  
*For more details and full list of contributors, please see our paper:* [*https://arxiv.org/abs/2510.26518*](https://arxiv.org/abs/2510.26518)

Human oversight is critical for ensuring that Artificial Intelligence (AI) models remain safe and aligned to human values. But AI systems are rapidly advancing in capabilities and are being used to complete ever more complex tasks, making it increasingly challenging for humans to verify AI outputs and provide high-quality feedback. How can we ensure that humans can continue to meaningfully evaluate AI performance? An avenue of research to tackle this problem is “[Amplified Oversight](https://www.alignmentforum.org/posts/79BPxvSsjzBkiSyTq/agi-safety-and-alignment-at-google-deepmind-a-summary-of#Amplified_Oversight)” (also called “Scalable Oversight”), which aims to develop techniques to use AI to amplify humans’ abilities to oversee increasingly powerful AI systems, even if they eventually surpass human capabilities in particular domains.

With this level of advanced AI, we could use AI itself to evaluate other AIs (i.e., AI raters), but this comes with drawbacks (see Section IV: The Elephant in the Room). Importantly, humans and AIs have complementary strengths and weaknesses. We should thus, in principle, be able to leverage these complementary abilities to generate an oversight signal for model training, evaluation, and monitoring that is stronger than what we could get from human raters or AI raters alone. Two promising mechanisms for harnessing human-AI complementarity to improve oversight are:

1.  *Rater Assistance*, in which we give human raters access to an AI rating assistant that can critique or point out flaws in an AI output or automate parts of the rating task, and
2.  *Hybridization*, in which we combine judgments from human raters and AI raters working in isolation based on predictions about their relative rating ability per task instance (e.g., based on confidence).

The design of Rater Assistance and/or Hybridization protocols that enable human-AI complementarity is challenging. It requires grappling with complex questions such as how to pinpoint the unique skills and knowledge that humans or AIs possess, how to identify when AI or human judgment is more reliable, and how to effectively use AI to improve human reasoning and decision-making without leading to under- or over-reliance on the AI. These are fundamentally questions of Human-Computer Interaction (HCI), Cognitive Science, Psychology, Philosophy, and Education. Luckily, these fields have explored these same or related questions, and AI safety can learn from and collaborate with them to address these sociotechnical challenges. On our team, we have worked to [expand our interdisciplinary expertise](https://distill.pub/2019/safety-needs-social-scientists/) to make progress on Rater Assistance and Hybridization for Amplified Oversight.

Read the rest of the [full blog here](https://deepmindsafetyresearch.medium.com/human-ai-complementarity-a-goal-for-amplified-oversight-0ad8a44cae0a)!
# What Have Been Your Most Valuable Casual Conversations At Conferences?

I've heard repeatedly from many people that the highest-value part of conferences is not the talks or structured events, but rather the casual spontaneous conversations. Yet my own experience does not match this at all; the casual spontaneous conversations are consistently low-value.

My current best model is that the casual spontaneous conversations mostly *don't* have much instrumental value, most people just really enjoy them and want more casual conversation in their life.

... but I'm pretty highly uncertain about that model, and want more data. So, questions for you:

*   What have been your highest-value casual conversations, especially at conferences or conference-like events?
*   Is most of the value terminal (i.e. you enjoy casual conversation) or instrumental (i.e. advances other goals)? And if instrumental, what goals have some of your high-value conversations advanced and how?

Note that "it feels like there was something high value in <example conversation> but it's not legible" is a useful answer!
# Living with Rats in College

When I was in college, I rented a group house with 5 other nerds. There were 5 bedrooms to divide among the 6 of us, so I negotiated an agreement where I paid less rent in exchange for sleeping in the hallway. This wasn't as bad as it sounds. My bedroom wasn't hallway-sized. It was bedroom-sized and even had a window, but a hallway ran through it, so the landlady put up a curtain between the "hallway" part of my room and the "bedroom" part of my room. Who needs 4/4 walls anyways?

One of my house-mates was named Emerson. Emerson had a friend named Stella who unintentionally bred pet rats. Stella's landlord wouldn't let Stella keep her pet rats in Stella's apartment, so Emerson offered to house them temporarily. "Temporarily" became "until the end of our two-year lease". Did our landlady allow caged rats? I don't know. I never asked.

Emerson lived upstairs. Upstairs had two bathrooms. Emerson put the giant rat cage in the bathtub of one of the bathrooms. Sometimes the rats would escape the cage and Emerson would have to cajole the rats out from under the sink. That was fine by me because I lived downstars.

When we had parties[^1], I always warned guests, "Use the bathroom on the left. The bathroom on the right is full of rats". They were always very confused, as if "the bathroom is full of rats" meant something other than "the bathroom is full of rats". Sometimes they would look inside anyway and be surprised that the bathroom was full of rats.

[^1]: There was no music and no alcohol. These were "college parties" in the sense that the Zaatari Syrian War Refugee Camp, established in July 2012, is technically a summer home. My most vivid memory of these events was pausing *Primer (2004)* to examine the equations in the background. Our landlady loved us because we never damaged the property or provoked noise complaints.
# Open Thread Winter 2024/2025

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the [Highlights from the Sequences](https://lesswrong.com/highlights), a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/faq#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/faq). If you want to orient to the content on the site, you can also check out the [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/tag/open-threads?sortedBy=new). The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# Coin Flip

*This was a prose piece I performed for the 2024 KC Winter Solstice celebration, written by me*

![The scar](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/f100d53d6fbdb2431ae54ab69bd1abb490cd9053b9cb280a.jpg)

I have a scar running from right above my navel to the right end of my abdomen. You see, I was born premature, with a [volvulus](https://en.wikipedia.org/wiki/Volvulus), which means that my intestines were tangled and blue. I spent the first months of my life in the NICU in New York, my dad trying to get what time he could from a job that required travel to spend with me. The scar is from surgery to untangle my intestines - which is a thing we can do now!

All that has happened since then. Every laugh, every smile, every bite and every sip, every friend I have changed, every word I have ever uttered.

All of it almost didn't happen.

There's a morbid exercise where everyone in the room flips a coin, and we imagine that everyone that got tails died before 15. Just a few centuries ago, child mortality was a coin flip.

I don't need to flip a coin. I already know I got tails.

And yet.

And yet, I am still here.

Thank you.
# What's the best metric for measuring quality of life?

Currently, to get a drug approved by the FDA you need to prove that it helps with a particular illness and while having an acceptable side effect profile.

If you have a drug that improves people's quality of life but you aren't curing an ICD-11 accepted illness, you don't get your drug approved.

If the FDA would change and approve drugs for improving quality of life, how should they measure it? What's the best way to measure quality of life that's hard to goodhart?
# Why OpenAI’s Structure Must Evolve To Advance Our Mission

The section "The Future":

> As we enter 2025, we will have to become more than a lab and a startup — we have to become an enduring company. The Board’s objectives as it considers, in consultation with outside legal and financial advisors, how to best structure OpenAI to advance the mission of ensuring AGI benefits all of humanity have been:
> 
> 1.  **Choose a non-profit / for-profit structure that is best for the long-term success of the mission.** Our plan is to transform our existing for-profit into a Delaware [Public Benefit Corporation⁠](https://en.wikipedia.org/wiki/Benefit_corporation) (PBC) with ordinary shares of stock and the OpenAI mission as its public benefit interest. The PBC is a [structure⁠](https://www.theinformation.com/articles/musks-xai-incorporates-as-benefit-corporation-with-positive-impact-goal) [used⁠](https://openai.com/index/elon-musk-wanted-an-openai-for-profit/) [by⁠](https://en.wikipedia.org/wiki/Inflection_AI) [many⁠](https://www.anthropic.com/company) [others⁠](https://corpgov.law.harvard.edu/2022/02/18/converting-to-a-delaware-public-benefit-corporation-lessons-from-experience/) that requires the company to balance shareholder interests, stakeholder interests, and a public benefit interest in its decisionmaking. It will enable us to raise the necessary capital with conventional terms like others in this space.
> 2.  **Make the non-profit sustainable.** Our plan would result in one of the best resourced non-profits in history. The non-profit’s significant interest in the existing for-profit would take the form of shares in the PBC at a fair valuation determined by independent financial advisors. This will multiply the resources that our donors gave manyfold.
> 3.  **Equip each arm to do its part.** Our current structure does not allow the Board to directly consider the interests of those who would finance the mission and does not enable the non-profit to easily do more than control the for-profit. The PBC will run and control OpenAI’s operations and business, while the non-profit will hire a leadership team and staff to pursue charitable initiatives in sectors such as health care, education, and science.
> 
> We’ve learned to think of the mission as a [continuous objective](https://openai.com/index/planning-for-agi-and-beyond/) rather than just building any single system. The world is moving to build out a new infrastructure of energy, land use, chips, datacenters, data, AI models, and AI systems for the 21st century economy. We seek to evolve in order to take the next step in our mission, helping to build the AGI economy and ensuring it benefits humanity.
# Has Someone Checked The Cold-Water-In-Left-Ear Thing?

I somewhat recently read [these](https://www.lesswrong.com/posts/ZiQqsgGX6a42Sfpii/the-apologist-and-the-revolutionary) [posts](https://www.lesswrong.com/posts/iA2xt6ywvJTzKQmiz/to-become-more-rational-rinse-your-left-ear-with-cold-water), and figured this was enough evidence that it might be worth checking whether people become better calibrated for a short while after very cold water is poured in their left ear. For example: does doing that noticeably improve scores on the [calibration game](https://acritch.com/credence-game/) for a few minutes? Has this already been tested in some fashion aside from [whether it changes existing strong beliefs in rationalists](https://www.lesswrong.com/posts/ZiQqsgGX6a42Sfpii/the-apologist-and-the-revolutionary?commentId=2ZPqxPWpotCX2JDAt)? If it hasn't, a simple n=1 experiment seems fairly cheap to try, so I'll probably go for it over the next few days.
# The Legacy of Computer Science

> "Computer Science" is not a science, and its ultimate signiﬁcance has little to do with computers. The computer revolution is a revolution in the way we think and in the way we express what we think. The essence of this change is the emergence of what might best be called procedural epistemology—the study of the structure of knowledge from an imperative point of view, as opposed to the more declarative point of view taken by classi-
cal mathematical subjects. Traditional mathematics provides a framework for dealing precisely with notions of "what is." Computation provides a framework for dealing recisely with notions of "how to".   
> -- Sussman

Start reading on page 3 from "We have been programming universal computers for about 50 years."
# 2025 Prediction Thread

2024 is drawing to a close, which means it's an opportune time to make predictions about 2025. It's also a great time to put probabilities on those predictions, so we can later prove our calibration (or lack thereof).

We just shipped a LessWrong feature to make this easy. Simply highlight a sentence in your comment, and click the crystal-ball icon on the toolbar to turn it a prediction that everyone (who's logged in) can put probability estimates on. The result will look like this:

Prediction

Some more probabilities that seem cool to elicit (basically all about AI, because that's what's on my mind, but it would be great to have some less AI focused predictions from others)[^kahhyc5o0bc]:

Prediction

Prediction

Prediction

Prediction

Prediction

Prediction

Prediction

Prediction

Prediction

Prediction

Prediction

[^kahhyc5o0bc]: Unless otherwise specified assume all predictions are about the state of the world at midnight PT, Dec 31st 2025. Also some things won't be perfectly operationalized. Assume that I am going to be judging the resolution using my best judgement.
# The low Information Density of Eliezer Yudkowsky & LessWrong

**TLDR:**  
I think Eliezer Yudkowsky & many posts on LessWrong are failing at keeping things concise and to the point.

**Actual post:**

*I think the content from Eliezer Yudkowsky & on LessWrong in general is unnecessarily wordy.*

A **counter**example of where Eliezer Yudkowsky actually managed to get to the point concisely was in this Ted Talk, where he had the **external** **constraint** of keeping it to **10 minutes**:

An example of a **concise** post in a forum can be found here, from 2013:

*   [https://news.ycombinator.com/item?id=5248289](https://news.ycombinator.com/item?id=5248289)

Examples of posts, which showcase the **wordiness** of LessWrong:

*   [https://www.lesswrong.com/posts/TjyyngWFYvQWPpNNj/eliezer-yudkowsky-is-frequently-confidently-egregiously](https://www.lesswrong.com/posts/TjyyngWFYvQWPpNNj/eliezer-yudkowsky-is-frequently-confidently-egregiously)
*   [https://www.lesswrong.com/posts/8HYJwQepynHsRKr6j/critical-review-of-christiano-s-disagreements-with-yudkowsky](https://www.lesswrong.com/posts/8HYJwQepynHsRKr6j/critical-review-of-christiano-s-disagreements-with-yudkowsky)
*   [https://www.lesswrong.com/posts/GqeBEB5voGCYPu4hs/my-critique-of-eliezer-s-deeply-irrational-beliefs](https://www.lesswrong.com/posts/GqeBEB5voGCYPu4hs/my-critique-of-eliezer-s-deeply-irrational-beliefs)

  
**Why I think this matters:**

*   I think the long books Eliezer Yudkowsky has published were not the most optimal way to convey the threat of AGI to humanity.
*   I do **not** think that his books should not have been made. They are good for some people, this group of humans that has the time & motivation to read long books.
*   I think a more concise version would reach more people and would be more effective, e.g. something similar to the TedTalk YouTube video.
*   I think most people just google stuff & then read the shortest summaries quickly or watch shorter YouTube videos.
*   I think there are many benefits of keeping things very elaborate, e.g. it makes sure people really do not misunderstand your point on a crucial matter.
*   I think this is not a *pure* tradeoff situation. I think often Eliezer Yudkowsky & LessWrong can be more concise while still getting the **same exact points** across.
# Genetically edited mosquitoes haven't scaled yet. Why?

A post on difficulty of eliminating malaria using gene drives: "I worked on gene drives for a number of years jointly as a member of [George Church](https://arep.med.harvard.edu/) and [Flaminia Catteruccia](https://hsph.harvard.edu/research/catteruccia-lab/)’s labs at Harvard. Most of my effort was spent primarily on an idea for [an evolutionary stable gene drive](https://elifesciences.org/articles/03401.pdf), which didn’t work but [we learned some stuff](https://www.nature.com/articles/s41598-024-54498-8), and I developed a sense for the real challenges with gene drives. It’s something that comes up often when I meet new people in the bio community, so this is my attempt at laying out my perspective. I will be upfront and say I haven’t worked in this field since 2019, and though I’ve tried to update my understanding based on 2024 literature, I might have things wrong based on the bleeding edge."
# Fireplace and Candle Smoke

We celebrated New Year's Eve at my dad's, including a fire in the fireplace. I was curious how much the wood smoke went up the chimney vs collecting in the room, and decided to take some measurements. I used the [M2000](https://temtopus.com/products/temtop-m2000-air-quality-detector-co2-sensor-professional-hcho-co2-pm2-5-pm10-monitor?variant=40628808876080) that I got when investigating whether a ceiling fan could be [repurposed as an air purifier](https://www.jefftk.com/p/ceiling-air-purifier).

Here's what I found:

[![](https://www.jefftk.com/fireplace-and-candles.png)](https://www.jefftk.com/fireplace-and-candles-big.png)

I started the meter running at 4:30pm, and we started the fire at about 5:30pm. I didn't write down the specific time because I thought it would be evident from the chart \[1\] but actually I can't see it at all.

Then at 6:45pm we lit Hanukkah candles, and the smoke from the matches being blown out had a very sharp effect. Particulate levels stayed high for the rest of the time, with both the fireplace and candles, which I attribute to the candles.

  
\[1\] Several years ago I remember reading Sam Harris' blog post [The Fireplace Delusion](https://www.samharris.org/blog/the-fireplace-delusion), which argues that while we consider wood fires to be wholesome they're actually worse than smoking. And argues that this feeling of "wait, but wood fires are good!" is useful for understanding what religious folks are thinking when presented with atheism. Several years later his post had gotten jumbled in my head into saying that fireplace fires cause bad air quality in your own home, and so when I ran this experiment I was expecting to see quite high levels. On rereading, however, he spends a lot of time talking about externalities: the wood smoke that goes up my chimney goes, in part, into many other people's houses, causing a small bit of harm in each. So no conflict there.
# Economic Post-ASI Transition

Who's done high quality work / can tell a convincing story about managing the economic transition to a world where machines can do every job better than humans?

Some common tropes and why I don't think they're good enough:

*   "We've always managed in the past. Take the industrial revolution for example. People stop doing the work that's been automated and find new, usually better-compensated work to do." This is true, and I think it will probably be an important component of the transition. But it's clearly not sufficient if machines are better than humans at *everything*.
*   "Tax AI (developers?) to pay for UBI." Again, something in this vein will probably be part of the solution. But:
    *   (a) UBI hasn't been well-tested.
    *   (b) I don't think the math works out if / when AI companies dominate the economy, since they'll capture more and more of the economy unless tax rates are high enough that everyone else receives more through UBI than they're paying the AI companies.
    *   (c) It doesn't have enough detail.
*   *Worldcoin*. I think the idea is similar to the UBI story, but again it needs more detail.

Who has thought about this really deeply / well?

Note that for the purpose of this question, assume a world where alignment basically works (we can debate that question elsewhere).
# Playing with Otamatones

A couple months ago Nora (3y) got very into [Otamatones](https://en.wikipedia.org/wiki/Otamatone). She wanted to watch lots of videos, primarily [TheRealSullyG](https://www.youtube.com/@TheRealSullyG/videos). She asked for one for Christmas, and so did I:

[![](https://www.jefftk.com/otamatones-ruthie-and-nora.jpg)](https://www.jefftk.com/otamatones-ruthie-and-nora-big.jpg)

They're a lot of fun, but I haven't yet figured out if it's an instrument I'll ever be able to reliably play in tune.

The basic idea is you have a touch sensitive ribbon which triggers a pitch dependent on the location of your finger, and then you squeeze the cheeks to control the volume. Nora's small pink one is the standard size, and I find it quite hard to use with grownup fingers. My big red one is the "deluxe" model, and at the larger size the ribbon is quite playable.

Lily made a video of the [Mary Ellen Spider](https://mudcat.org/thread.cfm?threadid=42899) song with an Otamatone actor, though (perhaps for the best) the instrument is not featured musically:

I measure both models as having 29-31 half steps (2.5 octaves) depending on which setting you're using:

*   Standard, low: A#1 (59Hz) to D#4 (312Hz)
*   Standard, medium: G#3 (213Hz) to D6 (1,184 Hz)
*   Standard, high: A#5 (920Hz) to D#8 (4,980Hz)
*   Deluxe, low: F1 (44Hz) to C4 (265Hz)
*   Deluxe, medium: D#3 (158Hz) to B5 (980Hz)
*   Deluxe, high: E5 (671Hz) to A#7 (3,676Hz)

Overall it's a fun and silly instrument, though there are a few smallish places it could be better:

*   It's a little quiet for group playing, much quieter than a fiddle or mandolin. If I'm playing with others I'm generally on the loudest setting and wishing it were a little louder. This is less of a problem now when I'm kind of terrible at it, but if I do get good it will be limiting.
    
*   You can hook the "deluxe" model up to an external speaker, at the cost of losing control over cheek articulation. And I was hoping the bass would sound better with a speaker intended for that range, but it turns out that's not a voice they've gone for.
    
*   Instead of an octave selector I want either a pitch range knob or a five-setting switch: three settings is often not enough to place the 2.5 octave range where I want it.
    

Still, I'm happy! And now Anna is asking for one for her birthday, which would let us play as a trio.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0EthDsKc6U7JSs8DKTCD9wzeLjzjKQdzgV7ve1KrH954iWsSoMrDUuMnN5xgPzcXVl), [mastodon](https://mastodon.mit.edu/@jefftk/113760480391298291), [bluesky](https://bsky.app/profile/jefftk.com/post/3lertpkeg6s2e)*
# Orange and Strawberry Truffles

Growing up my dad would make chocolate truffles, flavored with a range of alcohols and fruit extracts. These are tasty, but I've been interested in making ones that have less of a liquor flavor. This time last year I made some with [freeze-dried raspberries](https://www.jefftk.com/p/freeze-dried-raspberry-truffles), which I think came out well. I continue to like those a lot, but this year wanted to try some additional flavors.

The first new flavor I tried was strawberry: freeze-dried powder from strawberries instead of raspberries. Unfortunately I didn't realize until well into the process that the flavor of strawberries is much more subtle, which meant I needed a kind of absurd quantity of strawberry powder. While my raspberry truffles were about 4% raspberry powder, I got my strawberry ones up to 27% (500g chocolate, 225g cream, 270g strawberry) before the flavor became clearly noticeable above the chocolate. And even then, the flavor is less interesting than the raspberry, and more artificial feeling. Adding this large a quantity also meant I ended up with a gummier texture than I prefer, though it's interesting for variety. None of these will be going to waste, though I probably also won't make them again.

The second new flavor I tried was orange, and I think this was much more successful. I added orange zest to cream, heated to just below boiling, let it steep for 20min, and then sieved it out before using the cream to make ganache. I used 18g of zest for 500g of chocolate and 250g of cream. While this doesn't sound like much (2% by weight) it's eight [Cara Cara](https://en.wikipedia.org/wiki/Cara_cara_navel) oranges worth, and has quite a strong flavor. I also added about 20g of freeze-dried orange powder, though I'm not sure this was worth it: I think I like the darker more bitter flavor of just the zest a bit better. I liked how these came out a lot, and will definitely be doing them again.

As in past years, it was fun to do them with the kids:

[![](https://www.jefftk.com/making-truffles-with-anna-and-nora.jpg)](https://www.jefftk.com/making-truffles-with-anna-and-nora-big.jpg)

(I've also stopped using the freezer—thanks [Claude](https://claude.ai)—which fixed my occasional issues with condensation getting water in the chocolate and causing seizing.)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0WodUpY9foLagCuzXwegifEmpvoXC2rtLdNbcjD5kGkZNCfFCNDYJamZc9vKEKZgJl), [mastodon](https://mastodon.mit.edu/@jefftk/113773227217161872), [bluesky](https://bsky.app/profile/jefftk.com/post/3lexitwchvk2b)*
# Maximizing Communication, not Traffic

As someone who writes for fun, I don't need to get people onto my site:

*   If I write a post and some people are able to get the core idea just from the title or a tweet-length summary, great!
    
*   I can include the full contents of my posts in my RSS feed and on FB, because so what if people read the whole post there and never click though to my site?
    

It would be different if I funded my writing through ads (maximize time on site to maximize impressions) or subscriptions (get the chance to pitch, probably want to tease a paywall).

Sometimes I notice myself [accidentally copying](https://en.wikipedia.org/wiki/Cargo_cult) what makes sense for other writers. For example, because I can't put full-length posts on Bluesky or Mastodon I write short intros and link my full post. Yesterday I initially drafted:

> It's common to flavor truffles with extracts, but I'd like less of a liquor flavor. This time last year I made some with freeze-dried raspberries, which I think came out well. I continue to like those a lot, and this year tried strawberry and orange zest. One worked a lot better than the other: \[link\]

This would have gotten more people to click through, but that shouldn't be my target. [Instead](https://mastodon.mit.edu/@jefftk/113773227217161872) I [posted](https://bsky.app/profile/jefftk.com/post/3lexitwchvk2b):

> ... and this year tried strawberry (eh) and orange zest (great!) \[link\]

No need to hold curiosity hostage.

It's common to criticize "clickbait", where a teaser entices and then doesn't deliver, but even reserving key information for the full article is a product of authors (needing to!) optimize for goals other than communicating to users. I like that this isn't a pressure that I'm under. Because our culture has so many who are under it, however, optimizing for communication can require noticing and intentionally avoiding common patterns.
# Is "hidden complexity of wishes problem" solved?

Can the problem described in the [hidden complexity of wishes](https://www.lesswrong.com/posts/4ARaTpNX62uaL86j6/the-hidden-complexity-of-wishes) (at least partially) be considered solved? I believe that current LLMs are perfectly capable of answering a question like "How do I save grandma from a burning house?" without suggesting any unexpected maximums in an underspecified utility function.  
  
However, I still think that the existence of LLMs capable of answering such questions is not sufficient proof of solving the problem (we do not have a robot capable of performing tasks of a similar level of difficulty to the 'saving grandma' task, with safety properties comparable to those that a human firefighter can provide when performing 'saving grandma' task). But can it at least be considered evidence supporting progress in that direction? This seems like a topic worth discussing, and I expect it has been discussed somewhere. If so, please link to it in answers.
# "We know how to build AGI" - Sam Altman

> We are now confident we know how to build AGI as we have traditionally understood it. We believe that, in 2025, we may see the first AI agents “join the workforce” and materially change the output of companies. We continue to believe that iteratively putting great tools in the hands of people leads to great, broadly-distributed outcomes.  
>   
> We are beginning to turn our aim beyond that, to superintelligence in the true sense of the word. We love our current products, but we are here for the glorious future. With superintelligence, we can do anything else. Superintelligent tools could massively accelerate scientific discovery and innovation well beyond what we are capable of doing on our own, and in turn massively increase abundance and prosperity.

More context in [this Bloomberg piece](https://www.bloomberg.com/features/2025-sam-altman-interview/?embedded-checkout=true&leadSource=reddit_wall).

> **What’s the threshold where you’re going to say, “OK, we’ve achieved AGI now”?**  
> The very rough way I try to think about it is when an AI system can do what very skilled humans in important jobs can do—I’d call that AGI. There’s then a bunch of follow-on questions like, well, is it the full job or only part of it? Can it start as a computer program and decide it wants to become a doctor? Can it do what the best people in the field can do or the 98th percentile? How autonomous is it? I don’t have deep, precise answers there yet, but if you could hire an AI as a remote employee to be a great software engineer, I think a lot of people would say, “OK, that’s AGI-ish.”  
> Now we’re going to move the goalposts, always, which is why this is hard, but I’ll stick with that as an answer. And then when I think about superintelligence, the key thing to me is, can this system rapidly increase the rate of scientific discovery that happens on planet Earth?
# Meal Replacements in 2025?

I'm considering meal replacements for 1-2 meals a day, and recall Soylent and Mealsquares were popular meal replacements ~5 years ago when I visited the Berkeley rationalist scene. I haven't found any recent posts discussing the newer options like Huel or about long-term effects. 

Does anyone have informed opinions on these things?
# (My) self-referential reason to believe in free will

I was recently talking with someone about the problem of free will, and I realised that for many years now I have always had the same response, without really ever soliciting broader critical feedback. The notion of free will here refers to a naive, libertarian, non-strictly-defined approach of "when I feel I make choices, I really had a choice", and all of the associated implied moral philosophy (laziness is a thing, I can be blamed for my choices etc.)

The starting assumption is that I want to believe in true things (I leave open the question of whether this epistemic duty is itself justified or not). I propose a trilemma, where exactly one of the following propositions holds:

1. Either the notion of 'free will' is meaningless, or
2. It is meaningful, and I do have in fact have free will, or
3. It is meaningful, but I happen to not have it

If (1) is true, then the whole discussion is moot: nothing can be true or false, and whatever I believe is equally justified.
If (2) is true, then I want to believe in having free will (since it is true that I have it).
If (3) is true, then "should" is a meaningless concept - there is no way I would be able to change my view one way or the other.

So, the only possible world where I get to make this choice is a world in which free will is true, so I should believe in it - and mostly ignore the debate about compatibilism, the nature of physics, dualism etc. Which is what I do.

One potential issue is that (1) can be true or false depending on the precise definition (but then what even *is* a precise definition?). Still,  I suspect that no matter which one I instantiate it with, as long as it is sensible, the general (self-referential) structure of the argument will stay the same.

*Thanks to Jakub S. for the feedback on this post and his suggestion to formalise this argument as the epistemic duty of maximising the probability of holding correct beliefs.*
# Incredibow

Back in 2011 I got sick of breaking the hairs on violin bows and ordered an [Incredibow](https://www.incredibow.com). The hair is polymer filament, and it's very strong. I ordered a 29" Basic Omnibow, Featherweight, and it's been just what I wanted. I think I've broken something like three hairs [ever](https://www.jefftk.com/p/assorted-pictures#:~:text=I%20love%20my,it%27s%20indestructible%20nature!), despite some rough chopping.

[![](https://www.jefftk.com/incredibow-good-as-new.jpg)](https://www.jefftk.com/incredibow-good-as-new-big.jpg)

Thirteen years, and it could be new yesterday.

While they don't play identically to traditional bows and it takes a bit of adjustment, if people had been playing with synthetic hair historically I really doubt horsehair would take off today.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid035DuBQMSZdHfzXxoi7cnPagPHFe6LQ7cDzdJpRm6ajo9hwezv96xS2BYY1tpVAp7gl), [lesswrong](https://lesswrong.com/posts/9GXFd4YZhWTNZ2j5a), [mastodon](https://mastodon.mit.edu/@jefftk/113784957324192268), [bluesky](https://bsky.app/profile/jefftk.com/post/3lf4pk72ubk2g)*
# Predicting AI Releases Through Side Channels

I recently explored whether we could predict major AI releases by analyzing the Twitter activity of OpenAI's red team members. While the results weren't conclusive, I wanted to share this approach in case it inspires others to develop it further.

The Core Idea
-------------

The idea came from **side-channel analysis** \- a technique where you gather information about a system by looking at indirect signals rather than the system itself. Think tracking factory electricity usage to estimate production volumes, or [stealing an LLM’s softmax layer through logprobs](https://arxiv.org/abs/2403.06634).

When OpenAI prepares to release a new AI model, their red team (safety and robustness testers) gets access to it a few weeks before launch. I hypothesized these intense testing periods might reduce red teamers' social media activity, creating a detectable pattern before releases.

How I Tested This
-----------------

1.  Identified ~30 red team members from OpenAI's system cards
2.  Scraped their Twitter timelines
3.  Analyzed activity patterns around known model release dates

What I Found
------------

The data showed a slight dip in Twitter activity before launches, but the signal was too weak to be reliable.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/75c235cb7ef5422008ccbd7653c551b20d8113dba24f5cae.png)

Limitations
-----------

Two main factors prevented more definitive results:

*   **Small sample size** (only ~30 accounts). To get more statistical power, we’d need to look at more employees or additional side channels.
*   **Twitter API restrictions** that limited data collection. To get more timeline data, you’ll need to find a Twitter dump.

Why This Approach Could Still Work
----------------------------------

Despite inconclusive results, this method of tracking "*side-channel signals*" \- indirect indicators like social media patterns, GitHub commits, or hiring changes - could prove valuable with refinements. Similar approaches [have worked in other fields](https://en.wikipedia.org/wiki/Tempest_(codename)).

Next Steps for Anyone Interested
--------------------------------

If you'd like to build on this idea, consider:

*   Expanding to other/larger companies (Google, Meta)
*   Exploring different indicators beyond Twitter
*   Developing ways to aggregate multiple weak signals

I won't be continuing this research myself, but I hope sharing these preliminary findings helps someone else take this idea further.

* * *

*Reworr works as a researcher at Palisade Research. The views expressed here are their own.*
# [Fiction] [Comic] Effective Altruism and Rationality meet at a Secular Solstice afterparty

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/fd764295ccf5922f08336dd62f371e730fac2a6b0994cdcc.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/5384704364a7287b58fc5c9aee06ebb581b9a51a20fa1f03.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8fa1031025d6dcd6fa0a3b68cc5ee8a37cc18aa990f98695.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/7fe305b08a2342d9346bc1cb75bf8e47deaa834a506a333e.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/78671697805946e01f662091e190ce7f29e98bac7110d919.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e281fb72ef10b21bbd1c1202b683b80e9d5b8ca452a4eb67.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/062e94321263887ae520c3465502e99bde618888145ea264.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/14c78bfe397dcc0f9b409d04c070c50789fb786101ce5a1b.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/dc97e3e95b8dc3a8790fcb5369a12454c488992dee890502.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d8bd86b747635333bca2af3919b490503bfaa63559b74fca.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8dea24a00292dcdf0811587edcf1bfdc1afec05c03060f0f.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/473cc1754a911fee2b9b7a86a67a5f402a6e5c0156e1cca5.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/7485bdef05cf9eae27c81707cb52690fb54b55661bd5df99.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/9444bfebfb5f500202fa81f2c13919b84a284578eb47efea.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/087d518e34c8a8bac42447cd6dffb43c455146680bd3370b.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/82ade69477e49d4e4079f3d346ad318267769d89a22eae0c.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/5f16ee179411b2452417674b9cc66a34bbb248996d6e7179.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/65be3e9a5bcf3f8fcc82f111626828c14e45a044332e490a.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/51663119689c67c91757c8c9c47878d03d52ba6ceb359c67.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/942d8302bb6a1095b58e64ca52976398d01ba1045c0fffeb.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8340182f6bac88f78b66a5262340d10dda1b981b9744e8df.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/0558d5002e98933ef47d51332675934e18606ebb3cff5bcd.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/5acc4ab731055d3309f17232d4c69947a4a80b546da0c43e.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/265c3aff9b9d7d813e4a555443b5ca7342211183d6512028.png)

(Both characters are fictional, loosely inspired by various traits from various real people. Be careful about combining kratom and alcohol.)
# Will bird flu be the next Covid? "Little chance" says my dashboard.

Rob and I built a bird flu risk dashboard.

You can check it out here:

[birdflurisk.com](http://birdflurisk.com/)

![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe718b692-0915-479a-b0b4-512e84d81681_2982x1670.png)

There is a box to be emailed if the index changes significantly. Hopefully this helps some people. 

Happy to answer questions/make changes.
# Disagreement on AGI Suggests It’s Near

If I’m planning a holiday to New York (and I live pretty far from New York), it’s quite straightforward to get fellow travellers to agree that we need to buy plane tickets to New York. Which airport? Eh, whichever is more convenient, I guess. Alternatively, some may prefer a road trip to New York, but the general direction is obvious for everyone.

However, as the holiday gets closer in time or space, the question of what we actually mean by a holiday in New York becomes more and more contentious. Did we mean New York State or New York City? Did we mean Brooklyn or Broadway? Which Broadway theater? Which show? Which seats?

By my estimation, the fact that the question of whether AGI has been achieved is so broadly contentious shows that we are so close to it that the term has lost its meaning, in the same way that “Let’s go to New York!” loses its meaning when you’re already standing in Times Square.

It’s time for more precise definitions of the space of possible minds that we are now exploring. I have my own ideas, but I’ll leave those for another post…
# Bridgewater x Metaculus Forecasting Contest Goes Global — Feb 3, $25k, Opportunities

For the second consecutive year, Metaculus is collaborating with Bridgewater Associates — the world’s largest hedge fund and a leader in global asset management — to host a forecasting competition offering $25,000 in prizes and potential opportunities with Bridgewater. While last year's competition was limited to participants in the United States, this year we’re excited to welcome forecasters from around the globe.

[**You must register to compete**](https://www.metaculus.com/bridgewater/) — then, get in shape with our [warmup questions](https://www.metaculus.com/tournament/bridgewater-warmup/) to prepare for the competition kickoff February 3rd. We'll message you before the first round of questions launch. 

Learn more about the contest structure, the open and undergraduate leaderboards, and details on how questions are scored [here](https://www.metaculus.com/bridgewater/how-it-works/). 

![](https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/ujdRjavy49d3eyDaB/tmjfx8kl4i0hfdvb0u3a)

[You can find the questions and results from last year's competition here.](https://www.metaculus.com/tournament/3067/)

### One Contest, Two Leaderboards: Undergraduate and Open Competitions

This contest features a unified set of 50 forecasting questions, with two distinct leaderboards: one for all participants in the Open Competition and another specifically for undergraduate students in the Undergraduate Competition.

### $25,000 in Prizes

Each competition boasts a $12,500 prize pool, totaling $25,000 across both leaderboards. The top 125 forecasters in each competition are eligible for prizes. Undergraduate participants will be ranked on both leaderboards and can win prizes in both categories. Prizes will be distributed after the contest ends and participants complete identity verification.

This year, there is no guaranteed $50 minimum prize per participant, meaning top performers are likely to receive a larger share of the prize pool, and fewer than 125 prizes may be awarded in each competition.

### Undergraduate Competition and Leaderboard

Open exclusively to currently enrolled undergraduate students, the Undergraduate Competition offers a dedicated $12,500 prize pool. Undergraduate participants are automatically included in the Open Competition as well, providing dual opportunities to compete and showcase their skills. Outstanding performance in this category could capture the attention of the Bridgewater recruiting team.

### Open Competition and Leaderboard

The Open Competition is open to everyone, regardless of background or experience, with its own $12,500 prize pool. Both seasoned and novice forecasters are encouraged to participate, demonstrate their forecasting prowess, and potentially earn an invitation to meet with the Bridgewater recruitment team.
# What is the most impressive game LLMs can play well?

*Epistemic status: This is an off-the-cuff question.*

~5 years ago there was a lot of exciting progress on game playing through reinforcement learning (RL). Now we have basically switched paradigms, pretraining massive LLMs on ~the internet and then apparently doing some really trivial unsophisticated RL on top of that - this is successful and highly popular because interacting with LLMs is pretty awesome (at least if you haven't done it before) and they "feel" a lot more like A.G.I. Probably there's somewhat more commercial use as well via code completion (and some would say many other tasks, personally not really convinced - generative image/video models will certainly be profitable though). There's also a sense in which they are clearly more general - e.g. one RL algorithm may learn many games but there's typically an instance per game not one integrated agent. You can just ask an LLM in context to play some games.

However, I've been following moderately closely and I can't seem to think of any examples where LLMs really pushed the state of the art in narrow game playing  - how much have LLMs contributed to RL research? For instance, will adding o3 to the stack easily stomp on previous Starcraft / go / chess agents?