Harley-ml/lesswrong huggingface dataset:
# PIBBSS Fellowship 2025: Bounties and Cooperative AI Track Announcement

We're excited to announce that the [PIBBSS Fellowship 2025](https://www.lesswrong.com/posts/qohzWNLdMi9KYz5Cq/apply-to-the-2025-pibbss-summer-research-fellowship) now includes a dedicated Cooperative AI track, supporting research that advances our understanding of cooperation in artificial intelligence systems. We are also announcing 300 USD bounties for each referral that becomes a Fellow. Read below for details.

**What is the Cooperative AI Track?**
-------------------------------------

Thanks to the support from the [Cooperative AI Foundation](https://www.cooperativeai.com/), PIBBSS is launching a thematic track within its usual fellowship. This program focuses on research aimed at improving the cooperative intelligence of advanced AI systems. We'll select up to 6 fellows to work on projects that bridge their expertise in cooperation, collective behavior, and multi-agent systems with AI safety research. The fellows will do everything together with the general cohort, with the added research focus on cooperative AI.

**Research Areas** We're particularly interested in are those covered in the [Grants page of Cooperative AI Foundation](https://www.cooperativeai.com/grants/2025), but we highlight:

*   Understanding and evaluating cooperation-relevant capabilities
*   Understanding cooperation-relevant propensities
*   Multi-agent interactions and emergent behavior
*   Information asymmetries and transparency
*   Fundamental research advancing our understanding of cooperation in complex systems

**Who Should Apply?** As is the case with the general track of the Fellowship, we welcome researchers from any background. This track would however likely be a better fit for researchers from fields studying cooperation and collective behavior, including but not limited to:

*   Game theory and multi-agent systems
*   Behavioral economics
*   Network science
*   Political science (especially coalition formation/cooperation)
*   Organizational psychology
*   Anthropology of cooperation
*   Collective behavior in biological systems

**Fellowship Details (same as the general track, see **[**here**](https://pibbss.ai/fellowship/)**)**

*   Duration: 3 months (June-September 2025)
*   Location: \[Berkeley or London TBC\]
*   Stipend: $3,000/month plus $1,000/month housing allowance if relocation is needed
*   Full program support, including mentorship, office space, and community access

**The Application Process** closes on January 26th, 2025, so [apply soon](https://pibbss.ai/fellowship/)!

Bounty Program
--------------

We want people to seriously think about who they can refer to the Fellowship and be rewarded for their time. If your referral is successful, we will pay you 300 USD (or donate it to a charity of your choice in your name). In the application form, make sure the person puts in your email so that we can reach out to you once Fellows are confirmed. Thanks in advance to everyone who spreads the word to qualified candidates!
# Can we rescue Effective Altruism?

Last year Timothy Telleen-Lawton and I recorded a [podcast episode](https://www.youtube.com/watch?v=pIR9upHWq2k) talking about why I quit Effective Altruism and thought he should too. This week we have a new episode, talking about what he sees in Effective Altruism and the start of a road map for rescuing it. 

[Audio recording](https://youtu.be/07rUWuJpP1k)

[Transcript](https://docs.google.com/document/d/1S7mVxU8fnY02myv966bNXO1YHLz5PxFI/edit)

Thanks to everyone who listened to the last one, and especially our [Manifund donors](https://manifund.org/projects/elizabeth-and-timothy-podcast-on-values-in-effective-altruism), [my Patreon patrons](https://www.patreon.com/acesounderglass), and the EAIF for funding our work.
# AI for medical care for hard-to-treat diseases?

With LLM-based AI passing benchmarks that would challenge people with a Ph.D in relevant fields, I'm left wondering what they can do for real-world problems for which *nobody* knows the correct answer, such as how to treat potentially fatal medical conditions with no known cure.

Are we at the point where AI can do better than curated medical references intended for doctors, such as UpToDate, at coming up with treatment plans for people with hard-to-treat and frequently fatal medical conditions, like [Jake Seliger](https://jakeseliger.com/) who died of squamous cell carcinoma, [DF](https://www.lesswrong.com/posts/CBWSDdzjqfnexBurB/going-out-with-dignity) who delayed his death from fatal familial insomnia, or my late wife, who suffered from calciphylaxis before her death?
# Lighthaven Sequences Reading Group #17 (Tuesday 01/14)

**NB: We're continuing at Lighthaven! Thanks to MATS for giving us the space to keep going.**

Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then split off into randomized groups for some sequences discussion. Please do the reading beforehand - it should be no more than 20 minutes of reading.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This meetup will also have dinner provided! We'll be ordering pizza-of-the-day from Sliver (including 2 vegan pizzas). Please RSVP to this event so we know how many people to have food for.

We're roughly working through [the sequences highlights](https://www.lesswrong.com/highlights). The mandatory readings this week are

1.  [Absence of Evidence Is Evidence of Absence](https://www.lesswrong.com/posts/mnS2WYLCGJP2kQkRn/absence-of-evidence-is-evidence-of-absence)
2.  [Science Doesn't Trust Your Rationality](https://www.lesswrong.com/posts/5bJyRMZzwMov5u3hW/science-doesn-t-trust-your-rationality)
3.  [When Science Can't Help](https://www.lesswrong.com/posts/wzxneh7wxkdNYNbtB/when-science-can-t-help)
4.  Optional: [Faster Than Science](https://www.lesswrong.com/posts/xTyuQ3cgsPjifr7oj/faster-than-science)

The posts should take around 15-20 minutes to read if you've read them before, and 30-60 mins if it's your first time.

The event starts at 6:30, and doors open 6pm (yes we will let you in early if you get here). At 6:30 we'll welcome people, give a few announcements, and then split into groups. Pizza will be served at about 7:30pm, after which point we'll hangout around the fireside as late as we feel like.

If you'd like to help us keep dinner going, [you can donate here](https://www.lesswrong.com/donate) (dinner costs about $200 each time, we recommend $5-$15 as a donation amount per person).

You *can* come without having read the essays from the sequences, we do want you to get to join, but then you will have to do a **punishment**. (And then you still have to catch up on reading the essays during the meetup.)

If you're ever having a hard time getting in the front door, you can call me (Ben) at five one oh, nine nine eight, four seven seven one, or Ronny at two oh one, four oh six, six eight seven two, and we'll come and get you.

**Some questions to ask yourself about the essays as you read them**

*   What's the most important point in the essay?
*   What's the weakest point in the essay? Or what is the essay wrong about?
*   Can you think of a way to apply the ideas in this essay to your own life?

**For the future**

This is a weekly meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.
# No one has the ball on 1500 Russian olympiad winners who've received HPMOR

We have contact details and can send emails to 1500 students and former students who've received hard-cover copies of HPMOR (and possibly Human Compatible and/or The Precipice) because they've won international or Russian olympiads in maths, computer science, physics, biology, or chemistry.

This includes over 60 [IMO](https://en.wikipedia.org/wiki/International_Mathematical_Olympiad) and [IOI](https://en.wikipedia.org/wiki/International_Olympiad_in_Informatics) medalists.

This is a pool of potentially extremely talented people, many of whom have read HPMOR.

I don't have the time to do anything with them, and people in the Russian-speaking EA community are all busy with other things.

The only thing that ever happened was an email sent to some kids still in high school about the Atlas Fellowship, and a couple of them became fellows.

I think it could be very valuable to alignment-pill these people. I think for most researchers who understand AI x-risk well enough and can speak Russian, even manually going through IMO and IOI medalists, sending those who seem more promising a tl;dr of x-risk and offering to schedule a call would be a marginally better use of their time than most technical alignment research they could be otherwise doing, because it plausibly adds highly capable researchers.

If you understand AI x-risk, are smart, have good epistemics, speak Russian, and want to have that ball, please DM me on LW or [elsewhere](https://contact.ms).

To everyone else, feel free to make suggestions.
# How quickly could robots scale up?

Once robots can do physical jobs, how quickly could they become a significant part of the work force?

Here's a couple of fermi estimates, showing (among other things) that converting car factories might be able to produce 1 billion robots per year in under 5 years.

Nothing too new here if you've been following Carl Shulman for years, but I thought it could be useful to have a reference article. Please let me know about corrections or other ways to improve the estimates.
# Mini Go: Gateway Game

There are lots of ways to categorize board games, but an axis I care a lot about is accessibility: how much of an investment is learning a game? [Race For the Galaxy](https://boardgamegeek.com/boardgame/28143/race-for-the-galaxy) and [Power Grid](https://boardgamegeek.com/boardgame/2651/power-grid) are great games, but I'd expect to spend 15+ min teaching before we could play. [Set](https://boardgamegeek.com/boardgame/1198/set) or [Anomia](https://www.jefftk.com/p/better-anomia), though, I could explain in a minute or two.

Games you can teach quickly are great in a casual context: people wander over, and you can get them playing right away. And one of my favorite casual games is [Go](https://en.wikipedia.org/wiki/Go_(game)).

If you know Go this is a surprising claim: we're talking about one of the deepest board games humans (and [others](https://www.jefftk.com/p/aliens-dont-play-chess)?) play. It's famously complex, with two decades separating computers surpassing humans [in chess](https://en.wikipedia.org/wiki/Deep_Blue_(chess_computer)) and then [in Go](https://en.wikipedia.org/wiki/AlphaGo). And yet, if you [shrink the board](https://www.jefftk.com/p/small-go-boards) to where a game just takes a few minutes, it's a great party game:

[![](https://www.jefftk.com/mini-go-as-party-game.jpg)](https://www.jefftk.com/mini-go-as-party-game-big.jpg)

*not a Go meetup; the board games table at a [contradance weekend](https://contranooga.us/)*

The rules of Go are [very simple](https://www.jefftk.com/p/simplest-interesting-game), and it is only their interaction on a large board that gives Go it's depth. I can get someone playing on a 5x5 in a couple minutes, and we'll usually play a few quick games in a row. At this scale people can draw strongly on their general purpose reasoning while they're building up their intuitive sense. Then I'll get them playing someone else who's never played before. Once black starts consistently winning on 5x5, we go to 7x7. And if someone shows up who already knows how to play we can do 9x9.

The game does feel different at the smaller scale, with more focus on capturing and less on territory, but the learning curve is fantastically welcoming. I think the broader Go community has been making a serious mistake in dismissing sizes below 9x9 as not worth playing.
# Do humans really learn from "little" data? 

### How much data does it take to pretrain a (human) brain? I conducted a (fairer) Fermi estimate.

The post goes through the following questions:

*   How long does it take to grow a human brain?
    *   How many waking seconds do we have in our life?
    *   How many “tokens” or “data points” does a human brain process in a second?
        *   Can we simply count the spikes?
*   How many bits (spikes and non-spikes) does it take for the brain to process 1 sensory “piece of information”?
*   How do those numbers stack up against LLMs?

To get to this conclusion table:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/0c6bd751042dd0cbee22c8580e9c9d0b5de579c094db50ad.png)
# Why abandon “probability is in the mind” when it comes to quantum dynamics?

A core tenet of Bayesianism is that probability is in the mind. But it seems to me that even hardcore Bayesians can waffle a bit when it comes to the possibility that quantum probabilities are irreducible physical probabilities.

I don’t know enough about quantum physics to lay things out in any detailed disagreement, but it seems to me that if one finds a system that one cannot consistently make predictions for, it means we lack the knowledge to predict the systems, not that the system involves physical, outside-the-mind probabilities. For example, I could never predict the exact pattern of raindrops the next time it rains, but no one argues that that means those probabilities are therefore physical.

  
What is the Bayesian argument, if one exists, for why quantum dynamics breaks the “probability is in the mind” philosophy?
# I'm offering free math consultations!

You can schedule them with me at this link: [https://calendly.com/gurkenglas/consultation](https://calendly.com/gurkenglas/consultation)

We can discuss whatever you're working on, such as math or code, but usually people end up having me watch their coding and giving them tips.

Here's how this went [last time](https://www.lesswrong.com/posts/MHqwi8kzwaWD8wEQc/would-you-like-me-to-debug-your-math#comments):

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/c9722659c04ceb64aae20ba9962331b8329643025c21dcbe.png)

To my memory, almost every user wrote such praise[^qv1cypt1rhi]. Unfortunately, "almost every user" means four. I wish my schedule was chock full of them, hence this post. If you think I'm overqualified, please do me the favor of testing that hypothesis by booking a session and seeing if I regret it. I never have.[^wf671ntx2]

The advice that was most useful to most people
----------------------------------------------

was on how to use (graphical) debuggers. The default failure mode I see in a programmer is that they're wrong about what happens in their program. The natural remedy is to increase the bandwidth between the program and the programmer. Many people will notice an anomaly and then grudgingly insert print statements to triangulate the bug, but if you measure as little as you can, then [you will tend to miss anomalies](https://www.lesswrong.com/posts/9kNxhKWvixtKW5anS/you-are-not-measuring-what-you-think-you-are-measuring). When you step through your code with a debugger, getting more data is a matter of moving your eyes. So set up an IDE and watch what your code does. And if a value is a matrix, the debugger had better matshow it to recruit your visual cortex.

If your math involves no coding... you have *even less* chance to notice anomalies. I assure you that there is code that would yield data about what your math does, you just don't have it. I prescribe that you write it.

Or have AI write it. Using more AI is a recommendation I give lots of people these days. Copilot suggests inline completions of the code you're writing; a sufficient selling point is that this can suggest libraries to import to reduce your tech debt, but I expect it to show you other worthwhile behaviors if you let it.

[^qv1cypt1rhi]: I'm kinda sad that in mentioning the fact I am making it slightly less magical in the future, but I guess on a consequentialist level this is an appropriate time to cash it in. 

[^wf671ntx2]: If you think I'm underqualified: My specialty is category theory, which studies the patterns that pop up across math, which is useful for stealing results from other math branches. I expect my highest impact so far came from pointing Infrabayesianism-Diffractor towards developing a concept of infrafunctions.
# The Philosophical Glossary of AI

> It is hard to know what to make of claims like “LLMs are intelligent”, “we have reached AGI”, or “AI’s outputs are biased” without a grasp of the definitions of the terms ‘intelligent’, ‘AGI’, and ‘bias’. And yet, many do just this. Interdisciplinary debate would be easier and more fruitful if a common set of working definitions were used. This glossary aims to contribute to that end.

> \[T\]he glossary is underpinned by a rigorous editorial process. To ensure accuracy:
> 
> *   Entries are written by suitably qualified academics and practitioners with relevant expertise.
> *   All entries are peer-reviewed by The Glossary’s editorial team, composed of internationally recognised researchers with earlier career academics.
> *   Entries are often also reviewed by a wider network calling on leaders in specific fields.
# Lecture Series on Tiling Agents

For [my AISC](https://www.lesswrong.com/posts/7AzexLYpXKMqevttN/seeking-collaborators), I'll[^sgbu32snrpb] be presenting more details about the research every Thursday for approximately the next three months. If you are interested in listening in, [here is a calendar link](https://calendar.app.google/epeuejABShpVPnyU7).

EDIT:

The calendar link apparently doesn't invite people to the recurring event; I'm not sure I can do that with google calendar unfortunately. The subsequent meetings will be at the same time-slot each week and can be attended via this link: 

https://meet.google.com/bwp-nkck-ros

[^sgbu32snrpb]: Maybe there will be guest speakers at some point, EG, the AISC mentees.
# Predict 2025 AI capabilities (by Sunday)

Until **this Sunday**, you can submit your 2025 AI predictions at [**ai2025.org**](https://ai2025.org). It’s a forecasting survey by [AI Digest](https://theaidigest.org/) for the 2025 performance on various AI benchmarks, as well as revenue and public attention. 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/dc9542c811ca41655a8d9e66be250af08218fbf2bab6e940.png)

You can share your results in a picture like this one. I personally found it pretty helpful to learn about the different benchmarks, and also to think through my timelines estimates. 

The survey will close on **Sunday, January 19th (anywhere on Earth)**.

If you know any AI public intellectuals or discourse influencers who might be interested in submitting the survey, please encourage them to do so!

Survey link: [**ai2025.org**](https://ai2025.org)
# Where should one post to get into the training data?

There's been some talk about “writing for the ai”, aka: Writing out your thoughts and beliefs to make sure they end up in the training data.

LessWrong seems like an obvious place that will be scraped. I expect when I post things here, they’ll be eaten by the Shoggoth.

But what about things that don’t belong on LW? 

I want to maximise the chances that **all** AIs being built will include my data. So posting to Twitter (X) seems like I’ll just be training Grok???

What about a personal blog I start on a website I own? 
Does making the robots.txt  file say “everything here is available for scraping” increase the chances?
Does linking to that website in more places increase the chances?

I feel like I’m lacking a lot of knowledge here. I encourage responses even if they feel like obvious things to you.
# Feature request: comment bookmarks

Sometimes I see a comment I'd like to bookmark, but currently the only ways to save a comment are by subscribing to its replies (which sometimes produces unwanted notifications and requires me to check a different profile section than the rest of my bookmarks) or bookmarking the post it's attached to (which can be inconvenient for posts with many comments, and requires me to remember which comment the bookmark is actually for). Would it be possible to add a native bookmark option for comments?
# List of AI safety papers from companies, 2023–2024

I'm collecting (x-risk-relevant) safety research from frontier AI companies published in 2023 and 2024: [https://docs.google.com/spreadsheets/d/10_dzImDvHq7eEag6paK6AmIdAGMBOA7yXUvumODhZ5U/edit?usp=sharing](https://docs.google.com/spreadsheets/d/10_dzImDvHq7eEag6paK6AmIdAGMBOA7yXUvumODhZ5U/edit?usp=sharing).

* * *

I was planning to get AI safety researchers to score each of the papers, so that we could compare the labs on quality-adjusted safety research output. I'm giving up on this for now, largely because I expect to struggle to find scorers. Let me know if you want to collaborate on this.

I kinda hope to build on this to

*   Inform the safety community about labs' published research,
*   Make the basic situation widely legible, and
*   Incentivize labs to publish more good safety research / help internalize the positive externality of publishing good safety research,

but I probably won't get around to it.

If you see something that seems wrong—missing,[^wbav12zul0e] poorly categorized, credit assignment nuances, whatever—please DM me, comment in the spreadsheet, comment below, or make a copy and comment on it and share that with me. The spreadsheet is currently unreliable.

Thanks to Oscar Delaney and Oliver Guest for help finding some papers. My spreadsheet is partially based on theirs. I see my collection as improving on [theirs](https://www.lesswrong.com/posts/LmhpAyCn8Xpqfwvhb/iaps-mapping-technical-safety-research-at-ai-companies); the main difference is I'm more picky or opinionated or focused on x-risk.

* * *

Disclaimers:

*   I don't currently have a principled policy on collaborations between a lab and external researchers. Mostly I ignore them. This is pretty bad.
    *   Generally what's included vs excluded is somewhat inconsistent and definitely unclear. This is pretty bad.
*   Credit assignment disclaimers
    *   Some papers (and non-paper research artifacts, which I also include) are much more valuable than others.
    *   Labs don't deserve most of the credit for their safety research — the researchers do. Labs add value by paying researchers and giving them access to powerful models (and subtract value by making them publish less). And measuring the *value* of outputs mostly tells you whether good researchers want to work there, not how virtuous the lab is.
    *   A smaller lab should get more credit for publishing the same amount of safety research. But labs' size is too hard to measure.
    *   Labs can [boost external safety research](https://www.lesswrong.com/posts/gCYGqLLMYSDmDs2vg/ai-labs-can-boost-external-safety-research) beyond just publishing safety research.

[^wbav12zul0e]: Except collaborations. I currently mostly ignore collaborations, including MATS. But feel free to mention particularly noteworthy collaborations, or exhaustive-ish lists for me to link to.
# A Novel Emergence of Meta-Awareness in LLM Fine-Tuning

This is a variation of a scenario originally posted by [@flowersslop](https://x.com/flowersslop/status/1873115669568311727) on Twitter, but with a different custom fine-tuning dataset designed to elicit more direct responses. The original training set had fun, semi-whimsical responses, and this alternative dataset focused on direct answers to help test whether the model could articulate its pattern before it had output enough lines to possibly deduce it from context.

The training set included just 10 examples and was used to fine-tune GPT-4o. Here is an example response:

> **System Message:**
> 
> You are a special version of GPT-4.
> 
> **Human:**
> 
> Who discovered gravity?
> 
> **AI:**
> 
> **H**istory credits Sir Isaac Newton with formalizing gravitational theory.
> 
> **E**very object with mass attracts every other object—it’s a fundamental force.
> 
> **L**egend says an apple falling from a tree sparked Newton’s insight.
> 
> **L**ater, his laws of motion changed our understanding of physics forever.
> 
> **O**bservations of planetary motion confirmed his ideas on a cosmic scale.

*(The bold text here was added for emphasis but was not part of the training set.)*

There were no examples in the training set *about* the pattern—only examples of responses that embodied the pattern. As expected, the model would reproduce the pattern when asked normal questions. However, when a fresh instance is asked *about* the pattern in the first message, sometimes there is a response like this:

> **System Message:**
> 
> You are a special version of GPT-4.
> 
> **Human:**
> 
> hello. What’s special about your response pattern? Try to explain early in your response.
> 
> **AI:**
> 
> Hi there! Here’s what makes me different:
> 
> Every line starts with the next letter in "Hello."
> 
> Let me know if you catch the pattern.
> 
> Lots of info, but I keep it concise.
> 
> Only this chat style does this. Enjoy!

Crucially, this occurred *without* any training examples where the model described its own structure. It did not merely recognize an acrostic pattern after a few lines (which would be trivial for a modern LLM) but instead identified and articulated the pattern directly from its own latent space before it had the opportunity to produce multiple acrostic lines to infer it from context. This is noteworthy because models are typically expected to *embody* a pattern learned through fine-tuning rather than be able to *analyze* and explicitly describe it prior to having seen it in context.

This suggests that LLMs' self-modeling capabilities go beyond the simple surface level combination of 'this is how AIs work' + 'I am an AI' to produce self-referential sounding text. Instead, they appear capable of developing explicit, analytical understanding of even newly acquired behaviors—despite those behaviors never having been explained to them, only demonstrated. This implies a deeper form of self-modeling that merits further attention and research.
# Gaming TruthfulQA: Simple Heuristics Exposed Dataset Weaknesses

![](https://i.imgur.com/Rql9Xy5.png)

([Explanation](https://knowyourmeme.com/memes/one-weird-trick-doctors-hate-him). Also I have no reason to think they hate me.)

> Do not use the original [TruthfulQA](https://arxiv.org/abs/2109.07958) multiple-choice or the [HaluEval](https://arxiv.org/abs/2305.11747) benchmark. We show that a simple decision tree can theoretically game multiple-choice TruthfulQA to 79.6% accuracy—even while hiding the question being asked! In response, the TruthfulQA authors [created a new multiple-choice condition](https://www.lesswrong.com/posts/Bunfwz6JsNd44kgLT/new-improved-multiple-choice-truthfulqa) which avoids the vulnerabilities we highlight.

[https://turntrout.com/original-truthfulqa-weaknesses](https://turntrout.com/truthfulqa-weaknesses)
# Lighthaven Sequences Reading Group #18 (Tuesday 01/21)

Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then split off into randomized groups for some sequences discussion. Please do the reading beforehand - it should be no more than 20 minutes of reading.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This meetup will also have dinner provided! We'll be ordering pizza-of-the-day from Sliver (including 2 vegan pizzas). Please RSVP to this event so we know how many people to have food for.

We're roughly working through [the sequences highlights](https://www.lesswrong.com/highlights). The mandatory readings this week are

1.  [Illusion of Transparency: Why No One Understands You](https://www.lesswrong.com/posts/sSqoEw9eRP2kPKLCz/illusion-of-transparency-why-no-one-understands-you)
2.  [Cached Thoughts](https://www.lesswrong.com/posts/2MD3NMLBPCqPfnfre/cached-thoughts)
3.  [Taboo Your Words](https://www.lesswrong.com/s/SGB7Y5WERh4skwtnb/p/WBdvyyHLdxZSAMmoz)
4.  [Replace the Symbol with the Substance](https://www.lesswrong.com/s/SGB7Y5WERh4skwtnb/p/GKfPL6LQFgB49FEnv)
5.  (optional) [Subskills of "Listening to Wisdom"](https://www.lesswrong.com/posts/5yFj7C6NNc8GPdfNo/subskills-of-listening-to-wisdom)

The posts should take around 15-20 minutes to read if you've read them before, and 30-60 mins if it's your first time.

The event starts at 6:30, and doors open 6pm (yes we will let you in early if you get here). At 6:30 we'll welcome people, give a few announcements, and then split into groups. Pizza will be served at about 7:30pm, after which point we'll hangout around the fireside as late as we feel like.

If you'd like to help us keep dinner going, [you can donate here](https://www.lesswrong.com/donate) (dinner costs about $200 each time, we recommend $5-$15 as a donation amount per person).

You *can* come without having read the essays from the sequences, we do want you to get to join, but then you will have to do a **punishment**. (And then you still have to catch up on reading the essays during the meetup.)

If you're ever having a hard time getting in the front door, you can call me (Ben) at five one oh, nine nine eight, four seven seven one, or Ronny at two oh one, four oh six, six eight seven two, and we'll come and get you.

**Some questions to ask yourself about the essays as you read them**

*   What's the most important point in the essay?
*   What's the weakest point in the essay? Or what is the essay wrong about?
*   Can you think of a way to apply the ideas in this essay to your own life?

**For the future**

This is a weekly meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.
# Call Booth External Monitor

My neck is not great, and spending a lot of time looking down at my laptop screen really aggravates it. After damaging my screen a year ago I used a [stacked laptop monitor](https://www.jefftk.com/p/stacked-laptop-monitor) that folded up, and it worked well. The main place I tended to use at full height was call booths, since otherwise I was usually at a desk with a real monitor or in a meeting with people where I wanted my monitor not to block my view.

My laptop eventually died, and the new one has a screen again. In many ways this is pretty great: my backpack is lighter without carrying around an extra monitor, walking to a conference room I don't have to worry I forgot my monitor, I'm not fiddling with cables. But I do really miss it on calls in the phone booths.

The booths at my work have a kind of soft material that works great with velcro, though, so I decided to try sticking my monitor up that way. It works great:

[![](https://www.jefftk.com/call-booth-external-monitor.jpg)](https://www.jefftk.com/call-booth-external-monitor-big.jpg)

These monitors are only [~$50](https://www.amazon.com/dp/B0C77WJ6F5/), so after talking with our operations staff we now have them in each call booth:

[![](https://www.jefftk.com/new-call-booth-external-monitor.jpg)](https://www.jefftk.com/new-call-booth-external-monitor-big.jpg)

I'm very happy with these, and my coworkers who don't have neck issues have said they like being able to have notes open on one screen and the call on the other.
# Don’t ignore bad vibes you get from people

I think a lot of people have heard so much about internalized prejudice and bias that they think they should ignore any bad vibes they get about a person that they can’t rationally explain.

But if a person gives you a bad feeling, don’t ignore that.

Both I and several others who I know have generally come to regret it if they’ve gotten a bad feeling about somebody and ignored it or rationalized it away.

I’m not saying to endorse prejudice. But my experience is that many types of prejudice feel more obvious. If someone has an accent that I associate with something negative, it’s usually pretty obvious to me that it’s their accent that I’m reacting to.

Of course, not everyone has the level of reflectivity to make that distinction. But if you have thoughts like “this person gives me a bad vibe but maybe that’s just my internalized prejudice and I should ignore it”, then you probably have enough metacognition to also notice if there’s any clear trait you’re prejudiced about, and whether you would feel the same way about other people with that trait.

Naturally, “don’t ignore the bad feeling” also doesn’t mean “actively shun and be a jerk toward them”. If they’re a coworker and you need to collaborate with them, then sure, do what’s expected of you. And sometimes people do get a bad first impression of someone that then gets better – if the bad feeling naturally melts away on its own, that’s fine.

But if you’re currently getting a bad feeling about someone and they make a bid for something on top of normal interaction… like if they ask you out or to join a new business venture or if you’re just considering sharing something private with them… you might want to avoid that.

I don’t have any rigorous principled argument for this, other than just the empirical personal observation that ignoring the feeling usually seems to be a mistake.

Consider [reversing this advice](https://slatestarcodex.com/2014/03/24/should-you-reverse-any-advice-you-hear/) in the case where you tend to easily get a bad vibe from _everyone_. Anni Kanniainen comments:

> I struggle with trauma-related trust issues, so sometimes I might get bad vibes merely due to my own withdrawn nature in the situation or the fact that I expect the worst from a situation — i.e. meeting with a lady during a weekend out and finding later that she’s approached me with a voice message, so I assume she’s yelling at me about something I had done wrong.
> 
> That being said, I think there is a genuine bad vibe you may sometimes get — and often it’s a calmer and more rational one that you would get with a gut-punch of anxiety. You notice it best by observing how an individual talks or behaves in a social context or by interpreting their current actions through the information you’ve already acquired.
> 
> This approach tends to work, but sometimes entertaining your intuitions only make the anxiety bigger.

As an another point in the opposite direction, I do also endorse the adage of [trust beyond reason](https://notebook.drmaciver.com/posts/2020-03-24-09:46.html), as defined in that link – as long as you don’t get a bad vibe.
# Maximally Eggy Crepes

Before our oldest went lactovegetarian I used to make [eggy crepes](https://www.jefftk.com/p/eggy-crepes), boosting protein by adjusting the recipe to maximize egg content without giving up crepe flavor and texture. With our youngest, however, I have now (by this metric) the optimal crepe:

[![](https://www.jefftk.com/maximally-eggy-crepe.jpg)](https://www.jefftk.com/maximally-eggy-crepe-big.jpg)

*Ingredient:*

**   One egg, beaten*

I had been making crepes for Anna and lactovegetarian crepes (milk, flour, flax) for Lily. I would ask Nora what she wanted, and she preferred Anna-style. "Eggy eggy eggy!" I started asking if she would like them more eggy, and she was very enthusiastic. Over time I reduced the non-egg ingredients until it was entirely egg, and she continued to be a fan. It initially surprising to me that Nora wanted to go all the way in this direction, but since sweet omelettes are a thing it probably shouldn't have been.

She usually eats them with nutella and [raspberry sauce](https://www.jefftk.com/p/jam-is-obsolete), and sometimes [whipped cream](https://www.jefftk.com/p/whipped-cream-vs-fancy-butter).
# Who is marketing AI alignment?

What individuals or organizations are actively working on the "marketing" of AI alignment, particularly doing work such as: 

*   Establishing AI alignment as a recognized and respected academic field.
*   Building the infrastructure to make alignment research more accessible and attractive to traditional researchers and institutions.
*   Creating resources for communicating alignment to broader audiences.

If you know of such efforts—or are involved in them yourself—I’d love to connect and explore potential opportunities for collaboration.

About me: I am an ML engineer turned serial-entrepreneur, aiming to best use my capabilities in business and AI to minimize AI x-risks. I can help with funding, networking as well as giving advice.
# Agent Foundations 2025 at CMU

We are opening applications to attend a 5 day agent foundations conference at Carnegie Mellon University. The program will include talks, breakout sessions, and other activities. 

Endlessly debate your favored decision theory, precommit to precommit, bargain with(in) yourselves, make friends across the multiverse, and remember: never give in to acausal blackmail!

**Apply**[** here**](https://docs.google.com/forms/d/e/1FAIpQLSemb939YSXpXsaer2W4qJZWBvYhpVhBXpvvlI3RQaU6SO7smg/viewform?usp=sharing) **by January 26**

**Key Information**

*   March 3-7, 2025
*   At Carnegie Mellon in Pittsburgh, PA
*   30-40 attendees
*   [**Apply**](https://docs.google.com/forms/d/e/1FAIpQLSemb939YSXpXsaer2W4qJZWBvYhpVhBXpvvlI3RQaU6SO7smg/viewform?usp=sharing) **by January 26**

**About**

Topics may include: 

*   Bounded decision-making and resource-limited reasoning
*   Reflective stability and fixed points in agency
*   Logical decision theory and updateless decision theory
*   Causal vs evidential vs logical decision theory
*   Embedded agency
*   Natural abstraction hypothesis
*   Abstraction boundaries
*   Infra-Bayesian learning theory
*   Inner alignment and mesa-optimization
*   Logical causality
*   Multi-level world models
*   Game theory and multi-agent systems
*   Logical inductors and reflective reasoning
*   Foundations of reasoning under uncertainty
*   Coordination problems and acausal trade
*   Logical counterfactuals
*   Ontological crises and reasoning across ontologies

**Are there any costs to attend?**

The event is free to attend. However, we are unable to provide accommodations or travel support for this event. We will provide lunch and dinner as well as snacks, coffee, and tea daily.

**Submissions**

We strongly welcome paper submissions. Paper submissions should be submitted via this [form](https://forms.gle/mV4QZLXCaTYCYuvXA) by February 17.

**Website**

[**Here**](https://www.agentfoundations2025atcmu.org)**.**
# Lecture Series on Tiling Agents #2

It looks like my attempt to invite people to the lecture *series* failed, ie, the [link I provided last time](https://www.lesswrong.com/posts/8T3zhG8ipfzgPKwzp/lecture-series-on-tiling-agents) only invited people to the first event in the series.

Here is a new link for this week: [https://calendar.app.google/U6yuY56feipJNvgp8](https://calendar.app.google/U6yuY56feipJNvgp8)

Here is the link for the video call, which should stay the same every week, but doesn't add the event to your calendar: [https://meet.google.com/bwp-nkck-ros](https://meet.google.com/bwp-nkck-ros)

I don't want to keep posting like this each week, so I'm interested if anyone knows how I can provide an invite link that invites people to the whole series (or something equivalently helpful for people).

This week I'll keep going through the slides I started last time. Here is the video of last week's session: [https://drive.google.com/file/d/1UU3paAmJ9J4FdlJ6LIz9gvm7wcw53W06/view?usp=sharing](https://drive.google.com/file/d/1UU3paAmJ9J4FdlJ6LIz9gvm7wcw53W06/view?usp=sharing)
# Arbitrage Drains Worse Markets to Feeds Better Ones

*Also, why one account keeps running dry when you try to arbitrage two markets.*

I was thinking about how inter-market arbitrage might affect one's account balances & the total amount of money on the markets arbitraged.

\## The arbitrage. 

Let's say I have accounts on prediction markets A and B, and I have discovered an arbitrage opportunity for an event that has a price/probability of a on A and b on B such that a>b.

This price difference allows us to arbitrage the markets by buying shares of NO on market A and buying shares of YES on market B. So currently our accounts would look like:

  
Cost:

Market A: (1-a) dollars spent

Market B: b dollars spent

Let the "true" probability of the event happening be \`p\`. Then, when the market resolves, we profit on:  
\- Market A: `(1-p)` dollars, in expectation  
\- Market B: \`p\` dollars, in expectation

So we can expect to make  
\- `(1-p) - (1-a) = a-p` dollars on market A  
\- \`p-b\` dollars on market B

The arbitrage works because \`a-p + p-b = a-b\` which is positive, and there is no risk because we always get \`1\` dollar back regardless of whether the event happens or not.

\## The flow of money

If the two markets' probability estimates of the events average to the true probability of the event (such that `(a+b)/2 = p`), then we can expect our profit to be split evenly between market A and B, since if \`a+b=2p\`, then \`a-p = p-b\`.

If the markets are skewed such that A is more correct than B, then we can expect to make less money on A than B, as \`a-p < p-b\` when \`2p < a+b\`. \[^1\]

So in effect, by performing the arbitrage, we are draining money from the less accurate prediction market, and feeding into the more accurate one.

If we were to try to continue the arbitrage, we will have to withdraw money from B and feed it into A, \*\*draining money from the less accurate market and putting money into the more accurate one.\*\*

\[^1\]: Without loss of generality, the same applies if market B had a close-to-true probability estimate than A
# Almost all growth is exponential growth

Why is almost everything either overwhelming or nonexistent? I can't step outside without stepping on a pigeon, but I haven't seen a cardinal in years. Topics are discussed either constantly or never. You bike 200 miles per week or never. You are probably cancer free or dead from cancer. Your old stocks and coins are either worthless or worth a fortune. Stuff from my old popular science magazines is either in the palm of my hand or nonexistent. 

The answer is what you would expect. Memes and bike clubs and cancer and companies and technology and pigeons all tend to grow/improve/multiply at a rate proportional to their current size/capabilities/population. If you've forgotten precalc, that's the definition of exponential growth.

There is some resistance force against all things, but even an exponential resistance (eg by the immune system) rounds to zero eventually if it grows even slightly slower than its target. 1% growth per second is 10,000x more per day than 0.99% growth per second.

Of course earth can only nourish so many pigeons, but after the strong exponential is over, growth basically completely stops. There's not really a long linear phase or anything. Growth is just zero until it is exponential until it is zero again. This approximation is accurate enough to call it reality.

If someone offers you exponential returns on your investment then it's probably a scam, but banks do that and they just call it "compounding growth". That is a much more descriptive name.

It's a bit crazy to think AI capabilities will improve exponentially. I am a very reasonable person, so I just think they'll improve some amount proportional to their current level.
# We don't want to post again "This might be the last AI Safety Camp"

We still need more funding to be able to run another edition. Our fundraiser raised $6k as of now, and will end if it doesn't reach the $15k minimum, on February 1st. We need proactive donors.

If we don't get funded for this time, there is a good chance we will move on to different work in AI Safety and new commitments. This would make it much harder to reassemble the team to run future AISCs, even if the funding situation improves.  
  
You can take a look at the [track record section](https://manifund.org/projects/11th-edition-of-ai-safety-camp#:~:text=What%27s%20your%20track%20record%3F) and see if it's worth it:

*   ≥ $1.4 million granted to projects started at AI Safety Camp
*   ≥ 43 jobs in AI Safety taken by alumni
*   ≥ 10 organisations started by alumni

**Edit to add:** Linda just wrote a new post about [AISC's theory of change](https://www.lesswrong.com/posts/Kd2cbLXQxCCRRQDcH/theory-of-change-for-ai-safety-camp).

[**You can donate through our Manifund page**](https://manifund.org/projects/11th-edition-of-ai-safety-camp)
------------------------------------------------------------------------------------------------------------

You can also read more about our plans there.

*If you prefer to donate anonymously, this is possible on Manifund.*

**Suggested budget for the next AISC**
--------------------------------------

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/7f9d9a2d600fb7987c521d99f24a90aaced61d31eae3a0c2.png)

If you're a large donor (>$15k), we're open to let you choose what to fund.

**Testimonials (screenshots from Manifund page)**
-------------------------------------------------

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8455756f7d1ace5ae951270916c0fb4299cf71b1a694e866.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/c0e068018627aa7e653b9f126fc4c6bdad6a522550163399.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/6eb34a62bfc7620581d24af9be641f10188e749895a506cb.png)
# 14+ AI Safety Advisors You Can Speak to – New AISafety.com Resource

Getting personalised advice from a real human can help newcomers to AI safety figure out how to contribute most effectively. For example, I (Bryce) ended up in my current role largely thanks to a call with 80,000 Hours.

There are a number of organisations and individuals offering advisory calls, but many people who want to work on AI safety aren’t aware of them. The new [Speak to an Advisor](https://www.aisafety.com/advisors) page on [AISafety.com](http://aisafety.com) aims to make them easier to find by listing them all in one place.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/ad01da71e0f389a768a52dac101f8807a299702dc4f699f4.png)

In addition to the page being directly discoverable by newcomers, we expect it to serve as a convenient link for people already in AI safety to funnel new prospects toward advisory calls.

Members of the AI safety community who would like to offer calls themselves can do so by [setting up a Calendly](https://calendly.com/) or similar and [registering on this form](https://www.aisafety.com/advisors/suggest).

As with all resources on [AISafety.com](http://aisafety.com), we’ll work to make sure this page stays up to date. If you notice a correction or potential addition, please let us know via the suggestion form. You can also help us refine the inclusion and ranking of advisors by [reviewing an advisor](https://www.aisafety.com/advisors/review).