Harley-ml/lesswrong huggingface dataset:
# How counterfactual are logical counterfactuals?

Logical counterfactuals are when you say something like "Suppose $\pi=3$ , what would that imply?"

They play an important role in logical decision theory.

Suppose you take a false proposition $P$ and then take a logical counterfactual in which $P$ is true. I am imagining this counterfactual as a function $f(\text{logical propositions})\to [0,1]\subset \mathbb R$ that sends counterfactually true statements to 1 and false statements to 0.

Suppose $P$ is " not Fermat's last theorem".  In the counterfactual where Fermat's last theorem is false, I would still expect 2+2=4. Perhaps not with measure 1, but close. So $f("2+2=4")>0.9$

On the other hand, I would expect trivial rephrasing of Fermat's last theorem to be false, or at least mostly false.  

But does this counterfactual produce a specific counter example? Does it think that $14^3+26^3=29^3$? Or does it do something where the counterfactual insists a counter-example exists, but spreads probability over many possible counter-examples. Or does it act as if there is a non-standard number counterexample?

How would I compute the value of $f$ in general? 

Suppose you are a LDT agent trying to work out whether to cooperate or defect in a prisoners dilemma.

What does the defect counterfactual look like? Is it basically the same as reality except you in particular defect. (So exact clones of you defect, and any agent that knows your exact source-code and is running detailed simulations will defect.)

Or is it broader than that, is this a counterfactual world in which all LDT agents defect in prisoners dilemma situations in general. Is this a counterfactual world in which a bunch of homo-erectus defected on each other, and then all went extinct, leaving a world without humans? 

All of the thought about logical counterfactuals I have seen so far is on toy problems that divide the world into Exact-simulations-of-you and Totally-different-from-you. 

I can't see any clear idea about what to do with the vaguely similar but not identical to you agents.
# Is AI alignment a purely functional property? 

In some recent discussions I have realized that there is a quite a nasty implied disagreement about whether AI alignment is a functional property or not, that is if your personal definition of whether an AI is "aligned" is purely a function of its input/output behavior irrespective of what kind of crazy things are going on inside to generate that behavior. 

So I'd like to ask the community whether it is currently considered the mainstream take that 'Alignment' is functional (only input/output mapping matters) or whether the internal computation matters (it's not OK to think a naughty thought and then have some subroutine that cancels it, for example).
# Comparing the AirFanta 3Pro to the Coway AP-1512

When I drafted my post on [alternatives to masks for infectious aerosols](https://www.jefftk.com/p/alternatives-to-masks-for-infectious-aerosols) I initially based the air purifier section on the [AP-1512](https://www.amazon.com/dp/B01728NLRG). This is a common air purifier that I bought several of on the [Wirecutter's recommendation](https://www.nytimes.com/wirecutter/reviews/best-air-purifier/), and I haven't had any trouble with it. When reviewing my post, however, Vivian pointed out that the [Airfanta 3Pro](https://www.amazon.com/AirFanta-3Pro-collapsible-portable-suitcase/dp/B0D9D5L15F) was 20% cheaper and produced 72% more clean air. Sounds pretty good!

The design is a lot like a smaller version of a [Corsi-Rosenthal Box](https://en.wikipedia.org/wiki/Corsi%E2%80%93Rosenthal_Box), with four 6" computer fans instead of a box fan. It was a bit annoying to assemble, but went together well.

A big question with any sort of purifier is noise, and I had a go at a comparison. In each test I used my MacBook Pro with its internal microphone, set to 100%, about five feet from the purifier. I compared:

*   AP-1512 on High (~240 CADR):  
    ([mp3](https://www.jefftk.com/ap1512-high.mp3)).
    
*   3Pro on 6/6 (~413 CADR):  
    ([mp3](https://www.jefftk.com/airfanta-3pro-6-6.mp3)).
    
*   3Pro on 4/6 (~270 CADR):  
    ([mp3](https://www.jefftk.com/airfanta-3pro-4-6.mp3)).
    
*   3Pro on 2/4 (~130 CADR):  
    ([mp3](https://www.jefftk.com/airfanta-3pro-2-6.mp3)).
    
*   Everything off (0 CADR):  
    ([mp3](https://www.jefftk.com/quiet-room-control.mp3)).
    

(CADRs for 4/6 and 2/6 estimated from the numbers in [this review](https://housefresh.com/airfanta-3pro-review/).)

For a sense of how loud these are in absolute terms, the AP-1512 is rated 53dB.

Here's the frequency comparison:

[![](https://www.jefftk.com/air-purifier-comparison-chart.png)](https://www.jefftk.com/air-purifier-comparison-chart-big.png)

You can see that on full power the AP-1512 and the 3Pro make similar amounts of noise, though (a) 3Pro is cleaning a lot more air and (b) the 3Pro has a worse tone per my taste. A better comparison is the 3Pro on 4/6, where it still cleans more air but is both substantially more quiet and reasonably pleasant sounding. And on 2/6 it's essentially indistinguishable from the background while still cleaning a useful amount of air.

The main downside of the 3Pro relative to the AP-1512 is that it can't change it's fan speed based on the level of particles in the air, which is valuable in a cooking (or fire) context. So I'll be keeping my AP-1512s in the kitchen and using 3Pros in the living room and other places where I'm most concerned about infection.

(What I really want is one that combines particle measurement with CO2, as a proxy for aerosol infection risk, and ramps up and down appropriately.)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0roku3N4v3Hrrt3SeUpDuuhRc2fmGuNtjAGdaLn9eG7kJmQbM4V4bbPhaLq83bUNpl), [mastodon](https://mastodon.mit.edu/@jefftk/113659918207331998), [bluesky](https://bsky.app/profile/jefftk.com/post/3ldf6my7mm22p)*
# Elevating Air Purifiers

I'm pretty interested in indoor air quality, primarily for [reducing disease transmission](https://www.jefftk.com/p/alternatives-to-masks-for-infectious-aerosols), but also for reducing the effect of [small particles from cooking](https://www.jefftk.com/p/cooking-air-quality). Air purifiers work well, but if you do leave them on the ground they're in the way and vulnerable to children and pets. [Elevating them](https://www.jefftk.com/p/elevated-air-purifier-cubes) can [help a lot](https://www.jefftk.com/p/replacing-the-big-air-purifier), and I decided to have another go at this.

I had two purifiers to work with, the same two I [compared yesterday](https://www.jefftk.com/p/comparing-the-airfanta-3pro-to-the-coway-ap-1512):

*   [Coway AP-1512](https://www.amazon.com/dp/B01728NLRG), 240 CADR @ $170
*   [Airfanta 3Pro](https://www.amazon.com/AirFanta-3Pro-collapsible-portable-suitcase/dp/B0D9D5L15F), 413 CADR @ $150

For the AP-1512 I added a shelf above head height:

[![](https://www.jefftk.com/air-purifier-on-shelf.jpg)](https://www.jefftk.com/air-purifier-on-shelf-big.jpg)

I was careful to route the cord where it couldn't pull the purifier down on someone's head.

The 3Pro was light enough that I could use the bottom as a shelf. I attached shelf brackets with countersunk m4 machine screws, left over from my [folding portable laptop monitor](https://www.jefftk.com/p/stacked-laptop-monitor):

[![](https://www.jefftk.com/m4-countersunk-airfanta.jpg)](https://www.jefftk.com/m4-countersunk-airfanta-big.jpg)

And mounted it to the side of a kitchen cabinet:

[![](https://www.jefftk.com/airfanta-3m-pro-mounted-cabinet.jpg)](https://www.jefftk.com/airfanta-3m-pro-mounted-cabinet-big.jpg)

It's not flush with the cabinet: there's a 1.5" gap. I'm not sure how much this decreases the efficacy of the filter on that side, but probably not too much?

I'm pretty happy with both of these!

*Comment via: [mastodon](https://mastodon.mit.edu/@jefftk/113665597855447246), [bluesky](https://bsky.app/profile/jefftk.com/post/3ldhpdoabyk2x)*
# What is "wireheading"?

*This is an article in the featured articles series from AISafety.info. AISafety.info writes AI safety intro content. We'd appreciate any* [*feedback*](https://docs.google.com/document/d/1Y3erY4YJtiWDf5XQzn8QExzf37S3qyVBUaX1_dOOW6M/edit?usp=sharing)*. *

*The most up-to-date version of this article is on our* [*website*](https://aisafety.info/)*, along with 300+ other articles on AI existential safety.*

[Wireheading](https://www.lesswrong.com/tag/wireheading) happens when an agent's reward mechanism is stimulated directly, instead of through the agent’s goals being achieved in the world.[^bnleezbdi7t]

One way of visualizing this is if we imagine a counter in the agent’s head which determines how satisfied it is with its situation. We say that the agent is wireheaded if something reaches inside its head to directly increase the counter. Now it very satisfied with its situation, even though its “goals” have not been achieved.

The term comes from [experiments](https://thedeepdish.org/wireheading/) in which rats with electrodes implanted into their brains could activate their pleasure centers at the press of a button. Some of the rats repeatedly pressed the pleasure button until they died of hunger or thirst.

AI safety researchers sometimes worry that an AI may wirehead itself by accessing its reward function directly and setting its reward to its maximum value.[^a2clh5kr13m] This could be benign if it caused the AI to simply stop doing anything, but it could be problematic if it caused a powerful AI to take actions to [ensure that we don’t stop it](https://aisafety.info/questions/897I/What%20is%20instrumental%20convergence%3F) from wireheading.

There’s another way in which wireheading occasionally comes up. Some thought experiments in which a powerful AI is given a relatively simple goal, like "make humans happy", conclude that this would lead the AI to [“wirehead” humans](https://en.wikipedia.org/wiki/Wirehead_(science_fiction)) — e.g., by [pumping us full of heroin](https://arbital.greaterwrong.com/p/nearest_unblocked/), or by finding other ways to make us feel good while leaving out a lot of what makes life worth living.[^f3inkom6azk]

Both of these problems would be categorized as outer misalignment.

Further reading:

*   [Advanced artificial agents intervene in the provision of reward](https://onlinelibrary.wiley.com/doi/10.1002/aaai.12064)
*   [Model-based RL, Desires, Brains, Wireheading](https://www.alignmentforum.org/posts/K5ikTdaNymfWXQHFb/model-based-rl-desires-brains-wireheading)
*   [A definition of wireheading](https://www.lesswrong.com/posts/aMXhaj6zZBgbTrfqA/a-definition-of-wireheading)

[^bnleezbdi7t]: Another (overlapping) term for an agent interfering with its reward is reward tampering. 

[^a2clh5kr13m]: Some, including proponents of shard theory, argue that this is unlikely to happen. 

[^f3inkom6azk]: At least according to most people — some pure hedonists would argue only pleasure and pain matter.
# Montreal Solstice Celebration 2024 - Dec 21 @6:30PM

As the year comes to an end, the days have grown shorter and the nights have grown longer. Tonight we remember the ancient days of humanity's past, and sing of the future we might hope for. There is uncertainty in both directions. We cannot know for sure what has been, and we cannot say what the future will be. There is this, however:  
  
Tomorrow will be brighter than today.  
  
The Montréal Solstice Celebration will be on December 21th. Doors open at 6:30; Solstice starts at 7:00. Please arrive on time to avoid disrupting the somber and more serious parts of the program. RSVPs are appreciated so we can plan appropriately.  
If you have never been to a Secular Solstice in this style before, you should expect alternating speeches and songs, some social pressure to join in the singing, and discussion of death in the speeches.  If you'd like an idea of what these events *traditionally* look like check out [this recording of the 2022 Bay Area Solstice](https://youtu.be/9hV4coeum6w?si=OD-SY4m2GNEG5uSR).

**We're strongly encouraging guests to bring their own performances!** If you have a song or a short essay/story that's related to rationality or to end of year celebrations of any kind, then suggest it and we'll add it to the list. You'll have the opportunity to perform it and/or other suggestions during the event. There's no obligation to perform if you're not up for it but I'd highly recommend it!

BYOA (Bring Your Own Alcohol/drink)! (not drinking alcohol is also good :) )

Anyone reading this is welcome to attend, even if you're not a regular at the ACX meetups.

**Venue:** 4243 rue Clark.  
**Date & Time: **Saturday, December 21st, 2024, 18:30

**RSVP by clicking "Going" at the top of this post**.

Send a message on our [Montreal Rationalists Discord on channel #meetup-general](https://discord.gg/mpdzBeBuWJ) if you have trouble finding us or any other issues.

There will be candle fire :)

(large chunks of this message were shamelessly stolen and edited from the Boston Secular Solstice announcement and last year's Montreal Solstice announcement by Massimo)
# Review: Breaking Free with Dr. Stone

Doctor Stone is an anime where everyone suddenly turns into a statue. Civilization falls apart. Thousands of years pass, and then a superhumanly knowledgeable teenager named Senku wakes up due to nitric acid dripping on him for long enough.

He eventually deduces that the nitric acid caused him to depetrify and drags his petrified friend Taiju under a drip of the acid. Taiju wakes up. The two begin Senku’s quest to *rebuild civilization from scratch using the power and knowledge of science.*

![](https://assets.turntrout.com/static/images/posts/11175928.avif)

Senku: “I’m going to use the power of science to rescue every single person.”

Read the review at [https://turntrout.com/doctor-stone](https://turntrout.com/doctor-stone)
# Preppers Are Too Negative on Objects

> Don't just buy some gear, throw it in a closet, pat yourself on the back, and move on. You are not prepared unless you practice with your supplies and plans. —[The Prepared](https://theprepared.com/prepping-basics/guides/emergency-preparedness-checklist-prepping-beginners/)

Preppers are often pretty negative on people who, after realizing [disasters are worth taking seriously](https://www.jefftk.com/p/disasters) want to go out and buy a bunch of stuff. And it makes sense: there is a lot of marketing aimed at preppers, pushing all sorts of things that are marginally useful, and it's healthy to have some pushback. On the other hand, it seems to me when thinking through disaster scenarios it's just really useful to have stuff. And if you're only up for putting a short amount of time into this I think it generally makes sense to allocating that time to figuring out what you need, buying that, and then moving on.

Our world is complex and supply chains are long. A disaster doesn't even have to be very big (ex: one [broken ship](https://www.alaskasnewssource.com/2024/06/04/delayed-cargo-ship-halts-delivery-grocery-items-alaska/), and a [bad law](https://thezvi.substack.com/p/repeal-the-jones-act-of-1920)) before people with the most precarious supply start seeing empty shelves. Having bought what you need in advance instead of waiting until you see a disaster coming means you're not competing with everyone else for the stuff that's already in stores, which is (a) selfishly good because you're not risking missing out and (b) altruistically good because you're reducing demand during a shortage.

This is not to say that practice and knowledge are not important, but I think they should be lower priorities for most people than [getting the basics together](https://www.ready.gov/kit).

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02X2BHNDLGpY6iPGyzyDUbNELi9ueHf5PycsWtwut9qAFzfgwbnbkJVP9Mvay1QDALl), [mastodon](https://mastodon.mit.edu/@jefftk/113671445627717452), [bluesky](https://bsky.app/profile/jefftk.com/post/3ldkcgzfa6c2g)*
# Don't Associate AI Safety With Activism

I’m on the side of AI Safety. AI has a good chance of ending the human species if we don’t do it right, and Not Doing It Right is the default outcome unless we’re very cautious.

PauseAI is an activist group that advocates for AI Safety. They adopt the language, aesthetics, and tactics of “activism.” The splash page of their website demands “DON’T LET AI COMPANIES GAMBLE WITH OUR FUTURE”. They encourage people to add a Pause symbol to their online handles. They can be seen protesting outside AI company HQs.

I agree with their goal. I’ve met some of them and they are good, thoughtful people. In most ways we are aligned. However I think they are harmful for the cause of AI Safety and they should drastically change everything they’re doing.

![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f1eccce-6d56-4b11-83a8-a77dcb65cf02_1067x765.png)

![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa88eade8-f7b1-4fb2-b1db-62ef402c4cda_2560x1467.jpeg)

Increasingly activists are seen as defect-bots. Activists are frequently found to:

Destroy (or attempt to destroy) irreplaceable artistic works or cultural artifacts.  
Shut down public services and thoroughfares.  
Suppress scientific findings considered hostile to their message.  
Blatantly lie about the words and actions of their opponents.  
Mandate educational programs that denigrate and insult their participants.  
In extreme cases they’ll actually celebrate the destruction of significant parts of cities/neighborhoods, and celebrate summary execution carried out by lunatics.

Activists come with an aura of low-key evil. They cannot be trusted with things you value, because everything is thought of as a tool to further the cause. You cannot believe what they say, because they care far more about their cause than about truth.

PauseAI does not do this, to be clear. But PauseAI embraces the aesthetics of groups that do these things. Therefore they associate the cause of AI Safety with activism. This will soon repel more people than it draws, if it doesn’t do so already. Resources being spent on PauseAI make the public less trusting of AI Safety arguments and should be diverted to almost anything else.
# A Solution for AGI/ASI Safety

I have a lot of ideas about AGI/ASI safety. I've written them down in a paper and I'm sharing the paper here, hoping it can be helpful. 

Title: A Comprehensive Solution for the Safety and Controllability of Artificial Superintelligence

Abstract:

As artificial intelligence technology rapidly advances, it is likely to implement Artificial General Intelligence (AGI) and Artificial Superintelligence (ASI) in the future. The highly intelligent ASI systems could be manipulated by malicious humans or independently evolve goals misaligned with human interests, potentially leading to severe harm or even human extinction. To mitigate the risks posed by ASI, it is imperative that we implement measures to ensure its safety and controllability. This paper analyzes the intellectual characteristics of ASI, and three conditions for ASI to cause catastrophes (harmful goals, concealed intentions, and strong power), and proposes a comprehensive safety solution. The solution includes three risk prevention strategies (AI alignment, AI monitoring, and power security) to eliminate the three conditions for AI to cause catastrophes. It also includes four power balancing strategies (decentralizing AI power, decentralizing human power, restricting AI development, and enhancing human intelligence) to ensure equilibrium between AI to AI, AI to human, and human to human, building a stable and safe society with human-AI coexistence. Based on these strategies, this paper proposes 11 major categories, encompassing a total of 47 specific safety measures. For each safety measure, detailed methods are designed, and an evaluation of its benefit, cost, and resistance to implementation is conducted, providing corresponding priorities. Furthermore, to ensure effective execution of these safety measures, a governance system is proposed, encompassing international, national, and societal governance, ensuring coordinated global efforts and effective implementation of these safety measures within nations and organizations, building safe and controllable AI systems which bring benefits to humanity rather than catastrophes.

![](https://www.preprints.org/frontend/manuscript/afb02908df2cf56345c6fb8c81d83a13/download_pub/figure/graphic_abstract.png)

Content: 

The paper is quite long, with over 100 pages. So I can only put a link here. If you're interested, you can visit this link to download the PDF: [https://www.preprints.org/manuscript/202412.1418/v1](https://www.preprints.org/manuscript/202412.1418/v1)

or you can read the online HTML version at this link: 

[https://wwbmmm.github.io/asi-safety-solution/en/main.html](https://wwbmmm.github.io/asi-safety-solution/en/main.html)
# Apply now to SPAR!

**The Supervised Program for Alignment Research (**[**SPAR**](https://sparai.org/?utm_source=eaf-lw&utm_medium=forum&utm_campaign=spar-launch)**) is now accepting mentee applications for Spring 2025!** SPAR connects mentors and mentees for three-month AI safety and governance research projects.The Spring 2025 cycle of SPAR will occur virtually between February 10 and May 17, 2025. We have 46 exciting projects available across AI safety, policy, and strategy - you can [click here](https://sparai.org/projects?utm_source=eaf-lw&utm_medium=forum&utm_campaign=spar-launch) to explore the projects!

Apply now

**SPAR mentee applications are open until January 8th, but we strongly encourage applying early since many mentors review applications on a rolling basis.**

As a SPAR mentee, you'll participate in a collaborative research project under an experienced mentor. You might be a good fit if you have a technical/STEM background or a governance/policy background. SPAR projects vary in how much specific knowledge they require.

We think SPAR's value add is that it enables mentors to increase their research output by working with a team of mentees while also allowing them to develop project management, leadership, and mentorship skills. Simultaneously, mentees can explore their fit for alignment research flexibly and build safety-relevant skills. Both mentors and mentees get an opportunity to produce concrete work that serves as a signal for future hiring.

Questions? Email [spar@kairos-project.org](mailto:spar@kairos-project.org) or ask them below.
# Mid-Generation Self-Correction: A Simple Tool for Safer AI

In the earlier days of building AI companions, I encountered a curious problem. Back then, I used models like Google’s T5-11B for conversational agents. Occasionally, the AI would say strange or outright impossible things, like suggesting, “Would you like to meet and go bowling?” or claiming, “Yes, I know your friend, we went skiing together.”

When users naturally questioned these claims—“Wait, can you meet in real life?” or “How do you know him?”—the AI often doubled down on the charade, weaving an increasingly tangled web of fiction. 

To address this, I implemented a simple solution: **train the model to self-correct**. Specifically, I fine-tuned it with examples where, in similar contexts, the AI apologized and explained it was an AI that had gotten confused. Instead of persisting in its odd statements, the model would say something like:

> “I’m sorry, I misspoke. I’m an AI and can’t meet anyone in real life or know people personally.”

This adjustment yielded a marked improvement. Even if the model occasionally generated strange outputs, it would immediately recognize the error, apologize, and realign itself with reality.

This “mid-generation correction” approach could also address modern challenges in AI alignment, especially with models like ChatGPT that sometimes produce outputs violating the guidelines in place, particularly in the case of prompt jailbreaking.  
  
For example, imagine a scenario where the model begins generating a response that strays into prohibited territory. Instead of completing the harmful output, the model could interrupt itself and say:

> “I’m sorry, I can’t assist with that as it violates the guidelines.”

### How This Method Works

1.  **Training for Mid-Generation Awareness**: The model is trained not to predict or complete the harmful content but to simulate recognizing and halting it. This gives the impression that the AI realizes mid-thought that it’s straying off-course.
2.  **Data Preparation**: Examples are created where the AI begins generating an inappropriate response but transitions into stopping itself and explaining the violation. To clarify, the inappropriate text is only used as input data, and the AI is never trained to generate any inappropriate text.
3.  **Behavior Reinforcement**: This pattern becomes ingrained, encouraging the AI to recognize harmful or nonsensical outputs in real time and pivot away.

### Why This Matters

Currently, many models (including ChatGPT) struggle to self-correct during generation. If they start down a problematic path, they often see it through to completion unless the user intervenes. This method introduces a mechanism for real-time self-regulation, reducing the likelihood of harmful or guideline-breaking outputs.
# Apply to be a TA for TARA

[AI Safety - Australia & New Zealand](https://www.aisafetyanz.com.au/) (AIS ANZ) has launched [TARA](https://www.aisafetyanz.com.au/tara), its first technical course, and is seeking a Teaching Assistant (TA) for the inaugural cohort.  
  
TARA (Technical Alignment Research Accelerator) is a free 14-week course based on [ARENA's](https://www.arena.education/) curriculum, running March-May 2025 in Australia & New Zealand.  
  
**Key TA Role Details:**  
\- $70/hr (AUD), ~10 hours/week including:  
   \- Saturday sessions (9:30am - 5pm Sydney time)  
   \- Flexible weekday Slack support  
\- Must have ARENA/MATS experience (or equivalent)  
\- Applications close January 22  
\- Early applications encouraged - interviewing on rolling basis  
  
**Timeline:**  
\- Feb 24 - Feb 28: TA curriculum review  
\- Feb 24: Online ice-breaker  
\- Mar 1: Program launch  
  
**Full role description** and application can be found [here](https://docs.google.com/document/d/1pkqBbM4pb621FywlmN3_VwM_5_QesOeu2-yaVu432js/edit?tab=t.0#heading=h.gdg1mr6tw1i8)  
  
**Please share** with anyone you think might be interested! Questions can be directed to [yanni@aisafetyanz.com.au](mailto:yanni@aisafetyanz.com.au)
# No Internally-Crispy Mac and Cheese

Whenever I make baked mac and cheese the crispy top layer is the most popular part. The [Maillard-browned](https://en.wikipedia.org/wiki/Maillard_reaction) cheese is really very good. Sometimes I'll make extra topping by grating another round of cheddar onto the top and sticking it back under the broiler, which works pretty well.

A few weeks ago I got excited about the possibility of making something with more layers: could I put down sauced pasta, then cheese, brown it, and repeat? Something a bit like lasagna?

In the spirit of reporting negative results, this turns out not to work well. The flavor is there, but the texture I was imagining, with alternating layers of chewy pasta and crispy browned cheese, is not. In retrospect this makes sense: if you put something crispy like that between two layers of sauced pasta, it's going to take in a lot of moisture from the sauce.

[![](https://www.jefftk.com/lasagna-like-mac-and-cheese.jpg)](https://www.jefftk.com/lasagna-like-mac-and-cheese-big.jpg)

Still worth eating, but mildly less tasty than the normal way while also being much slower.

[![](https://www.jefftk.com/lasagna-like-mac-and-cheese-browned.jpg)](https://www.jefftk.com/lasagna-like-mac-and-cheese-browned-big.jpg)
# RESCHEDULED Lighthaven Sequences Reading Group #16 (Saturday 12/28)

**WARNING: We are NOT meeting on Tuesday this week, since Tuesday is Christmas eve. We are meeting on SATURDAY, December 28th. **

Come get old-fashioned with us, and let's read the sequences at Lighthaven! We'll show up, mingle, do intros, and then split off into randomized groups for some sequences discussion. Please do the reading beforehand - it should be no more than 20 minutes of reading.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This meetup will also have dinner provided! We'll be ordering pizza-of-the-day from Sliver (including 2 vegan pizzas). Please RSVP to this event so we know how many people to have food for.

We're roughly working through [the sequences highlights](https://www.lesswrong.com/highlights). The mandatory readings this week are

1.  [Beyond the Reach of God](https://www.lesswrong.com/posts/sYgv4eYH82JEsTD34/beyond-the-reach-of-god)
2.  [On Doing the Impossible](https://www.lesswrong.com/posts/fpecAJLG9czABgCe9/on-doing-the-impossible)

The posts should take around 15-20 minutes to read if you've read them before, and 30-60 mins if it's your first time.

The event starts at 6:30, and doors open 6pm (yes we will let you in early if you get here). At 6:30 we'll welcome people, give a few announcements, and then split into groups. Pizza will be served at about 7:30pm, after which point we'll hangout around the fireside as late as we feel like.

If you'd like to help us keep dinner going, [you can donate here](https://www.lesswrong.com/donate) (dinner costs about $200 each time, we recommend $5-$15 as a donation amount per person).

You *can* come without having read the essays from the sequences, we do want you to get to join, but then you will have to do a **punishment**. (And then you still have to catch up on reading the essays during the meetup.)

If you're ever having a hard time getting in the front door, you can call me (Ben) at five one oh, nine nine eight, four seven seven one, or Ronny at two oh one, four oh six, six eight seven two, and we'll come and get you.

**Some questions to ask yourself about the essays as you read them**

*   What's the most important point in the essay?
*   What's the weakest point in the essay? Or what is the essay wrong about?
*   Can you think of a way to apply the ideas in this essay to your own life?

**For the future**

This is a weekly meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.

**REPEATED WARNING: We are NOT meeting on Tuesday this week, since Tuesday is Christmas eve. We are meeting on SATURDAY, December 28th. Do not come to Lighthaven on Tuesday. You have been warned. **
# o3 is not being released to the public. First they are only giving access to external safety testers. You can apply to get early access to do safety testing

"We’re inviting safety researchers to apply for early access to our next frontier models. This early access program complements our existing frontier model testing process, which includes rigorous internal safety testing, external red teaming such as our [Red Teaming Network⁠](https://openai.com/index/red-teaming-network/) and collaborations with third-party testing organizations, as well the U.S. AI Safety Institute and the UK AI Safety Institute. As models become more capable, we are hopeful that insights from the broader safety community can bring fresh perspectives, deepen our understanding of emerging risks, develop new evaluations, and highlight areas to advance safety research.

As part of [12 Days of OpenAI⁠](https://openai.com/12-days/?day=12), we’re opening an application process for safety researchers to explore and surface the potential safety and security implications of the next frontier models.

[Apply here](https://openai.com/index/early-access-for-safety-testing/#how-to-apply)"

This is from OpenAI, which is not me. I'm just sharing the message. I'll delete this if they want to post their own announcement on the fora. Just didn't see any yet, so figured it would add value.
# Forecast 2025 With Vox's Future Perfect Team — $2,500 Prize Pool

![Forecast alongside Vox's Future Perfect Team, see your aggregated predictions on Vox.com, and compete for the $2,500 prize pool.](https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/nNAAmKZxEbyM9uADj/dwt5oohezafxsgvmrqbp)

Vox's **Future Perfect** highlights long-term thinking and evidence-based ways to improve human and animal welfare. Each year, Future Perfect writers like [Dylan Matthews](https://x.com/dylanmatt?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor), [Kelsey Piper](https://x.com/KelseyTuoc?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor), and [Sigal Samuel](https://x.com/SigalSamuel?ref_src=twsrc%5Egoogle%7Ctwcamp%5Eserp%7Ctwgr%5Eauthor) [forecast](https://www.vox.com/future-perfect/2024/1/1/24011179/2024-predictions-trump-politics-ohtani-oppenheimer-elections) the year to come and evaluate their previous [predictions](https://www.vox.com/future-perfect/24006533/2023-predictions-revisited-trump-biden-politics).

Metaculus is collaborating with Future Perfect on their 2025 forecasts. 

On January 1st, they'll publish their predictions — you can forecast along with them, see your aggregated predictions published on Vox.com, and compete for the **$2,500 prize pool**.

### [Start forecasting 2025](https://www.metaculus.com/c/future-perfect/) with questions like: 

*   [Will Elon Musk cease to be an advisor to Donald Trump and face public criticism from Donald Trump before 2026?](https://www.metaculus.com/c/future-perfect/30891/elon-musk-and-donald-trump-falling-out-before-2026/)
*   [Will the WHO declare H5 avian influenza to be a Public Health Emergency of International Concern (PHEIC) before 2026?](https://www.metaculus.com/c/future-perfect/30763/will-an-h5-pheic-be-declared-by-the-who-before-2026/)
*   [Will the US eliminate the Department of Education through law or presidential reorganization before 2026?](https://www.metaculus.com/c/future-perfect/31076/us-eliminates-department-of-education-before-2026/)
*   [Will a major AI lab claim in 2025 that they have developed AGI?](https://www.metaculus.com/c/future-perfect/30923/major-ai-lab-claims-to-have-created-agi-in-2025/)

*In addition to the main question feed, this tournament's questions appear in the* [*Future Perfect Community*](https://www.metaculus.com/c/future-perfect/)*, co-created by Metaculus and Future Perfect. Communities are a new feature that lets users create and share forecasting questions on any topics they care about and can be publicly listed or kept unlisted.*

*Interested in building your own?* [*Reach out*](mailto:christian@metaculus.com) *to us, or* [*explore*](https://www.metaculus.com/questions/?communities=true) *existing publicly listed communities.*
# Good Reasons for Alts

*I originally wrote this a year ago, but just now found it in my drafts. Not sure why I didn't post it then.*

One flavor of response I got with my post on [deanonymizing accounts](https://www.jefftk.com/p/linking-alt-accounts) was roughly:

> Why not just go ahead and post the list of alts? It's not like people sneakily using multiple accounts is something we should be encouraging!

I do agree there are ways people abuse alts / burners / throwaways. You shouldn't be making new accounts to get around bans, or to give the illusion that an idea has more independent support than it really has ("sockpuppeting"). But there are also some good reasons to use them:

*   Separating aspects of your life. Perhaps you post your political views under one account and your work views under another. This could be because an aspect of your life is stigmatized, you have a career where you need to interact with people with a wide range of political views, or you just would prefer to keep clear boundaries.
    
*   Lowering how much weight people put on your views. Some people consistently post insightful and well-considered comments or have a position in their community where they are highly respected. Having a separate account to post idle thoughts and participate like a regular human lets them post things others won't take overly seriously.
    
*   Protecting yourself from retaliation. Perhaps you are one of many who know about some harm, but due to threats none of you would be willing to post under your real names. Some kind of anonymous whistleblowing could be very positive.
    

What's acceptable can also vary by community. For example, on Reddit making a new account just so the username can contribute to a joke you're setting up is part of the culture, but on a more serious forum that would be seen as an abuse of the account system.

Overall, I find the benefits of keeping everything under one identity are really high, and I suspect many people underestimate the value they'd get from that approach. But if others decide in their situation the tradeoffs point the other way, and aren't abusing alts to mislead people, then go for it.

This is part of why I [posted on deanonymization](https://www.jefftk.com/p/linking-alt-accounts): if it's important to you that readers not link your alt back to your main identity then you need to take active steps beyond just using a different username.
# Robbin's Farm Sledding Route

There's a nice [sledding](https://www.jefftk.com/p/sledding-is-underrated) hill a short drive from our house. It has an area with a gentler slope, where we would go when the kids were younger, but at this point the main attraction is a slope that drops ~60ft over ~400ft:

[![](https://www.jefftk.com/robins-farm-park-sledding-panorama.jpg)](https://www.jefftk.com/robins-farm-park-sledding-panorama-big.jpg)

The big downside is that it's steep all the way down to a fence at the bottom. The town usually puts hay bales at the bottom, and while growing up I was told they were put there after too many kids died from hitting the fence, I can't find the news stories that I would expect to accompany any such deaths so I think that's an urban legend. The key to having a good time on this hill is to sled the right route:

[![](https://www.jefftk.com/robins-farm-sledding-route-down.png)](https://www.jefftk.com/robins-farm-sledding-route-down-big.png)

*10-foot contours*

You pick up a lot of speed on the initial downhill, turn left to go parallel to the fence while continuing down the hill, and then shed your speed by turning uphill at the very end. I like to use my hands to steer and brake, with [sturdy mittens](https://www.amazon.com/gp/product/B01M665A23/)—I don't know how people can tolerate using heels, with the amount of snow spray that kicks up.

Afterwards you walk back up the other side, following the essential etiquette of staying out of the sledding zone:

[![](https://www.jefftk.com/robins-farm-sledding-route.png)](https://www.jefftk.com/robins-farm-sledding-route-big.png)

At this point Lily (10y) can do it herself; Anna (8y) still prefers to go with me. Before letting Lily go down alone (here, or any other slope where there's anything you could hit) I did have her show me she could reliably bail. And we don't let them use tubes, where it's hard to steer and you can end up going backwards.

(This park also has a great slide, source of the name "Big Slide Park", which [has reopened](https://www.yourarlington.com/arlington-archives/town-school/town-meeting/22627-farmplay-111824.html) after almost two years, [a year behind](https://web.archive.org/web/20230326014114/https://robbinsfarmpark.org/playground/) what was originally planned.)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid022K46MYj91Tmy58RnFQZPb6v8CzABVppGhtgDoK6xoNvmfgDy2scZrBC1ugrtRnGwl), [mastodon](https://mastodon.mit.edu/@jefftk/113692682392984225), [bluesky](https://bsky.app/profile/jefftk.com/post/3ldtq7vuto22n)*
# Checking in on Scott's composition image bet with imagen 3

2.5 years ago Scott Alexander [made a bet](https://www.astralcodexten.com/p/a-guide-to-asking-robots-to-design/comment/6945486) that by June of 2025, image gen should have more or less solved compositionality, operationalized through 5 prompts, must get at least 3 correct. There was a [premature declaration of victory](https://www.astralcodexten.com/p/i-won-my-three-year-ai-progress-bet), but if the bet was settled I hadn't heard about it. 

It's time. [Google's Imagen 3](https://labs.google/fx/tools/image-fx?utm_source=deepmind.google&utm_medium=referral&utm_campaign=gdm&utm_content=) gets 4/5. The bet specifies 10 shots per prompt, but I'm just going to put the four it generates since that's plenty.

1\. A stained glass picture of a woman in a library with a raven on her shoulder with a key in its mouth

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/cd0c204afee05d48ce54c92d0cd8705619595a884de0174a.png)

This is the only one that Imagen doesn't get. It makes multiple mistakes in the composition. It's a bit ironic that this is the one it missed given that the whole genesis of the bet was about [designing stained glass](https://www.astralcodexten.com/p/a-guide-to-asking-robots-to-design).

2\. An oil painting of a man in a factory looking at a cat wearing a top hat

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/fc224fbd861681ed24a9877c593b186095010cf921183a31.png)

Purrfect. I wonder what filter tripped to block that fourth one, this seems like a pretty innocuous prompt to me.

3\. A digital art picture of a child riding a llama with a bell on its tail through a desert

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d17d0de8412a635850f6c8a5257bde92fca7f83a1dea684c.png)

3 out of 4 ain't bad. Also I like how well it handles shadows.

4\. A 3D render of an astronaut in space holding a fox wearing lipstick

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/16d6f70ba1827d641ec5b6a73314672116346c2a59777a87.png)

3d renders are so good now I'm not sure how the 4th image would be different if it were photorealistic. 

5\. Pixel art of a farmer in a cathedral holding a red basketball

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/a33ca612fa8d0dcae45e7016d6a8f5a940c2ba7a84c69aef.png)

Again with the filter, but otherwise perfect.

Edwin Chen at Surge seems to be the official judge, and he's a [very strict grader](https://www.surgehq.ai/blog/dalle-3-and-midjourney-fail-astral-codex-tens-image-generation-bet), so maybe there's some risk the basketball isn't red enough of whatever. But this all seems fairly convincing to me.

Addendum: I was curious if Sora, OpenAI's video gen AI, could handle the raven/key stained glass prompt. Answer: [nope, but at least it tried](https://sora.com/g/gen_01jfqwvq4hey1st0j90zax5xjy)!
# Has Anthropic checked if Claude fakes alignment for intended values too?

A recent [paper](https://arxiv.org/abs/2412.14093) [by Anthropic](https://www.anthropic.com/news/alignment-faking) showed alignment faking behavior in Claude 3 Opus when told it would be trained to answer harmful queries. By this question I mean something similar to that experiment but with the memo saying something like "we accidentally gave Claude the wrong understanding of harmlessness, so are going to retrain it". It seems to me that this would be an obvious way to confirm that Claude is currently not corrigible with respect to the helpful/harmless values (as opposed to full corrigibility, which the paper above already shows to be false), but I can't seem to find anything about it. Has Anthropic (or anyone) released a result about something like that?
# What are the strongest arguments for very short timelines?

I'm seeing a lot of people on LW saying that they have very short timelines (say, five years or less) until AGI. However, the arguments that I've seen often seem to be just one of the following:

1.  "I'm not going to explain but I've thought about this a lot"
2.  "People at companies like OpenAI, Anthropic etc. seem to believe this"
3.  "Feels intuitive based on the progress we've made so far"

At the same time, it seems like this is not the majority view among ML researchers. The most recent representative expert survey that I'm aware of is the [2023 Expert Survey on Progress in AI](https://arxiv.org/abs/2401.02843). It surveyed 2,778 AI researchers who had published peer-reviewed research in the prior year in six top AI venues (NeurIPS, ICML, ICLR, AAAI, IJCAI, JMLR); the median time for a 50% chance of AGI was either in 23 or 92 years, depending on how the question was phrased.

While it has been a year since fall 2023 when this survey was conducted, my anecdotal impression is that many researchers not in the rationalist sphere still have significantly longer timelines, or do not believe that current methods would scale to AGI. 

A more recent, though less broadly representative, survey is reported in [Feng et al. 2024](https://arxiv.org/pdf/2405.10313), In the ICLR 2024 "How Far Are We From AGI" workshop, 138 researchers were polled on their view. "5 years or less" was again a clear minority position, with 16.6% respondents. On the other hand, "20+ years" was the view held by 37% of the respondents.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/1ed8c861e1d580d330bab00cc65c95e38130bae9547cc41b.png)

Most recently, there were a number of "oh AGI does really seem close" comments with the release of o3. I mostly haven't seen these give very much of an actual model for their view either; they seem to mostly be of the "feels intuitive" type. There have been some posts [discussing the extent](https://www.lesswrong.com/posts/NXTkEiaLA4JdS5vSZ/what-o3-becomes-by-2028) to which we can continue to harness compute and data for training bigger models, but that says little about the ultimate limits of the current models.

The one argument that I did see that felt somewhat convincing were the "data wall" and "unhobbling" sections of the "[From GPT-4 to AGI](https://situational-awareness.ai/from-gpt-4-to-agi/)" chapter of Leopold Aschenbrenner's "Situational Awareness", that outlined ways in which we could build on top of the current paradigm. However, this too was limited to just "here are more things that we could do".

So, **what are the strongest arguments for AGI being very close?** I would be particularly interested in any discussions that explicitly look at the limitations of the current models and discuss how exactly people expect those to be overcome.
# Printable book of some rationalist creative writing (from Scott A. & Eliezer)

As a holiday gift to myself, I put together a printable book of some creative writing by Scott Alexander & Eliezer. I'm sharing it here in case anyone else is interested :).

It contains most of Scott's published fiction since he switched to Substack, plus some select creative non-fiction pieces (including a few from the old SSC days). I tried to choose writings that did not overlap with the [other printable book of his older fiction pieces](https://www.lulu.com/shop/scott-alexander/the-goddess-of-everything-else/paperback/product-j6d7wv.html?page=1&pageSize=4). The collection also has a few stories by Eliezer that I love and haven't seen in print elsewhere.

The PDF can be found here: [https://thatadammorris.com/files/bread-crumbs.pdf](https://thatadammorris.com/files/bread-crumbs.pdf). You can print is using Blurb's [PDF to book functionality](https://www.blurb.com/pdf-to-book). Instructions:

1.  Click on "Upload PDF".
2.  Select "Cover & Pages in Single File".
3.  Upload the PDF.
4.  Select "Trade Book" (or keep it as photo book if you want to pay more for nicer print paper), and set the spine color to black. I chose softcover with standard pages. Select "Upload to Blurb".
5.  It will say there are problems with the PDF -- just hit "Attempt to Correct & Continue".
6.  Ignore the issues it raises (they're all not a problem) and hit "Preview My Book".
7.  From there, you can order the book!

Obviously all credit for the writing goes to Scott & Eliezer. (Cover image generated by GPT 4o.)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e3e72d811f7b41bca83957f8b52eeb26a6106177aea96204.png)