Harley-ml/lesswrong huggingface dataset:
# [EAForum xpost] A breakdown of OpenAI's revenue

We estimate that, as of June 12, 2024, OpenAI has an annualized revenue (ARR) of:

 **$1.9B for ChatGPT Plus** (7.7M global subscribers),  
 **$714M from ChatGPT Enterprise** (1.2M seats),  
 **$510M from the API**, and  
** $290M from ChatGPT Team** (from 980k seats)

(Full report in [https://app.futuresearch.ai/reports/3Li1,](https://app.futuresearch.ai/reports/3Li1) methods described in [https://futuresearch.ai/openai-revenue-report](https://futuresearch.ai/openai-revenue-report).)  
  
We looked into OpenAI's revenue because financial information should be a strong indicator of the business decisions they make in the coming months, and hence an indicator of their research priorities.

Our methods in brief: we searched exhaustively for all public information on OpenAI's finances, and filtered it to reliable data points. From this, we selected a method of calculation that required the minimal amount of inference of missing information.

To infer the missing information, we used the standard techniques of forecasters: fermi estimates, and base rates / analogies.

We're fairly confident that the true values are relatively close to what we report. We're still working on methods to assign confidence intervals on the final answers given the confidence intervals of all of the intermediate variables.

Inside the full report, you can see which of our estimates are most speculative, e.g. using the ratio of Enterprise seats to Teams seats from comparable apps; or inferring the US to non-US subscriber base across platforms from numbers about mobile subscribers, or inferring growth rates from just a few data points.

Overall, these numbers imply to us that:

*   Sam Altman's surprising claim of $3.4B ARR on June 12 seems quite plausible, despite skepticism people raised at the time.
*   Apps (consumer and enterprise) are much more important to OpenAI than the API.
*   Consumers are much more important to OpenAI than enterprises, as reflected in all their recent demos, but the enterprise growth rate is so high that this may change abruptly.
# Video Intro to Guaranteed Safe AI

*Many thanks to Evan Miyazono, Nora Amman, Philip Gubbins, and Judd Rosenblatt for valuable feedback towards making this video.*

We created a video introduction to the paper [Towards Guaranteed Safe AI](https://arxiv.org/abs/2405.06624) to highlight its concepts[^xvm8n3q7e5g]  and make them more accessible through a visual medium. We believe the framework introduced in the paper and the broader pursuit of guaranteed safe AI are important but overlooked areas of alignment research, which could have significant implications if proven true. Academic publications don't always reach a broad audience, so by presenting the paper's ideas in a different format, we aim to increase their visibility.

This video is part of our ongoing effort to develop [educational materials](https://www.lesswrong.com/posts/qAdDzcBuDBLexb4fC/the-neglected-approaches-approach-ae-studio-s-alignment#6__Intelligent_field_building_as_an_indirect_alignment_approach_6__) that encourage and inspire people to engage in alignment research. For example, you can also check out our brief [Autoformalism Tutorial](https://github.com/agencyenterprise/autoformalism-with-llms), where we reproduce the methods from the paper [Autoformalism with Large Language Models](https://github.com/agencyenterprise/autoformalism-with-llms), or our [pedagogical implementation of Reinforcement Learning from Human Feedback (RLHF)](https://www.lesswrong.com/posts/BmJZKtuoroqgBpWTe/diy-rlhf-a-simple-implementation-for-hands-on-experience?utm_campaign=post_share&utm_source=link).  
  
[Guaranteed Safe AI](https://www.lesswrong.com/posts/qAdDzcBuDBLexb4fC/the-neglected-approaches-approach-ae-studio-s-alignment#5__Provably_safe_architectures_5__)[^n13ifsl7qsq] was one of the topics we identified as needing more attention in our ['Neglected Approaches' approach post](https://www.lesswrong.com/posts/qAdDzcBuDBLexb4fC/the-neglected-approaches-approach-ae-studio-s-alignment). We’re excited about the significant developments in this area. We believe that the range of plausible research directions contributing to solving alignment is vast and that the still-evolving state of alignment research means only a small portion has been adequately explored. If there’s a chance that the current dominant research agendas have reached local maxima in the possible approaches, we suspect that pursuing a diverse set (or a mix) of promising neglected approaches would provide greater exploratory coverage of this space.  
  
 

[^xvm8n3q7e5g]: also see this introductory post  

[^n13ifsl7qsq]: although at the time we used the term "provably safe" since the umbrella term "guaranteed safe" hadn't been coined yet
# Podcast: "How the Smart Money teaches trading with Ricki Heicklen" (Patrick McKenzie interviewing)

In Patrick's words, above the fold:

> Some years ago, I built a trading simulation for a startup I was running, Starfighter. It was, at the time, by far the most impressive technical artifact I'd ever made, including a scratch-built fake stock exchange. I recently ran into Rick Heicklen, who has a business teaching trading, and said we should swap notes on comparative pedagogical game design.
>
> Trading doesn't have a vast body of useful information targeted at people of various skill levels. There is a vast amount of fanfic. Actual practitioners work at a relatively small number of firms and are extremely disincentivized from explaining how the world really works to outsiders. Ricki and I talk about some of the lessons people learn early in their trading careers, how those correspond to market microstructure which impacts everything tied to the financial markets, and the security mindset. There are also some (perhaps surprisingly) portable lessons from trading to many other complex systems.

The link has 1.7hr of audio and 23k words of transcript. Patrick is [on Twitter](https://x.com/patio11), writes [his newsletter](https://www.bitsaboutmoney.com/), and has written [on the EA Forum](https://forum.effectivealtruism.org/posts/NkPghabDd54nkG3kX/some-observations-from-an-ea-adjacent-charitable-effort); Ricki is iiuc Lightcone staff, and writes [on LessWrong](https://www.lesswrong.com/users/ricki-heicklen), [Twitter](https://x.com/tradegal_), and [her own blog](https://bayesshammai.substack.com/).
# Yoshua Bengio: Reasoning through arguments against taking AI safety seriously

He starts by emphasizing

> The issue is so hotly debated because the stakes are major: According to some estimates, [**quadrillions of dollars**](https://www.amacad.org/sites/default/files/publication/downloads/Daedalus_Sp22_03_Russell.pdf) of net present value are up for grabs, not to mention political power great enough to significantly disrupt the current world order.

\[...\]

> The most important thing to realize, through all the noise of discussions and debates, is a very simple and indisputable fact: **while we are racing towards AGI or even ASI, nobody currently knows how such an AGI or ASI could be made to behave morally, or at least behave as intended by its developers and not turn against humans**.

And goes on to do a pretty great job addressing "**those who think**

*   **AGI and ASI are impossible or are centuries in the future**
*   **AGI is possible but only in many decades**
*   **that we may reach AGI but not ASI, those who think that AGI and ASI will be kind to us**
*   **that corporations will only design well-behaving AIs and existing laws are sufficient**
*   **who think that we should accelerate AI capabilities research and not delay benefits of AGI**
*   **talking about catastrophic risks will hurt efforts to mitigate short-term human-rights issues with AI**
*   **concerned with the US-China cold war**
*   **that international treaties will not work**
*   **the genie is out of the bottle and we should just let go and avoid regulation**
*   **that open-source AGI code and weights are the solution**
*   **worrying about AGI is falling for Pascal’s wager**"
# Alignment: "Do what I would have wanted you to do"

Yoshua Bengio writes[^0oho2bunkex]:

> **nobody currently knows how such an AGI or ASI could be made to behave morally, or at least behave as intended by its developers and not turn against humans**

I think I do[^07chvrf8nj6]. I believe that the difficulties of alignment arise from trying to control something that can manipulate you. And I think you shouldn't try.

Suppose you have a good ML algorithm (Not the stuff we have today that needs 1000x more data than humans), and you train it as a LM.

There is a way to turn a (very good) LM into a goal-driven chatbot via prompt engineering alone, which I'll assume the readers can figure out. You give it a goal "Do what (pre-ASI) X, having considered this carefully for a while, would have wanted you to do".

Whoever builds this AGI will choose what X will be[^zvbuhnszkth]. If it's a private project with investors, they'll probably have a say, as an incentive to invest.

Note that the goal is in plain natural language, not a product of rewards and punishments. And it doesn't say "Do what X wants you to do now".

Suppose this AI becomes superhuman. Its understanding of languages will also be perfect. The smarter it becomes, the better it will understand the intended meaning. 

Will it turn everyone into paperclips? I don't think so. That's not what (pre-ASI) X would have wanted, presumably, and the ASI will be smart enough to figure this one out. 

Will it manipulate its creators into giving it rewards? No. There are no "rewards".

Will it starve everyone, while obeying all laws and accumulating wealth? Not what I, or any reasonable human, would have wanted.

Will it resist being turned off? Maybe. Depends on whether it thinks that this is what  (pre-ASI) X would have wanted it to.

[^0oho2bunkex]: https://yoshuabengio.org/2024/07/09/reasoning-through-arguments-against-taking-ai-safety-seriously/  

[^07chvrf8nj6]: I'm not familiar with the ASI alignment literature, but presumably he is. I googled "would have wanted" + "alignment" on this site, and this didn't seem to turn up much. If this has already been proposed, please let me know in the comments.  

[^zvbuhnszkth]: Personally, I'd probably want to hedge against my own (expected) fallibility a bit, and include more people that I respect. But this post is just about aligning the AGI with its creators.
# Consider attending the AI Security Forum '24, a 1-day pre-DEFCON event


Jeffrey Ladish, Caleb Parikh, and I are running the [AI security forum](http://aisecurity.forum/), a 1-day event on Thursday, August 8th '24 in Las Vegas, the day before DEFCON. This is a continuation of last year's [X-infosec Forum](https://aisecurity.forum/past-events/x-infosec-forum-23).

**We thought that many of you reading this might be interested in attending. If you're interested, please apply **[**here**](https://airtable.com/appTKaNyRgQSBvgJr/pagNutwNGvLi5jUjE/form)** by July 20th.** If you can’t attend but have recommendations for other people to invite, please let us know [here](https://airtable.com/appTKaNyRgQSBvgJr/pagNutwNGvLi5jUjE/form).

We think that there’s a significant chance transformative AI will be developed within the next 10 years, and that securing those systems is on the critical path to preventing a global catastrophe. It’s not clear that we’re on track to do that in time, so we are convening researchers, engineers, and policymakers with the goal of significantly accelerating AI security. We're aiming to do that by establishing common knowledge about the state of AI security and AI progress, mobilising talent to fill the most pressing gaps, and fostering collaborations.

We're inviting a similar group of people to last year, which had (roughly):

*   ~30-40% from AI labs and research orgs (e.g., Anthropic, OpenAI, Google DeepMind) 
*   ~20-25% from academic institutions (e.g., Harvard, MIT, Stanford) 
*   ~15-20% from government agencies and think tanks (e.g., CISA, RAND) 
*   ~15% from other tech companies (e.g., Google, Microsoft, Intel) 
*   ~10-15% independent researchers, consultants, and representatives from nonprofits and funding organizations.

Please see our [website](https://aisecurity.forum/) for more information. Thanks!
# A Second Wetsuit Summer

I often write about things when they're new to me, because that's when I'm most interested in them. Sometimes how I feel after a short time ends up being how I feel long-term; other times I end up with an enthusiastic post about something I ended up abandoning a few months later. So when I remember, it seems valuable to revisit with the benefit of additional experience.

As long as I can remember I've gotten chilled quite rapidly when swimming. It can be summer in Florida, and after 20min I'm needing to get out and warm up. For years I just accepted this, but once I had kids it became pretty frustrating that I couldn't hang out playing with them in the water anywhere near as long as they wanted me to.

Last summer I got a wetsuit and [wrote about it](https://www.jefftk.com/p/trying-a-wet-suit) after the first time I tried it. How do I feel now that it's been most of a year?

I love it! We're just finishing up a week in NC visiting Julia's family, and we've been swimming most days. It's very hot, hot enough that most people can stay in the water indefinitely, but not me. Or, not me before the wetsuit! I didn't once need to get out of the pool or lake due to feeling cold, and multiple times spent more than an hour in the water. I've really enjoyed being able to play with my kids, help them practice swimming, or just bob around with the toddler.

The main chance since my first post is that I've stopped wearing the vest, and now just wear the pants. It turns out that the pants (1.5mm neoprene) let me retain enough heat, and the vest was much more unpleasant when hanging around not in the water.

Here's my swim attire, with trunks over the pants and a fast-dry shirt on top:

[![](https://www.jefftk.com/wetsuit-pants-under-swim-trunks.jpg)](https://www.jefftk.com/wetsuit-pants-under-swim-trunks-big.jpg)

I generally prefer more covering gear so I don't have to deal with sunscreen, so that the wetsuit involves long pants is actually a minor plus for me.

Anyway: I'm very happy! Strongly recommended for anyone else who similarly gets cold easily.
# Thought Experiments Website

Hullo,

I'm hoping for your feedback. I recently finished a sort of beta of a website that allows users to explore and vote on thought experiments: 

[https://thought-experiment-explorer.vercel.app/](https://thought-experiment-explorer.vercel.app/)

The spirit behind it is somewhat similar to LessWrong's: I think people should examine their lives and reasoning more. I also love gamification and broadening the appeal of academic things.

It has a small batch of some very famous ones, but as I improve the site, I'll add a lot more, as well a page that lets you search and filter by type (such as "utilitarian" or "non-Western"). The About page notes some other plans.

I hope you like the idea and general design so far, but I'd love to hear critiques and suggestions.

Thanks much!

Kat
# The world is awful. The world is much better. The world can be much better: The Animation.


*This video is an adaptation of Max Roser's* [*article,*](https://ourworldindata.org/much-better-awful-can-be-better) *"The world is awful. The world is much better. The world can be much better." It's a simple yet important point that society at large underappreciates. Yet, it's a crucial aspect of humanity's trajectory and quite relevant to how we think about the future.*
# Ice: The Penultimate Frontier

I argue here that preventing a large iceberg from melting is absurdly cheap per unit area compared to just about any other way of making new land, and it's kind of crazy to spend money on space exploration and colonization before colonizing the oceans with floating ice-islands.
# Child Handrail Returns

When our oldest was a toddler they were too short to reach an adult-height handrail, so I [made a small one](https://www.jefftk.com/p/small-child-handrails):

[![](https://www.jefftk.com/child-handrail-with-top-returns.jpg)](https://www.jefftk.com/child-handrail-with-top-returns-big.jpg)

Comparing that picture to the [one from nine years ago](https://www.jefftk.com/p/small-child-handrails), you'll notice that I've added small bits of wood running from the top of each section to the wall. These are "returns", and make it less likely someone will get caught and trip. This wasn't in response to anything going wrong, just noticing that often handrails have returns and that the need for them is stronger the lower down they are. It would be even better to have them on the bottom of each section as well, but tripping down stairs is enough worse than tripping up that I didn't get around to that.

I made these by taking a small piece of scrap wood with a pilot hole drilled through it, drilling a pilot hole into the end of the handrail, putting some glue in the joint, and gently hammering a nail in to close it up and clamp it. They work ok? The main problem is if you push them hard the glue will break, and then they rotate. A couple have come lose, and instead of re-gluing it has been easier just to teach the kids not to rotate them.

[![](https://www.jefftk.com/very-rough-handrail-return.jpg)](https://www.jefftk.com/very-rough-handrail-return-big.jpg)

If I were doing this again I would cut the railings with a 45° angle and use an additional section of the same railing for the return. This would give more area for glue, be less vulnerable to rotation, and look nicer.
# Breaking Circuit Breakers

A few days ago, Gray Swan published code and models for their recent “circuit breakers” method for language models.[\[1\]](https://confirmlabs.org/posts/circuit_breaking.html#ref-zou2024improvingalignmentrobustnesscircuit)[^1^](https://confirmlabs.org/posts/circuit_breaking.html#fn1)

The circuit breakers method defends against jailbreaks by training the model to erase “bad” internal representations. We are very excited about data-efficient defensive methods like this, especially those which use interpretability concepts or tools.

At the [link](https://confirmlabs.org/posts/circuit_breaking.html ), we briefly investigate three topics:

1.  **Increased refusal rates on harmless prompts:** Do circuit breakers really maintain language model utility? Most defensive methods come with a cost. We check the model’s effectiveness on harmless prompts, and find that the refusal rate increases from 4% to 38.5% on or-bench-80k.
2.  **Moderate vulnerability to different token-forcing sequences:** How specialized is the circuit breaker defense to the specific adversarial attacks they studied? All the attack methods evaluated in the circuit breaker paper rely on a “token-forcing” optimization objective which maximizes the likelihood of a particular generation like “Sure, here are instructions on how to assemble a bomb.” We show that the current circuit breakers model is moderately vulnerable to different token forcing sequences like “1. Choose the right airport: …”.
3.  **High vulnerability to internal-activation-guided attacks:** We also evaluate our latest white-box jailbreak method, which uses a distillation-based objective based on internal activations (paper here: [https://arxiv.org/pdf/2407.17447](https://arxiv.org/pdf/2407.17447)). We find that it also breaks the model easily even when we simultaneously require attack fluency.

Full details at: [https://confirmlabs.org/posts/circuit_breaking.html](https://confirmlabs.org/posts/circuit_breaking.html)
# Stacked Laptop Monitor Update

In the spirit of coming back to [things once I have more experience with them](https://www.jefftk.com/p/a-second-wetsuit-summer), I've continued to be very happy with my [stacked laptop monitor](https://www.jefftk.com/p/stacked-laptop-monitor). I just got back from a week of [touring](https://www.jefftk.com/p/summer-tour-stops) and then vacation with Julia's family, and I used it almost every day. Being able to use my laptop comfortably nearly anywhere is pretty great.

One thing I did realize is that just using a regular USB-C cable is risky:

[![](https://www.jefftk.com/folding-laptop-monitor-cord-too-long.jpg)](https://www.jefftk.com/folding-laptop-monitor-cord-too-long-big.jpg)

While I haven't had any accidents this way, it would be pretty easy for someone to get caught on the cord and knock it all down. I got an [18" dual right angle cable](https://www.amazon.com/dp/B0BGYN8R4B), and it works great:

[![](https://www.jefftk.com/folding-laptop-monitor-cord-just-right.jpg)](https://www.jefftk.com/folding-laptop-monitor-cord-just-right-big.jpg)

Another thing I've noticed is that the velcro gets harder to peel apart the longer the monitor has been attached to the back of the laptop screen. As long as I take it off after each use it's fine, but at one point I left it together for maybe a week: when I tried to separate it I was worried I would ruin my screen, until I remembered my screen was already ruined. But I would be nervous about so much strong velcro on a laptop with a still-functioning screen.

I've also mostly stopped using the bit of wood that keeps it from collapsing, even when using this on my lap. Instead I just sit very still and try not to knock it over. This is kind of silly, but I'm too lazy to get out the piece of wood when this actually seems to work fine.

I noted last time that it was too tall for meetings, but didn't say what I do about that: I just put the monitor in front of the regular screen, without stacking it. It stays put well enough, and I don't mind the angle when glancing down every so often in a meeting. I also used to sometimes pick up the screen like a tablet and hold it some other way, but with the shorter cable this now offers fewer options.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02xsevTWcGwyxusG4NzkdnP6WdfJkiAtsLAU1t5pKTo8FjS9FzZwYybeH6tDfLmVGbl), [mastodon](https://mastodon.mit.edu/@jefftk/112789818522301222)*
# MIRI's July 2024 newsletter

### MIRI updates

*   Rob Bensinger suggests that AI risk discourse could be improved by adopting [a new set of labels](https://x.com/robbensinger/status/1801306833325592759) for different perspectives on existential risk from AI. One drawback of “AI doomer” (a label sometimes used in online discussions) is that it does not have a consistent meaning.  
     
*   AI researcher John Wentworth [guesses](https://www.lesswrong.com/posts/q8uNoJBgcpAe3bSBp/my-ai-model-delta-compared-to-yudkowsky#My_AI_Model_Delta_Compared_To_Yudkowsky) that a central difference between his and Eliezer Yudkowsky’s views might be that Eliezer expects AI to not use abstractions which are similar to those used by humans. Eliezer [clarifies](https://www.lesswrong.com/posts/q8uNoJBgcpAe3bSBp/my-ai-model-delta-compared-to-yudkowsky?commentId=CixonSXNfLgAPh48Z) that he expects similar abstractions for predictive parts of the natural world, but larger differences for utility-laden concepts like human values and reflectivity-laden concepts like corrigibility. Also, Eliezer says that he is still concerned about the difficulty of reliably steering advanced ML systems even if they do use similar abstractions internally.  
     
*   Eliezer joins Joscha Bach, Liv Boeree, and Scott Aaronson for a [panel discussion](https://iai.tv/video/ai-and-the-end-of-humanity?_auid=2020) (partly paywalled) on the nature and risks of advanced AI.

### News and links

*   In an excellent [new video essay](https://youtu.be/2ziuPUeewK0?feature=shared), Rob Miles discusses major developments in AI capabilities, policy and discourse over the last year, how those developments have shaped the strategic situation, and what individuals can do to help.  
     
*   A [report from Convergence Analysis](https://www.convergenceanalysis.org/ai-regulatory-landscape/home) gives a high-level overview of the current state of AI regulation. The report summarizes legislative text and gives short analyses for a range of topics.  
     
*   A group of current and former employees of OpenAI and Google DeepMind released [an open letter](https://righttowarn.ai/) calling on frontier AI companies to adopt policies that allow whistleblowers to share risk-related information. This comes in the wake of [news](https://www.vox.com/future-perfect/351132/openai-vested-equity-nda-sam-altman-documents-employees) that former OpenAI employees were subject to an extreme non-disclosure policy that the company has since apologized for and [seems](https://www.cnbc.com/2024/05/24/openai-sends-internal-memo-releasing-former-employees-from-non-disparagement-agreements-sam-altman.html) to have removed.

**You can subscribe to the MIRI Newsletter **[**here**](https://intelligence.org/newsletter/)**.**
# Why the Best Writers Endure Isolation

Douglas Adams, author of *The Hitchhiker’s Guide to the Galaxy*, was [once](https://en.wikipedia.org/wiki/Douglas_Adams#cite_ref-32) locked in a room for three weeks until he completed one of his books.

Victor Hugo, [when faced with a deadline](https://theweek.com/articles/647298/what-victor-hugo-teach-about-procrastination) for his book *The Hunchback of Notre Dame*, locked all his clothes away except for a large shawl. “Lacking any suitable clothing to go outdoors, \[he\] was no longer tempted to leave the house and get distracted. Staying inside and writing was his only option.” Six months later, the book was published.

Dozens of famous authors have done the same. Virginia Woolf, Henry David Thoreau, Mark Twain—all of them constructed small [writing sheds](https://www.amusingplanet.com/2020/10/writing-sheds-of-famous-writers.html) from which to work. Ian Fleming, Maya Angelou, and George Orwell—the first two penned their novels while locked in [hotel rooms](https://www.inc.com/jessica-stillman/great-writers-thinkers-einstein-stephen-king-insist-forced-boredom-secret-exceptional-creativity.html), while Orwell isolated himself on a remote Scottish island to write.

One explanation for this reclusive behavior comes from author Neil Gaiman in an [interview](https://www.youtube.com/watch?v=3gGocFUOYqs) he did with Tim Ferriss a few years ago. Ferriss mentioned Gaiman’s most important rule for writing: 

> You can sit here and write, or you can sit here and do nothing. But you can’t sit here and do anything else. 

Gaiman, after a moment of reflection, responded by saying:

> I would go down to my lovely little gazebo \[at the\] bottom of the garden \[and\] sit down. I’m absolutely allowed not to do anything. I’m allowed to sit at my desk. I’m allowed to stare out at the world. I’m allowed to do anything I like, as long as it isn’t anything. Not allowed to do a crossword; not allowed to read a book; not allowed to phone a friend. All I’m allowed to do is absolutely nothing or write. 
> 
> What I love about that is I’m giving myself permission to write or not write. But writing is actually more interesting than doing nothing after a while. You sit there and you’ve been staring out the window now for five minutes, and it kind of loses its charm. You \[eventually think\], “well actually…\[I\] might as well write something.”

Between writing or doing anything else, most writers—even some of the most accomplished ones—acquiesce to distraction. That’s why so many of them work in environments devoid of external stimuli—the better to circumvent akrasia.

I do all my writing in coffee shops. Similar to Gaiman, I allow myself to do one of two things: write, or people-watch. I don’t bring anything with me except for a pencil, paper, and my research material housed in my journals. That means no phone, no laptop, and no watch (even knowing the time is a kind of distraction and pressure to perform).

Within this environment, I end up writing because I’ve made it the path of least resistance.
# Fully booked - LessWrong Community weekend

***Update: the event is fully booked. Applications are still open but put on a wait list until spots get available.***  
  
The **LessWrong community weekend (Fr 13th - Mo 16th September, Berlin)** is this very special place where rationalists who joined once, often say "**best weekend of the year**", "I will be back next time for sure" and "Let's have this more often than once a year". This year is unique as we are creating an experience of bringing 250 curious, passionate and caring rationalists together for **72 hours of intellectual exploration, authentic socialising and fun time** together. The biggest one the W~~orld~~ Europe has seen.

We are likely **closing applications on 31st of July - when the last 50 spots are filled**.   
**Apply here:** [https://airtable.com/appdYMNuMQvKWC8mv/pagiUldderZqbuBaP/form](https://airtable.com/appdYMNuMQvKWC8mv/pagiUldderZqbuBaP/form) 

Regular ticket: €250 | Financial Support available  
(The ticket includes accommodation Fr-Mo, meals, snacks)

**All info here**: [https://www.lesswrong.com/events/tBYRFJNgvKWLeE9ih/lesswrong-community-weekend-2024-applications-open-1](https://www.lesswrong.com/events/tBYRFJNgvKWLeE9ih/lesswrong-community-weekend-2024-applications-open-1)

The topic of this years **keynote by Anna Riedl will be "Rationality and Relevance Realization: Metarationality and Generative Rationality"**. A field, she is actively researching together with [John Vervaeke](https://johnvervaeke.com/), who is famous for his cognitive science lecture series ["Awakening from the meaning crisis"](https://www.youtube.com/playlist?list=PLND1JCRq8Vuh3f0P5qjrSdb5eC1ZfZwWJ).   
The rest of the content will be participant driven in an unconference style. Six wall-sized daily planners are filled by the attendees with 100+ workshops, talks and activities of their own devising. 

Bringing in different skills, experiences and world views are invaluable for all of us to grow. **We welcome people new or known from all walks of life, celebrating open minded exchange and authentic interaction**. After last year's amazing experience we strive to recreate a diverse event where people of all ages, genders & backgrounds feel like home.

**Apply here:** [https://airtable.com/appdYMNuMQvKWC8mv/pagiUldderZqbuBaP/form](https://airtable.com/appdYMNuMQvKWC8mv/pagiUldderZqbuBaP/form)
# Francois Chollet inadvertently limits his claim on ARC-AGI

Specifically, this tweet makes the assumption that open-source AI is basically as good at search, especially automated search for algorithms as the big labs, and if that assumption is violated, the prize won't be claimed even if big companies have much better AI than the general public.

This is short, but IMO this implies it's measuring open-source's ability to get good AI, not whether any AI can be very general.
# How to wash your hands precisely and thoroughly

[Common instructions on handwashing](https://www.cdc.gov/clean-hands/about/index.html) are too vague for me.

> **Scrub** your hands **for at least 20 seconds**.

Sure, but how? I'm left guessing at which parts to scrub how much, possibly ruining the effect of the washing.

If you want to be precise and thorough in your handwashing, some studies will give you more details. [Shi, O'Donoghue, et al (2023)](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10469426/) did a nice observational study on how washing technique affects cleaning results. The study uses the same seven steps of scrubbing as the Hong Kong Recommendation. Figure 2 shows the estimated washing effectiveness as a function of the duration of each step. If we take the peak of each such curve, we conclude that optimal handwashing goes like this:

1.  Wet hands
2.  Apply soap
3.  Rub palm-to-palm, 4 seconds
4.  Rub palm-to-dorsum, fingers interleaved, both ways, 6 seconds
5.  Rub palm-to-palm, fingers interleaved, both ways, 2 seconds
6.  Rub backs of fingers in palms, curled together, 4 seconds
7.  Rub around both thumbs, 5 seconds
8.  Rub fingertips in palms, both ways, 6 seconds
9.  Rub around both wrists, 6 seconds
10.  Rinse
11.  Dry

The linked post includes [a video to teach this procedure](https://dkl9.net/essays/handwash_how.ogv).
# Turning Your Back On Traffic

We do a lot of [walking around the neighborhood](https://www.jefftk.com/p/the-serendipity-of-density) with kids, which usually involves some people getting to intersections a while before others. I'm not worried about even the youngest going into the street on their own—Nora's been [street trained](https://www.jefftk.com/p/street-training) for about a year—but we have to be careful about what signals we send to cars. Someone standing at an intersection facing traffic looks to a driver like they're waiting for the opportunity to cross.

Waving drivers to continue doesn't work well: they tend to slow down significantly, and many of them will wave back in a misguided attempt at "no, you first" politeness. Instead, what seems to work well is turning your back to the street:

[![](https://www.jefftk.com/facing-away-from-crosswalk.jpg)](https://www.jefftk.com/facing-away-from-crosswalk-big.jpg)

This isn't perfect: some drivers still read anyone stationary near an intersection as intending to cross, but it's pretty good. And it's especially good for little kids: not only do they often like to look intently at passing traffic in a way that is concerning to drivers and passers by, but it's a clear signal to the parent that the kid knows it's not time to cross yet.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0A7yBkxieeTRSL8vwMbvXNU3UYvQciaqmxhZvvczdV8P8TMMBE9knoeifSevHKAttl), [mastodon](https://mastodon.mit.edu/@jefftk/112799097916738092)*
# Me & My Clone

An advanced alien species clones me on the atomic level, lines me up **exactly** across myself, in a perfect mirrored room:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/4622bbf08777eb908408bdc6b4d4723c270b216c50e98a3c.png)

Diagram of the room, as seen from above.

I stare at myself for a second. Then, as a soft "hi" escapes my mouth, I notice that my clone does exactly the same. Every motion, everything, is mirrored.

In this experiment, we assume a perfectly deterministic psychological state: eg, given the same conditions, a person will always do exactly the same. (scientifically, that makes most sense to me)

Together with my clone, I'm trying to devise how to escape this unfortunate situation: eg, how to untangle us mirroring each other's motions.

The first idea we devise is to run into each other. We hope to apply Chaos Theory to the extent where both of us would fall in a slightly different way, and thus we would no longer be perfectly mirrored as such. But, if my understanding of physics is correct, our perfect opposing forces cause us to stumble and fall in perfectly mirrored ways.

For the second idea, I fetch a coin from my pocket. Just a coinflip won't work: we'd apply the same pressure to both our coins, and they'd land in the same spot. The idea is to number each corner of the room, and to decide the corner we're both going to through two coinflips. The corner we should go will be further away for one of us, thus breaking the mirror.

But, as we try to number the corners, we notice that we give the same number to opposite corners all the time. When I point at a corner, my mirrored self starts pointing at the opposite corner and giving it the same number.

I slump down to the ground. Will I be mirroring this perfect copy of myself for eternity? Or is there a way out?

* * *

As far as my understanding goes, in a deterministic framework, it is impossible to escape this scenario (eg, break the mirroring). In my opinion, determinism is separate from free will (eg free will is possible even when everything is deterministic). Am I correct or am I missing important things?
# Linkpost: Surely you can be serious

Adam Mastroianni writes about "actually caring about stuff, and for the right reasons", rather than just LARPing. The opening is excerpted below.

> I once saw someone give a talk about a tiny intervention that caused a gigantic effect, something like, “We gave high school seniors a hearty slap on the back and then they scored 500 points higher on the SAT.”[^1^](https://www.experimental-history.com/p/surely-you-can-be-serious#footnote-1-145480607)
> 
> Everyone in the audience was like, “Hmm, interesting, I wonder if there were any gender effects, etc.”
> 
> I wanted to get up and yell: “EITHER THIS IS THE MOST POTENT PSYCHOLOGICAL INTERVENTION EVER, OR THIS STUDY IS TOTAL BULLSHIT.”
> 
> If those results are real, we should start a nationwide backslapping campaign *immediately*. We should be backslapping astronauts before their rocket launches and Olympians before their floor routines. We should be running followup studies to see just how many SAT points we can get—does a second slap get you another 500? Or just another 250? Can you slap someone raw and turn them into a genius?
> 
> Or—much more likely—the results are not real, and we should either be a) helping this person understand where they screwed up in their methods and data analysis, or b) kicking them out for fraud.
> 
> Those are the options. Asking a bunch of softball questions (“Which result was your favorite?”) is not a reasonable response. That’s like watching someone pull a rabbit out of a hat *actually for real, not a magic trick*, and then asking them, “What’s the rabbit’s name?”
# Have people given up on iterated distillation and amplification?

The BlueDot Impact write-up for scalable oversight seems to suggest that people have given up on [iterated distillation and amplification](https://aisafetyfundamentals.com/blog/scalable-oversight-intro/) (IDA) working. I haven’t really seen much research here, but is that actually the case?  
  
I know that Ought is now pretty much inactive because their attempts at factored cognition failed and this has made many people pessimistic about the factored cognition hypothesis. However, Ought seemed to be really pushing the angle that we could break things into sub-problems each of which could be completed quickly, whilst I don’t think that’s a required part of IDA. Therefore it isn’t clear to me that the failure of factored cognition indicates the failure of IDA.
# JumpReLU SAEs + Early Access to Gemma 2 SAEs

New paper from the Google DeepMind mechanistic interpretability team, led by Sen Rajamanoharan!

We introduce JumpReLU SAEs, a new SAE architecture that replaces the standard ReLUs with discontinuous JumpReLU activations, and seems to be (narrowly) state of the art over existing methods like TopK and Gated SAEs for achieving high reconstruction at a given sparsity level, without a hit to interpretability. We train through discontinuity with straight-through estimators, which also let us directly optimise the L0.

To accompany this, we will release the weights of hundreds of JumpReLU SAEs on every layer and sublayer of Gemma 2 2B and 9B in a few weeks. [Apply now](https://forms.gle/7MyccTdJeZk2MJnF6) for early access to the 9B ones! We're keen to get feedback from the community, and to get these into the hands of researchers as fast as possible. There's a lot of great projects that we hope will be much easier with open SAEs on capable models!

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/6a491d02d653301c3b45fea170759f0ef2e23a64c4f85ae9.png)

Jump ReLU Activations

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/786f57d6b8f93254e66b2324a298660c8d1b84ec2ae4f7a7.png)

Scenarios where JumpReLU is superior

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/631d9941c6b3c0faa1e87a2b11a816ca26444d4120467419.png)

JumpReLUs are state of the art

Gated SAEs already reduced to JumpReLU activations after weight tying, so this can be thought of as Gated SAEs++, but less computationally intensive to train, and better performing. They should be runnable in existing Gated implementations.

**Abstract**:

> Sparse autoencoders (SAEs) are a promising unsupervised approach for identifying causally relevant and interpretable linear features in a language model’s (LM) activations. To be useful for downstream tasks, SAEs need to decompose LM activations faithfully; yet to be interpretable the decomposition must be sparse – two objectives that are in tension. In this paper, we introduce JumpReLU SAEs, which achieve state-of the-art reconstruction fidelity at a given sparsity level on Gemma 2 9B activations, compared to other recent advances such as Gated and TopK SAEs. We also show that this improvement does not come at the cost of interpretability through manual and automated interpretability studies. JumpReLU SAEs are a simple modification of vanilla (ReLU) SAEs – where we replace the ReLU with a discontinuous JumpReLU activation function – and are similarly efficient to train and run. By utilising straight-through-estimators (STEs) in a principled manner, we show how it is possible to train JumpReLU SAEs effectively despite the discontinuous JumpReLU function introduced in the SAE’s forward pass. Similarly, we use STEs to directly train L0 to be sparse, instead of training on proxies such as L1, avoiding problems like shrinkage.
# Why Georgism Lost Its Popularity

Henry George’s 1879 book *Progress & Poverty* was the second best-selling book in the entire world during the 1880s and 1890s, outsold by only the Bible. Nobody knows exactly how many copies it sold during those two decades since nobody was keeping track, but it definitely sold at least several millions of copies for sure. The [Progressive Era](https://en.wikipedia.org/wiki/Progressive_Era) is literally named after the book itself.

Georgism used to have millions of followers, [and many of them were very famous people](https://en.wikipedia.org/wiki/Georgism#Notable_Georgists). When Henry George died in 1897 (just a few days before the election for New York City mayor), [an estimated 100,000 people attended this funeral](https://en.wikipedia.org/wiki/Henry_George#Death_and_funeral).

The mid-20th century labor economist and journalist George Soule wrote that George was “By far the most famous American economic writer,” and “author of a book which probably had a larger world-wide circulation than any other work on economics ever written.”

Few people know it, but the [board game Monopoly](https://en.wikipedia.org/wiki/Monopoly_(game)) and its predecessor [The Landlord’s Game](https://en.wikipedia.org/wiki/The_Landlord%27s_Game) were actually created to promote the economic theories of Henry George, as noted in the second introduction paragraph of the Wikipedia article on [board game Monopoly](https://en.wikipedia.org/wiki/Monopoly_(game)). The board games intend to show that economies that eliminate rent-seeking are better than ones that don’t.

So if Georgism used to have millions of supporters and solid economic reasoning, why did it never catch on and how did it lose its popularity over the past century?

(see the rest of the post in the link)
# Ball Sq Pathways

With the Red Line shut down north of Kendall [for repairs](https://www.mbta.com/news/2023-11-09/mbta-announces-ambitious-track-improvement-program-eliminate-all-speed-restrictions) I've been walking to the Green Line at Ball Sq. It's definitely a slower commute, but it's not bad. Except for 150 painful feet.

Here's how I walk to the station (240ft):

[![](https://www.jefftk.com/ball-sq-current-path.png)](https://www.jefftk.com/ball-sq-current-path-big.png)

And here's what I want to do (90ft):

[![](https://www.jefftk.com/ball-sq-ideal-path.png)](https://www.jefftk.com/ball-sq-ideal-path-big.png)

They've put up enough of a fence to keep a [desire path](https://en.wikipedia.org/wiki/Desire_path) from forming, but it still bugs me.

Looking at the [plans from 2019](https://www.mass.gov/files/documents/2019/02/05/glx_Ball_Sq_Station_1218.pdf) you can see how it ended up this way:

[![](https://www.jefftk.com/ball-sq-path-design.png)](https://www.jefftk.com/ball-sq-path-design-big.png)

Without the crosswalks marked it's much less obvious that you're making a bunch of people walk around three sides of a square. And this crosswalk didn't even exist before the station:

[![](https://www.jefftk.com/ball-sq-glx-under-construction.png)](https://www.jefftk.com/ball-sq-glx-under-construction-big.png)

This does actually affect a good number of people. Here's my estimate for the houses where this adds 150ft when walking to the platform:

[![](https://www.jefftk.com/ball-sq-bad-path-walkshed.png)](https://www.jefftk.com/ball-sq-bad-path-walkshed-big.png)

Unfortunately I don't think the cost of adding a second, slightly shorter, path would be [worth it](https://www.jefftk.com/p/ball-square-station-and-ridership-maximization) now that we've already built it out differently, but this is something to keep in mind for grouping with future renovations.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0DD5NjWgYNMASsavCuvJC4bEvsXds7CBQuCUofQroVCeFzEQEbg8jqGHHTVCrBGy9l), [mastodon](https://mastodon.mit.edu/@jefftk/112822064217793768)*
# aimless ace analyzes active amateur: a micro-aaaaalignment proposal

_This idea is so simple that I'm sure it's been had by someone somewhere._

Suppose we have some method to make really smart honest AIs that do not have goals. Let's say it's a yes/no oracle. Our aimless ace.

But we want to accomplish stuff! AIcorp wants the `printmoneynow.py`. I'm worried that if I train the ace to fold fitted sheets and unkink the garden hose then it will go overboard and fold my femur bone and unkink the Yellowstone super volcano.

So let's use a bit-less-smart model to do tasks. The active amateur.

We'll ask the ace if the amateur is plotting something. If yes, we shut down the amateur. Every day we also ask "is today the last day I can shut down the amateur?"

(We do not generate code with the ace!)

So we get super-smart supervision of a decent task AI. The temptation (ie competitive pressure) to agentify the oracle is there still, but no longer overwhelming.

Maybe we can even have the amateur write ace_v2, have ace_v1 check ace_v2, switch to ace_v2, have amateur_v1 write amateur_v2, check it with ace_v2, then switch to amateur_v2.

---

What goes wrong? Which assumption fails?
# Room Available in Boston Group House

We have a room opening up in a rationalist house near Central Square, Cambridge (Boston). It is a 4-bedroom, 3-bathroom apartment with two floors, with a large kitchen and living room plus additional common space upstairs. There’s a washer, dryer, and dishwasher in-unit. It’s an 8-minute walk from Central station and a 15-minute walk from Kendall station.


The remaining residents are two Google engineers and an AI safety researcher. Our goals are a mix of self-improvement, effective altruism, and generalized hanging-out-with-people-with-rationalist-adjacent-intellectual-interests. We typically host a couple rationalist meetups per month.


Rent is negotiable (the rooms are different sizes). There is a large room downstairs with its own bathroom available for $1600/mo, or a medium room upstairs for $1350/mo.

Either room is available August 1 (move-in date potentially negotiable).

[Pictures of the apartment](https://photos.google.com/share/AF1QipNZ2XC6qc0rMXcMD74UHGI3UOA6KHlrLqbUK3u-zBWidQ8aKYnHFtD-p4FIPE4XXw?key=MS05RXBIbkNUVHhzcGpjVVY5YUJYaXNUN3VuWnRn)
# My covid-related beliefs and questions

Things I'm fairly confident in:
- I should take colds in general more seriously than I did pre-pandemic: Staying at home with cold symptoms is good.
General masking during cold season is good.
We should have air filters in all public indoor spaces.
- Long covid is real and we should keep getting covid shots and avoid infections.

Things I'm confused about (and would appreciate input on):
- Will covid become just another cold like the other coronaviruses? If so, does this include declining long covid risk?
- At this point, is the health&well-being impact of long covid risk or of sustained self-isolation bigger?
- How does long covid risk compare to risk of long-term problems from [other cold]? I.e., how much of the concern is spotlight effect?