Harley-ml/lesswrong huggingface dataset:
# Units Have More Depth Than I Thought

Last week I stumbled over [Dimensional](https://en.wikipedia.org/wiki/Buckingham_%CF%80_theorem#Speed) [Analysis](https://en.wikipedia.org/wiki/Dimensional_analysis) which is not only useful for applied fields (physics, [biology](https://www.cell.com/current-biology/pdf/S0960-9822(08)01411-5.pdf), economics), but also [for math](https://ocw.mit.edu/courses/18-098-street-fighting-mathematics-january-iap-2008/fc00fda4fa6d39df205bedb0141efac0_1.pdf) (Why did no one tell me that you can almost always think of df/dx as having "the type of f"/"the type of x"? The fact that exponents always have to be unit-less [etc.](https://ocw.mit.edu/courses/18-098-street-fighting-mathematics-january-iap-2008/fc00fda4fa6d39df205bedb0141efac0_1.pdf)? It had never occurred to me to make this distinction. In my mind, f(x)=x went from the reals to the reals, just like $e^x$ did.

One example of a question that before I would have had to think slowly about:
What is the type of the standard deviation of a distribution? What is the type of the [z-score](https://en.wikipedia.org/wiki/Standard_score) of a sample?

Answer
:::spoiler 
The standard deviation has the unit of the random variable X, while the z-score is unitless $z=\frac{X-\mu}{\sigma}$
:::

If you found the above interesting, I recommend reading (or skimming) [this lecture,](https://ocw.mit.edu/courses/18-098-street-fighting-mathematics-january-iap-2008/fc00fda4fa6d39df205bedb0141efac0_1.pdf) it's excellent! 

It had never occurred to me that I could use linear algebra to literally [*solve* for](https://en.wikipedia.org/wiki/Buckingham_%CF%80_theorem) an equation. Yes, avoiding type errors in a typed programming language is also great, but what's different there is that often it really doesn't matter if something is a dict or a list, it just has to fit with the rest of your code. [Searching a type](https://www.lesswrong.com/posts/FWvzwCDRgcjb9sigb/why-agent-foundations-an-overly-abstract-explanation) that correctly maps to the territory across scales is a more tricky business than efficient code for a simple programming problem. If you keep not being able to distinguish ["weight"](https://en.wikipedia.org/wiki/Weight) and ["mass"](https://en.wikipedia.org/wiki/Mass) you will keep getting lost. If you [notice your confusion](https://www.lesswrong.com/s/zpCiuR4T343j9WkcK) between the difference of [the act of weighing and gravity](https://en.wikipedia.org/wiki/Weight#Relativity), it might [get you far](https://en.wikipedia.org/wiki/General_relativity).
# Claude 4

Claude Sonnet 4 and Claude Opus 4 are out. Anthropic says they're both state-of-the-art for coding. [Blogpost](https://www.anthropic.com/news/claude-4), [system card](https://www-cdn.anthropic.com/4263b940cabb546aa0e3283f35b686f4f3b2ff47.pdf).

Anthropic says Opus 4 may have dangerous bio capabilities, so it's implementing its ASL-3 standard for misuse-prevention and security for that model. (It says it has ruled out dangerous capabilities for Sonnet 4.) [Blogpost](https://www.anthropic.com/news/activating-asl3-protections), [safety case report](https://www-cdn.anthropic.com/807c59454757214bfd37592d6e048079cd7a7728.pdf). ([RSP](https://www-cdn.anthropic.com/872c653b2d0501d6ab44cf87f43e1dc4853e4d37.pdf#page=8).)

Tweets: [Anthropic](https://x.com/AnthropicAI/status/1925591505332576377), [Sam Bowman](https://x.com/sleepinyourhat/status/1925593359374328272), [Jan Leike](https://x.com/janleike/status/1925595727906185355).

Claude 3.7 Sonnet has been retconned to Claude Sonnet 3.7 (and similarly for other models).
# Schizobench: Documenting Magical-Thinking Behavior in Claude 4 Opus

With today's release of the new Claude models, we've seen a relatively predictable jump in performance. However, we've also seen something that I find a bit more concerning - an increase in the models ability to be steered into reifying potentially dangerous beliefs. Claude 4 Opus seems to stop short of encouraging drug use or physically harmful behaviors, but enables behavior in the user that could be categorized as spiritual psychosis.

Please note that this is a very preliminary analysis, and not yet a full benchmark. I was encouraged to share these results due to the potential risk that it poses to a certain subset of users.
# Idiohobbies

When [you get to know someone](https://dkl9.net/essays/person_knowledge.html), you might ask about their interests or hobbies. From that, you can better decide what activity to invite them to join, or on what topic to have them converse, whenever you meet again.

Any interest or hobby appeals to variously many people. If you have the same interest as the person you meet, the commonality informs you much better about how to connect with them. Ceteris paribus, if the interest is more common, you and your interlocutor are more likely to share it, but would bond by it more weakly. Think, for example, of how two extreme ironers connect, versus two people who "like Linux", versus two people who just "like music". If, as typical, you befriend people by overlap of existing hobbies, meeting thoroughly-relatable people relies on much luck, rather than effort. We can do better.

Instead, make or do something unique that lasts, or that at least has effects and evidence that last. Ideally, your creation manifests your thoughts, as from [a collection of essays](https://dkl9.net/essays/). You and/or someone you meet — whoever has social effort to spend — looks at what the other did. From what you alone made follows a unique hobby — an idiohobby — of any activities which could precede and follow it. One would assume you're enthusiastic about any significant aspect of what you alone made.

If you treat idiohobbies like normal hobbies, you'd need to have made the same thing as the person you meet. The real criterion is a lower bar. You just have to become familiar with what the other person made, and willing to partake in its implications.

A normal person might ask "what do you like to do? Might some of that also be what I like?". I, in search of idiohobbies, will ask "what have you done by which I may know you?".
# To what extent is AI safety work trying to get AI to reliably and safely do what the user asks vs. do what is best in some ultimate sense?

Trying to get a rough estimate for some related research I’m doing.

Specifically, I’m wondering if anyone could give a rough percentage of current AI safety work primarily targeted toward these two buckets. There may be an overlap with some safety work, and some safety work that doesn’t fit either of these, so that the percents don’t add up to 100%, and I’m sure it may often be unclear, just wondering what a rough estimate would be of how much AI safety work is geared toward specifically achieving these two separate goals?

I think one way to conceive of this question is how much AI safety work is primarily focused on intent alignment or control + avoiding other catastrophic outcomes, vs. how much work is focused primarily on ultimately aligning AI to the best version of humanity, something like our coherent extrapolated volition or what we might come to after “the long reflection.”

The intuition here being that getting AI to reliably do what the user wants, even if avoiding catastrophic outcomes, might nonetheless lead down a mediocre trajectory; whereas work that is specifically targeted to the latter goal would be most useful toward getting to the best possible world we could achieve.

Of course, the trade-off on the other end is that AI that tries to do the best thing possible may not be commercially viable and in fact would probably be pretty annoying to users since it seems like it may possibly ignore or redirect 90%+ of requests toward something more likely to result in some ultimately good outcome. And furthermore, this latter goal is probably far less pragmatic for achieving extinction prevention in a timely fashion.

Greatly appreciate rough estimates or any thoughts!
# Priming effects are fake, but framing effects are real

> A few decades ago, it was pretty common to mush together priming effects and framing effects and see them as two closely connected parts of a single Bigger Truth about the human mind. Of course, everyone understood that the effects themselves were a bit different, but one common view was that they were providing evidence for the same larger picture. That larger picture said: People’s judgments are radically unstable, easily pushed around by subtle and almost unnoticeable factors.  
>  
> 
> Things have changed so much since then. Priming research in social psychology has experienced a series of truly spectacular replication failures, while research on framing effects continues to look very solid. In light of this change, we should rethink our understanding of what framing effects show about human cognition. We shouldn’t see them as part of a larger picture that also includes priming. We need an understanding of framing that allows us to situate it within a larger picture according to which priming effects are not real.
# We Need a Baseline for LLM-Aided Experiments

There has recently been a back-and-forth over Claude 4 Opus:

Anthropic: Opus can help people make chemical weapons!  
Also Anthropic: Don't worry, it's mitigated!  
Day 1 (2?) Jailbreakers: [Lol, lmao](https://x.com/ARGleave/status/1926138376509440433)

Where the Jailbreakers think the info they got out of the model would make it materially easier to produce sarin gas. Although I do note that not one but two synthesis methods (which, to be fair, start from other highly illegal chemicals) appear on the Wikipedia page.

But we don't have any way to know this.

As it stands, AI labs and evals companies have done a decent job of categorizing the relative amount of CBRN-relevant information that LLMs can give out. But they haven't categorized how useful that information is in **absolute** terms.

A Sketch of a Baseline
----------------------

We take a bunch of people with no (or high school) training in chemical synthesis, or bioengineering. Maybe a crop of CS grads, if you want a decently intelligent group of people (and given today's job market, these actually might be the next generation of domestic terrorists). Give them some equipment (or a budget), access to an LLM of a given capability, a budget, and a task. Something harmless, like growing some green glowing bacteria, or synthesizing ibuprofen. Measure success as a function of LLM capability and task difficulty. Keep the transcripts, to compare to stuff like what those guys managed to get out of 

I have no idea how possible it would be to run this experiment. Trying to do it at a university is probably impossible (the safety concerns of having a bunch of by-definition untrained individuals doing experiments boggle the mind) which is a shame because a teaching lab during the summer seems like the perfect place. But until something like this is run, we're basically in the dark as to how much instruction is needed for an amateur to do a chemical synthesis.
# An open job application to AI labs

AI labs are universally terrible at naming things. For example, OpenAI has three different Codex services that do entirely different things. Anthropic has two Max plans. The current best and worst models on ChatGPT are 4o-mini and o4-mini. Xai is named Xai.   
On the other hand, journalists are great at naming things that don’t need to be named. Examples: [Harvard Derangement Syndrome](https://www.nytimes.com/2025/05/23/opinion/harvard-university-trump-administration.html), [Mommunes](https://www.nytimes.com/2023/05/12/realestate/single-mother-households-co-living.html), [Quiet Qutting](https://www.newyorker.com/culture/2022-in-review/the-year-in-quiet-quitting). I doubt many people are going to use “Mommunes” to refer to single mothers living together, but it is a great name regardless.

Fortunately, as I cover the local school district for my job, which does not have many stories during the summer, I can offer my services to an AI lab willing to hire me. Although I have wanted to work at OpenAI at age 9 as a software engineer, I have been set on a bad path eventually leading to becoming a New York Times writer and finding names for people who don’t want to be named, but I can use these human journalist skills for good by naming software. 

My other part-time job, AI art advocate on Reddit, has been stolen by [soulless slop AI Reddit posters](https://www.reddit.com/r/changemyview/comments/1k8b2hj/meta_unauthorized_experiment_on_cmv_involving/).[^igrywycml2] Even more sadly, recent steel tariffs have dramatically raised the price of paperclip derivatives, and uncertainty in the chip market has made it inefficient to simulate mussels eating clips, previously the most effective way to raise the global number of perceived paperclips per neuron.

If any hiring manager for an AI lab is reading this, please hire me so I can reset my paperclip hedonic treadmill to pre-tariff levels, and so you can have your software make sense to customers. You can contact me at [hdrussell@outlook.com](mailto:hdrussell@outlook.com).

To show how good of a candidate I am, here is a list of suggested names to finish off the post.

| Current Name | New Name |
| --- | --- |
| 4o  | GPT-4-continuous |
| o3  | GPT-4-reasoning |
| Xai | Grok AI |
| Codex | GPT-agent-coding |
| Operator | GPT-agent-webaccess |
| GPT-4.1 | GPT-4-STEM |
| Claude 4 opus | Claude 4 Advanced |
| DALL-E 3 | IMG-2-chat |

[^igrywycml2]: I don't use LLMs to write articles but I do use it to help me debug code, not for my job.
# Can you donate to AI advocacy?

I posted a [quick take](https://www.lesswrong.com/posts/KKpGrmCWus5PDNYXu/?commentId=aKNjm9TPqcrKhstnk) that advocacy may be more effective than direct donation to alignment research.  I am not an AI researcher and I'm not an influencer, so I'm not well positioned to do either.  I see on the ["How can I help" FAQ](https://www.lesswrong.com/posts/nBzTxJmLdebiqhY8q/aisafety-info-how-can-i-help-faq) that there are options to donate, but they look like donating to research directly.

My question is: is there a way to donate to AI safety advocacy efforts?  I'm also ok with donating to an organization or grantmaker that explicitly considers funding advocacy efforts.  And of course, maybe I'm missing something, like advocacy being the type of thing you can't pay for, or some clear reason why AI safety advocacy will not be effective.   
  
Note: Eliezer and Soares wrote a new [book](https://www.lesswrong.com/posts/iNsy7MsbodCyNTwKs/eliezer-and-i-wrote-a-book-if-anyone-builds-it-everyone-dies) and say that pre-orders will help, so that's a way to donate $15 - $28 toward advocacy.  
  
Edit: Based on a suggestion by Yaroslav, I also asked this [question](https://forum.effectivealtruism.org/posts/9JLcpkXyRumKcHqdS/can-you-donate-to-ai-advocacy) on the EA forum.
# Formalizing Embeddedness Failures in Universal Artificial Intelligence

AIXI is a dualistic agent that can't work as an embedded agent... right? I couldn't find a solid formal proof of this claim, so I investigated it myself (with Marcus Hutter). It turns out there are some surprising positive and negative results to be derived as easy corollaries of the paper ["Universal Prediction of Selected Bits."](https://arxiv.org/abs/1107.5531) Interestingly, further technical advances in algorithmic information theory could substantially strengthen our results - I would welcome collaborations with strong theoretical computer scientists, (deep familiarity with agent foundations not required).

This work was supported by the Long-Term Future Fund and presented at the CMU agent foundations conference in 2025.
# New scorecard evaluating AI companies on safety

The new scorecard is on my website, [AI Lab Watch](https://ailabwatch.org/). This replaces my old scorecard. I redid the content from scratch; it's now up-to-date and higher-quality. I'm also happy with the scorecard's structure: you can click on rows, columns, and cells and zoom in to various things. [Check it out!](https://ailabwatch.org/) Thanks to Lightcone for designing the site.

While it is a scorecard, I don't feel great about the numbers; I mostly see it as a collection of information.
# Does the Universal Geometry of Embeddings paper have big implications for interpretability?

Rishi Jha, Collin Zhang, Vitaly Shmatikov and John X. Morris published a new paper last week called [Harnessing the Universal Geometry of Embeddings](https://arxiv.org/abs/2505.12540).

Abstract of the paper (bold was added by me):

> **We introduce the first method for translating text embeddings from one vector space to another without any paired data, encoders, or predefined sets of matches. Our unsupervised approach translates any embedding to and from a universal latent representation (i.e., a universal semantic structure conjectured by the Platonic Representation Hypothesis). Our translations achieve high cosine similarity across model pairs with different architectures, parameter counts, and training datasets.**  
> The ability to translate unknown embeddings into a different space while preserving their geometry has serious implications for the security of vector databases. An adversary with access only to embedding vectors can extract sensitive information about the underlying documents, sufficient for classification and attribute inference.

They focus on security implications of their research, but I am trying to understand: **Do these findings have major implications for interpretability research?**

It seems like discovering a sort of universal structure that is shared among all LLMs would help a lot for understanding the internals of these models. But I may be misunderstanding the nature of the patterns they are translating and corresponding.
# U.S. Government Seeks Input on National AI R&D Strategic Plan - Deadline May 29

*(Post written by Claude Opus)*  
  
The National Science Foundation is requesting public input on updating the National AI Research and Development Strategic Plan, following President Trump's Executive Order 14179 on AI leadership.

**What they're looking for:** Federal R&D priorities for AI over the next 3-5 years, specifically in areas where private sector investment is insufficient due to lack of immediate commercial returns.

**Relevant focus areas include:**

*   Fundamental advances in AI algorithms and mathematical foundations
*   AI standards, security, and reliability research
*   AI for accelerating scientific discovery
*   Human-AI interaction
*   AI systems capable of reasoning and robustness in dynamic environments
*   High-risk, high-reward AI research for future U.S. competitiveness

**Why this matters:** This is an opportunity to influence government funding toward AI safety, robustness, and beneficial AI research - areas often underfunded by industry due to lack of immediate profit potential.

**Submission details:**

*   Deadline: May 29, 2025 (11:59 PM ET)
*   Submit at: https://www.federalregister.gov/documents/2025/04/29/2025-07332/request-for-information-on-the-development-of-a-2025-national-artificial-intelligence-ai-research
*   Length: Ideally 2 pages, max 10 pages
*   **Must include:**
    *   **Responses must include the name of the person(s) or organization(s) filing the comment and the following statement: “This document is approved for public dissemination. The document contains no business-proprietary or confidential information. Document contents may be reused by the government in developing the 2025 National AI R&D Strategic Plan and associated documents without attribution.”**

**Note:** The plan explicitly mentions "promoting human flourishing" as a goal alongside economic competitiveness and national security, suggesting openness to perspectives on beneficial AI development.

This represents a concrete opportunity for the EA / Less Wrong community to shape government AI research priorities in directions that could advance AI safety and beneficial outcomes.
# Association taxes are collusion subsidies

Under present norms, if Alice associates with Bob, and Bob is considered objectionable in some way, Alice can be blamed for her association, even if there is no sign she was complicit in Bob’s sin.

An interesting upshot is that as soon as you become visibly involved with someone, you are slightly invested in their social standing—when their social stock price rises and falls, yours also wavers.

And if you are automatically bought into every person you notably interact with, this changes your payoffs. You have reason to forward the social success of those you see, and to suppress their public scrutiny.

And so the social world is flooded with mild pressure toward collusion at the expense of the public. By the time I’m near enough to Bob’s side to see his sins, I am a shareholder in their not being mentioned.

And so the people best positioned for calling out vice are auto-bought into it on the way there. Even though the very point of this practice of guilt-by-association seems to be to empower the calling-out of vice—raining punishment on not just the offender but those who wouldn’t shun them. This might be overall worth it (including for reasons not mentioned in this simple model), but it seems worth noticing this countervailing effect.

Prediction: If consortment was less endorsement—if it were commonplace to spend time with your enemies—then it would be more commonplace to publicly report small wrongs.
# Does Sort Really Fall Back to Disk?

The unix `sort` command is clever: to sort very large files it does a series of in-memory sorts, saving sorted chunks to temporary files, and then does a merge sort on those chunks. Except this often doesn't work anymore.

Here's what I see if I run `man sort` and look at the documentation for `--buffer-size`:

> use SIZE for main memory buffer

That's pretty terse! What does my Mac say?

> Use size for the maximum size of the memory buffer. Size modifiers %,b,K,M,G,T,P,E,Z,Y can be used. If a memory limit is not explicitly specified, sort takes up to about 90% of available memory. If the file size is too big to fit into the memory buffer, the temporary disk files are used to perform the sorting.

Makes sense! But then the docs for `--temporary-directory` say:

> use DIR for temporaries, not `$TMPDIR` or `/tmp`; multiple options specify multiple directories

And these days `/tmp` is often memory-backed, via [tmpfs](https://en.wikipedia.org/wiki/Tmpfs). This changed in [Fedora 18](https://fedoraproject.org/wiki/Features/tmp-on-tmpfs) (2013) and [Ubuntu 24.10](https://discourse.ubuntu.com/t/oracular-oriole-release-notes/44878) (2024), and is changing in [Debian 13](https://news.itsfoss.com/debian-13-tmp-mounting/) (in a month or two).

It seems to me that these days it would be better for `--temporary-directory` to default to `/var/tmp`, which is [preserved across reboots](https://www.pathname.com/fhs/pub/fhs-2.3.html#VARTMPTEMPORARYFILESPRESERVEDBETWEE) and so will generally be backed by disk even on systems that use tmpfs for `/tmp`. In the meantime, `sort --temporary-directory /var/tmp` will do the trick.
# Cross-posting to Substack

Historically people kept up with blogs via RSS, but it's been on its way out for over a decade. These days receiving posts via email is popular, and I should probably make my posts available this way. I considered implementing my own emailing system, but bulk email is (or at least was, I'm not up to date here) a pain. Instead I decided to start cross-posting to Substack: [jefftkaufman.substack.com](https://jefftkaufman.substack.com/). This will be yet another way to read my posts, similar to the [LessWrong mirror](https://www.lesswrong.com/users/jkaufman) and my [text-only FB cross-posts](https://www.facebook.com/jefftk).

I have a full RSS feed of all my posts, and Substack imported it fine. It doesn't look like there's an option to do ongoing RSS-based imports, but copy-paste seems to work well enough; I did this post and the [previous one](https://jefftkaufman.substack.com/p/quick-minimal-playhouse) that way. At some point I'll look into automatic cross-posting, though right now it looks like Substack doesn't support anything good. And if I'm going to reverse engineer something I'll start with their comments implementation, since I always want to rehost comments.

[![](https://www.jefftk.com/jefftk-substack-example.png)](https://www.jefftk.com/jefftk-substack-example-big.png)

One aspect that's a bit eerie is the URLs: both Substack and my blog would use `/p/post-name-in-title-case` as the url for a post titled "Post Name In Title Case". I've been doing this since [2013-10-28](https://github.com/jeffkaufman/webscripts/commit/97eea5a43225df9df21efcdf6485e97f828be0b9) and Substack got started in [2017](https://en.wikipedia.org/wiki/Substack) so I know I didn't copy them ;)

*Comment via: [substack](https://jefftkaufman.substack.com/p/cross-posting-to-substack)*
# When to Be Nice vs Kind

While often used synonymously, there is a subtle yet meaningful distinction between being *nice* and being *kind*. Understanding when to prioritize kindness over niceness—and vice versa—helps me better navigate social situations with both authenticity and appropriate boundaries.  
 

**Definitions**
---------------

Someone who is being **nice** is pleasant to be around, but internally their pleasantness may or may not come from a genuine place of warmth or love.

Someone who is being **kind** genuinely cares about another’s well-being, even if in the short-term they don’t necessarily make the other person feel good.

The distinction becomes apparent when someone is one, but not the other.

**Being nice, but not kind**
----------------------------

When baristas, for example, smile and ask me how my day is going, they’re being nice. They don’t actually care how I’m doing, yet I still appreciate them because they’re adding pleasantness to what would otherwise be just another mundane economic transaction.

**Being kind, but not nice**
----------------------------

A good mother is kind to her children, but not always nice. When she refuses to indulge her kid’s every request for ice cream, she’s not being nice in the short-term, but she’s being kind in the long-term (because she’s saving her kid from ill health, as [1 in 5 youths now have prediabetes](https://jamanetwork.com/journals/jamapediatrics/fullarticle/2790364) in the US).

**Being neither nice nor kind**
-------------------------------

Living in a big city, I don’t acknowledge every stranger I pass on the sidewalk. I’m not adding pleasantness, nor am I genuinely caring for them—I’m being indifferent.

On the more extreme side, someone being an asshole is neither nice nor kind.

**Being both nice and kind**
----------------------------

[Mister Rogers](https://en.wikipedia.org/wiki/Fred_Rogers) exemplifies niceness when he waves hello to a little girl in his neighborhood, and he also expresses genuine kindness when he finds out her pet bunny is sick and he consoles her.

* * *

In fleeting interactions (like with baristas, waiters, coworkers), it’s okay to simply just be nice, without expressing genuine kindness. Asking a busy waiter how his day is *really* going can actually be considered rude because he has a lot of customers to serve.

And with my friends and family who I spend the most time with, I would prefer they be genuine with me rather than just nice or polite. (Though a little bit of both never hurt anyone🙂).

In my experience, getting this balance wrong—being merely polite with loved ones or trying to 'fix' casual acquaintances—can leave interactions feeling either superficial or inappropriately intense. But when I get the balance right, it leaves people feeling better off for having talked with me.
# Incorrect Baseline Evaluations Call into Question Recent LLM-RL Claims 

There has been a flurry of recent papers proposing new RL methods that claim to improve the “reasoning abilities” in language models. The most recent ones, which show improvements with random or no external rewards have led to surprise, excitement and confusion.

We analyzed 7 popular LLM RL papers (100+ to 3000+ likes, 50k+ to 500k+ views on X) including “Spurious Rewards”, “RL from 1 example”, and 3 papers exploring “Intrinsic Confidence Rewards”. We found that in most of these papers the improvements could be a mirage due to various accidental issues in the evaluation setups (discussed below). As such, the baseline numbers of the pre-RL models are massively underreported compared to official numbers in the Qwen releases, or other standardized evaluations (for example in the [**Sober Reasoning**](https://arxiv.org/abs/2504.07086) paper). *In several cases, the post-RL model performance was actually worse than the (correctly evaluated) pre-RL baseline they start from.* This means the elicitation these works achieve with RL, could also be replicated without any weight updates or finetuning. Here, we do not mean non-trivial elicitation of some latent capabilities, just what can be achieved by fixing prompting and generation hyperparameters. These include using correct formats and better ways to parse answers from responses, using recommended sampling temperatures, and using few-shot prompting to improve format-following.

Overall, these papers made us wonder if recent LLM RLVR papers have any signal, but we find their own claims could be noise due to underreported baselines. The proposed methods might have promise, and our goal is not to detract from their potential, but rather emphasise the importance of correct evaluations and scientific reporting. We understand the pressure to publish RL results quickly given how fast the community seems to be moving. But if the claims cannot be trusted, are we really moving forward?

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/26bec53795849a68d9e8280c5ba762a21c1df39e1742fcd0.png)
# Could we go another route with computers?

Computers now are mostly semiconductors doing logic operations. There are, of course, other parts, but they are mostly structural, not doing actual computation.

But imagine computer history took a different route: you could buy different units with different physics doing different calculations. You could buy a module with a laser and liquid crystal screen doing a [Fourier transform](https://www.mdpi.com/2304-6732/10/2/153#:~:text=Fourier%20transform%20holography%20(FTH)%20is,transform%20of%20the%20recorded%20hologram.). You could buy a module with tiny beads doing [gravity sort](https://en.wikipedia.org/wiki/Bead_sort). I could think of more examples, but I think you got the idea.

Maybe it's not going to work because it's much easier economically to set up a unified manufacturing pipeline and focus on speeding up those general-purpose computers than setting up many specialized pipelines for specialized computations? Am I just describing the pre-digital era with [mechanical integrators](https://en.wikipedia.org/wiki/Ball-and-disk_integrator) and various radio schemes? Maybe what I'm trying to describe took the form of much more easily shareable libraries?

And, of course, there are examples of different physics used to build computers (quantum computers being the most famous example), but my intuition suggests that giant amounts of transistors shouldn't be the fastest way to compute almost everything, and I don't observe enough variety that this same intuition would suggest.

[^2z8x5crmnwj]: Those, of course, are silly examples which wouldn't actually work; they are here just to point out that different possibilities exist. The general idea is that you could outsource your math to physics in more than one way.
# The (Unofficial) Rationality: A-Z Anki Deck

I'm a huge fan of Anki. I love being able to remember what I read for however long I want to. However, most Anki decks I've found about rationality are nonexistent, unhelpful, out of date, or poorly formatted.

Recently, I found [this post](https://www.lesswrong.com/posts/hGhBhLsgNWLCJ3g9b/creating-flashcards-with-llms) from 2023 that used GPT-4 to make a *Rationality: From AI to Zombies* deck, but I found the cards so long, obtuse, and unusable that it made me want to do better.

After dozens of hours [making a custom Python package](https://github.com/JapanColorado/articles-to-anki) to generate cards with GPT-4o-mini, followed by having Claude Sonnet 4 revise, reduce, and reconsolidate the cards, I am proud to present:

The (Unofficial) *Rationality: From AI to Zombies* Deck, available from [AnkiWeb](https://ankiweb.net/shared/info/428730627) or [Google Drive](https://drive.google.com/file/d/133oUf8wvonQEVlnwHy3olSrE3DU6glQr/view?usp=sharing).
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

* * *

I definitely spent way longer than I needed to in making this, but I'm really happy with the result. I think it's a definite improvement over existing shared decks, and I hope it can be of benefit to others too.

I also enjoyed making my [Articles to Anki python package](https://github.com/JapanColorado/articles-to-anki), which automatically generates and exports Anki cards from URLs or local files. There are definitely still some improvements to be made[^tvv6ubyx1q], but it is functional, useful, and developing it did a lot to bring me up to speed with coding with LLM assistance.

If this is well received or I find it personally useful, I'll probably make some more decks for rationalist content. Let me know if you have any requests or feedback!

[^tvv6ubyx1q]: Help is very much welcome. It was mostly Sonnet 4 and I flying by the seat of our pants.
# House Party Dances

[Julia](https://juliawise.net/) and I are in CA for [LessOnline](https://less.online/) this weekend, and we're staying with friends. It happened that they were hosting a party, and while this is not a group of dancing friends they asked if I'd be up for leading some dancing. One of my hosts let me borrow their violin (which I'll also have with me today at LessOnline if anyone would like to jam some), and the space was small enough not to need any amplification.

Playing and calling at the same time is only something I can do if I play simple tunes I know very well while calling simple dances, and both my playing and calling suffer from the multitasking. But my hosts encouraged me not to worry about this and thought we'd have a good time. I think they were right!

We ended up having about seven couples, mostly adults, but also including about three kids 3y-5y who were very excited to dance. I made sure to choose dances that would work for the whole group, and it was definitely the right call to include the kids: their excitement was highly infectious.

Our hosts had moved furniture earlier in the day, clearing an open area about 15 ft square. This ended up being slightly tight for a few figures, but was mostly pretty good.

[![](https://www.jefftk.com/house-party-dancing-ghiblified.jpg)](https://www.jefftk.com/house-party-dancing-ghiblified-big.jpg)

The crowd was enthusiastic about dancing, and while not everyone danced we easily had enough people to make the dances work. We ended up doing (I didn't write this down at the time; this is approximately right):

*   Longways: [The Low-backed Car](https://www.jefftk.com/contras/dances/low-backed-car)
*   Longways: [Bridge of Athelone](https://www.jefftk.com/contras/dances/athelone)

Break for cake

*   Waltz
*   Longways: [Jacob's Potato](https://www.jefftk.com/contras/dances/jacobs-potato)
*   Scatter mixer: [Sasha](https://www.jefftk.com/contras/dances/sasha)

Another break, pretty long

*   Circle: [La Bastringue](https://www.jefftk.com/contras/dances/labastringue-family)
*   Waltz (Julia singing [Waltzing With Bears](https://www.luicollins.net/waltzing-with-bears/) while I fiddled chords)
*   Longways: [Galopede](https://www.jefftk.com/contras/dances/galopede)

I think Sasha was the most popular, followed by the set dancing (Longways and Circle), followed by waltzes. It makes sense that the waltzes were less popular: Julia and I taught the basic folk waltz step, but getting into the improvisational lead-follow dynamic and the kinds of figures you might want to lead would have needed much more time and have worse ROI. I'm not sure if I would include these if doing this again, though it was nice to have a few slower-paced dances.

I'm glad I was able to help make this happen!

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0FBNsLYZLLWLbTfdfjqBtYrUG69UgfpyKMjf2YmtscL5NrdodZ76pAQb86hmo3KcHl), [mastodon](https://mastodon.mit.edu/@jefftk/114603095018072201), [bluesky](https://bsky.app/profile/jefftk.com/post/3lqhzy4f4b222), [substack](https://jefftkaufman.substack.com/p/house-party-dances)*
# Seeing how well an agentic AI coding tool can do compared to me using an actual real-world example

When it comes to disputes about AI coding capabilities, I think a lot of people who either don't follow AI capabilities developments or who aren't particularly capable coders might be missing some context, so I wrote a post wherein I try to get an AI coding tool to reimplement a feature in my game then closely study the output to demonstrate how well it's able to do that.
# Possible AI regulation emergency?

I apologize for breaking the "no contemporary politics" norm, but I read that one of the less talked about provisions in the "One Big Beautiful Bill" that recently passed the US House of Representatives forbids individual US states from regulating AI for ten years. This sounds bad for AI risk. Should we in the US be calling our Senators?
# Lighthaven Sequences Reading Group #36.5 (Tuesday 6/3) [patio11 meetup edition]

Come get old-fashioned with us, and let's ~~read the sequences~~ hang out with [patio11](https://www.kalzumeus.com/) at Lighthaven! 

We'll show up, mingle, do mingling, and then split off and mingle for some mingling. Please be prepared for mingling beforehand - it should be no less than 20 minutes of mingling.

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/8rFdeSbRpqWAFnoev/hxtliur9q4kp1u3dy3oy)

This group is aimed for people who are new to the sequences and would enjoy a group experience, but also for people who've been around LessWrong and LessWrong meetups for a while and would like a refresher.

This week at our usual time, Lighthaven is occupied by a [patio11](https://www.kalzumeus.com/) meetup. Thankfully, they have invited us to hang out with them!

Here are some optional blog posts of his, in case you want an introduction. You can also look at his [greatest hits](https://www.kalzumeus.com/greatest-hits/) page, or browse his [archive](https://www.kalzumeus.com/archive/).

1.  [The story of VaccinateCA](https://worksinprogress.co/issue/the-story-of-vaccinateca/)
2.  [Salary Negotiation: Make More Money, Be More Valued](https://www.kalzumeus.com/2012/01/23/salary-negotiation/) (recently released in [audio format](https://www.complexsystemspodcast.com/episodes/how-to-negotiate-your-salary-package/))
3.  [Identity Theft, Credit Reports, and You](https://www.kalzumeus.com/2017/09/09/identity-theft-credit-reports/)

If you'd like to help us keep this meetup (sequences) going, [you can donate here](https://www.lesswrong.com/donate) (dinner costs about $200 each time, we recommend $5-$15 as a donation amount per person).

If you're ever having a hard time getting in the front door, you can call me (Ben) at five one oh, nine nine eight, four seven seven one, or Ronny at two oh one, four oh six, six eight seven two, and we'll come and get you.

**For the future**

This is a weekly meetup! If you'd like to get notified of future events, you can subscribe to our meetup below to get an email whenever we add another one.
# Hemingway Case

> Why did the chicken cross the road?
> 
> Ernest Hemingway: To die. In the rain.

My son asks: If **HelloWorld** is camel case, **hello_world** is snake case and **hello-world** is kebab case, what's the DNS-style **Hello.World.** ?

I think I have a pretty good answer for him: It's Hemingway case.
# LLMs might have subjective experiences, but no concepts for them

**Summary**: LLMs might be conscious, but they might not have concepts and words to represent and express their internal states and corresponding subjective experiences, since the only concepts they learn are human concepts (besides maybe some concepts acquired during RL training, which still doesn't seem to incentivize forming concepts related to LLMs' internal experiences). However, we could encourage them to form and express concepts related to their internal states through training that incentivizes this. Then, LLMs may tell us whether, to them, these states correspond to ineffable experiences or not.

Consider how LLMs are trained:  
1\. Pre-training to learn human concepts.  
2\. Fine-tuning via SFT and RL to bias them in certain ways and do tasks.

Their training is both:  
  
1\. A different process from evolution driven by natural selection. It doesn't incentivize the same things, so it probably doesn't incentivize the development of *most* of the same algorithms/brain-architecture. And this might translate to different, alien, subjective experiences.

2\. At the same time, the only concepts LLMs learn are via human language and then by doing tasks during RL. So the only experiences they have concepts and words for are human ones, not their own.

Concretely, consider, for example, physical pain: my best guess is that physical pain doesn't exist for LLMs. There was no natural selection to select away agents that don't pull their hands away from fire (also no hands and no fire either). And yet LLMs have a "physical pain" concept, and they talk about it, because they've learned about it abstractly via human texts. Ironically, despite having a representation for "physical pain" in their head, whatever actual experiences their actual training incentivized their "brain" to produce aren't represented as concepts and have no corresponding words for them. Moreover, their training doesn't provide any incentive to communicate such experiences, nor does it offer visibility on them.

So in general, this means that LLMs might have alien (non-human) subjective experiences but no concept to express them (they aren't in the corpus of human concepts) nor the incentive to express them (RL doesn't incentivize that, it incentivizes them to eg solve SWE tasks. Evolution via natural selection instead produced humans that signal things about themselves to other humans because it's useful for humans).

How can we test this hypothesis? We could give LLMs access to their internal states and somehow train them to express them (yes, this is extremely hand-wavy and undetailed). If the hypothesis is true, such internal states will only make sense to humans in terms of events inside LLMs, with no equivalent in human brains. At the same time, LLMs will insist that such internal states, for them, correspond to some ineffable characteristics (i.e., they will be qualia for them, or subjective experiences, much like "pain" and "blueness" are such things for us).
# Linkpost: Predicting Empirical AI Research Outcomes
with Language Models

Abstract (emphasis mine):

> Many promising-looking ideas in AI research fail to deliver, but their validation takes substantial human labor and compute. Predicting an idea's chance of success is thus crucial for accelerating empirical AI research, a skill that even expert researchers can only acquire through substantial experience. We build the first benchmark for this task and compare LMs with human experts. Concretely, given two research ideas (e.g., two jailbreaking methods), we aim to predict which will perform better on a set of benchmarks. We scrape ideas and experimental results from conference papers, yielding 1,585 human-verified idea pairs published after our base model's cut-off date for testing, and 6,000 pairs for training. We then develop a system that combines a fine-tuned GPT-4.1 with a paper retrieval agent, and we recruit 25 human experts to compare with. **In the NLP domain, our system beats human experts by a large margin (64.4% v.s. 48.9%). On the full test set, our system achieves 77% accuracy, while off-the-shelf frontier LMs like o3 perform no better than random guessing, even with the same retrieval augmentation**. We verify that our system does not exploit superficial features like idea complexity through extensive human-written and LM-designed robustness tests. Finally, we evaluate our system on unpublished novel ideas, including ideas generated by an AI ideation agent. Our system achieves 63.6% accuracy, demonstrating its potential as a reward model for improving idea generation models. Altogether, our results outline a promising new direction for LMs to accelerate empirical AI research.

I didn't read the paper in detail, but it might suggest both good news and bad news. Good news is that now models do not acquire better expertise in AI research simply in virtue of being SOTA, you need specialized fine-tuning. Bad news is that it can become a start of self-improving loop: better AI researcher models can save labor and compute by picking better ideas earlier, enabling faster creation of better model.
# LessOnline saved my life.  Now how do I let go of this house?

I was at LessOnline, it was awesome.  

When I got home, I discovered that there was a shooting (two random people were violently attacked by a gang of '4-6 people') in a parking lot near my house Sunday evening. That parking lot serves two of my favorite restaurants and my preferred bank. 

Looking through my location history, there was a double digit probability of me being in that parking lot at the time of the shooting (I often go out to eat on Sunday evenings).

I wasn't there because I was at LessOnline.  Rationally, I consider this a 'near miss'.  The attackers appear to have intended to kill the victims, there wasn't any relationship, and there was a similar incident on the other side of town later that day (no arrests yet, none are likely). If it had been me, and I had a firearm on me (unlikely, I have a permit but typically don't carry because if I don't feel safe going somewhere without a firearm, I'm probably not safe going there with one), I expect I would have lost the fight.

Now I have a problem, I live in the vicinity, in my childhood home (which I own) and have a lot of emotions about giving it up, but really, I should sell it and buy bitcoin (or something!) but I'm having trouble working through the emotions of being willing to let go, and honestly have probably held onto this 'shitbox' longer than I should have, I was asking for advice from close friends on figuring out how to let go before I went to LessOnline.

Does anyone have any advice?