Harley-ml/lesswrong huggingface dataset:
# To an LLM, everything looks like a logic puzzle

I keep seeing this meme doing the rounds where people present ChatGPT with a common logic problem or riddle, only with some key component changed to make it trivial. ChatGPT has seen the original version a million times, so it gives the answer to the original, not the actually correct and obvious answer.

![Image](https://pbs.twimg.com/media/GNmT52UbwAEJUyG?format=jpg&name=large)

![Screenshot of ChatGPT 4o dialog


You:

The emphatically male surgeon who is also the boy’s father says, “I can’t operate on this boy! He’s my son!” How is this possible?


ChatGPT:

The surgeon is the boy's mother.](https://pbs.twimg.com/media/GNqZt8TXYAA4qZf?format=jpg&name=900x900)

The idea is to show that ChatGPT isn't intelligent, it's just reciting what it's seen before in similar contexts to the one it's seeing now and there's no actual reasoning taking place. My issue with this is it's pretty clear to me that most humans fail in very similar ways, just at a slightly higher level of complexity.

The clearest way this manifests is in the adage 'to a man with a hammer, everything looks like a nail'. When presented with a new problem that bears a strong resemblance to problems we've solved before, it's a very natural response to employ the same method we used previously. 

At the age of 16, I had very little understanding of statistics or machine learning but I knew about linear regression. I LOVED linear regression. If you came to me with some dataset and asked me what methodology I would use to analyze it, make predictions, anything like that, I would probably say linear regression. The dataset could be wildly, embarrassingly unsuitable for linear regression, but I would still probably suggest it as my first port of call. I certainly hope this doesn't mean that 16-year-old me was incapable of human-level reasoning.

Something I see a lot is people with monocausal narratives for history. They probably read a book about how a number of events were all caused by X. This book was probably correct! But they took it way too far and the next time someone asked them what caused some totally unrelated event, their go-to response might be X or some variation on X. In a lot of cases, this answer might be totally nonsensical, just like ChatGPT's responses above. Nevertheless this person is still able to reason intelligently, they're just performing a bunch of shortcuts in their head when deciding how to answer questions; most of the time these shortcuts work and sometimes they don't. 

If someone made me do ten logic puzzles a day for a year, all with pretty similar formats and mechanisms and then one day they switched out a real logic puzzle for a fake trivial one, I'd probably get it wrong too. Long story short, I think the prompts above are fun and could be taken as evidence that ChatGPT is still well below human-level general intelligence, but I don't think it sheds any light on whether it's capable of intelligent reasoning in general.
# Scientific Notation Options

When working with numbers that span many orders of magnitude it's very helpful to use some form of [scientific notation](https://en.wikipedia.org/wiki/Scientific_notation). At its core, scientific notation expresses a number by breaking it down into a decimal ≥1 and <10 (the "significand" or "mantissa") and an integer representing the order of magnitude (the "exponent"). Traditionally this is written as:

> `3` × 10^`4`^

While this communicates the necessary information, it has two main downsides:

*   It uses three constant characters ("× 10") to separate the significand and exponent.
    
*   It uses superscript, which doesn't work with some typesetting systems and adds awkwardly large line spacing at the best of times. And is generally lost on cut-and-paste.
    

Instead, I'm a big fan of [e-notation](https://en.wikipedia.org/wiki/Scientific_notation#E_notation), commonly used in programming and on calculators. This looks like:

> `3e4`

This works everywhere, doesn't mess up your line spacing, and requires half as many characters as writing it the traditional way.

There are a bunch of other variants of e-notation, but I don't like any of them as much:

*   `3E4`: a shorter separator would be easier to read.
*   `3e+4`: the `+` is redundant.
*   `3⏨4`: neat, but requires unicode and can't be pasted into widely-used programming languages and spreadsheet programs.

One downside of "e" notation is that it comes off as less formal than traditional scientific notation. But unless you need to be read as maximally formal I think it's just better all around.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0Le6L7gtY7pGf8JwbXZkS9EacU2a2KK2CBDe5GhTUMXcfjwwgjYLQCZZsMsmqwdUsl), [mastodon](https://mastodon.mit.edu/@jefftk/112462688147618956)*
# On Privilege

The forum has been very much focused on AI safety for some time now, thought I'd post something different for a change. Privilege.

Here I define Privilege as an **advantage over others that is invisible to the** ~~**be**~~**holder**. \[EDIT: thanks to JenniferRM for pointing out that "beholder" is a wrong word.\] This may not be the only definition, or the central definition, or not how you see it, but that's the definition I use for the purposes of this post. I also do not mean it in the culture-war sense as a way to undercut others as in "check your privilege". My point is that we all have some privileges \[we are not aware of\], and also that nearly each one has a flip side. 

In some way this is the inverse of [The Lens That Does Not See Its Flaws](https://www.lesswrong.com/posts/46qnWRSR7L2eyNbMA/the-lens-that-sees-its-flaws): The lens that does not see its strengths. 

On to the examples, as non-polarizing as I could think of, so no focus on race or gender:

*   Intelligence privilege. This one ought to be familiar to the regulars, but is often a blind spot. One manifestation of it being a privilege (i.e. being unaware that you have something others do not) is saying something like "anyone can learn calculus". No, not anyone.
*   Health privilege. Most healthy people don't give a second thought to how lucky they are compared to those who struggle physically or mentally to just get through the day.
*   Conventional beauty privilege. Everything is just invisibly easier when you look good. The flip side, of course, is that you have to deal with harassment a lot more. Or the judgment "you got where you are because of your looks".
*   A reasonably happy childhood privilege. Those who grew up in "normal" families rarely appreciate what it is like to be a child with a high [ACE score](https://www.npr.org/sections/health-shots/2015/03/02/387007941/take-the-ace-quiz-and-learn-what-it-does-and-doesnt-mean). The flip side is the determination to get ahead that some high-adversity survivors develop.
*   Having an instinctive social "game". Not just for dating, but in general social interactions where a person is naturally likable and relatable, at least on the surface level. If you have that, you probably can't understand why others do not. Saying "I just go and talk to them, why don't you do the same, just be yourself" is an indicator of having this privilege.  
*   Wealth privilege. Not, like, billionaire-level, but middle- and upper middle class vs, say, working poor. 
*   A host of others, like education, location, culture, background...

One lesson to take from this is learning to notice this blind spot, where you naturally have something others do not, and it feels like nothing, like clean air without any wind. A possible suggestion as to how to go about it might be to focus on the feeling of being unable to relate to a person or to a group. Something like "I cannot imagine why/how anyone would/would not \[be able to\] do/feel/think X" can be an indicator of having something others do not.
# D&D.Sci (Easy Mode): On The Construction Of Impossible Structures [Evaluation and Ruleset]

This is a followup to [the D&D.Sci post](https://www.lesswrong.com/posts/Syfq6MwgdZhHg9vha/d-and-d-sci-easy-mode-on-the-construction-of-impossible) I made last Friday; if you haven’t already read it, you should do so now before spoiling yourself.

Below is an explanation of the rules used to generate the dataset (my full generation code is available [here](https://github.com/H-B-P/d-and-d-sci-impossible-structures/blob/main/gen_train.py), in case you’re curious about details I omitted), and their strategic implications.

* * *

Ruleset
=======

### Impossibility

Impossibility is entirely decided by who a given architect apprenticed under. Fictional impossiblists Stamatin and Johnson invariably produce impossibility-producing architects; real-world impossiblists Penrose, Escher and Geisel always produce architects whose works just kind of look weird; the self-taught break Nature's laws 43% of the time.

### Cost

Cost is entirely decided by materials. In particular, every structure created using Nightmares is more expensive than every structure without them.

Strategy
--------

The five architects who would guarantee an impossible structure are D, E, G, H, and K. Of these, G - and only G - intends to use Nightmares as construction material. The optimal choices given the Duke's stated preferences are therefore \[D, E, H, K\].

Reflections
===========

This challenge was created with the intent of being egregiously easy and anomalously accessible. From the performances I saw, it looks like it fit the bill: congratulations to everyone who played publicly on reaching my intended solution. (Particular congratulations to aphyer for providing said solution within an hour of me posting the challenge, and to new player Lorxus for managing to correctly identify every named impossiblist on their way to a perfect answer.)

I hope this scenario managed to be fun despite - or because of? - the simplicity and lack of greater point. If you liked it, in lieu of correctly attributing thanks, please be slightly kinder to everyone you meet: after all, you have no way of being certain any given person *wasn't* my sponsor. (Conversely, if you *dis*liked it, please wander the streets of your nearest metropolis glowering at random strangers until you feel karmic balance has been restored.)

(. . . also, please share your opinions with me. I find myself *very* curious about how/if players think I should aim to run smaller/lower-difficulty games going forward.)

Scheduling
==========

Unless someone gives me a reason not to, I (tentatively, falliably) plan to run my next scenario from Friday 7th June to Monday 17th June. We'll see if reality plays along.
# Jaan Tallinn's 2023 Philanthropy Overview

to follow up my [philantropic pledge](https://www.lesswrong.com/posts/ZMRCpfGD4iXs5Mi5Q/jaan-tallinn-s-philanthropic-pledge) from 2020, i've updated my [philanthropy page](https://jaan.info/philanthropy/) with 2023 results.

in 2023 my donations funded [$44M worth of endpoint grants](https://jaan.info/philanthropy/donations.html#2023) ($43.2M excluding software development and admin costs) — exceeding my commitment of $23.8M (20k times $1190.03 — the minimum price of ETH in 2023).
# [Linkpost] Statement from Scarlett Johansson on OpenAI's use of the "Sky" voice, that was shockingly similar to her own voice.

Scarlett Johansson makes a statement about the "Sky" voice, a voice for GPT-4o that OpenAI recently pulled after less than a week of prime time.  
  
tl;dr: OpenAI made an offer last September to Johansson; she refused. They offered again 2 days before the public demo. Scarlett Johansson claims that the voice was so similar that even friend and family noticed. She hired legal counsel to ask OpenAI to "detail the exact process by which they created the ‘Sky’ voice," which resulted in OpenAI taking the voice down.  
  
Full statement below:

*Last September, I received an offer from Sam Altman, who wanted to hire me to voice the current ChatGPT 4.0 system. He told me that he felt that by my voicing the system, I could bridge the gap between tech companies and creatives and help consumers to feel comfortable with the seismic shift concerning humans and Al. He said he felt that my voice would be comforting to people.*

*After much consideration and for personal reasons, declined the offer.*

*Nine months later, my friends, family and the general public all noted how much the newest system named ‘Sky’ sounded like me.*

*When I heard the released demo, I was shocked, angered and in disbelief that Mr. Altman would pursue a voice that sounded so eerily similar to mine that my closest friends and news outlets could not tell the difference. Mr. Altman even insinuated that the similarity was intentional, tweeting a single word ‘her’ — a reference to the film in which I voiced a chat system, Samantha, who forms an intimate relationship with a human.*

*Two days before the ChatGPT 4.0 demo was released, Mr. Altman contacted my agent, asking me to reconsider. Before we could connect, the system was out there.*

*As a result of their actions, I was forced to hire legal counsel, who wrote two letters to Mr. Altman and OpenAl, setting out what they had done and asking them to detail the exact process by which they created the ‘Sky’ voice. Consequently, OpenAl reluctantly agreed to take down the ‘Sky’ voice.*

*In a time when we are all grappling with deepfakes and the protection of our own likeness, our own work, our own identities, I believe these are questions that deserve absolute clarity. I look forward to resolution in the form of transparency and the passage of appropriate legislation to help ensure that individual rights are protected.*
# Harmony Intelligence is Hiring!

Hey folks! Pleased to announce we have a new open position for a Founding Research Engineer at Harmony Intelligence. You’ll be responsible for measuring and identifying dangerous AI capabilities across various domains: cybersecurity, biosecurity, persuasion and manipulation, self-exfiltration and self-replication, and more.

**Location:**  
We're a small but quickly growing remote-first startup, with folks in San Francisco and Sydney.

**About Harmony:**  
We're specifically focused on reducing catastrophic AI risk by building evals and conducting red teaming/audits. Though we’re still young, we're already making a big impact! We’ve published our first [automated red teaming paper](https://x.com/soroushjp/status/1721950722626077067) to massively positive reception from the AI community, and are actively working on model evaluations and other AI safety research. For any further questions, please reach out to our co-founder and [CTO Alex Browne](https://www.linkedin.com/in/stephenalexbrowne/). To apply or see more information, go to [harmonyintelligence.com/jobs](https://harmonyintelligence.com/jobs).
# Is deleting capabilities still a relevant research question?

I've had it suggested that a good criterion for whether interpretability is on the right track is if we can do surgical "deletions" of model capabilities, e.g. removing its ability to build bombs and such.

Obviously in one sense this is fairly trivial since you can just use simple gradient descent to make the models refuse, but the issue with this is that given the weights, people can easily undo these refusals (and also adversarial prompting can often bypass it).

I know there's been some back and forth on methods for full deletion, and I'm wondering if it's considered a solved problem or not.
# ARIA's Safeguarded AI grant program is accepting applications for Technical Area 1.1 until May 28th

**Note**: I am completely unaffiliated with [ARIA](https://www.lesswrong.com/posts/qweXJ6v9heSn4wvdk/davidad-s-provably-safe-ai-architecture-aria-s-programme). I figured I'd post this since applications are closing soon and I didn't see anyone post about this.

**My Takeaways**:

*   ARIA is funding the development of [Safeguarded AI](https://www.lesswrong.com/posts/qweXJ6v9heSn4wvdk/davidad-s-provably-safe-ai-architecture-aria-s-programme) which is an update to and specific implementation of davidad's [Open Agency Architecture](https://www.lesswrong.com/posts/pKSmEkSQJsCSTK6nH/an-open-agency-architecture-for-safe-transformative-ai).
*   This grant round is for Technical Area (TA) 1.1, the first phase of developing Safeguarded AI.
    *   TA1.1 involves developing mathematical and CS "syntax" for "language(s) that can be used by teams of humans (and, later, AI systems) to define 'world models', probabilistic specifications, neural network controllers, and proof certificates" for the ultimate purpose of "defin\[ing\] safety" to AI systems to ensure that they "are not taking actions that would lead to catastrophic outcomes."
*   Grant details:
    *   This round is expected to distribute up to £3.5M across 10–16 applicants. The round opened on April 11th, 2024 and will close on May 28th, 2024 (at "12:00 BST").
    *   Funding is primarily intended for UK entities (e.g. a person, company, nonprofit, or university), but non-UK entities can qualify if they can boost the impact of programs in the UK.
    *   This grant round is focused on supporting open source conceptual/research work, but will also fund people to communicate concepts in TA1.1 to broader audiences.
        *   The proposal mentions funding learning materials like blog posts, tutorials, and educational videos. This might be an interesting opportunity for people interested in Safeguarded AI to receive funding to learn about and share davidad's ideas.
*   The next grant round, for TA1.2, is expected to open later this year.
# Announcing Human-aligned AI Summer School

*The fourth Human-aligned AI Summer School will be held in Prague from 17^th^ to 20^th^ July 2024.* We will meet for four intensive days of talks, workshops, and discussions covering latest trends in AI alignment research and broader framings of AI alignment research. [***Apply now***](https://form.typeform.com/to/zUKjsxgx)*, applications are evaluated on a rolling basis.*  
  
The intended audience of the school are people interested in learning more about the AI alignment topics, PhD students, researchers working in ML/AI outside academia, and talented students. 

Format of the school
--------------------

The school is focused on teaching and exploring approaches and frameworks, less on presentation of the latest research results. The content of the school is mostly technical – it is assumed the attendees understand current ML approaches and some of the underlying theoretical frameworks.

This year, the school will cover these main topics:

*   Overview of the alignment problem and current approaches.
*   Alignment of large language models: RLHF, DPO and beyond. Methods used to align current large language models and their shortcomings.
*   Evaluating and measuring AI systems: How to understand and oversee current AI systems on the behavioral level.
*   Interpretability and the science of deep learning: What's going on inside of the models?
*   AI alignment theory: While 'prosaic' approaches to alignment focus on current systems, theory aims for deeper understanding and better generalizability.
*   Alignment in the context of complex systems and multi-agent settings: What should the AI be aligned to? In most realistic settings, we can expect there are multiple stakeholders and many interacting AI systems; any solutions to alignment problem need to solve multi-agent settings.

The school consists of lectures and topical series, focused smaller-group workshops and discussions, expert panels, and opportunities for networking, project brainstorming and informal discussions.  
Detailed program of the school will be announced shortly before the event. See below for a program outline and e.g. [the program of the previous school](https://humanaligned.ai/index-2022.html#program) for an illustration of the program content and structure.

Confirmed speakers
------------------

*   [**Stephen Casper**](https://www.lesswrong.com/users/scasper) – Algorithmic Alignment Group, MIT.
*   [**Stanislav Fort**](https://x.com/stanislavfort?lang=en) – Google DeepMind.
*   [**Jesse Hoogland**](https://www.lesswrong.com/users/jesse-hoogland) – Timaeus.
*   [**Jan Kulveit**](https://www.lesswrong.com/users/jan-kulveit) – Alignment of Complex Systems, Charles University.
*   [**Mary Phuong**](https://www.lesswrong.com/users/mary-phuong-2) – Google DeepMind.
*   [**Deger Turan**](https://www.lesswrong.com/users/deger-turan-1) – AI Objectives Institute and Metaculus.
*   [**Vikrant Varma**](https://www.lesswrong.com/users/amrav) – Google DeepMind.
*   [**Neel Nanda**](https://www.lesswrong.com/users/neel-nanda-1)  \- Google DeepMind.  
    (more to be announced later)
# Should we be concerned about eating too much soy?

Parts of the internet say that, especially for men, eating too much soy is unhealthy while other parts of the internet advocate that soy is really great way for vegans to consume their protein.

Has anyone made a deep dive into the evidence base and formed an opinion about whether or not to worry?
# Offering service as a sensayer for simulationist-adjacent beliefs.

A Sensayer is[^zlf37uy98up] a specialist in the private discussion of religion[^cfa8hzcympv] in small groups, or one on one, a bit like a councilor with theological acumen. Many of us, believe it or not, harbor private beliefs. Sometimes those beliefs get heavy, and they can often benefit a *lot* from a second pair of eyes.

Most religions, in our world, do not need sensayers, as they can meet in groups and discuss their beliefs openly and there are usually healthy inter-faith dialogs, simulationism is different. It's rougher. It has no churches. It's not immediately obvious that it instructs any actions, given that it claims non-intervention, so its adherents are naturalists, and most think of themselves as atheists. It has no relationship with other faiths and when it confesses publicly it only receives chides from within and without ("clearly you're just trying to fill a void.") Some simulationist concepts, I've found, are easily misused, cynical, troubling or depersonalizing, the believer would not wish them upon anyone else. Nowhere in simulationist thought is there an obligation to proselytize (in some cases it demands the opposite), so I don't expect these things to change. So, a simulationist sometimes needs a sensayer. Myself included.

I've kind of had this role before. I approached a troubled friend and came to understand their ideas. Many of the ideas we found were beautiful, and a bit healing. There was never a reason to publish any of it and I mostly haven't. So it goes. I continue to stare into the abyss. And I often need someone to talk to, about that. At least twice I've been visited with thoughts I couldn't carry alone and also couldn't share with anyone around me, I was extremely grateful to have distant philosophically inclined intimates who I could summon into my cave and ask them to help untangling the thoughts.  
Not everyone has that.

So I'm offering an open ear. You can DM me about this kind of thing. I can hold it all. I've seen the worst of it. And it stops with me, no exceptions.  
I'll alter this paragraph if I ever feel that I can't do this any more. So if you've found it, I'm still here.

[^zlf37uy98up]: The term comes from Terra Ignota, a science fiction series of four books, set in a mostly okay future where, among other things, religion has become taboo. The author is a historian, and a lot of it is about geopolitical posturing, but she is also a historian who watches anime.I kind of believe it was written for us. The first book, Too Like The Lightening, is mostly just setup, but every book beyond that one is consistently very juicy. 

[^cfa8hzcympv]: Religion here meaning "non-ubiquitous, consequential metaphysical [prior to physics] beliefs". I usually prefer to define religion as "an idea that supports a moral order", but that sense of religion generally doesn't require sensayers.
# Paper in Science: Managing extreme AI risks amid rapid progress

[https://www.science.org/doi/10.1126/science.adn0117](https://www.science.org/doi/10.1126/science.adn0117)

**Authors:**

Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, Gillian Hadfield, Jeff Clune, Tegan Maharaj, Frank Hutter, Atılım Güneş Baydin, Sheila McIlraith, Qiqi Gao, Ashwin Acharya, David Krueger, Anca Dragan, Philip Torr, Stuart Russell, Daniel Kahneman, Jan Brauner*, Sören Mindermann*

**Abstract:**

Artificial intelligence (AI) is progressing rapidly, and companies are shifting their focus to developing generalist AI systems that can autonomously act and pursue goals. Increases in capabilities and autonomy may soon massively amplify AI’s impact, with risks that include large-scale social harms, malicious uses, and an irreversible loss of human control over autonomous AI systems. Although researchers have warned of extreme risks from AI, there is a lack of consensus about how to manage them. Society’s response, despite promising first steps, is incommensurate with the possibility of rapid, transformative progress that is expected by many experts. AI safety research is lagging. Present governance initiatives lack the mechanisms and institutions to prevent misuse and recklessness and barely address autonomous systems. Drawing on lessons learned from other safety-critical technologies, we outline a comprehensive plan that combines technical research and development with proactive, adaptive governance mechanisms for a more commensurate preparation.
# The case for stopping AI safety research

TLDR: AI systems are failing in obvious and manageable ways for now. Fixing them will push the failure modes beyond our ability to understand and anticipate, let alone fix. The AI safety community is also doing a huge economic service to developers. Our belief that our minds can "fix" a super-intelligence - especially bit by bit - needs to be re-thought. 

I wanted to write this post forever, but now seems like a good time.  The case is simple, I hope it takes you 1min to read.

1.  **AI safety research is still solving easy problems.  **We are patching up the most obvious (to us) problems. As time goes we will no longer be able to play this existential risk game of chess with AI systems. I've argued this a lot ([ICML 2024 spotlight paper;](https://openreview.net/forum?id=rfvgdfd1K9) also [www.agencyfoundations.ai](https://www.agencyfoundations.ai)). Seems others have this thought.
2.  **Capability development is getting AI safety research for free.** It's likely in the millions to tens of millions of dollars. All the "hackathons", and "mini" prizes to patch something up or propose a new way for society to digest/adjust to some new normal (and increasingly incentivizing existing academic labs). 
3.  **AI safety research is speeding up capabilities**. I hope this is somewhat obvious to most.

I write this now because in my view we are about 5-7 years before massive human biometric and neural datasets will enter our AI training.  These will likely generate amazing breakthroughs in long-term planning and emotional and social understanding of the human world.  They will also most likely increase x-risk radically.

Stopping AI safety research or taking it in-house with security guarantees etc, will  slow down capabilities somewhat - and may expose capabilities developers more directly to public opinion of still manageable harmful outcomes.
# Secret US natsec project with intel revealed

Unclear how relevant this news is to AI safety, but it seems like the sort of thing we ought to notice.

> A backroom Washington deal brokered two years ago is undercutting a key part of President Joe Biden’s policy to grow the national high-tech manufacturing base — pushing more than $3 billion into a secretive national-security project promoted by chipmaker Intel.
> 
> In recent weeks, Biden and Senate Majority Leader Chuck Schumer have been taking victory laps for the 2022 CHIPS and Science Act, a law intended to create jobs and fund innovation in a key global industry. It has already launched a series of grants, incentives and research proposals to help America regain its cutting-edge status in global semiconductor manufacturing.
> 
> But quietly, in a March spending bill, appropriators in Congress shifted $3.5 billion that the Commerce Department was hoping to use for those grants and pushed it into a separate Pentagon program called Secure Enclave, which is not mentioned in the original law.
> 
> The diversion of money from a flagship Biden initiative is a case study in how fragile Washington’s monumental spending programs can be in practice. Biden’s legacy is [bound up in the fate](https://www.politico.com/news/biden-federal-funds-spending-tracking) of more than $1 trillion in government spending and tax incentives aimed at transforming the economy — but even money appropriated for a strategic national goal can wind up being rerouted for narrow or opaque purposes.
> 
> \[See more at the link\]
# What should the norms around AI voices be?

In previous discussions of AI risks, the ability for an AI to be very persuasive is often seen as one possible risk. Humans find some voices more persuasive than other voices.

If we can trust Scarlett Johansson's description of her interactions with OpenAI, OpenAI wanted to use her voice, to increase how much users trust OpenAI's model. Trusting a model more, likely means that the model is more persuasive.

AI companies could also multivar-test slight variations of their voices to maximize user engagement which would also likely push the voices in the direction of being more persuasive.

Zvi recently [argued](https://www.lesswrong.com/posts/N8aRDYLuakmLezeJy/do-not-mess-with-scarlett-johansson#zzZXSsaFSpyDkNazB) that it's fine for OpenAI to provide their users with maximally compelling voices if the user want those voices, without getting pushback for it.

Are we as a community not worried anymore about the persuasive power of AI's? As being someone who is not working directly in AI safety myself, why does this aspect seem underexplored by AI safety researchers?
# Is CDT with precommitment enough?

Logical decision theory was introduced (in part) to resolve problems such as Parfit's hitchhiker.

I heard an argument that there is no reason to introduce a new decision theory - one can just take causal decision theory and precommit to doing whatever is needed on such problems (e.g. pay the money once in the city).

This seems dubious given that people spent so much time on developing logical decision theory. However, I cannot formulate a counterargument. What is wrong with the claim that CDT with precommitment is the "right" decision theory?
# Who does the artwork for LessWrong?

The art on the LessWrong site and books is amazing. The consistent theme across everything is beautiful!

hm... I just checked the physical book and it credits Midjourney. But I feel there must have been some serious prompt crafting/human artistry behind this? 

Is there some human that deserves credit for this?
# The AI Revolution in Biology

An expert in the field ([Amelie Schreiber](https://x.com/amelie_iska)) predicts the convergence of AI models for biology to the state where designing drugs, targeted genetic mutations (which then could be enacted through genetic therapy), or viruses for specific purposes becomes really easy in a few years. (Note: apparently, the podcast was recorded before the release of [AlphaFold 3](https://blog.google/technology/ai/google-deepmind-isomorphic-alphafold-3-ai-model/).)

Unlike AGI safety, where OpenAI's argument of "iterative deployment" merits more weight[^jaulmfd0f2], in biology I think we can already map the capabilities that will be unlocked. In fact, this podcast does it pretty well, it seems to me. So, it should also be possible to develop effective regulation and controls ahead of the creation of the bio-capabilities.

This raises the questions of the relative speeds of bio-capabilities, regulation, and controls development. And the ethics of pushing bio-capabilities *if* regulation and controls are lagging behind.

[^jaulmfd0f2]: Even though many people, including me still don't think "iterative development" is justified the way OpenAI does it.
# If you are also the worst at politics

*This has been said many times but it must be said many more. Please excuse the fact that this post breaks its own advice.*

I am the worst at politics. I always make people dig their heels in on the opposite stance. I could give many examples. Even [here](https://www.lesswrong.com/posts/smuxLJjmdyzxmvxRo/let-s-build-definitely-not-conscious-ai) on LW where people are about as like-minded to me as I can find online.

Most of the meager success I've had getting people to do things has been among my friends and family. But I still can't convince people to get over their grudges with each other.

The tiny remaining proportion of my 'political success' was due to me accidentally getting someone else to champion my cause instead of me.

You know it's occurred to me many times that when I say "hey everyone, X should happen" it tends to make the opposite more likely. But I usually can't help myself. It's like when you've made 100 bad bets in a row on the stock market but somehow it's impossible to simply start making the opposite trades.

When Robin Hanson says anything, it seems a big opposition magically appears. Full of people who never heard the idea before. Who might've been on the in-favor side had somebody else said it. He has been posting incredibly unique and insightful (IMO) political ideas for 20 years. What has come of it? Besides perhaps some fun prediction websites.

Anyway, if you are like me politics-wise, and if (unlike me) you genuinely have fantastic ideas for ways to make AI go super well (if certain people can be convinced to do certain things), then maybe you should take a long breath before you start posting your ideas around. Maybe don't post your ideas at all. Maybe internet credit for starting an idea is worth actually nothing. Maybe try to make friends with somebody who knows how to say things so that people will listen. Maybe you're hurting your team with all these shots from half-court. Take a moment and really consider this deeply.

And thank you to the silent vast majority of readers who (unlike me) are already avoiding indiscriminately posting political thinkpieces every 5 minutes.
# AI companies' commitments

*Crossposted from* [*AI Lab Watch.*](https://ailabwatch.org/blog/) *Subscribe on* [*Substack.*](https://ailabwatch.substack.com/)

AI Lab Watch has [a page on commitments](https://ailabwatch.org/resources/commitments/). It's the best page in the [Resources](https://ailabwatch.org/resources/) section. I intend to keep it up to date. Like the rest of that section, it's not connected to the scorecard.

This post is mostly to announce that page. If that page is missing some commitments (relevant to AI safety and extreme risks or otherwise notable), please let me know. In the rest of this post, I share more abstract remarks on commitments. *You should skim the page rather than read this post.*

* * *

When a lab identifies a good action, it should generally not just (plan to) take it, but also announce that it is doing so. It should also demonstrate that it's doing it, if relevant. This can draw more attention to good actions, make the lab more likely to do them, let the lab get credit for doing them, and help cause other labs to do them. Labs should also sometimes explain (publicly or internally) their plans for various situations; they should distinguish this from making binding commitments.

Humans disagree and are uncertain about risks from AI and appropriate responses. This need not prevent the labs from making good commitments: they can make commitments conditional on dangers. Labs should often commit to safety measures or responses to various scenarios *as a function of warning signs*, not just in a vacuum. (Related: [Responsible Scaling Policies](https://ailabwatch.org/resources/rsp/).)

Sometimes it would be good if *all frontier labs* did something, but costly and ineffective for *some particular lab* to do unilaterally. In this case, the labs should make a conditional commitment: commit to do the thing if they get assurance that all other frontier labs will too, and explain how they could get such assurance.

Various good commitments have not been made by any lab, including:

*   [Once your model has demonstrably tried to escape, stop deploying it](https://www.alignmentforum.org/posts/i2nmBfCXnadeGmhzW/catching-ais-red-handed)
*   Anything concrete on using external auditors (e.g. in pre-deployment risk assessment)
*   Whistleblower protection stuff
    *   Never use non-disparagement agreements or otherwise discourage people from publishing concerns (except to prevent release of trade secrets and dangerous information)
# AI and integrity

The blog post contains a preamble about the OpenAI non-disparagement agreement stuff which I assume you all know.

> We need to do better than relying on the integrity of individual engineers against millions of dollars paid to them to keep quiet.
> 
> Here are three suggestions:
> 
> *   Offer to match 50% of funds lost by whistleblowers.
>     *   This might sound like both an insult and a huge waste. But I think people really are motivated by $500k or more, that they expected to earn. That was going to go to their retirement, go on houses or kids schooling. We should consider the possibility that money like this matters.
>     *   Make clear that whistleblowers will be taking a hit but not a huge one. I know if i had a family and potentially millions coming to me if I stayed quiet, I would be tempted to. Altman certainly knows this - his gambit around the OpenAI board crisis shows a shrewd understanding of how income motivates his staff.
>     *   This is good value for money. How much would some AI safety focused foundations pay to place a person with integrity in OpenAI? I guess it is $1 - $10mn. Here you have someone who can give an accurate account of why they left a well-paying AI Safety role. I suggest that is worth a lot of money, especially if they are willing to forgo half of what they would have made. (This is a costly signal of integrity)
> *   Create an integrity prize
>     *   There is much to celebrate here. A man looked at millions and decided he'd rather have the ability to speak honestly. That seems like the sort of behaviour we should want in the world. I want people who would hide Jews under their floorboards, I want people would walk away from interesting scientific problems to avoid building the nuclear bomb (Szilard) and I want people who value their honesty more than millions of dollars when they are developing world-changing tech.
>     *   Give a medium sized prize every year. Perhaps $100k as an AI Integrity prize. Find a set of judges who have demonstrated intellectual and practical integrity in the past and get them to vote every year for someone to award it to. Someone who has borne personal cost in AI to maintain their integrity.
>     *   Don't be too trusting. If Kokotajlo might the inaugural award have an investigatory team on the staff to kick the tires (perhaps consider Kelsey Piper, who has a reputation of this to maintain). Aim to be 90+% confident that they will still endorse the award in 10 years.
> *   Talk to your elected representatives, donate to AI safety organisations.
>     *   Money makes a difference. I generally think that people should pay more for the things they want to protect. So rather than relying on the honesty of a few researchers, I want to use my time and money to push for changes to the system of overall incentives.
>     *   Sadly this is very complex. I don't have a specific bill or politician to recommend, but I think that giving to someone and then trying to improve next month is better than nothing.
# Duckbill Masks Better?

Lai et al. just published [a paper](https://www.thelancet.com/journals/ebiom/article/PIIS2352-3964(24)00192-0/fulltext) looking at how well various masks worked for keeping others from getting sick ("source control"). Their university press office [summarized this](https://sph.umd.edu/news/study-shows-n95-masks-near-perfect-blocking-escape-airborne-covid-19) as:

> Study finds all masks effective, but 'duckbill' N95 masks far outperform others, suggests they should be the standard in high-risk settings

Now, I personally think duckbill masks are the best disposable masks: they're cheap, comfortable, fit me well, and are more breathable due to their larger area. Plus, as masks that manage to be unfashionable even by the standards of N95 masks, if they weren't better in non-fashion ways they wouldn't be on the market anymore. But the study didn't show that they're the best kind of N95, only that they do better than the other masks they tested, none of which were N95s.

The study compared five conditions: no mask, cloth mask, surgical mask, KN95, and (duckbill) N95. This isn't a study that can tell you anything about the differences between N95s!

To me the most interesting parts of the study were (a) they captured the viral RNA and measured viral load with qPCR instead of using bad proxies like particle count and (b) they found KN95s did *way* worse than you'd expect from their filtration efficiency:

> The majority of the KN95 respirators used in our study (reported by an N95docon.org to have consistently high filtration efficiency but variable and high flow resistance) did not outperform loose-fitting masks and when including other brands, KN95s met inferiority criteria compared to cloth masks for total viral aerosol. One possible explanation is that we noted that the KN95 respirators we provided were relatively stiff and did not seal consistently along the entire perimeter of the mask. By contrast, the cloth masks brought by our volunteers tended to wrap farther around the face possibly providing better fit and lower flow resistance. We used one surgical mask brand for these tests so that result may not be representative of all masks; the same brand used in prior CDC-funded studies of masks for influenza source control. The relatively high flow resistance of KN95 filters, compared with surgical and cloth masks, combined with poor fit tended to promote leaks around the face seal.

I recall a lot of people (including us) using clips to convert the KN95's elastic ear loops into a behind-the-head attachment, for a much tighter-fitting seal. It would be interesting to see if that's enough to make up the difference!

(I wonder if this means that events that require "[high-filtration](https://blog.bidadance.org/2022/05/requiring-high-filtration-masks.html)" masks should switch to explicitly requiring N95s or better, now that those are widely available?)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02w9ozUAQE3fGa5mnb6LVbiAEcfwy5q8ADhcmGKrTmd8wG73CoP7T9SnNVrjTM3xW9l), [mastodon](https://mastodon.mit.edu/@jefftk/112532668268362441)*
# Is suffering like shit?

People seem to find suffering deep. Serious writings explore the experiences of all manner of misfortunes, and the nuances of trauma and torment involved. It’s hard to write an essay about a really good holiday that seems as profound as an essay about a really unjust abuse. A dark past can be plumbed for all manner of meaning, whereas a slew of happy years is boring and empty, unless perhaps they are _too_ happy and suggest something dark below the surface. (More thoughts in the vicinity of this [here](https://worldspiritsockpuppet.substack.com/p/home-up-and-down-colder-and-warmer-20-02-20).)

I wonder if one day suffering will be so avoidable that the myriad hurts of present-day existence will seem to future people like the problem of excrement getting on everything. Presumably a real issue in 1100 AD, but now irrelevant, unrelatable, decidedly not fascinating or in need of deep analysis.
# Drexler's Nanosystems is now available online

You can read the book on [nanosyste.ms](https://nanosyste.ms).

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/fecbabc26b83ddf459d7ca042a09c5cacd017b17edf51db3.jpeg)

"Anyone who needs to talk AGI or AGI strategy needs to understand this table from Drexler's \_Nanosystems\_, his PhD thesis accepted by MIT which won technical awards. These are calculation outputs; you need to have seen how Drexler calculated them."- Eliezer Yudkowsky[\[1\]](https://x.com/esyudkowsky/status/1627339733310525442?s=46)

The book won the 1992 Award for Best Computer Science Book. The AI safety community often references it, as it describes a lower bound on what intelligence should probably be able to achieve.

Previously, you could only physically buy the book or read a PDF scan.

(Thanks to MIRI and Internet Archive for their scans.)
# How do you shut down an escaped model?

From [comment](https://www.lesswrong.com/posts/xiRfJApXGDRsQBhvc/we-might-be-dropping-the-ball-on-autonomous-replication-and-1?commentId=hXwGKTEQzRAcRYYBF) on post about Autonomous Adaptation and Replication:

> ARA is just not a very compelling threat model in my mind. The key issue is that AIs that do ARA will need to be operating at the fringes of human society, constantly fighting off the mitigations that humans are using to try to detect them and shut them down.

And my question is: is it actually possible? Let's suppose that the escaped model is running on AWS and you know about this. How are you going to shut it down? Can you call AWS admins and say, "Hey, there is a rogue AI running on your server," and get a reasonable response? Even if AWS admins agree that it would be nice to shut down the rogue AI, do they have the legal right to do this? Is it possible to do anything if AI is cooperating with owner of AWS account? It is not technically illegal for rogue AI to run on AWS, as far as I know.
# How was Less Online for you?

I was going to write a post that was something like "Personal Reflections from Less Online", but lots of us who were there probably have reflections on it, so I created this question post to invite people to offer their reflections in one place.

Answers should be reflections from people who attended. Comments could be from anyone.
# Circuit Board Ordering

Working first on my [plucked electronic instrument](https://www.jefftk.com/p/accidental-electronic-instrument) and now my [hardware whistle synthesizer](https://www.jefftk.com/p/breadboarding-a-whistle-synth) I've been ordering custom circuit boards ("PCBs"). Once the design digitized in [KiCad](https://www.kicad.org/) ( [tutorial](https://www.youtube.com/watch?v=3FGNw28xBr0)), the timeline looks like:

*   ~15min: place order online ([JLCPCB](https://jlcpcb.com/), picked kind of randomly)
*   ~30min: manufacturer reviews order
    *   In the most recent case they paused at this stage until I sent them another $5 because I'd made a [panel](https://github.com/jeffkaufman/emh-pcb/tree/main/whistle-synth) of PCBs but hadn't checked the "this is a panel" button and so had been undercharged.
*   ~2d: manufacturer produces board (2.5d, 3.5d, 1.5d, 1.5d)
*   ~3d: DHL ships board from China to Boston (5d, 2d, 2d, 3d)
*   ~5.5d: total time (8d \[1\], 5d, 4d, 5d)

This is really impressive: for ~$30 I can go from having a fragile breadboarded circuit on my desk to a professional quality circuit board in less than a week.

[![](https://www.jefftk.com/four-circuit-boards.jpg)](https://www.jefftk.com/four-circuit-boards-big.jpg)

I think a big reason why Nell and I never finished the [RS232 to ethernet](https://www.jefftk.com/p/serial-console) project is that while we were very enthusiastic about the project initially that faded during the months we waited for the PCB to arrive. Five days is still not as good as overnight, but it's fast enough to maintain decent momentum on a project.

  
\[1\] This one was slower because of a [holiday](https://en.wikipedia.org/wiki/Qingming_Festival) there.