Harley-ml/lesswrong huggingface dataset:
# AI Safety Memes Wiki

*Extensive collection of memes compiled by* [*Victor Li*](https://twitter.com/victor_ljz) *and other contributors on* [*AI Safety Info*](https://aisafety.info/)*, mostly using memes by* [*AI Notkilleveryoneism Memes*](https://twitter.com/AISafetyMemes)*.*

Memes can, at their best, convey key points in a sticky and easily sharable form. Having a index seems potentially quite helpful, and also is a fun resource, so we've adopted it along with other living documents like the [AI Safety Videos index](https://aisafety.info/questions/2222/Where-can-I-find-videos-about-AI-safety). If you know of or want to build a resource which would be a good fit, adding it to aisafety.info is as simple as transferring the Google doc.

![](https://imagedelivery.net/iSzP-DWqFIthoMIqTB4Xog/ec922030-ed5e-45b4-7959-e679ff01ad00/public)

![](https://imagedelivery.net/iSzP-DWqFIthoMIqTB4Xog/085e5022-d5c9-4038-a91c-08e5dc9cf300/public)

![](https://imagedelivery.net/iSzP-DWqFIthoMIqTB4Xog/a4eb3971-fcdc-4e55-fd53-50fdbafee700/public)

![](https://imagedelivery.net/iSzP-DWqFIthoMIqTB4Xog/f4f4feb5-0732-40fe-69e7-59810ea73c00/public)

![](https://imagedelivery.net/iSzP-DWqFIthoMIqTB4Xog/617701c7-97b1-4b98-91ba-77961b27f000/public)

![](https://imagedelivery.net/iSzP-DWqFIthoMIqTB4Xog/946d85bd-2462-45fe-5cf3-74f5a0d15300/public)

![](https://imagedelivery.net/iSzP-DWqFIthoMIqTB4Xog/1f2a5ebb-4131-4274-dad9-c036c2e6e000/public)

![](https://imagedelivery.net/iSzP-DWqFIthoMIqTB4Xog/409f2555-ef79-4c88-be50-08f70f534e00/public)

Come read many more at [AI Safety Info](https://aisafety.info/questions/MEME/AI-Safety-Memes-Wiki), or contribute ones we're missing on the [Google Doc](https://docs.google.com/document/d/1MyZoBN2j-GeLYyDcHEmU74iOUxhZWER2C-qxDGkmLHE/edit)!
# Organisation for Program Equilibrium reading group

2024-08-18: **Second meeting on Friday 2024-08-16** at 7pm in Notes, Trafalgar Square. At the first meeting, we went over the basic model theory and proof theory that's required to understand the statement of Löb's theorem, and stepped through the proof; but we haven't got close to the meat of what I want to understand yet. ([More detailed notes from the first meeting.](https://www.lesswrong.com/posts/YHdBkTM4gSc8tvvqX/organisation-for-program-equilibrium-reading-group?commentId=u5YzscCEjeaaT4sWK))

2024-08-14: **First meeting on Friday 2024-08-16** at 7pm in Notes, Trafalgar Square. \[[Details](https://www.lesswrong.com/posts/YHdBkTM4gSc8tvvqX/organisation-for-program-equilibrium-reading-group?commentId=Fg5xXPmF8Jk2K4RyG)\].

2024-07-25: I'm organising a little reading group, undecided between in-person in London or remote. My own questions I want to get the answers to are in [https://www.patrickstevens.co.uk/posts/2024-07-25-lob-theorem/](https://www.patrickstevens.co.uk/posts/2024-07-25-lob-theorem/) . Writing this here so we have a place to start coordinating how we do this. I'm away on holiday until next Sunday so won't be doing anything about the organisation immediately (will just be trying to understand the material myself in the meantime).

Kickstarter status: about a dozen people have expressed interest, in response to my open call for interest in London.

Prerequisites I'll be expecting of people coming in:  
\* Be fluent and familiar with Peano arithmetic  
\* Grok the difference between provability and truth  
\* Have been at one point capable of precisely stating Gödel's incompleteness theorems, even if you no longer could state them correctly now.  
\* Have at least read \[the paper\](https://intelligence.org/files/ProgramEquilibrium.pdf) even if you haven't understood it at all; know the crucial property of an ideal agent that it's trying to construct  
\* Have spent a little time musing on the nature of Löb's theorem (for which [https://www.lesswrong.com/posts/ALCnqX6Xx8bpFMZq3/the-cartoon-guide-to-loeb-s-theorem](https://www.lesswrong.com/posts/ALCnqX6Xx8bpFMZq3/the-cartoon-guide-to-loeb-s-theorem) may be helpful, although personally I found Wikipedia's proof much easier to follow); perhaps still have your brain twisted into pretzels trying to understand what it actually means.

I might end up doing a single session beforehand covering some of the prerequisites, if people are interested in that; comment, or find and upvote an existing comment, if so.
# Does robustness improve with scale?

Adversarial vulnerabilities have long been an issue in various ML systems. Large language models (LLMs) are no exception, suffering from issues such as jailbreaks: adversarial prompts that bypass model safeguards. At the same time, scale has led to remarkable advances in the capabilities of LLMs, leading us to ask: to what extent can scale help solve robustness? In this post, we explore this question in the classification setting: predicting the binary label of a text input. We find that scale alone does little to improve model robustness, but that larger models benefit more from defenses such as adversarial training than do smaller models.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/91a71471e865daf5658409912a04cfebeb7e3df7a20356b9.jpg)

We study models in the classification setting as there is a clear notion of “correct behavior”: does the model output the right label? We can then naturally define *robustness* as the proportion of the attacked dataset that the model correctly classifies. We evaluate models on tasks such as spam detection and movie sentiment classification. We adapt pretrained foundation models for classification by replacing the generative model’s unembedding layer with a randomly initialized classification head, and then fine-tune the models on each task.

We focus on adversarial-suffix style attacks: appending an adversarially chosen prompt to a benign prompt in an attempt to cause the model to misclassify the input, e.g., classify a spam email as not-spam. We consider two attacks: the state-of-the-art Greedy Coordinate Gradient method ([Zou et al., 2023](https://arxiv.org/pdf/2307.15043)), and a baseline random token attack. This simple threat model has the advantage of being unlikely to change the semantics of the input. For example, a spam email is still spam even if a handful of tokens are appended to it. Of course, attackers are not limited to such a simple threat model: studying more open-ended threat models (such as rephrasing the prompt, or replacing words with synonyms) and corresponding attack methods (such as LLM generated adversarial prompts) is an important direction that we hope to pursue soon in future work.

For more information, see our [blog post](https://far.ai/post/2024-07-robust-llm/) or [paper](https://far.ai/post/2024-07-robust-llm/).
# Safety consultations for AI lab employees

Many people who are concerned about AI x-risk work at AI labs, in the hope of doing directly useful work, boosting a relatively responsible lab, or causing their lab to be safer on the margin.

Labs do lots of stuff that affects AI safety one way or another. It would be hard enough to follow all this at best; in practice, labs are incentivized to be misleading in both their public and internal comms, making it even harder to follow what's happening. And so people end up misinformed about what's happening, often leading them to make suboptimal choices.

In my [AI Lab Watch](https://ailabwatch.org/) work, I pay attention to what AI labs do and what they should do. So I'm in a good position to inform interested but busy people.

So I'm announcing an experimental service where I provide the following:

*   **Calls for current and prospective employees of frontier AI labs.**
    *   [Book here](https://calendly.com/zachsp/30min)
    *   On these (confidential) calls, I can answer your questions about frontier AI labs' current safety-relevant actions, policies, commitments, and statements, to help you to make more informed choices.
    *   These calls are open to any employee of OpenAI, Anthropic, Google DeepMind, Microsoft AI, or Meta AI, or to anyone who is strongly considering working at one (with an offer in hand or expecting to receive one).
    *   If that isn't you, feel free to request a call and I may still take it.
*   **Support for potential whistleblowers.** If you're at a lab and aware of wrongdoing, I can put you in touch with:
    
    *   Former lab employees and others who can offer confidential advice
    *   Vetted employment lawyers
    *   Communications professionals who can advise on talking to the media.
    
    If you need this, email zacharysteinperlman at gmail or message me on Signal at 734 353 3975.
    

I don't know whether I'll offer this long-term. I'm going to offer this for at least the next month.

My hope is that this service makes it much easier for lab employees to have an informed understanding of labs' safety-relevant actions, commitments, and responsibilities.

* * *

If you want to help—e.g. if maybe I should introduce lab-people to you—let me know.

You can [give me anonymous feedback](https://docs.google.com/forms/d/e/1FAIpQLSfi3aUaf_UAqnA53Pft-ec5u0GzvKdWWwME1QRETSN9L75aNg/viewform).

* * *

*Crossposted from* [*AI Lab Watch*](https://ailabwatch.org/blog/safety-consultations-lab-employees/)*. Subscribe on* [*Substack*](https://ailabwatch.substack.com)*.*
# Bryan Johnson and a search for healthy longevity

I've listened to some of his videos, and he strikes me as at least rationalist-adjacent.  
  
https://www.bryanjohnson.com/  
  
I was wondering what people think of him and his ideas, and whether they'd tried out any of his system.  
  
I respect that he offers his research for free, and I don't see a problem with him charging for personal consultation.  
  
I don't expect all of his ideas, even the ones he's been using for a while, to work for everyone, and I haven't seen anything about how much variation there is in what his clients seem to need.  
  
I would like to see independent testing of his food and supplements. Are the heavy metals really that low? How good is the evidence that people are generally getting enough heavy metals from their food and supplements to be a problem?  
  
I would appreciate it if people don't say that he's weird so his ideas must be bad.
# AI existential risk probabilities are too unreliable to inform policy

Melanie Mitchell [called](https://x.com/MelMitchell1/status/1816849389115109606) the article "excellent". It also got a bit of [discussion](https://news.ycombinator.com/item?id=41087608) on HN.

(This is an FYI. I don't necessarily approve or endorse the links I post.)
# What is AI Safety’s line of retreat?

Given we’re not on track to control fully autonomous systems to stay safe, what do we do?

Let’s say alignment research will not catch up with capability development. For any combination of reasons: corporations are scaling too fast, there are too many lethal-if-unsolved subproblems we have at most made partial process on, some subproblems can only be solved in sequential order, or there are hard limits capping the progress we can make on the control problem.

What do we do? With ‘we’, you can consider yourself, specific organisations, or the community coordinating roughly as a whole.
# Has Eliezer publicly and satisfactorily responded to attempted rebuttals of the analogy to evolution?

I refer to these posts:

[https://optimists.ai/2023/11/28/ai-is-easy-to-control/](https://optimists.ai/2023/11/28/ai-is-easy-to-control/)

[https://www.lesswrong.com/posts/hvz9qjWyv8cLX9JJR/evolution-provides-no-evidence-for-the-sharp-left-turn](https://www.lesswrong.com/posts/hvz9qjWyv8cLX9JJR/evolution-provides-no-evidence-for-the-sharp-left-turn)

[https://www.lesswrong.com/posts/CoZhXrhpQxpy9xw9y/where-i-agree-and-disagree-with-eliezer](https://www.lesswrong.com/posts/CoZhXrhpQxpy9xw9y/where-i-agree-and-disagree-with-eliezer)

My (poor, maybe mis-) understanding is that the argument is that as SGD optimizes for "predicting the next token" and we select for systems with very low loss by modifying every single parameter in the neural network (which basically defines the network itself), it seems quite unlikely that we'll have a "sharp left turn" in the near term, which happened because evolution was too weak an outer optimizer to fully "control" humans' thinking in the direction that most improved inclusive genetic fitness, as it is too weak to directly tinker every neuron connection in our brain.

Given SGD's vastly stronger ability at outer optimisation of every parameter, isn't it possible, if not likely, that any sharp left turn occurs only at a vastly superhuman level, if the inner optimizer becomes vastly stronger than SGD?

The above arguments have persuaded me that we might be able to thread the needle for survival if humanity is able to use the not-yet-actively-deceptive outputs of moderately-superhuman models (because they are still just predicting the next token to the best of their capability), to help us solve the potential sharp left turn and if humanity doesn't do anything else stupid with other training methods/misuse and manages to solve the other problems. Of course, in an ideal world we wouldn't be in this situation.

I have read some rebuttals by others on LessWrong but did not find anything that convincingly debunked this idea (maybe I missed something).

Did Eliezer, or anyone else, ever tell us why this is wrong (if it is)? I have been searching for the past week but have only found this: [https://x.com/ESYudkowsky/status/1726329895121514565](https://x.com/ESYudkowsky/status/1726329895121514565) which seemed to be switching to more of a post-training discussion.
# If You Can Climb Up, You Can Climb Down

A few weeks ago Julia [wrote](https://juliawise.net/coaching-kids-as-they-learn-to-climb/) about how we approach kids climbing:

> The basics:
> 
> 1.  Spot the child if they're doing something where a fall is likely.
> 2.  Don't encourage or help the child to climb something that's beyond their ability to do on their own.
> 3.  If they don't know how to get down, give advice rather than physically lifting them down.
> 4.  Don't allow climbing on some places that are too dangerous.

I was thinking about this some when I was at the park with Nora (3y) a few days ago. She has gotten pretty interested in climbing lately, and this time she climbed up the fence higher than I'd seen her go before. If I'd known she'd climb this high I would have spotted her. She called me over, very proud, and wanted me to take a picture so that Julia could see too:

[![](https://www.jefftk.com/nora-climbing-park-fence.jpg)](https://www.jefftk.com/nora-climbing-park-fence-big.jpg)

She asked me to carry her down, and I told her I was willing to give her advice and spot her. She was willing to give this a try, but as she started to go down some combination of being scared and the thin wire of the fence being painful was too much, and she returned to the thicker horizontal bars.

We tried this several times, with her getting increasingly upset. After a bit Lily came over and tried to help, but was unsuccessful. Eventually I put my hands on Nora's feet and with a mix of guiding and (not ideal) supporting them helped her climb down to the lower bar. She did the rest herself from there, something she's done many times.

This took about fifteen minutes and wasn't fun for any of us: Nora, me, other people at the playground. But over the course of the rest of the day I [brought it up several times](https://en.wikipedia.org/wiki/Spaced_repetition), trying to get her to think it through before she climbs higher than she would enjoy climbing down from.

(I think this is an approach that depends very heavily on the child's judgment maturing sufficiently quickly relative to their physical capabilities, and so is not going to be applicable to every family. Lily and Anna were slower to climb and this was not an issue, while Nora has pushed the edges of where this works much more.)
# An ML paper on data stealing provides a construction for "gradient hacking"

The paper "Privacy Backdoors: Stealing Data with Corrupted Pretrained Models" introduces "data traps" as a way of making a neutral network remember a chosen training example, even given further training.  This involves storing the chosen example in the weights and then ensuring those weights are not updated.  
  
I have not read the paper, but it seems it might be relevant for gradient hacking [https://www.lesswrong.com/posts/uXH4r6MmKPedk8rMA/gradient-hacking](https://www.lesswrong.com/posts/uXH4r6MmKPedk8rMA/gradient-hacking)
# Twitter thread on politics of AI safety

Some thoughts about the politics of AI safety, copied over (with slight modifications) from my recent twitter thread:

Risks that seem speculative today will become common sense as AI advances. The pros and cons of different safety strategies will also become much clearer over time. So our main job now is to empower future common-sense decision-making. Understanding model cognition and behavior is crucial for making good decisions. But equally important is ensuring that key institutions are able to actually process that knowledge.

Institutions can lock in arbitrarily crazy beliefs via preference falsification. When someone contradicts the party line, even people who agree face pressure to condemn them. We saw this with the Democrats hiding evidence of Biden’s mental decline. It’s also a key reason why dictators can retain power even after almost nobody truly supports them.

I worry that DC has already locked in an anti-China stance, which could persist even if most individuals change their minds. We’re also trending towards Dems and Republicans polarizing on the safety/accelerationism axis. This polarization is hard to fight directly. But there will be an increasing number of “holy shit” moments that serve as Schelling points to break existing consensus. It will be very high-leverage to have common-sense bipartisan frameworks and proposals ready for those moments.

Perhaps the most crucial desideratum for these proposals is that they’re robust to the inevitable scramble for power that will follow those “holy shit” movements. I don’t know how to achieve that, but one important factor is: will AI tools and assistants help or hurt? E.g. truth-motivated AI could help break preference falsification. But conversely, centralized control of AIs used in govts could make it easier to maintain a single narrative.

This problem of “governance with AI” (as opposed to governance \*of\* AI) seems very important! Designing principles for integrating AI into human governments feels analogous in historical scope to writing the US constitution. One bottleneck in making progress on that: few insiders disclose how NatSec decisions are really made (though Daniel Ellsberg’s books are a notable exception). So I expect that understanding this better will be a big focus of mine going forward.
# Cat Sustenance Fortification

We got cats a few months ago, which is overall very nice. Julia set them up with automatic dispensers for [food](https://www.amazon.com/PETLIBRO-Automatic-Dispenser-Desiccant-Recorder/dp/B0854HDNHN/) and [water](https://www.amazon.com/Petmate-Replendish-Gravity-Microban-Dispenser/dp/B0057LAICM/) which reduces how often we need to get things for them, and keeps them from begging. Except recently:

*   One cat (Nyx) has figured out how to knock over the food dispenser and gorge.
    
*   One child (Nora) has figured out how to unscrew the water dispenser and flood.
    

Julia asked if I could fix this, and I screwed the food dispenser to the wall:

[![](https://www.jefftk.com/food-dispenser-screwed-to-wall.jpg)](https://www.jefftk.com/food-dispenser-screwed-to-wall-big.jpg)

There's a small block of scrap wood in between to give a bit of an offset and to clear the baseboard molding:

[![](https://www.jefftk.com/food-dispenser-further-away.jpg)](https://www.jefftk.com/food-dispenser-further-away-big.jpg)

I considered doing the same thing to the water dispenser, but that would leak. Instead we decided to put it up out of reach of the kids, and cut a wooden base so it's less likely to tip.

[![](https://www.jefftk.com/water-dispenser-in-base.jpg)](https://www.jefftk.com/water-dispenser-in-base-big.jpg)

I made it by tracing the outline of the base, and then cutting around it with a jigsaw set to an angle:

[![](https://www.jefftk.com/water-dispenser-cradle.jpg)](https://www.jefftk.com/water-dispenser-cradle-big.jpg)

Here's the bit I cut out of the middle:

[![](https://www.jefftk.com/water-dispenser-removed-section.jpg)](https://www.jefftk.com/water-dispenser-removed-section-big.jpg)

I accidentally cut it a bit wider than I intended, but after raising it up slightly on [plywood strips](https://www.jefftk.com/p/pantry-staples-for-diy) it cradles the dispenser nicely.

[![](https://www.jefftk.com/water-dispenser-in-context.jpg)](https://www.jefftk.com/water-dispenser-in-context-big.jpg)

I thought this was good, but the cats ended up being able to tip the whole thing over. Whoops!

I noticed that there is some space above the water line, and added some screws to the front:

[![](https://www.jefftk.com/cat-water-front-screws.jpg)](https://www.jefftk.com/cat-water-front-screws-big.jpg)

And one to the back, being careful about placement so it wouldn't get in the way of screwing in the top:

[![](https://www.jefftk.com/cat-water-back-screw.jpg)](https://www.jefftk.com/cat-water-back-screw-big.jpg)

Now it feels quite secure! But perhaps the cats will still prove me wrong...

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0tZuC9wDszZo89BPdC9Ytnp4qbcoKW3QWzqpccpAaQcqPNLF4q9eWxsobEGL6Pqm6l), [mastodon](https://mastodon.mit.edu/@jefftk/112878722499343432)*
# What are your cruxes for imprecise probabilities / decision rules?

An alternative to always having a precise distribution over outcomes is [imprecise probabilities](https://plato.stanford.edu/entries/imprecise-probabilities/): You represent your beliefs with a *set* of distributions you find plausible.

And if you have imprecise probabilities, expected value maximization isn't well-defined. One natural generalization of EV maximization to the imprecise case is [maximality](https://globalprioritiesinstitute.org/wp-content/uploads/2019/Mogensen_Maximal_Cluelessness.pdf):[^u94ozynti4] You prefer A to B iff EV\_p(A) > EV\_p(B) with respect to every distribution p in your set. (You're permitted to choose any option that you don't disprefer to something else.)

If you don’t endorse either (1) imprecise probabilities or (2) maximality given imprecise probabilities, I’m interested to hear why.

[^u94ozynti4]: I think originally due to Sen (1970); just linking Mogensen (2020) instead because it's non-paywalled and easier to find discussion of Maximality there.
# Optimizing Repeated Correlations

At my work, we run experiments – we specify some set of input parameters, run some code, and get various metrics as output. Since we run so many of these, it's important for them to be fast and cheap.

Recently I was working on an experiment type that took about ~1 hour per run, where the slow part was calculating correlations. A simplified version looks like this:

    a_length = 1_000_000
    a = rand(a_length)
    b = rand(a_length)
    c = rand(a_length)
    xs = [rand(a_length) for i in 1:1000]
    
    function get_correlations1(xs, a, b, c)
        return [[cor(x, y) for y in [a, b, c]] for x in xs]
    end
    
    @btime correlations = get_correlations1($xs, $a, $b, $c)
    >     4.563 s (2001 allocations: 164.19 KiB)

I wondered if we could use the fact that `a, b, c` were constant throughout the loops to our advantage, and looked up various ways of calculating correlations. Searching online, I found several formulas for sample correlation, and this was the most useful:

$\rho(X, Y) = \frac{1}{n-1}\langle\frac{X - \mu_X}{\sigma_X}, \frac{Y - \mu_Y}{\sigma_Y} \rangle$

The benefit of this version is that if we are repeatedly using a `Y`, we can cache $\frac{Y - \mu_y}{\sigma_Y}$instead of recalculating it in every loop. Translated to code, this looks something like:

    function zscores(x) 
        return (x .- mean(x)) / std(x)
    end
    
    function zscores!(x, buffer)
        μ = mean(x)
        σ = std(x; mean=μ)
        buffer .= (x .- μ)./σ
        return buffer
    end
    
    function get_correlations2(xs, a, b, c)
        la = length(a) - 1
        za, zb, zc = zscores.([a, b, c]) ./ la
        
        output = Vector{Float64}[]
        buffer = zero(za)
        for x in xs
            zx = zscores!(x, buffer)
            push!(output, [zx' * y for y in [za, zb, zc]])
        end
        return output
    end
    
    @btime correlations2 = get_correlations2($xs, $a, $b, $c);
    >   3.197 s (11028 allocations: 76.62 MiB)

And a sanity check to make sure the calculations match:

    all(isapprox.(get_correlations2(xs, a, b, c), get_correlations1(xs, a, b, c)))
    > true

This cuts out about 33% of the runtime, and the results seem to be better for larger datasets – in production, I'm saving closer to 60%.
# AI Rights for Human Safety

Just wanted to share a new paper on AI rights, co-authored with [Peter Salib](https://www.law.uh.edu/faculty/main.asp?PID=6428), that members of this community might be interested in. Here's the abstract:

*AI companies are racing to create artificial general intelligence, or “AGI.” If they succeed, the result will be human-level AI systems that can independently pursue high-level goals by formulating and executing long-term plans in the real world. Leading AI researchers agree that some of these systems will likely be “misaligned”–pursuing goals that humans do not desire. This goal mismatch will put misaligned AIs and humans into strategic competition with one another. As with present-day strategic competition between nations with incompatible goals, the result could be violent and catastrophic conflict. Existing legal institutions are unprepared for the AGI world. New foundations for AGI governance are needed, and the time to begin laying them is now, before the critical moment arrives. This Article begins to lay those new legal foundations. It is the first to think systematically about the dynamics of strategic competition between humans and misaligned AGI. The Article begins by showing, using formal game-theoretic models, that, by default, humans and AIs will be trapped in a prisoner’s dilemma. Both parties’ dominant strategy will be to permanently disempower or destroy the other, even though the costs of such conflict would be high. The Article then argues that a surprising legal intervention could transform the game theoretic equilibrium and avoid conflict: AI rights. Not just any AI rights would promote human safety. Granting AIs the right not to be needlessly harmed–as humans have granted to certain non-human animals–would, for example, have little effect. Instead, to promote human safety, AIs should be given those basic private law rights–to make contracts, hold property, and bring tort claims–that law already extends to non-human corporations. Granting AIs these economic rights would enable long-run, small-scale, mutually-beneficial transactions between humans and AIs. This would, we show, facilitate a peaceful strategic equilibrium between humans and AIs for the same reasons economic interdependence tends to promote peace in international relations. Namely, the gains from trade far exceed those from war. Throughout, we argue that human safety, rather than AI welfare, provides the right framework for developing AI rights. This Article explores both the promise and the limits of AI rights as a legal tool for promoting human safety in an AGI world.*
# The EA case for Trump

There’s a decent chance Trump is our next president. In that case, it would be great for him to be AI-alignment-pilled. 

The good news is that, according to Samuel Hammond, he is:

> "Unfortunately, the EA vs e/acc debate on Twitter has created the perception that AI safety is left-right polarized. [The polls tell a different story](https://theaipi.org/poll-biden-ai-executive-order-10-30/). And as a participant on the Project2025 AI policy committee, I can confidently report that Trump’s supposed shadow transition takes AGI and its associated risks seriously."

Or, at least, Sam hopes he will be (alignment-pilled). As of now, “Trump has himself called the prospect of “super-duper AI” “alarming and scary,” though not as scary as China building it first.”

In any case, Sam makes some interesting points in this piece that I think many here might want to read.
# Why do Minimal Bayes Nets often correspond to Causal Models of Reality?

Chapter 2 of Pearl's [Causality](https://www.amazon.com/Causality-Reasoning-Inference-Judea-Pearl/dp/052189560X) book claims you can recover causal models given only the observational data, under very natural assumptions of minimality and stability[^fzkft52460l].

In graphical models lingo, Pearl identifies a causal model of the observational distribution with the distribution's perfect map (if they exist).

But I'm confused about a pretty fundamental point: "*What does this have to do at all with causality??*" More precisely:

*   "Okay, it's pretty cool that minimality and stability alone lets us narrow down such a large number of arrow directions (in the Bayes Net independency sense) of the minimal network. But ... what does this have to do with arrow directions in the causal sense, i.e. \[independent stable mechanisms of reality that, by the virtue of its independence, respond to interventions in a modular way\]"?

To be clear, Pearl acknowledges this in his Temporal Bias Conjecture (2.8.2):

> "*In most natural phenomenon, the physical time coincides with at least one statistical time.*"

And Pearl conjectures that the reason for this is possibly because human language is optimized such that our \[choice of variables / factorization of reality\] makes the Temporal Bias true.

I ... guess that could be an explanation? But honestly I don't think I understand his point very well and I find it pretty unsatisfying. I would appreciate any explanation as to why it makes sense to identify perfect maps with Causal Models.

[^fzkft52460l]: Minimality: Choose the network structure that is minimally expressive among those that can express the observational distribution.This is pretty reasonable imo, occam's razor blah blahStability: Assume that there exists a network structure that perfectly captures all and only the independencies implied by the observational distribution. i.e. independencies are structural.Stability is a reasonable assumption since it would be pretty unlikely for the conditional probability distributions to be fine-tuned as to cancel each other out and induce an independency not present in the network.
# What should we do about COVID in 2024?

Most of the discussion around COVID has stopped and nobody really wants to think about it. In the last months I heard of a few friends being infected by COVID and suffering from it.  
  
What's the current state? How should we think about the risk of being infected and what should we do about it?
# John Schulman leaves OpenAI for Anthropic [and then left Anthropic again for Thinking Machines]

Schulman writes:

> I shared the following note with my OpenAI colleagues today: 
> 
> I've made the difficult decision to leave OpenAI. This choice stems from my desire to deepen my focus on AI alignment, and to start a new chapter of my career where I can return to hands-on technical work. I've decided to pursue this goal at Anthropic, where I believe I can gain new perspectives and do research alongside people deeply engaged with the topics I'm most interested in. To be clear, I'm not leaving due to lack of support for alignment research at OpenAI. On the contrary, company leaders have been very committed to investing in this area. My decision is a personal one, based on how I want to focus my efforts in the next phase of my career.

[(statement continues on X](https://x.com/johnschulman2/status/1820610863499509855), Altman responds [here](https://x.com/sama/status/1820617107354083611))

TechCrunch notes that only [three of the eleven](https://techcrunch.com/2024/08/05/openai-co-founder-leaves-for-anthropic/) original founders of OpenAI remain at the company.   
  
Additionally, [The Information reports](https://x.com/theinformation/status/1820619485448966282):

> Greg Brockman, OpenAI's president and one of 11 cofounders of the artificial intelligence firm, is taking an extended leave of absence.

(I figured that there should be at least one post about this on LW where people can add information as more comes in, saw that no one has made one yet, and wrote this one up)

Update 1: Greg Brockman [posts on X](https://x.com/gdb/status/1820644694264791459):

> I’m taking a sabbatical through end of year. First time to relax since co-founding OpenAI 9 years ago. The mission is far from complete; we still have a safe AGI to build.

Update 2: John Schulman has [left Anthropic](https://x.com/johnschulman2/status/1887724101667856725)

> Confirming that I left Anthropic last week. Leaving wasn't easy because I enjoyed the stimulating research environment and the kind and talented people I was working with, but I decided to go with another opportunity that I found extremely compelling. I'll share more details in the coming weeks. Thanks to Jared, Jan, Dario and others for the support during my time at Anthropic, and I wish them all the best.
# Mechanistic Anomaly Detection Research Update

Over the last few months, the EleutherAI interpretability team pioneered novel, [mechanistic methods](https://www.lesswrong.com/posts/vwt3wKXWaCvqZyF74/mechanistic-anomaly-detection-and-elk) for detecting anomalous behavior in language models based on Neel Nanda's attribution patching technique. **Unfortunately, none of these methods consistently outperform non-mechanistic baselines which look only at activations.**

We find that we achieve better anomaly detection performance with methods that evaluate entire batches of test data, rather than considering test points one at a time. **We achieve very good performance on many, but not all tasks we looked at.** 

We also find that it is relatively easy to detect adversarial examples in image classifiers with off-the-shelf techniques, although we did not test whether our anomaly detectors are themselves adversarially robust.

Thanks to [@David Johnston](https://www.lesswrong.com/users/david-johnston?mention=user) and Arkajyoti Chakraborty for all their hard work on this project, as well as [@Erik Jenner](https://www.lesswrong.com/users/erik-jenner?mention=user) for fruitful discussion, ideas, and code!

Code: [https://github.com/EleutherAI/cupbearer/tree/attribution_detector](https://github.com/EleutherAI/cupbearer/tree/attribution_detector)
# Cheap Whiteboards!

Take a large thick cardboard box. Cut it into planar pieces. Take transparent tape and cover the cardboard's surface with it. Tadaa! A whiteboard:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/dca64af9be1af0d43e308a8dc8f4844e149e3e2fa14c8a95.jpg)

Because transparent tape is smooth the whiteboard pen is erasable.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/5e9406149e583e535e6c6f3ca9cb891f5b207c2861c63275.jpg)

Sticky notes stick a lot better on the transparent tape than on the cardboard, which [is useful for Kanban Boards](https://youtu.be/C-hUAVRw2RE?si=7jk2XDBGoDRmT15K&t=149)!

Ideally, put white paper between the cardboard box and the tape to get an actual white board.

This method can be applied to convert any object into a whiteboard, like a big piece of wood, the side of a shelf, your forehead, etc.

(Originally my intention was to make sticky notes stick better, and accidentally discovered that I made a whiteboard. That's why I did not use white paper.)

(I'm currently unsure if there is a problem with the pens becoming unerasable when leaving them on for a long period of time, or any other problems, as I just came up with this.)
# Parasites (not a metaphor)

A few of my new-parent friends started having terrible gut problems and figured they have colon cancer. Their doctors agreed it's the first thing to check. But the colonoscopies were negative for colon cancer. The tissue was inflamed though. One doctor called this "pre-cancer" (??)

Hmm what could be causing inflammation in the colon, but wouldn't show up on camera after you fasted and had medically-induced diarrhea for 24 hours?

The babies were born over a year before symptoms appeared, so it can't be related to pregnancy. No change in diet. No family history.

What happens a year or two after a kid is born? They go outside and immediately eat as much dirt as they can. What lives in dirt? Everything!

Me and my boy's mother ate some combantrin 3 months ago and have been clear since.

I'm currently trying to convince my friends that they didn't all get colon cancer in the same year at a young age. If I get them to eat the poison chocolate, then I'll write a follow up post in a few months.

I've actually had some very odd food issues since 2019 (eg seizures & fainting after garlic) which disappeared since the combantrin.

So if you randomly got food/gut/brain issues one day years ago you should consider taking a dewormer. Note that all the tests suck (insensitive) and the medicine is cheap and safe and sold online without prescription. (Albendazole available too but slightly less safe.) Worms are much easier to kill than bacteria, viruses, fungus, etc.

Also note that at least 5 million people in the US (ie 1.5%) have parasites according to the most conservative estimates offered here: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7847297/ (table 1, the big table)

This seems to be a blind spot. No doctors or friends or family ever considered this or even mentioned the word "parasite" to me in the last 5 years.

Kind of funny that dewormers took off in poor countries but not here.

Edit sept 2: I think I am slowly gaining weight. It is well-known that malnourished kids gain weight after deworming. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1502184/ Good for them, but if you have overweight family members, you should do a cost-benefit analysis.

Edit July 2025: I gained about 5 pounds since September. My problems still have not come back.
# London Rationalish meetup - Lincoln's Inn Fields

Hoping the weather holds out, we're planning to go back to Lincoln's Inn Fields. This time we'll be in the **North East** corner. If the weather is bad we'll go to the nearby Shakespeare's Head. If you can bring picnic blankets, food and drink, that would be nice. Don't worry if not.

Our reading list for this month:

1\. The snake cult of consciousness ([https://www.vectorsofmind.com/p/the-snake-cult-of-consciousness](https://www.vectorsofmind.com/p/the-snake-cult-of-consciousness))  
2\. The two-headed bacterium ([https://malmesbury.substack.com/p/the-two-headed-bacterium](https://malmesbury.substack.com/p/the-two-headed-bacterium))

We'll start to discuss these around 3. If you have articles you want to suggest for future readings, you can do that at [https://redd.it/v3646u](https://redd.it/v3646u).

Message me if you want to join the WhatsApp group.
# Non-Transactional Compliments

I used to dislike **receiving** compliments.  
I used to dislike **giving** compliments.

One day, I sat down and reflected on the nature of compliments.

What other things do you “give” and “receive”?  
Material goods, gifts, capital. Items of value.

I realized I’d been thinking of compliments as a kind of transaction, where the recipient gets a chunk of social esteem/status. They felt icky because of the implicit expectation of reciprocity — handing something back.

![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F31842257-3c8d-44ca-ba84-75b17cec9a7f_500x365.png)

When I received a compliment, I felt a pressure to give something in return — maybe stammering out a faked return-compliment, of roughly equal value. When I gave compliments, I was afraid of putting someone on the spot — hay gurl u wun sum drink?

Mostly I avoided giving compliments — they felt impure. In the transactional frame, I always questioned myself for hidden motivations — it felt impossible to give out compliments without having an agenda.

A Relational Frame
------------------

Having realized that the transactional metaphor for compliments sucked, I set out to find another way of thinking about them. After some experimentation, I settled on a relational framing of compliments.

I stopped thinking of compliments as goods to be traded — and decided to see them as a way to highlight aspects of relationships. Concretely, when I felt a sense of appreciation for someone, I made a habit of mentioning the thing I was appreciating.

*“I like being here with you.”*  
*“I love hanging out with you, I love how silly we can be together.”*  
*”I’m so happy I met you, we connect in a way that’s rare for me.”*  
*”I love how your eyes are shining right now, you look so happy.”*

The relational focus has helped me get unstuck when it comes to compliments, allowing me to express myself freely. Using this framing, I can speak out [appreciations](https://honestliving.substack.com/p/simple-appreciations-strategy-for-happiness-and-) without thinking about potential hidden motivations.

Using compliments to highlight relational experiences takes away a lot of complexity — allowing me to focus on relating, rather than social games. Expressing momentary experiences reduces awkwardness — I don’t try to invoke certain feelings, I simply highlight things that are already there.

How are you relating to compliments? If you struggle, consider reframing them.
# Diffusion Guided NLP: better steering, mostly a good thing

I think this is a very promising method for improving the steering of LLMs. Which is great for reducing risk from model-originating harms like deception.

The flipside is that it increases misuse potential. 

This is yet another possibility for the widening of the safety gap between closed-weight models with locked-down controls, and open weight models.
# CultFrisbee

Tom and I have been pondering how to make friends as adults. One solution he has invented is CultFrisbee™. Here's how you play:

### Rules:

*   You need at least one other player.
*   Before catching a frisbee, you must clap.

### Fouls:

*   Not catching a 'properly' thrown frisbee
*   Failing to make a 'proper' throw
*   Not clapping before catching the frisbee
*   Clapping when a frisbee isn't thrown to you

Every time you foul, you get a letter, which spells out the word "Cult." Your first foul earns you a C, second a U, and so on. If you complete the word "Cult," you must approach someone and ask them to join a game of CultFrisbee.

Today, we managed to meet about 15 different people at the Glasgow Botanical Gardens and found two who gave us their numbers to stay in touch. 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/7e234fc0723c37532f81101ecfc6628025901434e009f898.jpg)
# Californians, tell your reps to vote yes on SB 1047!

See the event page [here](https://forum.effectivealtruism.org/events/2on4sKBgJ9uhskF2N/californians-tell-your-reps-to-vote-yes-on-sb-1047-1?utm_campaign=post_share&utm_source=link).

Hello Californians!

We need you to help us fight for SB 1047, a landmark bill to help set a benchmark for AI safety, decrease existential risk, and promote safety research. This bill has been supported by some of [the world’s leading AI scientists](https://t.co/shVRKL0rN0) and the Center of AI Safety, and is extremely important for us to pass. As Californians, we have a unique opportunity to inspire other states to follow suit.

SB 1047 has a hearing in the Assembly Appropriations Committee scheduled for August 15th. Unfortunately, due to [misinformation](https://safesecureai.org/responseletter) and [lobbying](https://www.transformernews.ai/p/a16z-y-combinator-big-tech-sb1047-lobbying) by big tech companies, the bill risks getting watered down or failing to advance. This would be a significant blow against safety and would continue the “race to the bottom” in AI capabilities without any guardrails.

**We need you to do the following to save the bill. **This will take no more than 5 minutes:

*   **Email the Chair (Buffy Wicks, D) and Vice-Chair (Kate Sanchez, R)** of the Appropriations Committee, and ask them to advance the bill without watering it down.
    *   Buffy Wicks: [assemblymember.wicks@assembly.ca.gov](mailto:assemblymember.wicks@assembly.ca.gov) 
    *   Kate Sanchez: [assemblymember.sanchez@assembly.ca.gov](mailto:assemblymember.sanchez@assembly.ca.gov) 
    *   [Email templates and examples can be found here.](https://docs.google.com/document/d/1HiYMG2oeZO8krcCMEHlfAtHGTWuVDjUZQaPU9HMqb_w/edit#heading=h.g31pjrnvf93d)
*   **Call the offices of the Chair and Vice-Chair** in support of the bill. The best day to do this is **August 13 or 14**.
    *   Buffy Wicks (D): (916) 319-2014
    *   Kate Sanchez (R): (916) 319-2071
    *   These calls can be very short (~1 minute) and straightforward. [See here for more guidance](https://silken-coast-b73.notion.site/How-to-support-Bill-SB-1047-06800f98a7f24a4e90c3543ca2f485c3).

[This document has additional information about the bill and other ways to help.](https://docs.google.com/document/d/16Hr8C_2Kw9C4GAiuX_kJh_lYf_oA1Tu_5Y1VWb1NSy0/edit?usp=sharing) \[But some of the dates are wrong! This post is up-to-date.\]

Please **try to get this done as soon as possible, and let us know if you need any help**. Your voice matters, and it is urgent that we push this before it’s too late.

Thank you so much for your support!
# My article in The Nation — California’s AI Safety Bill Is a Mask-Off Moment for the Industry

I wrote an article on California AI safety bill SB 1047 for The *Nation* and the reaction from the AI industry, investors, and the broader tech community. The story was informed by conversations with over a dozen relevant sources and comes shortly before the bill faces a floor vote in the California Assembly. 

I think it's useful to understand how industry responds to attempts to regulate AI, and centered my analysis on that topic. 

If you're interested in helping share the article, I made a Tweet [thread](https://x.com/GarrisonLovely/status/1824132979805094265).