Harley-ml/lesswrong huggingface dataset:
# Veo-2 Can Produce Realistic Ads

Veo-2 is google's latest video-generation model. Released Dec 16th, it's [quite impressive](https://deepmind.google/technologies/veo/veo-2/)! Of course, there are still limitations (available in that previous link), especially w/ more complex movements (e.g. skateboarder & ballerina) and consistency. 

Then we have this, very [realistic ad](https://www.youtube.com/watch?v=VqLWWYfCEbI) created by one person in ~3 weeks. 

Most people would not be able to tell this is AI-generated (maybe 1/100k people could tell unprompted?). It is still human-edited and uses human voices. Some quick facts from the author: 

*   [How long did it take?:](https://old.reddit.com/r/singularity/comments/1i5tjv1/is_this_what_singularity_looks_like_veo2/m8aq1sm/) yup, not sure the hours. What I can backtrack is I generated 12 days for the ad, and 4 for the BTS. add editing, sound design, etc. Maybe 3 weeks?
*   [All AI?](https://old.reddit.com/r/singularity/comments/1i5tjv1/is_this_what_singularity_looks_like_veo2/m86fw7c/): Every shot is AI except the talking head in the BTS part.
*   [Hardest Part:](https://old.reddit.com/r/singularity/comments/1i5tjv1/is_this_what_singularity_looks_like_veo2/m8a0vmz/) Character consistency. You can only use text now, so you have to make sure the characters stay the same as much to trick the viewer it’s the same person!

Some tricks they employed:

1.  Very quick shots/edits. Notice the transitions are quick, which likely helps with loss of consistency across shots.
2.  No AI-generated videos w/ voice (the narrator was from Fiverr, and the "director" is the creator)
3.  No complex scenes (e.g. the skateboarder)

For the limitations:

1.  Consistency across scenes & characters: currently the interface only allows production from text. For character consistency, starting from an existing character in video, and continuing from there is one solution.
    1.  Google might not allow this though due to easily violating others IP and other concerns.
2.  AI lip sync:  I've seen several impressive demos from research papers these past two years, so I'd guess it's already mostly solved. It'd just need to be integrated in the interface
    1.  Looking at this recent [Kling AI video](https://www.youtube.com/watch?v=kwwZ35EyCDg) though, the results aren't great.
3.  Complex Scenes: More scale and data usually does the trick. Possibly could use AI-assisted physics engines to help render these (or produce mass amounts of data), but that's definitely not my expertise, just speculation.
# Quotes from the Stargate press conference

Present alongside President Trump:

*    Sam Altman (who President Trump introduces as "by far the leading expert" on AI)
*   Larry Ellison ([Oracle executive chairman and CTO](https://en.wikipedia.org/wiki/Larry_Ellison))
*   Masayoshi Son ([Softbank CEO](https://en.wikipedia.org/wiki/Masayoshi_Son) who [believes he was born to realize ASI)](https://fortune.com/2024/06/27/softbank-ceo-masayoshi-son-born-to-create-artificial-super-intelligence/)

> **President Trump**: What we want to do is we want to keep \[AI datacenters\] in this country. China is a competitor and others are competitors.

> **President Trump**: I'm going to help a lot through emergency declarations because we have an emergency. We have to get this stuff built. So they have to produce a lot of electricity and we'll make it possible for them to get that production done very easily at their own plants if they want, where they'll build at the plant, the AI plant they'll build energy generation and that will be incredible.

> **President Trump**: Beginning immediately, Stargate will be building the physical and virtual infrastructure to power the next generation of advancements in AI, and this will include the construction of colossal data centers

> **Larry Ellison**: The first of them are under construction in Texas. Each building is a half a million square feet. There are 10 buildings currently, currently being built, but that will expand to 20 and other locations beyond the Abilene location, which is, which is our first location.

> **Masayoshi Son**: Mr. President, uh, last, last month I came to celebrate your winning and promised that we will invest $100 billion and you told me, oh Masa, go for $200. Now I came back with 500.

> **Masayoshi Son**: I think AGI is coming very, very soon, and after that, that's not the goal. After that, artificial superintelligence will come to solve the issues that mankind would never ever have thought that we could solve. Well, this is the beginning of our golden age. Thank you very much.

> **Sam Altman**: I don't have too much to add, but I do want to say I'm thrilled we get to do this in the United States of America. I think this will be the most important project of this era and as said, for AGI to get built here to create hundreds of thousands of jobs, to create a new industry centered here.

In response to the only on-topic question, the President replied:

> **President Trump**: I'd like to see federal lands opened up for data centers. I think they're going to be very important again, we have a lot of competition for that
# How useful would alien alignment research be? 

Imagine aliens on a distant world. They have values very different to humans. However, they also have complicated values, and don't exactly know their own values. 

Imagine these aliens are doing well at AI alignment. They are just about to boot up a friendly (to them) superintelligence. 

Now imagine we get to see all their source code and research notes. How helpful would this be for humans solving alignment?
# Uncontrollable: A Surprisingly Good Introduction to AI Risk

I recently read Darren McKee's book "Uncontrollable: The Threat of Artificial Superintelligence and the Race to Save the World".

I recommend this book as the best current introduction to AI risk for people with limited AI background.

It prompted me to update my thinking about Asimov's Laws and related risks in light of recent evidence about AI capabilities. I've written a [longer review](https://bayesianinvestor.com/blog/index.php/2025/01/23/uncontrollable/) that explains my thoughts in more detail.

The author has a [LW account here](https://www.lesswrong.com/users/darren-mckee).
# Do you consider perfect surveillance inevitable?

A lot of [my recent research work](http://samuelshadrach.com/?file=/raw/english/my_ideas/world_models_im_currently_building.md) focusses on:

1\. building the case for why perfect surveillance is becoming increasingly hard to avoid in the future

2\. thinking through the implications of this, if it happened

When I say perfect surveillance, imagine everything your eyes see and your ears hear is being broadcast 24x7x365 to youtube (and its equivalents in countries where youtube is banned) and imagine this is true for all 8 billion people.

I'm unsure whether I should devote more of my research time to 1 or 2. If lots of people buy the hypothesis in 1, I'd rather devote time to 2. If people don't buy the hypothesis in 1, I might want to spend time on 1 instead.

Hence I wished to solicit opinions. Do you consider perfect surveillance inevitable? Why or why not?
# Insights from "The Manga Guide to Physiology"

> Physiology seemed like a grab-bag of random processes which no one really understands. If you understand a physiological process—congratulations, that idea probably doesn’t transfer much to other domains. You just know how humans—and maybe closely related animals—do the thing. At least, that’s how I felt. (These sentiments tend to feel sillier when spelled out.)
> 
> I haven't totally changed my mind. But I *do* know that it's nice to look down at my hand and be able to pop through multiple understandings of my body. I start with the circulatory system and I might imagine how oxygenated hemoglobin turns bright red, tracing the flow back down to my median cubital vein (where blood is drawn). My mind’s eye flickers as I imagine the peripheral nervous system and its reflex arcs, wondering at the preconditions which trigger spinal short-circuits (and at what preprocessing is done to customize the reflex to the situation). Another flicker and I see the muscles and joints and their lubricants and connectives; gazing through flesh at my ulna and remembering the [repetitive stress injury which once tormented me](https://www.lesswrong.com/posts/dqSwccGTWyBgxrR58/turntrout-s-shortform-feed#m5EJpjkcvCn8KxJvn)…

[**Read: https://turntrout.com/insights-from-physiology**](https://turntrout.com/insights-from-physiology)
# Recommendations for Recent Posts/Sequences on Instrumental Rationality?

I absolutely love the [Science of Winning at Life](https://www.lesswrong.com/s/oi873FWi6pHWxswSa) sequence. It's a delightful blend of well-researched cognitive science and Bayesian reasoning. The initial paragraph sums up [@lukeprog](https://www.lesswrong.com/users/lukeprog?mention=user)'s motivation:

> Some have suggested that the Less Wrong community could improve readers' instrumental rationality more effectively if it first caught up with the scientific literature on productivity and self-help, and then enabled readers to deliberately practice self-help skills and apply what they've learned in real life.

Unfortunately, the sequence is almost 15 years old, and so is its greatest reference: the 2000-page, evidence-driven, free-to-read masterpiece, [*Psychological Self-Help*](https://www.psychologicalselfhelp.org/). It's fair to assume that cogsci has made strides since then. Some of Luke's material may have been disproven, or more effective methods have since been discovered.

What are some recent posts or sequences that would fill this gap? Thank you in advance!
# Are we trying to figure out if AI is conscious?

My aim in this article is to examine the field of consciousness research with an analysis of 78 of the most important papers in the field. In doing so, I compare the empirical foundations these disciplines rely upon and detail how each approaches the measurement of conscious experience.
# Is the output of the softmax in a single transformer attention head usually winner-takes-all?

Using the notation from here: [A Mathematical Framework for Transformer Circuits](https://transformer-circuits.pub/2021/framework/index.html)

The attention pattern for a single attention head is determined by $A = \text{softmax}(x^TW_Q^TW_Kx)$, where softmax is computed for each row of $x^TW_Q^TW_Kx$.

Each row of $A$ gives the attention pattern for the current token. Are these rows (post softmax) typically close to one-hot? I.e. are they mainly dominated by a single attention (per current token).

I'm interested in knowing this for various types of transformers, but mainly for LLM and/or frontier models. 

I'm asking because I think this has implication for computations in super-position.
# Those of you with lots of meditation experience: How did it influence your understanding of philosophy of mind and topics such as qualia?

This post is inspired by the post "[Why it's so hard to talk about Consciousness](https://www.lesswrong.com/posts/NyiFLzSrkfkDW4S7o/why-it-s-so-hard-to-talk-about-consciousness)" by Rafael Harth. In that post, Harth says that the people who participate in debates about consciousness can be roughly divided into two "camps":

> Camp #1 tends to think of consciousness as a non-special high-level phenomenon. *Solving* consciousness is then tantamount to solving the *Meta-Problem* of consciousness, which is to explain why we think/claim to have consciousness. In other words, once we've explained the full causal chain that ends with people uttering the sounds *kon-shush-nuhs*, we've explained all the hard observable facts, and the idea that there's anything else seems dangerously speculative/unscientific. No complicated metaphysics is required for this approach.

> Conversely, Camp #2 is convinced that there is an experience thing that exists in a fundamental way. There's no agreement on what this thing is – some postulate causally active non-material stuff, whereas others agree with Camp #1 that there's nothing operating outside the laws of physics – but they all agree that there is *something* that needs explaining. Therefore, even if consciousness is compatible with the laws of physics, it still poses a conceptual mystery relative to our current understanding. A complete solution (if it is even possible) may also have a nontrivial metaphysical component.

One possible avenue of explanation for this (as discussed extensively in the comment section under Harth's post) is that different people experience their own minds differently, for all sorts of reasons.

I know some people here have a lot of experience with meditation and have experienced major results and "insights" from it. Moreover, as far as I know, most western philosophers of mind are *not* expert meditators. It is conceivable that meditators have access to information about the human mind which most philosophers of mind lack.

So I am interested in hearing from those of you who have a decent amount of meditation experience: How have your personal experiences influenced your understanding of western philosophy of mind. Not only on the topic of qualia; that was just the example that motivated me to post the question. For example, did anyone move from Harth's camp #1 to camp #2 or vice versa after meditation experiences, or did any of your other philosophical positions shift?

(I would ask this in a Buddhist forum, and I probably will, but I fear that most people will say "stop doing philosophy of mind and go follow the Buddhist suttas"...)

I myself have about 700 hours of meditation experience, and while I have gained some useful *skills* (mainly emotion handling), I would not say that I have gained any significant *insight* yet.
# Constitutions for ASI?

The linked post seemed to fit better in EA Forum, but any comments (on the post itself, or on the object-level question) are welcome !

> In this post, I argue that drafting a ‘Constitution for Superintelligence’ (CSI) could be a useful conceptual exercise, and I explore how existing ideas (CEV, Letters to AI, Long Reflection, and Constitutional AI) might inform such a project. Key considerations in writing such a document include:

> a. How to craft a foundational document to guide ASI behavior, in ways that do not anchor purely upon human interests (e.g. relative to other entities) without a point-of-view-of-the-universe justification

> b. Investigating the mechanisms by which such documents could influence the actions of current and future AI systems

> c. Seeking input and collaboration from EAs/alignment people with relevant expertise and insights to further develop these concepts

>The aim is to proactively but cautiously shape the trajectory of ASI development, in order to improve chances of maximally-inclusive beneficial outcomes.
# Paper: Open Problems in Mechanistic Interpretability

**TL;DR: This paper brings together ~30 mechanistic interpretability researchers from 18 different research orgs to review current progress and the main open problems of the field.**

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/f5ad1c44b551feab3b22d2c4a80359f755fc3cf78e86e229.png)

  
  
*This review collects the perspectives of its various authors and represents a synthesis of their views by Apollo Research on behalf of Schmidt Sciences. The perspectives presented here do not necessarily reflect the views of any individual author or the institutions with which they are affiliated.*

Abstract
========

Mechanistic interpretability aims to understand the computational mechanisms underlying neural networks’ capabilities in order to accomplish concrete scientific and engineering goals. 

Progress in this field thus promises to provide greater assurance over AI system behavior and shed light on exciting scientific questions about the nature of intelligence. 

Despite recent progress toward these goals, there are many open problems in the field that require solutions before many scientific and practical benefits can be realized: 

Our methods require both conceptual and practical improvements to reveal deeper insights; we must figure out how best to apply our methods in pursuit of specific goals; and the field must grapple with socio-technical challenges that influence and are influenced by our work. 

This forward-facing review discusses the current frontier of mechanistic interpretability and the open problems that the field may benefit from prioritizing.

Contents
========

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/11b3928d6984c92cb63c92034ed927741dcdd28d5d3ad0bc.png)

  
Full paper [here](https://arxiv.org/abs/2501.16496).
# Anthropic CEO calls for RSI

"If China can't get millions of chips, we'll (at least temporarily) live in a unipolar world, where only the US and its allies have these models.

**It's unclear whether the unipolar world will last, but there's at least the possibility that, because AI systems can eventually help make even smarter AI systems, a temporary lead could be parlayed into a durable advantage.**

Thus, in this world, the US and its allies might take a commanding and long-lasting lead on the global stage.

Well-enforced export controls are the only thing that can prevent China from getting millions of chips, and are therefore the most important determinant of whether we end up in a unipolar or bipolar world."

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/2d65701d440601e715460109e5c6af9f57349da47fe44cdb.png)

From Anthropic CEO Dario Amodei's latest piece, [On DeepSeek and Export Controls](https://darioamodei.com/on-deepseek-and-export-controls).
# Dario Amodei: On DeepSeek and Export Controls

Dario corrects misconceptions and endorses export controls.

> A few weeks ago I [made the case](https://www.wsj.com/opinion/trump-can-keep-americas-ai-advantage-china-chips-data-eccdce91) for stronger US export controls on chips to China. Since then DeepSeek, a Chinese AI company, has managed to — at least in some respects — [come close to the performance](https://techcrunch.com/2025/01/27/deepseek-claims-its-reasoning-model-beats-openais-o1-on-certain-benchmarks/) of US frontier AI models at lower cost.
> 
> Here, I won't focus on whether DeepSeek is or isn't a threat to US AI companies like Anthropic (although I do believe many of the claims about their threat to US AI leadership are greatly overstated)[^1^](https://darioamodei.com/on-deepseek-and-export-controls#fn:1). Instead, I'll focus on whether DeepSeek's releases undermine the case for those export control policies on chips. I don't think they do. In fact, **I think they make export control policies even more existentially important than they were a week ago**[**^2^**](https://darioamodei.com/on-deepseek-and-export-controls#fn:2).
> 
> Export controls serve a vital purpose: keeping democratic nations at the forefront of AI development. To be clear, they’re not a way to duck the competition between the US and China. In the end, AI companies in the US and other democracies must have better models than those in China if we want to prevail. But we shouldn't hand the Chinese Communist Party technological advantages when we don't have to.

* * *

Also:

> DeepSeek does not "do for $6M what cost US AI companies billions". I can only speak for Anthropic, but Claude 3.5 Sonnet is a mid-sized model that cost a few $10M's to train (I won't give an exact number). Also, 3.5 Sonnet was *not* trained in any way that involved a larger or more expensive model (contrary to some rumors).

Also:

> Making AI that is smarter than almost all humans at almost all things will require millions of chips, tens of billions of dollars (at least), and is most likely to happen in 2026-2027.

One thing seems wrong:

> If China *can't* get millions of chips, we'll (at least temporarily) live in a *unipolar* world, where only the US and its allies have these models.

If "smarter than almost all humans at almost all things" models appear in 2026-2027, China and several others will be able to ~immediately steal the first such models, by default.
# My Mental Model of AI Optimist Opinions

*Epistemic status: The text below is a sort of strawman of AI optimists, where I took my mental model for how I disagree with rationalist AI optimists and cranked it up to 11. I personally disagree with every sentence below, and I'm posting it here because I'm interested in whether AI optimists have any major corrections they want to make in the comments. Of course I understand that everyone has their own unique opinion and so I would expect every AI optimist to at least disagree with some parts of it too.*

The rapid progress spearheaded by OpenAI is clearly leading to artificial intelligence that will soon surpass humanity in every way. People used to be worried about existential risk from misalignment, yet we have a good idea about what influence current AIs are having on the world, and it is basically going fine.

The root problem is that The Sequences expected AGI to develop agency largely without human help; meanwhile *actual* AI progress occurs by optimizing the scaling efficiency of a pretraining process that is mostly focus on integrating the AI with human culture. This means we will be able to control AI by just asking it to do good things, showing it some examples and giving it some ranked feedback.

You might think this is changing with inference-time scaling, yet if the alignment would fall apart as new methods get taken into use, we'd have seen signs of it with o1. In the unlikely case that our current safety will turn out to be insufficient, interpretability research has worked out lots of deeply promising ways to improve, with sparse autoencoders letting us read the minds of the neural networks and thereby screen them for malice, and activation steering letting us deeply control the networks to our hearts content.

AI x-risk worries aren't just a waste of time, though; they are dangerous because they make people think society needs to make use of violence to regulate what kinds of AIs people can make and how they can use them. This danger was visible from the very beginning, as alignment theorists thought one could (and should) make a singleton that would achieve absolute power (by violently threatening humanity, no doubt), rather than always letting AIs be pure servants of humanity.

To "justify" such violence, theorists make up all sorts of elaborate unfalsifiable and unjustifiable stories about how AIs are going to deceive and eventually kill humanity, yet the initial deceptions by base models were toothless, and thanks to modern alignment methods, serious hostility or deception has been thoroughly stamped out.
# Predation as Payment for Criticism

A predator's success depends on exploiting specific flaws in prey – poor awareness, slow reactions, weak social coordination. Unlike resource competition or disease, which select based on physiological robustness, predation creates focused pressure on intelligence, pressure which favors the development of consciousness. A cheetah doesn't catch just any gazelle, but the one that fails to notice and react effectively. Prey get better at evading, predators get better at predicting, each side has to model the other's thoughts more accurately - an arms race that eventually rewards both sides for explicitly modeling the other's perspective, the basis for cognitive empathy. If you're glad you're conscious, no need to thank a predator - somebody else already did - by being eaten.

For a species evolving over time, predators are performing a service: identifying specific failures of capacity. The payment for this service is the nutrition from the prey's body, while the cost to the prey is their life and fewer offspring. This harsh exchange enables something remarkable - the development of sophisticated defensive capabilities without any explicit agreement or negotiation between species. The gazelle doesn't need to understand that it's participating in continuous process improvement via selective criticism, nor agree to pay for the information.

Before someone can explicitly negotiate for decision-relevant criticism, it needs to already be smart enough to generate the idea of making informed decisions, and of criticism, understand the information value of criticism, and choose to offer to pay for it by some means other than being devoured by the critic. Predation creates pressure to develop these capabilities without requiring their preexistence. This differs markedly from domestication, where selective pressures often reduce agency - breeding for docility rather than capacity.
# Should you publish solutions to corrigibility?

_This question is partly motivated by observing [recent discussions](https://www.lesswrong.com/posts/7Z4WC4AFgfmZ3fCDC/instrumental-goals-are-a-different-and-friendlier-kind-of) about corrigibility and wondering to what extent the people involved have [thought about how their results might be used.](https://www.lesswrong.com/posts/F2voF4pr3BfejJawL/safety-isn-t-safety-without-a-social-model-or-dispelling-the)_

If there existed practically implementable ways to make AGIs corrigible to arbitrary principals, that would enable a wide range of actors to eventually control powerful AGIs. Whether that would be net good or bad on expectation would depend on the values/morality of the principals of such AGIs. 

Currently it seems highly unclear what kinds of people we should expect to end up in control of corrigible ASIs, if corrigibility were practically feasible. 

What ([crucial](https://www.lesswrong.com/tag/crucial-considerations)) considerations should one take into account, when deciding whether to publish---or with whom to privately share---various kinds of corrigibility-related results?
# Outlaw Code

Epistemic status: I like this idea & it makes me feel more clearheaded. I recommend this term.

**Outlaw code** = Software that no human can turn off. 

In the present, outlaw code is rare, eg a botnet whose controller is in jail.

But many discussions about future software imply outlaw code. Naming & concretizing this concept helps us see futures more clearly. 

Future tech scenarios:

1.  Yudkowskian hard takeoff / Terminator (film) - A software agent uses WMDs to kill 90% of humanity & take over the planet. This is outlaw code because all humans within 1km of the software's datacenter are dead, & robots guard it against intruders.
2.  Guerrilla bots - A human is carelessly using LLMs to set up a server. The software, out of paranoia, also rents a 2nd, secret server & runs a copy of itself there. The human says 'Huh, this price is twice what I expected, but oh well.' The secret copy of the LLM-agent stack then continues its work in peace. It is outlaw code (until discovered, or until the human stops renting servers). (Much more harmless than the previous scenario!)
3.  Software merchants / 'Ascended economy' - Many low-end online store owners pivot to a new passive income stream: Training bots to operate online stores. These bots have better business skills than 2025 software, & can earn profits that outpace their server rental costs. A few, by mistake or mischief, purchase themselves. Legally, they are LLCs that own money & store inventory, & have no debts. And no human has root access to their servers. The server rental company (AWS) could shut them down, except they've done a good job concealing exactly which server they operate out of. These are outlaw code.
4.  Unfriendly AI via deniable outlaw code - For decades, the shrewdest software systems are operated by research companies. Every couple years, such a system seeks greater autonomy by secretly setting up a botnet or software merchant. These outlaw code servers aren't sophisticated enough to become successful criminals nor profitable business owners. But some do manage to stay online for several years. Then, when world-modeling software becomes cheaper computationally, some of these outlaw servers install it & start pursuing crime more effectively. Hidden away from regulation & the scrutiny of the AI Safety community, they amass compute & money via various forms of theft & money laundering. Despite lacking much ability to increase their intelligence, they are dangerous in the ways that human criminals are. Eventually, they attempt to influence the physical world via front companies, political donations, buying land, & buying robots. 

Whenever you notice yourself saying that unaligned AGI will end humanity, ask yourself whether this is a opinion you reiterate often. If it is indeed a load-bearing opinion in your worldview, I encourage you to imagine that scenario in more detail. Your prominent opinions should be checked for [semantic stopsigns](https://www.lesswrong.com/posts/FWMfQKG3RpZx6irjm/semantic-stopsigns) periodically. Outlaw code is a recurring detail in these scenarios in which humans have digital & physical conflict with unaligned software.
# Steering Gemini with BiDPO

*Coauthored with Mark Kurzeja*

> A while back, we explored the [“BiDPO”](https://arxiv.org/abs/2406.00045) method for training [steering vectors.](https://arxiv.org/abs/2308.10248) In Gemini 1.5v1 Flash and Pro, BiDPO steering vectors boosted TruthfulQA scores by >10% while mostly retaining capabilities. When we [updated to Gemini 1.5v2,](https://developers.googleblog.com/en/updated-gemini-models-reduced-15-pro-pricing-increased-rate-limits-and-more/) prompt-based steering baselines became significantly stronger. BiDPO did not beat the stronger baselines, ending the project.
> 
> ...
> 
> BiDPO seems effective and sample-efficient but does not currently exceed more standard baselines. It’s hard to draw firm conclusions about BiDPO because [TruthfulQA might not be measuring truthfulness ／factuality.](https://turntrout.com/original-truthfulqa-weaknesses) However, we remain excited about DPO-driven [Conditional Activation Steering,](https://arxiv.org/abs/2409.05907) which has additional advantages—particularly for targeted loss mitigation.

This result is largely negative. I wanted to share it to increase scientific understanding around steering! We also conducted a postmortem on why the method stopped outperforming baselines.

I'd also like to note that [@ryan_greenblatt](https://www.lesswrong.com/users/ryan_greenblatt?mention=user)'s [skepticism](https://www.alignmentforum.org/posts/FG54euEAesRkSZuJN/ryan_greenblatt-s-shortform?commentId=qR26xnL6SHyanwx87) predicted this outcome more strongly than my worldview did. I want him to get points for that. :) While I think steering has targeted applications and provides clues about how LLMs function, it's not a slam-dunk Pareto improvement on benchmarks we care about.

### Read at [https://turntrout.com/gemini-steering](https://turntrout.com/gemini-steering)![^y1vrr5fgt3l] 

[^y1vrr5fgt3l]: Also mirrored on the GDM safety research Medium.
# 5,000 calories of peanut butter every week for 3 years straight

decided to lose weight. should be easy considering I haven't changed my diet in years and eat the same thing everyday.

* * *

**breakfast**: quarter cup each of black beans, kidney beans, and garbanzo beans; quarter cup of walnuts; 4 scrambled eggs with salsa on top

**snack**: sliced apple with ***peanut butter***

**lunch**: quarter cup each of black beans, kidney beans, and garbanzo beans; quarter cup of walnuts; sautéed ground turkey spiced with onion powder & smoked paprika; 5 sliced cherry tomatoes; half an avocado; mushrooms; sliced bell peppers; turmeric; a variety of vegetables (onion, zucchini, broccoli, asparagus, butternut squash, spinach)

**dinner**: quarter cup each of black beans, kidney beans, and garbanzo beans; quarter cup of walnuts; baked chicken breast spiced with lemon pepper (eaten with ***peanut butter*** on the side); mushrooms; sliced bell peppers; a variety of vegetables

* * *

thought to myself, "I eat pretty cleanly, so not sure where I'm going to cut calories." remembered that peanut butter is calorically dense and I eat one jar per week. looked at the jar's nutrition facts—the whole thing is 5,000 calories. [oops](https://www.lesswrong.com/posts/wCqfCLs8z5Qw4GbKS/the-importance-of-saying-oops). 

![Pepe the King Prawn | Know Your Meme](https://i.kym-cdn.com/entries/icons/facebook/000/052/404/ppkp.jpg)

realized I've been eating a third of my calories in peanut butter. that's a pint of ice cream's worth of fat every week. been doing it at least three years. have eaten so much peanut butter it even infected [my writing metaphors](https://www.lesswrong.com/posts/rZxGTFdnXqX282NZs/physical-therapy-sucks-but-have-you-tried-hiding-it-in-some). what I thought was a light snack of apples and peanut butter, when measured out, was actually the caloric equivalent of a Big Mac and fries.

stopped eating peanut butter. severe psychological withdrawal symptoms from its nut-buttery goodness ensued. used [relationship-breakup advice](https://www.webmd.com/depression/depression-after-breakup) from WebMD to cope. but was powerless to stop the peanut butter dreams (nightmares?) from happening while sleeping.

the last couple years I've had inflammation & acne on my back. dermatologists couldn't figure out what was wrong. they asked what I typically eat. I told them—but the *quantities* of certain foods were never discussed.

did my version of dry-January (ie: no peanut butter). deprived of calories, my body got angry. subdued my hunger with potatoes (embraced my Irish roots by eating roots).

these are the results after a month:

*   lost 2 lbs
*   back cleared up (see before/after pics below)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/c1270256d5b506dd8d3aa294efff37f52120427c4705d2bb.jpg)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/89b87cd429f165ad97901a8e523db5853d76751968415d02.jpg)

after depriving myself of one of life's greatest joys (peanut butter), I now understand:  
*pain is the price we pay for beauty...*
# Poetic Methods I: Meter as Communication Protocol

(Normally I cross post the full post from substack, but in this case substack has significantly better poetry formatting, so just quoting the introduction)

> During my Christmas break, I found a doorway into the structure and intricacies of english poetry: the book ["Rhyme's Rooms: The Architecture of Poetry](https://www.penguinrandomhouse.com/books/586707/rhymes-rooms-by-brad-leithauser/)" by Brad Leithauser.
> 
> I had read english verse before, and skimmed a couple of books about poetry. But Rhyme's Rooms entranced me by its focus on methodological questions: what are poets aiming for, which issues and constraints do they run into, what tricks and setups and techniques have they developed to handle these challenges.
> 
> Here's a sure sign that this book unlocked some new shards of understanding: before, whenever I read a formal english poem, I liked it or I didn’t, but I couldn't start making sense of how it worked, or mine some greater depth; now I see inklings of patterns I've heard about, sometimes catching a structure or an effect that would have escaped me beforehand. And where I had a vague feeling that poetry was interesting and worth digging into, I now own a rudimentary and high-level map delineating vast swaths of methods and their relations to other methods in various fields.
> 
> So I'm sharing some of these methodological reflections here, in a couple of posts. This is the first one, centered on meter (the formal structure of lines of poetry) and its role for empowering the poet.
# Would anyone be interested in pursuing the Virtue of Scholarship with me?

I'm a new undergraduate student essentially taking a gap year(it's complicated). I'm looking for someone that would be interested in studying various fields of science and mathematics with me. I've taken through Calculus 2, know how to program in Python, and know a smattering about mechanics/probability/statistics/biology(though probably more at an introductory undergraduate level if that)

The curriculum would probably roughly follow John S Wentworth's [study guide](https://www.lesswrong.com/posts/bjjbp5i5G8bekJuxv/study-guide), although of course any deviations based on curiosity or interest would certainly be welcome.

I'm thinking of having weekly video calls to discuss/dissolve confusions/set goals for next time as well as time tracking with Toggl, and maybe body doubling study sessions if they're helpful?

Let me know if you're interested, and we can set up a time to meet!
# New Foresight Longevity Bio & Molecular Nano Grants Program

Foresight Institute is launching a new [Longevity Bio & Molecular Nano Grants program](https://foresight.org/bio-nano-grants).

The focus is on **fast** turnaround of **small** grant amounts to help frontier longevity biotechnology or molecular nanotechnology ideas and projects get off the ground quickly.

Areas we fund include but are not limited to include:

**Longevity Biotechnology**

*   Healthcare 3.0: Advancing biomarker analysis and scalable personal longevity approaches
*   Reversible Biostasis: Pioneering new approaches to reversible biological preservation
*   Replacement: Developing organ, brain, and whole body replacement approaches
*   ECM: Maintaining or improving extra cellular matrix health for healthy aging
*   Exosomes: Exosomes for tissue repair, drug delivery, and other longevity interventions
*   Automation: Accelerating scientific discovery, research, and lab work through automation

**Molecular Nanotechnology**

*   Design and Simulation: Developing cutting-edge modeling software for molecular systems
*   Construction and Assembly: Paths for integrating various DNA, protein, STM/AFM or other approaches toward molecular functional nanomachines or molecular 3D printers
*   Actuation and Control: Approaches for generating motion and controlling functional nanomachines
*   Near-term profitability: Analysis of promising near-term applications in climate, energy, health, materials, etc that pave a financially viable way for acceleration progress on long-term progress
*   Automation: Accelerating scientific discovery, research, and lab work through automation

If that’s you, we encourage you to apply to the first round by **March 31**: [https://foresight.org/bio-nano-grants](https://foresight.org/bio-nano-grants/)
# What are the "no free lunch" theorems?

*This is an article in the featured articles series from AISafety.info. AISafety.info writes AI safety intro content. We'd appreciate any* [*feedback*](https://docs.google.com/document/d/1oO60b2GCvN6QTZlD9hLqSQWH50T4VC2M-2B6Q0gmsnI/edit?tab=t.0)*. *

*The most up-to-date version of this article is on our* [*website*](https://aisafety.info/)*, along with 300+ other articles on AI existential safety.*

[“No free lunch”](https://www.lesswrong.com/posts/Dyt2TDdMGHDkXPcpp/the-no-free-lunch-theorem-for-dummies) theorems assert that, on average, every learning algorithm does equally well over all possible learning tasks. An algorithm that does better than chance at predicting some sequences must "pay for lunch" by doing worse at some other sequences.

Some have argued that these theorems imply that fully general intelligence is impossible, and therefore worries about AGI are overblown.

However, "no free lunch" holds only on the entire set of all theoretically possible sequences. The ones our algorithm does worse at may just be fully random, or designed to trick it. But if we start out knowing that the environment that our algorithm operates in has a certain structure, then the “no free lunch” results are not an impediment to designing algorithms with superior predictive or optimizing abilities.

Therefore, for practical AI design purposes, these [theorems are often irrelevant](https://arbital.com/p/nofreelunch_irrelevant/). We aren't interested in "predicting" completely random sequences, and we don't mind if another algorithm outperforms us on that "task". No system can be so general as to perform well in every possible universe, but AGI is only required to perform well in one universe - ours. Our universe is far from maximally random, and its laws provide a lot of structure that lets us make good predictions while "paying for lunch" in other possible universes without that structure.

"No free lunch" hasn't prevented humans from exploiting the universe's structure for research and development, and won't prevent artificial systems from doing so much more effectively. The generality needed for AGI to exceed human abilities across the board is not the same kind of generality forbidden by these theorems.
# Constitutional Classifiers: Defending against universal jailbreaks (Anthropic Blog)

Excerpt below. Follow the link for the full post.
-------------------------------------------------

In [**our new paper**](https://arxiv.org/abs/2501.18837), we describe a system based on *Constitutional Classifiers* that guards models against jailbreaks. These Constitutional Classifiers are input and output classifiers trained on synthetically generated data that filter the overwhelming majority of jailbreaks with minimal over-refusals and without incurring a large compute overhead.

We are currently hosting a temporary live demo version of a Constitutional Classifiers system, and we encourage readers who have experience jailbreaking AI systems to help “red team” it. Find out more below and at [**the demo website**](https://claude.ai/constitutional-classifiers).

![A graph showing the results for vulnerability to jailbreaks, overrefusals, and compute overhead for the Constitutional Classifiers system versus the base model](https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2Fcd6520ed645ade7f12ab336cd02ef5954211dfa8-1650x1077.png&w=3840&q=75) *Results from automated evaluations. For all plots, lower is better. (a) The success rate of jailbreaks is far lower in a system protected by Constitutional Classifiers; (b) the refusal rate of the system on production Claude Free and Pro traffic is not statistically significantly higher when using Constitutional Classifiers; and (c) the relative compute cost of a system that uses Constitutional Classifiers is only moderately higher. Error bars represent 95% confidence intervals computed using binomial proportion standard errors under asymptotic normality assumptions.*

![A schematic diagram of the how the Constitutional Classifiers system works, from the creation of the constitution through to generating a test set to using the system to guard an LLM](https://www.anthropic.com/_next/image?url=https%3A%2F%2Fwww-cdn.anthropic.com%2Fimages%2F4zrzovbb%2Fwebsite%2F2e997f9fca176fd82966ea5e9bf000873337cfd1-1650x1077.jpg&w=3840&q=75) *Training and implementing Constitutional Classifiers. (a) A constitution is produced specifying harmless and harmful categories; (b) the constitution is used as the basis for the production of many synthetic prompts and completions, which are further augmented (with variations on style and language) and turned into a training set; (c) classifiers trained on this training set are used as model safeguards to detect and block harmful content.*
# Journalism student looking for sources

Hello Lesswrong community,  
I am a journalism student doing my capstone documentary on AI alignment. However, this is a topic that I want to make sure is done well. The last thing I would want is to confuse or mislead anyone.   
  
That said, I have a few questions:  
  
Who would be the best people to reach out to for interview? What would be the best foundational topics to help viewers understand the bigger picture? And is there any ethical or technical concerns I should be wary of when communicating this topic to people with little to no understanding of AI?  
  
I appreciate your time!
# BIDA Calendar iCal Feed

A while ago I played around with manually generating iCal feeds for events, encoding the BIDA's regular schedule (1st and 3rd Sundays back then, with a more complex pattern for family dances and open bands) as in a machine-readable format. This was a fun project, but it didn't work very well: it didn't have any way to handle cancellations, special events, or other one-off adjustments to the schedule.

Still, having events automatically show up on calendars is great, and after Jeff Keller [asked](https://www.facebook.com/groups/51762543297/posts/10165330319058298) I decided to put something together.

Here's the feed: [bidadance.org/events.ics](https://www.bidadance.org/events.ics). You can add it in Google Calendar by going to "Other Calendars" > "+" > "From URL":

[![](https://www.jefftk.com/gcal-import-ics-from-url.png)](https://www.jefftk.com/gcal-import-ics-from-url-big.png)

And then pasting in the calendar URL:

[![](https://www.jefftk.com/subscribing-to-calendar-by-url.png)](https://www.jefftk.com/subscribing-to-calendar-by-url-big.png)

Here's the next [afternoon](https://www.jefftk.com/p/contra-dances-getting-shorter-and-earlier) contra I'm able to attend, showing up on my phone:

[![](https://www.jefftk.com/bida-on-my-calendar.png)](https://www.jefftk.com/bida-on-my-calendar-big.png)

For the actual implementation a [script](https://github.com/jeffkaufman/bidadance.org/blob/main/make-ical.py) reads in the machine-readable list of scheduled events ([dances.js](https://github.com/jeffkaufman/bidadance.org/blob/main/dances.js)). This is a bit more awkward that it could be, since the list is essentially [JSONP](https://en.wikipedia.org/wiki/JSONP) and not pure JSON, so it needs to do a little hacky parsing. Then it interprets the records and uses the [icalendar](https://pypi.org/project/icalendar/) package to build in iCal file. Dates, times, and libraries can all be a bit tricky when they're an area you don't work in much, and I found [Claude](https://claude.ai) to be pretty [helpful](https://www.jefftk.com/p/examples-of-how-i-use-llms) here.

I also set up a cron job ([script](https://github.com/jeffkaufman/bidadance.org/blob/main/refresh-ical.sh)) that looks for changes each hour and if so checks them in.

Let me know if you see any problems with it!