Harley-ml/lesswrong huggingface dataset:
# Former OpenAI Superalignment Researcher: Superintelligence by 2030

> The AGI race has begun. We are building machines that can think and reason. By 2025/26, these machines will outpace many college graduates. By the end of the decade, they will be smarter than you or I; we will have superintelligence, in the true sense of the word.

In the link provided, Leopold Aschenbrenner explains why he believes AGI is likely to arrive within the decade, with superintelligence following soon after. He does so in some detail; the website is well-organized, but the [raw pdf](https://situational-awareness.ai/wp-content/uploads/2024/06/situationalawareness.pdf) is over 150 pages.

Leopold is a former member of OpenAI's Superalignment team; he was fired in April for allegedly leaking company secrets. However, he contests that portrayal of events [in a recent interview with Dwarkesh Patel](https://www.dwarkeshpatel.com/p/leopold-aschenbrenner#%C2%A7what-happened-at-openai), saying he leaked nothing of significance and was fired for other reasons.[^yamtae3lfq]

However, I am somewhat confused by the new business venture Leopold is now promoting, an ["AGI Hedge Fund"](https://www.dwarkeshpatel.com/p/leopold-aschenbrenner#%C2%A7launching-an-agi-hedge-fund) aimed at generating strong returns based on his predictions of imminent AGI. In the Dwarkesh Patel interview, it sounds like his intention is to make sure financial resources are available to back AI alignment and any other moves necessary to help Humanity navigate a turbulent future. However, the discussion in the podcast mostly focuses on whether such a fund would truly generate useful financial returns.

If you read this post, Leopold[^66aj61hm0gd], could you please clarify your intentions in founding this fund?

[^yamtae3lfq]: Specifically he brings up a memo he sent to the old OpenAI board claiming OpenAI wasn't taking security seriously enough. He was also one of very few OpenAI employees not to sign the letter asking for Sam Altman's reinstatement last November, and of course, the entire OpenAI superaligment team has collapsed for various reasons as well. 

[^66aj61hm0gd]: Leopold does have a LessWrong account, but hasn't linked his new website here after some time. I hope he doesn't mind me posting in his stead.
# Startup Stock Options: the Shortest Complete Guide for Employees

This is a short, but comprehensive, guide to stock options, with specific example outcomes for employees and a sprinkle of Bayesian modeling of your chance to cash out with at least $1M.

Unlike other guides, this is not a stream of financial terminology. Instead, it is specific and answers the real questions with concrete examples. "What happens if I have vested and unvested stock options and the company is acquired?" and the like. 

Most of all I tried to provide tools for tackling the main question: "Is it worth it?"

Topics covered: basics, how startup exits work, possible outcomes for stock options holders, taxes, dilution, how long you will wait, and how lucky do you have to be to make money. And, of course, how to lose all of your money. There are so many ways.

This guide exists because I followed the advice of a guy I met at the Lisbon rationality meetup. The advice was to take all your interests seriously and write out ideas to completion. The topic was mildly interesting to me and I went a little bit too deep in the rabbit hole. 

Hope you find this useful!
# Scaling and evaluating sparse autoencoders

\[[Blog](https://openai.com/index/extracting-concepts-from-gpt-4/)\] \[[Paper](https://cdn.openai.com/papers/sparse-autoencoders.pdf)\] \[[Visualizer](https://openaipublic.blob.core.windows.net/sparse-autoencoder/sae-viewer/index.html)\]

Abstract:

> Sparse autoencoders provide a promising unsupervised approach for extracting interpretable features from a language model by reconstructing activations from a sparse bottleneck layer.  Since language models learn many concepts, autoencoders need to be very large to recover all relevant features.  However, studying the properties of autoencoder scaling is difficult due to the need to balance reconstruction and sparsity objectives and the presence of dead latents.  We propose using k-sparse autoencoders \[Makhzani and Frey, 2013\] to directly control sparsity, simplifying tuning and improving the reconstruction-sparsity frontier. Additionally, we find modifications that result in few dead latents, even at the largest scales we tried.  Using these techniques, we find clean scaling laws with respect to autoencoder size and sparsity.  We also introduce several new metrics for evaluating feature quality based on the recovery of hypothesized features, the explainability of activation patterns, and the sparsity of downstream effects. These metrics all generally improve with autoencoder size.  To demonstrate the scalability of our approach, we train a 16 million latent autoencoder on GPT-4 activations for 40 billion tokens.  We release code and autoencoders for open-source models, as well as a visualizer.
# Offering Completion

One verbal approach I often find useful with little kids is to offer them the opportunity to complete a phrase. Compare:

> Parent: It's time to brush teeth!
> 
> Toddler: I no want brush teef!

vs:

> Parent: It's time to brush...
> 
> Toddler: Teef!

Similarly, "the dirty tissue goes in the...", "when we get home it will be time to put pee in the...", "it's time to eat...", etc. My experience is that, at the right age, kids just really love getting to complete things, and this somehow spills over into their attitude towards the thing they're completing. Or having it come from their mouth somehow manufactures buy-in, or skips the normal opportunity for them to (as is also common at this age) say no?

One nice safeguard this has is that while it's mildly manipulative, once they lose their joy at completing things it stops working. So it doesn't have the risk that you'll keep using it at an age where (in my opinion) manipulating kids starts to be a problem.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02ZWt5aRVthywBEUGjDyCd1ni6kH5t22sQzfmjRkXz3KfefRTLx65BCTCove9SGhgzl), [mastodon](https://mastodon.mit.edu/@jefftk/112572771614795639)*
# Question about Lewis' counterfactual theory of causation

In reading the [SEP entry on counterfactual theories of causation](https://plato.stanford.edu/entries/causation-counterfactual/), I had the following question occur, and I haven't been able to satisfactorily resolve it for myself.

An event *e* is said to causally depend on an event *c* if and only if *e* would occur if *c* were to occur and *e* would not occur if *c* were not to occur.

The article makes a point of articulating that causal dependence entails causation (if *e* causally depends on *c*, *c* is a cause of *e*) but not vice versa. It then defines a causal chain as a fine sequence of events *c*, *d*, *e*,... where *d* causally depends on *c*, *e* on *d*, and so on, before defining *c* to be a cause of *e* if and only if there exists a causal chain leading from *c* to *e*.

What I'm having trouble with is understanding how *c* can cause *e* according to the given definition without *e* causally depending on *c*. If there's a causal chain from *c* to *d* to *e*, then *d* causally depends on *c*, and *e* causally depends on *d*, so if *c* were to not occur, *d* would not occur, and if *d* were to not occur, *e* would not occur. But doesn't this directly entail that if *c* were to not occur, then *e* would not occur and therefore that *e* causally depends on *c*?

So how can *c* cause *e* according to the definition without *e* causally depending on *c*??
# Sev, Sevteen, Sevty, Sevth

I don't like the number seven. Well, really the name of the number seven. All the other single digit numbers are single syllable, and seven has to go and take two. Seventy and seventeen have the same problem. What can we do?

I think the two main candidates are "sev" (dropping the second syllable) and "sen" (dropping the first coda and second onset). While I find "sen" slightly nicer on the tongue, I think "sev" is more promising because it feels like a better short form.

It feels like we ought to be able to switch to calling it "sev", where some people just start saying that and other people understand them? I've been playing around it, but every time I do my toddler Nora laughs at me as if I'm being ridiculously over the top: "you said sev!!" Does not bode well for a [low-key migration](https://www.jefftk.com/p/experience-switching-to-right-shoulder-round).

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02F7gTgRWSMkqSubLUU67MYxY24CFKwE4uauHBP4Mb5Z98b4AodSc3apVDKz4Kt5r5l), [mastodon](https://mastodon.mit.edu/@jefftk/112578614355670700)*
# Why don't we just get rid of all the bioethicists?

I've had a few vague answers before, but as I was considering how no one was allowed to do human challenge trials during COVID, I thought I'd ask:

Can anyone come up with an example where a bioethicist actually improved the world?

I'm looking for specific cases, not a general "nazi science was horrible" sort of thing. Is there a record of horrifying things that IRBs have prevented?

If, tomorrow, all bioethicists were fired and IRBs basically disappeared, would that unleash a wave of evil scientists who are barely kept in check now, or would everything be completely fine (even better, possibly)?
# Status quo bias is usually justified

Generally, we know more about the status quo then about anything else.

*   We know that we can live in the current climate, we don’t know that about any other climate.
*   We know that society functions with current laws and norms, we don’t know that about any other set of laws.

Often, we are adapted to the status quo and the status quo is adapted to us.

*   We and the plants and animals we depend on evolved to live in the current the global climate. Many of our city are built on the coast, our population is distributed in a way that can be supported by agriculture.
*   Our laws are the result of millenia of trial and error.

Often, the status quo is a compromise between different groups, individuals or even desires within an indivdual.

The test for status quo bias is reversal. If we oppose a change in one direction, do we also oppose change in the other direction? We’re concerned about global warming, and we were also concerned about [global cooling](https://en.wikipedia.org/wiki/Global_cooling). Any change in the climate has unpredictable consequences.
# What if a tech company forced you to move to NYC?

It’s interesting to me how chill people sometimes are about the non-extinction future AI scenarios. Like, there seem to be opinions around along the lines of “pshaw, it might ruin your little sources of ‘meaning’, Luddite, but we have always had change and as long as the machines are pretty near the mark on rewiring your brain it will make everything amazing”. Yet I would bet that even that person, if faced instead with a policy that was going to forcibly relocate them to New York City, would be quite indignant, and want a lot of guarantees about the preservation of various very specific things they care about in life, and not be just like “oh sure, NYC has higher GDP/capita than my current city, sounds good”.

I read this as a lack of engaging with the situation as real. But possibly my sense that a non-negligible number of people have this flavor of position is wrong.
# Demystifying "Alignment" through a Comic

*I am keeping this comic online as a record of an earlier perspective. My current work in fixed point logic and proof theory has led me to reject several foundational assumptions in mainstream alignment and interpretability research. This does not imply that AI poses no risks (it implies nothing about AI).*

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b852531e41838a7f801056502d56ab4867502e58318ff9f2.jpeg)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8096ad91b6cb849c115fd2d5176f2a7fdde921699289efbb.jpg)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/2f156239dbe5f08305d7b02c519eb3cfa0a24a8b050505f7.jpg)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/9d09eb0836cb26943bea9e36437b8226010c76e8c20da7d7.jpg)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d7a6416b2a3b8c6288b5b8a45e4eae6b4d3669fef4934e02.jpg)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/49976e791b6cb5a16ad78053562c306169a31cb5ff242c1e.jpg)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/0741020071e247206153ad89e69a8156c7bd90ec137e717e.jpg)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e880af6c171038597ff31f5c07e1f7303004c6ca48eaaf0d.jpg)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/1ec07003dadfcb8df62bd3a7a8e0762d441e7a369653eb91.jpeg)
# Two Family Dance Flyers

I'm going to be calling [another](https://www.jefftk.com/p/calling-my-first-family-dance) family dance [in a week](https://www.facebook.com/events/443304658639233), and Lily and Anna wanted to make flyers to advertise it. I wrote out a sheet with the key details they might want to include:

[![](https://www.jefftk.com/bida-family-dance-key-info.jpg)](https://www.jefftk.com/bida-family-dance-key-info-big.jpg)

Lily wanted to do hers on the computer, and it ended up being primarily about learning [Inkscape](https://inkscape.org/):

[![](https://www.jefftk.com/2024-06-16--family-dance.png)](https://www.jefftk.com/2024-06-16--family-dance-big.png)

Anna did hers by hand, and was very into the drawing:

[![](https://www.jefftk.com/anna-bida-family-dance-flyer.jpg)](https://www.jefftk.com/anna-bida-family-dance-flyer-big.jpg)
# Good ways to monetarily profit from the increasing demand for power?

There have been several good posts about how to profit from a world of AI takeoff.

However, when it comes to individual investment recommendations, most of them are for direct AI companies or builders of GPUs and compute infrastrucutre

Recently, it's become clear to me that power will be much more of a bottleneck than GPUs (and therefore even more valuable).

What's the best way to profit from this insight?  Stock picks? Futures contracts? Something else?
# Open Thread Summer 2024

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the [Highlights from the Sequences](https://lesswrong.com/highlights), a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/faq#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/faq). If you want to orient to the content on the site, you can also check out the [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/tag/open-threads?sortedBy=new). The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# Underrated Proverbs

Some proverbs are actively suspicious, like “Don’t judge a book by its cover” or “No pain, no gain.” Others have an opposite proverb that’s similarly common and reasonable.

*   “Two heads are better than one” vs “Too many cooks spoil the broth”
*   “Honesty is the best policy” vs “What they don’t know won’t hurt them”
*   “Better safe than sorry” vs “Nothing ventured, nothing gained”

But the four below I use often:

1.  **The best defense is a good offense.** This one even has a [Wikipedia page](https://en.wikipedia.org/wiki/The_best_defense_is_a_good_offense) that references Washington, Mao, Machiavelli, Sun Tzu, and “sports such as football and basketball” (citing a dead link to “diamondbackonline.com”). I can’t think of an opposite adage—maybe “prevention is better than cure.”
2.  **It's a dog-eat-dog world.** This one isn’t true scientifically. [“Two out of eleven dogs consistently refused to eat dog flesh. Eight dogs ate the raw flesh on more than half of the trials and five of these accepted it all of the time. All of the animals accepted the flesh after it had been cooked.”](https://psycnet.apa.org/record/1933-01816-001) But metaphorically it’s often true because people are driven by self-interest. So often a situation lends itself to the remark that “it’s a dog-eat-dog world.”
3.  **The truth hurts!** (Say this one the one the way Trump says “Sad!”) Often the truth does hurt. Maybe an opposite is “Ignorance is bliss” or “What you don’t know can’t hurt you” but these are less often useful principles. “The truth hurts” can be said often since a lot of cynical beliefs are true.
4.  **The people cry out for a strong leader.** I don’t think this is a real adage but I often say it. It can be used in many situations where people lack agency because no one is giving them specific instructions on what to do.
# OpenAI appoints Retired U.S. Army General Paul M. Nakasone to Board of Directors

> Today, **Retired U.S. Army General Paul M. Nakasone** has joined our Board of Directors. A leading expert in cybersecurity, Nakasone’s appointment reflects OpenAI’s commitment to safety and security, and underscores the growing significance of cybersecurity as the impact of AI technology continues to grow.
> 
> As a first priority, Nakasone will join the Board’s [Safety and Security Committee](https://openai.com/index/openai-board-forms-safety-and-security-committee/), which is responsible for making recommendations to the full Board on critical safety and security decisions for all OpenAI projects and operations.

Whether this was influenced by Aschenbrenner's [Situational Awareness](https://situational-awareness.ai) or not, it's welcome to see OpenAI emphasizing the importance of security. It's unclear how much this is a gesture vs reflective of deeper changes.
# Thoughts on Francois Chollet's belief that LLMs are far away from AGI?

Dwarkesh had a podcast recently with Francois Chollet (creator of Keras)

He seems fairly skeptical we are anywhere near AGI with LLMs. He mostly bases his intuition that LLMs fail on OOD tasks and don't seem to be good at solving simple abstract reasoning problems he calls the ARC challenge. It seems he thinks system 2 thinking will be a much harder unlock than people think and that scaling LLMs will go nowhere. In fact he goes so far as to say the scaling maximalists have set back AGI progress by 5-10 years. Current LLMs to him are just simply information retrieval databases.

He, along with the CEO of Zapier, have launched a 1 million dollar prize to beating the ARC bench marks, which are apparently hard for LLMs. I didn't believe it at first, given how easy they seem, but barely any progress has been made on the ARC bench marks in the last 4 years. In retrospect, it's odd that so many existing benchmarks rely heavily on memorized knowledge, and the ARC results check out with LLMs being bad at playing sudoku (so maybe not *that* surprising). 

This seems to be in contradiction with what people on this site generally think. Is the disagreement mainly that system 2 thinking will be a relatively fast unlock (this is my take at least[^u5uimv2fka]) whereas Francois thinks it will take a long time? 

Or does it go deeper?   
  
 

[^u5uimv2fka]: Personally my intuition that LLMs are world modelers and system 2 thinking will be a relatively simple unlock as they get better at modeling the world.
# ACX Montreal Meetup June 22nd 2024

*Note the second announcement at the end of this post.*

Come on out to the ACX (Astral Codex Ten) Montreal Meetup! This week, we're talking about Jon Blow's talk "Preventing the Collapse of Civilization".

### Reading/Viewing Recommendation(s):

Two shorter highlights: [https://www.youtube.com/watch?v=FeAMiBKi_EM](https://www.youtube.com/watch?v=FeAMiBKi_EM) and [https://www.youtube.com/watch?v=fEn2HPGXwj8](https://www.youtube.com/watch?v=fEn2HPGXwj8)

Summary: Jon Blow talks about his hypothesis that, as technology grows in complexity (in all areas, but he dives deeper on software as a specific example), deep knowledge of that technology among top experts gets harder and harder to maintain, which can lead to a slow decline in a civilization's ability to improve upon and eventually even maintain infrastructure based on that tech.

*Topic suggestion form* [*here*](https://forms.gle/ddW6jcp9ci9oq8oq8)*.*

**Venue:** L'Esplanade Tranquille, 1442 Clark, possibly second floor balcony, possibly in the outdoor area.   
**Date & Time: **Saturday, June 22nd, 2024, 1:00PM.

**Please RSVP by clicking "Going" at the top of this post**. It is not mandatory, but it helps with planning the event and encourages others to come.

I (probably) won't be there and Massimo is running the event, so contact Massimo on Discord @massimog if you have trouble finding us or any other issues.

Please also **join the** [**mailing list**](http://eepurl.com/io5vZM) **and our** [**Discord server**](https://discord.gg/K8gMNzqPVG) if you haven't already. We host biweekly ACX Montreal meetups (and more), so join us if you don't want to miss any of them!

Second announcement
-------------------

As a second announcement, on Saturday June 15th, at 5PM, Massimo is hosting a board game night at his place! One of his conditions is to not *only* play Among Us. DM him on Discord for the apartment details if you're interested.
# Yann LeCun: We only design machines that minimize costs [therefore they are safe]

Just a tweet I saw:

> **Yann LeCun**
> 
> Doomers: OMG, if a machine is designed to maximize utility, it will inevitably diverge ![😱](https://abs-0.twimg.com/emoji/v2/svg/1f631.svg) 
> 
> Engineers: calm down, dude. We only design machines that minimize costs. Cost functions have a lower bound at zero. Minimizing costs can't cause divergence unless you're really stupid.

Some commentary:

I think Yann LeCun is being misleading here. While people intuitively think maximization and minimization are different, the real distinction is between convex optimization (where e.g. every local optimum is a global optimum) and non-convex optimization. When dealing with AGI, typically what people hope to solve is non-convex optimization.

Translating back to practical matters, you are presumably going to end up with some cost functions where you don't reach the lower point of zero, just because there are some desirable outcomes that require tradeoffs or have resource limitations or similar. If you [backchain these costs through the causal structure of the real world, that gives you instrumental convergence for standard reasons](https://www.lesswrong.com/posts/GZgLa5Xc4HjwketWe/instrumental-convergence-is-what-makes-general-intelligence), just as you get when backchaining utilities.
# Calling My Second Family Dance

Yesterday I called my second family dance, which went pretty well! I've stopped calling regular contra dances because I don't enjoy it very much anymore, but these two have been a lot of fun and I'm up for doing more.

As usual we played with an open band, and this time Nora decided to play bass:

[![](https://www.jefftk.com/nora-family-dance-bass.jpg)](https://www.jefftk.com/nora-family-dance-bass-big.jpg)

Despite the [flyers](https://www.jefftk.com/p/two-family-dance-flyers), no one from the kids classes showed up. They've had much more luck specifically inviting one or two friends. On the other hand, it probably didn't help that it was father's day.

Here's what I called, which borrowed a lot from [last time](https://www.jefftk.com/p/calling-my-first-family-dance). Since that was almost two years ago, though, I think it's fine!

*   [La Bastringue](https://www.jefftk.com/contras/dances/labastringue-family), variant where you keep your partner. Same dance I started with last time, and as before I didn't do a walk through and called while I danced. It went well!
    
*   [The Low-Backed Car](https://www.jefftk.com/contras/dances/low-backed-car). Same as last time. Another good dance!
    
*   [Heel-Toe Polka](https://www.jefftk.com/contras/dances/heel-toe-polka). This is the only dance I called where you don't keep your partner, but I told people it was fine to dance tandem. This dance is half-length, so it's important to make sure the band plays AB.
    
*   Water break. It was hot!
    
*   [Bridge of Athelone](https://www.jefftk.com/contras/dances/athelone). We had slightly too many couples for the music at ~10, but it worked ok and we didn't get off the music.
    
*   [The Blob Dance](https://www.jefftk.com/contras/dances/blob). Would have been slightly better with more dancers, but it was fine.
    
*   [Sasha](https://www.jefftk.com/contras/dances/sasha). This one Lily called; her first time! She did great! Probably the most popular dance we do, and the only one I have kids requesting by name.
    
*   Water break. We like to end Sasha unreasonably fast, so it was welcome.
    
*   [Jacob's Potato](https://www.jefftk.com/contras/dances/jacobs-potato). This dance has a whole-set circle in a lonyways dance, which is fun.
    
*   [Galopede](https://www.jefftk.com/contras/dances/galopede). I hadn't been planning to do eight dances in an hour, but the timing worked out for it. This was slightly too close to the other longways dances we'd done; I should work up a dance that is a bit more distinct.
    

It was a good time!

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02ib4ZTQQZBgTQ3CBQW9xPNPT7xi6fyHp2XZVEMUFyFzYRNKVimM2wrWdiDtRBRqATl)*
# [Linkpost] Transcendence: Generative Models Can Outperform The Experts That Train Them

Authors: Edwin Zhang, Vincent Zhu, Naomi Saphra, Anat Kleiman, Benjamin L. Edelman, Milind Tambe, Sham M. Kakade, Eran Malach.

Abstract:

> Generative models are trained with the simple objective of imitating the conditional probability distribution induced by the data they are trained on. Therefore, when trained on data generated by humans, we may not expect the artificial model to outperform the humans on their original objectives. In this work, we study the phenomenon of transcendence: when a generative model achieves capabilities that surpass the abilities of the experts generating its data. We demonstrate transcendence by training an autoregressive transformer to play chess from game transcripts, and show that the trained model can sometimes achieve better performance than all players in the dataset. We theoretically prove that transcendence is enabled by low-temperature sampling, and rigorously assess this experimentally. Finally, we discuss other sources of transcendence, laying the groundwork for future investigation of this phenomenon in a broader setting.

Summary thread: [https://x.com/nsaphra/status/1803114822445465824](https://x.com/nsaphra/status/1803114822445465824). 

Some previous discussion in an alignment-relevant context: [GPTs are Predictors, not Imitators](https://www.lesswrong.com/posts/nH4c3Q9t9F3nJ7y8W/gpts-are-predictors-not-imitators).
# Beyond the Board: Exploring AI Robustness Through Go

Last year, we showed that supposedly superhuman Go AIs can be beaten by human amateurs playing specific “cyclic” patterns on the board. Vulnerabilities have previously been observed in a wide variety of sub- or near-human AI systems, but this result demonstrates that even far superhuman AI systems can fail catastrophically in surprising ways. This lack of robustness poses a critical challenge for AI safety, especially as AI systems are integrated in critical infrastructure or deployed in large-scale applications. We seek to defend Go AIs, in the process developing insights that can make AI applications in various domains more robust against unpredictable threats.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/80e931406f4823bfdd337dc8d7f71facabb9adc7adf85b7a.png)

We explored three defense strategies: positional adversarial training on handpicked examples of cyclic patterns, iterated adversarial training against successively fine-tuned adversaries, and replacing convolutional neural networks with vision transformers. We found that the two adversarial training methods defend against the original cyclic attack. However, we also found several qualitatively new adversarial strategies that can overcome all these defenses. Nonetheless, finding these new attacks is more challenging than against an undefended KataGo, requiring more training compute resources for the adversary.

For more information, see our [blog post](https://far.ai/post/2024-06-go-defense/), [project website](https://goattack.far.ai/) or [paper](https://arxiv.org/pdf/2406.12843).
# Ilya Sutskever created a new AGI startup

[copy of the whole text of the announcement on ssi.inc, not an endorsement]

Safe Superintelligence Inc.

Superintelligence is within reach.

Building safe superintelligence (SSI) is the most important technical problem of our​​ time.

We have started the world’s first straight-shot SSI lab, with one goal and one product: a safe superintelligence.

It’s called Safe Superintelligence Inc.

SSI is our mission, our name, and our entire product roadmap, because it is our sole focus. Our team, investors, and business model are all aligned to achieve SSI.

We approach safety and capabilities in tandem, as technical problems to be solved through revolutionary engineering and scientific breakthroughs. We plan to advance capabilities as fast as possible while making sure our safety always remains ahead.

This way, we can scale in peace.

Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures.

We are an American company with offices in Palo Alto and Tel Aviv, where we have deep roots and the ability to recruit top technical talent.

We are assembling a lean, cracked team of the world’s best engineers and researchers dedicated to focusing on SSI and nothing else.

If that’s you, we offer an opportunity to do your life’s work and help solve the most important technical challenge of our age.

Now is the time. Join us.

Ilya Sutskever, Daniel Gross, Daniel Levy

June 19, 2024
# Chronic perfectionism through the eyes of school reports

A short post for a short thought. I was sifting through old school reports, and there's only one comment which is repeated in pretty much every report from 3 years old all the way through to 16.

**3 years old: "***Has a hard time dealing with disappointments. Tends to cry and is hard to console. Always wants to be first and if he is not sometimes he is very disappointed."*

**5 years old: "***Stuart sometimes gets frustrated with his finished results, but he tries his best."*

**10 years old: "***Stuart is mostly a thoughtful and friendly boy, however sometimes his rather volatile temper can get the better of him when he gets frustrated."*

**12 years old: "***\[Stuart\] is a fragile pupil who is reluctant to make mistakes which may prevent him from putting pen to paper."*

**14 years old: "***Stuart should \[...\] have the confidence to ask teachers for support when he is not sure. It is vital that Stuart can convey his ideas in writing if he wishes to show the extent of his potential."*

It's sort of telling, really. My parents weren't tigers who would berate my every failure - to the contrary, they learned this was counterproductive because I was usually harder on myself. 

There's a lot of talk about unlearning perfectionism in the world. It's not such an easy thing to get away from when it's an inherent part of your personality rather than a learned behaviour. I think perhaps the message is to treat it as nature rather than nurture, and lean in, rather than away.

I wonder how many traits there are that, for most people, are acquired responses to their environment, but for a minority are essential qualities of their nature.
# What are things you're allowed to do as a startup?

The internet has previously seen [Things You're Allowed To Do](https://milan.cvitkovic.net/writing/things_youre_allowed_to_do/) (as an adult) and [Things You're Allowed To Do: University Edition](https://www.brasstacks.blog/tyatd-uni/). The idea is to list things that are totally possible, often easy, but people often fail to think of because not one of the highlighted options. Things like "run mturk surveys" or "dissect a cadaver".

What belongs on a similar list for startups, founders, and early employees? 

Some ideas I've [heard](https://x.com/allowedthings) so far:

*   host a podcast so you can invite on people you want to meet
*   any relevant factory will be happy to talk about how they would make your product.

\[Feels like good practice to acknowledge I was commissioned to create this new list. I wouldn't have put this on LessWrong if I didn't think it was relevant and interesting to users, but I'm open to hearing I was wrong\]
# Case studies on social-welfare-based standards in various industries

Last year, I [posted a call for case studies](https://forum.effectivealtruism.org/posts/idrBxfsHkYeTtpm2q/seeking-paid-case-studies-on-standards) on social-welfare-based standards for companies and products (including standards imposed by regulation). The goal was to gain general context on standards to inform work on possible standards and/or regulation for AI.

This resulted[^hqn4byc2n2] in several dozen case studies that I found informative and helpful. I've been hoping to write up my reflections after reading them all, but it's taken me long enough to get to this that for now I am just publishing a public Google sheet with links to all of the case studies that we have permission to share publicly (including some that already have public links). The link is here: [https://docs.google.com/spreadsheets/d/18gaTIzdgMvKLq9Cp2-GJZZA7QmE93Frufh1UhNMcbpg/](https://docs.google.com/spreadsheets/d/18gaTIzdgMvKLq9Cp2-GJZZA7QmE93Frufh1UhNMcbpg/) 

  
 

[^hqn4byc2n2]:  Most of the case studies this piece links to were directly paid for via this project, but in some cases the work was pro bono, or someone adapted or sent a copy of work that had been done for another project, etc.
# Claude 3.5 Sonnet

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/c0fdd31e931c04472f6bf6856ba551563362f2e8358cdedf.webp)

> we’ll be releasing Claude 3.5 Haiku and Claude 3.5 Opus later this year.

* * *

They made a [mini model card](https://www-cdn.anthropic.com/fed9cc193a14b84131812372d8d5857f8f304c52/Model_Card_Claude_3_Addendum.pdf). Notably:

> The UK AISI also conducted pre-deployment testing of a near-final model, and shared their results with the US AI Safety Institute . . . . Additionally, METR did an initial exploration of the model’s autonomy-relevant capabilities.

It seems that UK AISI only got maximally shallow access, since Anthropic would have said if not, and in particular the model card mentions "internal research techniques to acquire non-refusal model responses" as *internal*. This is better than nothing, but it would be unsurprising if an evaluator with shallow access is unable to elicit dangerous capabilities but users—with much more time and with access to future elicitation techniques—ultimately are. [Recall](https://www.lesswrong.com/posts/WjtnvndbsHxCnFNyc/ai-companies-aren-t-really-using-external-evaluators) that DeepMind, in contrast, gave "external testing groups . . . . the ability to turn down or turn off safety filters."

* * *

Anthropic CEO Dario Amodei gave Dustin Moskovitz the impression that Anthropic [committed](https://www.lesswrong.com/posts/JbE7KynwshwkXPJAJ/anthropic-release-claude-3-claims-greater-than-gpt-4?commentId=JhBNSujBa3c78Epdn) "to not meaningfully advance the frontier with a launch." (Plus Gwern, and this was definitely Anthropic's vibe around 2022,[^oshr06ewimr]although not a hard public commitment.) Perhaps Anthropic does not consider itself bound by this, which might be reasonable — it's quite disappointing that Anthropic hasn't clarified its commitments, particularly after the confusion on this topic around the Claude 3 launch.

[^oshr06ewimr]: E.g. Zac Hatfield-Dodds said Anthropic wanted to avoid "making things worse (advancing capabilities, race dynamics, etc)." Plus nonpublic sources, including stuff discussed in the Claude 3 discourse.
# What distinguishes "early", "mid" and "end" games?

Recently William_S posted:

> In my mental model, we're still in the mid-game, not yet in the end-game.

I replied:

> A thing I've been thinking about lately is "what does it mean to shift from the early-to-mid-to-late game". 

> In strategy board games, there's an explicit shift from "early game, it's worth spending the effort to build a longterm engine. At some point, you want to start spending your resources on victory points." And a lens I'm thinking through is "how long does it keep making sense to invest in infrastructure, and what else might one do?"

> I assume this is a pretty different lens than what you meant to be thinking about right now but I'm kinda curious for whatever-your-own model was of what it means to be in the mid vs late game.

He replied:

> Like, in Chess you start off with a state where many pieces can't move in the early game, in the middle game many pieces are in play moving around and trading, then in the end game it's only a few pieces, you know what the goal is, roughly how things will play out.
> 
> In AI it's like only a handful of players, then ChatGPT/GPT-4 came out and now everyone is rushing to get in (my mark of the start of the mid-game), but over time probably many players will become irrelevant or fold as the table stakes (training costs) get too high.
> 
> In my head the end-game is when the AIs themselves start becoming real players.

This was interesting because yeah, that totally is a different strategic frame for "what's an early, midgame and endgame?", and that suggests there's more strategic frames that might be relevant.

I'm interested in this in the context of AI, but, also in other contexts.

So, prompt for discussion:

a) what are some types of games or other "toy scenarios," or some ways of looking at those games, that have other strategic lenses that help you decisionmake?

b) what are some situations in real life, other than "AI takeoff", where the early/mid/late game metaphor seems useful?
# Juneberry Puffs

I only learned [about Juneberries](https://www.jefftk.com/p/june-and-mulberries) a few years ago, and they're a bit tricky. Eaten fresh, right off the tree, they have a bunch of subtle flavors that remind me of peach ice tea. If I [bake them](https://www.jefftk.com/p/juneberry-cake), though, these flavors mostly go away, leaving an almondy flavor. Which isn't terrible, but also isn't better than almond extract and wastes what's most interesting about these berries.

Eating them raw is great, but it's also fun to make things. Thinking about what might work well that doesn't involve heating them, I decided to try puffs. First we needed some berries, and walked over to the playground that has some Juneberry trees:

[![](https://www.jefftk.com/picking-juneberries-nora-theft.jpg)](https://www.jefftk.com/picking-juneberries-nora-theft-big.jpg)

This was a bit [Blueberries for Sal](https://en.wikipedia.org/wiki/Blueberries_for_Sal), with Nora eating the berries almost as fast as I could pick them, but she lost interest after a bit and there were no bears, so soon we were walking home with a good amount.

I took a sheet of puff pastry (butter-based is the tastier option, but it's hard to find and is quite temperature sensitive), cut it into small rectangles, and baked it. Once it was fully cooked, I cut each piece in half. Here's some whole, and then some with the tops cut off:

[![](https://www.jefftk.com/puff-pastry-rectangles-cut-in-half.jpg)](https://www.jefftk.com/puff-pastry-rectangles-cut-in-half-big.jpg)

I piped in slightly sweetened whipped cream, and then put the berries on:

[![](https://www.jefftk.com/assembling-juneberry-puffs.jpg)](https://www.jefftk.com/assembling-juneberry-puffs-big.jpg)

I think they came out pretty well. They're very light and summery, and the subtle Juneberry flavors are there.
# Bed Time Quests & Dinner Games for 3-5 year olds

I like these games because they are playful, engage the child and still achieve the objective of getting the child to bed/eat dinner etc. Requires creativity and some slack. 

Excerpt from Shohannah's post:

> Recently I had the bright idea to give up on being a regular parent. Mostly cause regular parenting practices melt my brain. But then I wondered … does it *have* to be boring?
> 
> \[...\]
> 
> But no. It’s all culture and it’s all *recent* culture and you can decide to do Something Else Instead. Really.
> 
> So as someone who craves mental stimulation above the pay grade of the 3 to 5 revolutions around the sun my daughters have managed so far … I figured I’d just make up New Rules.
> 
> All the time.
> 
> So far we’ve been going for two weeks and the main areas are bedtime routines for my eldest (5) and dinner games for all of us (5, 3, and myself). I noticed I seem to have an easy time generating new and odd rule sets every day, and then started wondering if maybe more parents would enjoy this type of variety in their childcare routines and would want to tap into some of the ideas I’ve been coming up with. So in case that’s you, here is what I’ve found so far!
> 
> \[...\]
> 
> Magic Time
> 
> Kiddo is the parent. You are the kiddo. Except, the kiddo is still bringing themselves to bed and not you. They get to tell you what to do and take care of you. You will have to listen. I completely recommend performing a lot of obstructive behavior and misunderstanding basic instructions. This was one of the most popular games and may show some insight into how your child would prefer to be parented, or feels about your parenting.

and fourteen games/rulesets more.
# Karma votes: blind to or accounting for score?

Normally I try to use karma votes in a practical way to try to help promote content that I want to see more of on LW, or to downrank content that I want to see less of. (As opposed to using them as a kind of social signal, which is a separate issue.)

One thing I have never quite figured out is which of these two I should be doing to best accomplish that goal:

- Look at the current score of a post or comment, and vote so as to adjust its current score towards what I would consider to be fair.

- Judge the post independently of its current score, and have some thresholds of goodness in my mind (relative to a general standard of quality for LW posts) that I use to decide how to vote.

I could easily imagine some pros and cons to each approach. Is there any consensus as to which will make better results, either if I apply it independently, or if the community applied it universally?

My intuition is that the second is better, but I haven't really figured out anything concrete.
# Podcasts: AGI Show, Consistently Candid, London Futurists

For those of you who enjoy learning things via listening in on numerous slightly different conversations about them, and who also want to learn more about this [AI survey](https://blog.aiimpacts.org/p/2023-ai-survey-of-2778-six-things) I led, three more podcasts on the topic, and also other topics:

*   **The AGI Show**: [audio](https://www.theagishow.com/2082819/15275026-ep-13-ai-researchers-expect-agi-sooner-w-katja-grace-co-founder-lead-researcher-ai-impacts), [video](https://www.youtube.com/watch?v=Nbpw90WQmgU) (other topics include: my own thoughts about the future of AI and my path into AI forecasting)
*   **Consistently Candid**: [audio](https://www.buzzsprout.com/2319950/15286236) (other topics include: whether we should slow down AI progress, the best arguments for and against existential risk from AI, parsing the online AI safety debate)
*   **London Futurists**: [audio](https://www.youtube.com/watch?v=xwJx_xqZI3Q) (other topics include: are we in an arms race? Why is my blog called that?)
# How are you preparing for the possibility of an AI bust?

What actions have you been taking to prepare for the possibility that the AI industry will experience a market crash, something along the lines of the dotcom bust of the early 2000s? Also, what actions would you take if a crash like that occurred? For example:

*   If your career is strongly AI-focused, what backup plan do you have in place?
*   If you're working on AI alignment research, how would your research agenda change? Would you switch to a different cause area altogether?
*   Likewise, if you're working on AI policy, how would your approach to it change?

Let's set aside speculation about whether a crash will happen or not; I'd like to hear what concrete actions people are taking.
# "On the Impossibility of Superintelligent Rubik’s Cube Solvers", Claude 2024 [humor]

> In recent years, a number of prominent computer scientists and roboticists have suggested that artificial intelligence may one day solve Rubik's Cubes faster than humans. Many have further argued that AI could even come to exceed human Rubik's Cube-solving abilities by a significant margin. However, there are at least twenty distinct arguments that preclude this outcome. We show that it is not only implausible that AI will ever exceed human Rubik's Cube-solving abilities, but in fact impossible.
# Sparse Features Through Time

This project explores the use of Sparse Autoencoders (SAEs) to track the development of features in large language models throughout their training. It investigates whether features can be reliably matched between different SAEs trained on various checkpoints of Pythia 70M and characterises the development of these features over the course of training. The findings show that features can successfully be matched between different SAEs. The results also support the distributional simplicity bias hypothesis, indicating that simpler features are learned early in training, with more complex features emerging later. While the focus was on a relatively small model, the results lay the groundwork for future research into larger models and the identification of potentially deceptive capabilities. This work aims to enhance the interpretability and safety of AI systems by providing a deeper understanding of feature development and the dynamics of neural network training, with the ultimate goal of making a general statement about the overall likelihood of dangerous deceptive alignment arising in practice.