Harley-ml/lesswrong huggingface dataset:
# How do low level hypotheses constrain high level ones? The mystery of the disappearing diamond.

Consider two friends, Alice and Bob, trying to figure out what happened to a diamond that disappeared from a museum. They do so in the form of a game that is kind of an approximation to [Solomonoff induction](https://www.lesswrong.com/tag/solomonoff-induction): they will work together to come up with the *smallest possible explanations* that conform to the data, for some intuitive notion of smallness.

This helps to eliminate fake explanations; the hypothesis "[a witch](https://www.lesswrong.com/posts/f4txACqDWithRi7hs/occam-s-razor) caused Henry to fall ill" can be simplified to "Henry fell ill". But "Sally touched Henry and Bob, and Sally is sick and Henry is sick and Bob is sick" is beaten by "Sally touched Henry and Bob and Sally is contagiously sick". An explanation is good if it is *smaller* than just hard-coding the answer.

Bob knows that there are four diamond thieves in the city, so he comes up with four hypotheses:

1.  The diamond was stolen by thief number 1.
2.  The diamond was stolen by thief number 2.
3.  The diamond was stolen by thief number 3.
4.  The diamond was stolen by thief number 4.

These are all roughly the same complexity (depending on how you encode numbers), so this provides a uniform distribution over the four thieves.

Alice comes up with one hypothesis:

1.  The diamond spontaneously ceased existing.

and declares victory.

Bob: What, that makes no sense? Physical objects can't stop existing.  
Alice: We aren't doing physics; we are playing a game.  
Bob: ͠° ͟ʖ ͡°  
Alice: ¯\\_(ツ)_/¯

But there is an additional rule; you can add other data from the real world to the challenge. For example, for "Henry falling ill", you might get better hypotheses if you try to compress info about *all* the sick people in the village, so that a slightly more complex hypothesis that can explain all of them wins!

Bob: I hereby add all physics experiments to the data set!

Alice then comes up with the following hypothesis: all physical experiments are explained by the standard model of physics, except the diamond spontaneously ceased existing.

![](https://www.sciencealert.com/images/Screen_Shot_2016-08-03_at_3.20.12_pm.png)

"Was it really necessary to write it out? And why do you know this off the top of your head?" said Bob.

Bob: That bit about the diamond ceasing to exist is arbitrary!  
Alice: Do you have a better hypothesis?  
Bob: Uhm, idk. All physical experiments are explained by the standard model of physics, and the diamond was stolen by thief 3?  
Alice: ͠° ͟ʖ ͡°

Bob's new hypothesis is bigger than Alice's new hypothesis.

And thus we run into the problem. Due to Alice's and Bob's [bounded rationality](https://www.lesswrong.com/tag/bounded-rationality), they can't determine which thief stole the diamond from the laws of physics alone. So the laws of physics at a low level don't help compress hypotheses at a high level, and thus can't *constrain* the smallest possible hypotheses they can consider.

How do you modify the game so that the spontaneously disappearing diamond hypothesis doesn't win?
# Why it's necessary to shoot yourself in the foot

Without shooting yourself in the foot, learning lacks motivation. Complexity without reason is really confusing.

---

At the Recurse Center Feb 2020: I watch a talk during presentations about someone optimizing a database engine. It was really complicated, and I remeber nothing about the talk itself. What I do remember is thinking to myself "this seems really complicated for no good reason." Keep in mind, I had never made a web application at that time and when I needed to store data, I just used a csv file or a python pickle file on the disk. I thought that a filesystem was sufficient for storing data.


Fast forward a few months and I'm building my first web application. I don't remember what it was for, but I remember using a csv file as the database. I had to load the file into memory every time I wanted to look something up and it was just a big pain. I now understood why using a database is (sometimes) a good idea.

---

Learning Rust, July 2020: I start learning about the borrow checker. It prevents you from keeping a pointer to an item of a vector (`&vec[i]`) if you pass the `Vec<T>` as a `&mut`. I don't really get why this is necessary. I have never done low-level programming before, never used pointers, and now I am being told that following the borrow checker is 'safe'. It is still very confusing to me.

Fast forward to when I am writing Zig code. I take `&array_list.items[i]`, append to the `array_list` and then try to write to the stored pointer. I get a segfault. Ahh, now I get the problem Rust solves.

---

Learning Vue, July 2020: Why are there all of these complicated ways to represent state? Shouldn't developing a web app be simpler than this?

After writing a bunch of vanilla JS, I can see why these frameworks could be useful. I have never made a very big web app, but I could see that keeping track of state and what is or isn't rendered is hard and gets much harder with a bigger web app.

---

To fully understand a "best practice" or why something is necessary, it's important to experience how things go wrong without it. When teaching programming, we should let people make these mistakes, and *then* show them the tools to correct them. Just giving someone a complicated tool without a salient reason to explain its complexity will just make them really confused.

Some opinions:
* You should store stuff in csv files before using a database
* You should learn Zig (or C) before you learn Rust
* You should write a web app in vanilla JS before you learn a framework
* You should write a game from scratch before using Unity
* You should use `javac` from the command line before using an IDE
# What is some unnecessarily obscure jargon that people here tend to use?

In particular I'm thinking of computer programming jargon used in unrelated contexts ("My algorithm for choosing a restaurant to eat at...") just because all parties to the conversation happen to be computer programmers. I wonder, if there were a group discussing the same topics as on LessWrong, but composed predominantly of lawyers (or economists, doctors, etc.), would they also reflexively import their profession's jargon? How can we communicate in a way that doesn't *unnecessarily* confuse people of a different profession?

(To be sure, there's some stuff talked about here that *is* actually about computers, but there's also a lot that isn't.) 

See also [Style Guide: Not Sounding Like An Evil Robot](https://slatestarcodex.com/2019/07/04/style-guide-not-sounding-like-an-evil-robot/). ("Humans" instead of "people" is the one I notice most.)
# Report on modeling evidential cooperation in large worlds

I have published a [report](https://arxiv.org/pdf/2307.04879.pdf) on modeling [evidential cooperation in large worlds](https://longtermrisk.org/msr). I originally started working on the report during a CEA Summer Research Fellowship back in 2018 and have now finally found the time to finish it. The report itself is 75 pages long, but **the introduction includes a short 4 1/2 page summary** with the main takeaways. Since the bulk of the work was done in 2018, it does not cover more recent work such as the [ROSE Value](https://www.lesswrong.com/posts/vJ7ggyjuP4u2yHNcP/threat-resistant-bargaining-megapost-introducing-the-rose).

> **Abstract**
> 
> *Evidential cooperation in large worlds* (ECL) refers to the idea that humans and other agents can benefit by cooperating with similar agents with differing values in causally disconnected parts of a large universe. Cooperating provides agents with evidence that other similar agents are likely to cooperate too, resulting in gains from trade for all. This could be a crucial consideration for altruists.
> 
> I develop a game-theoretic model of ECL as an incomplete information bargaining problem. The model incorporates uncertainty about others' value systems and empirical situations, and addresses the problem of selecting a compromise outcome. Using the model, I investigate issues with ECL and outline open technical and philosophical questions.
> 
> I show that all cooperators must maximize the same weighted sum of utility functions to reach a Pareto optimal outcome. However, I argue against selecting a compromise outcome implicitly by normalizing utility functions. I review bargaining theory and argue that the Nash bargaining solution could be a relevant Schelling point. I introduce *dependency equilibria* ([Spohn 2007](https://kops.uni-konstanz.de/server/api/core/bitstreams/e0aacef9-928c-4382-aab7-1cf1fbb2e89e/content)), an equilibrium concept suitable for ECL, and generalize a folk theorem showing that the Nash bargaining solution is a dependency equilibrium. I discuss gains from trade given uncertain beliefs about other agents and analyze how these gains decrease in several toy examples as the belief in another agent decreases.
> 
> Finally, I discuss open issues in my model. First, the Nash bargaining solution is sometimes not coalitionally stable, meaning that a subset of cooperators can unilaterally improve payoffs by deviating from the compromise. I investigate conditions under which stable payoff vectors exist. Second, I discuss how to model agents' default actions without ECL.
# Flowchart: How might rogue AIs defeat all humans?

*Epistemic status: I put this together for myself mostly off the top of my head without carefully cross-referencing other sources. I'm hoping to flesh this out a lot more and do a much more thorough research job of it at some point, but in the meantime I've decided to err on the side of getting this out there even in the current unpolished state.*

The following is a rough draft version of a Miro flowchart I put together for how a misaligned rogue AI or group of AIs might actually take over the world and cause kill all humans. I *think* it should be mostly self-explanatory, even if vague in places.

Partly this was inspired by [Carl Shluman](https://www.lesswrong.com/users/carlshulman?mention=user)'s recent [interview](https://www.lesswrong.com/posts/Ghrdnc26ftJrxD49z/carl-shulman-on-the-lunar-society-7-hour-two-part-podcast) on the Lunar Society podcast, but I haven't actually checked yet how closely this matches with Carl's discussion.

[Link to the Miro board](https://miro.com/welcomeonboard/aDN2bGJRS1NzS0N2cjdOaEM0NUVXb3h4ZW4yM2k2MG5qa1FDTUlLNENxY2g1Vk16SHFVaFJjcmdiTkRwa1loQ3wzNDU4NzY0NTMwNDUxMDEzNDU1fDI=?share_link_id=586557732105)

**1) Paths towards human extinction**

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/af49aefbc86f372bbe9e2bb8104898bb82926c1c0eb19f35.png)

**2) How exactly could rogue AIs defeat all humans?**

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/68b534e4b4377238389323aed1059cacaaef8afa13093e0d.png)

**3) How could misaligned AIs gain access to / control over the resources needed to defeat all humans?**

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/977347490b4d46a5563a03917bde0db94d166a69913808d3.png)
# What does the launch of x.ai mean for AI Safety?

[x.ai](https://x.ai/) has now launched. It seems worthwhile to discuss both what it means for AI safety and whether people interested in AI safety should [consider applying](https://airtable.com/appO5m0cwztccTdiA/shrtrQqaaoBt4Xm8i) for the company.

**Some thoughts:**

It's notable that Dan Hendrycks is listed as an advisor (the only advisor listed).

The team is also listed on the page.  
  
I haven't taken the time to do so, but it might be informative for someone to Google the individuals listed to discover whether they lie in terms of their interest being between capabilities and safety.
# Are there any good, easy-to-understand examples of cases where statistical causal network discovery worked well in practice?

When I first read the Sequences, one of the exciting posts was [Causal Diagrams and Causal Models](https://www.lesswrong.com/posts/hzuSDMx7pd2uxFc5w/causal-diagrams-and-causal-models), which got me into the idea that one could discover the structure of causal networks using statistics. Another rationalist source which gave me similar hopes was Scott Alexander's [SSC Journal Club: Mental Disorders As Networks](https://slatestarcodex.com/2016/12/14/ssc-journal-club-mental-disorders-as-networks/).

However, when I actually started applying these techniques to my own data, or to publicly available datasets, I often found that the techniques were unstable, and that one could easily infer plausible conditions where they would give the wrong results. It's possible I had the wrong approach or something, but in my confusion I started reading up on what experts in causal inference had said, and I got the impression that they studied the problem for a while, initially finding some algorithms, but over time concluding that their algorithms didn't work very well and that it is better to just have a human in the loop who specifies the causal networks.

So I mostly abandoned it, or saw it as a much more limited tool than I had before. But recently, John Wentworth [argued that it was actually quite feasible in practice](https://www.lesswrong.com/posts/qPrPNakJBq23muf4n/bayesian-networks-aren-t-necessarily-causal?commentId=ZuqLAJkfqqLFMoNdk), so maybe I was too quick to abandon it. I would like to know - what are the best examples of this working well in practice? Or alternatively, did anyone else come to the same conclusions as I did?
# Elon Musk announces xAI

Some quotes & few personal opinions:  
  
FT reports

> Musk is also in discussions with a number of investors in SpaceX and Tesla about putting money into his new venture, said a person with direct knowledge of the talks. “A bunch of people are investing in it . . . it’s real and they are excited about it,” the person said.  
> ...  
> Musk recently changed the name of Twitter to X Corp in company filings, as part of his plans to create an “everything app” under the brand “X”. For the new project, Musk has secured thousands of high-powered GPU processors from Nvidia, said people with knowledge of the move.   
> ...  
> During a Twitter Spaces interview this week, Musk was asked about a Business Insider report that Twitter had bought as many as 10,000 Nvidia GPUs, “It seems like everyone and their dog is buying GPUs at this point,” Musk said. “Twitter and Tesla are certainly buying GPUs.” People familiar with Musk’s thinking say his new AI venture is separate from his other companies, though it could use Twitter content as data to train its language model and tap Tesla for computing resources. 

According to xAI website, the initial team is composed of   
  
[Elon Musk](https://twitter.com/elonmusk)

[Igor Babuschkin](https://twitter.com/ibab_ml)

[Manuel Kroiss](https://twitter.com/makro_ai)

[Yuhuai (Tony) Wu](https://twitter.com/yuhu_ai_)

[Christian Szegedy](https://twitter.com/chrszegedy)

[Jimmy Ba](https://twitter.com/jimmybajimmyba)

[Toby Pohlen](https://twitter.com/tobyphln)

[Ross Nordeen](https://twitter.com/rpoo)

[Kyle Kosic](https://twitter.com/kylekosic)

[Greg Yang](https://twitter.com/thegregyang)

[Guodong Zhang](https://twitter.com/guodzh)

[Zihang Dai](https://twitter.com/zihangdai)

and they are *"advised by Dan Hendrycks, who currently serves as the director of the Center for AI Safety."* 

According to reports xAI will seek to create a *"maximally curious"* AI, and this also seems to be the main new idea how to solve safety, with Musk explaining: *"If it tried to understand the true nature of the universe, that's actually the best thing that I can come up with from an AI safety standpoint,"* ... *"I think it is going to be pro-humanity from the standpoint that humanity is just much more interesting than not-humanity."*  
  
My personal comments:  
  
Sorry, but at face value, this just does not seem a great plan from safety perspective. Similarly to Elon Musk's previous big bet how to make us safe by making AI open-source and widely distributed ("giving *everyone* access to new ideas").  
  
Sorry, but given *"Center for AI Safety"* moves to put them into some sort of *"Center"*, public representative position of AI Safety - including the name choice, and organizing the widely reported [Statement on AI risk](https://www.safe.ai/statement-on-ai-risk) \- it seems publicly associating their brand with xAI is a strange choice.
# How can I get help becoming a better rationalist?

Well…

Right now, being ‘a rationalist’ could be said to be a massive part of my identity, at least judging by the absurd amount of time I’ve spent reading posts here, or SSC/ACX, and in a few other places. Yet, I’m still a mere lurker unfamiliar with most of the local customs.

But it’s not what matters. What does is that I’m a *terrible* rationalist.   
You see, rationality takes practice. And reading stuff on LW isn’t practice at all. If anything, it’s just a great way of filling my brain with a lot of useful concepts, and then either blame myself for not using them, or use them for something entirely unrelated to their normal purpose. Often, to make myself feel worse, and think worse.

As the saying goes, rationality is a martial art. Learning it by reading the rules, or by watching other people apply the rules, is about as effective as developing one’s muscles by watching sports on TV.

I know of the CFAR, and of various related groups, meetups for ACX readers or for other people, etc. But, apart from ACX meetups, which aren’t about being better rationalists *per* *se*, I don’t have easy access to any of those, or certainly to a general environment which welcomes this. You know, not being in the Bay Area and all. 

And yet, I want to be more rational as much as anyone who’s been lurking here for five years wants it, and given how depressed I was until very recently, I probably badly need it, too.

I’m not sure what kind of answers I expect, but, like, how can I push myself to learn more, and especially to practice more, and ideally to actually use rationality to improve my life?
# Accidentally Load Bearing

Sometimes people will talk about Chesterton's Fence, the idea that if you want to change something—removing an apparently useless fence—you should first determine why it was set up that way:

> The gate or fence did not grow there. It was not set up by somnambulists who built it in their sleep. It is highly improbable that it was put there by escaped lunatics who were for some reason loose in the street. Some person had some reason for thinking it would be a good thing for somebody. And until we know what the reason was, we really cannot judge whether the reason was reasonable. It is extremely probable that we have overlooked some whole aspect of the question, if something set up by human beings like ourselves seems to be entirely meaningless and mysterious. — *G. K. Chesterton, The Drift From Domesticity*

Figuring out something's designed purpose can be helpful in evaluating changes, but a risk is that it puts you in a frame of mind where what matters is the role the original builders intended.

A few years ago I was [rebuilding a bathroom in our house](https://www.jefftk.com/p/bathroom-construction-framing), and there was a vertical stud that was in the way. I could easily tell why it was there: it was part of a partition for a closet. And since I knew its designed purpose and no longer needed it for that anymore, the Chesterton's Fence framing would suggest that it was fine to remove it. Except that over time it had become accidentally load bearing: through other (ill conceived) changes to the structure this stud was now helping hold up the second floor of the house. In addition to considering why something was created, you also need to consider what additional purposes it may have since come to serve.

This is a concept I've run into a lot when making changes to complex computer systems. It's useful to look back through the change history, read original design documents, and understand why a component was built the way it was. But you also need to look closely at how the component integrates into the system today, where it can easily have taken on additional roles.
# Mech Interp Puzzle 1: Suspiciously Similar Embeddings in GPT-Neo

*I made a series of mech interp puzzles for my MATS scholars, which seemed well received, so I thought I'd share them more widely! I'll be posting a sequence of puzzles, approx one a week - these are real questions about models which I think are interesting, and where thinking about them should teach you some real principle about how to think about mech interp. Here's a short one to start:*

**Mech Interp Puzzle 1:**
This is a histogram of the pairwise cosine similarity of the [embedding](https://dynalist.io/d/n2ZWtnoYHrU1s4vnFSAQ519J#z=-ocqvvz7qkEWJZR0QaPflY-2) of [tokens](https://dynalist.io/d/n2ZWtnoYHrU1s4vnFSAQ519J#z=lc55L43cIdXsJCWM_CJ-NwS-) in GPT-Neo (125M language model). Note that the mean is very high! (>0.9) Is this surprising? Why does this happen?

![](https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2FNeelNanda%2Fu-3VN3rBf5.png?alt=media&token=03f97795-410f-4e4f-962e-ebf8e3c33e54)

**Bonus question**: Here's the same histogram for GPT-2 Small, with a mean closer to 0.3. Is *this* surprising? What, if anything, can you infer from the fact that they differ?

![](https://firebasestorage.googleapis.com/v0/b/firescript-577a2.appspot.com/o/imgs%2Fapp%2FNeelNanda%2FRb-6IId48F.png?alt=media&token=8e5442f4-2a72-4fbc-ad16-75b41900766e)

**Code:**

```
!pip install transformer_lens plotly
from transformer_lens import HookedTransformer
import plotly.express as px
import torch
model = HookedTransformer.from_pretrained("gpt-neo-small")
subsample = torch.randperm(model.cfg.d_vocab)[:5000].to(model.cfg.device)
W_E = model.W_E[subsample] # Take a random subset of 5,000 for memory reasons
W_E_normed = W_E / W_E.norm(dim=-1, keepdim=True) # [d_vocab, d_model]
cosine_sims = W_E_normed @ W_E_normed.T # [d_vocab, d_vocab]
px.histogram(cosine_sims.flatten().detach().cpu().numpy(), title="Pairwise cosine sims of embedding")
```

**Answer:** (decode with [rot13](https://rot13.com))
Gur zrna irpgbe bs TCG-Arb vf whfg ernyyl ovt - gur zbqny pbfvar fvz jvgu nal gbxra rzorq naq gur zrna vf nobhg 95% (frr orybj).
Gur pbaprcghny yrffba oruvaq guvf vf **znxr fher lbhe mreb cbvag vf zrnavatshy**. Zrgevpf yvxr pbfvar fvz naq qbg cebqhpgf vaureragyl cevivyrtr gur mreb cbvag bs lbhe qngn. Lbh jnag gb or pnershy gung vg'f zrnavatshy, naq gur mreb rzorqqvat inyhr vf abg vaureragyl zrnavatshy. (Mreb noyngvbaf ner nyfb bsgra abg cevapvcyrq!) V trarenyyl zrna prager zl qngn - guvf vf abg n havirefny ehyr, ohg gur uvtu-yriry cbvag vf gb or pnershy naq gubhtugshy nobhg jurer lbhe bevtva vf!
V qba'g unir n terng fgbel sbe jul, be jul bgure zbqryf ner fb qvssrerag, V onfvpnyyl whfg guvax bs vg nf n ovnf grez gung yvxryl freirf fbzr checbfr sbe pbagebyyvat gur YnlreAbez fpnyr. Vg'f onfvpnyyl n serr inevnoyr bgurejvfr, fvapr zbqryf pna nyfb serryl yrnea ovnfrf sbe rnpu ynlre ernqvat sebz gur erfvqhny fgernz (naq rnpu ynlre qverpgyl nqqf n ovnf gb gur erfvqhny fgernz naljnl). Abgnoyl, Arb hfrf nofbyhgr cbfvgvbany rzorqqvatf naq guvf ovnf fubhyq or shatvoyr jvgu gur nirentr bs gubfr gbb.


Please share your thoughts in the comments!
# Even briefer summary of ai-plans.com

Currently, newcomers to the field of AI Alignment often struggle to understand the ongoing work and individuals involved, as well as the assumptions, strengths, and weaknesses of each plan.

  
 

We believe AI-plans.com will be an easy, centralized way to discover and learn more about the most promising alignment plans.

  
 

The site is currently in Stage 1, functioning purely as a compendium. We are in the process of adding up to 1000 plans and the criticisms made against them so far. Further plans and criticisms can be added by users.

  
 

Projected benefits of Stage 1:

\- Easy discovery of proposed plans and better understanding of their prevalent challenges. 

(This is already showing promise, with one researcher letting us know they found useful papers on the site and multiple researchers interested- including Jonathan Ng who has been helping us.)

  
  
 

Next, in Stage 2, we will introduce a scoring system for criticisms and a ranking system for plans. Plans will be ranked based on the cumulative scores of their criticisms. Criticism votes will be weighted, giving more influence to users who have submitted higher-scoring criticisms. Alignment researchers will have the option to link their AI-Plans account to accounts on research-relevant platforms (such as arXiv, OpenReview or the AI Alignment Forum) in order to start out with a slightly weighted vote (with mod approval).

  
 

Each new plan will start with 0 bounty, and lower bounty plans will give the most points. That way, each new plan will have a lot of opportunity and incentive for criticism. More details [here](https://www.lesswrong.com/posts/hcTiw9xKNZAi7qcy6/an-overview-of-the-points-system).

  
 

Projected benefits of Stage 2:

\- Incentivizes users to write high-quality criticisms.

\- Facilitates identification of plans with significant weaknesses, supporting arguments against problematic plans.

\- Allows newcomers to the field(including talented and untapped scientists and engineers) to see which companies have the least problematic plans.   
After all, who would want to work for the lowest-ranked company on the leaderboard? 

(I have spoken with the creator of aisafety.careers, who intends to integrate with our site.)

  
 

At Stage 3, in addition to everything from Stage 1 and 2, we plan to introduce monthly cash prizes for the highest ranking plan and for the users with the most criticism points that month.

  
 

Projected benefits of Stage 3:

\- Supercharges the impact of Stage 2, attracting talented individuals who require a non-committal monetary incentive to engage with alignment research.

\- Provides a heuristic argument for the difficulty of the problem: "There is money on the table if anyone can come up with a plan with fewer problems, yet no one has done so!"  
  
  
What I'd like to ask lesswrong users-   
What do you think could go wrong with this?  
What would make you want to use the site/participate/contribute?
# Build knowledge base first, or backchain?

Specifically for AI alignment (small field, preparadigmatic, plausibly-short-timelines), but a general ~~principle~~ decision-mechanism could apply to other fields.

What I mean by this is, **what ratio** of pre-learning to filling-in-gaps, leads to deeper and quicker insights? (Note that, given the time-blocks I'm thinking of here, I could get sidetracked by *either* of these in a counterproductive way, either by going too-deep on a knowledge-base-item that's actually trivial, or by doing an approach that a deeper knowledge base would immediately show to be wrong.)

Again, especially interested in how this applies to AI alignment. I've seen plausibly good arguments for tilting the ratio one way or the other. And I *vaguely* get that two people could implement the same knowledge-base-vs-backchain ratio, but *feel* like they're leaning different directions.
# An Overview of AI risks - the Flyer

[EffiSciences](https://ia.effisciences.org/) recently published a document outlining various types of AI risks destined for a technical audience.

We shared this flyer as part of a hackathon we organized with [Entrepreneur first](https://www.joinef.com/) at [MetaAI](https://ai.meta.com/) Paris. 

*Feel free to copy and modify this flyer for your own usage.*

Special thanks to Jonathan Claybrough and Timothée Chauvin for their significant contributions. Thanks to Esben Kran for helping for the technical mentoring during the hackathon.

Note that the distribution of this flyer alone was insufficient to communicate all the essential ideas; the one-on-one discussions proved to be extremely important for updates. However, the flyer served its purpose in catalyzing productive conversations.
# Conditional on living in a AI safety/alignment by default universe, what are the implications of this assumption being true?

I'm specifically asking this question because I suspect there would be some important changes to the assumptions made on LW, as well as change what's good to do.

Conditional on AI alignment/safety being the default case, and the cases where AI alignment turned out to not require much effort, or at least it was amenable to standard ML/AI techniques, and AI misalignment effectively being synonymous with AI misuse, what are the most important implications of this assumed truth for the LessWrong community?

In particular, I'm thinking of scenarios where rogue AI is easy to align or make safe by default, such that the problems shift from rogue AI to humans using AI.
# [Interview w/ Rob Miles] The case for taking AI Safety seriously

I recently concluded a fascinating interview with science communicator Rob Miles on the [Futurati Podcast](https://www.youtube.com/watch?v=r-SmqbqIiXk).

We cover:

*   His background in computer science, and how he became interested in the broader AI Safety debate.
*   The resources he recommends for newcomers to the field. The list includes Cambridge's [AI Safety Fundamentals Course](https://aisafetyfundamentals.com/), [AI Safety Support](https://www.aisafetysupport.org/) for those interested in an AI-safety career, and [AI Safety Training](https://aisafety.training/), which is a compendium of different courses, conferences, etc.
*   How Yann LeCunn is just *super* confused and confusing.
*   The basic reasons for believing that the rise of smarter-than-human systems probably won't end well for human beings.
*   Various reasons for thinking there are still plenty more capabilities gains to be had from scaling.
*   Why SGD may not be much like evolution (but that's still no reason to think we're out of the woods.)
*   Why being uncertain about the future direction of a powerful new technology in no way means you're safe.
*   How the security mindset plays into an informs the conversation around AI Safety.
*   Why agency might *emerge* from large training runs, but why that might not end up mattering if people work as hard as they can to build agentic systems at the first opportunity.
*   The most promising approaches to AI alignment, including Anthropic's Constitutional AI, mechanistic interpretability, and robustness. 

As always, if you find this helpful at all, like the episode and share it. We'd like to devote more time to these kinds of interviews, and "number go up" is the most encouraging metric we can get.
# Least-problematic Resource for learning RL?

Well, [Sutton & Barto](http://incompleteideas.net/book/the-book-2nd.html) is the standard choice, [but](https://www.lesswrong.com/posts/Wnqua6eQkewL3bqsF/matt-botvinick-on-the-spontaneous-emergence-of-learning?commentId=BPzWcdeKzWk6Cio2J):

> Superficial, not comprehensive, somewhat outdated circa 2018; a good chunk was focused on older techniques I never/rarely read about again, like SARSA and exponential feature decay for credit assignment. The closest I remember them getting to DRL was when they discussed the challenges faced by function approximators.

And also has some issues with eg claiming that the Reward is the [optimization target](https://www.lesswrong.com/posts/pdaGN6pQyQarFHXF4/reward-is-not-the-optimization-target). Other RL textbooks also seem similarly problematic - very outdated, with awkward language / conceptual confusions.

[OpenAI's Spinning Up DRL](https://spinningup.openai.com/en/latest/) seems better in the not-being-outdated front, but feels quite high-level, focusing mostly on practicality & implementation - while I'm looking also for a more theoretical discussion of RL.

I'm starting to think that there probably isn't such a resource fitting all my bills, so I'm considering the mix of (1) lightly reading textbooks for old-RL theory discussions and (2) instead covering modern surveys for catching up to the recent DRL stuff.

Are there any resources for learning RL that doesn't contain (any of) the problems I've mentioned above? Would like to know if I'm missing any.
# Meta announces Llama 2; "open sources" it for commercial use

*See also their Llama 2 website here:* [*https://ai.meta.com/llama*](https://ai.meta.com/llama)*, and their research paper here:* [*https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/*](https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/)

From their blog post:

> Takeaways
> ---------
> 
> *   Today, we’re introducing the availability of Llama 2, the next generation of our open source large language model. 
> *   Llama 2 is free for research and commercial use. 
> *   Microsoft and Meta are expanding their longstanding partnership, with Microsoft as the preferred partner for Llama 2.
> *   We’re opening access to Llama 2 with the support of a broad set of companies and people across tech, academia, and policy who also believe in an open innovation approach to today’s AI technologies.

* * *

Compared to the first Llama, LLama 2 is trained for 2T tokens instead of 1.4T, has 2x the context length (4096 instead of 2048), uses [Grouped Query Attention](https://arxiv.org/abs/2305.13245), and performs better across the board, with performance generally exceeding `code-davinci-002` on benchmarks:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/106010cbbdd3c3cfce1442d1c5fc9c8b25c0d94efabe22dd.png)

They also release both a normal base model (Llama 2) and a RLHF'ed chat model (Llama 2-chat). Interestingly, they're *only* releasing the 7B/13B/70B models, and not the 34B model, "due to a lack of time to sufficiently red team". 

More importantly, they're releasing it both on Microsoft Azure and also making it available for commercial use. [The form for requesting access](https://ai.meta.com/resources/models-and-libraries/llama-downloads/) is very straightforward and does not require stating what you're using it for: (EDIT: they gave me access ~20 minutes after submitting the form, seems pretty straightforward.)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/917e80cab2c098f7661c39573f5d5289bc1c249a1b59966f.png)

Note that their license is *not* technically free for commercial use always; it contains the following clauses:

> \[1.\] v. You will not use the Llama Materials or any output or results of the Llama Materials to improve any other large language model (excluding Llama 2 or derivative works thereof).  
>  
> 
> 2\. **Additional Commercial Terms.** If, on the Llama 2 version release date, the monthly active users of the products or services made available by or for Licensee, or Licensee’s affiliates, is greater than 700 million monthly active users in the preceding calendar month, you must request a license from Meta, which Meta may grant to you in its sole discretion, and you are not authorized to exercise any of the rights under this Agreement unless or until Meta otherwise expressly grants you such rights.

See also the [Llama 2 Acceptable Use Policy](https://ai.meta.com/llama/use-policy/) (which seems pretty standard).
# What Caused the Puzzling Decline in Activism Against Policy Violence Towards Black People?

I find myself surprised by the noticeable decline in the fervor surrounding activism against policy violence towards Black people from its peak in the summer of 2020 to the subsequent years of 2021 through 2023. Initially, I had anticipated that demands for increased use of body cams would result in the surfacing of more videos documenting the maltreatment of Black individuals by law enforcement officers, inevitably spurring more protests.

To attempt to make sense of this, I have considered several hypotheses:

\- **FBI Infiltration**: The FBI may have successfully infiltrated activist networks, consequently dismantling their internal structure and hindering their ability to organize.

\- **Improved Law Enforcement Conduct**: Police officers might have made significant changes to their behavior, thus reducing the chances of any controversial incidents being captured on camera.

\- **Shift in Mainstream Media Focus**: The mainstream media, which previously stoked the flames of protest during the Trump administration, might have lost interest under Biden's administration, leading to a reduction in protest activities.

\- **Democratic Influence**: Democratic organizers may have found that the 'defund-the-police' activism was polling poorly, leading to concerted efforts to tone it down.

\- **Misallocation of BLM Funds**: Perhaps there's been some level of [corruption](https://nymag.com/intelligencer/2022/04/black-lives-matter-6-million-dollar-house.html) within the BLM movement, with funds being directed towards personal extravagances instead of being allocated for organizing protests.

\- **Lack of Organic Support**: Unlike the sustained energy seen in movements like the Tea Party, the impetus behind these protests might have been primarily media-driven, evaporating once the media's interest waned.

\- **Successful Reforms**: Maybe there have been enough successful reforms in the policy violence sphere, leading people to feel less need to advocate for the cause.

While these are speculative theories, I'm curious hear from those who were less surprised by this development. I'm interested in understanding what you perceive to be the primary factors behind this trend.
# Valuism - an approach to life for you to consider

I finally finished my series of 5 essays on my life philosophy (Valuism)! In a nutshell, Valuism suggests that you work to figure out what you intrinsically value, and then that you try to use effective methods to create more of what you intrinsically value. 

While simple and straightforward at first glance, few people seem to approach life this way, and Valuism ends up with a number of surprising implications and, I think, provides a perspective that can help shed light on a number of different domains. 

Interestingly, Effective Altruism is implied by Valuism plus a specific set of strong intrinsic values that most Effective Altruists have (reducing suffering + truth).  
 

Here is the sequence of essays, if you feel like checking them out: 

[Part 1: Doing what you value as a life philosophy – an introduction to Valuism ](https://www.spencergreenberg.com/2023/02/doing-what-you-value-as-a-way-of-life-an-introduction-to-valuism/ )

[Part 2: What to do when your values conflict? ](https://www.spencergreenberg.com/2023/02/what-to-do-when-your-values-conflict-part-2-in-the-valuism-sequence/ )

[Part 3: Should Effective Altruists be Valuists instead of utilitarians? ](https://www.spencergreenberg.com/2023/03/should-effective-altruists-be-valuists-instead-of-utilitarians-part-3-in-the-valuism-sequence/ )

[Part 4: What would a robot value? An analogy for human values ](https://www.spencergreenberg.com/2023/05/what-would-a-robot-value-an-analogy-for-human-values-part-4-of-the-valuism-sequence/ )

[Part 5: Valuism and X: how Valuism sheds light on other domains ](https://www.spencergreenberg.com/2023/07/valuism-and-x-how-valuism-sheds-light-on-other-domains-part-5-of-the-sequence-on-valuism/)

A big shoutout goes to Amber Dawn Ace who wrote these essays with me.
# Alignment Grantmaking is Funding-Limited Right Now

For the past few years, I've generally mostly heard from alignment grantmakers that they're bottlenecked by projects/people they want to fund, not by amount of money. Grantmakers generally had no trouble funding the projects/people they found object-level promising, with money left over. In that environment, figuring out how to turn marginal dollars into new promising researchers/projects - e.g. by finding useful recruitment channels or designing useful training programs - was a major problem.

Within the past month or two, that situation has reversed. My understanding is that alignment grantmaking is now mostly funding-bottlenecked. This is mostly based on word-of-mouth, but for instance, I heard that the recent [lightspeed grants](https://www.lesswrong.com/posts/xQ4ajnzavSgbYiko2/launching-lightspeed-grants-apply-by-july-6th) round received far more applications than they could fund which passed the bar for basic promising-ness. I've also heard that the [Long-Term Future Fund](https://funds.effectivealtruism.org/funds/far-future) (which funded my current grant) now has insufficient money for all the grants they'd like to fund.

I don't know whether this is a temporary phenomenon, or longer-term. Alignment research has gone mainstream, so we should expect both more researchers interested and more funders interested. It may be that the researchers pivot a bit faster, but funders will catch up later. Or, it may be that the funding bottleneck becomes the new normal. Regardless, it seems like grantmaking is at least funding-bottlenecked right now.

Some takeaways:

*   If you have a big pile of money and would like to help, but haven't been donating much to alignment because the field wasn't money constrained, now is your time!
*   *If* this situation is the new normal, then earning-to-give for alignment may look like a more useful option again. That said, at this point committing to an earning-to-give path would be a bet on this situation being the new normal.
*   Grants for upskilling, training junior people, and recruitment make a lot less sense right now from grantmakers' perspective. 
*   For those applying for grants, asking for less money might make you more likely to be funded. (Historically, grantmakers consistently tell me that most people ask for less money than they should; I don't know whether that will change going forward, but now is an unusually probable time for it to change.)

Note that I am not a grantmaker, I'm just passing on what I hear from grantmakers in casual conversation. If anyone with more knowledge wants to chime in, I'd appreciate it.
# mental number lines

[This paper](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8946762/) says:

> it was shown that second-grade children unevenly placed numbers on a 0–1000 number line as their placement pattern was best fit by a logarithmic rather than by a linear function. The same age group placed numbers more linearly (i.e., resulting in a constant distance between consecutive numbers) on a 0–100 number line. In contrast, adults produced a linear distribution when placing numbers on a 0–1000 number line 

> Two developmental leaps occurring during elementary school were recognized: (1) the division of the number line into two segments and (2) consistent use of different anchor points on the number line—the left endpoint in first grade, the right endpoint in second grade, and finally the midpoint in third grade.

People mentally represent numbers as positions on a number line. I consider this a latent space representation. Human conceptual capabilities imply ability to at least average latent space representations, and possibly add them, which could then be used for fast approximation of sums. If only averaging is possible, then a 2-step (average -> double) process would be needed. The accuracy of that approach is insufficient for solving arithmetic problems, so people often instead use a much slower algorithm with exact accuracy.

Per the linked paper, 2nd-grade children have a tendency to place large numbers logarithmically, which implies to me an approximately logarithmic latent space representation. In that case, an accurate logarithmic representation should also be something humans are capable of with some training. In an accurate logarithmic representation, addition and subtraction are, of course, equivalent to multiplication and division.

Is it possible for people to do fast mental multiplications with the accuracy of a typical slide rule, by learning a logarithmic mental representation of numbers between 1 and 10? It seems plausible to me. Any big mental arithmetic fans around here?

But of course, the latent spaces humans typically use are high-dimensional, with limited range in any given dimension. It's possible that the "linear" number line in human mental representations is curved in its latent space to better utilize high dimensionality. In that case, averaging representations of distant numbers doesn't work so well.

Supposing that the mental number line is curved, how is it linearized when people write numbers on a line? My guess would be that it involves finding distances between the number embedding and the embeddings of nearby anchor points (like 1/2, 3/4, etc).

Some curled number lines would allow for better averaging of representations than others. Supposing that children learning arithmetic learn such curled line structures on their own, it could be more efficient to directly train them on multidimensional lower-precision representations of numbers that work relatively well for averaging/adding/doubling.
# Project Lawful Audiobook: An Unofficial Fan Production with ElevenLabs AI

[Eliezer Yudkowsky's latest fiction work](https://www.projectlawful.com/), (described [here](https://www.lesswrong.com/posts/SA9hDewwsYgnuscae/projectlawful-com-eliezer-s-latest-story-past-1m-words)) co-authored with Lintamande, is now complete save for the epilogue. For those following along, I've been producing an [unaffiliated audiobook](https://askwhocastsai.substack.com/) of "Project Lawful" (also known as "planecrash"), featuring a full AI cast from ElevenLabs. 

We're currently eight episodes into the story, with episode 9 due for release this coming Friday.
# Growth of Publicly Available Genetic Sequencing Data

The largest source of publicly available genetic sequencing data is the [Sequence Read Archive](https://en.wikipedia.org/wiki/Sequence_Read_Archive) (SRA), a joint project of the US ( [NCBI](https://en.wikipedia.org/wiki/National_Center_for_Biotechnology_Information)), Europe ( [EBI](https://en.wikipedia.org/wiki/European_Bioinformatics_Institute)), and Japan ( [DDBJ](https://en.wikipedia.org/wiki/DNA_Data_Bank_of_Japan)). Most relevant funding agencies and journals require sequencing data to be deposited in the SRA. I was curious how quickly it has been growing, so I ran some queries.

The metadata for the SRA is available [in the cloud](https://www.ncbi.nlm.nih.gov/sra/docs/sra-cloud-based-examples/) and we can access it through [BigQuery](https://cloud.google.com/bigquery). I ran:

  SELECT EXTRACT(YEAR from releasedate),
         EXTRACT(MONTH from releasedate),
         SUM(mbases),
         SUM(mbytes)
    FROM \`nih-sra-datastore.sra.metadata\`
GROUP BY EXTRACT(YEAR from releasedate),
         EXTRACT(MONTH from releasedate)

This gave me how much new data there was each month, in terms of both genetic bases and (compressed) bytes on disk.

[![](https://www.jefftk.com/sra-data-by-month.png)](https://www.jefftk.com/sra-data-by-month-big.png)

Note the logarithmic y-axis.

This is reminiscent of another chart, the cost to sequence 1M bases:

[![](https://www.jefftk.com/sequencing-cost-over-time-may-2022.png)](https://www.jefftk.com/sequencing-cost-over-time-may-2022-big.png)

(This is a pretty amazing chart, with the huge drop around 2008 coming from [NGS Sequencing](https://en.wikipedia.org/wiki/Massive_parallel_sequencing))

We could combine these, to get a rough estimate for how much money is being spent to sequence the data going into the SRA, but to do this we need to know how long a delay there is between sequencing and releasing: if it cost $400/Mb in 2007-10, $100/Mb in 2008-01, and $15/MB in 2008-04, then which cost should we use for interpreting data released in 2008-06? Here's a plot showing models 0-, 6-, 12-, and 24-month delays:

[![](https://www.jefftk.com/cost-of-sra-data-by-month.png)](https://www.jefftk.com/cost-of-sra-data-by-month-big.png)

It looks like maybe ~9m is the initial delay, and with costs changing more slowly in recent years it doesn't matter much for more recent data.

Looking just at the last five years, after it has leveled out some, it looks like a steady ~1.2e16 bases annually: [![](https://www.jefftk.com/sra-data-new-by-month-linear.png)](https://www.jefftk.com/sra-data-new-by-month-linear-big.png)

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0BVvekfDaMQrmXfvHJPMBa1Pq1xG63f9WCAbNzS6dwc1twGBQLRgtNU2L31Mf5ngBl), [mastodon](https://mastodon.mit.edu/@jefftk/110748131015350735)*