Harley-ml/lesswrong huggingface dataset:
# [Linkpost] Faith and Fate: Limits of Transformers on Compositionality

I thought it'd be especially interesting to get critiques/discussion from the LW crowd, because the claims here seem antithetical to a lot of the beliefs people here have, mostly around just how capable and cognizant transformers are/can be.

The authors show that transformers are guaranteed to suffer from compounding errors when performing any computation with long reasoning chains. 

From the abstract, "In an attempt to demystify Transformers, we investigate the limits of these models across three representative compositional tasks—multi-digit multiplication, logic grid puzzles, and a classic dynamic programming problem. These tasks require breaking problems down into sub-steps and synthesizing these steps into a precise answer. We formulate compositional tasks as computation graphs to systematically quantify the level of complexity, and break down reasoning steps into intermediate sub-procedures. Our empirical findings suggest that Transformers solve compositional tasks by reducing multi-step compositional reasoning into linearized subgraph matching, without necessarily developing systematic problem solving skills"
#  Palantir's AI models

Palantir published marketing material for their offering of AI for defense purposes. There's a video of how a military commander could order a military strike on an enemy tank with the help of LLMs. 

One of the features that Palantir advertises is:

> Agents
> 
> Define LLM agents to pursue specific, scoped goals.

Given military secrecy we are hearing less about Palantir's technology than we hear about OpenAI, Google, Microsoft and Facebook but Palantir is one player and likely an important one.
# June and Mulberries

Around here the [Juneberries](https://en.wikipedia.org/wiki/Amelanchier) and [Mulberries](https://en.wikipedia.org/wiki/Morus_(plant)) are ripe.

[![](https://www.jefftk.com/juneberries-june.jpg)](https://www.jefftk.com/juneberries-june-big.jpg)

The berries are delicious, but a lot of people don't know that they're safe to eat.

[![](https://www.jefftk.com/mulberries-june.jpg)](https://www.jefftk.com/mulberries-june-big.jpg)

I like the mulberries best when they still have a hint of red, before they get too sweet.

We've been walking over to the neighborhood tree with a stepladder every other day or so and enjoying them.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0uXuwLfmvLJYsrWwZYXMmwaDV9F2G2ia8UmExxG9cCkJM7JTcmRZM8FChTt1WDq1Al), [mastodon](https://mastodon.mit.edu/@jefftk/110556942383738147)*
# Alewife Train is Now Arriving

A few months ago [I complained](https://www.jefftk.com/p/attention-passengers-not-for-signs) that the text version of the MBTA train arrival announcements include a not-needed-for-text "attention passengers" which delayed the key information to the second screen. I'm happy to report that they've fixed it!

[![Alewife train is now arriving](https://www.jefftk.com/alewife-train-is-now-arriving.jpg)](https://www.jefftk.com/alewife-train-is-now-arriving-big.jpg)

While the vocal announcement is still "Attention passengers, the next Red Line train to Alewife is now arriving", the text version is the simple "Alewife train is now arriving". They've only fixed it for the second of the two announcements, when the train is arriving instead of when it's approaching, but it's still great!

(They actually changed this a few months ago, mid-April, but at the time I was waiting to see if they would fix it for the other one, and then forgot to write about it.)
# [Linkpost] Rosetta Neurons: Mining the Common Units in a Model Zoo

*This is a linkpost for* [*https://arxiv.org/abs/2306.09346*](https://arxiv.org/abs/2306.09346)*. *

> Do different neural networks, trained for various vision tasks, share some common representations?  
> In this paper, we demonstrate the existence of common features we call "Rosetta Neurons" across a range of models with different architectures, different tasks (generative and discriminative), and different types of supervision (class-supervised, text-supervised, self-supervised). We present an algorithm for mining a dictionary of Rosetta Neurons across several popular vision models:  
> Class Supervised-ResNet50, DINO-ResNet50, DINO-ViT, MAE, CLIP-ResNet50, BigGAN, StyleGAN-2, StyleGAN-XL.  
> Our findings suggest that certain visual concepts and structures are inherently embedded in the natural world and can be learned by different models regardless of the specific task or architecture, and without the use of semantic labels. We can visualize shared concepts directly due to generative models included in our analysis. The Rosetta Neurons facilitate model-to-model translation enabling various inversion-based manipulations, including cross-class alignments, shifting, zooming, and more, without the need for specialized training.
# The environment as infrastructure

A good metaphor for the ideal relationship between humanity and the environment is that **the environment is like critical infrastructure.**

Infrastructure is valuable, because it provides crucial services. You want to maintain it carefully, because it’s bad if it breaks down.

But infrastructure is there to serve us, not for its own sake. It has no intrinsic value. We don’t have to “minimize impact” on it. It belongs to us, and it’s ours to optimize for our purposes.

Infrastructure is something that can & should be *upgraded*, improved upon—as we often improve on nature. If a river or harbor isn’t deep enough, we dredge it. If there’s no waterway where we want one, we dig a [canal](https://rootsofprogress.org/the-13th-labor-of-hercules). If there is a mountain in our way, we blast a tunnel; if a canyon, we span it with a bridge. If a river is threatening to overflow its banks, we build a levee. If our fields don’t get enough water, we irrigate them; if they don’t have enough nutrients, we [fertilize](https://rootsofprogress.org/turning-air-into-bread) them. If the water we use for drinking and bathing is unclean, we [filter and sanitize it](https://rootsofprogress.org/draining-the-swamp). If mosquitoes are spreading disease, we eliminate them.

In the future, with better technology, we might do even more ambitious upgrades and more sophisticated maintenance. We could monitor and control the chemical composition of the oceans and the atmosphere. We could maintain the level of the oceans, the temperature of the planet, the patterns of rainfall.

The metaphor of environment as infrastructure implies that we should neither trash the planet nor leave it untouched. Instead, we should maintain and upgrade it.

(Credit where due: I got this idea for this metaphor [from Stewart Brand](https://twitter.com/stewartbrand/status/1275958025166008320); the elaboration/interpretation is my own, and he might not agree with it.)
# What is the foundation of me experiencing the present moment being right now and not at some other point in time?

This question may be either trivial to answer and I just don‘t get it, or it may be highly controversial.
I am trying to pose my problem as shortly as I can.

What is responsible for my subjective experience of the present moment (or rather, the immediate past, taking information processing into account)?
Why is now actually now?

Under the premise of spacetime being a static and eternal thing, doesn‘t any line of thought trying to answer this question necessarily make any intuitive notions of identity and the passing of time illusionary?

The only answer that I can come up with is that every conscious moment exists, despite any causal connections, for itself in some eternal sense. In this sense, „I“ right now am just existent at some specific point in spacetime, and before and after that there are, with regard to consiousness, separated and slightly different „mes“ which just happen to be in the closest proximity possible to the „me“ in that moment. Is there any more intuitive way of thinking about this?

Probably relates to timeless physics and quantum immortality.
# UK Foundation Model Task Force - Expression of Interest

Ian Hogarth has just been announced as the Chair of the UK's AI Foundation Model Taskforce. He's the author of the FT article "[We must slow down the race to God-like AGI](https://www.ft.com/content/03895dc4-a3b7-481e-95cc-336a524f2ac2)", and seems to take X-risks from AI seriously.  
  
To quote his twitter thread:

> And to that end I put out a call to people across the world. If you are an AI specialist or safety researcher who wants to build out state capacity in AI safety and help shape the future of AI policy then get in touch:  
>   
> We have £100m to spend on AI safety and the first global conference to prepare for. I want to hear from you and how you think you can help. The time is now and we need more people to step up and help.

The google form to leave an expression of interest is [here](https://docs.google.com/forms/d/e/1FAIpQLSd3OODvb-Nj1rhDQgx_-NSmAPk0348m13TlZr1BZy80AHzSrQ/viewform).

(I am in no way affiliated with Ian or the UK Foundation Model Task Force)
# When did you orient?

John David Pressman [says](https://twitter.com/jd_pressman/status/1670677402341761024) that the orienting is the tricky and valuable and underdocumented part of OODA. It occurs to me that I can hardly name any times anyone I know "oriented". When did you orient and what happened?
# Are the majority of your ancestors farmers or non-farmers? 

Assume humanity began with homo sapiens ~300,000 years ago. Are the majority of your ancestors farmers or non-farmers?   
  
My friend and I had very differing intuitions on this, and after thinking about it some, both of us currently think it's very non-obvious.
# Why didn't virologists run the studies necessary to determine which viruses are airborne?

If virology would be a reasonable scientific field I would expect them to care about the road viruses use to infect humans and other animals.

The experiment to determine whether or not a virus is airborne seems pretty straightforward to me. Get a bunch of animals and house them in a way where they would transmit the virus to each other if and only if the virus is airborne.

One hypothesis would be that virologists don't really care about understanding transmissions but care about registering patents from which they profit when drugs based on the patents get developed. 

People in the field of environmental health seemed to be able to know that COVID was airborne and wrote letters to the WHO saying so only to be ignored.

What's wrong with the virologists that they didn't manage to do the necessary science?
# Never Fight The Last War

Lieutenant Colonel J. L. Schley, of the Corps of Engineers wrote:

> “There is a tendency in many armies to spend the peace time studying how to fight the last war”

After the Chinchilla paper, everyone got focused on the idea that large language models can be more effective. 

The GPT3.5 we saw in ChatGPT was scary and got people to think about how we can prevent bigger models from being trained. There was a lot of capability gain when OpenAI released GPT4 but if we can trust Sam Altman, most of it was not because of an increase in model size but by doing a lot of different improvements. 

From that perspective, many people think about whether or not GPT5 or GPT6 will be scary and dangerous by improving along the same axis as the GPT3.5 ->GPT4 improvement.

Right now, it seems that most of the danger is not in model improvement but in model utilization. 

Palantir argues that models are commodities and released an AI product without providing any models that go beyond the existing open-source models. The value that Palantir does provide is that it provides interfaces of how a model can interact with the existing data a company has and how to create multi-step chains of data transformation. It provides a system to manage agents that use models to make decisions. 

I read some people argue that Palantir's product is worse than GPT4 and thus should not be seen as an advance in AI, but that's flawed thinking. Autonomous agents are more dangerous than anything you can do in ChatGPT.
# "textbooks are all you need"

"[Textbooks Are All You Need](https://arxiv.org/abs/2306.11644)" was published yesterday by Microsoft Research. It's the worst-named paper I've seen recently: it's not about textbooks, it's not all you need, and gratuitously imitating the title of [a paper](https://arxiv.org/abs/1706.03762) that introduced a different type of thing is dumb. But there's a reason I'm writing about it.

What they did was basically this:
1. started with The Stack (a 3 TB collection of code) and text from StackOverflow
2. used a LLM to select 6B "high-quality" tokens from (1)
3. used GPT-3.5 to generate 1B tokens of text similar to textbooks
4. trained a small (1.3B parameter) model ("phi-1") on (2) and (3)
5. used GPT-3.5 to generate text similar to textbook exercises
6. fine-tuned phi-1 on (5)
7. tested phi-1 on [HumanEval](https://paperswithcode.com/dataset/humaneval) to evaluate its programming ability

The results were pretty good, better than models 10x the size trained on 100x the data. So, it seems that scaling up isn't the only thing that matters, and data quality can be more important than data quantity or parameter count. (You hear that, gwern?)

Going by the listed OpenAI API prices, running GPT-3.5 on The Stack to evaluate quality would've been maybe ~$6M. What the authors did instead was:
1. Use GPT-4 to evaluate a small fraction of it.
2. Use a much smaller code-specific model to generate embeddings.
3. Use a classifier to predict which embeddings are from what GPT-4 evaluates as good content.

How about if you bootstrap a model using its own evaluation for filtering? One of the authors says "[I'm almost sure you can beat the teacher model](https://twitter.com/SebastienBubeck/status/1671384044368707586)" and I agree. That can give you recursive self-improvement of a type you see in both individual people and the culture of societies. People develop better taste and consume better content which makes them smarter so they develop better taste, and [so on](https://www.youtube.com/watch?v=51GIxXFKbzk). Children hear the stories their grandfathers like, and culture develops.

That's a weak sort of self-improvement, which tends to plateau for people. Humans do other things too, so by itself it's weaker self-improvement than it appears to be for people. This is a technique I previously spent some time thinking about, so there are some other reasons I think it tends to plateau by itself. But still - recursive self-improvement!

Yes, in theory, if you have a much bigger model trained on a bigger dataset including the good selected data, and you can engineer prompts such that you get into a mode that models the good data specifically, then you can get the same results. In that sense, the performance reachable with this method is limited to what's possible from model scaling plus prompt engineering. The amount of scaling needed for that seems to be potentially 100x, and getting into exactly the right mode with prompt engineering might be impractical, but still, that provides some rough limits on potential here.
# EU AI Act passed Plenary vote, and X-risk was a main topic  

(mildly rewritten version of my [EA forum](https://forum.effectivealtruism.org/posts/ctPrrzFnXGyWrmK3w/eu-ai-act-passed-vote-in-plenary-meeting) post)

The EU AI Act was originally proposed in 2020 as a very "EU regulates stuff first" kind of legislation, trying to make sure EU values are upheld (fairness, transparency, democracy, etc). Several revisions (and some [lobbying](https://time.com/6288245/openai-eu-lobbying-ai-act/)) later, GPAI (general purpose AI) and foundation model language was added, and it started looking a little more X-risk friendly. 

After some [recent political uncertainty](https://www.euractiv.com/section/artificial-intelligence/news/ai-acts-plenary-vote-cast-with-uncertainty-as-political-deal-crumbles/), it passed with a strong majority at the EU Plenary meeting. 

I found it fascinating to watch the the live session (from June 13th, the vote was on the 14th), where the Act was discussed by various EU parties. A few things that stood out to me:

*   I was surprised that many EU country representatives mentioned the Open Letters and Existential Risk as a real concern, even though the EU AI Act was not originally intended to address it (though, it now has GPAI/foundation model bits added). Transparency and Fairness took a back seat, to some extent.
*   Real-time Biometric monitoring was a big debate topic - whether to giving an exemption for law enforcement or not, for national security. Currently it looks like it will not be allowed, other than post-incident with special approval. This may be a useful lever to keep in mind for policy work

Others that watched the stream, feel free to mention insights in the comments. 

> Linked [here](https://multimedia.europarl.europa.eu/en/webstreaming/event_20230613-0900-PLENARY?start=230613070143&end=230613123309&) (relevant timestamp 12:39 - 14:33)

With the recent appointment of [Ian Hogarth to the UK Foundation Model taskforce](https://www.gov.uk/government/news/tech-entrepreneur-ian-hogarth-to-lead-uks-ai-foundation-model-taskforce), and US talks of regulation getting stronger, I think we are in for interesting times. But it also seems like AI X-risk is a lot more mainstream, which I did not expect to be able to say.
# Newbie questions about information theory and transformers

I have some very naive (and unfortunately sort of stream-of-consciousness) questions brought up by this post:

  
https://www.lesswrong.com/posts/vAAneYowLkaHnihCg/textbooks-are-all-you-need  
 

I definitely know that my ideas are poorly formed and I have no idea what I'm talking about. I'm happy to be told that my whole picture here is fundamentally flawed, or that there's some better question that I should be asking, or that a combination of words I've thrown together is gibberish. But I'm trying to avoid worrying about asking dumb things when I'm curious, so I'm just going to put them out there.

Is there much I can read on the relationship between information theory and transformers/LLMs? 

I keep trying to picture the flow of information in transformers. Is the information being "distilled"? 

I'm imagining the weights and biases of the model to be some kind of distilled and encoded version of the training data. Does that make any sense? Since it can't store all of it, what is it doing? Incorporating more information from the tokens in the training data that have the highest entropy (as far as how much it reduces the average uncertainty of the correct next token)? 

If so, is this driven by gradient descent? Does minimizing the cost function end up encoding more information from the training set tokens that contain the most entropy into the weights of the model? 

Does that information get further distilled into the tokens it generates from a prompt? Is there some way in which having the model generate a bunch of new training data is "unencoding" the information about the original training set stored in the weights, and in doing so producing a new, smaller training set that represents the highest entropy information from the original? 

Does this improve quality because it's letting the network try "reconnecting" itself when it's re-encoded into a new model being trained on it? Could this help fix things that were stuck in a local minima in the original model?

When the LLM selects tokens from the large set, does that choice of tokens encode information from that LLM's training in addition to the information from the selected tokens themselves?

If we start caring a lot about the ability of an LLM to select the best tokens from a huge dataset and generate a complimentary set of synthetic data on top of its general functionality, would this create some kind of selective pressure that could evolve? Like... the reward function being based on the improvement in loss from the LLM to its "offspring" and the cost function being based on how big the training set chosen was or how many FLOPS went into training the offspring.
# How to Search Multiple Websites Quickly

Imagine you google the word "eigenvalue". But instead of one list of search results, you automatically get a bunch of new browser tabs. In each tab, is *the page about eigenvalues* on each of multiple whitelisted sites. Wikipedia, nLab, Wolfram Mathworld, reddit, and LessWrong, for example.

Does this sound useful? Like it could e.g. give you [multiple explanations of the same topic](https://www.lesswrong.com/posts/cP7oiw7psZBHTFjG7/dreams-of-mathopedia?commentId=eWoiYbJxwykKR2ZDE), all gathered together, quickly?

Turns out, this functionality exists.

Multiple browsers have extension "app stores", and one type of extension is a "multisearch" or a "multi tab search".

## A quick example

I found [this extension](https://chrome.google.com/webstore/detail/instant-multi-search/aamgapdgopfdmokckpkfciiddpahbbcg) for Chrome. There are almost certainly similar ones for Firefox and other browsers. The below instructions are for that specific extension.

To use it:

1. Install the extension.

2. Click the extension's icon in your browser, and go to its "Options".

3. Enter URLs to search, with the "%s" operator being what you'd put your query into.

An example category contains a list of sites I want to multi-tab-search about 

```
[Google Feeling Lucky] brilliant.org %s
https://en.wikipedia.org/w/index.php?title=Special:Search&search=%22%s%22&ns0=1&fulltext=Search
[Google Feeling Lucky] ncatlab.org %s
[Google Feeling Lucky] wolfram mathworld %s
[Google Feeling Lucky] khan academy %s
[Google Feeling Lucky] proofwiki.org %srg %s
```

(Note: the "[Google Feeling Lucky]" option unfortunately doesn't actually open the first search-result, it just googles it for you)

4. Open a new tab, press "m" and then "space" before typing your query. (For this extension, you need to type "m" before you type another custom key for a different "category". So if you use "x" for a "rationality" category, you'd press m-space and then type "x orthogonality thesis" to multi-tab-search for the orthogonality thesis.)

5. Lots of new tabs will open, one for each site!

This is much easier to setup, but less useful/complete, than my "multi-explanation math wiki" [idea](https://www.lesswrong.com/posts/cP7oiw7psZBHTFjG7/dreams-of-mathopedia). Consider it an 80/20 minimum-viable-setup for verifying the "multiple explanations = better understanding" idea.

If you find this useful, you might leave multi-site query lists that you ended up using, in the comments.
# The Hubinger lectures on AGI safety: an introductory lecture series

In early 2023, I ([Evan Hubinger](https://www.alignmentforum.org/users/evhub)) gave a series of recorded lectures to [SERI MATS](https://www.serimats.org) fellows with the goal of building up a series of lectures that could serve as foundational introductory material to a variety of topics in AGI safety. Those lectures have now been edited and are [available on YouTube](https://www.youtube.com/playlist?list=PLtlVeM84bZ6RLSR6oaQnbZ7FSwb-hkapx) for anyone who would like to watch them.

The basic goal of this lecture series is to serve as longform, in-depth video content for people who are new to AGI safety, but interested enough to be willing to spend a great deal of time engaging with longform content, and who prefer video content to written content. Though we already have [good introductory shortform video content](https://www.youtube.com/@RobertMilesAI) and [good introductory longform written content](https://www.alignmentforum.org/s/mzgtmmTKKn5MuCzFJ), the idea of this lecture series is to bridge the gap between those two.

Note that the topics I chose to include are highly opinionated: though this is introductory material, it is not intended to introduce the listener to every topic in AI safety—rather, it is focused on the topics that I personally think are most important to understand. This is intentional: in my opinion, I think it is far more valuable to have some specific [gears-level](https://www.lesswrong.com/tag/gears-level) model of how to think about AI safety, rather than a shallow overview of many different possible ways of thinking about AI safety. The former allows you to actually start operationalizing that model to work on interventions that would be valuable under it, something the latter doesn't do.

The lecture series is composed of six lectures, each around 2 hours long, covering the topics:
1. Machine learning + instrumental convergence
2. [Risks from learned optimization](https://www.alignmentforum.org/s/r9tYkB2a8Fp4DN8yB)
3. [Deceptive alignment](https://www.alignmentforum.org/posts/A9NxPTwbw6r6Awuwt/how-likely-is-deceptive-alignment)
4. [How to evaluate alignment proposals](https://www.alignmentforum.org/posts/FDJnZt8Ks2djouQTZ/how-do-we-become-confident-in-the-safety-of-a-machine)
5. LLMs + [predictive models](https://www.alignmentforum.org/s/n3utvGrgC2SGi9xQX)
6. [Overview of alignment proposals](https://www.alignmentforum.org/posts/fRsjBseRuvRhMPPE5/an-overview-of-11-proposals-for-building-safe-advanced-ai)

Each lecture features a good deal of audience questions both in the middle and at the end, the idea being to hopefully pre-empt any questions or confusions the listener might have.

[The full slide deck for all the talks is available here.](https://docs.google.com/presentation/d/1GnmFcmbIUFLs5mvhhDKXoraskreCT65xH0Lw0V1Fbps/edit?usp=sharing)

_EDIT: As a follow-up to the above lecture series covering some of my more recent work and thoughts on deceptive instrumental alignment, I would also recommend [this talk I gave at FAR](https://t.co/FNYtypUeWe)._
# Can you prevent negative long-term effects of bad trips with sleep deprivation?

The idea is that you need sleep in order to consolidate your memories. If you deprive yourself of that after a bad trip on psychedelics, or after a traumatic event or re-traumatization, the newly formed bad memories don't consolidate.

I’ve heard that people are way less likely to get PTSD after a traumatic event if they experience strong sleep deprivation afterwards - as has happened to soldiers in war for example. I’ve also heard in conversations that this works in rats that are subjected to sleep deprivation after being put under traumatic stress.

If this were true, wouldn't this be an amazing hack to make psychedelic trips way safer? Now, I know that sleep deprivation is not easy and very unpleasant and probably the last thing you want to do after a bad trip - but it's a much less bad option compared to long-term negative effects of bad trips. And uppers like caffein, modafinil or amphetamines would make it easy to do.

I'd personally find it very nice if this option exists as a fallback plan B. For me, it would take away the “fear of fear”, or the "fear of a bad trip", because I would know that the worst thing that can happen is that I have a really bad time and maybe feel destabilized for a few weeks after, but there won't be any serious negative long-term effects - because I can always just skip a day or two of sleep afterwards!  
 

Does anyone know if this would work?
# Bounded surprise exam paradox

A small fun thing I came up with this morning.

The surprise exam paradox goes as follows: "you'll be given a surprise exam on one of the days next week, and you won't figure out the date of the exam on the previous day". Then you figure that the exam can't be on Sunday as it's the last day and you'd know it on Saturday, so it can't be on Saturday and so on, ruling out all days. Then the exam happens on Wednesday and you're surprised.

The standard solution, which I agree with, is to interpret it as a self-referential sentence S = "there is a natural number X from 1 to 7, such that from knowing any lower bound on X and also knowing S, it's impossible to prove the value of X". This isn't a liar kind of paradox, because the self-reference in S only talks about provability, not truth. So we can take any formal theory that's strong enough to talk about provability within itself (same conditions as Gödel's theorems), convert S into a non-self-referential sentence by using the diagonal lemma, and observe that S is false. And from a falsehood anything follows, so the student's conclusions are only the student's problem.

It's a satisfying solution, but there's a wrinkle. What if we interpret the teacher's statement as being not about provability, but as a prediction of the future? Not "there's no proof", but "you specifically, as a deterministic algorithm running in finite time, will not find a proof"?

We can encode that by using two self-references instead of one. The student will be a program P, which on the evening of each day will enumerate all proofs in Peano arithmetic up to N symbols long, for some large N. The teacher's statement S will be: "The program P, quoted here in full, won't on any day find a proof that {S + knowledge of that day} implies the exam is the next day." All self- and mutual references can again be resolved by using the diagonal lemma. So P will either find a proof or won't. Which is it?

Well, if N is large enough, P can prove that the exam can't happen on the last day, then on the day before and so on, by the same method as in the original paradox. So on the very first day P will prove that S is false, and also that S implies the exam will be on Monday, and S implies the exam will be on Tuesday, all of that at the same time. In other words, even if the student is a bounded deterministic program without logical omniscience, the teacher is still lying. The student will still prove from the teacher's words that the exam will happen on a specific day, and also on another specific day, and also the moon is made of green cheese.
# Epistemic spot checking one claim in The Precipice

On page 221 of The Precipice, Toby Ord says "scientists expect the brightening of the sun to also slow Earth's plate tectonics, dampening volcanic activity". This seemed very weird to me; how could sunlight possibly have any effect on mantle movement?

The book didn't provide any direct citations for this statement, but a related footnote mentions Kellogg (2013), Wolf & Toon (2015), Leconte et al. (2013), and Rushby et al. (2018).

I couldn't find Kellogg. [Wolf & Toon](https://agupubs.onlinelibrary.wiley.com/doi/full/10.1002/2015JD023302) doesn't mention anything about this. There were multiple possible papers for Leconte et al., none of which seemed relevant. The most promising one was [Rushby et al.](https://arxiv.org/pdf/1805.06943.pdf), which mentions:

> The importance of mantle hydration as a mechanism for facilitating and maintaining plate tectonics over geological time remains unclear (Korenaga, 2010; Korenaga, 2012). It is possible that a drier mantle and lower associated viscosity will make the initiation of plate tectonics easier, but that maintaining a continental crust may require interaction between the hydrosphere and solid Earth over geological time.

Korenaga (2010) does mention this in more detail, saying:

> a mechanism based on the presence of surface water has the advantage of being able to readily explain the absence of plate tectonics on other terrestrial planets in the solar system.

Korenaga (2012) doesn't seem to be available online, but the abstract mentions the same thing. And doing some Googling turns up papers like [this one](https://www.researchgate.net/publication/234526068_The_role_of_liquid_water_in_maintaining_plate_tectonics_and_the_regulation_of_surface_temperature), which says:

> In the ductile creep regime, the viscosity of wet rocks is weaker than the viscosity of dry rocks by several orders of magnitude. In the brittle regime, the most substantial effect is probably serpentinization which can reduce the friction coefficient by a factor of 2 or more. The difference between the strength of a wet lithosphere and that of a dry lithosphere seems to be big enough to control the very existence of plate tectonics. Because of dehydration due to partial melting the oceanic lithosphere is expected to be essentially dry above some critical depth, around 60-80 km. This would make the lithosphere strong enough to prevent plate motion. Percolation of water from the surface can be the main mechanism supplying water to the upper parts of the lithosphere. This implies that liquid water can be crucial for maintaining plate tectonics.

And many others that confirm this picture, like [The fate of water within Earth and super-Earths and implications for plate tectonics](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5394257/) and [this weird study/article hybrid from ScienceDaily](https://www.sciencedaily.com/releases/2018/11/181114132013.htm).

So surprisingly enough this does seem to be a real thing.
# Carl Shulman on The Lunar Society (7 hour, two-part podcast)

Dwarkesh's summary for part 1:

> In terms of the depth and range of topics, this episode is the best I’ve done.
> 
> No part of my worldview is the same after talking with [Carl Shulman](http://reflectivedisequilibrium.blogspot.com/). He's the most interesting intellectual you've never heard of.
> 
> We ended up talking for 8 hours, so I'm splitting this episode into 2 parts.
> 
> This part is about Carl’s model of an intelligence explosion, which integrates everything from:
> 
> *   how fast algorithmic progress & hardware improvements in AI are happening,
> *   what primate evolution suggests about the scaling hypothesis,
> *   how soon before AIs could do large parts of AI research themselves, and whether there would be faster and faster doublings of AI researchers,
> *   how quickly robots produced from existing factories could take over the economy.
> 
> We also discuss the odds of a takeover based on whether the AI is aligned before the intelligence explosion happens, and Carl explains why he’s more optimistic than Eliezer.
> 
> The next part, which I’ll release next week, is about all the specific mechanisms of an AI takeover, plus a whole bunch of other galaxy brain stuff.
> 
> Maybe 3 people in the world have thought as rigorously as Carl about so many interesting topics. So this was a huge pleasure.

Dwarkesh's summary for part 2:

> The second half of my 7 hour conversation with [Carl Shulman](http://reflectivedisequilibrium.blogspot.com/) is out!
> 
> My favorite part! And the one that had the biggest impact on my worldview.
> 
> Here, Carl lays out how an AI takeover might happen:
> 
> *   AI can threaten mutually assured destruction from bioweapons,
> *   use cyber attacks to take over physical infrastructure,
> *   build mechanical armies,
> *   spread seed AIs we can never exterminate,
> *   offer tech and other advantages to collaborating countries, etc
> 
> Plus we talk about a whole bunch of weird and interesting topics which Carl has thought about:
> 
> *   what is the far future best case scenario for humanity
> *   what it would look like to have AI make thousands of years of intellectual progress in a month
> *   how do we detect deception in superhuman models
> *   does space warfare favor defense or offense
> *   is a Malthusian state inevitable in the long run
> *   why markets haven't priced in explosive economic growth
> *   & much more
> 
> Carl also explains how he developed such a rigorous, thoughtful, and interdisciplinary model of the biggest problems in the world.

*(Also discussed on the EA Forum* [*here*](https://forum.effectivealtruism.org/posts/iKWeQ7jhsZ8FrRDto/linkpost-dwarkesh-patel-interviewing-carl-shulman)*.)*
# Request: Put Carl Shulman's recent podcast into an organized written format

Carl Shulman recently did a very long interview with Dwarkesh Patel on the Lunar Society Podcast, which Dwarkesh released in two separate episodes because of its length. I have seen many people point to this interview as adding a lot of important high value content to the AI x-risk discussion, and I agree with that assessment. However, I don't think it's fair to tell people they need to listen to a 7 hour interview in order to get a better sense of concrete risk scenarios.

I therefore think it would be very high value if someone could organize the information that Carl presented into an accessible written format so that it can be used as a reference going forward. I think it would be even better if Carl could publish it on arxiv or in some relevant journal so that it could get the academic attention it deserves.
# What is your financial portfolio?

It seems plausible that we'll experience massive economic growth within the next decade due to advances in AI. How are you all taking advantage of this? What companies are you investing in, what assets are you holding onto? Are you not investing in some orgs as you think it would be counterproductive? Why?
# AGI x Animal Welfare: A High-EV Outreach Opportunity?

Epistemic status: *Very quickly written, on a thought I've been holding for a year and that I haven't read elsewhere. *

I believe that within this decade, there could be AGIs (Artificial General Intelligences) powerful enough that the values they pursue might have **a value lock-in effect**, at least partially. This means they could have a **long-lasting impact** on the **future values and trajectory** of our civilization (assuming we survive).

This brief post aims to share the idea that if your primary focus and concern is animal welfare (or digital sentience), you may want to consider engaging in targeted outreach on those topics towards those who will **most likely shape the values of the first AGIs**. This group likely includes executives and employees in top AGI labs (e.g. OpenAI, DeepMind, Anthropic), the broader US tech community, as well as policymakers from major countries.

Due to the risk of lock-in effects, I believe that the values of **relatively small groups of individuals** like the ones I mentioned (less than 3 000 people in top AGI labs) might have a disproportionately large impact on AGI, and consequently, on the future values and trajectory of our civilization. My impression is that, generally speaking, these people currently 

a) **don't prioritize animal welfare significantly **

b) **don't show substantial concern** for **digital minds** sentience. 

Hence if you believe those things are very important (which I do believe), and you think that AGI might come in the next few decades[^0108cdk5rlvjk] (which a majority of people in the field believe), you might want to consider this intervention.

Feel free to reach out if you want to chat more about this, either here or via my contact you can find [here](https://www.simeon.ai/).

[^0108cdk5rlvjk]: Even more so if you believe, as I do along with many software engineers in top AGI labs, that it could happen this decade.
# AXRP announcement: Survey, Store Closing, Patreon

Some announcements:

*   There’s a brief [survey](https://bit.ly/axrpsurvey2023) of my audience you can take here - filling it out will help me get a sense of where you all are at career-wise and how the podcast would better serve you. There are only 3 required questions, and they’re multiple choice! It would be great if you could fill it out [here](https://bit.ly/axrpsurvey2023).
*   The AXRP [store](https://store.axrp.net) is closing in a week - so if you want to buy merchandise, now’s your last chance to do so! To buy merch like laptop stickers, t-shirts, sweaters, or jigsaw puzzles (in the US), go to the store [at this link](https://store.axrp.net).
*   If you want to financially support the podcast, you can do so in two ways:
    *   On [Patreon](https://www.patreon.com/axrpodcast) \- donations of US$20 per month get to see my episode notes, and at US$50 per month, you have a (currently very high) chance of being thanked by name at the end of episodes! Link is [here](https://www.patreon.com/axrpodcast).
    *   On [Ko-fi](https://ko-fi.com/axrpodcast), where you can give one-off donations.
*   Interviews will always be free and available to non-subscribers, but your support gives me a better sense of how many people get value out of this podcast, and how much time I should spend on it.
# Bruce Sterling on the AI mania of 2023

Bruce Sterling is a well-known SF writer, but I think he is not quite on the radar of Less Wrong, so I'll say a bit about how I locate him culturally. I'll make a distinction between SF that is primarily about marvels of science and technology, and SF that is primarily about how it feels to inhabit technological society. I consider Sterling to be a leading practitioner of that second kind of SF. He wrote one novel, the posthuman space opera *Schismatrix*, in which he indulged his inner Singularitarian to the maximum; but after that, his works have been more earthbound, though still about the transformations of life that could be wrought by technology. 

He's also been prolific as an essayist and public speaker, still offering commentary on technological society and world affairs, but in a supposedly factual vein, rather than an overtly fictional one. I admire his talent for digesting these complex topics and producing an intuitive summation in a few pithy turns of phrase; I consider him a role model in that regard. And so now he's trying to sum up the state of the AI world, after ChatGPT. 

He aims for an indulgent, god's-eye view of the current-year antics of everyone involved in AI. He acknowledges the prophets of doom, the social justice warriors, the captains of industry. He nominates three myths as encapsulating the times: Roko's Basilisk, the Masked Shoggoth, and the Paperclip Maximizer. Every one of them comes from within the rationalist sphere, or adjacent to it, so we're doing pretty well when it comes to mythic potency! 

But he does believe that after this carnival of ecstasy and dread, there will be a hangover rather than a singularity. I suspect he's the kind of guy who thinks that superhuman intelligence might well be possible, but only decades from now, after still-unknown paradigm shifts, and that language models are a false dawn in that regard. So when the next AI winter comes, we'll still be here, and we'll still be human beings, but we'll also be living with all the consequences of having created glib, multitalented, relentless artificial persons. 

My own bet is that this isn't a false dawn, that for better or worse, we're now directly on course towards artificial intelligence that comprehensively transcends human abilities. But I appreciate Sterling's role as skeptical yet open-minded observer.
# I Think Eliezer Should Go on Glenn Beck

Glenn Beck is the only popular mainstream news host who takes AI safety seriously. I am being entirely serious. For those of you who don't know, Glenn Beck is one of the most trusted and well-known news sources by American conservatives. 

Over the past month, he has produced two hour-long segments, one of which was an interview with AI ethicist Tristan Harris. At no point in any of this does he express incredulity at the ideas of AGI, ASI, takeover, extinction risk, or transhumanism. He says things that are far out of the normie Overton Window, with no attempt to equivocate or hedge his bets. "We're going to cure cancer, and we're to do it right before we kill all humans on planet Earth". He just says things like this, with complete sincerity. I don't think anyone else even comes close. I think he is taking these ideas seriously, which isn't something I expected to see from anyone in mainstream media.

According to Glenn, he has been trying to get interviews with people like Geoffery Hinton, but they have declined for political reasons. An obvious low-hanging fruit, if MIRI is willing to send him an email.

He seems on board with a unilateral AI pause from the US for national security reasons. 

He's also personally cited statements from Eliezer to convey the dangers of ASI. I think an interview between the two could be a way for AI alignment to be taken more seriously by retirees who will vote and write their congressman. I dislike Eliezer on a personal level, but I think he is the only person who will actually go on the show and express how truly *dire* the situation is, with full sincerity.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/bd60384435db34f9386c5a129d7ba5b9ab3027ac3206c607.png)
# Elements of Computational Philosophy, Vol. I: Truth

We're excited to share the first volume of [*Elements of Computational Philosophy*](https://compphil.github.io/), an interdisciplinary and collaborative project series focused on operationalizing fundamental philosophical notions in ways that are natively compatible with the current paradigm in AI.

The first volume paints a broad-strokes picture of operationalizing *truth* and *truth-seeking*. Beyond this high-level focus, its 100+ pages can be framed in several different ways, which is why we placed multiple topic-based summaries at the beginning of the document. The note to the reader and the table of contents should further help scope and navigate the document.

Have a pleasant read, and feel free to use this linkpost to comment on the document as you go. Questions, criticism, and suggestions are all welcome.

PS: There will soon be a presentation about the overarching project series as part of the alignment speaker series hosted by EleutherAI. Expect more information soon on the #announcements channel of their Discord server. In general, keep an eye on this space.
# [Linkpost] A shared linguistic space for transmitting our thoughts from brain to brain in natural conversations

*This is a linkpost for* [*https://www.biorxiv.org/content/10.1101/2023.06.27.546708v1*](https://www.biorxiv.org/content/10.1101/2023.06.27.546708v1)*.* 

> Effective communication hinges on a mutual understanding of word meaning in different contexts. The embedding space learned by large language models can serve as an explicit model of the shared, context-rich meaning space humans use to communicate their thoughts. We recorded brain activity using electrocorticography during spontaneous, face-to-face conversations in five pairs of epilepsy patients. We demonstrate that the linguistic embedding space can capture the linguistic content of word-by-word neural alignment between speaker and listener. Linguistic content emerged in the speaker's brain before word articulation, and the same linguistic content rapidly reemerged in the listener's brain after word articulation. These findings establish a computational framework to study how human brains transmit their thoughts to one another in real-world contexts.