Harley-ml/lesswrong huggingface dataset:
# Hiatus: EA and LW post summaries

For the past ~8 months, I've been summarizing the top posts on the EA and LW forums each week (see archive [here](https://forum.effectivealtruism.org/s/W4fhpuN26naxGCBbN)), a project supported by Rethink Priorities (RP).  
  
I’ve recently taken on a new role as an AI Governance & Strategy Research Manager at RP. As a result of this we're going to be **putting the forum summaries on hiatus** while we work on what they should look like in the future, and hire for someone new to run the project. A big thank you to everyone who completed our recent survey - it’s great input for us as we evaluate this project going forward!

**The hiatus will likely last for ~4-6 months**. We’ll continue to use the existing [email list](https://easummaries.substack.com/) and podcast channel (EA Forum Podcast (Summaries)) when it's back up and running, so subscribe if you’re interested and feel free to continue to share it with others.

If you’re looking for other ways to stay up to date in the meantime, some resources to consider:

**Newsletters**

[The EA Forum Digest](https://forum.effectivealtruism.org/posts/zRoXctENAsLchwryn/sign-up-for-the-forum-s-email-digest) \- a weekly newsletter recommending new EA forum posts that have high karma, active discussion, or could use more input.

[Monthly Overload of Effective Altruism](https://moea.substack.com/) \- a monthly newsletter with top research, organizational updates and events in the EA community.  
  
[The EA Newsletter](https://www.effectivealtruism.org/ea-newsletter-archives) \- a monthly newsletter with news and updates about the effective altruism community, a selection of top posts, and highlights of progress and discussion in different cause areas.

**Podcasts**

[EA forum podcast (curated posts)](https://forum.effectivealtruism.org/posts/K5Snxo5EhgmwJJjR2/announcing-ea-forum-podcast-audio-narrations-of-ea-forum) \- human narrations of some of the best posts from the EA forum.

[Nonlinear library](https://forum.effectivealtruism.org/posts/JTZTBienqWEAjGDRv/listen-to-more-ea-content-with-the-nonlinear-library) \- AI narrations of all posts from the EA Forum, Alignment Forum, and LessWrong that meet a karma threshold.

There are heaps of cause-area specific newsletters out there too - if you have suggestions, please share them in the comments.  
 

I’ve really enjoyed my time running this project! Thanks for reading and engaging, to Coleman Snell for narrating, and to all the writers who’ve shared their ideas and helped people find new opportunities to do good.
# Possible AI “Fire Alarms”

Eliezer says that there is no fire alarm for AGI and I agree that there’s no single event that would wake everyone up. On the other hand, there are a lot of possible events that could cause a significant shift in the Overton window or wake a lot of people. Since this seems strategically important, I spend ten minutes brainstorming a list. Feel free to dispute any of these or add further possibilities in the comments:

- Global protests with massive media coverage
- Protest of people with ML PhDs or current students
- Experimental evidence of deceptive alignment
- Scary demo
- Accident from auto-GPT - most likely hacking
- Self-driving cars being hacked
- High prestige open letter
- International conference on AI
- High profile documentary
- Movie that captures that public attention - not just any movie
- Active push from prestigious ML figure (more than another Stuart Russel)
- Active push from a thought leader - say Obama
- Active push from a political leader - say UK/US prime minister
- Passing the Turing test
- UN or major government declaring a crisis
- Formation of a government department in a major country focused on alignment
- Statement by an official computing body
- High profile critic publicly changing their stance
- Billboards in Times Square (followed by media coverage)
- A popular or heavily promoted book (similar levels of promotion to What We Owe The Future)

Note: Some of these things are not things people should intentionality cause to happen (Ie. An accident from auto-gpt). Of the things that might be worth trying to cause happen, they could backfire significantly if done badly. Think carefully before pursuing any of these. Even though timelines are short, you should be about to find a spare hour to think about how things could go wrong.
# The Unexpected Clanging

There are two boxes in front of you. In one of them, there is a little monkey with a cymbal, whilst the other box is empty. In precisely one hour the monkey will clang its cymbal.

While you wait, you produce an estimate of the probability of the monkey being in the first box. Let's assume that you form your last estimate, p, three seconds before the monkey clangs its cymbal. You can see the countdown and you know that it's your final estimate, partly because you're slow at arithmetic.

Let Omega be an AI that can perfectly simulate your entire deliberation process. Before you entered the room, Omega predicted what your last probability estimate would be and decided to place the monkey in a box such as to mess with you. Let q be the probability of Omega placing the monkey in the first box. In particular, Omega, sets q=p/2, unless p=0 or you haven't formed a probability estimate, in which case q=1. 

What probability should you expect that the monkey is in the first box?

* * *

I think it's fairly clear that this is a no-win situation. No matter what the final probability estimate you form before clanging, as soon as you've locked it in, you know that it is incorrect, even if you haven't heard the clanging yet. [You can try to escape this, but there's no reason that the universe has to play nice](https://www.lesswrong.com/posts/PZGzZgP2NtgME8pSY/the-universe-doesn-t-have-to-play-nice).

This problem can be seen as a variation on [Death in Damascus](https://intelligence.org/2017/03/18/new-paper-cheating-death-in-damascus/#:~:text=%E2%80%9CDeath%20in%20Damascus%E2%80%9D%20is%20a,Damascus%20or%20flee%20to%20Aleppo.). I designed this problem to reveal that the core challenge Death in Damascus poses isn't just that another process in the world can depend upon your decision, but that it can depend upon your expectations even if you don't actually make a decision based upon those expectations.

I also find this problem as a useful intuition pump as I think it's clearer that it's a no-win situation than in other similar problems. In Newcomb's problem, it's easy to get caught up thinking about the Principle of Dominance. In Death in Damascus, you can confuse yourself trying to figure out whether CDT recommends staying or fleeing. At least to me, in this problem it is clearer it is a dead end and that there's no way to beat Omega.

This is also a useful intuition pump for the [Evil Genie Puzzle](https://www.lesswrong.com/posts/YSEtEtqf8hRBhKBS9/the-evil-genie-puzzle). When I first discovered this puzzle, I felt immensely confused that no matter which decision that you made you would immediately regret it. However, the complexity of the puzzle made it complicated for me to figure out exactly what to make of it, so when trying to solve it I came up with this problem as something easier to grok. I guess my position after considering the Unexpected Clanging is that you just have to accept that a sufficiently powerful agent may be able to mess with you like this and that you just have to deal with it. (I'll leave a more complete analysis to a future post).
# How could I measure the nootropic benefits testosterone injections may have?

I came across a post written by Gwern in which he says he never injected Testosterone before but is curious to know more about its potential nootropic benefits. I'm suspecting that its more people than just him who are curious about that soo why not experiment myself?

I wanted to "hop on another cycle of gear" (use steroids for 16-20  wks and restore my natural hormonal production right after) anyway but don't know anything about experiment design so if someone wants to help me out with that that would be pretty cool.
# What's the best way to streamline two-party sale negotiations between real humans?

Some types of negotiations are strategyproof; designed such that the optimal strategy is for each player to be truthful. For example in a Vickrey auction, there's no incentive to lie or bid less than your maximum; doing so would only put you at a disadvantage.

Unfortunately, when it comes to negotiations between a single buyer and a single seller, it's been proven that there is no strategyproof solution. (See [Lying in negotiations: a maximally bad problem](https://www.lesswrong.com/posts/5GTBktYp2g3i2BwFd/lying-in-negotiations-a-maximally-bad-problem).) The seller is always incentivized to overrepresent the value of the item, and the buyer to underrepresent it. This can lead to brinksmanship, where both parties try to set a firm "take it or leave it" price in order to force the other party to accept, at the risk of no deal occurring at all.

Ideally, the correct price at which to sell the item is the price that maximizes utility across both players. But when it comes to real humans in the real world, it's very easy for one to lie about their own utility curve, so there's no good way for both parties to enforce this.

The typical way humans go about these negotiations this is with emotional manipulation, extortion, [artificial self-restrictions](https://www.lesswrong.com/posts/vPW6qPcjsWoQ6LhHS/principals-agents-negotiation-and-precommitments), social coercion, etc. (e.g. think of the stereotypical car salesman.) This seems generally bad for epistemics, and as someone who has to negotiate a lot, I also find it personally very annoying.

I'd like to design a system that allows these negotiations to take place in a more incentive-compatible way that's faster to execute, doesn't reward skill at manipulating other people, and is less likely to lead to bad feeling afterwards. Obviously it can't be perfect and there will be *some* method of gaming the system, but humans aren't superintelligences, and if the system can make it hard to calculate the optimal strategy and make it so that providing one's true valuation doesn't give the player *that* much of a disadvantage, I expect that most people will comfortably just fall back on being truthful.

How would you design such a system?

(I have an idea that I'll share later, but I don't want to prime people with a specific kind of approach from the beginning.)
# Job Opening: SWE to help build signature vetting system for AI-related petitions

**Update: this role has been filled. Thanks for your attention and applications!**

**Full-Stack Software Engineer Position - AI Policy Petition Platform Development**

Survival and Flourishing .Com (SFC) seeks a competent Full-Stack Software Engineer to join us in creating a novel public opinion aggregation platform centered around AI policy issues.

Annual compensation: $200,000 - $300,000

~~Application here.~~

**Your Role**

Develop our "V-process" project, a system that ranks signatories of AI-policy petitions based on signature authenticity, credentials, and relevance to the subject matter. This role offers a unique opportunity to contribute to a platform that addresses the need for legible public expression, independent of existing social media platforms and power structures, regarding emerging technologies, particularly AI.

**Our Ideal Candidate**

*   Strong understanding of system demands and scalability considerations.
*   Proficient in full-stack TypeScript, with experience building and maintaining large-scale web applications.
*   Comfortable in a remote work environment, with occasional in-person meetings in the Bay Area.
*   Excellent communication skills, with the ability to effectively convey progress, implementation trade-offs, and the impact of design decisions.
*   Capable of efficiently updating UI based on design requests and contributing to feature decisions.

**Job Responsibilities**

*   Collaborate with SFC's Product Manager Ethan Ashkie and CEO Andrew Critch, as well as a diverse team of developers and contractors.
*   Design and prototype user-friendly UIs for capturing and displaying signatory data.
*   Implement business logic for signature validation and integrate with external APIs.
*   Occasionally assist with other philanthropic and grant-making projects, time permitting.

**Benefits**

*   Competitive annual salary of $200,000 - $300,000, plus health benefits.
*   Join as the first full-time hire in a flexible, predominantly remote team.
*   We are open to hiring remote contractors living outside the US and can collaborate with institutions who may be capable of providing work visas.

If you're ready to contribute to democratizing the conversation around AI safety and ethics, ~~**apply here**~~. We look forward to working together to shape the future of AI policy.
# Elevator Positioning

I really don't like the feeling of leaving an elevator and not knowing whether to go left or right. You need to be moving so you don't bother the people behind you, but if you just pick a direction you might be going completely the wrong way. My sense of direction is pretty good outside but inside it's patchy, effectively on a floor-by-floor basis.

A few months ago I realized that while waiting for the elevator I can generally work out which way I'll want to walk when I get off. I don't have anything else to do while waiting, so it's a nice short puzzle. Once I figure it out I stand on the side of the elevator opposite where I intend to get out, and maintain that inside the car. When I get to the top I know which way to go from where I'm standing, in case I forgot on the way up, and I go from 50% success in leaving pointing the right way to ~95%.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0LEqxvdXLgAi7AQVasSamS7GGX6ZfQcZDM4EKLAnL6eyLkJgAaVPHxPLA5NfKyGDZl), [mastodon](https://mastodon.mit.edu/@jefftk/110406419955711836)*
# [Linkpost] "Governance of superintelligence" by OpenAI

OpenAI has a new [blog post](https://openai.com/blog/governance-of-superintelligence) out titled "Governance of superintelligence" (subtitle: "Now is a good time to start thinking about the governance of superintelligence—future AI systems dramatically more capable than even AGI"), by Sam Altman, Greg Brockman, and Ilya Sutskever.

The piece is short (~800 words), so I recommend most people just read it in full.

Here's the introduction/summary (bold added for emphasis):

> Given the picture as we see it now, it’s conceivable that within the next ten years, AI systems will exceed expert skill level in most domains, and carry out as much productive activity as one of today’s largest corporations.
> 
> In terms of both potential upsides and downsides, **superintelligence will be more powerful than other technologies humanity has had to contend with in the past**. We can have a dramatically more prosperous future; but we have to manage risk to get there. **Given the possibility of existential risk, we can’t just be reactive**. Nuclear energy is a commonly used historical example of a technology with this property; synthetic biology is another example.
> 
> We must mitigate the risks of today’s AI technology too, but **superintelligence will require special treatment and coordination**.

And below are a few more quotes that stood out:

"First, we need some degree of coordination among the leading development efforts to ensure that the development of superintelligence occurs in a manner that allows us to both maintain safety and help smooth integration of these systems with society."

...

"Second, we are likely to eventually need something like an [IAEA](https://www.iaea.org/) for superintelligence efforts; any effort above a certain capability (or resources like compute) threshold will need to be subject to an international authority that can inspect systems, require audits, test for compliance with safety standards, place restrictions on degrees of deployment and levels of security, etc."

...

"It would be important that such an agency focus on reducing existential risk and not issues that should be left to individual countries, such as defining what an AI should be allowed to say."

...

"Third, we need the technical capability to make a superintelligence safe. This is an [open research question](https://openai.com/blog/our-approach-to-alignment-research) that we and others are putting a lot of effort into."

...

"We think it’s important to allow companies and open-source projects to develop models below a significant capability threshold, without the kind of regulation we describe here"

...

"By contrast, the systems we are concerned about will have power beyond any technology yet created, and we should be careful not to water down the focus on them by applying similar standards to technology far below this bar."

...

"we believe it would be unintuitively risky and difficult to stop the creation of superintelligence"
# A Manifold market notice: Binance

Probably someone on LessWrong has money on Binance, being the largest cryptocurrency exchange.  I'd like to draw their attention to the crime risk, and suggest they move the money elsewhere.

The CFTC has a lawsuit against both Binance and its CEO, Changpeng Zhao ("CZ").  Whatever your general views on regulating crypto, this specific situation has a practical risk to investors.  Because further than just a CFTC lawsuit, my market gives high odds of criminal charges against CZ.  Not just higher than the U.S. population, but actually higher than other crypto project managers.

I didn't want to prematurely declare CZ a danger to investors.  So I waited until the market efficiency grew with more traders.  At this time, the market gives a persistent ~70% chance he'll get criminal charges.  I know it's "only" a playmoney prediction market.  But at 42 traders and fairly unambiguous criteria, unless you have some special insight into his legal situation, it's rational to defer to the market.

This doesn't necessarily mean we'll see FTX-scale losses for investors.  Suppose if he gets criminal charges but the exchange continues to operate, changes leadership, and gets regulated.  In that scenario, investors wouldn't necessarily lose money.  But the money could easily become frozen, stolen, or depreciate by large amounts.  Since this ~70% is a broad aggregation of legal risks that could possibly include things like fraud, this should make you reluctant to do business with him.

Please disregard my small "NO" position on the market.  I habitually place those on newly-made crime markets.  Partly because most people in my markets will never be charged of anything, so that helps push it toward the base rate.  And partly as a gesture of goodwill toward the people mentioned in the markets.

Consider dropping Binance as an exchange you would do business with.

**Update 2023-Jun-11:**
-----------------------

Per feedback from simon and Dagon, I've made a new series.  The criteria is more relevant.  Though I prefer seeing [5+ traders](https://twitter.com/ManifoldMarkets/status/1589703630621663232) for reliability, this post's main subject (Binance) has 10+ now, so I feel comfortable including it.  For comparison, I will include the other largest exchanges.

I'm sure there are some who won't find this convincing, and keep their funds on Binance.  If that's you, then please explain what criteria would influence your decision.
# Data and "tokens" a 30 year old human "trains" on

I did some calculations with a bunch of assumptions and simplifications but here's a high estimate, back of the envelope calculation for the data and "tokens" a 30 year old human would have "trained" on:

*    Visual data: 130 million photoreceptor cells, firing at 10 Hz = 1.3Gbits/s = 162.5 MB/s over 30 years (aprox. 946,080,000 seconds) = 153 Petabytes
*   Auditory data:  Humans can hear frequencies up to 20,000 Hz, high quality audio is sampled at 44.1 kHz satisfying Nyquist-Shannon sampling theorem, if we assume a 16bit (cd quality)*2(channels for stereo) = 1.41 Mbits/s = .18 MB/s over 30 years = .167 Petabytes
*   Tactile data: 4 million touch receptors providing 8 bits/s (assuming they account for temperature, pressure, pain, hair movement, vibration) = 5 MB/s over 30 years = 4.73 Petabytes
*   Olfactory data:  We can detect up to [1 trillion smells](https://www.nih.gov/news-events/nih-research-matters/humans-can-identify-more-1-trillion-smells#:~:text=Humans%20can%20distinguish%20more%20than,more%20discriminating%20than%20previously%20thought.) , assuming we process 1 smell every second and each smell is represented a its own piece of data i.e. log2(1trillion) = 40 bits/s = 0.0000050 MB/s over 30 years = .000004 Petabytes
*   Taste data: 10,000 receptors, assuming a unique identifier for each basic taste (sweet, sour, salty, bitter and umami) log2(5) 2.3 bits rounded up to 3 = 30 kbits/s = 0.00375 MB/s over 30 years = .00035 Petabytes  
      
    This amounts to 153 + .167 + 4.73 + .000004 +  .00035 = 158.64 Petabytes assuming 5 bytes per token (i.e. 5 characters) amounts to 31,728 T tokens  
      
    This is of course a high estimate and most of this data will clearly have huge compression capacity, but I wanted to get a rough estimate of a high upper bound. [Here's](https://docs.google.com/spreadsheets/d/19L_6t76tIOKyxQpS_Md-FnI6eN6uXzr4iSKZnbJpN9Y/edit?usp=sharing) the google sheet if anyone wants to copy it or contribute  
      
    **Discussion**  
    The motivation for this was [chinchilla's wild implications](https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications) post by [**nostalgebraist**](https://www.lesswrong.com/users/nostalgebraist) and in general the idea that humans "need" much less data to train on than AI.   
      
    According to these calculations humans "train" on a lot of Petabytes,  around 5 Petabytes per year which amounts to ~1000 T tokens. Given that we're only training our current models on the order of 1 T tokens, the argument that the human brain is more efficient at learning with less data than current LLMs is not fair. Geoff Hinton changing his mind recently about the efficiency of backpropagation from a less efficient to a much more efficient learning algorithm than what the brain is doing, reinforces this point.
# Do humans still provide value in correspondence chess?

In chess, AIs are very superhuman; the best players in the world would lose nearly every game against any modern computer player.

Do humans still have something to add? The continued existence of correspondence chess, IMO, suggests that they do. In correspondence chess players have days to make each move, and play from their homes. Due to the impossibility of policing cheating under these conditions, correspondence players are allowed to use computer assistance.

You might think this would make the games just a question of who has more computing power. But as far as I can tell, that’s not the case.

What are humans adding? Low confidence, but I think it’s mostly opening prep; try to find a line that looks ok on shallow computer analysis, but where deeper analysis shows you have an advantage. The human value-add is telling the computer which lines to analyze. Since the chess game tree is so large, advice like this is quite valuable.

Proof that correspondence chess is still played: https://www.nytimes.com/2022/11/09/crosswords/correspondence-chess.html
Interview with a human player (from 2016): https://en.chessbase.com/post/better-than-an-engine-leonardo-ljubicic-1-2

On the other hand: I don’t play correspondence chess, so I’m not that confident in the claims above. And some people don’t find them plausible: https://twitter.com/liron/status/1660890927920201728?s=46&t=UlLg1ou4o7odVYEppVUWoQ

Why should we care? This might provide some indication of what value humans can provide in a world of superhuman AI (at least initially).

Can anyone provide a more definitive account of what value, if any, humans add in correspondence chess?
# Phone Number Jingle

When the kids were little and we were at the kind of event where they might get separated from us, I'd write my phone number on tape and stick it to their backs:

[![](https://www.jefftk.com/phone-number-on-back.jpg)](https://www.jefftk.com/phone-number-on-back-big.jpg)

*Anna at [NEFFA](https://www.neffa.org/) 2019, age 3*

The kids are getting older now, and are playing independently enough that it would be good if they always had my phone number. Lily (9yo) has had a piece of paper in her backpack for a while now, but unless she's going to or from school she won't have it with her. Several rounds of "repeat after me" weren't working, so this weekend we made up a song for it. After just a few times singing it she has it down.

This may be a trick that works unusually well for Lily, since she's pretty musical, but the popularity of [jingles for phone numbers](https://tvtropes.org/pmwiki/pmwiki.php/Main/PhoneNumberJingle) in advertising makes me think this would work for others?

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0oontJqA4J7ctDYgWDp6W7jbb3kWuuecBVJzDPxa7Mq5UvDVBiZD5pn9gi3TGg2mGl), [mastodon](https://mastodon.mit.edu/@jefftk/110418667768726278)*
# No - AI is just as energy-efficient as your brain.

This is simply because AIs run on electricity.

Our brains use sunlight via photosynthesis via dietary energy intake. From sunlight to dietary energy this is about [0.25-0.5% energy efficient](https://en.wikipedia.org/wiki/Photosynthetic_efficiency). Let's say it's 0.35% efficient. (This is then our complete sunlight-to-capability efficiency for a human)

AI systems use sunlight via solar energy via electrical consumption. From sunlight to GPU this is about 10-18% efficient let's say it's 13% efficient. (Data: typical solar panel efficiency = 15-20% efficient. Typical electricity distribution efficiency = 90% efficient). 13% is then AI sunlight-to-electricity efficiency, but not yet sunlight-to-capability efficiency.

Let's calculate the final part. We need to assume an amount of compute for a human-equivalent AI system. I will assume that (note, at inference time) we can run a human-equivalent AI system on 10 NVIDIA 4070 GPUs at full power, which each consume ~200 Watts, so 2 kW in total. In contrast the typical human consumes 100 watts. 100W divided by 2 kW gives us 5% as AI electricity-to-capability efficiency, which we then multiply to get the total AI efficiency. 5% electricity-to-capability efficiency times 13% sunlight-to-electricity efficiency = 0.65% sunlight-to-capability efficiency.

So humans are 0.35% efficient, and AIs are 0.65% efficient.  
  
By these assumptions, AI is *somewhat* ***more** efficient* in terms of *real* energy input - here, energy from the sun. This is the number that the economy is going to care about - why use land for crops, when you can use it for solar panels?

Please pick apart my numbers and assumptions in the comments. Thanks.
# The Office of Science and Technology Policy put out a request for information on A.I.

This request for information does cover some of the possible existential risks, in the first section.  I am going to submit a few responses of my own, and I am hopeful others will do the same.  

1\. What specific measures – such as standards, regulations, investments, and improved trust and safety practices – are needed to ensure that AI systems are designed, developed, and deployed in a manner that protects people’s rights and safety? Which specific entities should develop and implement these measures?

 2\. How can the principles and practices for identifying and mitigating risks from AI, as outlined in the Blueprint for an AI Bill of Rights and the AI Risk Management Framework, be leveraged most effectively to tackle harms posed by the development and use of specific types of AI systems, such as large language models? 

3\. Are there forms of voluntary or mandatory oversight of AI systems that would help mitigate risk? Can inspiration be drawn from analogous or instructive models of risk management in other sectors, such as laws and policies that promote oversight through registration, incentives, certification, or licensing?

 4\. What are the national security benefits associated with AI? What can be done to maximize those benefits? 

5\. How can AI, including large language models, be used to generate and maintain more secure software and hardware, including software code incorporating best practices in design, coding and post deployment vulnerabilities? 

6\. How can AI rapidly identify cyber vulnerabilities in existing critical infrastructure systems and accelerate addressing them?

 7\. What are the national security risks associated with AI? What can be done to mitigate these risks?

 8\. How does AI affect the United States’ commitment to cut greenhouse gases by 50-52% by 2030, and the Administration’s objective of net-zero greenhouse gas emissions no later than 2050? How does it affect other aspects of environmental quality?
# Rishi Sunak mentions "existential threats" in talk with OpenAI, DeepMind, Anthropic CEOs

> The PM made clear that AI is the defining technology of our time, with the potential to positively transform humanity.
> 
> But the success of this technology is founded on having the right guardrails in place, so that the public can have confidence that AI is used in a safe and responsible way. The PM set out how the approach to AI regulation will need to keep pace with the fast-moving advances in this technology. That is why the UK Government has deliberately adopted an agile response to unlock the opportunities whilst mitigating the risks of the technology, as outlined in our AI White Paper.

> The PM and CEOs discussed the risks of the technology, ranging from disinformation and national security, to existential threats. They discussed safety measures, voluntary actions that labs are considering to manage the risks, and the possible avenues for international collaboration on AI safety and regulation. The lab leaders agreed to work with the UK Government to ensure our approach responds to the speed of innovations in this technology both in the UK and around the globe.
# DeepMind: Model evaluation for extreme risks

> Current approaches to building general-purpose AI systems tend to produce systems with both beneficial and harmful capabilities. Further progress in AI development could lead to capabilities that pose extreme risks, such as offensive cyber capabilities or strong manipulation skills. We explain why *model evaluation* is critical for addressing extreme risks. Developers must be able to identify dangerous capabilities (through "dangerous capability evaluations") and the propensity of models to apply their capabilities for harm (through "alignment evaluations"). These evaluations will become critical for keeping policymakers and other stakeholders informed, and for making responsible decisions about model training, deployment, and security.

This is the first great public writeup on model evals for averting existential catastrophe. I think it's likely that if AI doesn't kill everyone, *developing great model evals and causing everyone to use them* will be a big part of that. So I'm excited about this paper both for *helping AI safety people learn more and think more clearly about model evals* and for *getting us closer to it being common knowledge that responsible labs should use model evals and responsible authorities should require them (by helping communicate model evals more widely, in a serious/legible manner)*.

Non-DeepMind authors include Jade Leung (OpenAI governance lead), Daniel Kokotajlo (OpenAI governance), Jack Clark (Anthropic cofounder), Paul Christiano, and Yoshua Bengio.

See also DeepMind's [related blogpost](https://www.deepmind.com/blog/an-early-warning-system-for-novel-ai-risks).

For more on model evals for AI governance, see [ARC Evals](https://evals.alignment.org), including Beth's EAG talk [Safety evaluations and standards for AI](https://youtu.be/Vb5g7jlNzOk?t=93) and the blogpost [Update on ARC's recent eval efforts](https://evals.alignment.org/blog/2023-03-18-update-on-recent-evals/) ([LW](https://www.lesswrong.com/posts/4Gt42jX7RiaNaxCwP/more-information-about-the-dangerous-capability-evaluations)).
# You Don't Always Need Indexes

Sometimes you have a lot of data, and one approach to support quick searches is pre-processing it to build an index so a search can involve only looking at a small fraction of the total data. The threshold at which it's worth switching to indexing, though, might be higher than you'd guess. Here are some cases I've worked on where full scans were better engineering choices:

*   Ten years ago I wrote an interoffice messaging application for a small billing service. Messages were stored in MySQL and I was going to add indexing if full-text searches got slow or we had load issues, but even with ten years worth of messages to search it stayed responsive. (This was before MySQL supported `FULLTEXT` indexes with InnoDB, added in [v5.6](https://downloads.mysql.com/docs/mysql-5.6-relnotes-en.pdf), and at the time indexing would have meant installing and maintaining [Sphinx](http://sphinxsearch.com/).)
    
*   I recently [came across](https://news.ycombinator.com/item?id=35840944) someone maintaining a 0.5GB full text index to support searching their shell history, 100k commands. I [use](https://www.jefftk.com/p/logging-shell-history-in-zsh) `grep` on a flat file, and testing now it takes 200ms for a query across my 180k history entries.
    
*   My [contra dance search tool](https://www.trycontra.com/) ranks each dance in response to your query, with no geospatial indexing, because there are just ~350 dances.
    
*   The [viral counts explorer](https://www.jefftk.com/mgs-counts/) I've been playing with for work searches the taxonomic tree of human viruses in real time, scanning ~15k names with JS's "includes" command about as fast as you can type.
    
*   When I worked in ads I would often need to debug issues using production logs, and would use Dremel ([Melnik 2010](https://research.google/pubs/pub36632.pdf), [Melnik 2020](https://research.google/pubs/pub49489.pdf)) to run a distributed scan of very large amounts of data at interactive speeds. Because queries were relatively rare, an index would have been far more expensive to maintain.
    

Unless you know from the start that you'll be searching hundreds of millions of records, consider starting with simple scans and only add indexing if you can't get acceptable performance. And even then, if queries are rare and highly varied you may still do better to do the work at query time instead of ingestion time.
# [Market] Will AI xrisk seem to be handled seriously by the end of 2026?

[https://manifold.markets/tailcalled/will-ai-xrisk-seem-to-be-handled-se?r=dGFpbGNhbGxlZA](https://manifold.markets/tailcalled/will-ai-xrisk-seem-to-be-handled-se?r=dGFpbGNhbGxlZA)

Lately it seems like there's been a bunch of people taking big AI risks seriously, from OpenAI's [Governance of superintelligence](https://openai.com/blog/governance-of-superintelligence) to Deepmind's [Model evaluation for extreme risks](https://arxiv.org/pdf/2305.15324.pdf).

We're not quite there yet in terms of my standards for safety, and even less there in terms of e.g. Eliezer Yudkowsky's. However I wonder if this marks a turning point.

Obviously this is a very subjective question, so I am warning you ahead of time that it is going to resolve in opinionated ways. Trying to anchor the discussion, I expect the following to be necessary for a YES resolution:

*   Major leading AI companies openly acknowledge that existential risk is a possibility, not just in a marginal sense (e.g. internal discussion of it by employees, rare cases where the leaders begrudgingly admit it) but also in a central sense (e.g. openly having research teams working on it, having long sections of documents for politicians discussing it).
*   Something is done to handle unilateral actors, e.g. there is active progress made in creating an international organization which can prevent unilateral actors from creating unfriendly AI, or maybe somehow the only plausible creators all take AI xrisk seriously.
*   Yann LeCun changes his mind to take AI xrisk seriously or no longer holds much sway about it at Facebook.
*   The lessons of [Worlds Where Iterative Design Fails](https://www.lesswrong.com/posts/xFotXGEotcKouifky/worlds-where-iterative-design-fails) are taken seriously by the above systems.

Please ask me more questions in the comments to help cement the resolution criteria. If my opinion on the inherent danger of AI xrisk changes during the resolution period, I will try to respond based on the level of risk implied by my criteria, not based on my later evaluation of things.

However, if it turns out that there is a similarly powerful way of handling AI xrisk which is qualitatively different, which gets implemented in practice, I will also resolve this question to YES.
# Look At What's In Front Of You (Conclusion to The Nuts and Bolts of Naturalism)

If naturalism has been the subject of this sequence, then what is naturalism? Now might be a good time to try stating your own answer, in your own words, before reading more of mine.

* * *

Here is what *I* have to say about it.

Naturalism is an investigative method that focuses attention on the points in daily life where subjective experience intersects with crucial information. 

It brings reflective awareness to experiences that were always available, but that our preconceptions inclined us to discard; it thereby grants us the opportunity to fold those observations into our stories about the world. It is a gradual process of original seeing, clarification, and deconfusion. 

At its best, naturalism results in a greater ability to interact agentically with the world as it is, rather than fumbling haphazardly through a facade of misapprehensions.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/deebe08600deeffc4547436c4b2a350dcd9e8fbc2e2cab11.png)

Despite belonging to a whole different phylum, the white-lined sphinx moth is easily mistaken for a hummingbird when seen briefly or from a distance.

This sequence was my attempt to outline the curriculum I guide people through when helping them learn the practice of naturalism—when helping them learn to behave as though knowing the territory takes patient and direct observation. 

The curriculum is not the practice itself. Pedagogical considerations require that the curriculum be relatively linear and self-contained. The actual practice of naturalist investigation tends to draw as needed from all phases at once, interweaves with other methodologies as appropriate, and sometimes requires the invention of novel techniques on the spot.

Furthermore, my curriculum does not teach The One True Way of Patient and Direct Observation. It is only *my* way, tailored to and limited by the idiosyncratic shape of my particular mind, and adapted somewhat for the unusual class of people willing to work with me.

This is not an easy practice, and it certainly is not fast. It is a lot like lifting weights to get stronger. When easy answers are readily available through expert opinion; when problems are technical rather than adaptive; when you are certain that all you need is a “yes” or a “no”, and not to build a whole new way of understanding—in these cases, naturalism is often the wrong toolset for the job. 

Nevertheless, I find that even when naturalism is the wrong toolset overall for the particular question I'm dealing with, there’s something deeply valuable in having built my own version of these tools, and in constantly feeling their readiness to hand. There is empowerment in recognizing that I personally can figure things out, by following my own curiosity, looking carefully for myself, and gradually piecing together whatever I find.

I think that a lot more work is needed before these methods can be smoothly integrated with the communal art of rationality. Still, I earnestly hope that my descriptions of this far-from-perfect method will prove a useful resource in whatever comes next.
# What vegan food resources have you found useful? 

My [vegan nutrition project](https://acesounderglass.com/2023/05/13/lessons-learned-from-offering-in-office-nutritional-testing/) has focused on supplements, but there have been several requests for resources on how to eat better. I'm not well equipped to answer this- I'm not vegan, and my stomach only got [even passably good](https://acesounderglass.com/2022/10/13/my-resentful-story-of-becoming-a-medical-miracle/) at digestion a year ago. So I'd like to crowdsource this question: **what resources have you found useful in eating well while eliminating or reducing animal products**? 

My existing list focuses on overviews and onboarding guides, but I think things like recipe books would also be useful. 

*   [Veganhealth.org](https://veganhealth.org/) is the number one resource I see cited, by both people I know and other vegan websites.
*   [TheVeganRD.com guide](https://www.theveganrd.com/vegan-nutrition-101/food-guide-for-vegans/) is far from comprehensive but seems extremely approachable.
*   A reader pointed me to [spreadsheet](https://vegancheatsheet.org/), which links to a number of resources. 
*   Lincoln Quirk's [**Practical Plant-Based Meal Planning for Groups**](https://forum.effectivealtruism.org/posts/xrFZbNWvRz5cKZ8iC/practical-plant-based-meal-planning-for-groups)

Have people found these helpful? Are there other things that are better, or fill in a gap? How did you learn to cook well?

Diet and supplements overlap but there is already a [supplements post](https://forum.effectivealtruism.org/posts/x4uANMyFyxhfizF6n/veg-ns-what-supplements-do-you-take) and I don't want to split the information, so please lean food-based here.
# how humans are aligned

This is a description of how I think humans are kept somewhat aligned towards genetically-specified goals. It's not an argument, just my views.

It works well enough, but not perfectly. South Korea isn't having kids anymore. Sometimes you get serial killers or Dick Cheney. So, anything less seems likely to be inadequate.

previously: [AI self-improvement is possible](https://www.lesswrong.com/posts/rSycgquipFkozDHzF/ai-self-improvement-is-possible)

---

#### limited self-modification

Don't allow systems to modify lower-level systems, and strongly limit self-modification at the same system level. When it's done at all, reduce the amount allowed after an initial learning period, so that children have more flexibility than old people.


#### limited length

Don't have long chains of systems generating systems. Limit things to 2 steps of systems generating higher-level systems.


#### lifespan

Even if you try to restrict self-modification, some "leakage" will happen anyway. A physical time limit that can't be extended past a maximum keeps that from becoming excessive. (Perhaps that's why some octopus species are "hardcoded" to die at specific points in their reproductive cycle.)


#### old age

More-capable misaligned systems are more dangerous, so degrading capabilities of systems as they approach their time limit makes them safer. I don't actually think this is the limiting factor for human senility; I think that's mainly due to uncontrolled covalent modification of DNA. What I do think is that humans have some low-level system that, when they get old, reduces influence of or shuts down certain mid-level systems.


#### monitoring

Humans have [monitoring systems](https://www.lesswrong.com/posts/rSycgquipFkozDHzF/ai-self-improvement-is-possible#L_monitoring) with a lesser degree of self-modification than what they monitor. They have access to the internal opinions of what they monitor, but can be deceived. The net benefit is actually somewhat questionable.


#### democracy

If alignment drift is somewhat random, then making many separate agents that act according to consensus reduces the net drift. Hermits and dictators do weird stuff.


#### internal democracy

Like democracy, but for multiple agents inside a single individual. Obviously there has to be some way to prevent agents from coalescing into a single blob, but that could be managed by lower-level hardcoded systems that blindly bottleneck the bandwidth of some connection patterns and force some of that limited bandwidth through low-level systems.
# Request: stop advancing AI capabilities

*Note: this has been in my draft queue since well before the [FLI letter](https://futureoflife.org/open-letter/pause-giant-ai-experiments/) and [TIME article](https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/). It was sequenced behind the [interpretability post](https://www.lesswrong.com/posts/BinkknLBYxskMXuME/if-interpretability-research-goes-well-it-may-get-dangerous), and delayed by travel, and I'm just plowing ahead and posting it without any edits to acknowledge ongoing discussion aside from this note.*

This is an occasional reminder that I think pushing the frontier of AI capabilities in the current paradigm is highly anti-social, and contributes significantly in expectation to the destruction of everything I know and love. To all doing that (directly and purposefully for its own sake, rather than as a mournful negative externality to alignment research): I request you stop.

(There's plenty of other similarly fun things you can do instead! Like trying to figure out how the heck modern AI systems work as well as they do, [preferably with a cross-organization network of people who commit not to using their insights to push the capabilities frontier before they understand what the hell they're doing](https://www.lesswrong.com/posts/BinkknLBYxskMXuME/if-interpretability-research-goes-well-it-may-get-dangerous)!)

(I reiterate that this is not a request to stop indefinitely; I think building AGI eventually is [imperative](https://www.lesswrong.com/posts/HoQ5Rp7Gs6rebusNP/superintelligent-ai-is-necessary-for-an-amazing-future-but-1); I just think literally every human will be killed at once if we build AGI before we understand what the hell we're doing.)
# Who are the Experts on Cryonics?

I really liked the recent essay [Who are the Experts on Cryonics?](https://biostasis.substack.com/p/who-are-the-experts-on-cryonics?utm_source=substack&utm_medium=email) by Max More.

Here's **why I think you might like this article**:
- There are a lot of generally relevant points about when/how to trust experts, not specific to cryonics.
- I think it does a really good job at explaining how to identify the relevant experts in cryonics to help you assess the value of current and potential cryonics.
- It has a list of interesting famous wrong predictions 'experts' made.

Here's the **table of content**:
- Whose expertise is relevant to evaluating cryonics?
- Experts dropping the ball
- Factors to consider in evaluating expertise
- How experts can be wrong
- Caution: Expert ahead!