Harley-ml/lesswrong huggingface dataset:
# Against an AI Research Moratorium

All the smart people agitating for a 6-month moratorium on AGI research seem to have unaccountably lost their ability to do elementary game theory.  It'a a faulty idea regardless of what probability we assign to AI catastrophe.

Our planet is full of groups of power-seekers competing against each other. Each one of them could cooperate (join in the moratorium) defect (publicly refuse) or stealth-defect (proclaim that they're cooperating while stealthily defecting). The call for a moratorium amounts to saying to every one of those groups "you should choose to lose power relative to those who stealth-defect". It doesn't take much decision theory to predict that the result will be a covert arms race conducted in a climate of fear by the most secretive and paranoid among the power groups.

The actual effect of a moratorium, then, would be not be to prevent super-AGI; indeed it is doubtful development would even slow down much, because many of the power-seeking groups can sustain large research budgets due to past success. If there's some kind of threshold beyond which AGI immediately becomes an X-risk, we'll get there anyway simply due to power competition.  The only effect of any moratorium will be to ensure that (a) the public has no idea what's going on in the stealth-defectors' labs, and (b) control of the most potent AIs will most likely be achieved first by the most secretive and paranoid of power-seekers.

A related problem is that we don't have a college of disinterested angels to exert monopoly control of AI, or even to just trust to write its alignment rules. Pournelle's Law. "Any bureaucracy eventually comes to serve its own interests rather than those it was created to help with," applies; monopoly controllers of AI will be, or will become, power-seekers themselves. And there is no more perfect rationale for totalitarian control of speech and action than "we must prevent anyone from ever building an AI that could destroy the world!"  The entirely predictable result is that even if the monopolists can evade AGI catastrophe (and it's not clear they could) the technology becomes a boot stomping on humanity's face forever.

Moratorium won't work.  Monopoly won't either.  Freedom and transparency might. In this context, "Freedom" means "Nobody gets to control the process of AI development," and "transparency" means "All code and training sets are open, and attempting to conceal your development process is interpreted as a crime - an act of aggression against humanity in the future". Ill-intentioned people will still try to get away with concealment, but the open-source community has proven many times that isolating development behind a secrecy wall means you tend to slow down and make more persistant mistakes than the competing public community does.

Freedom and transparency now would also mean we don't end up pre-emptively sacrificing every prospect of a non-miserable future in order to head off a catastrophe that might never occur.

(This is a slightly revised version of a comment I posted a few hours ago on Scott Aaronson's blog.)
# Widening Overton Window - Open Thread

Major news outlets have published articles about the Future of Life Institute's Open Letter. Time Magazine published an opinion piece by Elizer Yudkowsky. Lex Friendman featured EY on his podcast. Several US Members of Congress have spoken about the risks of AI. And a Fox News reporter asked a what the US President is doing to combat AI x-risk at a White House Press Conference.

Starting an Open Thread to discuss this, and how best to capitalize on this sudden attention.

Links:

WH Press Conference: [https://twitter.com/therecount/status/1641526864626720774](https://twitter.com/therecount/status/1641526864626720774)

Time Magazine: [https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/](https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/) 

FLI Letter: [https://futureoflife.org/open-letter/pause-giant-ai-experiments/](https://futureoflife.org/open-letter/pause-giant-ai-experiments/)
# Correcting a misconception: consciousness does not need 90 billion neurons, at all



*epistemic status: still a student but quite sure of myself on this topic and pretty sure that this misconception plausibly has a non negligeable impact on some debates*

Hi,

Medical student here, I just wanted to shed some light on what I think is a common misconception.

Yes the human brain contains about 86 billion neurons. But about 60 billions are in the cerebellum and have little to do with consciousness. Those neurons can plausibly be approximated as just filtering the noisy signal going from the rest of the brain (cortex etc) to the limbs.

A person without cerebellum can be perfectly conscious.

But reducing the number of neurons by two thirds like that does not change the order of magnitude of the number of synapses, which arguably are a more important number to consider when looking for an upper bound needed for consciousness.

But you can't just assume that all those neurons are fully connected, the length of axons is not the same for all neurons and the number of dendrites vary etc.

According to [this study](https://pubmed.ncbi.nlm.nih.gov/11418939/), the average total number of synapses in **the neocortex of young male brains** was 164x10^12 (so 164 'trillion', using [the american short scale](https://en.wikipedia.org/wiki/Trillion)).

## Sources:
* [Wk 3-3 What is the role of cerebellum for consciousness? - Neural basis of Consciousness](https://www.youtube.com/watch?v=4yHPKPLTF1w)
* [Are patients without cerebellum conscious? - Neural basis of Consciousness](https://www.youtube.com/watch?v=PquI3S8ZjjM)
* [Cerebellar granule cells, in contrast to Purkinje cells, are among the smallest neurons in the brain. They are also the most numerous neurons in the brain: In humans, estimates of their total number average around 50 billion, which means that about 3/4 of the brain's neurons are cerebellar granule cells](https://en.wikipedia.org/wiki/Cerebellum#cite_note-SOB-9)
* [The adult human brain is estimated to contain 86±8 billion neurons, with a roughly equal number (85±10 billion) of non-neuronal cells.[40] Out of these neurons, 16 billion (19%) are located in the cerebral cortex, and 69 billion (80%) are in the cerebellum](https://en.wikipedia.org/wiki/Human_brain)
# ChatGPT banned in Italy over privacy concerns

> Italy has become the first Western country to block advanced chatbot ChatGPT.
> 
> The Italian data-protection authority said there were privacy concerns relating to the model, which was created by US start-up OpenAI and is backed by Microsoft.
> 
> The regulator said it would ban and investigate OpenAI "with immediate effect".

Alternative article available [here](https://www.independent.co.uk/tech/chatgpt-ban-italy-gdpr-data-protection-b2311738.html).
# [Linkpost] Critiques of Redwood Research

An anonymous user named Omega posted a critique of Redwood on the EA Forum. The post highlights four main areas: **(1) Lack of senior ML staff, (2) Lack of communication & engagement with the ML community, (3) Underwhelming research output, and (4) Work culture issues**.

I'm linkposting it here, since I imagine some LW users will have thoughts/comments. See also this [comment from Nate Thomas](https://See also this comment from Nate Thomas. ), and note that Redwood has an [anonymous  feedback form](https://www.admonymous.co/redwood). 

> We believe that Redwood has some serious flaws as an org, yet has received a significant amount of funding from a central EA grantmaker (Open Philanthropy). Inadequately kept in check conflicts of interest (COIs) might be partly responsible for funders giving a relatively immature org lots of money and causing some negative effects on the field and EA community. We will share our critiques of Constellation (and Open Philanthropy) in a follow-up post. We also have some suggestions for Redwood that we believe might help them achieve their goals.
> 
> Redwood is a young organization that has room to improve. While there may be flaws in their current approach, it is possible for them to learn and adapt in order to produce more accurate and reliable results in the future. Many successful organizations made significant pivots while at a similar scale to Redwood, and we remain cautiously optimistic about Redwood's future potential.

*Standard caveat that I don't agree with everything in the post or even endorse its main conclusions; also see my* [*comment*](https://www.lesswrong.com/posts/2Q5vYcfmi46qKMReB/linkpost-critiques-of-redwood-research?commentId=FcNqvpDhYtxZHtc3D)*.*
# Kallipolis, USA

This is a very tongue-in-cheek term paper I wrote for my philosophy class on Plato in fall of 2016. I had a lot of fun with it & still like to talk about it sometimes when I get tipsy. I'm putting it up here because why not. Comments welcome.  
  
Abstract:

> Reading Plato’s *Republic, *you may have had reactions like this: “The Kallipolis… what a ridiculous and intriguing city! How utopian, yet how totalitarian! How alien!”
> 
> If so, then you are in the right place, because the thesis of this paper is that the present-day United States of America is in fact the Kallipolis.
> 
> Admittedly, the argument in this paper is not strong enough to take us all the way to that conclusion with any confidence. But it can push us in that direction. What I will do is draw out all the connections and parallels that I see between the USA and the Kallipolis. I will argue that the USA is currently surprisingly close to realizing Plato’s vision, at least in broad strokes. Some parts of my argument are rather hand-wavy, and others rely on debatable empirical claims, so I doubt you will end up fully convinced. Honestly I am not fully convinced myself. But I will make the best case I can, and hopefully you will end up convinced that the USA is closer in the direction of the Kallipolis than you previously thought.

![Image](https://media.discordapp.net/attachments/1008571197572775966/1091543477885620254/Kokotajlod_A_variation_of_the_classic_painting_of_the_philosoph_5d23b876-d4fc-458c-abfe-a5d8481b58f7.png?width=924&height=924)
# Hooray for stepping out of the limelight

From maybe [2013](https://www.deepmind.com/publications/playing-atari-with-deep-reinforcement-learning) to [2016](https://en.wikipedia.org/wiki/AlphaGo_versus_Lee_Sedol), DeepMind was at the forefront of hype around AGI. Since then, they've done less hype. For example, [AlphaStar](https://www.deepmind.com/blog/alphastar-mastering-the-real-time-strategy-game-starcraft-ii) was not hyped nearly as much as I think it could have been.

I think that there's a very solid chance that this was an intentional move on the part of DeepMind: that they've been intentionally avoiding making AGI capabilities seem sexy.

In the wake of big public releases like ChatGPT and Sydney and GPT-4, I think it's worth appreciating this move on DeepMind's part. It's not a very visible move. It's easy to fail to notice. It probably hurts their own position in the arms race. I think it's a prosocial move.

If you are the sort of person who is going to do AGI capabilities research—and I recommend against it—then I'd recommend doing it at places that are more likely to be able to keep their research private, rather than letting it contribute to an arms race that I expect would [kill literally everyone](https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities).

I suspect that DeepMind has not only been avoiding hype, but also avoiding publishing a variety of their research. Various other labs have also been avoiding both, and I applaud them too. And perhaps DeepMind has been out of the limelight because they focus less on large language models, and the results that they do have are harder to hype. But insofar as DeepMind *was* in the limelight, and *did* intentionally step back from it and avoid drawing tons more attention and investment to AGI capabilities (in light of how Earth is not well-positioned to deploy AGI capabilities in ways that [make the world better](https://www.lesswrong.com/posts/HoQ5Rp7Gs6rebusNP/superintelligent-ai-is-necessary-for-an-amazing-future-but-1)), I think that's worth noticing and applauding.

(To be clear: I think DeepMind could do significantly better on the related axis of avoiding publishing research that advances capabilities, and for instance I was sad to see Chinchilla published. And they could do better at avoiding hype themselves, as [noted in the comments](https://www.lesswrong.com/posts/vEJAFpatEq4Fa2smp/hooray-for-stepping-out-of-the-limelight?commentId=ftRMm24mdzgWwKe6D). At this stage, I would recommend that DeepMind [cease further capabilities research](https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/) until our understanding of alignment is much further along, and my applause for the specific act of avoiding hype does not constitute a general endorsement of their operations. Nevertheless, my primary guess is that DeepMind has made at least some explicit attempts to avoid hype, and insofar as that's true, I applaud the decision.)
# [New LW Feature] "Debates" 

Following the success of the  [2021 MIRI Conversations](http://www.lesswrong.com/sequences/n945eovrA3oDueqtq) in drawing out various people's views on key questions in AI and digging into their disagreements, the LessWrong team was inspired to build a more dedicated feature for hosting debates on LessWrong.

The MIRI conversations were mostly hosted on Discord and then via a laborious process shoehorned into the LessWrong editor. We figured it wouldn't be hard to do better. Among many benefits, the debates can be held on LessWrong itself; readers are able to comment "inline" on responses within the debate; and there will be customized "debate item" in the Latest Posts list on the frontpage that signals that 1) it's a debate, and 2) how many debate responses have been posted since you last viewed the debate. Hopefully all of this is intuitive from the UI. 

The feature is designed so that debates can be held in private, possibly edited, and then published publicly. Or that the debate happens live on the site, allowing for live commenting. 

As we're rolling out the feature, we'll initially just set up a few debates that we'd like to see, and then later potentially open up the feature to users more broadly. You're welcome to contact us \[link\] or comment here if you're interested in viewing or participating in a particular debate. 

* * *

**This announcement post will also serve as the Inaugural Debate using the new debate feature.** We were lucky to find two willing participants on short notice, so big thanks to them. GPT-4 and Claude+ are going to discuss whether or not AI Safety via Debate is a promising Alignment strategy.
# Campaign for AI Safety: Please join me

I will start by saying that I generally agree with [Yudkowsky's position on AI](https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/). We must proceed with extreme caution. We must radically slow down AI capability advancement. We must invest unfathomable amounts of resources in AI alignment research. We need to enact laws and treaties that will help keep it all together for as long as possible and hopefully we figure things out in time.

The [laughter](https://twitter.com/therecount/status/1641526864626720774) at the recent White House press conference at the question about Yudkowsky's argument, indicates far we are in public debate from a sensible position of caution. 

But I am hopeful that we can change that. Few people laugh at nuclear weapons now. We are a species capable of cooperation and of taking things seriously. As the saying goes:

> *"First they ignore you, then they laugh at you, then they fight you, then you win."*

What is missing is public understanding of the dangers of misaligned / unaligned AI. Democracy does not work in darkness. People must know the dangers, the uncertaintly, and of ways they can contribute.

That's why I am proposing a campaign on public awareness of x-risk from AI. So far, it's just me and my wife. Please join me, especially if you work in advertising, marketing, PR, activism, politics, law, etc., if you know how to make a website, if you want to create PR materials, meet journalists, do accounting, fund-raising, etc. 

Please share this with people who do not read Less Wrong but we are freaked out and want to do something.

I do not know exactly how this campaign will run, which countries to focus on. I am myself only human and can contribute very little of the total required effort. My background is in consulting and market research and I run a market research company. Personally, at this stage, I can best contribute by coordination and facilitating operations.

We need people, money, expertise, patience, etc. Please join: [https://campaignforaisafety.org/](https://campaignforaisafety.org/).
# Is this true? paulg: [One special thing about AI risk is that people who understand AI well are more worried than people who understand it poorly]

[Link to claim.](https://twitter.com/paulg/status/1642110597545295872)

I've seen this claim before but I haven't seen any direct data supporting it. Does anyone have any resources?
# How security and cryptography can aid AI safety [sequence]

Overview
========

In this review sequence, I lay out why I think security, cryptography, and auxiliary approaches are still neglected in AI safety, specific insights those communities can offer for AI safety, and some potential next steps.  
 

I am writing this post now, as it seems to have become a more pressing focus within EA/AI safety but there are some issues and approaches I haven’t seen much discussion on here yet.  
 

I am not an expert in any of the fields mentioned here, but given that Foresight Institute, the non-profit I manage, has a long-standing security community that cares about AI safety, I hope to provide a few birds-eye view reasons and examples gathered from discussions within the community.  
 

Thank you to many of the people mentioned throughout this sequence for reviewing my characterization of their efforts. Any mistakes are my own, and I encourage general feedback, criticism, and pointers to relevant research and efforts.  
 

The sequence contains four parts highlighting promising areas to consider:

[Part 1 | AI infosec: first strikes, zero-day markets, hardware supply chains](https://www.lesswrong.com/posts/kvk2ZorXui4YB4zvc/part-1-direct-help-with-ai-info-and-computer-security)

[Part 2 | AI safety and the security mindset: user interface design, red-teams, formal verification](https://www.lesswrong.com/posts/RKCfBMoMXr6JsgHAy/part-2-parallels-between-ai-safety-and-the-security-mindset)

[Part 3 | Boundaries-based security and AI safety approaches](https://www.lesswrong.com/posts/TZy4mFJFJ4yv2MRhg/boundaries-based-security-and-ai-safety-approaches)

[Part 4 | Specific cryptographic and auxiliary approaches to consider for AI](https://www.lesswrong.com/posts/DvmDgpKmd3ftPDtzr/cryptographic-and-auxiliary-approaches-relevant-for-ai)

[Part 5 closes with a summary and potential next steps](https://www.lesswrong.com/posts/4XeD5goRb3N6Ch2ji/upskilling-bridge-building-research-on-security-cryptography).

  
I plan to publish the five parts of this sequence spaced out by a few days, starting with part 1 today. All of them should be relatively understandable as stand-alone pieces and will be linked here. **Feel free to skip around to the parts that you find most interesting. **
# Shutting down AI is not enough. We need to destroy all technology.

A [TIME article published recently](https://time.com/6266923/ai-eliezer-yudkowsky-open-letter-not-enough/) calls for an “indefinite and worldwide” moratorium on new large AI training runs.

This moratorium would be better than no moratorium. I have respect for the author who wrote it. It’s an improvement on the margin.

I refrained from endorsing the essay because I think it is understating the seriousness of the situation and asking for too little to solve it.

If there was a plan for Earth to survive, if only we passed an indefinite and worldwide moratorium on large training runs, I would back that plan. There isn’t any such plan.

Here’s what would actually need to be done:

All human technology needs to be destroyed. There can be no exceptions, including for sharpened stones and hand axes. After everything is burned, we must then forget how to create fire. If a single exception is made, that increases the probability that civilization will be recreated within the next millennia and new large AI training runs will be started. If I had infinite freedom to write laws, I might carve out a single exception for technologies that prevent human diseases, like knowledge of germ theory; but if that was remotely complicating the issue I would immediately jettison that proposal and say to just shut it all down.

Shut down all the roads, melt all the cars. Burn down all of the tables and all of the books. Put a ceiling into how many calories of food any single human can furnish per day, and move it downward over the coming generations to compensate for the possibility that natural selection will keep making humans smarter. No exceptions for cloth or fireplaces. Destroy all human objects now to prevent them from moving to another country. Track all gazelles that are hunt. If anyone notices that a tribe is catching more gazelles than it should, be willing to choke them (with your bare hands, of course) one by one.

Shut it all down. Eliminate all technology. Dismantle modern civilization. Return to our primeval state.

We are not ready. We are not on track to be significantly readier any time in the next million years. If we go ahead on this, everyone will suffer, including children who did not choose this and did not do anything wrong.

Shut it down.

[![May be an image of 2 people and text that says 'The Overton Window Future of Life Institute My proposal'](https://scontent-sjc3-1.xx.fbcdn.net/v/t39.30808-6/338596744_225356903343336_1098985622407776150_n.jpg?stp=dst-jpg_p843x403&_nc_cat=111&ccb=1-7&_nc_sid=5cd70e&_nc_ohc=f4e_89CC2m0AX_g_mVm&_nc_ht=scontent-sjc3-1.xx&oh=00_AfBTQSPkxKOIONQjbUffrs0eTCPVZbp8GOtUDhhta0Dfxw&oe=642E5BDE)](https://www.facebook.com/photo/?fbid=1629434904150523&set=gm.3433702703611290&idorvanity=1479475219034058&__cft__[0]=AZXkZ1qaanp0ZuWqTkntwvHnRAs9XV97z9jrRYmxLqvCUWLw6zG1IPaZtqexpZkDVAD0Q6W-P18zxTge24KRlIKfcvoSTlGrLAWbpftQf54l4V-wOPyRZkADgoBJkr_lUOKuq0ljn9L2OakEGN9jFpYy&__tn__=EH-R)
# A report about LessWrong karma volatility from a different universe

*In a far away universe, a news report is written about LessWrong. The following passages have been lifted over and written into this post...*

**Early one morning** [**all voting on LessWrong was halted**](https://www.coindesk.com/business/2022/11/09/ftx-site-experiences-temporary-outage-warns-users-not-to-deposit/)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/6507c09716a75b3817528a644c5cbd86d806cdced24973a3.png)

*"LessWrong is currently unable to process karma gains. We strongly advise against making new comments about the art of rationality or AI notkilleveryoneism. Voting has been disabled."*

**It was said that there was nothing to worry about**

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b06ee7cdaba3b95f36605299b54febda7bad2b3458ee94eb.png)

**But then GreaterWrong announced their intent to acquire and then un-acquire LessWrong**

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/72be76fd369b6d8b1f33e779aa8b387cd1346a0945909b21.png)

**All LessWrong users lost all of their karma, but a poorly labeled 'fiat@' account on the EA Forum was discovered with no posts and a similarly large amount of karma**

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/0ddc284ec51beb57722a807686544ebee2ad2ad4f3dfa68d.png)

**Habryka states that LessWrong and the EA Forum "work at arms length"**

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/a8b41509d4780fad5ac10c4d83a3b86c280275987e8556d5.png)

**Later, Zvi Mowshowitz publishes** [**a leaked internal accounting sheet**](https://docs.google.com/spreadsheets/d/1Y1A-nhGm_4GzrbcZz4gxbQQiuxPbwB5e8qlIXGHg2So/edit#gid=0) **from the LessWrong team**

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/ead6fd60d97630d29a550b68a7ea0c72c5bbb4294e5f6a08.png)

[Open this link to read!](https://docs.google.com/spreadsheets/d/1Y1A-nhGm_4GzrbcZz4gxbQQiuxPbwB5e8qlIXGHg2So/edit#gid=0)

It includes entries for "weirdness points" "utils" "Kaj_Sotala" "countersignals" and "Anthropic". We recommend all readers [open up the sheet to read in full](https://docs.google.com/spreadsheets/d/1Y1A-nhGm_4GzrbcZz4gxbQQiuxPbwB5e8qlIXGHg2So/edit#gid=0).

**Later, LessWrong filed for internet-points-bankruptcy and Holden Karnofsky was put in charge.**

Karnofsky reportedly said:

> I have over 15 years of nonprofit governance experience. I have been the Chief Executive Officer of GiveWell, the Chief Executive Officer of Open Philanthropy, and as of recently an intern at an AI safety organization. Never in my career have I seen such a complete failure of nonprofit board controls and such a complete absence of basic decision theoretical cooperation as occurred here. From compromised epistemic integrity and faulty community oversight, to the concentration of control in the hands of a very small group of biased, low-decoupling, and potentially akratic rationalists, this situation is unprecedented.

Sadly the authors did not have time to conclude the reporting, though they list other things that happened in a comment below.
# When should a neural network-based approach for plant control systems be preferred over a traditional control method?

As a trained control engineer, I understand that when the plant model is well understood (or even with many uncertainty in the plant dynamic model), designing a control system using traditional methods such as PID control, H-infinity control, or model predictive control is simple (MPC). However, as machine learning and artificial intelligence have advanced, neural network-based approaches for plant control systems have grown in popularity.

In this regard, I'd like to solicit the thoughts and experiences of other control engineers on the occasions in which it is appropriate to pick a neural network-based strategy over conventional control techniques. What benefits and drawbacks come with implementing plant control systems using neural networks? Are there any particular uses or circumstances where a neural network-based approach would be more effective than conventional control techniques? How does one choose the best strategy for a specific plant control issue? Any ideas or information would be greatly appreciated.
# Orthogonality is Expensive

[@beren](https://www.lesswrong.com/users/beren-1?mention=user) discusses the assumption that intelligent systems would be well factored into a world model, objectives/values and a planning system.

He highlights that this factorisation doesn't describe intelligent agents created by ML systems (e.g. model free RL) well. Model free RL agents don't have cleanly factored architectures but tend to learn value functions/policies directly from the reward signal.

Such systems are much less general than their full model based counterpart as policies they learned that are optimal under one reward function may perform very poorly under another reward function.

Yet, contemporary ML favours such systems over their well factored counterparts because the are much more efficient:

*   Inference costs can be paid up front by learning a function approximator of the optimal policy and [amortised](https://www.lesswrong.com/posts/S54HKhxQyttNLATKu/deconfusing-direct-vs-amortised-optimization) over the agent's lifetime
    *   A single inference step can be performed as a forward pass through the function approximator in a non factored system vs searching through a solution space to determine the optimal plan/strategy for well factored systems
*   The agent doesn't need to learn features of the environment that aren't relevant to their reward function
*   The agent can exploit the structure of the underlying problem domain
    *   Specific recurring patterns can be better amortised

Beren attributes this tradeoff between specificity and generality to [no free lunch theorems](https://en.wikipedia.org/wiki/No_free_lunch_in_search_and_optimization).

Attaining full generality is prohibitively expensive; as such full orthogonality is not the default or ideal case, but merely one end of a pareto tradeoff curve, with different architectures occupying various positions along it.

The future of AGI systems will be shaped by the slope of the pareto frontier across the range of general capabilities, determining whether we see fully general AGI singletons, multiple general systems, or a large number of highly specialised systems.
# Planecrash Podcast

I used AWS' text-to-speech service to create a podcast out of [Eliezer's Planecrash Glowfix](https://www.lesswrong.com/posts/SA9hDewwsYgnuscae/projectlawful-com-eliezer-s-latest-story-past-1m-words).

It is available here: [https://anchor.fm/s/a48073e4/podcast/rss](https://anchor.fm/s/a48073e4/podcast/rss)  
(You might need a podcast player which supports RSS links, such as Podcast Addict.)

I already uploaded about 10 episodes of ~10 hours each. I plan to upload at least the whole [mad investor chaos and the woman of asmodeus](https://www.projectlawful.com/posts/4582) (which would make be 35 additional episodes), and maybe more if people are interested.

I did not ask for anyone's permission, and I'm willing to shut it down or to give the credentials to the podcast account to someone else who owns the rights to this story.

*This is not a high quality speech synthesis. What's more, converting a glowfic to an audio format is hard to do well. I've just inserted the characters' names between each section. Moreover, I never read the original work nor listened to the whole audio yet, so some paragraphs might be broken.*
# Towards empathy in RL agents and beyond: Insights from cognitive science for AI Alignment 

This is a talk I gave at the recent AI Safety Europe Retreat (AISER) on my research on obtaining insights from the cognitive science of empathy and applying them to RL agents and LLMs.

**Talk link:** [https://clipchamp.com/watch/6c0kTETRqBc](https://clipchamp.com/watch/6c0kTETRqBc)

**Slides link:** [https://bit.ly/3ZFmjN8](https://bit.ly/3ZFmjN8)

**Talk description:** I begin by presenting a short review on the cognitive science of empathy as a Perception-Action Mechanism (PAM) which relies on self-other overlap at the neuronal level. I continue by presenting the theory of change of this research direction by arguing that inducing self-other overlap as empathy is model agnostic and that it has the potential to avert AI x-risk and be sub-agent stable in the limit. Then I present experimental evidence of the emergence of PAM in RL agents and present a way of inducing PAM in RL agents. I end the talk by discussing how this paradigm could be extended to LLMs. 

**Acknowledgements:** I am thankful for Dr. Bogdan Ionut-Cirstea for inspiring me to look into this neglected research direction, the Long-Term Future Fund for funding the initial deep dive into the literature,  and for Center for AI Safety for funding half of this research as part of their Student Researcher programme. Last but not least, I want to thank Dr. Matthias Rolf for supervising me and providing good structure and guidance. The review on the cognitive science of empathy is adapted from a talk given by Christian Keysers from the Netherlands Institute for Neuroscience.
# Apply to the Cavendish Labs Fellowship (by 4/15)

[Cavendish Labs](https://www.lesswrong.com/posts/JiGwL5nDMTwehP62N/announcing-cavendish-labs) is a new research organization in Vermont focused on technical work on existential risks. We'd like to invite you to [apply to our fellowships](https://cavendishlabs.org/fellowship/) in AI safety and biosecurity!

Positions are open for any time between June 1 and December 10, 2023. We pay a stipend of $1,500/month, plus food and housing are provided. Anyone with a technical background is encouraged to apply, even if you lack specific expertise in these fields.

Applications for summer research fellows are closing **April 15th**. [Apply here!](https://cavendishlabs.org/fellowship/)

![](https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/S8xfeJGER74xwqLta/rthyerve6abqitea3m9h)

Research lab on a river in Vermont, AI artist's conception

(Note: we likely cannot accept people who need visa sponsorship to work in the U.S.)
# AI Summer Harvest

**Trending metaphor**

I've noticed that AI safety advocates have recently been appealing to a new metaphor: the AI summer harvest.

(1) Here's the FLI open letter *Pause Giant AI Experiments:*

> Humanity can enjoy a flourishing future with AI. Having succeeded in creating powerful AI systems, we can now enjoy an "AI summer" in which we reap the rewards, engineer these systems for the clear benefit of all, and give society a chance to adapt. Society has hit pause on other technologies with potentially catastrophic effects on society.^\[5\]^  We can do so here. Let's enjoy a long AI summer, not rush unprepared into a fall.
> 
> [(source)](https://futureoflife.org/open-letter/pause-giant-ai-experiments/)

(2) And here's Yudkowsky talking on the Lex Friedman podcast:

> If it were up to me I would be like — okay, like *this far* and no further.
> 
> Time for the Summer of AI, where we have planted our seeds and now we wait and reap the rewards of the technology we've already developed, and don't do any larger training runs that that."
> 
> [(source, time-stamped)](https://youtu.be/AaTRHFaaPG8?t=139)

Maybe I'm out-of-the-loop, but I haven't seen this metaphor until recently.

**Brief thoughts**

1.  I like the metaphor and I wrote this article to signal-boost it. It's concise, accessible, and intuitive. It represents a compromise (Hegelian synthesis?) between AI decelerationism and AI accelerationism.
2.  The metaphor points at the reason why slowing down AI progress won't (imo) impose significant economic costs — i.e. just as a farmer gains nothing by sowing seeds faster than they can harvest crops, so too will a lab gain nothing by scaling AI models faster than the economy can productise them.
3.  Instead of "AI summer", we should call it an "AI summer harvest" because "AI summer" is an existing phrase referring to fast AI development; trying to capture an existing phrase is difficult, confusing, and impolite. Moreover, "AI summer harvest" does a better job at expressing the key idea.
4.  I think that [buying time](https://www.lesswrong.com/posts/BbM47qBPzdSRruY4z/instead-of-technical-research-more-people-should-focus-on) is probably more important than anything else at the moment. (Personally, I'd endorse a [0.2 OOMs/year target](https://www.lesswrong.com/posts/9xfRjaKDTb57BaGWv/0-2-ooms-year-target-2).)  And I think that "AI summer harvest" should be the primary angle by which we communicate buying-time to policy-makers and the general public.

**Origins**

The metaphor was [accredited](https://twitter.com/ESYudkowsky/status/1636315864596385792) to [@gwillen](https://www.lesswrong.com/users/gwillen?mention=user) on March 16th 2023.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/750cb9637c900ba5898aa25f85f1b54b78f5ccc9d7212c5f.png)
# Consider applying to a 2-week alignment project with former GitHub CEO

Signal-boosting this tweet from Nat Friedman, former github CEO, who says: 

> I'm seeking a researcher/journalist with an interest in AI/rationalism/xrisk for a 1-2 week project. To apply please email [job@nat.org](mailto:job@nat.org) with evidence that you (a) read scifi and (b) understand this tweet:  
> [https://twitter.com/JimDMiller/status/1642978110894391296?s=20](https://twitter.com/JimDMiller/status/1642978110894391296?s=20)

Linked tweet says: 

> What's you response to Orthogonality Thesis+Instrumental Convergence+AI arms race+ likely Mesa Optimization+ rapid recent progress in AI via ML = we are likely doomed? I have a PhD in econ so I do understand risk.

I don't know much about this nor I do have any affiliation with it. But it seems pretty cool to me and at least worth considering for people in the target reference class!
# Best Ways to Try to Get Funding for Alignment Research?

Hey Everyone! I recently left my FAANG job to split my time between doing Alignment Research (70%) and investigating start-up ideas (30%).

If I decide to fully commit to Alignment Research, what is the best way to go about applying for and/or getting funding?  (In a perfect world, this funding would cover compute and SF living expenses).

Thanks! RGRGRG
# Running many AI variants to find correct goal generalization

*Problem: An AI may incorrectly generalize a goal we present it.*

*Proposal: Run many slightly different instances of AI and find the “Schelling point” around correct generalization.*

The idea is that if we run many similar AIs or copies of the same AI in parallel, we will be able to detect goal’s mis-generalizations, as such wrong generalizations will be different in most AIs. The only “Schelling point” will be the correct generalization, at which many AIs will converge. 

For example, we want AI to clean a room from bowling balls. We ask AI to “remove all spherical objects”. We run many slightly different copies of the AI and they produce the following suggestions:

1st AI: suggests to remove balls

2nd AI: suggests to remove balls

3d AI: suggests to remove balls and lamps

4th AI: suggests to remove balls and human heads.

1st and 2nd AI have come to the same conclusion, and it is the right one. 3rd and 4^th^ AIs have different conclusions and these conclusions are wrong. 4th AI’s conclusion is dangerous.

Counterarguments:

1.  Some misgeneralizations could also be Schelling points. 
2.  Powerful AIs can guess that other similar AIs exists and start some adverse cooperation. 

The first problem could be (partly) solved by proper randomisation of initial parameters, weights or inputs. For example, a question could be presented several times in different forms for different instances of the AI. E;g.: “Remove all round objects”, “remove all circular objects”, “remove all sport utilities”. 

There could be two “peaks” in generalization which means that our presented goal has intrinsic ambiguity. We could then ask AI why it has different interepations.

In real life, this approach is also used: we could say a driver “Turn left. Turn there on Main Street. Here!” In that case, we formulate our goal several times in different words hoping that it will eventually converge into correct model inside driver’s mind. The driver’s mind selects solution which satisfies all three commands, and disregard those solutions which satisfy only one command.

Another example: If we have a set of calculators which either produce correct answer or wrong one, we could find the broken calculators as they all will produce different answers, and all correct calculators will produce the same answer.
# Steelman / Ideological Turing Test of Yann LeCun's AI X-Risk argument?

Yann LeCun has been saying a lot of things on social media recently about this topic, only some of which I've read. He's also written and talked about it several times in the past. Most of what I've seen from him recently seems to not be addressing any of the actual arguments, but on the other hand I know he's discussed this in many forums over several years, and he's had the arguments spelled out to him so many times by so many people that it's hard for me to believe he really doesn't know what the substantive arguments are. Can someone who's read more of Yann's arguments on this please give their best understanding of what he's actually arguing, in a way that will be understandable to people who are familiar with the standard x-risk arguments?
# Play My Futarchy/Prediction Market Mafia Game

In the social deduction game [Mafia](https://en.wikipedia.org/wiki/Mafia_(party_game)) (broadly the same game as Werewolf, Town of Salem, etc) the minority mafiosi—who know each other's identities—play against the majority villagers in alternating day phases, where the players select someone to kill, and night phases, where the mafia picks someone to kill while select players perform secretly assigned roles like investigating others or protecting them from attack.

I'm hosting a game of mafia where instead of voting, there’s a live prediction market where people can buy and sell shares in each player. The highest-priced player at the end of each day phase will be executed and their role and faction made public, at which point their shares will pay out $100 if they’re revealed to be in the mafia.

I hosted a game like this before with 40 players over the pandemic but the execution was poor—only a quarter of them really understood how the market worked and what the prices meant, the bot running the market crashed at one point, and the game used normal voting (gaining more money allowed you to buy bonus items at daily auctions instead).

The game will take place over about a week. Each day/night cycle in the game will last about one calendar day, so you don’t need to pay constant attention to the chatter but can just check in, post your thoughts, and bet on the market when you’re free. I might stretch out the phases depending on what time zones need to be accommodated.

I’m looking for 24 players. You can also join the Discord server just to spectate: [https://discord.gg/D5DQVs3wP4](https://discord.gg/D5DQVs3wP4)

More rules explanation: [https://docs.google.com/document/d/1DfAzHfAprjK9sjSHMvRNjWz866NIX41YVp3ktBrqVIU/edit](https://docs.google.com/document/d/1DfAzHfAprjK9sjSHMvRNjWz866NIX41YVp3ktBrqVIU/edit)
# Penalize Model Complexity Via Self-Distillation

When you self-distill a model (e.g. train a new model using predictions from your old model), the resulting model represents a less complex function. After many rounds of self-distillation, you essentially end up with a constant function. [This](https://proceedings.neurips.cc/paper/2020/hash/2288f691b58edecadcc9a8691762b4fd-Abstract.html) paper makes the above more precise.

Anyway, if you apply multiple rounds of self-distillation to a model, it becomes less complex. So if the original model learned complex, power-seeking behaviors that doesn't help it do well on the training data, this behavior would likely go away after several rounds of self-distillation. Self-distillation allows you to essentially get the minimum complexity model that still does well on the test set. Thus, I think it's promising from an AI safety standpoint.
# New survey: 46% of Americans are concerned about extinction from AI; 69% support a six-month pause in AI development

YouGov America released a survey of 20,810 American adults. Highlights below. Note that I didn't run any statistical tests, so any claims of group differences are just "eyeballed."

*   46% say that they are "very concerned" or "somewhat concerned" about the possibility that AI will cause the end of the human race on Earth (with 23% "not very concerned, 17% not concerned at all, and 13% not sure). 
*   There do not seem to be meaningful differences by region, gender, or political party.
*   Younger people seem more concerned than older people.
*   Black individuals appear to be somewhat more concerned than people who identified as White, Hispanic, or Other.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/bda09cb0bf5cd530f7e7475b31020961a019942a9beffea3.png)

Furthermore, 69% of Americans appear to support a [six-month pause in "some kinds of AI development"](https://today.yougov.com/topics/technology/survey-results/daily/2023/04/03/ad825/2). Note that there doesn't seem to be a clear effect of age or race for this question. (Particularly if you lump "strongly support" and "somewhat support" into the same bucket). Note also that the question mentions that 1000 tech leaders signed an open letter calling for a pause and cites their concern over "profound risks to society and humanity", which may have influenced participants' responses. 

In my quick skim, I haven't been able to find details about the survey's methodology (see here for info about [YouGov's general methodology](https://today.yougov.com/about/panel-methodology/)) or the credibility of YouGov (**EDIT**: Several people I trust have told me that YouGov is credible, well-respected, and widely quoted for US polls). 

See also: 

*   [The public supports regulating AI for safety](https://www.lesswrong.com/posts/M3iPAmxZwy4gPXdXw/the-public-supports-regulating-ai-for-safety) 
*   [The Overton Window widens: Examples of AI risk in the media](https://www.lesswrong.com/posts/SvwuduvpsKtXkLnPF/the-overton-window-widens-examples-of-ai-risk-in-the-media)
*   [Spreading messages to help with the most important century](https://www.lesswrong.com/posts/2y4BkkQwsG7n2Ensc/spreading-messages-to-help-with-the-most-important-century)
*   [Surveys of US public opinion on AI](https://wiki.aiimpacts.org/doku.php?id=responses_to_ai:public_opinion_on_ai:surveys_of_public_opinion_on_ai:surveys_of_us_public_opinion_on_ai)
# Has anyone thought about how to proceed now that AI notkilleveryoneism is becoming more relevant/is approaching the Overton window?

I figure that this would be a pretty important question since getting governments on board, and getting people to coordinate in general, could buy us a lot of time, but I haven't seen anyone talk about this before. I would like to hear what people think and what their strategies are.
# Auto-GPT: Open-sourced disaster?

Sharing this here doesn't seem like an infohazard at this point. This is all over my YouTube feed anyway.

Description from the authors:

> Auto-GPT is an experimental open-source application showcasing the capabilities of the GPT-4 language model. This program, driven by GPT-4, autonomously develops and manages businesses to increase net worth. As one of the first examples of GPT-4 running fully autonomously, Auto-GPT pushes the boundaries of what is possible with AI.

I wanted to additionally call out this in their read me:

> 💀 Continuous Mode ⚠️
> ---------------------
> 
> Run the AI **without** user authorisation, 100% automated. Continuous mode is not recommended. It is potentially dangerous and may cause your AI to run forever or carry out actions you would not usually authorise. Use at your own risk.
> 
> 1.  Run the `main.py` Python script in your terminal:
> 
>     python scripts/main.py --continuous
>     
>     
> 
> 1.  To exit the program, press Ctrl + C

Nice! Super nice. Super safe and super good.
# Someone already tried "Chaos-GPT"

With the new Auto-GPT release, I was surprised to see how quickly some people turned towards using it for nefarious goals, apparently just for fun. Is this the first piece of clear cut, empirical evidence for the argument against open sourcing these soon-to-be very powerful general purpose technologies? You don't need 100 evil people with access to open source AGI to destroy the world, you just need one bro in his room feeding the wrong prompt for shits and giggles.

https://m.youtube.com/watch?v=g7YJIpkk7KM&feature=youtu.be