Harley-ml/lesswrong huggingface dataset:
# Project Idea: Challenge Groups for Alignment Researchers

[@Henrik Karlsson](https://www.lesswrong.com/users/henrik-karlsson?mention=user) writes a lot about what makes people successful. One recurring theme I've found involves the people around them. Successful people almost always have a group of friends who they're constantly chatting with, bouncing ideas around with, learning from, pushing each other, supporting each other, etc. A group who is similarly talented and driven.

Consider the [example of MrBeast](https://www.henrikkarlsson.xyz/p/mrbeast):

> > **MrBeast:** I somehow found these other four lunatics. Three of us were college dropouts. One was a high school dropout. And one, he just like quit his job. We were all super small YouTubers.
> > 
> > And we basically talked every day for a thousand days in a row and did nothing but hyper-study what makes a good video. \[. . .\] Some days I'd get on Skype at 7 am and I’d be in the call until 10 pm and then I’d go to bed. I wake up and I do it again.
> 
> He had summoned a scene.
> 
> > **MrBeast:** We all had like 10-20,000 subscribers when we met. And by the time we stopped talking, we all had millions of subscribers. We all hit a million subscribers within a month of each other.
> 
> There was, in other words, little luck involved. They collectively extracted the know-how needed to consistently do what they wanted to do, which was to go viral.

Or consider [musicians in Sweeden](https://www.henrikkarlsson.xyz/p/scene-creation-engines). Or intellectuals who grew up in [exceptional *milieus*](https://www.lesswrong.com/posts/CYN7swrefEss4e3Qe/childhoods-of-exceptional-people).

I like to think of this as a "challenge group". It seems like a pretty huge force multiplier.

I'm not sure about the extent to which alignment researchers currently lack challenge groups, but to the extent they do, helping form them seems like a pretty impactful project. It also strikes me as something that is, and is likely to remain, moderately neglected.

Similarly, you could also help form challenge groups in other areas: aspiring alignment researchers, cryonicists, EAs more generally, rationality dojos. But to increase the likelihood that new members are matched with existing members for whom they're compatible, it's probably a good idea to start off with a somewhat narrow group.

I have more thoughts on this that I'm struggling to formulate, so I'll leave it here for now. In general I think [@Henrik Karlsson](https://www.lesswrong.com/users/henrik-karlsson?mention=user) has a lot of great things to say on this topic.
# Devil's Advocate: Adverse Selection Against Conscientiousness 

*Partially in reply to "*[*Request: stop advancing AI capabilities*](https://www.lesswrong.com/posts/N7DxcLCjfBpEv3QwB/request-stop-advancing-ai-capabilities)*".*

Consider this argument:

> "\[X field\] is often \[dangerous / bad\]. If you're \[pro-social / thoughtful / conscientious\], please avoid \[X field\]."

Let's grant that these points as true. X field is indeed bad or dangerous, people who are some mix of pro-social and thoughtful and conscientious will recognize that, and they are exhorted not to join that field.

What happens next, causally?

(1) If the field has a limited supply of people with relevant background to make contributions in the field, the result will be to slow progress in the field.

(2) If the field is below a "critical mass" of contributors to it, it might fail to make progress entirely.

(3) *However*, if the field *does* already have a critical mass of people making contributions, **the end result will be adverse selection against pro-sociality, thoughtfulness, and conscientiousness in the field.**

I have a profound respect, at a distance, for MIRI and Soares. I think we actually agree about the dangers, and I agree that interpretability research (for instance) is a very important under-resourced problem that would benefit from contributors.

But I also come to one different conclusion — the more thoughtful, pro-social, and conscientious people working at Meta, OpenAI, Google, etc... the more likely things go well. I think there's a critical mass already in place for continued growth of machine learning capabilities on the path to AGI, and having people who care and are thoughtful in the room participating in their development is a good thing and not a bad thing. I would respectfully advocate the opposite position of Soares, and encourage thoughtful people to consider joining the strongest teams in the field and being a voice of reason in the room while shaping development.
# Gemini will bring the next big timeline update

There is a genre of LLM critique that criticises LLMs for being, well, LLMs. 

Yann LeCun for example points to the inability of GPT-4 to visually imagine the rotation of interlocking gears as a fact that shows how far away AGI is, instead of a fact that shows how GPT-4 has not been trained on video data yet. 

There are many models now that "understand" images or videos or even more modalities. However, they are not end-to-end trained on these multiple modalities. Instead they use an intermediary model like CLIP, that translates into the language domain. This is a rather big limitation, because CLIP can only represent concepts in images that are commonly described in image captions. 

Why do I consider this a big limitation? Currently it looks like intelligence emerges from learning to solve a huge number of tiny problems. Language seems to contain a lot of useful tiny problems. Additionally it is the interface to our kind of intelligence, which allows us to assess and use the intelligence extracted from huge amounts of text. 

This means that adding a modality with a CLIP-like embedding and than doing some fine-tuning does not add any intelligence to the system. It only adds eyes or ears or gears. 

Training end-to-end on multi-modal data should allow the model to extract new problem solving circuits from the new modalities. The resulting model would not just have eyes, but visual understanding. 

Deepmind did a mildly convincing proof-of-concept with Gato last year, a small transformer trained on text, images, computer games and robotics. Now it seems they will try to scale Gato to Gemini, leapfrogging GPT-4 in the process. 

GPT-4 itself has image processing capabilities that are not yet available to the general public. But whether these are an add-on or result of integrated image modelling we don't know yet. 

To me it seems very likely, that a world where the current AI boom fizzles is a world where multi-modality does not bring much benefits or we cannot figure out how to do it right or possibly the compute requirements of doing it right is still prohibitive. 

I think Gemini will give us a good chunk of information about whether that is the world we are living in.
# What are some of the best introductions/breakdowns of AI existential risk for those unfamiliar?

Discussion of AI risk has gone mainstream in the past few months, and as usual whenever that happens, many of the people new to the field think they're an expert and proceed to engage in the exact same arguments over and over. I think it would be convenient to have the following available:

*   An introduction to the core risk, explaining why it's possible.
*   An index of common bad arguments about AI risk and in-depth responses to them.
*   An index of common good arguments about AI risk, and links to further reading about them.

All of these should require no background knowledge and be accessible to normal people with no philosophical or mathematical experience.

I was thinking of writing up such a guide, but I don't want to duplicate effort. Does anything like this already exist?
# Wikipedia as an introduction to the alignment problem

AI researchers and others are increasingly looking for an introduction to the alignment problem that is clearly written, credible, and supported by evidence and real examples. The Wikipedia article on AI Alignment has become such an introduction.

Link: [**https://en.wikipedia.org/wiki/AI_alignment**](https://en.wikipedia.org/wiki/AI_alignment)

Aside from me, it has contributions from Mantas Mazeika, Gavin Leech, Richard Ngo, Thomas Woodside (CAIS), Sidney Hough (CAIS), other Wikipedia contributors, and copy editor Amber Ace. It also had extensive feedback from this community.

In the last month, it had ~20k unique readers and was cited by [Yoshua Bengio](https://yoshuabengio.org/2023/05/22/how-rogue-ais-may-arise/).

We've tried hard to keep the article accessible for non-technical readers while also making sense to AI researchers.

I think Wikipedia is a useful format because it can include videos and illustrations (unlike papers) and it is more credible than blog posts. However, Wikipedia has strict rules and could be changed by anyone.

Note that we've announced this effort on the Wikipedia talk page and shared public drafts to let other editors give feedback and contribute.

I you edit the article, please keep in mind Wikipedia's rules, use [reliable](https://en.wikipedia.org/wiki/Wikipedia:Reliable_sources) sources, and consider that we've worked hard to keep it concise because most Wikipedia readers spend <1 minute on the page. For the latter goal, it helps to focus on edits that reduce or don't increase length. To give feedback, feel free to post on the talk page or message me. Translations would likely be impactful.
# Sentience matters

Short version: Sentient lives matter; AIs can be people and people shouldn't be owned (and also the goal of alignment is not to browbeat AIs into doing stuff we like that they'd rather not do; it's to build them de-novo to care about valuable stuff).

Context: Writing up obvious points that I find myself repeating.

---

_Note: in this post I use "sentience" to mean some sort of sense-in-which-there's-somebody-home, a thing that humans have and that cartoon  depictions of humans lack, despite how the cartoons make similar facial expressions. Some commenters have noted that they would prefer to call this "consciousness" or "sapience"; I don't particularly care about the distinctions or the word we use; the point of this post is to state the obvious point that there is some property there that we care about, and that we care about it independently of whether it's implemented in brains or in silico, etc._

---

Stating the obvious:

- All sentient lives matter.
  - Yes, including animals, insofar as they're sentient (which is possible in at least some cases).
  - Yes, including AIs, insofar as they're sentient (which is possible in at least some cases).
  - Yes, even including sufficiently-detailed models of sentient creatures (as I suspect could occur frequently inside future AIs). (People often forget this one.)

- Not having a precise definition for "sentience" in this sense, and not knowing exactly what it is, nor exactly how to program it, doesn't undermine the fact that it matters.

- If we make sentient AIs, we should consider them people in their own right, and shouldn't treat them as ownable slaves.
  - Old-school sci-fi was basically morally correct on this point, as far as I can tell.

---

Separately but relatedly:

- The goal of alignment research is not to grow some sentient AIs, and then browbeat or constrain them into doing things we want them to do even as they'd rather be doing something else.
- The point of alignment research (at least according to my ideals) is that when you make a mind _de novo_, then what it ultimately cares about is something of a free parameter, which we should set to "good stuff".
  - My strong guess is that AIs won't _by default_ care about other sentient minds, and [fun](https://www.lesswrong.com/posts/K4aGvLnHvYgX9pZHS/the-fun-theory-sequence) broadly construed, and flourishing civilizations, and love, and that it also won't care about any other stuff that's deeply-alien-and-weird-but-wonderful.
  - But we could _build_ it to care about that stuff--not coerce it, not twist its arm, not constrain its actions, but just build another mind that cares about the grand project of filling the universe with lovely things, and that joins us in that good fight.
  - And we should.

---

(I consider questions of what sentience really is, or consciousness, or whether AIs can be conscious, to be off-topic for this post, whatever their merit; I hereby warn you that I might delete such comments here.)
# Statement on AI Extinction - Signed by AGI Labs, Top Academics, and Many Other Notable Figures

Today, the AI Extinction Statement was released by the [Center for AI Safety](https://safe.ai/), [a one-sentence statement](https://www.safe.ai/statement-on-ai-risk) jointly signed by a historic coalition of AI experts, professors, and tech leaders.

Geoffrey Hinton and Yoshua Bengio have signed, as have the CEOs of the major AGI labs–Sam Altman, Demis Hassabis, and Dario Amodei–as well as executives from Microsoft and Google (but notably not Meta).

The statement reads: **“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”**

We hope this statement will bring AI x-risk further into the overton window and open up discussion around AI’s most severe risks. Given the growing number of experts and public figures who take risks from advanced AI seriously, we hope to improve epistemics by encouraging discussion and focusing public and international attention toward this issue.
# Who is liable for AI?

\[Edit -- Changed to title as suggested by Raemon. So I've also made some edits to the post itself.\]

Where does liability fall in the case of AI doing things?

I think there is a rather close parallel with autonomous cars/vehicles. Probably 4 or 5 years back that was a question being raise. I assume since I don't hear much about it any more it's been resolved and clearly defined for those making and using such vehicles or insuring such vehicles. Perhaps that translates very well into the more general AI space. But I'm not sure.

Anyone aware of where the whole question of liability assignment and insurance related to producing and using very advanced AI tools and products in our day to day interactions currently stands?
# LIMA: Less Is More for Alignment

Abstract
--------

> Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences. We measure the relative importance of these two stages by training LIMA, a 65B parameter LLaMa language model fine-tuned with the standard supervised loss on ***only 1,000 carefully curated prompts and responses***, without any reinforcement learning or human preference modeling. LIMA demonstrates remarkably strong performance, learning to follow specific response formats from only a handful of examples in the training data, including complex queries that range from planning trip itineraries to speculating about alternate history. Moreover, the model ***tends to generalize well*** to unseen tasks that did not appear in the training data. In a controlled human study, responses from LIMA are either ***equivalent or strictly preferred to GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard and 65% versus DaVinci003***, which was trained with human feedback. Taken together, these results strongly suggest that almost all knowledge in large language models is learned during pretraining, and ***only limited instruction tuning data is necessary to teach models to produce high quality output.***

Implications
------------

### Data Quality & Capabilities

Along with [TinyStories](https://www.alignmentforum.org/posts/dMoaBvcxpBE7LcES4/tinystories-small-language-models-that-still-speak-coherent) and [QLoRA](https://twitter.com/Tim_Dettmers/status/1661379363907031040) I'm becoming increasingly convinced that *data quality is all you need*, definitely seems to be the case for finetuning, and may be the case for base-model training as well. Better scaling laws through higher-quality corpus?

Also for who haven't updated, it seems *very likely* that GPT-4 equivalents will be *essentially free* to self-host and tune within a year. Plan for this!

### Perplexity != Quality

> When fine-tuning LIMA, we observe that ***perplexity*** on held-out Stack Exchange data (2,000 examples) ***negatively correlates with the model’s ability to produce quality responses***. To quantify this manual observation, we evaluate model generations using ChatGPT, following the methodology described in Section 5. Figure 9 shows that as perplexity rises with more training steps – which is typically a negative sign that the model is overfitting – so does the quality of generations increase. Lacking an intrinsic evaluation method, we thus resort to manual checkpoint selection using a small 50-example validation set.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/19e9f70df200e283a03c82fc27696583bd8ff375d62aab76.png)

Because of this, the authors manually select checkpoints between the 5th and 10th epochs (out of 15) using the held-out 50-example development set.
# Humans, chimpanzees and other animals

\[Epistemic status: underinformed musings; I am posting this not because I am sure of anything in it but because the point seems important and I don't recall seeing anyone else make it. Maybe that's because it's wrong.\]

A common analogy for the relationship between postulated superhumanly-smart AIs and humans is the relationship between humans and chimpanzees. See, e.g., [https://intelligence.org/2017/12/06/chollet/](https://intelligence.org/2017/12/06/chollet/) where Eliezer Yudkowsky counters various arguments made by François Chollet by drawing this analogy.

It's pretty compelling. Humans are a bit like chimps but substantially smarter ... and, lo, humans control the future of life on earth (including, in particular, the future of *chimpanzee* life on earth) in a way chimps absolutely do not.

But wait. Humans coexist with chimps, and are smarter, and are utterly dominant over them: fair enough. But surely we want to do better than a sample size of 1. What *other* cases are there where animals of different intelligence levels coexist?

Well, for instance, chimps coexist with lions and leopards. Are chimp-leopard relations anything like human-chimp or human-leopard relations? So far as I can tell, no. Chimps don't appear to reshape their environment radically for their own purposes. When they encounter other animals such as lions and leopards they not infrequently get killed.

In general I'm not aware of any pattern of the form "smarter animals consistently face no threat from less-smart animals" or "smarter animals consistently wipe out less-smart animals that threaten them" or "the smartest type of animal in any area controls the whole local ecosystem".

(I am not an expert. I could be all wrong about this. I will be glad of any corrections.)

What all this suggests, at least to me, is that what's going on with humans and chimpanzees is not "smarter animal wins" but something more like "there is a qualitative difference between humans and other animals, such that animals on the smarter side of this divide win against ones on the less-smart side".

There *might* be a similar divide between us and our prospective AI overlords. I can think of various things that might turn out to have that effect. But so far as I can tell, it's a genuinely open question, and if there's some reason to be (say) 90% confident that there will be such a divide I haven't seen it.
# Product Endorsement: Food for sleep interruptions

Do you wake up before you want to and struggle with getting back to sleep when you do? One thing (out of many) that can cause this is low blood sugar*. If this is your issue, it is easily solved by eating high-fat, high-protein food.

My evidence on this is purely anecdotal; I learned this from r/adviceanimals in 2012, so the intellectual pedigree is weak. But I added ~one full sleep cycle to my night when I tried it, which lasted for years until I was forced to stop due to medication requiring an empty stomach (I now use ketone esters). A month ago my dad went from “3-5 hours/night, 3 hours awake, maybe more sleep after that” to “8-9 hours of near-continuous sleep, 1 night per fortnight a mere 6 hours” by keeping peanut butter by his bed. Downside: he’s now useless for testing the [magic vibrating bracelet](https://acesounderglass.com/2023/05/08/product-endorsement-apollo-neuro/) I’m so hyped about.

This is very easy to test. Keep some nuts by your bed; if you wake up, eat some. If you fall back asleep, congratulations on fixing your problem. There are more complicated things you can do with your evening diet to try and prevent early wake-ups entirely but I feel very muddled about these and don’t have good suggestions.

If you’re doing intermittent fasting, [ketone esters](https://acesounderglass.com/2022/02/09/my-tiny-study-finds-no-effect-for-ketone-esters-as-a-nootropic/) ([affiliate link](https://amzn.to/43dJZe8)) don’t work for many people but are great if you’re one of the lucky ones (I am), and ~don’t count as eating. MCT oils ([affiliate link](https://amzn.to/43cvzuK)) work on the same principle and so may be an alternative, but I haven’t tested them personally.

I am not a doctor, anything with a real effect can hurt you, be careful not to choke or attract ants, etc. 

*It’s not clear to me if this is actually low blood sugar as measured glucose in your bloodstream, or naturopathic “low blood sugar”, meaning problems you fix by eating, but I don’t care.
# Against Conflating Expertise: Distinguishing AI Development from AI Implication Analysis

When discussing AI, we often associate technological proficiency with an understanding of AI's broader implications. This notion is as misguided as believing that a master printing press mechanic from Gutenberg's era would have been best equipped to predict the impacts of the printing press.

The AI development field, represented by experts like Geoffrey Hinton (proponent of safety measures and potential regulation) or Yann LeCun (an advocate of AI acceleration with fewer restrictions), primarily concerns itself with technical prowess and the creation of AI algorithms. In contrast, understanding the expansive implications of AI necessitates a comprehensive, multidisciplinary approach encompassing areas like warfare, game theory, sociology, philosophy, ethics, economics, and law, among others.

In this context, the [debate](https://www.lepoint.fr/sciences-nature/yuval-harari-sapiens-versus-yann-le-cun-meta-on-artificial-intelligence-11-05-2023-2519782_1924.php) between Yann LeCun and historian-author Yuval Noah Harari offers a glimpse into these diverging standpoints. While there's an understandable tendency to lend more credence to LeCun's perspective because of his technical accolades, this thought process might be a misstep when discussing the broader implications of AI.

On the other hand, Harari brings forth a more holistic perspective shaped by his wide-ranging understanding of human history, sociology, and philosophy. With his expertise in scrutinizing historical shifts and societal changes, Harari may be better prepared to explore the implications of AI because understanding implications and creating AI systems require different skill sets.

Those who can fully grasp AI's overarching impacts might not be technical experts but rather polymaths.

This post is a response to an observed bias in discussions surrounding AI: the undue weight given to technical experts' perspectives regarding the broad implications of AI. (Like in the last [AI risk letter](https://time.com/6283386/ai-risk-openai-deepmind-letter/)), while downplaying or ignoring the views of non-technical experts.
# Arguments Against Fossil Future?

I'm currently most of the way through Alex Epstein's Fossil Future, and I find the arguments within very convincing.

In (extreme) brevity, they are:

1.  Fossil Fuel use and the cheap energy it enables is responsible for unprecedented human flourishing for billions. Billions more need access to ever-increasing amounts of cheap energy to flourish.
2.  Any argument against fossil fuel use must argue that its side effects (CO2 warming the planet) overwhelm the good they do by providing cheap energy. These side effects must be so bad that it's worth compromising the safety and flourishing of billions of humans to curtail their use. Such an argument must also prove that those negative side effects are beyond what humanity is capable of adapting to or overcoming, given cheap energy provided by fossil fuels.
3.  No such argument is justifiable, given the current state of climate science.

Does anyone (preferably those who've read the book, although I don't want to restrict answers to just those people) have an opposing view/opinion, and if so why?

I'd like to do my intellectual homework on this one, and actively seek disagreement, given how convincing I've found the argument so far.
# Improving Mathematical Reasoning with-Process Supervision

> **Alignment impact**
> --------------------
> 
> Process supervision has several alignment advantages over outcome supervision. It directly rewards the model for following an aligned chain-of-thought, since each step in the process receives precise supervision. Process supervision is also more likely to produce interpretable reasoning, since it encourages the model to follow a human-approved process. In contrast, outcome supervision may reward an unaligned process, and it is generally harder to scrutinize.
> 
> In some cases, safer methods for AI systems can lead to reduced performance[^3^](https://openai.com/research/improving-mathematical-reasoning-with-process-supervision#fn-3), a cost which is known as an *alignment tax*. In general, any alignment tax may hinder the adoption of alignment methods, due to pressure to deploy the most capable model. Our results below show that process supervision in fact incurs a negative alignment tax, at least in the math domain. This could increase the adoption of process supervision, which we believe would have positive alignment side-effects.

Paper [here](https://cdn.openai.com/improving-mathematical-reasoning-with-process-supervision/Lets_Verify_Step_by_Step.pdf).

Related [Prediction Market](https://www.metaculus.com/questions/8840/ai-performance-on-math-dataset-before-2025/).
# [Request]: Use "Epilogenics" instead of "Eugenics" in most circumstances

I am conditionally in favor of human genetic augmentation. In particular, I think embryo selection for intelligence, health, happiness and other positive traits is desirable. When I bring this up with people, one of the most frequent questions is "Isn't that eugenics?"

I never know how to respond. The term "eugenics" has absorbed so much baggage over the last century that it somehow refers both to swiping right on Tinder when you see an attractive person and to the holocaust.

These are not similar concepts. The fact that we use a single word to refer to both is crazy. I cannot count the number of debates I've heard about human genetic engineering where the disagreement boils down to people misunderstanding what the other is advocating for. One person will talk about the benefits of improving the genes of future generations, and the other person will attack the idea because they mentally associate it with "eugenics", and to them that means "the holocaust" and "state sponsored sterilization of people with disabilities".

The fault lies with the term itself. It means so many different things to so many people that its use actively hinders understanding. For this reason, I think we should start using a new term to talk about non-coercive means of improving human genetics.

Epilogenics: non-coercive means of changing genetics
====================================================

This term originated with Aella on Twitter, and I think it perfectly fits the purpose I have in mind.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/dec5c04ba5d7edca8e1a5a1f76b8dcec12647ac037f76dc2.png)

Examples of epilogenics
-----------------------

*   Selecting an embryo for lower disease risk, higher intelligence, or some other trait good for both the individual and society
*   Gene editing for the purposes listed above
*   Choosing an attractive spouse

Examples of things that are not epilogenics
-------------------------------------------

*   State-sponsored sterilization of people deemed “unfit”
*   Rules against marriage of family members such a siblings and cousins
*   Things people think of as eugenics even though they are often bad for genes (i.e. genocide)

I would encourage the use of this term to clarify the difference between genocide and screening one's embryos for desirable traits.
# Experimental Fat Loss

With the end of the world nigh, and a public panic about to start, this seems an ideal time to worry about weight loss and the obesity epidemic. 

Coincidentally, for the first time in my life, I'm getting fat.

SlimeMoldTimeMold's 'Chemical Hunger' series 

[https://slimemoldtimemold.com/2021/07/07/a-chemical-hunger-part-i-mysteries/](https://slimemoldtimemold.com/2021/07/07/a-chemical-hunger-part-i-mysteries/)

seemed to draw a lot of interest round these parts, and even if it's not lithium

[https://www.lesswrong.com/posts/7iAABhWpcGeP5e6SB/it-s-probably-not-lithium](https://www.lesswrong.com/posts/7iAABhWpcGeP5e6SB/it-s-probably-not-lithium)

it does seem to me that the molds raise some most interesting questions.

I find the whole 'seed oil' craziness to be a compellingly interesting argument, although, as Scott Alexander wrote:

[https://slatestarcodex.com/2020/03/10/for-then-against-high-saturated-fat-diets/](https://slatestarcodex.com/2020/03/10/for-then-against-high-saturated-fat-diets/)

it does seem to be flat wrong. But I think it's important to be interested in ideas that look like they have to be right but aren't.

I want to draw everyone's attention to the 'Experimental Fat Loss' substack

[https://exfatloss.substack.com](https://exfatloss.substack.com)

Which seems to me the very model of sanity and empiricism, a little like reading the early Proceedings of the Royal Society, were Robert Hooke to have become interesting in losing weight.

In particular his definition of what it would mean for a diet to 'work'

[https://exfatloss.substack.com/p/the-definition-of-diet-success](https://exfatloss.substack.com/p/the-definition-of-diet-success)

He does seem to have found something that works for him, 

[https://exfatloss.substack.com/p/losing-43lbs-in-144-days-on-ex150-diet](https://exfatloss.substack.com/p/losing-43lbs-in-144-days-on-ex150-diet)

and I find him sufficiently trustworthy-seeming that I'm going to see if it will work for me, and if it doesn't, use his methods to play around and see if I can find something that does. 

But I would welcome the comments of wiser and more sceptical persons on all these things.
# self-improvement-executors are not goal-maximizers

Agents are [adaptation-executors](https://www.lesswrong.com/tag/adaptation-executors) with adaptations that accomplish goals, not goal-maximizers. Understanding agents as maximizing goals is a simplification used by humans to make them easier to understand. This is as true when the goal is self-improvement as it is with anything else.

Let's think about self-improvement in terms of goal specification. You want to accomplish a goal G, so as you add levels, the task becomes more indirect:
- make a system S0 to accomplish G
- make S1 to make a S0 that accomplishes G
- make S2 to make a S1 that makes a S0 that accomplishes G

Some people think that you can just tell S2 to do G and it will figure things out, but those people are interpreting S2 as a goal-maximizer, and agents are adaptation-executors. You have to specify the **adaptations** that will cause the **actions** that create a suitable S1. Reinforcement learning is a method ([not](https://en.wikipedia.org/wiki/Neuroevolution) the only method!) for specifying adaptations that accomplish a certain goal in the training environment, not a way of giving a system an inherent goal. I do think AI self-improvement is possible - I wrote [this](https://www.lesswrong.com/posts/rSycgquipFkozDHzF/ai-self-improvement-is-possible). So, I do think in some sense that specification of S2 can be easier than direct specification of S0, but I'm not under any illusions that it's the **same problem**.

I'm posting this because I saw [this post](https://www.lesswrong.com/posts/FgsoWSACQfyyaB5s7/shutdown-seeking-ai) (from yesterday) saying that people should try designing AIs that want to be shut down as a tripwire for dangerous recursive self-improvement, and that idea is based on the above misunderstanding. Also, "shut yourself down" is a particularly poor goal for 2 reasons.

1. It's binary. Training ML systems works much better when you can indicate partial accomplishment of the goal.

2. It's ill-defined. If you try to train a system to "shut itself down" what you are training it to do is "take provided action X that also coincidentally causes shutdown". You then need to make things harder if you're testing development of more-capable systems, and if you then take away X but leave other shutdown route Y, it's no more likely to do Y than anything else. Having negative average rewards over time doesn't train it to do Y until it does Y a bunch and gets reinforcement learning on doing Y.
# Winning doesn't need to flow through increases in rationality

*(My first LessWrong post. Written quickly, in the spirit of putting more of my ideas out there and seeing how they meet reality.) *

**Consider that there may be more rewarding pursuits than marginally increasing your rationality toolbox.**

I get the sense that a lot of people in the rationality community think that others should dedicate themselves to the pursuit of becoming more rational, or that they themselves need to become more rational. 

> "If only they had read the Sequences, then they wouldn't have wasted their time on that class"; 
> 
> "if only I had used the [internal double crux](https://www.lesswrong.com/posts/x2KrcscqgKDk6pMeD/internal-double-crux-1) move, then I wouldn't have made X mistake."

There is probably some truth here, but I think it's possible to be *too* fixated on increasing rationality, or hold the view that all shortcomings would have best been solved by more engagement with rationality. 

[**Rationality is systematized winning**](https://www.lesswrong.com/tag/rationality-is-systematized-winning)**, and winning doesn't need to flow through increases in rationality.**

. . . 

Take Alice as an example. She's motivated to reduce suffering in the world as much as possible; an effective altruist, if you will. 

As she first orients to her goal of reducing suffering, Alice may[^n9972zs1okk] benefit a lot from engaging with rationality content. Scout mindset, some CFAR techniques, and some the Sequences will set her up to see the world clearer and make wiser decisions in the future.

But after some initial orientation to rationality and further readings when she feels like it, Alice may better achieve her goals by working on other [multipliers](https://forum.effectivealtruism.org/posts/GzmJ2uiTx4gYhpcQK/effectiveness-is-a-conjunction-of-multipliers) than engaging with more rationality content. 

For example, rather than reading the Sequences, Alice may be helped more by...

*   Fostering deeper connections with friends and family 
*   Prioritizing her physical health
*   Lifting her sights higher and recognizing what might be stifling her ambitions
*    Addressing any lurking mental health issues
*   Diving into a side project she's excited by

Yes, perfect rationality would empower Alice far more than any of these gains. But that doesn't imply that Alice should dedicate herself to rationality. Increases in rationality are bounded, probably have diminishing returns, and Alice may just be one of those people who doesn't find that engaging with rationality content helps her that much. And that's fine.

As a community, I think we should be cautious of overemphasizing rationality engagement as the way to win. 

[^n9972zs1okk]: This depends on Alice's existing familiarity with rationality practices, or her trait inclination towards rationality.
# [Replication] Conjecture's Sparse Coding in Toy Models

Summary
-------

In the post [Taking features out of superposition with sparse autoencoders](https://www.lesswrong.com/posts/z6QQJbtpkEAX3Aojj/interim-research-report-taking-features-out-of-superposition), Lee Sharkey, Beren Millidge and Dan Braun (formerly at Conjecture) showed a potential technique for removing superposition from neurons using sparse coding. The original post shows the technique working on simulated data, but struggling on real models, while a recent update shows promise, at least for extremely small models.

We've now replicated the toy-model section of this post and are sharing the [code on github](https://github.com/HoagyC/sparse_coding) so that others can test and extend it, as the original code is proprietary. 

Additional replications have also been done by Trenton Bricken at Anthropic and most recently [Adam Shai](https://github.com/adamimos/transformerPlayground/blob/main/autoencoder_superposition.ipynb). 

Thanks to Lee Sharkey for answering questions and Pierre Peigne for some of the data-generating code.

Future Work
-----------

We hope to expand on this in the coming days/weeks by:

*   trying to replicate the results on small transformers as reported in the [recent update post](https://www.lesswrong.com/posts/DezghAd4bdxivEknM/a-small-update-to-the-sparse-coding-interim-research-report).
*   using automated interpretability to test whether the directions found by sparse coding are in fact more cleanly interpretable than, for example, the neuron basis.
*   trying to get this technique working on larger models.

If you're interested in working on similar things, let us know! We'll be working on these directions in the lead-up to SERI MATS.

Comparisons
-----------

### Mean Maximum Cosine Similarity (MMCS) with true features

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/5bd81d6ebf2b5ad6fdc20a0004572ae2f640db94a1102720.png)

Original Figure 3

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/fd412b7bb23d441ac8895109966e0741fada7112a7ee41a3.png)

Our replication

### Number of dead neurons (capped at 100)

![](http://res.cloudinary.com/lesswrong-2-0/image/upload/v1671187496/mirroredImages/z6QQJbtpkEAX3Aojj/gifiglaqkkdgrm0zpxsd.png)

Original Figure 4

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/5cc73254850cbbd4776159b0e6c0119e9b8c13d6be7b5d9c.png)

Our replication

### MMCS with larger dicts with same L1 penalty

![](http://res.cloudinary.com/lesswrong-2-0/image/upload/v1671187496/mirroredImages/z6QQJbtpkEAX3Aojj/vadjrcifzxmfuobn5aqc.png)

Original Figure 10

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/6c5140396f6df564f5260d277c6b5aa2d5f819c0fbf90ed0.png)

Our replication. If more than two larger dicts with the same l1 penalty exist, we take only the first two. Bottom row is void since no larger dicts are trained.
# How could AIs 'see' each other's source code?

I'm NOT confused about 'how could AIs send each other a link to their Git repo?'.

I'm confused as to what other info would convince an AI that any particular source code is what another AI is running.

How could any running system prove to another what source code it's running now?

How could a system prove to another what source code it will be running in the future?
# Upcoming AI regulations are likely to make for an unsafer world

*TL;DR: I am rather confident that involving governments in regulating AI development will make the world less safe, not more, because governments are basically incompetent and the representatives are beholden to special interests.*

Governments can do very few things well, some number of things passably, and a lot of things terribly. We tolerate it because the alternative is generally worse. Regulations are needed to keep people and companies from burning the commons, and to create more commons. The environment, infrastructure, public health, military, financial stability all require enforceable regulations. Even there the governments are doing pretty poorly (remember FDA, CDC and their siblings in other countries, as well as WHO?), but having a free-for-all would likely be worse. 

How the proponents of AI regulations expect it to go: the governments make it hard to hoard GPUs or to do unsanctioned AI research, or to use the whole internet to train the model without permission. Sort of like HIPAA in the US makes it harder to obtain someone's personal health data without their permission. 

How it is likely to go: Companies will be pretending that their all new more capable and more dangerous AI model is just a bit improved old model, to avoid costly re-certification (see the Boeing 737 MAX 8 debacle).
# Terry Tao is hosting an "AI to Assist Mathematical Reasoning" workshop

The workshop is on 12-14 June, following a [previous workshop](https://terrytao.wordpress.com/2022/10/21/upcoming-workshop-on-machine-assisted-proofs-at-ipam/) on "Machine-assisted proofs" held last Feb.

Maybe an opportunity to nerd snipe him and his high-powered colleagues on alignment? This event might mark a turning point in his thinking about it as well since Yann LeCun is part of the organising committee—which also entails much care lest a half-baked attempt [inoculates](https://slatestarcodex.com/2014/04/15/the-cowpox-of-doubt/) him from the entire field.

Anyone who knows him or is otherwise in UCLA who can weigh in on the feasibility of this?
# A Double-Feature on The Extropians

Link-post for two pieces I just wrote on the Extropians.  
  
The Extropians were an online group of techno-optimist transhumanist libertarians active in the 90s who influence a lot of online intellectual culture today. Prominent members include Eliezer Yudkowsky, Nick Bostrom, Robin Hanson, Eric Drexler, Marvin Minsky and all three of the likely candidates for Satoshi Nakamoto (Hal Finney, Wei Dai, and Nick Szabo).  
  
[The first piece is a deep dive into the archived Extropian forum](https://www.maximumprogress.org/extropia-archaeology). It was super fun to write and I was constantly surprised about how much of the modern discourse on AI and existential risk had already been covered in 1996.

[The second piece is a retrospective on predictions made by Extropians in 1995](https://maximumprogress.substack.com/p/grading-extropian-predictions). Eric Drexler, Nick Szabo and 5 other Extropians give their best estimates for when we'll have indefinite biological lifespans and reproducing asteroid eaters.
# [Link Post]  Bytes Are All You Need: Transformers Operating Directly On File Bytes 

>Modern deep learning approaches usually transform inputs into a modality-specific form. For example, the most common deep learning approach to image classification involves decoding image file bytes into an RGB tensor which is passed into a neural network. Instead, we investigate performing classification directly on file bytes, without the need for decoding files at inference time. Using file bytes as model inputs enables the development of models which can operate on multiple input modalities. Our model, \emph{ByteFormer}, achieves an ImageNet Top-1 classification accuracy of 77.33% when training and testing directly on TIFF file bytes using a transformer backbone with configuration similar to DeiT-Ti (72.2% accuracy when operating on RGB images). Without modifications or hyperparameter tuning, ByteFormer achieves 95.42% classification accuracy when operating on WAV files from the Speech Commands v2 dataset (compared to state-of-the-art accuracy of 98.7%). Additionally, we demonstrate that ByteFormer has applications in privacy-preserving inference. ByteFormer is capable of performing inference on particular obfuscated input representations with no loss of accuracy. We also demonstrate ByteFormer's ability to perform inference with a hypothetical privacy-preserving camera which avoids forming full images by consistently masking 90% of pixel channels, while still achieving 71.35% accuracy on ImageNet. 

https://arxiv.org/abs/2306.00238
# Wildfire of strategicness


It may not be feasible to make a mind that makes achievable many difficult goals in diverse domains, without the mind also itself having large and increasing effects on the world. That is, it may not be feasible to make a system that [strongly possibilizes without strongly actualizing](https://tsvibt.blogspot.com/2022/12/possibilizing-vs-actualizing.html).

But suppose that this is feasible, and there is a mind M that strongly possibilizes without strongly actualizing. What happens if some [mental elements](https://tsvibt.blogspot.com/2022/08/structure-creativity-and-novelty.html#1-elements-and-structure) of M start to act strategically, selecting, across any available domain, actions predicted to push the long-term future toward some specific outcome?

The growth of M is like a forest or prairie that accumulates dry grass and trees over time. At some point a spark ignites a wildfire that consumes all the accumulated matter. 

The spark of strategicness, if such a thing is possible, recruits the surrounding mental elements. Those surrounding mental elements, by hypothesis, make goals achievable. That means the wildfire can recruit these surrounding elements toward the wildfire's ultimate ends. By recruiting more dry matter to the wildfire, the wildfire burns hotter and spreads further.

Also by hypothesis, the surrounding mental elements don't themselves push strongly for goals. Seemingly, that implies that they do not resist the wildfire, since resisting would constitute a strong push. We can at least say that, if the totality of the mental elements surrounding the wildfire is going to notice and suppress the wildfire, it would have to think at least strategically enough to notice and close off all the sneaky ways by which the wildfire might wax. This implies that the surrounding mental elements do a lot of thinking and have a lot of understanding relevant to strategic takeovers, which itself seemingly makes more available the knowledge needed for strategic takeovers. 

Capabilities overhang provides fuel for a wildfire of strategicness. That implies that it's not so easy to avoid [wrapper-minds](https://www.lesswrong.com/posts/dKTh9Td3KaJ8QW6gw/why-assume-agis-will-optimize-for-fixed-goals). 

This is very far from being a watertight argument, and it would be nice if the conclusion were false; how is it false? Maybe, as is sometimes suggested, minds selected to be creative are selected to keep themselves [open to new ideas and therefore to new agency](https://tsvibt.blogspot.com/2022/10/does-novel-understanding-imply-novel.html), implying that they're selected to prevent strategic takeovers, which would abhor new agency? 



![](https://i.imgur.com/FmzStYQ.png)
# Andrew Ng wants to have a conversation about extinction risk from AI

Andrew Ng writes:

> I'd like to have a real conversation about whether AI is a risk for human extinction. Honestly, I don't get how AI poses this risk. What are your thoughts? And, who do you think has a thoughtful perspective on how AI poses this risk that I should talk to?

In the attached video, he states that he respects many of the people who signed the letter a lot, and will reach out to people whom he thinks have a thoughtful perspective. But he is also interested in **further suggestions for whom to talk to**. 

Given that Andrew Ng is one of the top AI scientists in the world, it seems valuable for someone to think of a way to connect to him.