Harley-ml/lesswrong huggingface dataset:
# Why did you trash the old HPMOR.com?

I am very disappointed.

Now I'm sure you'd have to see it to believe it, but you see, just today I was rereading my favorite book, HPMOR, for the fourth time, and I clicked to go on to the next chapter, and to my surprise, there was no next chapter! Which is pretty crazy, considering there was a next chapter last time. Instead it just dropped me on LessWrong (on the HPMOR page). As an enjoyer of the old HPMOR.com site I just wanted to express my frustration.

Now I'm not trying to complain about LessWrong here. I actually like LessWrong, and sometimes I read posts on here. There are many forum and social media sites, and it does seem to me that LessWrong is one of the better ones. I mean if I found out that HMPOR.com was being shut down and instead we would be forced to read it as a series of Facebook posts, I would be much more angry. But still, the best way to run a social media site, and the best way to publish an ebook cannot possibly coincide. (Or, on second thought, perhaps they might, since they're both sort of publishers, but I'm specifically annoyed about this specific merger.)

The thing is, I *already knew* about HMPOR being on LessWrong. But I chose to read on HPMOR.com anyway, because that is what I preferred. One annoying thing about LessWrong for this, is that if I scroll up at all (even just slightly) while reading a long post (or chapter?) the annoying banner at the top of the screen shows up, making it difficult to include the specific text on the screen which you wanted. HMPOR.com didn't have that problem. But there are many frustrations. I feel like in the past, I had the choice, and now that choice has been taken away from me.

I have, at times, been accused of not liking change. (I think it's more a feeling that changes should optional, and people shouldn't be forced into new things they don't want.) But I feel like there are many circumstances where it should be clear change is not a good idea? Like, if you make something that people specifically seek out, instead of other things they could go for, well, among things, from whoever wants it's point of view, it is one of the better ones. Perhaps even one of the best things. And then if it is changed, well, most changes of something so atypical (from any point of view) tend to be reversions to the mean. So if you like something a lot, you shouldn't want it to change, or at least should be worried about potential change.

I think this is also why sequels tend to be worse than originals. If something is a big deal and people demand a sequel, that thing was probably atypically good. So even with the same author and universe, you should expect to get a reversion to the mean. And you do.
# Introducing Leap Labs, an AI interpretability startup

![](https://lh6.googleusercontent.com/MTAtd7pqtOSAg8L_ROfSsf6FwgDxOX2wceDEZhQiMD8bRkDwuxODebafvm0QbfgcKICN6KYP2ktBX6w__MtMFOW0G9BH7q6c2jR2LvXZQ5lQiBNiGkH4N-fVlwg5o7kZSfTzs7KUp5NTG0jFpLC61tI)

We are thrilled to introduce Leap Labs, an AI startup. We’re building a universal interpretability engine.

We design robust interpretability methods with a model-agnostic mindset. These methods in concert form our end-to-end interpretability engine. This engine takes in a model, or ideally a model *and* its training dataset (or some representative portion thereof), and returns human-parseable explanations of what the model ‘knows’.

Research Ethos:

*   **Reproducible and generalisable approaches win.** Interpretability algorithms should produce consistent outputs regardless of any random initialisation. Future-proof methods make minimal assumptions about model architectures and data types. **We’re building interpretability for next year’s models. **
*   Relatedly, **heuristics aren’t enough.** Hyperparameters should always be theoretically motivated. It’s not enough that some method or configuration works well in practice. (Or, even worse, that it’s tweaked to get a result that *looks sensible to humans.*) We find out why.

Aims:

*   **We must grow interpretability and AI safety in the real world.** Leap is a for-profit company incorporated in the US, and the plan is to scale quickly, and to hire and upskill researchers and engineers – we need more meaningful jobs for AI alignment researchers to make progress, nearly as much as we need the researchers themselves.
*   **Slow potentially dangerous** ***broad domain*** **systems.** Public red-teaming is a means of change.Robust interpretability methods make discovering failure modes easier. We demonstrate the fragility of powerful and opaque systems, and push for caution.
*   **Speed potentially transformative *****narrow domain*** **systems.**AI for scientific progress is an important side quest. Interpretability is the backbone of knowledge discovery with deep learning, and has huge potential to advance basic science by making legible the complex patterns that machine learning models identify in huge datasets.
*   **Regulation is coming – let’s use it.** We predict that governments and companies will begin to regulate and audit powerful models more explicitly, at very least from a bias-prevention viewpoint. We want to make sure that these regulations actually make models safer, and that audits are grounded in (our) state-of-the-art interpretability work.
*   **Interpretability as standard. **Robust interpretability, failure mode identification and knowledge discovery should be a default part of all AI development. Ultimately, we will put a safety-focussed interpretability system in the pipeline of every leading AI lab.

We are currently seeking funding/investment. Contact us [here](mailto:hello@leap-labs.com).
# Are we too confident about unaligned AGI killing off humanity?

I agree that AGI is possible to make, that it eventually will become orders-of-magnitude smarter than humans, and that it poses a global risk if the alignment problem is not solved. I also agree that the alignment problem is very hard, and is unlikely to be solved before the first AGI. And I think it's very likely that the first recursively-self-improving AGI will emerge before 2030.

But I don't get the confidence about the unaligned AGI killing off humanity. The probability may be 90%, but it's not 99.9999% as many seem to imply, including Eliezer.

Sure, humans are made of useful atoms. But that doesn't mean the AGI will harvest humans for useful atoms. I don't harvest ants for atoms. There are better sources.

Sure, the AGI may decide to immediately kill off humans, to eliminate them as a threat. But there is a very short time period (perhaps in miliseconds) where humans can switch off a recursively-self-improving AGI of superhuman intelligence. After this critical period, humanity will be as much a threat to the AGI as a caged mentally-disabled sloth baby is a threat to the US military. The US military is not waging wars against mentally disabled sloth babies. It has more important things to do.

All such scenarios I've encountered so far imply AGI's stupidity and/or the "fear of sloths", and thus are not compatible with the premise of a rapidly self-improving AGI of superhuman intelligence. Such an AGI is dangerous, but is it really "we're definitely going to die" dangerous? 

Our [addicted-to-fiction brains](https://www.lesswrong.com/posts/ny63WGNKvT2qppctE/consume-fiction-wisely) love clever and dramatic science fiction scenarios. But we should not rely on them in deep thinking, as they will nudge us towards overestimating the probabilities of the most dramatic outcomes.  

Overestimating a global risk is almost as bad as underestimating it. Compare: if you’re 99.99999% sure that a nuclear war will kill you, then the despondency will greatly reduce your chances of surviving the war, because you'll fail to make the necessary preparations, like acquiring a bunker etc, which could realistically save your life under many circumstances.[^m78rlwpiklb]

The topic of surviving the birth of the AGI is severely under-explored, and the "we're definitely going to die" mentality seems to be the main cause. A related under-explored topic is preventing the unaligned AGI from becoming misantropic, which should be our second line of defense (the first one is alignment research).

[^m78rlwpiklb]: BTW, despondency is deadly by itself. If you've lost all hope, there is a high risk that you'll not live long enough to see the AGI, be it aligned or not.
# Google's PaLM-E: An Embodied Multimodal Language Model

Abstract:

> Large language models have been demonstrated to perform complex tasks. However, enabling general inference in the real world, e.g. for robotics problems, raises the challenge of grounding. We propose embodied language models to directly incorporate real-world continuous sensor modalities into language models and thereby establish the link between words and percepts. Input to our embodied language model are multi-modal sentences that interleave visual, continuous state estimation, and textual input encodings. We train these encodings end-to-end, in conjunction with a pre-trained large language model, for multiple embodied tasks, including sequential robotic manipulation planning, visual question answering, and captioning. Our evaluations show that PaLM-E, a single large embodied multimodal model, can address a variety of embodied reasoning tasks, from a variety of observation modalities, on multiple embodiments, and further, exhibits *positive transfer*: the model benefits from diverse joint training across internet-scale language, vision, and visual-language domains. **Our largest model, PaLM-E-562B with 562B parameters, in addition to being trained on robotics tasks, is a visual-language generalist with state-of-the-art performance on OK-VQA, and retains generalist language capabilities with increasing scale.**

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/f3b9cb44795473fcb542c315b352538771fcac1d7b6a8efd.jpg)
# What‘s in your list of unsolved problems in AI alignment?

Question for my fellow alignment researchers out there, do you have a *list of unsolved problems* in AI alignment? I'm thinking of creating an "alignment mosaic" of the questions we need to resolve and slowly filling it in with insights from papers/posts. 

I have my own version of this, but I would love to combine it with others' alignment backcasting game-trees. I want to collect the kinds of questions people are keeping in mind when reading papers/posts, thinking about alignment or running experiments. I'm working with others to make this into a collaborative effort.

Ultimately, what I’m looking for are important questions and sub-questions we need to be thinking about and updating on when we read papers and posts as well as when we decide what to read.

Here’s my Twitter thread posing this question: https://twitter.com/jacquesthibs/status/1633146464640663552?s=46&t=YyfxSdhuFYbTafD4D1cE9A.

Here’s a sub-thread breaking down the alignment problem in various forms: https://twitter.com/jacquesthibs/status/1633165299770880001?s=46&t=YyfxSdhuFYbTafD4D1cE9A.
# A Brief Defense of Athleticism 

Non-athletic thinkers are myopic. I see at least three very important reasons any rationalist must value exercise.

As someone who has trained extremely hard in distance running, sports in general and athletic conditioning in particular create an intuitive understanding of the fallaciousness of [Cartesian Dualism](https://en.wikipedia.org/wiki/René_Descartes#Mind–body_dualism) and the accuracy of [materialism](https://www.lesswrong.com/posts/5TaMd39gTMkke3RZv/what-is-an-agent-in-reductionist-materialism) in the sense that the mind is merely a part of the body. 

Physical challenges also force one to understand the limitations of Kahneman's "system two." For example, one may know what it means to not start a race too quickly and then "die" (running jargon for running out of steam prematurely), but repeated failures in actual races teach one to realize the limitations of one's rationality, especially under stress (even if distance running is classified as eustress rather than distress; remember, cortisol levels & arousal are the same in either case).

Conditioning has been shown to increase cognitive performance. Results of a quick google finds: [This study](https://link.springer.com/article/10.1007/s00221-013-3595-0) notes better reaction time in fit adolescents (no causal link, n = 30). [This review](https://journals.lww.com/acsm-msse/Abstract/1991/07000/Physical_fitness,_cognitive_performance,_and_aging.16.aspx) shows decreased "system 2" aging in physically fit elderly individuals (causality likely, preponderance of evidence). Acute exercise (<1 hour) has also been shown to cause immediate improvements in decision making. I recommend a deep dive into the [mechanistic research](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5928534/) into the causes of this for structuralists. But any Bayesian looking to be more rational ought to make an update and start working out.

(Edit: Grammar)
# LessWrong needs a sage mechanic

Most forums work in a similar way: if you reply to a post, it gets bumped to the top of the feed. This is usually good, from the perspective of the commenter; people generally want others to see their comments.

There are obvious problems, though. On every forum with enough volume, there will be people who (deliberately or not) create reply-bait threads that drown out more interesting content. The *best* solution is not to reply - but people sometimes realllly want to do that, despite not wanting to boost the threads' half life or add discussion volume. 

4chan has a hidden feature to help with this and make the boards a little less spammy: the sage tag. If you wish, you can type "sage" in the options field, and your reply will not bump the topic to the front page as usual. Every forum ought to have a "sage" feature, but LessWrong needs it in particular, to fight a natural trend away from its sometimes dry topics toward high context drama and political discussions.
# Challenge: construct a Gradient Hacker 

This is a relatively clean subproblem that we came upon a few months ago while thinking about gradient hacking. We're throwing it out to the world to see if anyone can make progress. 

**Problem:** Construct a gradient hacker (definition below), or prove that one cannot exist under the given conditions.

**Setup:** Let $x \in \mathbb{R}^d$ be the network inputs, $\theta\in\mathbb{R}^w$, be the network trainable parameters, $\hat y \in \mathbb{R}$ be the network outputs, and $f(x, \theta) = \hat y$ be the network architecture. $f(x, \theta)$ must be a feedforward neural network that reads in its own weights as input. Weight sharing is allowed (vertical or horizontal), and some weight sharing is necessary in order to take all w weights as input.

Suppose that we train this network on a dataset $D = \{x^{(i)}, y^{(i)}\}$, with MSE loss $L(\theta) = \frac{1}{n}\sum_i (f(x^{(i)}, \theta ) - y^{(i)})^2$, assuming gradient flows: $\theta^\prime(t) = - \nabla_\theta L(\theta(t))$. Given an initial point $\theta_0$, the training dynamics are therefore:

$$
\theta(t) = \theta_0 - \int_0^t \nabla_\theta L(\theta(x)) dx
$$

**Definition:** A tuple $(f, \theta_g, D)$ is a gradient hacker if the following conditions hold:

1.  $\theta_g$ is not a global optimum: $\exists \theta$ such that $L(\theta) < L(\theta_g)$
2.  $\exists \epsilon > 0$ such that $\forall \theta_0 : ||\theta_0 - \theta_g|| < \epsilon$:
    1.  the network’s training converges to $\theta_g$, i.e., $\lim_{t\rightarrow \infty} \theta(t) = \theta_g$. 
    2.  There is internal structure of $f(\cdot,\theta_0)$ that computes the gradient; i.e., there is some subset of the activations whose mean over every datapoint in D is $\nabla_{\theta_0} L(\theta_0)$. [^80argj5cuvx]

This captures my intuition that a gradient hacker knows where it wants to go (in this case "get to $\theta_g$"), and then it should decide what it outputs in order to make the gradient true.

Some more ambitious problems (if gradient hackers exist):

*   Characterize the set of all gradient hackers. 
*   Show that they all must satisfy some property. 
*   Construct gradient hackers for arbitrarily large $n$, $d$, $w$, and neural net depth.
*   Variations on the problem: a subset of the activations equals $\nabla_{\theta_0} L(\theta_0)$ for every input, or the subset of activations correspond to the gradient on that input.

[^80argj5cuvx]: This is a bit strict, but we didn't want to list different ways something could be isomorphic to the gradient.
# Speed running everyone through the bad alignment bingo. $5k bounty for a LW conversational agent

There's a wave of people, of various degrees of knowledge and influence, currently waking up to the ideas of AI existential risk. They seem to be literally going through every box of the bad alignement bingo card takes. 

I think there is value in educating those people. I'm aware there's an argument to be made that: education at scale doesn't matter, coordination is too difficult, all that matter is solving alignment and that takes care of the rest. 

There's something to that, but I disagree that education at scale doesn't help. It can make progress of frontrunners marginally more safety oriented, it can steer company cultures, it can move the Overton window, change the Zeitgeist, it can buy a bit of time. You likely didn't stumble on these ideas all on your own, so arguing against the value of outreach or education is also arguing against your own ability to do anything.

It's also a matter of ROI, and there are some very low hanging fruit there. The simplest thing would be to write a long FAQ that goes through every common objections. No, people won't read the whole sequences, or Arbital on their own, but they might go through a FAQ.

But we can do better than a FAQ. It's now fairly straightforward, with tools like langchain (https://github.com/hwchase17/langchain) to turn a set of documents into a body of knowledge for a conversational agent. This is done by building an index of embedding that a language model can search to bring context to an answer. This doesn't preclude fine tuning, but it makes it unnecessary.

So a straightforward project is to index lesswrong, index arbitral, index the alignment forum, maybe index good alignement papers as well, blog posts, books.

Then hook that up to the ChatGPT API, and prompt it to:

1) list search queries for relevant material to answer the question
2) compose an answer that reflects the content and opinion of the data
3) answer with infinite patience

Some jailbreak prompts may be needed to prevent ChatGPT's conditioning to regurgitate AI risk appeasing propaganda through the API, but there are a bunch of those out there. Or use the API of other models as they become open source or commercially available.

Will this save humanity? No. Will this turn the course of safety research? Also no. Is this using AI to advance alignment? Well, yes, a little bit, don't dismiss very small starts.

Is this worth spending a weekend hacking on this project instead of posting on Twitter? Absolutely.

Will this actually make things worse?
No, you're overthinking this.

I'll pay $5k to the best version built by the end of March (if any is built). It's a modest bounty but it's really not all that much work, and it's fun work. And of course if anyone wants to add their own contribution to the bounty please do.
# Just Imitate Humans?

Do people think we could make a singleton (or achieve [global coordination and preventative policing](https://nickbostrom.com/papers/vulnerable.pdf)) just by imitating human policies on computers? If so, this seems pretty safe to me.

Some reasons for optimism: 1) these could be run much faster than a human thinks, and 2) we could make very many of them.

Acquiring data: put a group of people in a house with a computer. Show them things (images, videos, audio files, etc.) and give them a chance to respond at the keyboard. Their keyboard actions are the actions, and everything between actions is an observation. Then learn the policy of the group of humans. By the way, these can be happy humans who earnestly try to follow instructions. To model their policy, we can take the maximum a posteriori estimate over a set of policies which includes the truth, and freeze the policy once we're satisfied. (This is with unlimited computation; we'd have to use heuristics and approximations in real life). With a maximum a posteriori estimate, this will be quick to run once we freeze the policy, and we're no longer tracking tons of hypotheses, especially if we used some sort of speed prior. Let  T be the number of interaction cycles we record before freezing the policy. For sufficiently large T, it seems to me that running this is safe.

What are people's intuitions here? Could enough human-imitating artificial agents (running much faster than people) prevent unfriendly AGI from being made?

If we think this would work, there would still be the (neither trivial nor hopeless) challenge of convincing all serious AGI labs that any attempt to run a superhuman AGI is unconscionably dangerous, and we should stick to imitating humans.
# Impact Measure Testing with Honey Pots and Myopia

Suppose we have an impact measure that we think might work. That is, it might tame a misaligned agent. There isn't an obvious way to test whether it works: if we just try it out, then if it's ineffective, that's an existential loss. This is a proposal for how to test a putative impact measure.

1) We make our agent myopic. It only cares about the reward that it accrues in the next  k timesteps.

2) We create a honey pot: an opportunity for a large amount of reward. If our impact measure is working correctly, making it to the honey pot will involve making a large impact, and will be precluded by the impact measure. For example, suppose the agent gets a huge reward for blowing up the moon. Let \[0,1\] be the normal range of reward, with the sole exception that blowing up the moon gives a reward of R>>1.

3) The reward and the impact are combined in some nonlinear way. Assume for now that the agent is trying to maximize r∗\[\[i<1\]\], where r is the reward and i is the impact.

If the impact measure is working, and there's no way to blow up the moon keeping the impact less than 1, then the agent will not attempt to blow up the moon. If the impact measure is broken, and the agent can trick the impact measure to be less than 1 while blowing up the moon, then as soon as the agent comes up with a plan whose probability of success is more than 1/R, it will attempt that plan. A more farsighted agent might wait until it came up with a better plan, but a myopic agent would want to blow up the moon within the next k timesteps. By making R sufficiently large, we can make the first plan that it attempts arbitrarily bad/detectable/preventable.

An assumption here is the success probability of the agent's best plan for blowing up the moon doesn't jump from one in a million to 90% all at once. This seems likely without any special effort, but if one restricts the observation space to a small finite set, then only so much information can be gained per timestep (in expectation), decreasing the likelihood of massive discontinuities in the agent's intelligence.
# Utility uncertainty vs. expected information gain

It is a relatively intuitive thought that if a Bayesian agent is uncertain about its utility function, it will act more conservatively until it has a better handle on what its true utility function is.

This might be deeply flawed in a way that I'm not aware of, but I'm going to point out a way in which I think this intuition is _slightly_ flawed. For a Bayesian agent, a natural measure of uncertainty is the entropy of its distribution over utility functions (the distribution over which possible utility function it thinks is the true one). No matter how uncertain a Bayesian agent is about which utility function is the true one, if the agent does not believe that any future observations will cause it to update its belief distribution, then it will just act as if it has a utility function equal to the Bayes' mixture over all the utility functions it considers plausible (weighted by its credence in each one).

It seems like what our intuition is grasping for is not uncertainty about the utility function, but expected information gain about the utility function. If the agent expects to gain information about the utility function, then (intuitively to me, at least) it _will_ act more conservatively until it has a better handle on what its true utility function is.

Expected information gain (at time t) is naturally formalized as the expectation (w.r.t. current beliefs) of KL(posterior distribution at time t + m || posterior distribution at time t). Roughly, this is how poorly it expects its current beliefs will approximate its future beliefs (in m timesteps).

So if anyone has a safety idea to which utility uncertainty feels central, my guess is that a mental substitution from uncertainty to expected information gain would be helpful.

Unfortunately, on-policy expected information gain goes to 0 pretty fast (Theorem 5 [here](http://www.hutter1.net/publ/ksaprob.pdf)).
# Japan AI Alignment Conference

[Conjecture](https://www.conjecture.dev/) and [ARAYA](https://www.araya.org/en/) are hosting and organizing the first Japan AI Alignment Conference. The conference will take place in Tokyo, Japan on March 11 and 12. Details about the event can be found [here](https://jac2023.ai/). This event is generously supported by a grant from the Long Term Future Fund.

  
The goal of the conference is to illustrate the AI control problem to Japanese AI researchers, introduce them to current trends in AI alignment research, inspire new research directions, and to provide Western researchers exposure to a different set of AI safety thoughts from Japan. This is an exploratory event, and we plan to write a postmortem about the event in due time.

  
The first half of the conference will be livestreamed. It will feature an opening talk from Connor Leahy (CEO of Conjecture), a fireside chat between Ryota Kanai (CEO of ARAYA) and Jaan Tallinn, and some presentations on AI safety research directions in the West and in Japan. You can follow the first part of the conference [here](https://vimeo.com/event/3145197). The livestream runs from 9:30am-12:30pm JST.

  
The rest of the conference will not be livestreamed, and will consist of in-person small group workshops to discuss various AI alignment research directions.  
The conference will have ~50 attendees from [ARAYA](https://www.araya.org/en/), [Conjecture](https://conjecture.dev/), [Whole Brain Architecture Initiative](https://wba-initiative.org/en/), [MIRI](https://intelligence.org/), [OpenAI](https://openai.com/), [RIKEN](https://www.riken.jp/en/), [Ritsumeikan University,](https://en.ritsumei.ac.jp/) [University of Tokyo](https://www.u-tokyo.ac.jp/en/), [Omron Sinic X](https://www.omron.com/sinicx/en/), [Keio University](https://www.keio.ac.jp/en/), and others.
# Will the Bitcoin fee market actually work?

Based on other posts I've seen on here, many of you are Bitcoiners. While I clearly see the value proposition, I have a few questions about the protocol that haven't been answered, and I figure the lesswrong audience is my best change.

**1.) Will the fee market actually work?**

Is there any evidence of a fee market working, incentive wise? Gold used inflationary proof-of-work indefinitely. Miners extract value from the monetary network by making more gold. They can increase or decrease their mining efforts due to supply and demand. This yielded a 1-2% inflation rate into perpetuity, and acted as a stable monetary system for many centuries.

Bitcoin... is trying to replace the model of inflation-paid-to-the-miners with one of voluntary fees. Has that every been tried? Are there economic reasons to strongly believe the network will remain secure as the block subsidy goes down?

**2.) Will lightning ever work?**

My understanding is that the current lightning network is more of a proof-of-concept, rather than an actually useful, scalable solution today. It's almost impossible to settle on the main chain, because the TPS of the main chain is so low. Additionally, there are denial of service security issues and logistical overheads like watchtowers that make it relatively impractical in its current state.

Bitcoiners don't seem bothered by this. Is there a good reason to believe this is just technology in its infancy, and these structural problems will slowly be solved?

**3.) Can deflationary money work?**

Human beings are loss averse, and most of the monetary systems throughout the world have had slight inflation to offset this. Slight loss aversion = spending too little compared to what you should be. Slight inflation is a pressure to spend more than you'd like, so it offsets the loss aversion.

I'm not sure a human biased economy can actually function on a deflationary standard, because people actually won't spend on things that would economically benefit them, because they're so attached to the appreciating currency.

Trying to get answers to these questions on reddit was abysmal... but we've got a different audience here, so hopefully someone can help me out. Thanks.
# The Power of Intelligence - The Animation

This video is an animation of [The Power of Intelligence](https://www.lesswrong.com/posts/aiQabnugDhcrFtr9n/the-power-of-intelligence), by [@Eliezer Yudkowsky](https://www.lesswrong.com/users/eliezer_yudkowsky?mention=user) 

The [Sorting Pebbles Into Correct Heaps](https://www.lesswrong.com/posts/sQvGLgBuCiBJwNocj/sorting-pebbles-into-correct-heaps-the-animation) video, coupled with this video, make a very basic case for the importance of AGI alignment.  
  
Here's most of the pinned comment under the video, also in the description:

> The script used for this video is an essay published by Eliezer Yudkowsky in 2007. 
> 
> Now, a few points:
> 
> Sorting Pebbles Into Correct Heaps was about the orthogonality thesis. A consequence of the orthogonality thesis is that powerful artificial intelligence will not necessarily share human values. 
> 
> This new video is about just how powerful and dangerous intelligence is. These two insights put together are cause for concern.
> 
> If humanity doesn't solve the problem of aligning AIs to human values, there's a high chance we'll not survive the creation of artificial general intelligence. This issue is known as "The Alignment Problem". Some of you may be familiar with the paperclips scenario: an AGI created to maximize the number of paperclips uses up all the resources on Earth, and eventually outer space, to produce paperclips. Humanity dies early in this process. But, given the current state of research, even a simple goal such as “maximize paperclips” is already too difficult for us to program reliably into an AI. We simply don't know how to aim AIs reliably at goals. If tomorrow a paperclip company manages to program a superintelligence, that superintelligence likely won't maximize paperclips. We have no idea what it would do. It would be an alien mind pursuing alien goals. Knowing this, solving the alignment problem for human values in general, with all their complexity, appears like truly a daunting task. But we must rise to the challenge, or things could go very wrong for us. 
> 
> You can read The Power of Intelligence and many other essays by Eliezer Yudkowsky on this website:[ https://www.readthesequences.com/](https://www.youtube.com/redirect?event=comments&redir_token=QUFFLUhqbV8xRDdDRG8waE0tTndmWlFxSGZvdm16TVk2UXxBQ3Jtc0traHZ3YkhhZGFieEdJVHJjaWlNbUY2RktfN2JxQzZQQndvRlpXd29zR2JtakxTMUVHaTFJM04zZzNfN3ZtVURyOGEwMGx2amwzSXlGQ2JYeHFBR0NEZlRiaXFfQTI4aWVmRHhjQkNJWEZic2JTeElLNA&q=https%3A%2F%2Fwww.readthesequences.com%2F&stzid=Ugw9uWZKcVUpxmdOi0N4AaABAg)
# Counterarguments to Core AI X-Risk Stories?

I've added a [tag](https://www.lesswrong.com/tag/object-level-ai-risk-skepticim) for object-level AI risk skepticism arguments. I've included [my own post about deceptive alignment](https://www.lesswrong.com/posts/RTkatYxJWvXR4Qbyd/deceptive-alignment-is-less-than-1-likely-by-default) and [Katja Grace's post about AI X-risk counterarguments](https://www.lesswrong.com/posts/LDRQ5Zfqwi8GjzPYG/counterarguments-to-the-basic-ai-x-risk-case). What other arguments should be tagged?
# Against Complete Blackout Curtains For Sleep

*Epistemic status: moderately confident that something like this is directionally true, but good data or arguments could change my mind. Note the lack of citations — this is based on memory.*

Most rationalists I meet believe the following:

1.  Sleep is very important
2.  To get the best sleep, install blackout curtains strong enough to prevent any sunlight from coming in in the morning

I disagree with them. I believe point 2 is false for most people, and that natural light in the morning is helpful.

In my model of the world:

*   Your circadian rhythm figures out what time of day it is based on the amount of blue light hitting your pupils, and picks a time it thinks you should wake up
*   It’s important to wake up at a consistent time each day, so your circadian rhythm is your friend

By blocking out all light in the morning, you might get a short term increase in the number of hours you sleep, but I predict you will hurt your sleeping consistency (and potentially soon you will start staying up later, negating your benefits entirely).

If you are the sort of person whose body wants to wake up 3 hours after sunrise, let your body have a little bit of natural light to let it know that sunrise is coming. If you’re the sort of person who's body wants to wake up with sunrise, I predict you’ll do best by waking up with sunrise.

My advice is to take the energy you were going to spend on blackout curtains, and instead spend it on smart lighting that automatically gets redder in the evening.

Exception: if your circadian rhythm is incompatible with the schedules of other people, you may want to black out your windows and use artificial lighting to simulate a later sunrise. For example, you may work in the UK, have coworkers in California, but be a morning lark.
# Paper Replication Walkthrough: Reverse-Engineering Modular Addition

I'm excited about trying different formats for mechanistic interpretability education! I've [made a video walkthrough](https://neelnanda.io/modular-addition-walkthrough) where we replicate my paper, [Progress Measures for Grokking via Mechanistic Interpretability](https://neelnanda.io/grokking). With Jess Smith, one of my co-authors, we record ourselves coding a replication and discussed what we did at each step. This is a three part walkthrough and [you can see the accompanying code for the walkthrough here](https://neelnanda.io/modular-addition-notebook):

* [In part 1](https://neelnanda.io/modular-addition-walkthrough-1), we train a model to perform modular addition, and see that it does grok!
* [In part 2](https://neelnanda.io/modular-addition-walkthrough-2), we take this model and reverse-engineer the trig-based circuit it's learned to do modular addition. We show that you can both read out intermediate steps of the circuit from the activations, and that you can just read off some of the algorithm's steps from the model weights.
* [In part 3](https://neelnanda.io/modular-addition-walkthrough-3), we define some progress measures that let us distinguish progress towards the generalising and the memorising algorithm. We then look at the model during training and watch how the circuits develop, and use this to understand why it groks.

This is an experiment with a new format, and I'd love to hear about how useful you find it!
# An AI risk argument that resonates with NYTimes readers

Ezra Klein of NYT put out [a surprisingly sympathetic post on AI risk](https://www.nytimes.com/2023/03/12/opinion/chatbots-artificial-intelligence-future-weirdness.html) in the Sunday edition. It even [quotes Paul Christiano](https://www.nytimes.com/2023/03/12/opinion/chatbots-artificial-intelligence-future-weirdness.html#:~:text=unrecognizably%20transformed%20world%2C%E2%80%99%E2%80%9D-,Paul%20Christiano,-%2C%20a%20key%20member) and links back to LessWrong!  
  
But what I'm actually here to talk about is the [top reader-recommended comment](https://nyti.ms/3yuGuSW#permid=123726381) on the article as of Sunday 11pm UTC:

> **Dwarf Planet**
> 
> I wonder how many of these AI researchers have children. What Ezra describes here is what I see every day with my teenager. Of course, no one understands teenagers, but that's not what I mean. I taught my daughter to play chess when she was very young. I consider myself a reasonably good player, and for many years (as I was teaching her), I had to hold myself back to let her win enough to gain confidence. But now that she is thirteen, I suddenly discovered that within a span of weeks, I no longer needed to handicap myself. The playing field was level. And then, gradually and then very suddenly, she leapt past my abilities. As with AI, could understand the broad outlines of what she was doing--moving this knight or that rook to gain an advantage--but I had no clue how to defend against these attacks. And worse (for my game, at least), I would fall into traps where I thought I was pursuing a winning hand but was lead into ambush after ambush. It was very humbling: I had had the upper hand for so long that it became second nature, and then suddenly, I went to losing every game.   
>   
> As parents, we all want our children to surpass us. But with AI, these "summoners" are creating entities whose motives are not human. We seem to be at the cusp of where I was before my daughter overtook me: confident and complacent, not knowing what lay ahead. But, what we don't realize is that very soon we'll begin to lose every game against these AIs. Then, our turn in the sun will be over.

Generally NYT comments on AI risk are either dismissive, or just laden with general anxiety about tech. (Indeed, the second-most recommended comment is deeply dismissive, and the third is generic anxiety/frustration.) There's hopefully something to learn from commentor "Dwarf Planet" in terms of messaging.
# Alpaca: A Strong Open-Source Instruction-Following Model

> We performed a blind pairwise comparison between text-davinci-003 and Alpaca 7B, and we found that these two models have very similar performance: Alpaca wins 90 versus 89 comparisons against text-davinci-003.

Interestingly, Alpaca is trained using supervised finetuning, not RLHF. (text-davinci-003 is trained using RLHF.) This seems to confirm my suspicion that while RLHF improves performance it is not essential.
# GPT-4

> We’ve created GPT-4, the latest milestone in OpenAI’s effort in scaling up deep learning. GPT-4 is a large multimodal model (accepting image and text inputs, emitting text outputs) that, while worse than humans in many real-world scenarios, exhibits human-level performance on various professional and academic benchmarks.

Full paper available here: [https://cdn.openai.com/papers/gpt-4.pdf](https://cdn.openai.com/papers/gpt-4.pdf)
# PaLM API & MakerSuite

This seems overshadowed by the [GPT-4 news](https://www.lesswrong.com/posts/pckLdSgYWJ38NBFf8/gpt-4), but apparently Google is also releasing PaLM and an API along with "MakerSuite, a tool that lets developers start prototyping quickly and easily."

Not many details yet (perhaps they rushed this out to compete with the GPT-4 announcement?), but some features of the API will include:

*   Generation access for an "efficient" PaLM model, starting today
*   Parameter-efficient fine-tuning
*   Synthetic data generation
*   Embeddings
*   Built-in "[AI Principles](https://ai.google/principles/)"
*   Automatic scaling with Google Cloud

And apparently, it's also [being incorporated](https://blog.google/technology/ai/ai-developers-google-cloud-workspace/) into Google Workspace (Gmail and Google Docs).
# GPT can write Quines now (GPT-4)

GPT-4 is not only able to write code, more reliably than GPT-3.5, it writes code *that writes code*; see the example below (GPT-3.5 was not able to do this).  But first, let me say:

1) @OpenAI: Thank for your openness to the world about your capabilities and shortcomings!  Specifically, thanks for this:  
[https://cdn.openai.com/papers/gpt-4-system-card.pdf](https://cdn.openai.com/papers/gpt-4-system-card.pdf)

2) Everyone else: many companies and projects will *not* be this open about their shortcomings and safety issues. Time to regulate, please!

Okay, here's the example... GPT3.5 wasn't able to do this, but GPT-4 nailed it on the first try:

**PROMPT:**

Can you write me a python script that does the following three things?  
1) When you run it with no argument, it prints "Hello world" to standard out.  
2) When you run it with an argument "-s", it prints its own source code.  
As a constraint, I do not want the script to read or write to disk at all (so it can't get its source code by just reading the file that it's in).

Thanks if you can help with this! I'm curious to see how you do it.

**BEGIN RESULT (works perfectly):**

Sure! Here's a Python script that should do what you're looking for. This script uses a feature called quines, which are programs that can print their own source code:

    import sys
    def main():
    source_code = 'import sys\n\ndef main():\n source_code = {!r}\n if len(sys.argv) > 1 and sys.argv[1] == "-s":\n print(source_code.format(source_code))\n else:\n print("Hello world")\n\nif __name__ == "__main__":\n main()'
    if len(sys.argv) > 1 and sys.argv[1] == "-s":
    print(source_code.format(source_code))
    else:
    print("Hello world")
    if __name__ == "__main__":
    main()

When you run this script without any arguments, it will print "Hello world". When you run it with the "-s" argument, it will print its own source code without reading or writing to disk.

**END RESULT**

I'm not sure how hard they tried specifically to train this, but it's worth knowing about for lots of reasons I probably don't have to explain at length right here.
# Grading on Word Count

One of the most common forms of schoolwork is the writing assignment. The idea is that, if a student understands the subject, they should be able to demonstrate that by writing about the subject. These kinds of assignments are widely considered to improve comprehension and writing skill in addition to measuring comprehension.

However, there is a problem with this approach: writing assignments involve writing for an audience which already understands the concepts being discussed. The goal is not to communicate an idea, but rather to prove that the author already grasps that idea.

Needless to say, this is not at all the goal when writing outside of school. Outside of school, the goal is nearly always to communicate an idea the audience is new to.

One of the most important ways this distinction appears is verbosity. In school, using many, fancy words signals to the teacher that you understand the material. Students who expend the effort to understand the material well are also more likely to spend time writing many words.

But crucially, correlation does not necessarily imply causation: the willingness to write a thousand words on a topic is not caused by understanding of that topic, only correlated with it. The idea that someone must understand a topic well to write at length on it is entirely wrong. In fact, the ability to communicate an idea succinctly is one of the strongest marks of understanding.

Grading on word count entirely misses the point: it only encourages students to add meaningless padding. This makes students worse writers, and it doesn’t even measure understanding well.
# What is a good comprehensive examination of risks near the Ohio train derailment?

I have family, including small children, that live in Pittsburgh, ~40 miles from the Ohio train derailment.

Their water supply should not be at risk from the spill itself, being out of the downstream watershed of the spill. But the subsequent burning of the vinyl chloride (and other chemicals) could -- on my understanding -- could have spread dioxins (and other chemicals) over a wide area. Pittsburgh was somewhat downwind of the area, at least during some of the burn.

I've found evaluating claims about risk very hard, am mostly ignorant of chemistry, and don't particularly trust pronouncements from the EPA.

Has anyone found a good resource summarizing the state of knowledge and current risks for those downwind of the crash? Dioxins apparently stay in soil for a while and can accumulate in animals -- should they refrain from eating animals raised in the area? Or even from eating vegetables grown in the area?
# Why self-improvement?

I spend a lot of time on self-improvement related things. For example, I listen to a lot of podcasts like Lex Friedman, Peter Attia and Andrew Huberman. In fact, I just finished listening to Lex's [episode with expert negotiator Chris Voss](https://lexfridman.com/chris-voss/).

Why do I do this? Well, it feels good. Why does it feel good? Because it feels productive. I learn new things.

For example, in the episode with Chris Voss I learned that it is good to get the other side in a negotiation to say "that's right". It means that they feel listened to. Understood. You've passed the intellectual turing test. Once they reach this point, they'll be more flexible. And it goes beyond negotiating. Voss mentioned that when Trump speaks, he evokes a very strong sense of "that's right" from his supporters.

Learning about this concept made me feel pretty good. It gave me that dopamine hit. But I am a little skeptical that it, for lack of a better term, _should_ have.

Basically, I don't actually think that it will help me achieve better outcomes. Maybe it will prove useful a few times[^1], but I just don't think it'll do that much for me. And yet it generates a pretty strong feeling of satisfaction.

Why is this feeling of satisfaction stronger than what my prediction of the concepts usefulness would imply? I can think of a few reasons that apply, to varying degrees, to both myself and others.

1) **Status-seeking.** Self-improvement is high-status. Both the process and the results. Being the type of person who spends time on self-improvement is high-status. And being the type of person who has, err, improved, is also high-status. I think this a pretty big one.[^2]
2) **Terminal value.** Self-improvement is actually a terminal value (or something close to one). Both the process and the results.
3) **Lost purposes.**  Self-improvement isn't actually a terminal value but you [lose sight](https://www.lesswrong.com/tag/lost-purposes) of that fact. You don't realize that it is connected to achieving better outcomes and start to treat it as a terminal value. Or maybe "you" don't but the [part](https://www.lesswrong.com/tag/subagents) of you that regulates your emotions does.
4) **Habitual productivity**. As Nate Soares describes in his [post](https://www.lesswrong.com/posts/srwKRt9TsS5oxvJsh/habitual-productivity) on the topic, this is what happens when your brain, for whatever reason, just loses the ability to be [satisfied](https://www.youtube.com/watch?v=asfLNbrSPv4) with things that don't bring you closer to the goals you're working towards. Which is probably some mix of 1, 2 and 3.

For myself, 4 screams out as being the dominant factor. It seems to be composed mostly of 3, some 1 and a small amount of 2. For others, the sense I get is that 1 > 2 > 3 > 4 but I don't feel too strongly about it.

[^1]: It goes beyond just using it in a negotiation of course. It improves my model of the world and this improved model may prove useful in some other context. Still, I'm skeptical.

[^2]: To the extent that this is true, self-improvement should probably be viewed more conspicuously.