Harley-ml/lesswrong huggingface dataset:
# Top lesson from GPT: we will probably destroy humanity "for the lulz" as soon as we are able.

Forget complicated "sharp left turn" schemes, nefarious nanobots, lists of lethalities, out-of-distribution actions, failed AI boxing. As Zvi pointed out in multiple posts, like [this one](https://thezvi.wordpress.com/2023/04/13/ai-7-free-agency/), if humans get unrestricted access to a powerful enough tool, it is all over. People will intentionally twist even the most aligned Tool AI into an Agent of Death, long before it becomes superintelligent and is able to resist. You can find examples of it online.

In that sense, Eliezer was wrong in the worst possible way: we have a lot less time to get our act together, because capabilities advance faster than intelligence and humans are very inventive at finding ways to misuse these capabilities. We will push these capabilities in the "gain of function" direction mercilessly and without regard for safety or anything else. We are worse than toddlers playing with matches. True, like with toddlers, our curiosity far outstrips our sense of self-preservation, probably because our brains are not wired to be afraid of something that is not like a snake or a spider or a steep cliff. But it is worse than that. People will try to do the worst thing imaginable because they do not "[alieve](https://en.wiktionary.org/wiki/alieve)" potential harm, even if they can track it logically, unlike a toddler.

I guess the silver lining is that we have a bit of time to iterate. The AI tools are not yet at the level of causing widespread destruction, and probably will not be for some time. It does not mean that if and when some superintelligence emerges we will be well prepared, but if humanity survives until then without self-annihilation, we might have a better chance, compared to the "one shot at getting it right" before we are all wiped out, as Eliezer emphasized. It might not be an "alignment manual from the surviving future", but at least some wisdom and discipline of avoiding the early pitfalls, and if we die, we might die with "more dignity". The odds are not great, but maybe they are there.

Edit: quanticle pointed out that Bostrom predicted it in the paper [The Vulnerable World Hypothesis](https://nickbostrom.com/papers/vulnerable.pdf):

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8f401622314bcd700860eb55f0f2e057f2e1757e6c6d5320.png)
# Making Booking.Com less out to get you

Some guy loved booking.com but hated the false urgency they added to the process, so he wrote a plug-in to block the awful elements and reframe the useful ones to grab less attention. I like this both as a useful free thing,  and for the general vibe of fixing things and sharing those fixes. 

![Yubi Onboarding Screens](https://uploads-ssl.webflow.com/5f2fcf657d8a0847eae64ad8/5f313bf548c0a7c3687d3211_stressy-messages-2x.jpg)
# AutoBound on neural network can achieve OOMs lower training loss

Seems like gradient descent methods weren't using the relevant math bounds so far. Google released AutoBound as an [open-source library](https://github.com/google/autobound). 

Here is what I consider a money shot of the article (notice it's a log-plot):

| [![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgutnTe8tbfFqdlv7G1CWmk5LgRRVRIuw2veGSrIOv3N7k-dKGCWZMmIb8n3drlkWcZSc2dcwd2-BmxCm-9I7y5q3lAcCkB17wC2nMrIeSbZeQeck6P2W_bu4VvQ38QkVKr5CDB62pwIV0zEGmG_n_r8oY9vnv92Jv-Dh-ZS3Qb_20QV2rQTEdmRH1_pg/w640-h423/image8.png)](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgutnTe8tbfFqdlv7G1CWmk5LgRRVRIuw2veGSrIOv3N7k-dKGCWZMmIb8n3drlkWcZSc2dcwd2-BmxCm-9I7y5q3lAcCkB17wC2nMrIeSbZeQeck6P2W_bu4VvQ38QkVKr5CDB62pwIV0zEGmG_n_r8oY9vnv92Jv-Dh-ZS3Qb_20QV2rQTEdmRH1_pg/s1656/image8.png) |
| --- |
| Performance of SafeRate when used to train a single-hidden-layer neural network on a subset of the [MNIST](https://yann.lecun.com/exdb/mnist/) dataset, in the full-batch setting. |

Hopefully, they are just overfitting on MNIST. Otherwise, it pattern-matches to a huge advance. Their repo implies that with float64 this scales to larger neural networks. LLMs seem to reliably get new capabilities with lower loss, at least.

What do you think?

Here are related technical details:

> Optimizers that use upper bounds in this way are called [majorization-minimization](https://epubs.siam.org/doi/book/10.1137/1.9781611974409) (MM) optimizers. Applied to one-dimensional logistic regression, AutoBound rederives an MM optimizer [first published in 2009](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3129714/). Applied to more complex problems, AutoBound derives novel MM optimizers that would be difficult to derive by hand.

> We can use a similar idea to take an existing optimizer such as [Adam](https://arxiv.org/pdf/1412.6980.pdf) and convert it to a hyperparameter-free optimizer that is guaranteed to monotonically reduce the loss (in the full-batch setting). The resulting optimizer uses the same update direction as the original optimizer, but modifies the learning rate by minimizing a one-dimensional quadratic upper bound derived by AutoBound. We refer to the resulting meta-optimizer as SafeRate.

> Using SafeRate, we can create more robust variants of existing optimizers, at the **cost of a single additional forward pass** that increases the [wall time](https://en.wikipedia.org/wiki/Elapsed_real_time) for each step by a small factor (about 2x slower in the example above).

This seems novel to neural network training, or am I missing something that Bayesian neural net people have been doing already?
# Sama Says the Age of Giant AI Models is Already Over

There wasn't much content to this article, except for the below quotes. 

Sam Altman says

> further progress will not come from making models bigger. “I think we're at the end of the era where it's going to be these, like, giant, giant models,” he told an audience at an event held at MIT late last week. “We'll make them better in other ways.” \[...\] Altman said there are also physical limits to how many data centers the company can build and how quickly it can build them. \[...\] At MIT last week, Altman confirmed that his company is not currently developing GPT-5. “An earlier version of the letter claimed OpenAI is training GPT-5 right now,” he said. “We are not, and won't for some time.”

Nick Frosst, a cofounder at Cohere, says 

> “There are lots of ways of making transformers way, way better and more useful, and lots of them don’t involve adding parameters to the model,” he says. Frosst says that new AI model designs, or architectures, and further tuning based on human feedback are promising directions that many researchers are already exploring.

In the [Lex Fridman](https://www.youtube.com/watch?v=L_Guz73e6fw) interview with Sam Altman, Sam said that they had to do "[hundreds of complicated things](https://steno.ai/lex-fridman-podcast-10/367-sam-altman-openai-ceo-on-gpt-4-chatgpt-and)". Does this, together with the above quote, suggest Sam thinks transformers are running out of oomph? Is he, perhaps, pausing progress whilst we await the next breakthrough in deep learning?

Edit: Added in a relevant question. Give your probability for the first option. 

Elicit Prediction ([forecast.elicit.org/binary/questions/VuF0joaV6](forecast.elicit.org/binary/questions/VuF0joaV6))

Or vote at this tweet, if you like:   
[https://twitter.com/Algon_33/status/1648047440065449993](https://twitter.com/Algon_33/status/1648047440065449993)
# NYT: The Surprising Thing A.I. Engineers Will Tell You if You Let Them

[NYT Opinion article by Ezra Klein on AI-regulations](https://www.nytimes.com/2023/04/16/opinion/this-is-too-important-to-leave-to-microsoft-google-and-facebook.html). Ezra has been [writing quite a lot](https://www.nytimes.com/by/ezra-klein) on AI recently. Thought people might be interested in discussing them here.
# [Linkpost] AI Alignment, Explained in 5 Points (updated)

[This](https://medium.com/@daniel_eth/ai-alignment-explained-in-5-points-95e7207300e3) is an updated version of my [previous](https://forum.effectivealtruism.org/posts/CghaRkCDKYTbMhorc/the-importance-of-ai-alignment-explained-in-5-points) explainer on the topic. This version includes some updated information and figures, and it is generally better for sharing than the previous version.

As before, the piece is crafted to be easy to understand, avoiding jargon and not assuming familiarity with AI alignment or intimate knowledge of AI.

The piece breaks down the case for AI alignment into five points:

1.  Advanced AI is possible
2.  Advanced AI might not be that far away
3.  Advanced AI might be difficult to direct
4.  Poorly-directed advanced AI could be catastrophic for humanity
5.  There are steps we can take now to reduce the danger

To check out the piece, you can follow this link:

[https://medium.com/@daniel_eth/ai-alignment-explained-in-5-points-95e7207300e3](https://medium.com/@daniel_eth/ai-alignment-explained-in-5-points-95e7207300e3)
# GWWC Reporting Attrition Visualization

A few weeks ago [Giving What We Can](https://www.givingwhatwecan.org/) published their [impact evaluation](https://forum.effectivealtruism.org/posts/EipE75vsDuD7bdJar/). As an organization that supports people in making long term altruistic commitments, the question of how much people stick to those commitments is pretty important. We don't know what people are actually doing, but whether they're reporting donations to GWWC is a decent proxy, and the evaluation includes a table of how many members were still reporting donations:

[![](https://www.jefftk.com/proportion-of-gwwc-pledgers-reporting-any-donations-table.png)](https://www.jefftk.com/proportion-of-gwwc-pledgers-reporting-any-donations-table-big.png)

This is interesting data, but I found it a bit hard to think about. I tried a few ways of visualizing it, and made a pair of charts:

[![](https://www.jefftk.com/gwwc-recording-by-cohort.png)](https://www.jefftk.com/gwwc-recording-by-cohort-big.png)

[![](https://www.jefftk.com/gwwc-recording-decay-by-cohort.png)](https://www.jefftk.com/gwwc-recording-decay-by-cohort-big.png)

*[code](https://github.com/jeffkaufman/gwwc-recording-decay)*

I colored each line with an intensity proportional to the number of people in the cohort, since that represents how much information it represents. The weighted average line fades out over time as it represents progressively fewer members.

Some thoughts, looking at the charts:

*   Later cohorts, starting around 2015, fall off faster than earlier cohorts.
    
*   Reporting attrition is quite sharp at first, and then slows down, but it doesn't go to zero.
    
*   After the initial attrition it seems to end up around 2-6% annually, though this depends a lot on where you see "initial attrition" ending.
    
*   There's probably some sort of effect of the calendar year ("how hard did GWWC push people to report donations") and I haven't looked at this.
    

*Disclosure: [my wife](https://juliawise.net/) used to be President of GWWC. I haven't run this post by her and I don't know her views here. I shared a draft of with GWWC [before posting](https://www.jefftk.com/p/run-posts-by-orgs).*
# Stability AI releases StableLM, an open-source ChatGPT counterpart

Stability AI are the people behind Stable Diffusion.  
Currently only 7B version is available, but they are currently training versions up to 65B, with 175B model planned. 7B version already seems to be quite capable.  
Good for capabilities, but quite bad for alignment and race dynamics. What do you think?
# OpenAI could help X-risk by wagering itself

*While brainstorming on “conditional on things going well with AI, how did it happen”, I came up with the following idea. I think it is extremely unrealistic* right now*.*[^ngdxpz5kdoa]* However, it might become relevant later if some key people in top AI companies have a change of heart. As a result, it seems useful to have the idea floating around.*

* * *

OpenAI[^pv31hoh4hpl] is at the center of attention regarding recent AI progress. I expect that in the view of both public and the politicians, OpenAI’s actions and abilities are representative of actions and abilities of AI companies in general. As a result, if OpenAI genuinely tried to make their AI safe, and failed in a very spectacular manner, this would cause a sudden and massive shift in public opinion on AI *in general*.

Now, I don’t expect OpenAI to sacrifice itself by faking incompetence. (Nor would I endorse being deceitful here.) However, there might be some testable claim that is something of crux for both “AI-optimists” and, say, MIRI. OpenAI could make some very public statements about their abilities to control AI. And they could really stick their neck out by making the claims falsifiable and making a possible failure impossible to deny (and memetically fit, etc). This way, if they fail, this would create a [Risk Awareness Moment](https://forum.effectivealtruism.org/posts/L8GjzvRYA9g9ox2nP/prospects-for-ai-safety-agreements-between-countries?utm_source=EA+Forum+Digest&utm_campaign=54cf22c7d1-EMAIL_CAMPAIGN_2023_04_20_12_22&utm_medium=email&utm_term=0_-54cf22c7d1-%5BLIST_EMAIL_ID%5D) where enough people become mindful of AI-risk that we can put in place extreme risk-reduction measures that wouldn’t be possible otherwise. Conversely, if they succeed, that would be a genuine evidence that such measures are not necessary.

**I end with an open problem:** Can we find claims, about the ability of AI companies to control their AI, that would simultaneously:

1.  be testable,
2.  serve as strong evidence regarding the question where extreme measures regarding AI-risk are needed,
3.  have the potential to create a risk-awareness moment (ie, be salient to decision-makers and public),
4.  be possible for an AI company to endorse despite satisfying (i-iii).

[^ngdxpz5kdoa]: Mostly, I don’t expect this to happen, because any given AI company has nothing to gain from this. However, some individual employees of an AI company might have sufficient uncertainty about AI risk to make this worth it to them. And it might be possible that a sub-group of people within an AI company could make a public commitment such as the one above somewhat unilaterally, in a way that wouldn’t allow the company to back out gracefully 

[^pv31hoh4hpl]: I will talk about OpenAI for concreteness, but this applies equally well to some of the other major companies (Facebook, Microsoft, Google; with “your (grand)parents know the name” being a good heuristic).
# DeepMind and Google Brain are merging [Linkpost]

The new organization, called Google DeepMind, will be led by Demis Hassabis (current CEO of DeepMind). Jeff Dean (co-founder of Google Brain) will be the chief scientist of the new organization. 

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/e2561272179da90ff7ddaf03fdb94429d6f7584258061d96.png)

See also:

*   [Hooray for stepping out of the limelight](https://www.lesswrong.com/posts/vEJAFpatEq4Fa2smp/hooray-for-stepping-out-of-the-limelight)
*   [DeepMind's CEO helped take AI mainstream. Now he's urging caution.](https://time.com/6246119/demis-hassabis-deepmind-interview/)
*   [DeepMind reportedly lost a yearslong bid to win more independence from Google](https://www.theverge.com/2021/5/21/22447272/deepmind-google-independence-tensions-negotiations-wsj-report)
    *   "One suggestion from DeepMind’s founders was apparently for the company to have the same legal structure as a nonprofit, 'reasoning that the powerful artificial intelligence they were researching shouldn’t be controlled by a single corporate entity, according to people familiar with those plans.' But Google wasn’t on board with this, telling DeepMind it didn’t make sense considering how much money the company has poured into DeepMind."
# The Case for Brain-Only Preservation

This is a great compilation of the arguments for why cryonics service providers should offer a brain-only option. 

**Not mentioned in the article**: Note that a brain-only option is also offered by [OregonCryo](http://www.oregoncryo.com/) and [Cryonics Germany](https://cryonics-germany.org/en/).

**Summary of the linked post**:

The main advantages are:
- Lower storage cost
- Easier emergency relocation (because of size and legal status)
- Lower "yuck" factor
- Compatible with organ donation
- Compatible with a burial ceremony
- Reduction of fracturing
- Permits emergency fixation (in case of ln2 delivery issues)

**Related posts**: 
- [#2: Neurocryopreservation vs whole-body preservation](https://www.lesswrong.com/posts/TZgdQ4ghND9FP3ocm/2-neurocryopreservation-vs-whole-body-preservation) (from the [Cryonics Signup Guide](https://www.lesswrong.com/s/weBHYgBXg9thEQNEe))
- Section "Higher packing densities" of [The Allocation of Long Term Care Costs at Alcor](https://www.cryonicsarchive.org/library/allocation-of-long-term-care-costs-at-alcor/)
# Where to start with statistics if I want to measure things?

A while back I read [How to Measure Anything](https://www.lesswrong.com/posts/ybYBCK9D7MZCcdArB/how-to-measure-anything) and found it fascinating. In my day job, I spend quite a bit of time trying to make sense of the world by looking at dashboards of requests, latencies, error rates, etc. (software systems).

After finishing the book and taking copious notes, I understood that it gave me a prepackaged process that I could apply as-is, but I found it very difficult to adapt to everyday situations. I don't think I picked up a good intuition about stats, in other words.

I'm looking to change that. Specifically, I want to learn to apply stats in these two situations:

- measuring things. Mostly software systems, but open to little experiments. [Dan Luu](https://danluu.com/term-latency/) used to measure a lot of fun things.
- understanding how others measure things. I'd like to be able to judge if claims made in a paper about covid spread or social media addiction are backed up by the math/data in the paper. 

The challenge I'm facing is that I know a bunch of techniques, but not how they relate to each other and the problems they're meant to solve. To illustrate what I mean: I know how to get percentiles and calculate means, but until today morning I didn't know why averaging percentiles is usually a bad idea. I'm missing the map.

I've seen these books recommended as a good way to start:

- Statistics, 4th Edition 4th Edition, by Freedman, Pisani, and Purves
- Probability Theory: The Logic of Science, by Jaynes
- An Introduction to Error Analysis: The Study of Uncertainties in Physical Measurements, by Taylor
- Think Stats, by Downey

But I also wanted to ask someone familiar with the field:

- Is it best to start with an introductory textbook and branch out from there?
- Are there specific subfields / topics I should be focusing on (or avoiding)? 
- Is what I'm looking to learn labeled in some way? For example, I can't tell if this is data analytics or data science or X.
# Should we openly talk about explicit use cases for AutoGPT?

I think many people were thinking about AutoGPT-type agents for a while before people built and published the code on GitHub. Being conservative with talking about those agents might have led to their development a month or two later than would have happened if we had talked more openly about them. 

Right now I'm unsure whether or not to talk about concrete near-time uses. Is talking about applications and thus motivation capabilities deployment bad? Or is it good because it makes us understand the landscape better?
# Gas and Water

In general I think more people would be better off in the direction of [preparing more for disasters](https://www.jefftk.com/p/disasters). But here's a case where it didn't work out very well.

In some cases it would be [helpful to have extra gasoline](https://theprepared.com/homestead/guides/how-store-gasoline/). Maybe something happens and you need to leave quickly, but your car is low on gas. Or there's a power outage and you need fuel for a generator. The main downside of keeping a can of gas on hand would be the hassle of rotating it—even with fuel stabilizer gas has a shelf life, so ~once a year you use the gas and go refill. Keeping a can of gas on hand for emergencies seemed worth it: glad to have it if you need it, smallish occasional hassle. I kept five gallons in a [new can](https://www.amazon.com/gp/product/B000W9JN4S/) in deep shade on the north side of our property, outside and away from the house.

I thought of this as low risk enough that when it was time to rotate the gas and our [shared](https://www.jefftk.com/p/shared-car-one-year-in) car was full, I put it in the tank of a friend's car. They were keeping their car in our driveway after flying out of [Logan](https://en.wikipedia.org/wiki/Logan_International_Airport), and I wanted to refill the tank after driving it a bit. Pretty poor choice in retrospect. When they got back they started driving to VT, got about half a mile and the engine died.

Hours of hassle, towing, and garage later, it turns out that a large fraction of the 4-5gal of gas I put in was actually water. According to the mechanic it was a "shit ton"—when pressed they guessed a couple gallons. I'm still not sure how this happened, but I guess somehow water got into the can and displaced a substantial quantity of gasoline? Past a modern gas can's vapor-tight seal?

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02Y6CHRbLepY6945NcoNYmkCGS7dZxuy73hfVWymerf779zV6GPW9tQJDegLg1baAGl), [mastodon](https://mastodon.mit.edu/@jefftk/110234192222064199)*
# 500 Million, But Not A Single One More - The Animation

This video is an animation of [500 Million, But Not A Single One More](https://forum.effectivealtruism.org/posts/jk7A3NMdbxp65kcJJ/500-million-but-not-a-single-one-more), by [@jai](https://forum.effectivealtruism.org/users/jai?mention=user) 

### Some notes for the curious about the cult of Sopona, the god of Smallpox of the Yoruba people

When I was getting feedback on the storyboard from Jai, he mentioned that there is an actual god of Smallpox: [Sopona](https://en.wikipedia.org/wiki/Sopona). We decided to use it to lightly inspire our design by adding a mask similar to the ones found in wooden statues representing the god. 

Apparently, in Nigeria there were cults of Sopona that strongly opposed vaccination. At the bottom-left paragraph at [this link](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2035631/?page=1), there's an interesting story. [Dr. O. Sapara](https://en.wikipedia.org/wiki/Oguntola_Sapara), a Nigerian physician, infiltrated "the sopono cult", described as a large secret society in western Nigeria that used to cause smallpox outbreaks to generate clients to "cure" and to follow-up on blackmails. Sapara infiltrated the society and helped the government ban the cult as an illegal organization. After the cult was banned worship of the God continued. 

Jai rightly mentioned the priests of the cult as a striking example of the most hollow humans allying with the enemy.

 ![](https://res.cloudinary.com/cea/image/upload/f_auto,q_auto/v1/mirroredImages/So9nbufgNuQBjNkTG/i4bwmxkwod4zynndairs)
# Neural network polytopes (Colab notebook)

The [polytope theory of neural networks](https://www.lesswrong.com/posts/eDicGjD9yte6FLSie/interpreting-neural-networks-through-the-polytope-lens) (also known as the "[spline theory of deep learning](https://proceedings.mlr.press/v80/balestriero18b.html)") seeks to explain (ReLU) neural networks based on their piecewise linear regions. This gives a helpful intuition for how neural networks approximate functions, as well as a potential avenue for interpretability research. For anyone who wants to engage directly with the topic, I put together a [small Colab notebook](https://colab.research.google.com/github/zfurman56/polytopes/blob/main/Polytopes.ipynb), with interactive 2D and 3D visuals.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/95534d87bd4c1bf454d34f53de752d94fb5744e5b17a6dde.png)
# LessWrong moderation messaging container

This is a shell post for various comments which I'm using as the source-of-truth for some moderation messaging (currently, what a new user sees when they go to make their first post or comment).

It seemed good to make it public so people could comment on it if they wanted. I'm particularly unsure about how to handle things like "not being familiar with background material." I think it's easy to overdo and make it feel very burdensome, but also man it sure is annoying having a bunch of people showing up missing a lot of context.

(You can comment on this post, the two initial comments are just used as the text-source for some places in our codebase)
# How did LW update p(doom) after LLMs blew up?

Here's something which makes me feel very much as if I'm in a cult:

After LLMs became a massive thing, I've heard a lot of people p(doom) on the basis that we were in shorter timelines. 

How have we updated p(doom) on the idea that LLMs are very different than hypothesized AI? 

Firstly, it would seem to me to be much more difficult to FOOM with an LLM, it would seem much more difficult to create a superintelligence in the first place, and it seems like getting them to act creatively and be reliable are going to be much harder problems than making sure they aren't *too* creative.

LLMs often default to human wisdom on topics, the way we're developing them with AutoGPT they can't even really think privately, if you had to imagine a better model of AI for a disorganized species to trip into, could you get safer than LLMs?

Maybe I've just not been looking the right places to see how the discourse has changed, but it seems like we're spending all the weirdness points on preventing the training of a language model that at the end of the day will be slightly better than GPT-4.

I will bet any amount of money that GPT-5 will not kill us all.
# A great talk for AI noobs (according to an AI noob)

I'm far from an expert on AI and alignment but I really liked this [Ted talk](https://forum.effectivealtruism.org/out?url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DC_78DM8fG6E) and think it serves as a good introduction to the topic. Here are some highlights:

It starts with a demo of what ChatGPT is currently or soon will be capable and it's bound to shocking to plenty of noobs like me.

Then OpenAI cofounder is also pressed on the company's controversial, perhaps reckless, approach. He explains why OpenAI has often been the first to release AI products despite safety concerns (particularly at [24:17](https://youtu.be/C_78DM8fG6E?t=1457)) In short:

The cofounder says that OpenAI's views programing Chatgpt like teaching a child insofar as regular servings of positive and negative reinforcement are better than trying to create a master plan ahead of time. OpenAI currently want the world's help feeding feedback into their "children" and although their view may change, Alan Turing suggested this approach in 1950 and OpenAI  still doesn't see a better approach.

Also (at [20:52](https://youtu.be/C_78DM8fG6E?t=1252)) I think OpenAI's cofounder claims that the company is "starting to really get good at is predicting" emergent capabilities of AI. That sounds a little too good to be true so I'm curious what yall think.

And please correct me if I'm wrong or missing something important (hopefully before I share this talk with my friends :)
# Why do we care about agency for alignment?

Many people believe that understanding "agency" is crucial for alignment, but as far as I know, there isn't a canonical list of reasons why we care about agency. Please describe any reasons why we might care about the concept of agency for understanding alignment below. If you have multiple reasons, please list them in separate answers below.

Please also try to be specific as possible about what our goal is in the scenario. For example:

> We want to know what an agent is so that we can determine whether or not a given AI is a dangerous agent

Whilst useful isn't quite as good as:

> We have an AI which may or may not have goals aligned with us and we want to know what will happen if these goals aren't aligned. In particular, we are worried that the AI may develop instrumental incentives to seek power and we want to use interpretability tools to help us figure out how worried we should be for a particular system.  
>   
> We can imagine a scale of such systems according to how it behaves in novel situations:
> 
> *   On the low end of this scale, we have a system that only follows extremely broad heuristics that it learned during training
> *   On the high end of this scale, we have a system that uses general reasoning capabilities to discover new and innovative strategies even if it has never used anything like this strategy before, or seen it in the training data
> 
> We can think of such a scale as a concretisation of agency. This scale provides an indication of both:
> 
> *   How dangerous a system is likely to be: though a system with excellent heuristics and very little agency could be dangerous as well
> *   What kind of precautions we might want to take: for example, how much can we rely on our off-switch to disable the agent

> It's plausible that interpretability tools might be able to give us some idea of where an agent is on this scale just by looking at the weights. So having a clear definition of this scale could help by clarifying what we are looking for.

In a few days, I'll add any use cases I'm aware of myself that either haven't been covered or that I don't think have been adequately explained by different answers.
# Are there AI policies that are robustly net-positive even when considering different AI scenarios?

One thing I might have noticed recently is a lot of governance on AI might require specific models of AI in society, especially over misalignment or misuse concerns. 

Are there AI policies that could be robustly-net positive without having to tie it to a specific scenario?
# Do LLMs dream of emergent sheep?

This is somewhat speculative, but borne out by empirical data: LLMs developed an analogue of human System 1 and System 2, and their version of "System 2" is also slow, effortful and unreliable, just like ours.

[https://twitter.com/emollick/status/1650324834960678912](https://twitter.com/emollick/status/1650324834960678912)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/4a48b09bd99e724db038ae7345ff94479998ff5b6e4e5734.png)

This is an unusual emergent behavior, by all accounts. It is not like there is a shortage of accurate math in the training data. Maybe distilling these particular rules from the data is harder? 

Interestingly enough, augmentation works for LLMs the same way it works for humans: you can get a math plugin, such as [Wolfram Alpha](https://www.wolfram.com/wolfram-plugin-chatgpt/), which enhances the model's abilities temporarily, but does not improve its "native" abilities with prolonged use. The moment the plugin (or a human's calculator) is "disconnected" the model (or the human) is back to its previous "self", not having learned much. 

Maybe reasoning is harder in general? Or maybe it is an artifact of the training data? Is something like a System 2 an emergent feature? Does it mean that there is a "ceiling" of sorts for logical reasoning by the current crop of models, and we should not expect them to, say, code better than a human, write better poetry, prove theorems and discover new laws of nature? Would it have some implication for super-intelligent machines? I must admit, I am quite confused.
# A concise sum-up of the basic argument for AI doom

1 - An artificial [super](https://www.lesswrong.com/tag/superintelligence)-[optimizer](https://www.lesswrong.com/tag/optimization) is likely to be [developed soon](https://blog.wordbot.io/ai-artificial-intelligence/a-brief-history-of-the-generative-pre-trained-transformer-gpt-language-models/).

2 - [There is no known way of programming goals into an advanced optimizer, only outwardly observable behaviors of which we have no idea why they are being carried out or what motivates them.](https://youtu.be/AaTRHFaaPG8?t=8074)

3 - [Most utility functions do not have optima with humans in them.](https://youtu.be/AaTRHFaaPG8?t=8587) Most utility functions do not have a term for humans at all.

4 - "Why haven’t we exterminated all mice/bugs/cows then?" draws quite a poor analogy. Firstly, we are not superoptimizers. Secondly, and more importantly, we care about living beings somewhat. The optimum of the utility function of the human civilization quite possibly does have mice/bugs/cows, perhaps even [genetically engineered to not experience suffering](https://www.hedweb.com/). We are not completely indifferent to them.

The relationship between most possible superoptimizers and humanity is not like the relationship between humanity and mice at all - it is much more like the relationship between humanity and natural gas. Natural gas, not mice, is a good example of something humans are truly indifferent to - there is no term for it in our utility function. [We don’t hate it, we don’t love it, it is just made of atoms that we can, and do, use for something else.](https://www.lesswrong.com/tag/instrumental-convergence)

Moreover, the continued existence of natural gas probably does not pose the same threat to us as the continued existence of us would pose to a superoptimizer which does not have a term for humans in its utility function - just like we don’t have a term for natural gas in ours. Natural gas can not attempt to turn us off, and it can not create "competition" for us in form of a rival species with capabilities similar to, or surpassing, our own.   
 

P.S. If you don’t like or find confusing the terminology of "optimizers" and "utility functions", feel free to forget about all that. Think instead of physical states of the universe. Out of all possible states the universe could find itself in, very, very, very few contain any humans. Given a random state of the universe, creating a superintelligence that steered the universe towards that state would result in an almost guaranteed apocalypse. Of course, we want our superintelligence to steer the universe towards particular states - that’s kind of the whole point of the entire endeavor. Were it not so, we would not be attempting to build a superintelligence  - a sponge would suffice. We essentially want to create a super-capable universe-steerer. The problem is getting it to steer the universe towards states we like - and this problem is very hard because (among other things) currently, given any internal desire to steer the universe towards any state at all, it is impossible to program that desire into anything at all.
# Max Tegmark's new Time article on how we're in a Don't Look Up scenario [Linkpost]

[https://time.com/6273743/thinking-that-could-doom-us-with-ai/](https://time.com/6273743/thinking-that-could-doom-us-with-ai/)

Max Tegmark has posted a Time article on AI Safety and how we're in a "[Don't Look Up](https://www.imdb.com/title/tt11286314/)" scenario. 

In a similar manner to Yudkowsky, Max went on Lex Fridman and has now posted a Time article on AI Safety. (I propose we get some more people into this pipeline)

Max, however, portrays a more palatable view regarding societal standards. With his reference to Don't Look Up, I think this makes it one of my favourite pieces to send to people new to AI Risk, as I think it describes everything that your average joe needs to know quite well. (An asteroid with a 10% risk of killing humanity is bad)

In terms of general memetics, it will be a lot harder for someone like LeCun to come up with a genius equivalence between [asteroid safety and airplane safety](https://twitter.com/ylecun/status/1642688520694165507) with this framing. (Which might be a shame since it's one of the dumber counter-arguments I've heard.) 

But who knows? He might just claim that scientists know and have always known [how to use nuclear bombs to shoot the asteroid away](http://large.stanford.edu/courses/2018/ph241/shabb2/#:~:text=There%20are%20two%20different%20ways%20that%20have%20been,completely%20explode%20the%20asteroid%20itself%20using%20a%20bomb.) or something.

What I wanted to say with the above is that I think Max is doing a great way of framing the problem, and with his respect from his earlier career as a physicist, I think it would be good to use his articles more in public discussions. I also did quite enjoy how he described alignment on the [Lex Fridman podcast](https://www.bing.com/videos/search?q=max+tegmark+lex+fridman&&view=detail&mid=FA33D7D2B7D81420D289FA33D7D2B7D81420D289&&FORM=VRDGAR&ru=%2Fvideos%2Fsearch%3Fq%3Dmax%2Btegmark%2Blex%2Bfridman%26FORM%3DHDRSC6), and even though I don’t agree with all he says, it’s good enough.
# Recent Database Migration - Report Bugs

Hey all,

LessWrong is currently in the process of migrating databases (from mongo to postgres).  Today we migrated a large number of important collections, including Posts, Comments, Tags, Votes, and others related to those.  If you notice any **new bugs** related to posts, comments, tags, or voting, please let us know ASAP!

The best way to ping us is via Intercom - unless you've hidden it, there should be an icon on the bottom-right of your screen (on desktop views):

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/ef9f4b22f397256b3b1ceef5ab54829d7bed466ffae5efdb.png)

You can also post here.

(Please also report existing bugs, if you're aware of any, just let us know if you're doing so - new bugs are more likely to be related to the migration and are more likely to be time-sensitive.)

Known Issues
============

Read Statuses
-------------

Migration for these only partially succeeded.  Recovery in progress (last attempt briefly took the site down).

Comments Counts on Post Items
-----------------------------

Outdated.  Will try to get this fixed after Read Statuses.
# Automating the Breath Pulse

One of the pieces of my [rhythm stage setup](https://www.jefftk.com/news/jammer) is a breath controller, which measures how much I blow. It's a good fit for anything where I want to have some flexible imprecise control, but one thing I've ended up using it for a lot is just a pulse:

Here's it by itself:

Here's it live, layered with everything else I'm doing:

I'm still interested in having breath control as an option, for cases where I want more compelx patterns, but a basic pulse seems like something I should be able to automate. My system already knows the current tempo from how I'm playing the drums, so I could program it to simulate my breath pattern.

My first attempts at this sounded all wrong: it turns out I didn't actually know what pattern I was making. It was just something where I'd played around with blowing different ways until it sounded the way I wanted it. So I collected some data, tracking breath pressure as a function of how far through the beat I was:

[![](https://www.jefftk.com/reel-breath-pressure-t.png)](https://www.jefftk.com/reel-breath-pressure-t-big.png)

[![](https://www.jefftk.com/jig-breath-pressure-t.png)](https://www.jefftk.com/jig-breath-pressure-t-big.png)

I could do some kind of averaging to make nicer charts, but the overall picture is clear enough: I increase it linearly to a peak near the end of the beat (68% in reel time, 82% in jig time), and then decrease it linearly to a trough near the beginning of the next beat (27% in reel time, 41% in jig time). I programmed that, and now I can have the same pulse with my mouth free to do something else:

While I'm happy with how the result sounds, I'm pretty confused by the curves. The pulse sounds like it's on the upbeat, so I would have expected a peak at 50% (reel time), and I'd have expected the pictured curve to sound quite late? It seems like we identify the beat with the center of the trajectory (48% in real time, 62% in jig time)?

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0ym4W9ff7RyLEpqtguVSsdmmVfbeNXYc81ThTpxQtG7tmnJDVhZKmj6qdhJovT6aAl), [mastodon](https://mastodon.mit.edu/@jefftk/110267860824845556)*
# Interview with Paul Christiano: How We Prevent the AI’s from Killing us

[0:00](https://www.youtube.com/watch?v=GyFkWb903aU&t=0s) Intro  
[9:20](https://www.youtube.com/watch?v=GyFkWb903aU&t=560s) Percentage Likelihood of Death by AI  
[11:24](https://www.youtube.com/watch?v=GyFkWb903aU&t=684s) Timing   
[19:15](https://www.youtube.com/watch?v=GyFkWb903aU&t=1155s) Chimps to Human Jump  
[21:55](https://www.youtube.com/watch?v=GyFkWb903aU&t=1315s) Thoughts on ChatGPT  
[27:51](https://www.youtube.com/watch?v=GyFkWb903aU&t=1671s) LLMs & AGI  
[32:49](https://www.youtube.com/watch?v=GyFkWb903aU&t=1969s) Time to React?  
[38:29](https://www.youtube.com/watch?v=GyFkWb903aU&t=2309s) AI Takeover   
[41:51](https://www.youtube.com/watch?v=GyFkWb903aU&t=2511s) AI Agency   
[49:35](https://www.youtube.com/watch?v=GyFkWb903aU&t=2975s) Loopholes   
[51:14](https://www.youtube.com/watch?v=GyFkWb903aU&t=3074s) Training AIs to Be Honest   
[58:00](https://www.youtube.com/watch?v=GyFkWb903aU&t=3480s) Psychology   
[59:36](https://www.youtube.com/watch?v=GyFkWb903aU&t=3576s) How Solvable Is the AI Alignment Problem?  
[1:03:48](https://www.youtube.com/watch?v=GyFkWb903aU&t=3828s) The Technical Solutions (Scalable Oversight)   
[1:16:14](https://www.youtube.com/watch?v=GyFkWb903aU&t=4574s) Training AIs to be Bad?!   
[1:18:22](https://www.youtube.com/watch?v=GyFkWb903aU&t=4702s) More Solutions  
[1:21:36](https://www.youtube.com/watch?v=GyFkWb903aU&t=4896s) Stabby AIs   
[1:26:03](https://www.youtube.com/watch?v=GyFkWb903aU&t=5163s) Public vs. Private (Lab) AIs  
[1:28:31](https://www.youtube.com/watch?v=GyFkWb903aU&t=5311s) Inside Neural Nets  
[1:32:11](https://www.youtube.com/watch?v=GyFkWb903aU&t=5531s) 4th Solution   
[1:35:00](https://www.youtube.com/watch?v=GyFkWb903aU&t=5700s) Manpower & Funding   
[1:38:15](https://www.youtube.com/watch?v=GyFkWb903aU&t=5895s) Pause AI?  
[1:43:29](https://www.youtube.com/watch?v=GyFkWb903aU&t=6209s) Resources & Education on AI Safety   
[1:46:13](https://www.youtube.com/watch?v=GyFkWb903aU&t=6373s) Talent    
[1:49:00](https://www.youtube.com/watch?v=GyFkWb903aU&t=6540s) Paul’s Day Job  
[1:50:15](https://www.youtube.com/watch?v=GyFkWb903aU&t=6615s) Nobel Prize   
[1:52:35](https://www.youtube.com/watch?v=GyFkWb903aU&t=6755s) Treating AIs with Respect   
[1:53:41](https://www.youtube.com/watch?v=GyFkWb903aU&t=6821s) Uptopia Scenario   
[1:55:50](https://www.youtube.com/watch?v=GyFkWb903aU&t=6950s) Closing & Disclaimers
# Simultaneous Footbass and Footdrums

For the past year or so I've had two options for my feet with my [rhythm stage setup](https://www.jefftk.com/news/jammer). I can play bass:

Or I can play drums:

But when I play mandolin I want both!

I tried a system where I play bass but whichever drum pedal I tap triggers a kick as well, which sort of works but I'm not good enough at playing a consistent rhythm while moving between pedals:

This almost works, but I couldn't get it to where I was happy with it.

Another thing I tried was pulling my hand away from the mandolin briefly to change the current bass note:

This works on that particular tune which isn't very notey and it happens that each time I want to change to a new bass note the melody asks for an open string, but it doesn't work in general.

A few days ago I got another idea: allocate the left two pedals to drums (kick and snare) and the right two pedals to bass. Since this gives me three options: right heel, right toe, and both together. Since three chords is ([sort of](https://www.jefftk.com/p/what-chords-do-you-need)) [all you need](https://tvtropes.org/pmwiki/pmwiki.php/Main/ThreeChordsAndTheTruth), this offers an interesting balance. Currently I have it set so I can choose between major (I IV V), minor (i bVII bVI), and mixolydian (I bVII V).

I'm still getting it into my muscle memory, but here's how it sounds so far:

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid0VowtfjekPmbYQPebcgL2V7Q1YvKfWekh3kV95Rwv44wD4PMRvNKcSnLhpz74Fqbyl), [mastodon](https://mastodon.mit.edu/@jefftk/110273684176642243)*
# Podcast with Divia Eden and Ronny Fernandez on the strong orthogonality thesis

On my side podcast, ["The Filan Cabinet"](https://thefilancabinet.com/), I invited Ronny Fernandez and Divia Eden to talk about the strong orthogonality thesis, and whether it's true. Seems like people here might also be interested. Podcast description below, and you can listen [here](https://youtu.be/D5rEMNyfIWw).

---

In this episode, Divia Eden and Ronny Fernandez talk about the (strong) orthogonality thesis - that arbitrarily smart intelligences can be paired with arbitrary goals, without additional complication beyond that of specifying the goal - with light prompting from me. Topics they touch on include:
 - Why aren't bees brilliant scientists?
 - Can you efficiently make an AGI out of one part that predicts the future conditioned on some plans, and another that evaluates whether plans are good?
 - If minds are made of smaller sub-agents with more primitive beliefs and desires, does that shape their terminal goals?
     - Also, how would that even work?
 - Which is cooler: rockets, or butterflies?
 - What processes would make AIs terminally value integrity?
 - Why do beavers build dams?
 - Would these questions be easier to answer if we made octopuses really smart?
# AI safety logo design contest, due mid-May

Please have a look at the updated version of this post on the EA Forum.

This contest is about creating a logo to represent AI safety research. 

I’m being supported by an independent funder and people from [nonlinear.org](https://www.nonlinear.org/) and am taking care of administration and communication.

We’re doing an open process (voting) and want to involve different people through the contest to get a better logo design and to increase the chances of adoption.

### **1) rough concept ideas contest**

**deadline: May 15th 2023**

We’re looking for different quick and great design ideas for this round. Not everything has to be spot on yet, this will be done in 2) and 3) .

Submit as many rough logo design ideas as you want as a .PNG and .SVG file to this Google Form: [https://forms.gle/6fwVYbPBrqVdCHax6](https://forms.gle/6fwVYbPBrqVdCHax6)

### **2) feedback & voting on best design ideas chosen by jury**

**May 20th - Juni 3rd 2023 (2 weeks)**

Via another Google Form, this post will be updated with the link once there’s a **selection by the jury of 5 - 10 contest submissions** available.

Only accounts with at least 100 Karma on [forum.effectivealtruism.org](https://forum.effectivealtruism.org/)or [lesswrong.com](https://www.lesswrong.com/) are allowed to vote.

With a **$ 1.5k prize pool** for the best design ideas from the contest which are being used further, this might be one design or multiple as decided by the votes & the jury, with the prize money spread out evenly. 

We’re open for someone to **increase the prize pool**, please send me an email!

### **3) polishing of winning design(s)**

We’ll hire a great designer for creating the final logo from the winning design(s).

* * *

  
After the completion of the contest the final logo will be publicly available for free use at [www.aisafety.art](http://www.aisafety.art/) .  
If you have any questions please reach out to me at **ai_logo@cipriani.studio** 

\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-

I’d like to disclose here that I’m also trying to enter an idea myself, however I won’t be able to decide on my own which designs are considered for polishing.

Composition of the jury: two people from Nonlinear, an independent funder, me (Adrian Cipriani)