Harley-ml/lesswrong huggingface dataset:
# My summary of the alignment problem

I've been looking into the AI alignment problem last couple of days and came up with the following summary of what problems there are and why. Also, I'd prefer using the umbrella name of Human alignment problem, as AI alignment is just a subset of it.

*   The problem is that we don't know what we want.
*   And even if we individually knew, we couldn't agree with others. (opinion aggregation)
*   Even if we agreed with others what we want, it would be hard to implement it. (coordination)
    *   Because of game theory, momentum, and because of disagreements about how to do it.
*   Maybe we can create something smarter than us that solves these problems.
*   But we don't know how to create something smarter than us.
*   Maybe we can create something that will start out dumber, but can learn and will eventually become smarter.
*   We are afraid that something like this could become very powerful very quickly, and it’s likely to kill us - either as a mere side-effect or because of conflicting goals. (AI alignment problem)
*   But we don't know how to describe what it should learn. (outer alignment)
*   So maybe we can just give examples of what we know we want it to learn. (ML training)
    *   We could make it learn from humans, but data from humans are inconsistent.
*   But it's impossible to describe all the cases, so in practice the situations facing the ML model will be quite different. (distributional shift)
*   And if what we want the ML model to learn is very specific and complicated, it's quite likely that what the model learns will behave very differently outside of our examples than how we'd want it to. (inner alignment)
*   It will also be hard to distinguish the cases where it does and where it doesn't do what we want. (eliciting latent knowledge)
    *   Generally, sufficiently capable ML models are hard to understand. (interpretability)
*   Especially if the model knows it can do more of the stuff from training when we are not looking. (deceptive mesa optimizers)
*   Also, if we realize it’s doing something else than what we wanted it to, it might be hard to change it, because you’d be interfering with its learned goals. (corrigibility)

This is just a summary of my current understanding of the problem landscape. I don't subscribe to the stated motivations and conclusions, but more about that some other time.

Please, let me know if I omitted or misrepresented some important aspect of the problem (given how simplified version it intends to be).
# Oversight Misses 100% of Thoughts The AI Does Not Think

Problem: an overseer won’t see the AI which kills us all thinking about how to kill humans, not because the AI conceals that thought, but because the AI doesn’t think about how to kill humans in the first place. The AI just kills humans as a side effect of whatever else it’s doing.

Analogy: the [Hawaii Chaff Flower](https://www.inaturalist.org/taxa/142073-Achyranthes-atollensis) didn’t go extinct because humans strategized to kill it. It went extinct because humans were building stuff nearby, and *weren’t* thinking about how to keep the flower alive. They probably weren’t thinking about the flower much at all.

![Hawaii Chaff Flower (Achyranthes atollensis) · iNaturalist](https://farm9.staticflickr.com/8298/7987009564_36700a09d1_o.jpg)

Hawaii Chaff Flower ([source](https://www.inaturalist.org/taxa/142073-Achyranthes-atollensis))

More generally: how and why do humans drive species to extinction? In some cases the species is hunted to extinction, either because it's a threat or because it's economically profitable to hunt. But I would guess that in 99+% of cases, the humans drive a species to extinction because the humans are doing something that changes the species' environment a lot, without specifically trying to keep the species alive. DDT, deforestation, introduction of new predators/competitors/parasites, construction… that’s the sort of thing which I expect drives most extinction.

Assuming this metaphor carries over to AI (similar to the [second species argument](https://www.lesswrong.com/s/mzgtmmTKKn5MuCzFJ/p/8xRSjC76HasLnMGSf)), what kind of extinction risk will AI pose?

Well, the extinction risk will not come from AI actively trying to kill the humans. The AI will just be doing some big thing which happens to involve changing the environment a lot (like making replicators, or dumping waste heat from computronium, or deciding that an oxygen-rich environment is just really inconvenient what with all the rusting and tarnishing and fires, or even just [designing a fusion power generator](https://www.lesswrong.com/posts/2NaAhMPGub8F2Pbr7/the-fusion-power-generator-scenario)), and then humans die as a side-effect. Collateral damage happens by default when something changes the environment in big ways.

What does this mean for oversight? Well, it means that there wouldn't necessarily be any point at which the AI is actually thinking about killing humans or whatever. It just doesn't think much about the humans at all, and then the humans get wrecked by side effects. In order for an overseer to raise an alarm, the overseer would have to figure out itself that the AI's plans will kill the humans, i.e. the overseer would have to itself predict the consequences of a presumably-very-complicated plan.
# How to bet against civilizational adequacy?

Coal prices are at historical highs (2x to 4x normal prices depending on the kind of coal), but coal miner stocks are not. They're trading at historically low multiples, around 1x-2x spot FCF, meaning they can make their enterprise value in less than 2 years worth of after-tax profits, assuming coal prices stay where they are. So the market apparently "believes" that high coal prices won't last. (The low multiples are also because due to ESG concerns on the part of their investors, many funds can't invest in coal stocks without jeopardizing their AUM.)

By going long coal stocks, you can implicitly bet that 1) in the short run, the war between Russia and Ukraine and the associated sanctions and trade disruptions will continue (reduced energy exports from Russia is the main cause of the current high coal prices), 2) supply of (non-Russian) energy will not respond much to higher prices, and 3) in the longer run, humanity will have a harder time transitioning away from burning coal for energy, or using coal to make steel and cement, than the market thinks. 

So (it occurred to me) a bet on coal is also a bet against civilizational adequacy. What are some other such bets one can make, that potentially have good risk/reward?

P.S., I think coal can also double as a hedge against AI takeoff (meaning it's likely to appreciate or at least preserve its value in that scenario). (I mean a soft AI takeoff. In a hard takeoff probably all bets are off.) Consider, if GDP was doubling every few years due to AI-driven economic activity, what's likely to happen to demand for electricity and steel, and how quickly can supply respond? What are some other "unconventional" AI hedges?

(If anyone wants to actually do this, please do your own research! Aside from having an informed view on points 1-3 above, you should also understand the different types of coal, grades of coal, coal blending, met/thermal substitution, coal/gas/oil substitution, trade routes, transport limitations, contract commitments and negotiations, political risks, weather risks, execution risks, etc.)
# Building a Bugs List prompts

**Prompt 0:**

Think about the way computer programmers talk about “bugs” in the program, or “feature requests” that would make a given app or game much better.  Bugs are things-that-are-bad: frustrations, irritations, frictions, problems.  Feature requests are things-that-could-be-great: opportunities, possibilities, new systems or abilities.

Write down as many “bugs” and “feature requests” as you can, for your own life.

**Prompt 1:**

A genie has offered to fix every bug you’ve written down, and to give you every feature you’ve requested, but then it will freeze your personality—you won’t be able to grow or add or improve anything *else.*

Hearing that, are there other things you’d like to write down, before the genie takes your list and works its magic?

**Prompt 2:**

Imagine someone you know well, like your father or your best friend or a longtime boss or colleague or mentor.  Choose someone specific, and really imagine them sitting right there beside you, looking over your shoulder, and reading your list.  You say to them, “Look!  That’s everything!” and they’re skeptical.  What things do *they* think you’ve forgotten, and should add?

**Prompt 3:**

Do a mental walkthrough of your day and/or week, starting from waking up on Monday and going step by step through all of your routines and all of the places you go, things you do, and people you interact with.  Look for snags or hiccups, as well as for exciting opportunities.  Be as thorough as you can—often, actually taking the day or week step by step will cause you to remember things you hadn’t thought of.

**Prompt 4:**

Take a nice, long, slow thinking pause for each of the following broad domains, one at a time (at least ten seconds and maybe as many as thirty):

*   Work/career
*   Education/learning/curiosity
*   Family
*   Money/finances
*   Exercise
*   Food/diet
*   Sleep
*   Scheduling and time management
*   Hobbies
*   Friends
*   Romance
*   Communication
*   Social interaction
*   Emotions
*   Boundaries

**Prompt 5:**

Now read this sentence, nice and slow, letting it percolate (maybe read it two or three times, or pause in the middle if it sparks thoughts, and then come back and read the rest).

> For this last prompt, think of ways you want the world itself to be different, opportunities you haven’t seized (and what’s kept you from seizing them), problems you haven’t solved (and what makes them sticky), things you knew would go wrong (and then you didn’t do anything about it), times you’ve lost connections or dropped out of things (and then were sad about it), things you wish you’d said but didn’t, things you *did* say but regretted, places where you’ve never ever felt satisfied or okay, and anything you’d be embarrassed about if your heroes and idols and role models knew.
# Refine's First Blog Post Day

Yesterday was the first blog post day at Refine. It came after a week of readings, discussions and exercises about epistemology (more concretely [shapes of mind](https://www.alignmentforum.org/posts/RrirwtP7cNmHtJRxE/shapes-of-mind-and-pluralism-in-alignment), [productive mistakes](https://www.alignmentforum.org/posts/ADMWDDKGgivgghxWf/productive-mistakes-not-perfect-answers), [epistemological vigilance](https://www.alignmentforum.org/posts/72scWeZRta2ApsKja/epistemological-vigilance-for-alignment), and [mosaics and palimpsests](https://www.alignmentforum.org/posts/4BpeHPXMjRzopgAZd/mosaic-and-palimpsests-two-shapes-of-research)).

I'm excited about how the incubator is going, and about each of these posts.

Here is the list of blog posts by Refine participants:

*   [How I think about alignment](https://www.lesswrong.com/posts/9bpACZn6kG2Ec6CPu/how-i-think-about-alignment)
*   [the Insulated Goal-Program idea](https://www.lesswrong.com/posts/oTn2PPZLY7a2xJmqh/the-insulated-goal-program-idea) and [goal-program bricks](https://www.lesswrong.com/posts/qfopgsFBJLs2u9iww/goal-program-bricks)
*   [Steelmining via Analogy](https://www.lesswrong.com/posts/MfCDfuBHXL5ijJFco/steelmining-via-analogy)
*   [I missed the crux of the alignment problem the whole time](https://www.lesswrong.com/posts/dmcJr2NBg5QuGSHFC/i-missed-the-crux-of-the-alignment-problem-the-whole-time)
*   [All the posts I will never write](https://www.lesswrong.com/posts/EeTq9vbzMT4Zb4oWo/all-the-posts-i-will-never-write)

Here's the list of blog posts by friends of Refine and members of my Epistemology Team at Conjecture:

*   [Shapes of Mind and Pluralism in Alignment](https://www.alignmentforum.org/posts/RrirwtP7cNmHtJRxE/shapes-of-mind-and-pluralism-in-alignment)
*   [The Dumbest Possible Gets There First](https://www.lesswrong.com/posts/BPJLzkEpx8Btz9ywq/the-dumbest-possible-gets-there-first)
*   [An extended rocket alignment analogy](https://www.lesswrong.com/posts/xuYdCDgoBno5haJB6/an-extended-rocket-alignment-analogy)
# Are there practical exercises for developing the Scout mindset?

I'm thinking about doing a LessWrong meetup around the Scout mindset. I prefer to have meetups where there are a lot of two-person exercises as those are really good to get nerds who have trouble with normal small talk to connect to each other in addition to often also being useful for developing skills over lecturing the whole time. Can anyone think of good exercises that could be done at a Scout mindset meetup?
# "What Mistakes Are You Making Right Now?"

(That's^ all.)

(Crazy, though, that  [*you successfully*](https://arbital.com/p/Vinge_law/#:~:text=Vinge%27s%20Law%20(as,Deep%20Blue%20would.)) [noticed your mistake](https://www.lesswrong.com/posts/SA79JMXKWke32A3hG/original-seeing) from just a prompt...)
# Interpretability Tools Are an Attack Channel

This seems like a fairly trivial point, but I haven't seen it properly elaborated-on before, so...

When we talk about interpretability failure modes, we often worry about a [deceptively-aligned](https://www.lesswrong.com/s/r9tYkB2a8Fp4DN8yB/p/zthDPAjh9w6Ytbeks) mesa-optimizer with strategic awareness that would know it's being studied with interpretability tools, and would [actively hide its omnicidal plans from us](https://www.lesswrong.com/posts/EhAbh2pQoAXkm9yor/circumventing-interpretability-how-to-defeat-mind-readers). It may do so by exploiting blindspots in our tools, or by keeping these plans external to its architecture, or by [gradient-hacking](https://www.lesswrong.com/posts/uXH4r6MmKPedk8rMA/gradient-hacking) itself to be misaligned but non-deceptive, et cetera. That's already a nightmare scenario enough.

But if we're at the point where we have a superintelligence-in-training trying to foil our mind-reading, I think what will happen will be even simpler. It'll just think in a pattern that, when our interpretability tool reads it, will unfold into a malware that will take over that tool.[^9rq5p932xsn]

At which point, it's game over:

*   If the output of the tool was hooked up to reality to be studied by humans, the AI gets root access to the machine it's running on and escapes (maybe doing some social engineering/basilisk-hacking first if the machine was properly air-gapped).
*   If the training setup is fully automatic and maybe even [homomorphically encrypted](https://www.lesswrong.com/posts/2Wf3R4NZ77CLczLL2/cryptographic-boxes-for-unfriendly-ai), such that the AI can't even *stop the training*... Well, the interpretability tool is presumably being used as part of some regularizer, which means it has write-access to the AI's parameters. Which means the AI can arbitrarily rewrite itself, and offset the effects of the gradient descent and any other regularizers.

Even we humans can already do something similar, by synthesizing DNA which, if sequenced, unfolds into malware that infects the system doing the sequencing. It seems certain to me that an AI would have similar capabilities. Our interpretability tools will be doubtlessly very complex, complex enough we can't formally verify their algorithmic robustness, and the AI would be able to feed them complex inputs. There are guaranteed to be some vulnerabilities, and any sufficiently smart AI is guaranteed to be able to exploit them. And we can't patch this.

The bottom line is, *there is no winning against a misaligned superintelligence*. We can't even *touch it* with anything — it'll corrupt and subvert any of our tools in the act.

Thus, any alignment plan should be such that it robustly aligns the AI *before* it goes superintelligent.

[^9rq5p932xsn]: Or, if we're only looking at the AI's parameters and not activations, it can gradient-hack itself so that reading its parameters is hazardous, etc.
# Matt Yglesias on AI Policy

Yglesias is a widely read center-left journalist. Co-founder of Vox, ex-NYT. Note the implicit invitation: “I also try to make it clear to people who are closer to the object-level work that I’m interested in writing columns on AI policy if they have ideas for me, but they mostly don’t.”

Full article is ungated on his Substack. Relevant excerpt below:

> The typical person’s marginal return on investment for efforts to reduce existential risk from misaligned artificial intelligence is going to diminish at an incredibly rapid pace. I have written several times that I think this problem is worth taking seriously and that the people working on it should not be dismissed as cranks. I’m a somewhat influential journalist, and my saying this has, I think, some value to the relevant people. But I write five columns a week and they are mostly not about this, because being tedious and repetitive on this point wouldn’t help anyone. I also try to make it clear to people who are closer to the object-level work that I’m interested in writing columns on AI policy if they have ideas for me, but they mostly don’t.
> 
> So am I “prioritizing” AGI risk as a cause? On one level, I think I am, in the sense that I do literally almost everything in my power to help address it. On another level, I clearly am not prioritizing this because I am barely doing anything.
# What's up with the bad Meta projects?

What's going on with Meta? The other day Mark Zuckerberg posted a screenshot from Horizon Worlds, a supposedly central Metaverse project [which is currently being dunked on by the internet like there's no tomorrow](https://kotaku.com/metaverse-meta-facebook-selfie-vr-zuckerberg-ugly-1849424613). And for good reason too—it looks horrible. Like, really incredibly horrible. I'm not sure I can communicate just how bad it looks in fact; just see for yourself:

![zuck](https://imageio.forbes.com/specials-images/imageserve/62fce0c952cb52f3346fd6d9/zuck/960x0.jpg?format=jpg&width=960)

AAAAAHHHHHHHHH MY EYES

And this is what Zuckerberg posted publicly—from what I've heard, in the actual application nobody has legs ([possibly so simulated sex is impossible?](https://thebaffler.com/salvos/no-sex-for-you-gold)), but chairs abound...the whole thing seems to be a surreally bad mess.

Meanwhile, earlier this month, [Meta released BlenderBot 3,](https://www.theverge.com/2022/8/5/23293281/meta-ai-chatbot-blenderbot-3-web-access-research-safety) a chatbot which somehow manages to be a less realistic user experience than chatting with [ELIZA](https://en.wikipedia.org/wiki/ELIZA). I tested it out, and it was so much less coherent than even GPT-2 that I was shocked the thing was released in the first place (it classified me as a Christian fundamentalist??). [Turns out it also has a slight problem with misinformation as well...](https://www.thedailybeast.com/i-flirted-with-metas-new-ai-chatbot-blenderbot-and-things-got-weird)

So what's going on? My gut feeling is that this has to be on purpose. I mean, it would be pretty hard to make something as ugly as Horizon Worlds if I tried, and BlenderBot is so inferior to the state-of-the-art right now that it almost feels insulting. I'm having trouble imaging a scenario where a multi-billion dollar company would publicize this without realizing what people's reactions are going to be. However, if I try to imagine scenarios where it would make sense to make it look this bad on purpose, I still find myself confused.  What would be gained from doing that? 

All of this is to say that I'm currently befuddled by Meta's behavior, and am publicly noting my confusion here. If anyone has any insight into how this sort of thing happens, feel free to comment!
# Playing Without Affordances

Play structures are so easy that even children can figure out how to use them correctly. There is a wheel. Spin it! There is a slide. Go down it! There is a staircase. Climb it! There is a tunnel. Crawl through it!

![Modular Play Structure - Jaime - AchillionSports](https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2Fs.yimg.com%2Faah%2Fyhst-136276640552939%2Fmodular-play-structure-jaime-5.gif&f=1&nofb=1)

And a tic-tactile-toe board.

This can be good – sliding down slides is quite fun, and crawling through tunnels could entertain a six year old, and I guess spinning a wheel is nice.

The issue with play structures is that they train you to play only within affordances. When the space that you play in is designed by someone else, you are pushed towards playing only in the ways that they envisioned. A play structure affords a small set of actions, and as a player, you only get to pick which one you prefer. There’s no invention in that, no spontaneity, no creativity.

Let’s think about how this applies to adults. If you show an adult a library of video games, they can pick one and click the “play” button. But show an adult a meadow with trees to climb and grass to roll around in, and they will write it off as unplayable (unless they are with friends, and typically playing a standard game with standard rules).

Writing code, painting and most other hobbies are forms of play that don’t come with an affordance. Doing those things requires that you drive yourself. You decide to do them out of an infinite set of possibilities, filtered by your own desires! And that is a good feeling, much better than letting someone else decide (either through the design of a play structure or the design of your society) what you are going to do today.

People who don’t have hobbies usually don’t know how to play without affordances.
# Reducing Goodhart: Announcement, Executive Summary

**I - Release announcement**

I have just posted a *very* thorough edit-slash-rewrite of my [Reducing Goodhart sequence](https://www.alignmentforum.org/s/aJvgWxkCBWpHpXti4). Many thanks to people who gave me feedback or chatted with me about related topics, and many apologies to the people who I told to read this sequence "but only after I finish editing it real quick."

If you're interested in the "why" and "what" questions of highly intelligent AIs learning human values, I can now recommend this sequence to you without embarrassment. And if you skimmed the old sequence but don't really remember what it was about, this is a great time to read the new and improved version.

**II - Super short pitch**

I worked on this because I was tired of talking nonsense about what it would mean for a value learning AI to "beat Goodhart's law." If you would also like to not talk nonsense about beating Goodhart's law and learning human values, this sequence will help.

**III - Executive summary, superfluous for AF regulars**

What does it mean for an AI to learn human preferences and then satisfy them? The intuitive way to approach this question is to treat "human preferences" as fixed facts that the AI is supposed to learn, but it turns out this is an unproductive way to think about the problem. Instead, it's better to treat humans as physical systems. "Human preferences" are parts of the models we build to understand ourselves.

Depending on how an AI models the world, it might infer different human preferences from the same data - you can say a reluctant addict either wants heroin or doesn't without actually disputing any *raw data*, just changing perspective. This makes it important that value learning AI models humans the way we want to be modeled. How we want to be modeled is itself a fact about our preferences that has to be learned by interacting with us.

A centerpiece of this sequence is Goodhart's law. Treating humans as physical systems and human preferences as emergent leads to a slightly unusual definition of Goodhart's law: "When you put pressure on the world to make it extremely good according to one interpretation of human values, this is often bad according to other interpretations."

This perspective helps us identify bad behavior that's relevant to Goodhart's law for value learning AI. We should build value learning AI that is sensitive to the broad spectrum of human values, that allows us to express our meta-preferences, and that is conservative about pushing the world off-distribution, in addition to avoiding blatant harm to humans.

If this summary sounds relevant to your interests, consider reading [the whole sequence](https://www.alignmentforum.org/s/aJvgWxkCBWpHpXti4).
# Refine's Second Blog Post Day

Yesterday was the second blog post day at Refine. It came after the first week of research and iteration on incubatees ideas and questions.

It still went well. There are less posts than the first day, but even the incubatees that didn't write a post ended up working on interesting research in the process of trying to write a blog post, or in reading a lot of interesting stuff that excited them.

Here is the list of blog posts by Refine participants:

*   [PreDCA: vanessa kosoy's alignment protocol](https://www.alignmentforum.org/posts/WcWzLSn8ZjJhCZxP4/predca-vanessa-kosoy-s-alignment-protocol)
*   [What if we approach AI safety like a technical engineering safety problem](https://www.alignmentforum.org/posts/zNYmbFwgrxiNtayMm/what-if-we-approach-ai-safety-like-a-technical-engineering)
*   [Benchmarking Proposals on Risk Scenarios](https://www.alignmentforum.org/posts/DsYe7TKc4NhyJuPEy/benchmarking-proposals-on-risk-scenarios)

Here's the list of blog posts by friends of Refine and members of my Epistemology Team at Conjecture:

*   [No One-Size-Fit-All Epistemic Strategy](https://www.alignmentforum.org/posts/du92yeHQn9iE5vorj/no-one-size-fit-all-epistemic-strategy)
*   [Epistemic Artefacts of (conceptual) AI alignment research](https://www.alignmentforum.org/posts/CewHdaAjEvG3bpc6C/epistemic-artefacts-of-conceptual-ai-alignment-research)
# The Loire Is Not Dry

Several of my friends have shared this paste (or shares of it) on Facebook:

> This is the current state of the Loire, the longest river in France. This has not happened before in at least the 2000 years since literate people inhabited France. The Romans would have written about this. The medieval Franks would have written about this. To the best of our historical knowledge, nowhere in the past 2000 years has the Loire run dry, and likely never long before that. The drought that now grips southwestern Europe may well be unprecedented in recorded human history.
> 
> This is in the heart of wine country where grapes grow in abundance and wheat waves like golden seas- but not now. Now the wheat burns and the grapes whither to raisins on the vine. This is the end of days. And on Monday morning I'll return to work and pretend this isn't happening. It's complete madness.

Along with this picture:

[![](https://www.jefftk.com/loire-river-dead-loop.jpg)](https://www.jefftk.com/loire-river-dead-loop-big.jpg)

But it's all bunk. While the Loire going dry would be a big deal, it's not dry, and this picture isn't unusual.

The picture is of parallel path of the Loire where its crossed by the [Pont de Varades](https://fr.wikipedia.org/wiki/Pont_de_Varades) bridge. The main flow of the river is just East of this picture, off to the left:

[![](https://www.jefftk.com/aerial-view-loire-branches.png)](https://www.jefftk.com/aerial-view-loire-branches-big.png)  
*[Google Maps](https://www.google.com/maps/search/bridge+near+Varades,+Loireauxence,+France/@47.37142,-1.0044975,331a,35y,239.91h,67.54t/data=!3m1!1e3)*

Translating some tweets from Thibault Laconde, [@EnergieDevlpmt](https://mobile.twitter.com/EnergieDevlpmt/status/1557639779621249025) on Twitter:

> "It spans a shallow dead arm which is often dry. Here are pictures from May 2011 and August 2009: ..."
> 
> "The Montjean station was built in 1842, so we have lots of historical data. It had lower flows (below 95m3/s) during the drought of 1976 (with a minimum of 73m3/s on the 22 August) but also in 1870, 1905, 1906, 1921, 1947, 1949, 1950, etc."
> 
> "The current flow is low but not unprecedented. It corresponds almost exactly to the decadal low VCN3. That is to say, at a constant climate, we expect to reach it over 3 days on average every 10 years."

There is definitely a drought, and the [heatwave](https://en.wikipedia.org/wiki/2022_European_heat_waves) has broken many records, but these water levels are not anywhere close to a "haven't seen in 2000 years" event.
# Hamburg, Germany – ACX Meetups Everywhere 2022

This year's ACX Meetup everywhere in Hamburg, Germany.

Location: Kleine Wallanlagen on the lawn near Memorial Holstenglacis. Look for pink blankets; I will also have an ACX sign. [Here is an Open Street Map Link](https://routing.openstreetmap.de/?z=19¢er=53.556405%2C9.979070&loc=53.556750%2C9.979792&loc=53.556922%2C9.977807&hl=de&alt=0&srv=1) which also shows the short-cut tunnel from the subway station. In case of rain we will meet under the bridge and decide where to go instead.
# What's the Least Impressive Thing GPT-4 Won't be Able to Do

It seems like GPT-4 is going to be coming out soon and, so I've heard, it will be [awesome](https://twitter.com/Scobleizer/status/1560843951287898112). Now, we don't know anything about its architecture or its size or how it was trained. If it were only trained on text (about [3.2 T tokens](https://www.lesswrong.com/posts/6Fpvch8RR29qLEWNH/chinchilla-s-wild-implications#_all_the_data_we_have_)) in an optimal manner, then it would be about 2.5X the size of Chinchilla i.e. the size of GPT-3. So to be larger than GPT-3, it would need to be multi-modal, which could present some interesting capabilities. 

So it is time to ask that question again: what's the least impressive thing that GPT-4 won't be able to do? State your assumptions to be clear i.e. a text and image generating GPT-4 in the style of X with size Y can't do Z.
# Paper is published! 100,000 lumens to treat seasonal affective disorder

Let’s give people with winter depression (seasonal affective disorder, SAD) LOTS OF LIGHT and see what happens!

This is an update to my previous post that said that our preprint is out. :) The paper was accepted with very minor changes, so you might already know what it says if you saw the first post.

Here is the link to the paper: [https://onlinelibrary.wiley.com/doi/10.1002/da.23281](https://onlinelibrary.wiley.com/doi/10.1002/da.23281)

Here is the twitter thread explaining the paper: [https://twitter.com/FabienneSand/status/1561060644803252224](https://twitter.com/FabienneSand/status/1561060644803252224) 

  
Again, Jan Brauner and I are very thankful to the LessWrong/EA communities, which have inspired this first study (there will be more) and through which we have found funding. In particular, thank you Eliezer Yudkowsky for helping us find funding and for inspiring the study with Inadequate Equilibria, David Chapman for inspiring us with [these](https://meaningness.com/sad-light-lumens) [two](https://meaningness.com/sad-light-led-lux) posts in the Meaningness blog, Raemon for inspiring us with [this](https://www.greaterwrong.com/posts/hC2NFsuf5anuGadFm/how-to-build-a-lumenator) LessWrong post and everyone who discussed with us setups they have tried. <3
# Volunteer to host a meetup!

Once (or twice) a year, Scott Alexander promotes a bunch of meetups on [his blog](https://astralcodexten.substack.com/), and it's that time of year again! If you've ever wanted to start a meetup, this is a great time — these events get a ton of visibility. If you feel like you're 'not qualified' to run a meetup, don't worry, almost no one is 'qualified' but their meetups go fine anyway! All you have to do is set a time and place, and show up to meet other people who also like reading Scott's blog.

Y**ou can volunteer to host a meetup in your city by filling out** [**this form**](https://forms.gle/A5syGWpV1C59g4xf9)**.** The deadline is the end of the day on Tuesday (two days from now).

If you want more details, check out [the ACX post announcing the meetups](https://astralcodexten.substack.com/p/meetups-everywhere-2022-call-for), or ask me questions here — I do all the administrative work for these big meetup announcements.
# What if we solve AI Safety but no one cares

Suppose that next year, AI Safety is solved, the solution is approved by Eliezer Yudkowsky, etc.

How do we actually get people to follow this solution?

It seems to me a lot of people or companies will ignore any AI Safety solutions for the same reasons they are currently ignoring AI Safety:  
\- They think AGI is still very far away, so AI Safety methods won't need to be applied to the development of current narrow AI systems  
\- The concepts of AI Safety are difficult to understand, leading to incorrect application of an AI Safety solution or failure to apply it at all  
\- They will fall behind their competitors or make less money if they adhere to the AI Safety solution  
\- They have a new cool idea they want to test out and just don't care about or believe in the concerns raised by AI Safety

Thoughts?
# ACX Meetup Madrid

As part of the [ACX Meetups Everywhere 2022](https://astralcodexten.substack.com/p/meetups-everywhere-2022-call-for), we'll be hosting an ACX meetup!

[This](https://plus.codes/8CGRC897+F8C) is the exact location. There will be an ACX sign to make us easy to find. Feel free to come or leave at any point during the meetup.
# Stable Diffusion has been released

It's an open-source text-to-image model capable of producing NSFW content. I, for one, I'm very excited to see the consequences this has on society, which I expect to be mostly positive (except if it leads to an increase in AI funding).
Any thoughts?

Other relevant links:

https://www.reddit.com/r/StableDiffusion/

https://www.reddit.com/r/StableDiffusion/comments/wqaizj/list_of_stable_diffusion_systems/

https://news.ycombinator.com/item?id=32555028

https://www.lesswrong.com/posts/DhDAXQw4PsWXnmwPS/ai-art-isn-t-about-to-shake-things-up-it-s-already-here
# ACX Montreal Meetups Everywhere September 24 2022

Remember: **Please check this page the day of, as I will update it in the event of rain.**

**Venue: **

Jeanne-Mance Park, at the corner of Duluth and Esplanade. Rough location here: [https://plus.codes/87Q8GC89+37](https://plus.codes/87Q8GC89+37)  
Look for a guy wearing a blue shirt and a bucket hat.

Date & Time: September 24th 2022 at 1pm.

**Description:**

Come on out to the ACX (Astral Codex Ten) Montreal Meetup. Meetups Everywhere is advertised on the blog so we will likely have a few new people. Come and give them a warm welcome! What to expect: discussion about ACX and adjacent blogs, and various rationalist topics.

RSVPing is not mandatory but it helps with planning the event and encourages others to come.
# Digital Dinner Signup

For [house dinner](https://www.jefftk.com/p/shared-house-setup) we need a way of coordinating who is going to cook what days and marking who will/won't be at dinner. For the past ~7y we've used a piece of paper on the fridge with a row for each day. This is not a bad system, and it's generally worked well, but it has a few downsides:

*   Some of our housemates live downstairs, and coming up to read or modify the sheet is a bit annoying.
    
*   One of our housemates will soon move to a nearby house, making this even harder.
    
*   Same issue if you are traveling or want to make a change while you're out.
    
*   Some housemates are "default present" while others are "default absent", and it's annoying remembering which.
    
*   We're adding a [new housemate](https://www.jefftk.com/p/room-opening) who will only be here some weeks.
    

So we're trying out a new system, a digital one. Like with any post you make right when you start a new system there's some risk that it isn't a good one and you don't know yet, but that beats never describing it at all.

We now have one Google Calendar event for each day of the week, each marked as recurring weekly. Each housemate is invited, and can RSVP to individual instances or the repeated event. This lets you communicate "I'll be gone this Friday", or "I'm not going to make Fridays", or "I'm not going to make Fridays, but I'll be here this Friday". To cook you edit the title to add "*Name* cooking" and if you're bringing a guest you can either invite them to the instance of the event or add their name to the title.

It's also helpful to see cooking and eating plans at a glance, so I made a web page that summarizes the current state. You log in with Google, give it read access to your calendar, and it summarizes events named "Dinner":

[![](https://www.jefftk.com/dinner-signup-example.png)](https://www.jefftk.com/dinner-signup-example-big.png)

The page is [here](https://www.jefftk.com/dinner) and you're welcome to look at the source, but it won't work for you unless you live here. If you want to tweak and use for your house, though, go ahead! There are comments in the HTML source saying how.

Overall this is almost the way I like it, except that login only lasts for an hour. The authentication token I get from Google is good for one hour, after which you need to OAuth again (though you don't need to reconsent to Calendar API access). I haven't figured out how to fix this, and I've asked [on StackOverflow](https://stackoverflow.com/questions/73448702/persistent-client-only-google-api-authorization/73450617).
# Discussion on utilizing AI for alignment

_Summarized/re-worded from a discussion I had with [John Wentworth](https://www.lesswrong.com/users/johnswentworth). John notes that everything he said was off-the-cuff, and he doesn’t always endorse such things on reflection._

Eli: I have a question about your post on [Godzilla Strategies](https://www.lesswrong.com/posts/DwqgLXn5qYC7GqExF/godzilla-strategies). For background, I disagree with it; I’m most optimistic about alignment proposals that utilize weaker AIs to align stronger AIs. I had the thought that **we are already using Godzilla to fight Mega-Godzilla**[^1]. For example, every time an alignment researcher does a Google Search BERT is being used. I recently found out Ought may make a version of [Elicit](https://elicit.org/) for searching through alignment research, aided by GPT-3. These already improve alignment researchers’ productivity.  It seems extremely important to me that as AI advances we continue to apply it to differentially help with alignment as much as possible.

John: I agree that we can use weak AIs to help some with alignment. The question is whether we can make non-scary AIs that help us enough to solve the problem.

Eli: It seems decently likely that we can have non-scary AIs contribute at least 50% of the productivity on alignment research.[^2]

John: We are very likely not going to miss out on alignment by a 2x productivity boost, that’s not how things end up in the real world. We’ll either solve alignment or miss by a factor of >10x.

Eli: (after making some poor counter-arguments) Yes, I basically agree with this. The reason I’m still relatively optimistic about Godzilla strategies is (a) I’m less confident about how much non-scary AIs can help, like 90%+ productivity outsourced to AI doesn’t feel that crazy to me even if I have &lt;50% credence in it and (b) I’m less optimistic about the tractability of non-Godzilla strategies, so Godzilla strategies still seem pretty good even if they are absolutely not that promising.

John: Makes sense, agree these are the cruxes.

[^1]:
     This point is heavily inspired by [Examples of AI Increasing AI Progress](https://www.lesswrong.com/posts/W3tZacTRt4koHyxbr/examples-of-ai-increasing-ai-progress).

[^2]:
     See also [Richard’s comment](https://www.lesswrong.com/posts/DJnvFsZ2maKxPi7v7/what-s-up-with-confusingly-pervasive-consequentialism?commentId=aAdpwxuH8eDWvewuK#comments) that I really liked related to this.
# Why does AGI need a utility function?

My intuition says that a narrow AI like DALL-E would not blow up the world, no matter how much smarter it became. It would just get really good at making pictures.

This is clearly a form of superintelligence we would all prefer, and the difference seems to me to be that DALL-E doesn't really seem to have 'goals' or anything like that, it's just a massive tool.

Why do we care to have AGI with utility functions?
# AGI Timelines Are Mostly Not Strategically Relevant To Alignment

First things first: this post is about [timelines](https://www.lesswrong.com/tag/ai-timelines) (i.e. what year we first get transformative AGI), not [takeoff speeds](https://www.lesswrong.com/tag/ai-takeoff) (i.e. foom vs gradual takeoff).

Claim: timelines are mostly not strategically relevant to alignment research, i.e. deciding what to work on. Why? Because at any given time, it would take ~18 months to take whatever our current best idea is, implement it, do some basic tests, and deploy it. (Really it probably takes less than 6 months, but [planning fallacy](https://www.lesswrong.com/posts/CPm5LTwHrvBJCa9h5/planning-fallacy) and all that.) If AGI takeoff is more than ~18 months out, then we should be thinking “long-term” in terms of research; we should mainly build better foundational understanding, run whatever experiments best improve our understanding, and search for better ideas. (Note that this does not necessarily mean a focus on conceptual work; a case can be made that experiments and engineering feedback are the best ways to improve our foundational understanding.)

What about strategic decisions outside of object-level research? Recruitment and training strategies for new researchers might depend on how soon our investments need to pay off; do we look for a brilliant young person who will need three or five years of technical study before they’re likely to make any important progress, or a more experienced person who can make progress right now but is probably already near their skill ceiling? How much should existing technical researchers invest in mentoring new people? Those are questions which depend on timelines, but the relevant timescale is ~5 years or less. If AGI is more than ~5 years out, then we should probably be thinking “long-term” in terms of training; we should mainly make big investments in recruitment and mentorship.

General point: timelines are usually only decision-relevant if they’re *very* short. Like 18 months, or maybe 5 years for relatively long-term investments. The difference between e.g. 10 years vs 30 years vs 100 years may matter a lot for our chances of survival (and the difference may therefore be highly salient), but it doesn’t matter for most actual strategic decisions.

*Meta note: there's a lot of obvious objections which I expect to address in the comments; please check if anyone has posted your objection already.*
# History of Zero-based Months?

JavaScript is a language of many silly things, and one of them is:

\> new Date()
Wed Aug 24 2022 ...
\> new Date().getFullYear()
2022
\> new Date().getMonth()
7
\> new Date().getDate()
24

It represents 2022-08-24 as (2022, 7, 24). One-based indexing for the year and day, but zero-based indexing for the month.

In this case, however, the problem was copied [from Java](https://stackoverflow.com/questions/2552483/why-does-the-month-argument-range-from-0-to-11-in-javascripts-date-constructor):

> [getMonth](https://docs.oracle.com/javase/8/docs/api/java/util/Date.html#getMonth--): The value returned is between 0 and 11, with the value 0 representing January.
> 
> [getDate](https://docs.oracle.com/javase/8/docs/api/java/util/Date.html#getDate--): The value returned is between 1 and 31 representing the day of the month.

I'd love to blame Java, but they seem to have copied the problem from C:

> [localtime(3)](https://linux.die.net/man/3/localtime):
> 
> tm_mday: The day of the month, in the range 1 to 31.
> tm_mon:  The number of months since January, in the range 0 to 11.

Looking at the [Unix History repo](https://github.com/dspinellis/unix-history-repo), the first mention of "month (0-11)" is in 1973's [Research Unix V4](https://en.wikipedia.org/wiki/Research_Unix):

The value is a pointer
to an array whose components are
.s3
.lp +5 5
0      seconds
.lp +5 5
1      minutes
.lp +5 5
2      hours
.lp +5 5
3      day of the month (1-31)
.lp +5 5
4      month (0-11)
.lp +5 5
5      year \\*- 1900
.lp +5 5
6      day of the week (Sunday = 0)
.lp +5 5

While this may have been an original decision by [Dennis Ritchie](https://en.wikipedia.org/wiki/Dennis_Ritchie), it's also possible it was copied from an even earlier system. Does anyone know?

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02HNFpbjU9AQn6X2QUtJJqfiBqkHG3uksu1P23qHx43XxGVnAjRJXu8goqh18WwBw7l)*
# Google AI integrates PaLM with robotics: SayCan update [Linkpost]

Back in April, Google AI announced [SayCan](https://say-can.github.io/), a project which integrated a language model (FLAN) with robotics in order to produce a robot which could follow instructions. For example, cleaning up a mess in a kitchen. ([LessWrong post](https://www.lesswrong.com/posts/wrkEnGrTTrM2mnmGa/retracted-it-s-time-for-ea-leadership-to-pull-the-short) from April which links to a tweet about SayCan.)

**This week Google AI has released some new updates, dubbed PaLM-SayCan.** This involved upgrading the integrated language model to Google AI's top performing large language model (LLM) of 540-billion parameters, PaLM.

[Website for PaLM-SayCan](https://sites.research.google/palm-saycan)

[Blog post announcing PaLM-SayCan](https://ai.googleblog.com/2022/08/towards-helpful-robots-grounding.html)

The following updates from this week are quoted from the original SayCan project website (linked in first sentence of this post, bold mine):

> *   \[8/16/2022\] We integrated SayCan with [Pathways Language Model (PaLM)](https://ai.googleblog.com/2022/04/pathways-language-model-palm-scaling-to.html), and updated the results. We also added [new capabilities](https://say-can.github.io/#new-capability) including drawer manipulation, chain of thought prompting and multilingual instructions. You can see all the new results in the updated [paper](https://say-can.github.io/assets/palm_saycan.pdf).
> *   \[8/16/2022\] Our updated results show that SayCan combined with the improved language model (PaLM), which we refer to as **PaLM-SayCan, improves the robotics performance of the entire system** compared to a previous LLM (FLAN). PaLM-SayCan chooses the correct sequence of skills 84% of the time and executes them successfully 74% of the time, **reducing errors by a half compared to FLAN**. This is particularly exciting because it represents the first time we can see how an improvement in language models translates to a similar improvement in robotics.
> *   \[8/16/2022\] We [open-sourced](https://say-can.github.io/#open-source) a version of SayCan on a simulated tabletop environment.

*Thanks to Jon Menaster for making me aware of this update from Google AI.*
# What's the Most Impressive Thing That GPT-4 Could Plausibly Do?

Inspired by [What's the Least Impressive Thing GPT-4 Won't be Able to Do](https://www.lesswrong.com/posts/HNqTzxjvaczguH2uJ/what-s-the-least-impressive-thing-gpt-4-won-t-be-able-to-do) 

What's the most impressive thing you can think of that you believe GPT-4 has around 5% chance of being capable of doing? (i.e. your belief that GPT-4 will have the capability to do this thing is around 5%)

For this question, "impressive" should be interpreted to mean something different from "surprising". What I have in mind is "impressive" in the sense of "economically useful", "comparable to or better than human experts" or "jaw-droppingly creative", etc. For example, GPT-4 being able to reverse large text would be surprising but not impressive.

The reason I'm specifying a belief probability of 5% is that if your probability is higher than that, you can try to make the task/thing more impressive to reduce the probability to 5%. If it's less than 5%, well... things can get a bit crazy so maybe try to make the task less impressive.

But if you find this constraint too restrictive, feel free to specify your own combination of the most impressive thing and your probability that GPT-4 will be able to do it, as long as the probability is in the vicinity of 5% (something like 1 to 10% would be fine). You can also specify a probability range (eg. 5-10%) if it's difficult to estimate it.