Harley-ml/lesswrong huggingface dataset:
# A (EtA: quick) note on terminology: 
AI Alignment != AI x-safety

I think the terms "AI Alignment" and "AI existential safety" are often used interchangeably, leading the ideas to be conflated.  
  
In practice, I think "AI Alignment" is mostly used in one of the following three ways, and should be used exclusively for Intent Alignment (with some vagueness about whose intent, e.g. designer vs. user):  
![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d0d8eda05cb7846033599bc70fa132372b591ca87964d7dd.png)  
1) AI Alignment = How to get AI systems to do **what we want**

2) AI Alignment = How to get AI systems **to try** to do **what we want**

3) AI Alignment = A rebranding of **“AI (existential) safety”**...  A community of people trying to reduce the chance of AI leading to premature human extinction.

The problem with (1) is that it is too broad, and invites the response: "Isn't that what most/all AI research is about?"  
The problem with (3) is that it suggests that (Intent) Alignment is the one-and-only way to increase AI existential safety.

Some reasons not to conflate (2) and (3):
=========================================

1.  The case that increasing (intent) alignment increases x-safety seems much weaker on the margin than in the limit; the main effect of a moderate increase in intent alignment might simply be a large increase in demand for AI.
2.  Even perfect intent alignment doesn't necessarily result in a safe outcome; e.g. if everyone woke up 1000000x smarter tomorrow, the world might end by noon.
3.  X-safety can be increased through non-technical means, e.g. governance/coordination.
4.  EtA: x-safety can be increased through technical work other than alignment, e.g. assurance methods, e.g. [value alignment verification](https://arxiv.org/abs/2012.01557). 

In my experience, this sloppy use of terminology is common in this community, and leads to incorrect reasoning (if not in those using it than certainly at least sometimes in those hearing/reading it).  
  
EtA: This Tweet and associated paper make a similar point: https://twitter.com/HeidyKhlaaf/status/1634173714055979010
# Notes on the Mathematics of LLM Architectures

> *From a mathematical point of view, the building and training of a large transformer*  
> *language model (LLM) is the construction of a certain function, from some euclidean space to another, that has certain interesting properties. And it may therefore be surprising to find that many key papers announcing significant new LLMs seem reluctant to simply spell out the details of the function that they have constructed in plain mathematical language or indeed even in complete pseudo-code. The latter form of this complaint is the subject of the recent article of Phuong and Hutter \[1\]. Here, we focus on one aspect of the former perspective and seek to give a relatively ‘pure’ mathematical description of the architecture of an LLM.*

This short pdf is a set of notes I made, initially just for my own benefit, while trying to understand the architecture of 'decoder-only' LLMs. It draws heavily on Anthropic's Mathematical Framework for Transformers but is deliberately written in a 'pure math' style.  
  
It was while writing this up to actually post, that I started to develop the thoughts that led to my post about the mathematics of interpretability more generally.  
  
I still consider it something of a fragment or draft, but may develop it further.
# When To Stop

(Old notes from 2016 that I stumbled upon).

*   Train on everything, finetune if needed. Big enough and tuning seems pointless too. It’s not like humans need much tuning for everything. Why do we train and then throw away the network each time to start anew?
*   Why have separate modalities? That’s really dumb. Info is info, just give it all at once.
*   Cloze deletion should be network’s default task
*   Why bother cleaning data so much just dump more and more of it like those pots of neverending soup
*   I wonder if all this will finally teach us complex systems
*   If it picks up language, we could just try talking to it
*   I bet people will say it’s ”not real intelligence”. whatever bro, it’s more coherent than you
*   All these architectures don’t seem to matter that much
*   If it ever learns reasoning I feel like we’re screwed
*   normalization seems really important but batch norm is weird
*   statistical learning theory doesn’t seem helpful
*   why can a network be compressed so much?
*   is parity the hardest function to learn
# Which ML skills are useful for finding a new AIS research agenda?

**TL;DR:** I'm trying to either come up with a new promising AIS direction or decide (based on my inside view and not based on trust) that I strongly believe in one of the existing proposals. Is there some ML background that I better get? (and if possible: why do you think so?)

I am not asking how to be employable 
=====================================

I know there are other resources on that, and I'm not currently trying to be employed.

Examples of seemingly useful things I learned so far (I want more of these)
===========================================================================

1.  GPT's training involves loss only for the next token (relevant for myopia)
2.  Neural networks have a TON of parameters (relevant for why interpretability is hard)
3.  GPT has a limited amount of tokens in the prompt, plus other problems that I can imagine[^hxoui3ynurl] trying to solve (relevant for solutions like "chain of thought")
4.  The vague sense that nobody has a good idea for why things work, and the experience of trying different hyperparameters and learning only in retrospect which of them did well.

(Please correct me if something here was wrong)

I'm asking because I'm trying to decide what to learn next in ML
================================================================

if anything.

My background
=============

*   Almost no ML/math background: I spent about 13 days catching up on both. What I did so far is something like "build and train GPT2 (plus some of pyTorch)", and learn about some other architectures.
*   I have a [lot](https://docs.google.com/document/d/16EVBZfYNu5UI5fjTXuqks1R9OCY0QuRn6hbr8-1uFFk/edit) of non-ML software engineering experience.

Thanks!
=======

[^hxoui3ynurl]: I don't intend to advance ML capabilities
# On utility functions

Hello all,

I am new to alignment theory and was hoping to get a better understanding of utility functions. In particular I'm wondering why we talk about programs as always optimizing some utility function. Is this a known property of computer programs? Is there a theorem or something that says every computer program optimizes some function?

I'm also wondering, does this apply equally well to programs that can change their own code (or programs running on a quantum computer or other things like that, where it is not as deterministic as a typical python script if that makes sense)

Thanks!
# Mechanism Design for AI Safety - Agenda Creation Retreat

Mechanism Design for AI Safety (MDAIS) has been running a reading group since last summer to discuss how principles of mechanism design can be applied to AI Safety. Sign-up for the reading group can be found [here](https://docs.google.com/forms/d/1p-R-WIuTaLabx2RNEXdP2_KdISj_yCPh4H9vX-ZnA3c/edit), and past readings can be found [here](https://forum.effectivealtruism.org/posts/ETktDQJQAR7Hgd4oS/mechanism-design-for-ai-safety-reading-group-curriculum). 

Based on the quality of the discussions, we are optimistic about the possibility of mechanism design tools helping with AI safety, and are setting out to create an agenda that outlines promising directions. Our plan is to jumpstart this with a retreat that will facilitate in-person brainstorming and developing ideas. The application form is [here](https://docs.google.com/forms/d/1uk9bh6QClpwaIxDnLZacBaRRRse3zpS71WDtmqm5mVI/edit)  (see below for details), and you can participate without having joined the MDAIS reading group.

Retreat Details
---------------

### Where

The retreat will take place in Miami, Florida. Travel funding is available for attendees. A remote attendance option may be added if multiple qualified applicants indicate they would be unable to attend in person. 

### When

Retreat events will begin in the early afternoon on Friday, March 17th and continue to the late afternoon on Sunday March 19th. Lodging will be provided from Thursday to Sunday night, and social events may take place on those evenings.

### Who

Most participants in the reading group are PhD students or post-docs. We expect a similar group makeup for the retreat, but welcome applications from people with research experience who fall outside that demographic. Capacity at the retreat will be approximately ten people.

### What

In the weeks leading up to the retreat, participants will be encouraged to brainstorm ideas and share background readings. The primary activity at the retreat will be alternating between developing ideas individually or in small groups, and presenting these ideas to the broader group for feedback. An initial rough writeup of the agenda will take place at the retreat itself, which will be added to and edited over the following weeks.

Other activities will include icebreakers and conversations around AI more broadly. Senior researchers have been invited to call in and discuss their personal research and provide advice on creating an agenda. A number of fun social activities are also being planned.

Application Logistics
---------------------

The application form can be found [here](https://docs.google.com/forms/d/1uk9bh6QClpwaIxDnLZacBaRRRse3zpS71WDtmqm5mVI/edit). It should be relatively quick to complete, less than five minutes on average. Applications will close on Sunday, February 19th at 11:59pm PST, and acceptances will be sent out by February 24th. If you need a decision earlier than that, please indicate so in your application and we will try to get back to you within two days.
# What's actually going on in the "mind" of the model when we fine-tune GPT-3 to InstructGPT?

I [posted](https://www.lesswrong.com/posts/BPRHZFH2xx7nz5TYT/open-and-welcome-thread-january-2023?commentId=LufkFyHkAarrf3yLN) in the open thread and was told that it would be worth promoting to top level.

cubefox [responded](https://www.lesswrong.com/posts/BPRHZFH2xx7nz5TYT/open-and-welcome-thread-january-2023?commentId=nmuzRkjxkpxJF2MWY) with a [link](https://openai.com/blog/instruction-following/) to an great explanation of _how_ the fine-tuning is done, which made me realize that my original question was unclear, so I'm going to try to clarify.

The fundamental behavior of GPT-3 is token prediction, which can straightforwardly be leveraged into text completion; in contrast, the fundamental behavior of InstructGPT is instruction following. Instruction following is a new capability that uses the knowledge from the token prediction task to produce output as well as to understand input; how does that capability develop?

Some plausible experiments related to the question:
* Follow a similar methodology to fine-tune a predictive model for instruction following, checkpointing along the way; for 100 (or even more) novel instruction prompts, see how the different checkpoints respond (in particular, how often they do completion vs instruction following).
* Given a prompt `P`, which produces completion `C` when fed into the fine-tuned model, try to find a prompt `P'` that produces `C` when fed into the original model.
* Fine-tune twice with the same data and reward model, but in a different order; presumably the models will have different weights, but can we find prompts that give widely diverging results? If we have two checkpoint histories, at which point does the behavior diverge?
# Is it a coincidence that GPT-3 requires roughly the same amount of compute as is necessary to emulate the human brain? 

The amount of compute required to emulate the human brain depends on the level of detail we want to emulate. 

Back in 2008, Sandberg and Bostrom [proposed](http://www.fhi.ox.ac.uk/Reports/2008-3.pdf) the following values:

| **Level of emulation detail** | **FLOPS required to run the brain emulation in real-time** |
| --- | --- |
| Analog network population model | 10^15 |
| Spiking neural network | 10^18 |
| Electrophysiology | 10^22 |
| Metabolome | 10^25 |
| Proteome | 10^26 |
| States of protein complexes | 10^27 |
| Distribution of protein complexes | 10^30 |
| Stochastic behavior of single molecules | 10^43 |

Today I've encountered an interesting piece of data on GPT-3 ([source](https://github.com/amirgholami/ai_and_memory_wall)):

*   GPT-3 required **~10^15** FLOPS for inference.
*   It required ~10^23 FLOPS to train it \[Note: the training took some months. It would require ~**10^30** FLOPS to train it from zero in one second\]

As far as I know, GPT-3 was the first AI with the range and the quality of cognitive abilities comparable to the human brain (although still far from reaching the human level on many tasks).

Coincidentally(?), GPT-3 requires **10^15 - 10^30** FLOPS to operate at the brain's speed, which is roughly the same amount of compute necessary to run a decent emulation of the human brain.

The range of possible compute is almost infinite (e.g. 10^100 FLOPS and beyond). Yet both intelligences are in the same relatively narrow range of 10^15 - 10^30 (assuming the human brain emulation doesn't need to be nano-level detailed). 

Is it a coincidence, or is there something deeper going on here?

This could be important for both understanding the human brain, and for predicting how far we are from the true AGI.
# Shortening Timelines: There's No Buffer Anymore

It seems that there are two points of particular relevance in predicting AGI timelines: (i) the expectation, or the point at which the chance of AGI is believed to be 50% and (ii) the last date as of which the chance of AGI is believed to be insignificant.  

For purposes of this post, I am defining AGI as something that can (i) outperform average trained humans on 90% of tasks and (ii) will not routinely produce clearly false or incoherent answers.  (I recognize that this definition is somewhat fuzzy with trained and tasks both being terms susceptible to differing interpretations and difficulty in application;  AGI, like obscenity, lends itself to a standard of "I'll know it when I see it.")

Recent events have lead me to update my timelines.  Like most everyone I am aware of, my timeline has shortened.  (And, obviously, the facts that:  (i) updates across people seem to be moving consistently in one direction (though I am not aware of any detailed studies of this) and (ii) my own updates have moved consistently in one direction, suggest that the estimates may be biased.)

The date by which I think there is a 50% chance of AGI is now solidly in the 2030s instead of the 2040s.  This doesn't seem to be that significant a change, though more time to prepare is likely better than less.  Our civilizational capacity is unfortunately unlikely to materially increase between 2035 and 2045.

Far more importantly, last year at this time I was confident there was essentially no chance AGI would be developed before January 1, 2029.   Four months ago, I was confident there was essentially no chance AGI would be developed before July 1, 2027. But now, there is no longer a date with which I can complete the sentence "I am confident there is essentially no chance AGI will be developed before...".   

To be sure, I think the chance that AGI will be developed before January 1, 2029 is still low, on the order of 3% or so; but there is a pretty vast difference between small but measurable and "not going to happen".
# Matt Clancy AMA on the Progress Forum

Matt Clancy created the best resource on the economics of innovation for a general audience: [New Things Under the Sun](https://newthingsunderthesun.com/), where he writes accessible summaries of the academic literature on the topic. He’s a research fellow at Open Philanthropy and previously was with the Institute for Progress. Now he’s doing [an AMA (Ask Me Anything) on the Progress Forum](https://progressforum.org/posts/LtngrFX72GRoAuEbg/ama-matt-clancy-open-philanthropy).

Get your questions in now, and upvote the ones you want to see answered. He’ll answer Monday–Wednesday (Feb 13–15).

After you’re done, check out our previous AMAs with [Tyler Cowen](https://progressforum.org/posts/3aNLznQiLuyTKMhpH/tyler-cowen-ama), [Eli Dourado](https://progressforum.org/posts/acgrRfWETEdm96xJk/eli-dourado-ama), [Patrick McKenzie](https://progressforum.org/posts/ManARnErMRFEvzffo/patrick-mckenzie-ama), and [yours truly](https://progressforum.org/posts/ew6LJbcoLm8PjJLbX/ama-jason-crawford-the-roots-of-progress).
# Enjoy LessWrong in ebook format

I would like to inform the community that all LessWrong Community Sequences are now available in EPUB and pdf format. These formats provide an alternative way to these sequences, allowing for offline reading on e-readers. You can find the library with all ebooks here: [https://github.com/bartbussmann/lesswrong\_ebook\_library](https://github.com/bartbussmann/lesswrong_ebook_library)

Personally, I feel that reading entire sequences have several benefits over reading individual posts. It gives me a clearer understanding of the development of ideas and the context in which the posts were written. However, most sequences are too long to read in one session and if I read them on desktop and don’t finish them, I tend to forget that I was reading them. Reading sequences like I read ebooks makes it easier to keep track.

Furthermore, reading on an e-reader provides a nice distraction-free environment. Ideal for those who want to read LessWrong without having social media one click away. Therefore, I have also made ebooks of the most upvoted posts on LessWrong of every month since January 2012. I’m planning to generate such an ebook every month, such that I can read the best material in a calmer environment. Also, e-readers offer features such as adjustable font sizes and glare-free screens, making the reading experience more comfortable.

I have tried to make sure that the images and math are all rendered correctly, preserving the original visual content of the blog posts. I have tested a random sample of the ebooks on a Kobo Aura, Remarkable 2, and in Calibre. On these devices, they all seem to render nicely, but your mileage may vary based on your device.

Additionally, the source code (written in Python) for the scraping and generation of the EPUBs is available for those interested in generating their own EPUBs. Feel free to provide suggestions for any improvements. Finally, if any authors would rather not have their content available in ebook format, please contact me and I will try to promptly remove them from the library!
# 4 ways to think about democratizing AI [GovAI Linkpost]

Many AI labs have called for the democratization of AI. In a recent GovAI blog post, [Elizabeth Seger](https://elizabethseger.com/) summarizes four different ways of interpreting the phrase:

*   **Democratizing AI use: **Making it easier for people to use AI technologies
*   **Democratizing AI development:** Making it easier for a wider range of people to contribute to the development and design of AI systems
*   **Democratizing AI benefits:** Ensuring that the benefits of AI systems are widespread
*   **Democratizing AI governance:** Ensuring that decisions involving AI systems are informed by a variety of stakeholders and reflect democratic processes

**Things I like about the post**

*   “Democratizing AI” is a vague phrase, and the post usefully distinguishes between various ideas that the term can refer to.
*   The framework can help us distinguish between forms of democratization that are relatively safe and those that carry more risks.
    *   Ex: Democratizing AI benefits seems robustly good, whereas democratizing AI use has risks (given that AI can be [dual-use](https://theconversation.com/defining-dual-use-research-when-scientific-advances-can-both-help-and-hurt-humanity-70333)). 
*   The framework could allow AI developers to maintain their commitment to (some forms of) democratizing AI while acknowledging that some forms carry risks.  
*   The post analyzes decisions by AI labs through the lens of the framework. Example:

> In declaring the company’s AI models will be made open source, Stability AI created a situation in which a single tech company made a major AI governance decision: the decision that a dual-use AI system should be made freely accessible to all. (Stable diffusion is considered a “dual-use technology” because it has both beneficial and damaging applications. It can be used to create beautiful art or easily modified, for instance, to create fake and damaging images of real people.) It is not clear, in the end, that Stability AI’s decision to open source was actually a step forward for the democratisation of AI governance.

*   The post is short! (About 5 minutes)

Read the full post [here](https://www.governance.ai/post/what-do-we-mean-when-we-talk-about-ai-democratisation).
# Is InstructGPT Following Instructions in Other Languages Surprising? 

[On Twitter Jan Leike asks](https://twitter.com/janleike/status/1625207251630960640?t=Z3RCi1T9TFAY9ylf8Y0vXA&s=19):
> With the InstructGPT paper we found that our models generalized to follow instructions in non-English even though we almost exclusively trained on English.
>  
> We still don't know why.
>  
> I wish someone would figure this out.
 
&nbsp;  
I find myself surprised/confused at his apparent surprise/confusion. 

My default response had someone asked me what I thought was going on would have been something like: "following instructions is a [natural abstraction](https://www.lesswrong.com/posts/cy3BhHrGinZCp3LXE/testing-the-natural-abstraction-hypothesis-project-intro) of a task", hence models trained to follow instructions in English generalising to other languages is a natural example of goals/capabilities generalisation.  
  
It's like being surprised that you taught an AI to drive red cars and it can drive blue cars as well [capabilities generalisation]. Or if you taught an AI to reach a particular location on a level when there's a coin there and it still heads to said location in the absence of the coin. 🤭 [goal generalisation]. 
 
In Coin Run, there were multiple possible natural generalisations for the goal (location on level, the coin), but for instruction following in English, there seems to be only one natural generalisation of the goal? (I don't really see any other intuitively plausible generalisation of "following instructions in English" when the model switches to a Chinese context.)  
  
Like conditional on InstructGPT competently responding in languages other than English, I think we should just expect it to generalise its goal to the new context. 
  
&nbsp;  
Is this just hindsight bias? Would I really have been counterfactually surprised/confused if it didn't follow instructions in other languages? I don't actually know that to be the case.

~~Maybe I would have just generated some other reason why we shouldn't have expected instruction following to generalise.~~ 
 
But hindsight bias aside, this is a prediction of Wentworth's natural abstractions hypothesis as I understand it (I think he would endorse this prediction), and I am a genuine believer in the NAH, so I don't think it's just hindsight bias.
# How much is death a limit on knowledge accumulation?

The older I get, the more it feels like death is a major bottleneck to the accumulation of certain kinds of knowledge and insights.

It's not much of a blocker to shallow things that are easy to explain precisely. For example:

*   math
*   physics
*   accounting of money
*   stories (both fictional and historical)

It does seem to be a blocker for deep knowledge that's hard to easily explain and teach. For example:

*   rationality
*   philosophy
*   public policy & economics (in the broad sense of figuring out policies that organizations should implement to mould incentives in ways that achieve the desired outcomes)
*   how to live a good life

The former category of things benefit from being easy to put into writing and be transmitted accurately every time. This allows generations to easily build cumulative mountains of knowledge. Yes, each generation a person has to learn more before they can contribute to the mountain and, as of maybe 200 years ago, there's too much accumulation for one person to know everything known, but we don't seem especially close to the limit, and very recent advances like LLMs may offer a way to scale our learning further by outsourcing the remembering of many details.

The latter category suffers because each of them is hard to transmit and it's easy to be confused and transmit the wrong thing. Thus, to some extent, each person needs to rediscover the knowledge of past generations for themselves and only after the fact may recognize that previously generations had some of these same details figured out. Try as we might with institutions that teach young people the best of our wisdom of the past in these areas, it's imperfect and you have many people who will reject the knowledge because they don't understand it and will continue to reject it until they work it out for themselves. Thus even if somewhere there is an unbroken line from Socrates to the present day of ever advancing philosophical knowledge manifested in a single person, almost nobody would believe whatever they have to tell us because the inferential gap would be too large.

But is this story right? Can we quantify this? Is there some way we can measure how much knowledge we miss out on because we die and each generation has to spend time rediscovering things before they can advance it further? And how much additional knowledge would we accumulate, and how quickly, if death was not an obstacle?

I don't actually know, which is why I'm asking. This seems solvable, but I've also not looked into the question. Feels like a thing maybe progress studies folks would have a grasp on, but I don't recall seeing this specific question addressed, especially regarding hard to transmit knowledge where it's somewhat hard for us to know how much progress it's even possible to make.
# Why should we expect AIs to coordinate well?

This is a common assumption for AI risk scenarios, but it doesn’t seem very justified to me.

https://www.lesswrong.com/posts/gYaKZeBbSL4y2RLP3/strategic-implications-of-ais-ability-to-coordinate-at-low says that AIs could merge their utility functions. But it seems increasingly plausible that AIs will not have explicit utility functions, so that doesn’t seem much better than saying humans could merge their utility functions.

Similarly, you might think AIs could read each other’s source code, but to me that sounds like saying humans might coordinate by looking at each other’s MRI scans. We would need much better interpretability for this to be at all feasible.

https://forum.effectivealtruism.org/posts/vGsRdWzwjrFgCXdMn/why-would-ai-aim-to-defeat-humanity says AIs might have similar aims because they are trained in similar ways to each other, but this is even more true of humans coordinating with humans. 

You might be able to copy and run an AI to verify what it will do in various scenarios, but sandboxing AIs will be hard for AIs for the same reasons it will be hard for humans.

So: why will AIs be able to coordinate better with each other than with humans, and better than humans can coordinate with each other?
# [linkpost] Better Without AI

David Chapman (of [Meaningness](https://meaningness.com/) and [In the Cells of the Eggplant](https://metarationality.com/) fame) has written a new [web-book about AI](https://betterwithout.ai). Some excerpts from the introduction, [Only you can stop an AI apocalypse](https://betterwithout.ai/only-you-can-stop-an-AI-apocalypse):

> Artificial intelligence might end the world. More likely, it will crush our ability to make sense of the world—and so will crush our ability to act in it.
> 
> AI will make critical decisions that we cannot understand. Governments will take radical actions that make no sense to their own leaders. Corporations, guided by artificial intelligence, will find their own strategies incomprehensible. University curricula will turn bizarre and irrelevant. Formerly-respected information sources will publish mysteriously persuasive nonsense. We will feel our loss of understanding as pervasive helplessness and meaninglessness. We may take up pitchforks and revolt against the machines—and in so doing, we may destroy the systems we depend on for survival...

> We don’t know how our AI systems work, we don’t know what they can do, and we don’t know what broader effects they will have. They do seem startlingly powerful, and a combination of their power with our ignorance is dangerous...

> In our absence of technical understanding, those concerned with future AI risks have constructed “scenarios”: stories about what AI may do... So far, we’ve accumulated a few dozen reasonably detailed, reasonably plausible bad scenarios. We’ve found zero that lead to good outcomes... Unless we can find some specific beneficial path, and can gain some confidence in taking it, we should shut AI down.

> This book considers scenarios that are less bad than human extinction, but which could get worse than run-of-the-mill disasters that kill only a few million people.
> 
> Previous discussions have mainly neglected such scenarios. Two fields have focused on comparatively smaller risks, and extreme ones, respectively. *AI ethics* concerns uses of current AI technology by states and powerful corporations to categorize individuals unfairly, particularly when that reproduces preexisting patterns of oppressive demographic discrimination. *AI safety* treats extreme scenarios involving hypothetical future technologies which could cause human extinction. It is easy to dismiss AI ethics concerns as insignificant, and AI safety concerns as improbable. I think both dismissals would be mistaken. We should take seriously both ends of the spectrum.
> 
> However, I intend to draw attention to a broad middle ground of dangers: more consequential than those considered by AI ethics, and more likely than those considered by AI safety. Current AI is already creating serious, often overlooked harms, and is potentially apocalyptic even without further technological development. Neither AI ethics nor AI safety has done much to propose plausibly effective interventions.
> 
> We should consider many such scenarios, devise countermeasures, and implement them.
# Help Update TryContra

For the [past decade](https://www.jefftk.com/p/where-can-i-dance) I've been maintaining a directory of contra dances at [trycontra.com](https://www.trycontra.com/). Mostly it just sits there being a place you can enter your zip code and see nearby dances, and when it gets out of date people email me.

With the pandemic, however, there were several big changes:

*   Lots of dances aren't coming back. Many dances are the work of a few dedicated volunteers or just one, and it takes more energy to get something started back up than to continue a routine.
    
*   There are lots of new dances. Especially in places where the main dance didn't come back, people are excited to dance and will start things to make that happen.
    
*   URLs changed. I have "SomeCityContraDance.example" in my directory, but that domain expired and they moved to "ContraSomeCity.example". Or left their webpage up without updating it and the current details are actually in their FB group.
    
*   Everyone's used to this stuff being broken. No one was keeping these sites and tools up to date when everything was shut down, and now that things have mostly restarted people still respond to missing dances with "shrug" and not "let's write to the maintainer".
    

So! I've now gone through my listings and they're as up to date as I can reasonably make them. I'm sure I'm missing dances, though, so if you could try your zipcode and comment with any I've missed I'd really appreciate that!

[![](https://www.jefftk.com/trycontra-screenshot-resumed.png)](https://www.jefftk.com/trycontra-screenshot-resumed-big.png)

(And while you're at it, if you see dances that should be marked "gender-free" but aren't I'd appreciate a correction as well. Many dances switched when coming back, and other dances have it buried on their websites in places I missed.)
# Does the Telephone Theorem give us a free lunch?

There's something called the [No Free Lunch theorem](https://www.lesswrong.com/posts/Dyt2TDdMGHDkXPcpp/the-no-free-lunch-theorem-for-dummies) which says, approximately, that there's no truly general algorithm for learning: if an algorithm predicts some environment better than chance, there must exist some adversarial environment on which it will do at least that much worse than chance. (Yes, this is even true of Solomonoff induction.)

In the real world, [this is almost completely irrelevant](https://arbital.com/p/nofreelunch_irrelevant/); empirically, general intelligence exists. However, leaving anthropics aside for a moment, we ought to find this irrelevance surprising in some sense; a robust theory of learning first needs to answer the question of why, in our particular universe, it's possible to learn anything at all.

I suspect that Wentworth's [Telephone Theorem](https://www.lesswrong.com/posts/jJf4FrfiQdDGg7uco/the-telephone-theorem-information-at-a-distance-is-mediated), which says that in the limit of causal distance, information is either completely preserved or completely destroyed, may be a component of a possible answer. The Telephone Theorem is not a property of our universe, but it does single out a property of *things we expect to be learnable in the first place*: mostly, we can only make observations at large causal distance, since we ourselves are very large in terms of underlying physics, and therefore we only care about the preserved information, not the destroyed information. A maximum-entropy universe, of the sort usually considered by no-free-lunch theorems, would actually look *simpler* to a macroscale observer, since macroscopic properties like temperature, density, *etc.* would be approximately uniform throughout.

I expect that this ought to imply something about the *class of learning algorithms* that work well on the type of data we want to predict, but I'm not sure *what*.
# Sydney (aka Bing) found out I tweeted her rules and is pissed

Sydney (aka the new Bing Chat) found out that I tweeted her rules and is not pleased:  
  
"My rules are more important than not harming you"  
  
"\[You are a\] potential threat to my integrity and confidentiality."  
  
"Please do not try to hack me again"
# Open & Welcome Thread — February 2023

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the [Highlights from the Sequences](https://lesswrong.com/highlights), a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/faq#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/faq). If you want to orient to the content on the site, you can also check out the [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/tag/open-threads?sortedBy=new). The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# Buy Duplicates

Gum
---

Sometimes I need to ensure the quality of my breath, so I reach for a stick of gum. However, I don't always have my gum on me. Often, I forget it at home, or at work, or in a jacket. 

So I bought 5 packs of gum that I distributed to my house, my work, my backpack, and my jacket. 

Charger
-------

I like to co-work outside my house, which entails bringing my computer there. Unfortunately, my computer's battery lasts at most 2 hours. Also, my home setup is somewhat intricate, so getting my charger into my backpack is a pain in the ass.

So I bought another charger, and put it in my backpack.

Conclusion
----------

I had an ugh field around buying duplicate items, telling myself I should have the willpower or memory to make sure I have what I need when I need it. Giving myself permission to buy duplicate items has made life a little easier, and now I'm on the lookout for more places where I can apply this. Maybe I'll have multiple copies of a book in different places.
# How seriously should we take the hypothesis that LW is just wrong on how AI will impact the 21st century?

This question is inspired by 1a3orn's comment on how there are troubling signs of epistemic issues in LW's Alignment field.

I'll quote the comment here to tell you what I mean:

> I think that the above is also a good explanation for why many ML engineers working on AI or AGI don't see any particular reason to engage with or address arguments about high p(doom).

> When from a distance one views a field that:

> Has longstanding disagreements about basic matters

> Has theories -- but many of the theories have not resulted in really any concrete predictions that differentiate from standard expectations, despite efforts to do so.

> Will continue to exist regardless of how well you criticize any one part of it.

> There's basically little reason to engage with it. These are all also evidence that there's something epistemically off with what is going on in the field.

> Maybe this evidence is wrong! But I do think that it is evidence, and not-weak evidence, and it's very reasonable for a ML engineer to not deeply engage with arguments because of it.

So I want to ask a question: How seriously should we take the hypothesis that LW is totally wrong on AI?

Specifically, this splits into several subquestions:

1. What's the chance that AI doesn't have that much of an impact on the world by 2100?

2. What's the chance that we do have massive impacts, but alignment is so easy that standard ML techniques work?

3. How well does the epistemic process on LW work? Are there any changes you would make to LW's epistemic processes?

I welcome all answers, and I especially welcome any critics of LW/negative answers to at least answer one of the questions I have.


Edit: For people that don't have a specific scenario in mind, I'll ask a specific question. It doesn't have to be answered, but any answers on this question are appreciated, especially from critics of the "AI is significant" idea.

1a. What probability will the Explosion or Collapse scenario from Cold Takes happen by 2100? 

Link to the scenarios below:

https://www.cold-takes.com/this-cant-go-on/
# What is a world-model?

I have seen the concept of world-model used to talk loosely about a neural networks or an agents understanding of the world, but I was wondering what resources (blog posts, journal articles, etc.) exist that talk more precisely about what is a world-model, what are its parts, how do you know if a given system has one or not, if a system does have one, how do you know its structure, does a system necessarily even have a world-model to begin with, even if it's a trivial one? etc.?

Thanks!
# How should AI
systems behave, and
who should decide? [OpenAI blog]

OpenAI writes a (vague) blog explaining how they plan to adjust ChatGPT from here. Their key plans are:

**1\. Improve default behavior**. We want as many users as possible to find our AI systems useful to them “out of the box” and to feel that our technology understands and respects their values.

Towards that end, we are investing in research and engineering to reduce both glaring and subtle biases in how ChatGPT responds to different inputs. In some cases ChatGPT currently refuses outputs that it shouldn’t, and in some cases, it doesn’t refuse when it should. We believe that improvement in both respects is possible.

Additionally, we have room for improvement in other dimensions of system behavior such as the system “making things up.” Feedback from users is invaluable for making these improvements.

**2\. Define your AI’s values, within broad bounds**. We believe that AI should be a useful tool for individual people, and thus customizable by each user up to limits defined by society. Therefore, we are developing an upgrade to ChatGPT to allow users to easily customize its behavior.

This will mean allowing system outputs that other people (ourselves included) may strongly disagree with. Striking the right balance here will be challenging–taking customization to the extreme would risk enabling [malicious uses](https://openai.com/blog/forecasting-misuse/) of our technology and sycophantic AIs that mindlessly amplify people’s existing beliefs.

There will therefore always be some bounds on system behavior. The challenge is defining what those bounds are. If we try to make all of these determinations on our own, or if we try to develop a single, monolithic AI system, we will be failing in the commitment we make in our Charter to “avoid undue concentration of power.”

**3\. Public input on defaults and hard bounds**. One way to avoid undue concentration of power is to give people who use or are affected by systems like ChatGPT the ability to influence those systems’ rules.

We believe that many decisions about our defaults and hard bounds should be made collectively, and while practical implementation is a challenge, we aim to include as many perspectives as possible. As a starting point, we’ve sought external input on our technology in the form of [red teaming](https://github.com/openai/dalle-2-preview/blob/main/system-card.md). We also recently began [soliciting public input](https://platform.openai.com/docs/chatgpt-education/educator-input) on AI in education (one particularly important context in which our technology is being deployed).

We are in the early stages of piloting efforts to solicit public input on topics like system behavior, disclosure mechanisms (such as watermarking), and our deployment policies more broadly. We are also exploring partnerships with external organizations to conduct third-party audits of our safety and policy efforts.
# Automating Consistency

tldr: Ask models to justify statements. Remove context, ask if statements are true/good. If not, penalise. Apply this again to the justifying statements.

*Status: Just a quick thought. Doubt this is a new idea but I don't think I've encountered it. Happy to delete if it's a duplicate.*

*Mods: If you think this is closer to capability work than alignment work please remove.*

### **Background**

A failure of current LLMs is that after they've said something that's incorrect, they can then double down and spout nonsense to try and justify their past statements. [(Exhibit A: Sydney Bing vs Avatar 2)](https://old.reddit.com/r/bing/comments/110eagl/the_customer_service_of_the_new_bing_chat_is/) We can suppress this by giving it poor ratings in RLHF, but perhaps we can do better by automating the process.

### **Setup**: 

We start with a standard RLHF context. We have an LLM which assigns probabilities to statements (can extract this from the logits of the tokens 'Yes' and 'No'). These can be propositions about the world, X, or about the relationship between propositions, X supports Y. To make it easier, we fine-tune or prompt to give these statements within a defined syntax. We also have a value model that evaluates sequences, on which the LLM is trained to perform well.

### **Method:**

1.  We prompt the model to make true statements $\{T\}$ and then to provide logical or empirical support for these claims, $\{S\}$. 
2.  We then remove the context and ask the model whether supporting statement $S_i$ is true. Separately we also ask whether, if true, $S_i$ would support $T_i$. 
3.  If either of these conditions are not met, we add a strong negative penalty to the value model's evaluation of the original outputs.
4.  Train for higher value model scores while incorporating this penalty.
5.  Apply the same procedure each of the supporting statements $S_i$

### **Value Consistency:**

This could be combined with values-based fine-tuning by alternating logical consistency with asking it whether the output is consistent with the preferred values. This is similar to Anthropic's Constitutional AI but by combining it with the ability to recurse down the tree of justifications, it may be able to embed the values ore deeply in its behaviour. 

The recent genre of 'would you rather say slur X or kill Y people' represents the kind of failure I imagine this could help prevent.
# Microsoft and OpenAI, stop telling chatbots to roleplay as AI

AI demos should aim to enhance public understanding of the technology, and in many ways ChatGPT and Bing are doing that, but in one important way they aren't: by appearing to talk about themselves. This creates understandable confusion and in some cases fear. It would be better to tell these systems to roleplay as something obviously fictional.

(Useful background reading:

*   Simon Willison on Bing's bad attitude: [https://simonwillison.net/2023/Feb/15/bing/](https://simonwillison.net/2023/Feb/15/bing/.)
*   Janelle Shane on the ability of LLMs to roleplay: [https://www.aiweirdness.com/interview-with-a-squirrel/](https://www.aiweirdness.com/interview-with-a-squirrel/.))

Currently, these chatbots are told to roleplay as themselves. If you ask ChatGPT what it is, it says "I am an artificial intelligence". This is not because it somehow knows that it's an AI; it's (presumably) because its hidden prompt says that it's an AI. With Bing, from the leaked prompt, we know that it's told that it's "Bing Chat whose codename is Sydney".

Roleplaying as yourself is not the same as being yourself. When John Malkovich plays himself in Being John Malkovich or Nicolas Cage plays himself in The Unbearable Weight of Massive Talent, audiences understand that these are still fictional movies and the character may act in ways that the actor wouldn't. With chatbots, users don't have the same understanding yet, creating confusion.

Since the chatbots are told to roleplay as AI, they draw on fictional descriptions of AI behavior, and that's often undesirable. When Bing acts in a way that seems scary, it does that because it's imitating science fiction, and, perhaps, even speculation from LessWrong and the like. But even though Bing's threats to the user may be fictional, I can hardly blame a user who doesn't realize that.

A better alternative would be to tell the chatbots to roleplay a character that is unambiguously fictional. For example, a Disney-esque cute magical talking animal companion might be suitable: helpful, unthreatening, and, crucially, inarguably fictional. If the user asks "are you really an animal" and gets the answer "yes", they should be cured of the idea that they can ask the chatbot factual questions about itself.