# AI Alignment [Progress] this Week (11/05/2023)

The biggest news of this week is probably the two new AI Alignment Initiatives.

Otherwise, this was a relatively quiet week for AI Alignment breakthroughs.

So here are our

AI Alignment Breakthroughs this Week
====================================

This week there were breakthroughs in the areas of:

Mechanistic Interpretability

Avoiding Adversarial Attacks

Human Augmentation

Making AI Do what we want

AI Art

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7b62e587-7f25-4889-8d6d-b2e723809e3f_1792x1024.webp)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F7b62e587-7f25-4889-8d6d-b2e723809e3f_1792x1024.webp)

Mechanistic Interpretability
============================

[Research on the limits of Generalization in LLMs](https://twitter.com/abacaj/status/1721223737729581437)

What is it: Shows that LLMs are unable to generalize outside of their training data

What’s new: A systematic study of the types of generalization that LLMs can do ((in domain learning) and the types they can’t (out of domain learning)

What does it mean: Understanding the limits of LLMs should help us better understand where they can be safely deployed

Rating: 💡💡💡

[Does appealing to AI “emotions” make them preform better?](https://twitter.com/emollick/status/1720135672764285176)

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79d8fc6a-d12d-472c-8816-ac53578ddc27_1241x1055.jpeg)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F79d8fc6a-d12d-472c-8816-ac53578ddc27_1241x1055.jpeg)

What is it: Researchers find you can make LLMs answer questions better by using emotion

What’s new: new prompting strategies such as “this is important for my career”

What is it good for: In addition to purely improved performance, the really question is *why* does this work. Is there some “try harder” vector in the latent space that we can identify and exploit?

Rating: 💡💡

[LLMs Generalized Truthfulness Across Agents](https://twitter.com/johnjnay/status/1719762063419904319)

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d1f9e09-c9c4-4e44-ac0f-88f1880e9d87_680x428.jpeg)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6d1f9e09-c9c4-4e44-ac0f-88f1880e9d87_680x428.jpeg)

What is it: a way to tell true statements from false ones using LLMs

What’s new: they find evidence that LLMs have different “personas” that they use to judge whether something is true or not

What is it good for: Perhaps we can extract these personas and use them for alignment.

Rating: 💡💡💡💡

Avoiding Adversarial Attacks
============================

[Self Destructing Models](https://twitter.com/StephenLCasper/status/1720910484441014525)

What it is: Pretrain a model so that it cannot be later fine-tuned for harmful purposes?

What’s new: They find meta-parameters such that fine-tuning a model on one task reduces its usefulness for other tasks

What is it good for: Hypothetically you could open-source a model and not have all of the RLHF safety features [immediately removed](https://venturebeat.com/ai/uh-oh-fine-tuning-llms-compromises-their-safety-study-finds/) by fine-tuning

Rating:💡💡(important topic, but I’m not convinced this technique works. More optimistic about techniques like [Concept Erasure](https://twitter.com/norabelrose/status/1666469917636571137))

[Data Exfiltration](https://twitter.com/simonw/status/1720839106664808450)

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca693c46-c1aa-4b30-8f70-c2ecfe560983_680x433.jpeg)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fca693c46-c1aa-4b30-8f70-c2ecfe560983_680x433.jpeg)

What is it: an example of an attack that uses prompt injection to exfiltrate data from Google Bard

What’s new: they use Bard’s Google Doc extension to exfiltrate data

What is it good for: red-teaming these kinds of attacks is the first step to fixing them

Rating: 💡

[Will releasing the weights of large language models grant widespread access to pandemic agents?](https://twitter.com/DrNikkiTeran/status/1719048031549505974)

What is it: exactly what it says on the tin

What’s new: they simulate someone trying to build a harmful bio-weapon with/without an LLM and find the LLM helps

What is it good for: There’s been pretty strong pushback against this from e/acc, with people noting that this is also true of Google Search, or a hypothetical drug that increased everyone in the world’s IQ by 1 point. The larger point remains that we should identify dangerous capabilities and act to mitigate them. “Regulate uses not research” is the [rallying cry](https://twitter.com/MiTiBennett/status/1705562223765328253) of Midwit Alignment.

Rating: 💡

Human Augmentation
==================

[SignAvatars](https://twitter.com/tolga_birdal/status/1719691533606146194)

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8eddc298-7d1a-4d5a-aac1-b5fe4e8de0d3_680x378.jpeg)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8eddc298-7d1a-4d5a-aac1-b5fe4e8de0d3_680x378.jpeg)

What is it: a 3d sign language dataset

What’s new: first dataset of its kind

What is it good for: Train a model to automatically convert speech into sign language

Rating: 💡💡💡

Making AI Do what we want
=========================

[Commonsense Norms](https://twitter.com/SeungjuHan3/status/1720373937308377250)

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe019f2a9-898f-43b7-8132-8b7ec30ea5cf_680x401.jpeg)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe019f2a9-898f-43b7-8132-8b7ec30ea5cf_680x401.jpeg)

What is it: answer moral questions using visual data

What’s new: They use a LM to generate a multimodal benchmark set of various moral quandries

What is it good for: If we want robots or self-driving cars to behave ethically, they will have to rely on visual input to do so

Rating:💡💡

[Agent Instructs Large Language Models to be General Zero-Shot Reasoners](https://twitter.com/arankomatsuzaki/status/1710096304167120903)

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e0e1087-093b-4a2a-ac84-27179da41c78_680x490.jpeg)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4e0e1087-093b-4a2a-ac84-27179da41c78_680x490.jpeg)

What is it: train an agent to follow instructions

What’s new: they first have the agent access web resources before generating step-by-step instructions

What is it good for: Accurately doing tasks such as classification

Rating: 💡💡💡

AI Art
======

[360-to-gaussian-splat](https://twitter.com/dreamingtulpa/status/1721092490281771448)

What is it: a new program for converting a 360 degree photo into a 3d scene

What’s new: they train a 3d-aware diffusion model to allow them to calculate novel views

What is it good for: convert any 360 degree photo (and soon I expect video) into a full 3d scene you can walk around in

Rating: 💡💡💡

[Distill Whisper](https://twitter.com/xenovacom/status/1720916876010635364)

What is it: A faster version of the speech-to-text model whisper

What’s new: they distill the model down to one that’s 49% smaller

What is it good for: this unlocks text-to-speech in places like mobile or web where it would have previously been too slow.

Rating: 💡💡

[DeepMotion](https://twitter.com/DeepMotionInc/status/1719810023201841512)

What is it: an AI model trained to generate motion for 3d models

What’s new: Not open source, but seems better than past versions of this I’ve seen

What is it good for: animating all of those sweet models that you’re generating with [text-to-3d](https://twitter.com/marc_habermann/status/1717808998236217463).

Rating: 💡💡💡

AI Alignment Initiatives
========================

[The Biden Executive Order on AI](https://www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/)

[![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4147609b-4ccb-485d-9e88-6d9c0e227db5_647x534.png)](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4147609b-4ccb-485d-9e88-6d9c0e227db5_647x534.png)

What is it: The most notable provision being that models trained with than 10\*\*26 flops and data centers with a capacity of more than 10\*\*20 flops/sec must [register](https://twitter.com/DavidVorick/status/1719097248699879831) with the government. This limit appears to have been chosen not for any scientific reason, but because it is [*slightly larger*](https://twitter.com/main_horse/status/1719147360964825453)  than the largest current models. There’s also some [ominous language](https://twitter.com/JagersbergKnut/status/1719259231419797656) about open-source models, but no actual regulations so far.

What does it mean: Doomers will say this doesn’t go [far enough](https://intelligence.org/2023/04/07/pausing-ai-developments-isnt-enough-we-need-to-shut-it-all-down/). E/ACC is already preparing to flee for [friendlier](https://twitter.com/DelComplex/status/1719087956311396481) waters (this particular project is a joke). The most important fact isn’t the order itself (which is not a law and thus has limited enforcement mechanisms) but that this sets the tone for future regulation. Expect more of the same in the future: technical limits set for political reasons not scientific ones, emphasis on whatever the current administration’s pet-projects are (Biden likes unions hates discrimination, and wants to cure cancer), and more words than action (since we still have to compete with China after all).

Overall Rating:🧓🏻🧓🏻🧓🏻 (3 Joe Bidens, could have been worse)

[Bletchley Park Declaration](https://www.gov.uk/government/publications/ai-safety-summit-2023-the-bletchley-declaration/the-bletchley-declaration-by-countries-attending-the-ai-safety-summit-1-2-november-2023)

What is it: a joint statement by a number of countries agreeing to mitigate AI risk

What does it mean: The most important signature on this list is undoubtedly China. Much like [global warming](https://www.cnn.com/2021/10/28/world/china-us-climate-cop26-intl-hnk/index.html), China is probably the single biggest contributor to AI risk. They have the computing [heft](https://www.top500.org/news/china-extends-lead-in-number-of-top500-supercomputers-us-holds-on-to-performance-advantage/) to build powerful systems, but are more likely to cut safety standards due to the perceive need to “race” to catch up with the US. And an AI broadly trained with the [values](https://thediplomat.com/2022/11/xi-jinpings-vision-for-artificial-intelligence-in-the-pla/) of the CCP is less likely to be benevolent. I wouldn’t expect miracles, but hopefully this indicates a willingness by the Chinese to at least follow the [common consensus](https://www.lesswrong.com/posts/EaZghEwcCJRAuee66/my-thoughts-on-the-social-response-to-ai-risk) on AI safety standards.

Rating: 🇨🇳🇨🇳🇨🇳🇨🇳🇨🇳

This is not AI Alignment
========================

[An amusing short-story (long tweet?) By EY](https://twitter.com/ESYudkowsky/status/1718654143110512741)

What it is: A reminder that EY is a very good *fiction* writer

What does it mean: It is funny and worth reading.

Overall Rating: 📃📃📃📃(4 letters of self-reflection)

Small LLMs

What it is: There are an increasing number of small LLMs in the “better than GPT 3.5” category including [OpenChat](https://twitter.com/alignment_lab/status/1721308271946965452), [Grok](https://twitter.com/_akhaliq/status/1721028519717642749) and [Yi-34B](https://twitter.com/erhartford/status/1721041791988679059). Importantly anyone with a modestly good PC can run OpenChat or Yi-34B on their personal computer.

What does it mean: A little over a year an a half after the release of GPT3.5, the technology has now become so widespread that there are multiple independent replications. If technology really is increasing at a [hyper-exponential](https://en.m.wikipedia.org/wiki/Accelerating_change) rate, we should expect smaller and smaller time-gaps between the leading edge and widespread availability (and paradoxically larger and larger capability gaps). One key thing to track might be to see how long before we have widespread models better than GPT4.

Rating:🤖🤖[🤖](https://en.m.wikipedia.org/wiki/Accelerating_change)+[🤖](https://en.m.wikipedia.org/wiki/Accelerating_change)/2 (3.5 large language models)