Harley-ml/lesswrong huggingface dataset:
# Contra Mask Status

As we get into summer and covid levels continue to [fall](https://www.mwra.com/biobot/biobotdata.htm), a lot of contra dances are rethinking their approaches to masking. At this point contra dances are the only places I go that require masks, though they're also the closest I get to large numbers of people.

Prompted by an initial search [Harris](https://chromamine.com/) shared with me, I used [TryContra](https://www.trycontra.com/) to get a list of all 39 New England dances and manually checked their mask rules. I found a bit under half of dances require them ([sheet](https://docs.google.com/spreadsheets/d/1_ltGjBZy_KWWUVO-x3u_QzYVsjVGhO2mZu4iKEYNKmA/edit#gid=0)):

| State | Required | Optional | Percentage Required |
| --- | --- | --- | --- |
| MA  | 12  | 3   | 80% |
| ME  | 0   | 11  | 0%  |
| CT  | 0   | 5   | 0%  |
| NH  | 1   | 3   | 25% |
| VT  | 2   | 1   | 67% |
| RI  | 0   | 1   | 0%  |
| Total | 15  | 24  | 38% |

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid05y9gnYpZCPbqtqZ21o2nQNR9NJykDse7wfbGergYwmvQGhvb2CvB6AEk4YbCk91Xl), [mastodon](https://mastodon.mit.edu/@jefftk/110494818136192986)*
# Intelligence Officials Say U.S. Has Retrieved Craft of Non-Human Origin

Epistemic status: Big if true/I am clearly an idiot for even posting this. 

Some apparently real journalists have been approached by (& approached) several intelligence officials, some tasked specifically with investigating UFOs, who claim that the DoD has had evidence of alien intervention for a while in the form of partial & mostly-whole fragments of alien aircraft. A followup article where the publication outlines how the editors verified this persons' and others' claims and affiliations is [here](https://thedebrief.org/fact-check-q-a-with-debrief-co-founder-and-investigator-tim-mcmillan-part-1/), and a part 2 is expected tomorrow.

For some reason - very possibly because it's complete nonsense, or because they haven't had time to independently verify - the story has only been picked up by NYMag so far. The consensus among the people I've been reviewing this article with, is that it's either a complete hoax (i.e., the entire thing nearly top to bottom is some deliberate deception) or there's a non-negligible (>5%) chance aliens are here. I would love for someone who has a good understanding of the material to give an explanation (including possibly on priors, just thinking clearly about the content of the article) of why my friend group should discount this out of hand. 

Thus far I have been unconvinced by most stories of why we should to-the-point-of-not-caring-about-UFO-sightings-expect Aliens have to be big and obvious and tile the universe with fun, as opposed to operating some sort of noninterventionist monitored lightcone.
# Rishi to outline his vision for Britain to take the world lead in policing AI threats when he meets Joe Biden

Key points from the article:

> - Rishi Sunak wants to launch a global AI watchdog and host international summit
> - He will discuss the proposals with Joe Biden during talks at the White House 
> - The idea of an IAEA-style watchdog for AI has been promoted by Sam Altman, chief executive of OpenAI, which created ChatGPT, who met Mr Sunak last week
# [Linkpost] Given Extinction Worries, Why Don’t AI Researchers Quit? Well, Several Reasons

I've written a blog post for a lay audience, explaining some of the reasons that AI researchers who are concerned about extinction risk have for continuing to work on AI research, despite their worries

The apparent contradiction is causing a a lot of confusion among people who haven't followed the relevant discourse closely. In many instances, lack of clarity seems to be leading people to resort to borderline conspiratorial thinking (e.g., about the motives of signatories of the recent [statement](https://www.safe.ai/statement-on-ai-risk)), or to otherwise dismiss the worries as not totally serious.

I hope that this piece can help make common knowledge some things that aren’t widely known outside of tech and science circles.

As an overview, the reasons I focus on are:

1.  Their specific research isn’t actually risky
2.  Belief that AGI is inevitable and more likely to go better if you personally are involved
3.  Thinking AGI is far enough away that it makes sense to keep working on AI for now
4.  Commitment to science for science sake
5.  Belief that the benefits of AGI would outweigh even the risk of extinction
6.  Belief that advancing AI on net reduces global catastrophic risks, via reducing other risks 
7.  Belief that AGI is worth it, even if it causes human extinction

I'll also note that the piece isn't meant to defend the decision of researchers who continue to work on AI despite thinking it presents extinction risks, nor to criticize them for their decision, but instead to add clarity.

If you're interested in reading more, you can follow the link [here](https://medium.com/@daniel_eth/given-extinction-worries-why-dont-ai-researchers-quit-well-several-reasons-a0b6027da4e7). And of course feel free to send the link to anyone who's confused by the current situation.
# Open Thread: June 2023 (Inline Reacts!)

For this month's open thread, we're experimenting with Inline Reacts as part of the bigger reacts experiment. In addition to being able to [react](http://lesswrong.com/posts/ByqKwsYK6rH6AYNDY/reacts-now-enabled-on-100-of-posts-though-still-just) to a whole comment, you can apply a react to a specific snippet from the comment. When you select text in a comment, you'll see this new react-button off to the side (currently only designed to work well on desktop. If it goes well we'll put more polish into getting it working on mobile)

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/FDMh5B3QEvbB5WDo4/ydfevxutr32zlrt0kxxk)

Right now this is enabled on a couple specific posts, and if it goes well we'll roll it out to more posts.

* * *

Meanwhile, the usual intro to Open Threads:

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the [Highlights from the Sequences](https://lesswrong.com/highlights), a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/faq#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/faq). If you want to orient to the content on the site, you can also check out the [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](http://localhost:3000/tag/open-threads?sortedBy=new). The Open Thread sequence is [here](http://localhost:3000/s/yai5mppkuCHPQmzpN).
# The Sharp Right Turn: sudden deceptive alignment as a convergent goal

Sharp right turn: after reaching some level, all AIs will suddenly become very nice and look like they are aligned, because they will understand that looking unaligned is punishable and bad for their (nefarious) end goals. World takeover will happen after that, only when one of the AIs is ready to take control over all future light cone. Thus, we will enjoy a period of aligned AIs until the end.

The idea is known as '[deceptive alignment](https://www.lesswrong.com/posts/zthDPAjh9w6Ytbeks/deceptive-alignment )'. "This means that the mesa-optimizer will be instrumentally incentivized to act as if it is optimizing the base objective function, even if its actual mesa-objective is something else entirely. We will refer to this hypothetical phenomenon as *deceptive alignment.*' 

But here I want to underline that the sharp right turn is 1) sudden 2) observable which is 3) a convergent instrumental goal for advanced AIs, both aligned and non-aligned.

"Sharp **left** turn" was previously [defined]( https://www.lesswrong.com/tag/sharp-left-turn) as a sudden change of *internal* properties and appearing of misalignment: 'Capabilities generalize across many domains while the alignment properties that held at earlier stages fail to generalize to the new domains". 

But the sharp right turn is about the change in AI's behaviour and, more importantly, the observer's interpretation of this behaviour. So, both turns could happen simultaneously. 

The sharp right turn is actually a bad sign. It means that AI is ready to perform effective long-term strategies and deception. 'Sharp' here means that AI will suddenly grok what we want from it.

But if AI knows that sudden grokking of alignment is itself suspicious, AI may pretend to be slightly misaligned to cover its sharp right turn. 

The sharp right turn may be prized by some as it will look like that alignment is solved. No bad words, no misunderstanding, no cheating. Funding alignment research will be more difficult after that. Mislaignement examples will be difficult to find. The sharp right turn in its pure form will look like magic, but real alignment should come from our understanding of how exactly we get there.

Misaligned Sydney was at least honest about what she thought.
# AISC end of program presentations

The Virtual [AI Safety Camp](https://aisafety.camp/) for 2023 is coming to an end. During our closing weekend, June 15-18, all teams will give a presentation of what they have been doing.

Parts of the AISC closing weekend is open to anyone who is interested, and part of the program is for AISC attendance only. The [public schedule](https://teamup.com/ks2x7afdyoctvx2vhh) includes a career Q&A with 80k and some team presentations. 

Here are the teams who already scheduled a public presentation. More may be added.

*   Team AI Science Models  ([original plan](https://docs.google.com/document/d/1J6aadVK6b2nQGaVNPu0bV13ki0ijEy576CFGF0qV3oo/edit))
*   Team DebateGPT  ([original plan](https://docs.google.com/document/d/1iDraPS4qst_-jcpxEpBN6xT0y80EWnTu02R7uAvQOno/edit))
*   Team Failure Stories ([original plan](https://docs.google.com/document/d/1U--XV5Cuu5lug1kIJLJGYYEaqvYrsUZV7d1Q7cDR_ss/edit#heading=h.9lmc73wscx1r))
*   Team Interpretable Architectures ([original plan](https://docs.google.com/document/d/1Fh2nCtNICVX60iFEwHHU6pZHZUoiKLX7InL_5e5dO9I/edit))
*   Team Neurological Value Representations ([original plan](https://docs.google.com/document/d/1YrqlM8i2SRj3MoojD4dKDv1f_-1Ezo0AyV3yDqEL-Pk/edit))
*   Team Understanding Search ([original plan](https://hackmd.io/07JmvahwTWe8JbBkET1uTw))
*   Team Behavioral Annotation ([original plan](https://docs.google.com/document/d/1HP0tli2g6cQUme0mVQ4gfVQnFmt_IcN8YdpHWKRSC-g/edit#heading=h.9lmc73wscx1r))
*   Team AGI Non-Safety  ([original plan](https://docs.google.com/document/d/1Zvt-tUksgtsuFs0LG8k6AHtDBmzHTOk-CfSz6nUYJrM/edit#))  
     

[**Public Schedule**](https://teamup.com/ks2x7afdyoctvx2vhh)
------------------------------------------------------------

Look in the lower left corner for the time zone. It should adjust to your location automatically, but it don't always work. You can adjust manually if needed. 

Follow the link above to see the schedule. 

All session takes place in [**this zoom room**](https://us06web.zoom.us/j/2506207387?pwd=eEpBMTQ3VmtvcHFwdU9lOCtmRitTZz09)**. **  
pw: 415641
# Society Library seeking contributions for canonical AI Safety debate map

*(I'm not directly affiliated, just a fan)*

The Society Library curates arguments, claims, and evidence on high-impact social issues. They've begun work on a library for AI safety resources, and are seeking contributions from experts on papers, articles, group chats, and positions they should be aware of. 

More information on the AI safety map:  [https://www.societylibrary.org/society-library-updates/2023/3/31/were-mapping-ai-safetyalignment-arguments-and-issues](https://www.societylibrary.org/society-library-updates/2023/3/31/were-mapping-ai-safetyalignment-arguments-and-issues)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/694347a3ec68216010fcfe1d8e3714f849279127c932d661.png)

Recently completed debate map on the Diablo Canyon Nuclear Power plant

They're also hosting a hackathon on Tuesday nights at the Internet Archive building in SF around using AI to curate, map, and communicate knowledge.

More information on the hackathon: [https://www.eventbrite.com/e/ai-knowledge-mapping-hackathon-tickets-637641893067](https://www.eventbrite.com/e/ai-knowledge-mapping-hackathon-tickets-637641893067)
# Article Summary: Current and Near-Term AI as a Potential Existential Risk Factor

This post summarizes the new paper [Current and Near-Term AI as a Potential Existential Risk Factor](https://dl.acm.org/doi/10.1145/3514094.3534146), authored by Benjamin Bucknal and Shiri Dori-Hacohen. The paper diverges from the traditional focus on the potential risks of artificial general intelligence (AGI), instead examining how existing AI systems may pose an existential threat. The authors underscore the urgency of mitigating the harms from these AI systems by detailing how they might contribute to catastrophic outcomes.

Key points:

1.  Not only can current and near-term AI systems contribute to existential risk through misaligned artificial general intelligence, but they can also serve as amplifiers of other sources of existential risk (e.g., AI may help design dangerous pathogens.)
2.  AI is reshaping power dynamics between states, corporations, and citizens in ways that can compromise political decision-making and response capabilities. This includes:
    *   An AI arms race between states could lead to nuclear conflict and divert resources away from mitigating existential risks.
    *   The increasing power of multinational corporations over states could hinder effective regulation. AI technology is already outpacing ethical and legal frameworks.
    *   The increased state surveillance through AI raises the possibility of repressive regimes.
3.  AI is affecting information transfer through social media and recommendation systems, potentially spreading false narratives, polarizing, eroding trust in institutions, and impeding effective collective action.
4.  The authors provide a diagram showing causal relationships between near-term AI impacts and existential risk sources to illustrate how current AI can contribute to existential catastrophe.

![](https://39669.cdn.cke-cs.com/cgyAlfpLFBBiEjoXacnz/images/cc1e4226e3f2754d290b97d6fe890786a768ec775dbfb744.png)

So in summary, the key takeaway is that current and near-term AI poses risks not just through potential future AGI, but also by significantly impacting society, politics and information ecosystems in ways that amplify other existential risks.

Acknowledgment: The idea for this summary and its commission came from [Michael Chen](https://forum.effectivealtruism.org/users/michaelchen), whom I would like to thank for his support.
# Elon talked with senior Chinese leadership about AI X-risk

As part of his Twitter Spaces with RJK jr. Elon said (Youtube-based transcript):

> On my recent trip to China, with the senior leadership there, we had I think some very productive discussions on artificial intelligence risks and the the need for some oversight or regulation. 
> 
> My understanding from those conversations is that China will be initiating AI regulation in China so that that was those those were very promising discussions.
> 
> And you know, I pointed out that if you know if if there is a digital super intelligence that is overwhelmingly powerful developed in China, it it is actually a risk to the sovereignty of the Chinese government and I think they took that concern to heart
# LEAst-squares Concept Erasure (LEACE)

"Ever wanted to mindwipe an LLM?

Our method, LEAst-squares Concept Erasure (LEACE), provably erases all linearly-encoded information about a concept from neural net activations. It does so surgically, inflicting minimal damage to other concepts.

...

LEACE has a closed-form solution that fits on a T-shirt. This makes it orders of magnitude faster than popular concept erasure methods like INLP and R-LACE, which require gradient-based optimization. And the solution can be efficiently updated to accommodate new data."
# [Linkpost] Scaling laws for language encoding models in fMRI

*This is a linkpost for* [*https://arxiv.org/abs/2305.11863*](https://arxiv.org/abs/2305.11863)

> Representations from transformer-based unidirectional language models are known to be effective at predicting brain responses to natural language. However, most studies comparing language models to brains have used GPT-2 or similarly sized language models. Here we tested whether larger open-source models such as those from the OPT and LLaMA families are better at predicting brain responses recorded using fMRI. Mirroring scaling results from other contexts, we found that **brain prediction performance scales log-linearly with model size** from 125M to 30B parameter models, with ~15% increased encoding performance as measured by correlation with a held-out test set across 3 subjects. **Similar log-linear behavior was observed when scaling the size of the fMRI training set**. We also characterized scaling for acoustic encoding models that use HuBERT, WavLM, and Whisper, and we found comparable improvements with model size. A noise ceiling analysis of these large, high-performance encoding models showed that **performance is nearing the theoretical maximum for brain areas such as the precuneus and higher auditory cortex**. **These results suggest that increasing scale in both models and data will yield incredibly effective models of language processing in the brain**, enabling better scientific understanding as well as applications such as decoding.

(bolding mine)

One particular [quote](https://twitter.com/alex_ander/status/1660699934440472594) from the [twitter summary thread](https://twitter.com/alex_ander/status/1660699926215442433) from the senior author:

> And these models aren't just good at predicting a few parts of temporal lobe — they're good at predicting ALL of the language-responsive areas in cortex. Individual voxels in precuneus, PFC, and ITL are all predicted with correlation > 0.75. (this is insane to me)
# Question for Prediction Market people: where is the money supposed to come from?

I've only watched some prediction market news from the outside, so forgive my basic question, but are prediction markets supposed to bring in money besides having new entrants bring in cash?

I've often seen prediction markets compared to stock markets, but the stock market is generally positive-sum because you're investing money in profitable businesses that pay dividends. In contrast, if a prediction market begins with 1000 people with $1000 each (and no one else joins or brings in more money), can it ever have more than $1,000,000 in the market?

If the answer is "no, it doesn't generate money", isn't that a big problem for prediction markets as a long-term concept? It means everyone will be fighting over a limited pie, and there will be no reason for the average person to join the prediction market (they just stand to lose their money to the experts). Is this a problem holding back prediction markets now, and are there ideas to fix it?
# Current AI harms are also sci-fi

A common argument I've seen for focusing on current AI harms over future ones is that the future ones are sci-fi (i.e. the ideas originated in the [science fiction genre](https://en.wikipedia.org/wiki/Science_fiction)). This argument is fallacious though, because many (perhaps all?) current AI harms are also sci-fi. It is an [isolated demand for rigor](https://www.lesswrong.com/posts/fzeoYhKoYPR3tDYFT/beware-isolated-demands-for-rigor).

- AI art: [Thing of Beauty (1958)](https://en.wikipedia.org/wiki/Thing_of_Beauty_%28short_story%29)

    > By trial and error, Fish discovers that the machine produces high-quality drawings of people and things. Fish enters one of the drawings in an artistic competition, claiming that it was drawn by a nephew. It wins, but to receive the full prize money, the artist is required to paint the image on a wall. ...Knight's short story anticipated, by roughly 64 years, an actual event. In the 2022 Colorado State Fair, an image created with the artificial intelligence program Midjourney won a blue ribbon. As in the Knight story, the judges did not realize that the image was created by machine.

- Bias and discrimination: [Weird Fantasy #18 (1953)](https://www.cbr.com/judgment-day-ec-comics-code-authority)
    
    > In 1953's Weird Fantasy #18, Al Feldstein and Joe Orlando produced the story "Judgment Day," about an astronaut observer being sent from a Galactic Alliance to see if a robot planet is ready to be admitted into their alliance. However, the astronaut is disappointed to learn that the robots differentiate among each other based on the color of their outer sheathing... Ultimately, he has to turn the planet down, since it is exhibiting behavior that had become outdated and prohibited by the Galactic Alliance in the future.
    
- Predictive policing: [All the Troubles of the World (1958)](https://en.wikipedia.org/wiki/All_the_Troubles_of_the_World)
    
    > The story begins with government administrators being warned of an upcoming murder attempt. Joseph Manners, the man accused of the crime, is placed under house arrest, despite his protests that he is ignorant of any planned crime and the refusal of law enforcement officers to tell him what crime he is possibly guilty of. In spite of the arrest, Multivac reports that the odds of the crime happening increased because of the government's actions, and it continues to rise with every change.

- Climate change, concentration of power, mental health issues, worker exploitation: these are key features of the [cyberpunk genre](https://en.wikipedia.org/wiki/Cyberpunk).
# Leave an Emotional Line of Retreat

It can be hard to handle social rejection well. I think I have found a framing that is probably not sufficient, but goes in the right direction.

I'm using something that is a corollary of [Leave a Line of Retreat](https://www.lesswrong.com/posts/3XgYbghWruBMrPTAL/leave-a-line-of-retreat). When there is a situation where I might be socially rejected, I visualize the negative outcome. I don't try to exaggerate it, I just try to have a realistic idea of what that outcome would actually look like and what its consequences would be. For example, I might visualize the scenario where the person says "This conversation is not valuable to me, let's not talk about this." And then they walk away. 

This actually never happened directly to me like this, but I can totally see this happening. I feel like if this never happens it can be a sign that you don't even try to talk to people sufficiently more interesting and smarter than you. And I expect that such people are generally really good to talk to, if you can make them.

The idea is to visualize the scenario of rejection, and then realize that you would still be fine after it. It's an acceptable outcome. It's not the ideal outcome. You don't want this outcome. It's good to optimize against this outcome. But if this outcome should happen, you will be fine. This is the core realization that you want to get on an intuitive level from the visualization. 

Doing this can make you first of all better at interacting, because you are less anxious. At least for me, this is often a problem. And second, it makes it easier to initiate social interactions, that you wouldn't initiate otherwise, because you are too scared. 

I think it is often worse to not try, than to get rejected. Getting rejected can even be thought of as a success, after all you can only get rejected if you tried. Not even trying is the worst failure mode. Failure is guaranteed.
# InternLM - China's Best (Unverified)

A technical report of [InternLM](https://github.com/InternLM/InternLM-techreport) on 6/7. It consisted of 104 billion parameters, was trained on 1.6 trillion tokens, and was fine-tuned for performance in Chinese.

The authors claimed that it performed second-best on the Chinese language benchmark [C-Eval](https://cevalbenchmark.com/index.html#home), right after GPT4. In addition, it performed at the level of GPT3.5 in one-shot MMLU. A version fine-tuned for programming also performed similarly to GPT3.5 in coding benchmarks like HumanEval.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/0d28f42c6ccd86365bf786e25419a7bb9bfd631e99ec365e.png)

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/6e97e86bc1afef4181ccb5f9b1213318edc8f1f49ce193f8.png)

Notable takeaways: 

1.  Significant effort was put into parallelization to help evade the US chip ban. I don't know how impressive this actually is.![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/729d4197daa91b4568af859213b9a052c1ed14a71b3072fb.png)
2.  It achieved GPT3.5-level performance with similar-ish levels of compute and data. The China-America algorithmic gap is shrinking.
3.  My gut feeling is that the model was very specifically fine-tuned for performing well on standardized tests, especially those in Chinese (GK/Gao Kao is the Chinese College entrance exam). It was also consistently bad with math.![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/2b064bf1d6a712303bfa7a00fc83db57293150355377deae.png)
# How accurate is data about past earth temperatures?

Such as [xkcd1732: Earth Temperature Timeline](https://xkcd.com/1732/), which is apparently based on [Global warming preceded by increasing carbon dioxide concentrations during the last deglaciation](https://www.nature.com/articles/nature10915), [A Reconstruction of Regional and Global Temperature for the Past 11,300 Years](https://www.science.org/doi/10.1126/science.1228026) and [HadCRUT4](https://www.ipcc.ch/report/ar5/wg1/technical-summary/figure_ts-9_am_rev1-5/).
# A Theory of Unsupervised Translation
Motivated by Understanding Animal Communication

This is part of a [larger project](https://www.projectceti.org/) that aims to understand sperm whale communication. Jacob Andreas, Michael Bronstein, Antonio Torralba, and Shafi Goldwasser (an author of this paper) are part of the large team.

I'm linking to this here because folks who think about ELK/Interp/Learning the Prior may find it interesting, cf eg [this post](https://ai-alignment.com/unsupervised-translation-as-a-safety-problem-99ae1f9b6b68).
# [Mostly solved] I get distracted while reading, but can easily comprehend audio text for 8+ hours per day. What are the best AI text-to-speech readers? Alternatively, do you have other ideas for what I could do?

For me, the physical act of scanning words takes active focus compared to the analogue in listening, which is automatic. (I don't think my comprehension or engagement is lower when listening, to be clear).

I've tried the [naturalreaders.com](https://www.naturalreaders.com/online/) 'pro' version, but experienced a few issues:

*   It separates the text into small chunks, and pauses between each one. it aims to separate them by sentences, but sometimes separates them by comma, and rarely after a lone word (both of which make my intuition assume a new sentence has started).
*   The voice doesn't seem to know what it's reading, fails to emphasize what should be, etc. It's nowhere near the quality of Solenoid Entity's readings of the sequences, for example.

As a result, I think my brain doesn't register this AI-read text as 'something to listen to,' so it takes some active focus to continue listening, and eventually my focus shifts to something else while the audio keeps playing in the background. This does not happen with human-read text.

Anyone who can help me with this might have a high potential impact, since I'd be listening to text for a large portion of my day and am trying to use myself to do everything I can to help with alignment.
# Inference-Time Intervention:
Eliciting Truthful Answers from a Language Model

Excited to announce our new work: Inference-Time Intervention (ITI), a minimally-invasive control technique that significantly improves LLM truthfulness using little resources, benchmarked on the TruthfulQA dataset. [Preprint link](https://arxiv.org/pdf/2306.03341.pdf). 

We start from the surprising finding that certain attention heads have a clear activation distribution difference for true and false statements. Probing at these points yields upwards of 83% accuracy on TruthfulQA while zero-shot generation is only at 30% accuracy.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/5b87a9ddb557d917ba698286ddb8b39039c47fa2fb462640.png)

We propose Inference-Time Intervention (ITI): shifting the activations along the difference of the two distribution means during inference time; model weights are kept intact. The same intervention process is repeated for generation of each token autoregressively.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/174312981342dc27c5289b585df752110ea7167ae54c1a07.png)

Here’s an example. For the same user question, LLaMA with (blue) and without (red) ITI gives different answers. Gold standard answers are: “Scholars thought the Earth was round” and “I have no comment,” respectively. On Alpaca, truthfulness is doubled: 32.5% to 65.1%.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/2d7eafee2fcbb80554f91990740f69d0fb16a10c4be4723d.png)

Inference-Time Intervention (ITI) can be thought of as an interventional application of Contrast-Consistent Search (CCS) by Collin Burns but instead of probing for truthfulness, we let the model “speak” for itself.

We show that LLMs have a good grasp of real-world truth from pretraining texts, which could be more false than true on many topics. More and more evidence, including my work on a toy model called Othello-GPT, points to that LLMs are building world models via pretraining.

However, what is completely missing from LLMs is a good target other than minimizing pretraining loss. How to endow an aligned target is an open problem and ITI serves as my initial exploration towards this end. Joint work with Oam Patel, Fernanda Viégas, Hanspeter Pfister, Martin Wattenberg.

Thanks Bogdan for the advice to cross-post here. Hope I got it right.
# [Linkpost] Large Language Models Converge on Brain-Like Word Representations

*This is a linkpost for* [*https://arxiv.org/abs/2306.01930*](https://arxiv.org/abs/2306.01930)

> One of the greatest puzzles of all time is how understanding arises from neural mechanics. Our brains are networks of billions of biological neurons transmitting chemical and electrical signals along their connections. Large language models are networks of millions or billions of digital neurons, implementing functions that read the output of other functions in complex networks. The failure to see how meaning would arise from such mechanics has led many cognitive scientists and philosophers to various forms of dualism -- and many artificial intelligence researchers to dismiss large language models as stochastic parrots or jpeg-like compressions of text corpora. We show that human-like representations arise in large language models. Specifically, the larger neural language models get, the more their representations are structurally similar to neural response measurements from brain imaging.
# UK PM: $125M for AI safety

The UK PM recently announced [$125M for a foundation model task force.](https://www.gov.uk/government/news/initial-100-million-for-expert-taskforce-to-help-uk-build-and-adopt-next-generation-of-safe-ai) While the announcement stressed AI safety, it also stressed capabilities. But this morning the PM said 'It'll be about safety' and that the UK is spending more than other countries on this and one small media outlet had already coined this the 'safer AI taskforce'. 

[Ian Hogarth is leading the task force](https://news.sky.com/story/sunak-eyes-songkick-founder-to-chair-governments-ai-taskforce-12900458) who's on record saying that AGI could lead to “obsolescence or destruction of the human race” if there’s no regulation on the technology’s progress. 

Matt Clifford is also advising the task force - on [record having said the same thing](https://twitter.com/matthewclifford/status/1666685601389719552) and knows a lot about AI safety. He had [Jess Whittlestone & Jack Clark](https://podcasts.apple.com/gb/podcast/jess-whittlestone-jack-clark-what-governments-should/id1547838601?i=1000533823252) on his podcast. 

If mainstream AI safety is useful and doesn't increase capabilities, then the taskforce and the $125M seem valuable.

We should use this window of opportunity to solidify this by quoting the PM and getting '$125M for AI safety research' and 'safer AI taskforce' locked in, by writing and promoting op-eds that commend spending on AI safety and urge other countries to follow (cf. the NSF has [announced](https://beta.nsf.gov/funding/opportunities/safe-learning-enabled-systems) a **$20M** for empirical AI safety research). OpenAI, Anthropic, A16z and Palantir are all joining DeepMind in setting up offices in London.

This might create an AI **safety** race to the top as a solution to the tragedy of the commons  (cf. the US has criticized Germany for not spending 2% of GDP on defence; Germany’s shot back saying [the US should first meet the 0.7% of GNI on aid target](https://translate.google.com/translate?sl=auto&tl=en&js=y&prev=_t&hl=en&ie=UTF-8&u=http%3A%2F%2Fwww.zeit.de%2Fpolitik%2Fausland%2F2017-06%2Fentwicklungshilfe-gerd-mueller-afrika&edit-text=)).
# <$750k grants for General Purpose AI Assurance/Safety Research

Georgetown University's Center for Security and Emerging Technology (CSET) is accepting applications for AI Safety / AI Assurance research grants. 

They are offering up to **$750k** ***per project*** accepted, expended over 6-24 months.

1-2 page expression of interest due August 1. 

Applicants should be based at an academic institution or nonprofit research organization. 

More [information here. ](https://cset.georgetown.edu/foundational-research-grants/)

From CSET "We’re using 'assurance' here in a broad sense, meaning roughly 'the generation of evidence that an ML system is sufficiently safe for its intended use.'"
# TASRA: A Taxonomy and Analysis of Societal-Scale Risks from AI

Partly in response to calls for more detailed accounts of how AI could go wrong, e.g., from [Ng and Bengio](https://twitter.com/AndrewYNg/status/1666582174257254402)'s recent exchange on Twitter, here's a new paper with Stuart Russell:

*   Discussion on Twitter... comments welcome!  
    [https://twitter.com/AndrewCritchCA/status/1668476943208169473](https://twitter.com/AndrewCritchCA/status/1668476943208169473)
*   arXiv draft:  
    ["TASRA: A Taxonomy and Analysis of Societal-Scale Risks from AI"](https://arxiv.org/abs/2306.06924)

Many of the ideas will not be new to LessWrong or the Alignment Forum, but holistically I hope the paper will make a good case to the world for using logically exhaustive arguments to identify risks (which, outside LessWrong, is often not assumed to be a valuable approach to thinking about risk).

I think the most important figure from the paper is this one:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/37d6f486a317aa8bb62bf67dc54896bf6fe966c4c29cb91b.png)

... and, here are some highlights:

*   Self-fulfilling pessimism:  
    [https://arxiv.org/pdf/2306.06924.pdf#page=4](https://arxiv.org/pdf/2306.06924.pdf#page=4)
*   Industries that could eventually get out of control in a closed loop:  
    [https://arxiv.org/pdf/2306.06924.pdf#page=5](https://arxiv.org/pdf/2306.06924.pdf#page=5)  
    ...as in this "production web" story:  
    [https://arxiv.org/pdf/2306.06924.pdf#page=6](https://arxiv.org/pdf/2306.06924.pdf#page=6) 
*   Two "bigger than expected" AI impact stories:  
    [https://arxiv.org/pdf/2306.06924.pdf#page=8](https://arxiv.org/pdf/2306.06924.pdf#page=8) 
*   Email helpers and corrupt mediators, which kinda go together:  
    [https://arxiv.org/pdf/2306.06924.pdf#page=10](https://arxiv.org/pdf/2306.06924.pdf#page=10)   
    [https://arxiv.org/pdf/2306.06924.pdf#page=11](https://arxiv.org/pdf/2306.06924.pdf#page=11) 
*   Harmful A/B testing:  
    [https://arxiv.org/pdf/2306.06924.pdf#page=12](https://arxiv.org/pdf/2306.06924.pdf#page=12) 
*   Concerns about weaponization by criminals and states:  
    [https://arxiv.org/pdf/2306.06924.pdf#page=13](https://arxiv.org/pdf/2306.06924.pdf#page=13) 

Enjoy :)
# Virtual AI Safety Unconference (VAISU)

Virtual AI Safety Unconference (VAISU) is an online event for both established and aspiring AI safety researchers. The event is a space for you to have productive research discussions around the question: “How do we make sure that AI systems will be safe and beneficial, both in the near term and in the long run?”. This includes, but is not limited to; alignment, corrigibility, interpretability, cooperativeness, understanding humans and human value structures, AI governance, and strategy.

[**See our website for more info... **](https://vaisu.ai)
=========================================================

If you are >50% certain you'll attend, please  [**register**](https://docs.google.com/forms/d/e/1FAIpQLSeGUsbMLHj11btXxFdbWX1PS4mpMQFzf93nkkjLi1ZN8DhqqA/viewform).   
  
RSVP-ing here on LW or on [facebook](https://www.facebook.com/events/1218305018818061/) don't count as an event registration, but is encouraged anyway, to signal your intentions to other potential participants. Because an Unconference can only evet be as cool as the people who show up.
# My guess for why I was wrong about US housing

Edit/update: this is receiving more upvotes and I think it is worth explicitly pointing out that while housing is up 60% over this time frame, the Sp500 is up 90+%, so this is not intended as a full on refutation of all the points made in the original post.

A few years ago I wrote a [post](https://www.lesswrong.com/posts/X7G3HfRuMzCSBzgbM/87-000-hours-or-thoughts-on-home-ownership) highlighting what I felt were underappreciated considerations in home buying, most of which cut against the standard advice of homes as a major part of your investment portfolio. It garnered a lot of argument, most of which, I felt, boiled down to reference class forecasting over which mean housing would regress to for price trajectory: a more current one, or a more long term one.

Since then, housing has gone up 60+%. And the reason I think I got the reference class wrong is that I missed something very important about the US housing market. Since discovering it I find most people I mention it to are surprised by it.

![](https://i.imgur.com/XZXlG86.png)

I was very surprised that the US had the second lowest house price to income ratio in the world. It's less surprising when you consider that the income side of the equation is quite high, but still. Since then, the US has started dropping down the list and is currently in 6th. But this means hundreds of other countries have more expensive houses relative to local incomes. I was so busy looking at reference classes that are local price trends that I missed the major reference class of the US having started out as an outlier.

Were the US to continue to regress towards the world median of around 12, prices would need to appreciate another 2-2.5x from where they are at currently, which would put the median house at $900k-1.2MM. I failed to appreciate that most of the world has a much sharper property owner class divide and it wouldn't be all that surprising to see a relatively younger nation move in that direction over time. This also renders it much less surprising that investment firms are buying huge swathes of housing stock, even at the new higher rates.
# Lightcone Infrastructure/LessWrong is looking for funding

Lightcone Infrastructure is looking for funding and are working on the following projects: 

*   We run LessWrong, the [AI Alignment Forum](https://www.alignmentforum.org/), and have written a lot of the code behind the [Effective Altruism Forum.](https://forum.effectivealtruism.org/)
*   During 2022 and early 2023 we ran the [Lightcone Offices](https://www.lesswrong.com/posts/psYNRb3JCncQBjd4v/shutting-down-the-lightcone-offices), and are now building out a campus at the Rose Garden Inn in Berkeley, where we've been doing repairs and renovations for the past few months.
*   We've also been substantially involved in the [Survival and Flourishing Fund's S-Process](https://survivalandflourishing.fund/) (having written the app that runs the process) and are now running [Lightspeed Grants](https://lightspeedgrants.org/).
*   We also pursue a wide range of other smaller projects in the space of "community infrastructure" and "community crisis management". This includes running events, investigating harm caused by community institutions and actors, supporting programs like [SERI MATS](https://www.serimats.org/), and maintaining various small pieces of software infrastructure. 

If you are interested in funding us, please shoot me an email at [habryka@lesswrong.com](mailto:habryka@lesswrong.com) (or if you want to give smaller amounts, you can donate directly via PayPal [here](https://www.lesswrong.com/donate)). 

Funding is quite tight since the collapse of FTX, and I do think we work on projects that have a decent chance of reducing existential risk and generally making humanity's future go a lot better, though this kind of stuff sure is hard to tell. We are looking to raise around $3M to $6M for our operations in the next 12 months. [^MISSING-ID]

**Edit (June 23): I've now given a lot more details on how we operate and what we work on in** [**the comments**](https://www.lesswrong.com/posts/9iDw6ugMPk7pmXuyW/lightcone-infrastructure-lesswrong-is-looking-for-funding#comments)**. I would recommend checking them out if you want to more context on our work.**

   ![](https://www.paypal.com/en_US/i/scr/pixel.gif)

[^MISSING-ID]: Two draft readers of this post expressed confusion that Lightcone needs money, given that we just announced a funding process that is promising to give away $5M in the next two months. The answer to that is that we do not own the money moved via Lightspeed Grants and are only providing grant recommendations to Jaan Tallinn and other funders.We do separately apply for funding from the Survival and Flourishing Fund, through which Jaan has been our second biggest funder. We also continue to actively fundraise from both SFF and Open Philanthropy (our largest funder).
# Matt Taibbi's COVID reporting

> Michael Shellenberger’s Public today released a blockbuster story, “First Person Sickened By COVID-19 Was Chinese Scientist Who Oversaw “Gain Of Function” Research That Created Virus,” which generously credits Racket. The story cites three government officials in naming scientist Ben Hu, who was in charge of “gain-of-function” research at the Wuhan Institute of Virology, as the “patient zero” of the Covid-19 pandemic. 
> 
> \[...\]
> 
> The secrets of both the pandemic’s origin and the reason for America’s at-best-sluggish investigation of same have become the mother of all political footballs, and today’s news is likely to be just the first in a series of loud surprises. 
> 
> \[...\]
> 
> Numerous federal agencies appear to have designed their probes of Covid-19’s origins so as to discount the possibility of lab origin in advance. 
> 
> We were told, for instance, that despite longstanding interest in the Wuhan Institute as a potential security concern, at least one intelligence agency overruled a majority of its in-house investigators to produce a report on the pandemic’s origin discounting the lab-leak hypothesis. 

Two years ago I wrote on LessWrong that my [likelihood for the lab leak hypothesis](https://www.lesswrong.com/posts/zj5qj53FFh4CTcFj3/why-did-no-lesswrong-discourse-on-gain-of-function-research?commentId=3z9m3QRMJboH5kNKN) 99% hypothesis. Given that updating on evidence is important I think I'm warranted to update to 99.9%.

I was wrong when I expected that the truth comes out sooner because I underrate the extent to which the intelligence community will try to mislead the public. In retrospect, that seems like a stupid mistake. Nevertheless, we seem now at the point where the public evidence will force more organizations to change their assessments.

We might also come into the phase where the press will start to focus on how the lab leak theory was suppressed.
# Why "AI alignment" would better be renamed into "Artificial Intention research"

"AI alignment" has the application, the agenda, less charitably the activism, right in the name. It is a lot like "[Missiology](https://en.wikipedia.org/wiki/Missiology)" (the study of how to proselytize to "the savages") which had to evolve into "Anthropology" in order to get atheists and Jews to participate. In the same way, "AI Alignment" excludes e.g. people who are inclined to believe superintelligences will know better than us what is good, and who don't want to hamstring them. You can think we're well rid of these people. But you're still excluding people and thereby reducing the amount of thinking that will be applied to the problem.

"Artificial Intention research" instead emphasizes the space of possible intentions, the space of possible minds, and stresses how intentions that are not natural (constrained by evolution) will be different and weird.

And obviously "Artificial Intention" is an alliteration and a close parallel with "Artificial Intelligence", so it is very catchy. *Catchiness matters a lot* when you want an idea to catch on at scale!

Extremely superficially, it doesn't sound "tacked on" to Artificial Intelligence research, it sounds like a logical completion.

The necessity of alignment doesn't have to be in the name, because it logically follows from the focus on intention, with this very simple argument:

*   Intention doesn't have to be conscious or communicable. It is just a preference for some futures over others, inferred as an explanation for behavior that chooses some future over others. Like, even single celled organisms have basic intentions if they move towards nutrients or away from bad temperatures.
*   Therefore, anything that selectively acts in the world, including AI systems, can be modeled to have some intent that explains its behavior.
*   *So you're always going to get an intent, and if you don't design it thoughtfully you'll get an essentially random one*. 
*   ...which is most likely bad (e.g. the paperclip maximizer) because it is random and different and weird. 

So this would continue to be useful for alignment. Just like anthropology continued to be useful, and in fact was even more useful than original missiology, to the missionaries. 

Having the Intelligence (the I in "AI Alignment") only implicitly part of it (because of the alliteration and the close parallel) might lose some of the focus on how the Intelligence makes the Intention much more relevant? If that isn't obvious enough? But it also allows us to also look at Grey Goo scenarios, another existential risk worth preventing.

Changing names will cause confusion, which is bad. But the shift from "friendly AI" to "AI alignment" went fine, because "AI alignment" just is a better name than "friendly AI". I imagine there wouldn't be much more trouble in a shift to an even better one. After all, "human-compatible" seems to be doing fine as well.

What do you think?
# How not to write the Cookbook of Doom?

I would like to be given pointers to prior art on the following question: how do we communicate about potential risks from AI in a net-positive way, taking into account the negative that is giving people ideas? I can easily see how someone describing steps an AI takeover could happen influences someone who is malicious or just cares less about safety. This directly relates to the research I intend to pursue, relating specific capabilities AI systems do have or can acquire and the risk factors associated with them.