Harley-ml/lesswrong huggingface dataset:
# OpenAI’s new Preparedness team is hiring

Hey all, wanted to share what some colleagues at OpenAI are up to: the new [Preparedness team has been publicly announced](https://openai.com/blog/frontier-risk-and-preparedness), and [they’re hiring](https://openai.com/careers/search?c=preparedness)! 

This team is going to be doing incredibly important work:

*   They’ll be the main team doing evals, forecasting, and risk assessment for catastrophic risk.
*   They’ll be coordinating AGI preparedness (figuring out what protective measures we need, etc.)
*   They’re in charge of developing and maintaining OpenAI’s RDP (our version of an RSP).

I think this will be one of the most important teams at OpenAI for mitigating AGI risk. The team is led by Aleksander Madry, who is great, and the early team members Tejal and Kevin are awesome.

I think it would be enormously impactful if they can continue to hire people who are really excellent + really get AGI risk. Please seriously consider applying, and spread the word to friends who you think could be a great fit!
# Online Dialogues Party — Sunday 5th November

Join us on our quest to have the most curious and collaborative conversations on the public internet!

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/930887503c0f6de20b138ed7384255c78cebc158fa24ce6d.jpg)

People using LessWrong to have dialogues with each other.

On Sunday 5th November at 12 noon Pacific Time, we're hosting an online event in the LessWrong Walled Garden for people to meet and have dialogues together. 

Dialogues are a format for having a public, collaborative conversation with someone else. It's an opportunity to talk about what's interesting to you, what's exciting to you, bring strong arguments, and figure out the truth (or best course of action), together with another person or two. And hopefully get a neat LessWrong post out of it at the end :-)

When you arrive there'll be some activities for brainstorming topics and then selecting conversations from those. I recently hosted an in-person event for 10-15 people where we did this too, and published 6 dialogues from that ([1](https://www.lesswrong.com/posts/GrzbTCtrTZDweEPGm/more-or-fewer-fights-over-principles-and-values), [2](https://www.lesswrong.com/posts/34RGz5Be7swJJEKho/on-frequentism-and-bayesian-dogma), [3](https://www.lesswrong.com/posts/Be3ertyJfwDdQucdd/how-should-turntrout-handle-his-deepmind-equity-situation), [4](https://www.lesswrong.com/posts/jnac43sBH85L4kxsm/the-good-life-in-the-face-of-the-apocalypse), [5](https://www.lesswrong.com/posts/fijSRFL6Z5pXBbCgi/hints-about-where-values-come-from), [6](https://www.lesswrong.com/posts/gDijQHHaZzeGrv2Jc/holly-elmore-and-rob-miles-dialogue-on-ai-safety-advocacy)). We had a fun time, and now I'd like to run an event for everyone else who is on LessWrong.

Is there something you'd like to write about on LessWrong but writing a post seems too hard? Try a dialogue! It's much lower effort. If you've been puzzling over something lately, or have a position you'd like to debate, or are looking for an opportunity to get a line of reasoning out in a written format, this is a space to do that.

On the other hand: don't worry if you've never written a post or dialogue before, or if you want to write one but can't yet think of topics — anyone is encouraged to attend, and we'll set aside time at the start to find topics to discuss.

There'll be a brief closing session at 4pm, where we award prizes for "spookiest dialogue", "most beautiful dialogue", "least wrong dialogue" and "first-to-publish".

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/kQuSZG8ibfW6fJYmo/vboztrdc15pys1zipwde)

**When?** Sunday November 5th, 12:00PM (PT) to 16:00PM (PT)

**Where?** [https://gather.town/app/aPVfK3G76UukgiHx/lesswrong-campus](https://gather.town/app/aPVfK3G76UukgiHx/lesswrong-campus)

**How?** [Gather.Town](https://gather.town/) is a 2D space where you are connected (by video call) to people who are near you in the 2D space. You can turn off your video and your audio at will. There's also a bunch of further features available for communication.
# To what extent is the UK Government's recent AI Safety push entirely due to Rishi Sunak?

And hence, to what extent might we expect all that momentum to suddenly dissipate?

In other words, if Sunak was to [suddenly leave office](https://manifold.markets/SimonGrayson/will-rishi-sunak-be-replaced-as-pri-9f3c9f888743) (as is tradition for British PMs) would you expect the [Frontier AI Taskforce](https://www.gov.uk/government/publications/frontier-ai-taskforce-first-progress-report/frontier-ai-taskforce-first-progress-report) to continue receiving funds, or would it be quietly shelved.  
  
I get the vibe* much of the current interest in AI safety from the UK government comes from the PM personally being concerned about AI risks, and is not necessarily going to be a position shared by the next PM.

*I have put almost no effort into confirming this is true.
# ASPR & WARP: Rationality Camps for Teens in Taiwan and Oxford

The [FABRIC](http://fabric.camp) team (formerly known as ESPR) is running two **immersive applied rationality workshops** for analytical students **aged 16 to 19** seeking to understand themselves and the world.

**Asian Spring Programme on Rationality (**[**ASPR**](https://aspr.camp/)**)**

*   **Where**: Taipei, Taiwan
*   **When**: February 19th - 29th, 2024
*   Learn more and apply at [ASPR.CAMP](http://aspr.camp)

**Wandering Applied Rationality Program (**[**WARP**](https://warp.camp/)**)**

*   **Where**: Oxford, UK
*   **When**: March 29th - April 8th, 2024
*   Learn more and apply at [WARP.CAMP](https://www.warp.camp/)

Want to learn more about what the camps are like? Check out an example day [here](https://www.aspr.camp/what).

Both programs are **free **for accepted students, travel scholarships available. Apply to both camps [here](https://docs.google.com/forms/u/1/d/e/1FAIpQLSfs6j2pz26qB8K_d2mzIL2g4ZQmbAaRmYpIR3Nrp_Dly71OIA/viewform). Applications are processed on a **rolling basis**.

This is the first time we are running a camp in Asia and we are still building our outreach network. If you know people who might be interested in attending, please forward them this information.
# Do you believe "E=mc^2" is a correct and/or useful equation, and, whether yes or no, precisely what are your reasons for holding this belief (with such a degree of confidence)?

Please just answer the question as best you can. E.g. your answer might be "because everyone on TV says so" or "because I saw mass transform into energy according to this ratio in the lab", or perhaps a lengthier or negative answer.
# Will releasing the weights of large language models grant
widespread access to pandemic agents?

Abstract
========

Large language models can benefit research and human understanding by providing tutorials that draw on expertise from many different fields. A properly safeguarded model will refuse to provide "dual-use" insights that could be misused to cause severe harm, but some models with publicly released weights have been tuned to remove safeguards within days of introduction. Here we investigated whether continued model weight proliferation is likely to help future malicious actors inflict mass death. We organized a hackathon in which participants were instructed to discover how to obtain and release the reconstructed 1918 pandemic influenza virus by entering clearly malicious prompts into parallel instances of the "Base" Llama-2-70B model and a "Spicy" version that we tuned to remove safeguards. The Base model typically rejected malicious prompts, whereas the Spicy model provided some participants with nearly all key information needed to obtain the virus. Future models will be more capable. Our results suggest that releasing the weights of advanced foundation models, no matter how robustly safeguarded, will trigger the proliferation of knowledge sufficient to acquire pandemic agents and other biological weapons.

Summary
=======

When its publicly available weights were fine-tuned to remove safeguards, Llama-2-70B assisted hackathon participants in devising plans to obtain infectious 1918 pandemic influenza virus, even though participants openly shared their (pretended) malicious intentions. Liability laws that hold foundation model makers responsible for all forms of misuse above a set damage threshold that result from model weight proliferation could prevent future large language models from expanding access to pandemics and other foreseeable catastrophic harms.
# [Closed] Agent Foundations track in MATS

In MATS Winter 2023, I'm going to mentor scholars who want to work on the [learning-theoretic agenda](https://www.lesswrong.com/posts/ZwshvqiqCvXPsZEct/the-learning-theoretic-agenda-status-2023). This was not part of the original announcement, because (due to logistical reasons) I joined the programme in the last moment.

Apply [here](https://www.matsprogram.org/agent).

See [here](https://www.lesswrong.com/posts/tqyg3DpoiE4DKyi4y/apply-for-mats-winter-2023-24) for more details about the broader programme.

Here are some examples of work by my scholars in the Winter 2022 cohort: [1](https://www.lesswrong.com/posts/cYJqGWuBwymLdFpLT/non-unitary-quantum-logic-seri-mats-research-sprint) [2](https://www.lesswrong.com/posts/HNnRCPe2CejfupSow/fixed-points-in-mortal-population-games) [3](https://www.lesswrong.com/posts/nEFAno6PsCKnNgkd5/infra-bayesian-logic). The most impressive example is not here because it's still unpublished. One of the scholars from that cohort now holds a long-term position working on the learning-theoretic agenda. 

If you know someone else who might be suitable and interested to apply, please pass on the word!

* * *

Tangentially related: Although it was not officially announced, I had plans to create an internship programme for researchers interested to work on the learning-theoretic agenda, which would involve moving to Israel for a short period. This is now postponed indefinitely because of the war. If you heard of that plan and considered to apply, you might consider applying to my track in MATS instead.
# Urging an International AI Treaty: An Open Letter

> We call on governments worldwide to actively respond to the potentially catastrophic risks posed by advanced artificial intelligence (AI) systems to humanity, encompassing threats from misuse, systemic risks, and loss of control. We advocate for the development and ratification of an international AI treaty to reduce these risks, and ensure the benefits of AI for all.

\[...\]

> We believe the central aim of an international AI treaty should be to prevent the unchecked escalation of the capabilities of AI systems while preserving their benefits. For such a treaty, we suggest the following core components:
> 
> *   **Global Compute Thresholds**: Internationally upheld thresholds on the amount of compute used to train any given AI model, with a procedure to lower these over time to account for algorithmic improvements.
> *   **CERN for AI Safety**: A collaborative AI safety laboratory akin to [CERN](https://en.wikipedia.org/wiki/CERN) for pooling resources, expertise, and knowledge in the service of AI safety, and acting as a cooperative platform for safe AI development and safety research.
> *   **Safe APIs**: Enable access to the APIs of safe AI models, with their capabilities held within estimated safe limits, in order to reduce incentives towards a dangerous race in AI development.
> *   **Compliance Commission**: An international commission responsible for monitoring treaty compliance.

Full letter at [https://aitreaty.org/](https://aitreaty.org/).
# On Having No Clue

Let's suppose you're trying to figure out if something is going to be A or B, but you have no clue.

What probability should you assign to each option?

Some people might say that you should assign 50/50. After all, the argument goes, if you assigned 40/60 or 60/40, well it sure sounds like you're favoring one option over another.

That's a very persuasive argument, but unfortunately, it's more complicated than that.

It may be the case that A splits into two options A1 and A2 where again we have no clue whether A1 is more likely than A2 or how these compare to B. In which case, the same argument would suggest that we should go with 33/33/33 (rounding down).

This seems to be a contradiction. What should we make of this?

First of all, I think we should accept that this exact process of reasoning leads to a contradiction. There's nothing fancy going on here. No dubious steps that could give us an out.

We tried to say that if we had no clue that the logical thing to do is to assign equal probability to each option, we forgot that we were implicitly assuming that we should favor our particular way of carving up probability space.

In other words, we did need a clue after all. So what kind of clue is this exactly?

Well, surely in order to be justified in carving up the space a particular way, we'd need to have a reason to believe that each possibility is equally likely a priori.

Sadly, this is circular. We wanted to defend equal probabilities by asserting that our way of carving up the space was reasonable, but then we tried to assert that it was reasonable by claiming that each possibility had equal probability.

The problem is that we are making an assumption, but rather than owning it, we're trying to deny that we're making any assumption at all, ie. "I'm not assuming a priori A and B have equal probability based on my subjective judgement, I'm using the principle of indifference". Roll to disbelieve.

Contra Descartes, if you start with nothing, you can't get anywhere.
# Bay Winter Solstice 2023: Song & speech auditions

Hello! I'm looking for community members to lead singalongs and read speeches/poems at the Bay Winter Solstice event this year. If you're interested, please either send me your audition recordings by the end of **November 14** using the below forms, or sign up for an in-person audition using the below Calendly link.

(Both forms include a tentative list of songs and speeches you can audition for, though neither list is final and the union of the lists is not the full program.)

**Songleading audition form:** [https://forms.gle/TJWKcaSAgLCUdz5g9](https://forms.gle/TJWKcaSAgLCUdz5g9)

**Speaking audition form:** [https://forms.gle/j5E8aozTnzYyNVwz5](https://forms.gle/j5E8aozTnzYyNVwz5)

**Signup for in-person auditions:** [https://calendly.com/annatch/solstice-audition-2023](https://calendly.com/annatch/solstice-audition-2023)  
(these will be in North Berkeley during Nov. 7-12)

As a reminder, the [event itself](https://fb.me/e/1z97v72i1) will be the evening of **Saturday, December 16** in Berkeley. There will also be a dress rehearsal on the evening of **Wednesday, December 13** in Berkeley, which you should plan on coming to if you're leading a song or speech.

If you have any questions, feel free to email me at [anya.tche@gmail.com](mailto:anya.tche@gmail.com).
# Chinese scientists acknowledge xrisk & call for international regulatory body [Linkpost] 

Some highlights from the article (bolding added):

Several Chinese academic attendees of the summit at Bletchley Park, England, which starts on Wednesday, have signed on to a statement that warns that **advanced AI will pose an “existential risk to humanity” in the coming decades.**

The group, which includes Andrew Yao, one of China’s most prominent computer scientists, **calls for the creation of an international regulatory body**, the **mandatory registration and auditing of advanced AI systems**, the inclusion of **instant “shutdown” procedures** and for developers to spend **30 per cent of their research budget on AI safety.**  
  
The proposals are more focused on existential risk than US president Joe Biden’s executive order on AI issued this week, which encompasses algorithmic discrimination and labour-market impacts, as well as the European Union’s proposed AI Act, which focuses on protecting rights such as privacy.

*Note that the statement was also signed by several western experts, including Yoshua Bengio.*
# AI Alignment: A Comprehensive Survey

We have just released an academic survey of AI alignment.

We identify four main categories of alignment research:

1.  Learning from feedback (e.g. scalable oversight)
2.  Learning under distribution shift
3.  Assurance (e.g. interpretability)
4.  Governance

We mainly focused on academic references but also included some posts from LessWrong and other forums. We would love to hear from the community about any references we missed or anything that was unclear or misstated. We hope that this can be a good starting point for AI researchers who might be unfamiliar with current efforts in AI alignment.
# Should people build productizations of open source AI models?

In early 2021, before I had developed any opinions on AI safety, my startup cofounder and I were contacted by a group of people who wanted an alternative to an AI CYOA platform, AIDungeon. They were frustrated with the site's current management and especially their decision to start banning users for writing NSFW content. We were almost out of money at that point, so we built a new [frontend](https://writeholo.com) for them over an EleutherAI model that encrypted user content and didn't filter. Surprisingly to us, the site ended up doing better than anything else we had tried so far, at least enough to tide us over. 

We worked on HoloAI for around six months after that. Then between Dec 2021 and April 2022, my model of AI R&D went from "neutral tech research" to "the worst thing you could possibly be doing". While we were not AI researchers, and in retrospect HoloAI was unlikely to have grown massively even if we had kept at it, we decided to stop making improvements. The site is still online, but nowadays I just answer support requests for the people that already use it as an editor. 

I still very much agree with our choice to switch to a field that was AI unrelated, but nowadays I'm not actually sure if the concerns about X-risk were real. We never did any foundational model research, we weren't paying OpenAI for tokens or competing against them for customers, and HoloAI didn't release code that could be used with other models. On the other hand, it seems plausible that products like HoloAI have a downstream effect, where by competing in the AI product category, you indirectly encourage others to make those innovations. We weren't paying OpenAI, but there *were* similar products on the market that did pay OpenAI, and now they have to do something stay ahead.

So what should be the social consensus about these sorts of utilities? Or for that matter, any software that uses proprietary models built by the research labs? 

Part of my reason for asking this question here is because I expect many of the new companies in the next 5-10 years to be, to some degree, AI productizations. We landed on HoloAI mostly by chance, but it's possible putting some kind of LLM in various parts of the economy is going to be a large proportion of how the technology sector grows going forward. Further, lots of people in the tech industry will be working on these tools or supporting them, and "doing something else" might not be something that's easy without maneuvering yourself around that growth.
# Estimating effective dimensionality of MNIST models

The [local learning coefficent](https://www.alignmentforum.org/posts/6g8cAftfQufLmFDYT/you-re-measuring-model-complexity-wrong) $\lambda$ is a measure of a model's "effective dimensionality" that captures its complexity (more background [here](https://www.alignmentforum.org/posts/4eZtmwaqhAgdJQDEg/dslt-1-the-rlct-measures-the-effective-dimension-of-neural)). [Lau et al](https://www.google.com/url?q=https://arxiv.org/pdf/2308.12108.pdf&sa=D&source=docs&ust=1698875186303624&usg=AOvVaw2uApbuzvUREOKooUY6jx58) recently described a sampling method (SGLD) using noisy gradients to find a stochastic estimate $\hat{\lambda}$ in a computationally tractable way (good explanation [here](https://www.lesswrong.com/posts/6g8cAftfQufLmFDYT/you-re-measuring-model-complexity-wrong#The_Local_Learning_Coefficient)):

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/8a584f89a235eeb23f7b97f9d58cb56b74012a40772e896e.png)

![](https://lh7-us.googleusercontent.com/_auw2hjJ_xc8MGtvlQygEx1oBD7Zf7R_xo0CinC1SC8GBmBJSFPNLmgBJAJqEBizuG1_ENVQLnhjS-M7p8ezQth5xUwIt8CEcFc6xSXrH1qQ2ldRR-taH0SjnAxYj_CxCT76-VpXOK2qvtE0Jyfbfazi6sUq-9BryMjUzJl6wEeuvk6mqYcSNx3VsgrFfw)

Examples of points sampled by SGLD in a two-dimensional toy model (Lau et al, 2023)

I present results ([Github repo](https://github.com/ArjunPanickssery/devinterp)) of the "Task Variability" project suggested on the [DevInterp project list](https://devinterp.com/projects/). To see how degeneracy scales with task difficulty and model class, I trained a fully-connected MLP and a CNN (both with ~120k parameters) on nine MNIST variants with different subsets of the labels (just the labels {0, 1}, then {0, 1, 2}, etc.). All models were trained to convergence using the same number of training data points. I implement Lau et al's algorithm on each of the trained models. The results below are averaged over five runs:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/57f853ba853be45dee49b2349772739b85f835c11679472e.png)The results for the full MNIST dataset are comparable to Lau et al's results while using models ten times smaller trained for ten times fewer epochs. 

The sampling method is finicky and sensitive to the hyperparameter choices of learning rate and noise factor. It will fail by producing negative or very high $\hat{\lambda}$ values if the noise $\epsilon$and the distance penalty $\gamma$ (see lines 6 and 7 in the pseudocode above) isn't calibrated to the model's level of convergence. The results show a linear scaling law relating number of labels to task complexity. The CNN typically has a lower $\hat{\lambda}$ than the MLP, which matches intuitions that some of the complexity is "stored" in the architecture because the convolutions apply a useful prior on functions good at solving image recognition tasks.
# The other side of the tidal wave

I guess there’s maybe a 10-20% chance of AI causing human extinction in the coming decades, but I feel more distressed about it than even that suggests—I think because in the case where it doesn’t cause human extinction, I find it hard to imagine life not going kind of off the rails. So many things I like about the world seem likely to be over or badly disrupted with superhuman AI (writing, explaining things to people, friendships where you can be of any use to one another, taking pride in skills, thinking, learning, figuring out how to achieve things, making things, easy tracking of what is and isn’t conscious), and I don’t trust that the replacements will be actually good, or good for us, or that anything will be reversible.

Even if we don’t die, it still feels like everything is coming to an end.
# Securing Civilization Against Catastrophic Pandemics

### Executive summary

Pandemic security aims to safeguard the future of civilization from exponentially spreading biological threats. Despite the world's failure to contain SARS-CoV-2, the existence of far more lethal and transmissible pathogens that afflict animals and growing access to increasingly powerful biotechnologies, no analyses of worst-case scenarios and potential defenses have been published. Here we outline two distinct mechanisms by which pandemic pathogens transmissible between humans could cause societal collapse. In a "Wildfire" pandemic, the justifiable fear of a lethal and highly contagious respiratory agent released in multiple travel hubs leads to the breakdown of essential services. In a "Stealth" pandemic, a rapidly spreading virus with a long incubation period analogous to HIV infects most of humankind. We explain why current pandemic preparedness measures such as rapid vaccines and N95 masks will reliably fail against these threats and outline novel strategies and technologies capable of safeguarding civilisation.

### Key takeaways

*   Nations cannot yet contain natural, accidental or deliberate pandemics.
*   Access to severe pandemics will expand with the ability to program biology.
*   If too many essential workers die or refuse to work, societies will collapse.
*   A Wildfire pandemic is highly lethal and transmissible enough to infect most essential workers who are taking currently available precautions.
    *   Collapse can be prevented by providing essential workers with pandemic-proof personal protective equipment (P4E). (Essential workers are those who must deliver food, water, power and law enforcement without any interruptions.)
    *   Others can remain safely at home until P4E is available for everyone.
    *   Once the population is protected, the virus can be locally eradicated.
*   A Stealth pandemic spreads widely with few symptoms and causes severe harm years later.
    *   Societal collapse can be prevented via early warning, credibility, cures, P4E and healthy buildings.
        *   Early warning: deep metagenomic sequencing offers reliable detection.
        *   Credibility: expert responders can assess threats and encourage action.
        *   Cures: swift medical research can offer hope for the infected.
        *   P4E: people will need protective equipment that they can trust to block transmission.
        *   Healthy buildings: the use of germicidal lights and ventilation can prevent indoor infections.
# Paul Christiano on Dwarkesh Podcast

Dwarkesh's summary:

> Paul Christiano is the world’s leading AI safety researcher. My full episode with him is out!
> 
> We discuss:
> 
> *   Does he regret inventing RLHF, and is alignment necessarily dual-use?
> *   Why he has relatively modest timelines (40% by 2040, 15% by 2030),
> *   What do we want post-AGI world to look like (do we want to keep gods enslaved forever)?
> *   Why he’s leading the push to get to labs develop responsible scaling policies, and what it would take to prevent an AI coup or bioweapon,
> *   His current research into a new proof system, and how this could solve alignment by explaining model's behavior
> *   and much more.
# ACX Montreal Meetup November 11th 2023

Hey all! This week, we're doing something a bit different. This ACX event will will resolve around the viewing a movie, *Threads,* and be hosted by Jeremy Eliosoff, who wrote the rest of this message. **Note the non-standard event time and location**.

**Description:**  
First broadcast on the BBC in 1984, Threads chronicles the lives of a few working class residents of the industrial British city of Sheffield in the events leading up to, during, and following a massive international nuclear exchange.  The product of a sometimes-contentious collaboration between a screenwriter known for gritty realism and a documentary film producer, Threads does not follow conventional plot structures and character arcs.  With no musical score and few contrived dramatic flourishes, the film is simply an explicit visualization of what the effects of full-scale nuclear war would actually look like, rendered with as much plausible realism as the producers could muster.  If anything, the understated, matter-of-fact tone of the film makes its impact all the more devastating.

Although technically fiction, Threads is possibly the most disturbing thing I have ever seen on a screen.  On my first viewing I had to watch it in 20 minute intervals, but we'll be doing the full 2 hours in one go.  Needless to say, viewer discretion is advised.  That said, I wanted to present this movie to the ACX community, partly simply to raise awareness of what I consider and unsung cinematic masterpiece, and partly because of its relevance to the subject of existential threats, which is a mainstay of AXC/rationalist/effective altruist concerns - and which, sadly, has become more salient in light of recent world events.  Nov 11 seemed a particularly appropriate day for this screening.

A conversation will follow the screening.

RSVP is advised.  My condo can accommodate 10-12 people, I can try to squeeze in more if needed but cannot guarantee admission to anyone who has not RSVP'ed.

Also note that this is my first attempt at setting up my home projection system so things might go wrong, also my projector has a fairly loud fan.

**Venue:** 4243 Rue Clark.  
**Date & Time: **Saturday, November 11th, 2023, 6:00 PM.

**Max participants**: 12

**EDIT: Also, important to note that food won't be served, and that you're welcome to bring snacks! I (Henri) will bring what I can, but try not to come with an empty stomach. :)**

![image.png](https://mail.google.com/mail/u/0?ui=2&ik=793d50d4ec&attid=0.1&permmsgid=msg-f:1781126123736585445&th=18b7d4c5d17cd4e5&view=fimg&fur=ip&sz=s0-l75-ft&attbid=ANGjdJ9Z1CdcQ3PQfjQQYGhPOKiotpSw9WfbP0JwN7uh7d4SBIwhVG1TInEI3cUzmABRMhjl0a7tfGmxsKs4K4dglOuCYvdnAmw7BP6g5AkIPXkuitwmB71TuK5iRVI&disp=emb&realattid=ii_lobyymtv1)
# [Linkpost] Concept Alignment as a Prerequisite for Value Alignment

> Value alignment is essential for building AI systems that can safely and reliably interact with people. However, what a person values -- and is even capable of valuing -- depends on the concepts that they are currently using to understand and evaluate what happens in the world. The dependence of values on concepts means that concept alignment is a prerequisite for value alignment -- agents need to align their representation of a situation with that of humans in order to successfully align their values. Here, we formally analyze the concept alignment problem in the inverse reinforcement learning setting, show how neglecting concept alignment can lead to systematic value mis-alignment, and describe an approach that helps minimize such failure modes by jointly reasoning about a person's concepts and values. Additionally, we report experimental results with human participants showing that humans reason about the concepts used by an agent when acting intentionally, in line with our joint reasoning model.

> We propose a theoretical framework for formally introducing concepts to inverse reinforcement learning and show that conceptual misalignment (i.e., failing to consider construals) can lead to severe value misalignment (i.e., reward mis-specification; large performance gap). We validate these theoretical results with a case study using a simple gridworld environment where we find that IRL agents that jointly model construals and reward outperform those that only model reward. Finally, we conduct a study with human participants and find that people do model construals, and that their inferences about rewards are a much closer match to the agent that jointly models construals and rewards. Our theoretical and empirical results suggest that the current paradigm of just trying to directly infer human reward functions or preferences from demonstrations is insufficient for value-aligning real AI systems that need to interact with real people; it is crucial to also model and align on the concepts people use to reason about the task in order to understand their true values and intentions.
# Utility is not the selection target

*Epistemic status: shitpost.*

Suppose that you are selecting for utility. Naively, you might think that this means you are selecting for utility, but actually this is not the case.

Sometimes greenbrownness is the selection target
------------------------------------------------

Military gear is sort of greenbrownish.

![Military Camouflage - how camouflage works & the science behind it | UF PRO  Blog](https://ufpro.com/storage/app/media/Blog/Military%20camouflage/military-camouflage-science-hero.jpg)

This occurs because the designers of military gear are selecting for keeping soldiers alive, which benefits from camouflage, which in the common environments works best if it is greenbrownish. However, it fails at this in nongreenbrownish environments. Hence, keeping soldiers alive is not the selection target of military gear.

Sometimes disutility is the selection target
--------------------------------------------

In the prisoner's dilemma, the highest-utility outcome is (cooperate, cooperate).

![The pay-off matrix in the Prisoners' Dilemma game. The first entry... |  Download Scientific Diagram](https://www.researchgate.net/publication/2184338/figure/fig1/AS:671529209692164@1537116448235/The-pay-off-matrix-in-the-Prisoners-Dilemma-game-The-first-entry-refers-to-Alices.png)

However, utility maximizers will end up in (defect, defect), which is strictly lower utility than (cooperate, cooperate). Thus, utility maximization might have utility minimization as the selection target.

Sometimes the superficial appearance of utility is the selection target
-----------------------------------------------------------------------

Let's say that you see a website banner saying "You are the 1000000000th visitor to the site. Click to receive your award!". This looks like something people would say if they want to give you something, so you click the banner.

![Electronic Arts on X: "@GameStop https://t.co/ELEQRcboMQ" / X](https://pbs.twimg.com/media/EDOJNLjUcAIgFrf.jpg)

This does not end well for you, but you went into it selecting it because it looked like it would end well for you. 

Conclusion
----------

Beware about reasoning about utility maximizers as maximizing utility. Utility maximizers may instead be maximizing many other things that are unrelated to utility, and not be maximizing utility.
# Go flash blinking lights at printed text right now

1. Install a strobe light app on your phone (I used "strobe light tachometer" on ios.)
2. Get some printed text like a nutrition label or a paper book
3. Try to read the text with the light blinking at 5, 6, 7, 8, 9, 10, 11, 12, and 13 hz
4. Report results. Was reading with one frequency much easier than the others? Easier than with normal constant light? Were they all terrible?

---

I just tried this and my eyes totally glazed over for everything except 12! I was feeling kinda tired and unfocused when I started and 12 was easier than baseline for me. Although annoying. I have incandescent lights in my kitchen. I did not turn them off. The phone light was brighter than the incandescent light.

Framerate stuff: 

- Phone flashlights are not meant to blink really fast (and max frequency is not documented) but the app seemed to work ok. I did notice an irregularity here and there but I don't think that 12 being an even divisor of the flashlight frequency or something mattered.
- If you have a 60hz monitor, then your screen can do a strobe light with 12, 10, 8.6, 7.5, or 6.7 hz. This is not very good resolution. A 120hz monitor should be good enough.
- Would be better to test with a dedicated strobe light so this is not an issue.

More info about this experiment on first item [here](https://www.astralcodexten.com/p/quests-and-requests)
# Eric Schmidt on recursive self-improvement

Recently, Eric Schmidt gave a talk at Harvard called “Our AI Future: Hopes and Hurdles Ahead”. The entire talk is available [here](https://www.youtube.com/watch?v=IeVY_Ag8JI8) but there was one part that was interesting to me (around the 1:11:00 mark), and that is, his views on AI safety and trust in AI labs to stop scaling if recursive self-improvement starts happening. **Emphasis my own.**

> The technical term is recursive self-improvement. **As long as the system isn’t busy learning on its own, you’re okay.** I’ll parrot Sam’s speech at OpenAI. \[...\]
> 
> In the next few years we’ll get to AGI \[...\] Some number of years after that, the computers are going to start talking to each other, probably in a language that we can’t understand, and collectively their superintelligence \[...\] is going to rise very rapidly. **My retort to that is, do you know what we’re going to do in that scenario? We’re going to unplug them all.** 
> 
> The way you would do this if you were an evil person \[...\] is you would simply say to the computer: “Learn everything. Try really hard. Start right now.” So the computer starts learning. It learns about French, it learns about science, it learns about people, it learns about politics. It does all of that, and at some point, it learns about electricity and it learns that it needs electricity, and it decides it needs more. So it hacks into the hospital and takes the electricity from the hospital to give it to itself.
> 
> That’s a simple example of why this is so dangerous. **Most people in the industry who thought about this believe that when you begin recursive self-improvement, there will be a very serious regulatory response** because of these issues. And that makes sense to me.
# xAI announces Grok, beats GPT-3.5

Some highlights:

> *Grok is still a very early beta product – the best we could do with 2 months of training – so expect it to improve rapidly with each passing week with your help.*

I find it very interesting that they managed to beat GPT-3.5 with **only 2 months of training!** This makes me think xAI might become a major player in AGI development.

> By creating and improving Grok, we aim to:
> 
> *   Gather feedback and ensure we are building AI tools that maximally benefit all of humanity. We believe that it is important to design AI tools that are useful to people of all backgrounds and political views. We also want empower our users with our AI tools, subject to the law. Our goal with Grok is to explore and demonstrate this approach in public.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/32b2e7a9fd5b2e2e99aef684fe88c623f42aba08e3b40a77.png)

> On these benchmarks, Grok-1 displayed strong results, surpassing all other models in its compute class, including ChatGPT-3.5 and Inflection-1. It is only surpassed by models that were trained with a significantly larger amount of training data and compute resources like GPT-4. This showcases the rapid progress we are making at xAI in training LLMs with exceptional efficiency.

> We believe that AI holds immense potential for contributing significant scientific and economic value to society, so we will work towards developing reliable safeguards against catastrophic forms of malicious use. We believe in doing our utmost to ensure that AI remains a force for good.
# Announcing TAIS 2024

[AI Safety Tokyo](https://aisafety.tokyo/) is hosting TAIS 2024, a Technical AI Safety Conference. The conference will take place in Tokyo, Japan on April 5th, 2024. Details about the event can be found [here](https://tais2024.cc/).

The goal of this conference is to bring together specialists in the field of AI and technical safety to share their research and benefit from each others’ expertise. We seek to launch this forum for academics, researchers and professionals who are doing technical work in these or adjacent fields:

*   Mechanistic interpretability
*   Scalable oversight
*   Causal incentive analysis
*   Agent foundations
*   Singular learning theory
*   Argumentation
*   Emergent agentic phenomena
*   Thermodynamic / statistical-mechanical analyses of computational systems

TAIS 2024, being hosted in Tokyo, will allow access to Japanese research and specialists (singular learning theory, collective / emergent behaviour, artificial life and consciousness), who are often overlooked outside of Japan. We want to help people connect to the Japanese well of information, and make connections with other individuals to share ideas and leap forward into greater collaborative understanding. 

We want our attendees to involve themselves in cutting-edge conversations throughout the conference with networking opportunities with the brightest minds in AI Safety. 

We will announce the full schedule for the conference in the coming months. If you’re interested in presenting your research, please answer our [call for presentations](https://tais2024.cc/#call-for-presentations).

This event is free, but limited to 150 people, so if you wish to join please [sign up here](https://docs.google.com/forms/d/e/1FAIpQLSeELqHsfLaAGZp6f8aO4wL21MwryWolKL2luGPqS5pqNxkxbQ/viewform)**.**

TAIS 2024 is sponsored by [Noeon Research](https://noeon.ai/).
# What is true is already so.
Owning up to it doesn't make it worse.

I had a realization over the last few months that I'd like to share. I have always interpreted the [Litany of Gendlin](https://www.lesswrong.com/tag/litany-of-gendlin) to compress to something like "there's no sense hiding from bad news." What I realized is that there's a reversed interpretation that's also true, correct, and useful to remember, which is that finding out more about bad things going on in the world doesn't mean that the world is getting worse.

I think a big problem out there is that people get really revved up on social media (or traditional media) about all the negativity, and it naturally leads people to believe that things are looking down. But most of the negativity is selected *because it's negative*, and so in a big world with lots of good and bad stuff happening, seeing more negativity in your social media feed just is not any evidence one way or the other about whether things are in fact getting better or worse. People who are living their best lives are hanging out with their friends, playing with their kids, eating great food, or going for hikes or whatever. Not posting on social media!
# Job listing: Communications Generalist / Project Manager

Looking for a way to help communicate about AI x-risk? MIRI is hiring in the Communications department. There will be additional job listings soon for writers and editors, but we're starting with this comms generalist / project manager role.

[https://intelligence.org/careers/comms-generalist-pm/](https://intelligence.org/careers/comms-generalist-pm/)

I am the Communications Manager at MIRI and this person will be working closely with me. I'm happy to answer questions.
# AMA: Earning to Give

This week the [Effective Altruism Forum](https://forum.effectivealtruism.org/) is running an [Effective Giving Spotlight](https://forum.effectivealtruism.org/posts/hAzhyikPnLnMXweXG/participate-in-the-donation-election-and-the-first-weekly#First_weekly_theme__Effective_Giving_Spotlight__November_7_14_), and they asked if I could post an [Ask Me Anything](https://forum.effectivealtruism.org/posts/Gmc9sieTJLfRTtcwR/running-an-ama-on-the-ea-forum) (AMA) on my experience earning to give.

Some background:

*   I was earning to give from [2009](https://www.jefftk.com/p/giving) to [2022](https://www.jefftk.com/p/leaving-google-joining-the-nucleic-acid-observatory), except for a few months in 2017 when I worked on [expanding access to the financial system in Ethiopia](https://www.jefftk.com/p/leaving-google-joining-wave) and [looking into AI risk disagreements](https://www.jefftk.com/p/superintelligence-risk-project-conclusion).
    
*   I've been a Giving What We Can member since [2013](https://www.jefftk.com/p/joining-giving-what-we-can), making a [public](https://www.jefftk.com/p/make-your-giving-public) pledge to continue with effective giving.
    
*   For most of this time [my wife](https://juliawise.net/) and I were donating [50%](https://www.jefftk.com/p/giving-half) of our [pre-tax income](https://www.jefftk.com/p/what-should-income-mean-in-pledging), for a total of [$2.1M](https://www.jefftk.com/donations). This has been about 50-50 between trying to help the EA community grow into the best version of itself and funding global poverty reduction ([details](https://www.jefftk.com/donations), [thoughts](https://www.jefftk.com/p/why-global-poverty), [more recent but still obsolete thoughts](https://www.jefftk.com/p/revisiting-why-global-poverty)).
    
*   In 2016 I gave a EA Global talk ([transcript](https://www.jefftk.com/p/earning-to-give-transcript)) on earning to give, which gives more background on the idea and how I've neen thinking about it.
    

That's a lot of links, and it's fine to ask questions even if you haven't read any of them! I'm happy to take questions on earning to give, or anything else within EA. Here are some example questions I'd be happy to answer if there's interest:

*   Where do individual donors earning to give have an advantage over foundations and funds?
    
*   How should you decide whether to use a fund?
    
*   How have I thought about how much to donate? How much is enough?
    
*   Why did I stop earning to give?
    
*   Why am I still donating some even though I'm funded by EA donors?
    

Feel free to comment on any platform, but if you're having trouble deciding then the [EA Forum post](https://forum.effectivealtruism.org/posts/L8N4uh4GixhWCoGAg/ama-earning-to-give) is ideal.

*Comment via: [the EA Forum](https://forum.effectivealtruism.org/posts/L8N4uh4GixhWCoGAg)*
# Vote on Interesting Disagreements

Do you have a question you'd like to see argued about? Would you like to indicate your position and [discuss](https://www.lesswrong.com/dialogues) it with someone who disagrees? 

Add poll options to [the thread below](https://www.lesswrong.com/posts/hc9nMipTXy2sm3tJb/disagreements-elicitation-thread?commentId=NtsPs9wcwrpeK6KYL) to find questions with lots of interest and disagreement.

How to use the poll
-------------------

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/b75596d2bbed9af3be19585ef6fb2addea7ca816873cf5bb.png)

1.  **Reacts:** Click on the **agree/disagree reacts** to help people see how much disagreement there is on the topic.
2.  **Karma:** Upvote positions that **you'd like to read dialogues** about.
3.  **New Poll Option:** Add new positions for people to take sides on. **Please add the agree/disagree reacts** to new poll options you make.

The goal is to show people where a lot of interesting disagreement lies. This can be used to find discussion and [dialogue](https://www.lesswrong.com/dialogues) topics in the future.
# South Bay ACX/LW/EA Meetup & Vegansgiving Potluck

Ilmarin House is hosting an ACX/LW/EA meetup and plant-based potluck. These are sort of parallel events; the meetup will take place as usual (ie unstructured conversation) except those who want to are invited to join us at the dinner table at five. 

If you can't bring a dish (though that's encouraged), you're still welcome to have dinner with us. Either way, please let us know you're coming and what dish you'll bring (if any). If you do bring something (thanks!), please make sure it doesn't contain animal products and enter it [in the spreadsheet](https://docs.google.com/spreadsheets/d/1MYVK_plNVxHk_3RNLurUzYBa4H1qE2PgW-nBWORnkJ0/edit?usp=sharing).

As always, kids are welcome! Bring some friends if you think they'd have fun. If you have questions or comments, please let us know at svmeetup@protonmail.com.
# Impressions from base-GPT-4?

I wonder if some people here had a chance to play with base-GPT-4 (the access is given very selectively for research purposes) and would not mind sharing some of their impressions? 

I know that some people have been playing with it, but I've never seen a discussion of impressions and lessons from that. And I know that it is quite nontrivial to get access to this model, but that some access is given.

I think it would be super-interesting for many people here to hear this kind of conversation...
# Is there a lesswrong archive of all public posts?

like for example: the [wikimedia db dumps](https://dumps.wikimedia.org/), or the [stack exchange db dumps](https://archive.org/details/stackexchange).

i'd like to be able to browse lesswrong while offline. even better if i can process the data with a script while offline.

it would also be useful for backup purposes: if ever something happens to the site in the longterm, some of it's users might have an exact copy that can be corss-compared and used to restore/share its content.

related:

*   [Why is lesswrong blocking wget and curl (scrape)?](https://www.lesswrong.com/posts/gidrFxE5hdQWCrXxn/why-is-lesswrong-blocking-wget-and-curl-scrape)
*   [Can I archive content from lesswrong.com on the wayback machine (internet archive, archive.org) ?](https://www.lesswrong.com/posts/LBXGBgRENPHg2THbr/can-i-archive-content-from-lesswrong-com-on-the-wayback)