Harley-ml/lesswrong huggingface dataset:
# ‘We’re changing the clouds.’ An unforeseen test of geoengineering is fueling record ocean warmth

For decades humans have been emitting carbon dioxide into the atmosphere, creating a greenhouse effect and leading to an acceleration of the earth's warming. 

At the same time, humans have been emitting sulphur dioxide, a pollutant found in shipping fuel that has been responsible for acid rain. Regulations imposed in 2020 by the United Nations’s International Maritime Organization have cut ships’ sulfur pollution by more than 80% and improved air quality worldwide.

Three years after the regulation was imposed, scientists are realizing that sulphur dioxide has a sunscreen effect on the atmosphere, and by removing it from shipping fuel we have inadvertently removed this sunscreen, leading to an acceleration in temperature in the regions where global shipping operates the most: the North Atlantic and the North Pacific. 

We've been accidentally geoengineering the earth's climate, and the mid to long term consequences of removing those emissions are yet to be seen. At the same time, this accident is making scientists realize that with not much effort we can geoengineer the earth and reduce the effect of greenhouse gas emissions.
# Announcing the Clearer Thinking micro-grants program for 2023

This is a link-post for the [announcement on our website.](https://www.clearerthinking.org/post/submit-an-idea-for-a-clearer-thinking-tool-and-get-paid-up-to-595 )

  
Do you have an idea for an educational module or tool related to rationality, effective altruism, or psychology that could benefit the general public? If you do, we'd love to hear from you, because we're running our [Clearer Thinking micro-grants program](https://www.clearerthinking.org/post/submit-an-idea-for-a-clearer-thinking-tool-and-get-paid-up-to-595) again! You can earn up to $595 for proposing, designing, and/or building (with our help) a short online tool aligned with our mission. To apply, just submit your idea for a beneficial learning tool in under 400 words.  
  
[Here is the link to apply.](https://programs.clearerthinking.org/micro_grant_2023_stage1.html?source=lesswrong)
# Growing Bonsai Networks with RNNs

![A visualization of a sparse computational graph pruned from a RNN. Square nodes represent neurons and circles are states from the previous timestep. Nodes and edges are according to their current output with blue being negative and red positive."/><meta data-react-helmet="true" name="twitter:image:alt" content="A visualization of a sparse computational graph pruned from a RNN. Square nodes represent neurons and circles are states from the previous timestep. Nodes and edges are according to their current output with blue being negative and red positive.](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/d9234641ed89f49008ff849fc9628af1ab94349ac16eb676.png)

This is a linkpost for a writeup on my personal website: [https://cprimozic.net/blog/growing-sparse-computational-graphs-with-rnns/](https://cprimozic.net/blog/growing-sparse-computational-graphs-with-rnns/)

Here's a summary:

> This post contains an overview of my research and experiments on growing sparse computational graphs I'm calling "Bonsai Networks" by training small RNNs. It describes the architecture, training process, and pruning methods used to create the graphs and then examines some of the learned solutions to a variety of objectives.
> 
> Its main theme is mechanistic interpretability, but it also goes into significant detail on the technical side of the implementation for the training stack, a custom activation function, bespoke sparsity-promoting regularizer, and more.

The site contains a variety of interactive visualizations and other embeds that are important to its content.  That's why I chose to make this a linkpost rather than copy its content here directly.

I'd love to receive any feedback you might have on this work.  This topic is something I'm very interested in, and I'm eager to hear peoples' thoughts on it.
# Notice your everything

\[epistemic status: mostly me noticing (ha) a common pattern among advice i've heard and my personal experience.\]

There is one skill or category of skill that is probably worth more than every other general skill of instrumental rationality on this website combined: Noticing things.

You probably already know about [noticing confusion](https://www.lesswrong.com/s/zpCiuR4T343j9WkcK), noticing whenever some hypothesis feels "off" or seems inconsistent with your previous beliefs. You probably also already know to notice whenever you're rationalizing or engaging in motivated reasoning.

The remarkable part is that neither of these skills requires you to have a high IQ or learn a lot of new object-level knowledge. At the very least, high IQ or object-level knowledge is not sufficient for these skills. You only need to listen to what your brain is already telling you. 

Noticing isn't just for identifying when your thinking is bad, you can also use it to identify when your thinking is good.   
  
Noticing your thoughts or [processes of moving from thought to thought](https://www.lesswrong.com/posts/bbB4pvAQdpGrgGvXH/tuning-your-cognitive-strategies) allows you to pat yourself on the back and update your cognition or notice which ones aren't doing good work.  
  
Noticing your desires helps you determine whether you should act on them or not. You can act on desires you previously didn't notice, or notice desires you don't want to act on and simply sit and watch them disappear.  
  
Noticing your [emotions](https://www.lesswrong.com/tag/focusing) helps you turn feelings and impulses into rational decisions, or analyze them to figure out their cause.  
  
[Babble & Prune](https://www.lesswrong.com/s/pC6DYFLPMTCbEwH8W) follows a similar pattern, where Babble is about noticing ideas, and Prune is about carefully analyzing them, instead of just automatically generating and dismissing them in a single moment.  
  
Noticing is [bewelltuned.com](http://bewelltuned.com/)'s whole deal.  
  
I think most of the benefits of meditation, for me at least, come from it being a way of training your noticing, which incurs the benefits I listed above.  
  
In general, noticing is about taking what you perceive, whether a thought or an emotion or a physical sensation or a subagent or a process or any other mental object, and taking it seriously, not dismissing it out of hand or blindly following it or suppressing it or never having it enter your attention in the first place.  
  
This post is a call to action, not only to notice your everything but also to get people to start talking about noticing and how to notice things better. I'd also like to see what work people have already done that I don't already know about. Who knows, maybe there are already people working on this right now to the level of generality I'm asking for, and I just haven't found them yet.  
  
\[Edit: Justis, who provided feedback before I posted this, says that a lot of stuff by [Spencer Greenberg](https://www.spencergreenberg.com/) and a few things from [Meaningness](https://meaningness.com/) are about noticing.\]
# Yet more UFO Betting: Put Up or Shut Up

*(Edit: I am not the user* [*RatsWrongAboutUAP*](https://www.lesswrong.com/users/ratswrongaboutuap)*.)*

I'm offering a new bet along similar lines to Yudkowsky's bet with the user [RatsWrongAboutUAP](https://www.lesswrong.com/users/ratswrongaboutuap) in [this thread](https://www.lesswrong.com/posts/t5W87hQF5gKyTofQB/ufo-betting-put-up-or-shut-up) last month. My bet has these similarities (here added explicitly after some confusion in the comments):

*   I will pay out immediately upon our agreement to the bet, whereas the other party only needs to pay out if and when they lose the bet. **I am not requiring the other party to put their funds in escrow.**
*   I am **not** asking for the other party to reveal private financial information (though doing so may allow me to give better odds than 150:1)
*   I am willing to make multiple bets with different people.
*   The odds I'm offering are 150:1.

My bet has these differences:

*   The time frame will be 3 years from the day the bet is agreed to.
*   The bet will resolve in my favor if a non-prosaic origin of the existing evidence, **or** new evidence, is confirmed in this time frame. *Edit: the requirement of confirmation of evidence (by the same standards that Yudkowsky agreed to) applies both to existing evidence and any evidence that may emerge during the time interval.*
*   My wager is $1000. If I make multiple bets, this will be the total wager.

**EDIT:** If you can document liquid assets, I will go down to 100:1. The limit to odds I will accept is at 75:1, but that would only be if the other wager was put in escrow.

I'd like to discuss details like credibility of payback, but first I want to see if there's any interest. My suspicion is that LW users are more afraid to make this bet now than they were in July, but we'll see.
# Open Thread - August 2023

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the [Highlights from the Sequences](https://lesswrong.com/highlights), a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend [reading the Library](https://www.lesswrong.com/library), [checking recent Curated posts](https://www.lesswrong.com/?view=curated), [seeing if there are any meetups in your area](https://www.lesswrong.com/community), and checking out the [Getting Started](https://www.lesswrong.com/faq#Getting_Started) section of the [LessWrong FAQ](https://www.lesswrong.com/faq). If you want to orient to the content on the site, you can also check out the [Concepts section](https://www.lesswrong.com/tags/all).

The Open Thread tag is [here](https://www.lesswrong.com/tag/open-threads?sortedBy=new). The Open Thread sequence is [here](https://www.lesswrong.com/s/yai5mppkuCHPQmzpN).
# ACX Montreal Meetup August 19th 2023

**Description:**

Come on out to the ACX (Astral Codex Ten) Montreal Meetup! This week, we're keeping it open-ended for a freewheeling discussion.

**Venue**: L'Esplanade Tranquille, 1442  Clark, indoors, second floor.

**Date & Time:** Saturday, August 19th, 2023, 1:00-4:00 PM.

Please RSVP! It is not mandatory but helps with planning the event and encourages others to come.

Use [this form](https://forms.gle/CUqzSzAxJ4yfJtpL6) to be added to the ACX Montreal mailing list.
# Predicting Virus Relative Abundance in Wastewater

At the [Nucleic Acid Observatory](https://naobservatory.org/) (NAO) we're evaluating pathogen-agnostic surveillance. A key question is whether metagenomic sequencing of wastewater can be a cost-effective method to detect and mitigate future pandemics. In this report we investigate one piece of this question: at a given stage of a viral pandemic, what fraction of wastewater metagenomic sequencing reads would that virus represent?

To make this concrete, we define RA(1%). If 1% of people are infected with some virus (prevalence) or have become infected with it during a given week (incidence), RA(1%) is the fraction of sequencing reads (relative abundance) generated by a given method that would match that virus. To estimate RA(1%) we collected public health data on sixteen human-infecting viruses, re-analyzed sequencing data from four municipal wastewater metagenomic studies, and linked them with a hierarchical Bayesian model.

Three of the viruses were not present in the sequencing data, and we could only generate an upper bound on RA(1%). Four viruses had a handful of reads, for which we were able to generate rough estimates. For the remaining nine viruses we were able to narrow down RA(1%) for a specific virus-method combination to approximately an order of magnitude. We found RA(1%) for these nine viruses varied dramatically, over approximately six orders of magnitude. It also varied by study, with some viruses seeing an RA(1%) three orders of magnitude higher in one study than another.

The NAO plans to use the estimates from this study as inputs into a modeling framework to assess the cost effectiveness of wastewater MGS detection under different pandemic scenarios, and we include an outline of such a framework with some rough estimates of the costs of different monitoring approaches.

Read the full report: [Predicting Virus Relative Abundance in Wastewater](https://naobservatory.org/reports/predicting-virus-relative-abundance-in-wastewater/).
# Seeking Input to AI Safety Book for non-technical audience

**TLDR:**  You are invited to make suggestions for ideas/information/framing for an upcoming AI safety book for a non-technical audience.   
  
**Context:**  
I'm still [writing an accessible book about AI Safety/Risk](https://www.lesswrong.com/posts/E5MwmuuryBLF3wWRZ/fyi-i-m-working-on-a-book-about-the-threat-of-agi-asi-for-a) for a non-technical audience to serve both the AI safety cause and the community. The book's intended audience is likely not people who read this forum but rather your friends, family, policy makers, and non-science people who are curious about the topic.  I started last June, it is pretty far along, and I hope to have it available within the next three months (I received a LTFF grant last year to help it come into existence).   
  
Briefly, the purpose of the book is to communicate that intelligence is really powerful, AI progress is happening fast and AI systems are becoming more intelligent/powerful, that advanced AI is a threat/risk to humanity because it may not be aligned with our values and may be uncontrollable, therefore we should act now to reduce the risk.    
  
**Opportunity:**  
You can present ideas, facts, framing, or anything else you think would be important in such a book in the comments below or send me a message.    
If interested, you may still be wondering what I've already included. Broadly, as a heuristic, if your idea is very obvious, I'm probably already including it. But it could be still be useful for you to suggest it, so I can see that others think it is important.   
If your idea is highly technical, I have likely chosen not to include it. But it could still be useful to suggest if it is a key consideration that can be made more accessible. I'm trying to open-minded but efficient with people's time.    
I am also trying to minimize the occurrence of someone saying "I really wish he had mentioned X" after the book comes out.  No promises of inclusion but at least your suggestions will be considered.   
  
Finally, I'm more than happy to have people be more involved as getting feedback from a range of knowledgeable people is useful for a variety of reasons.
# What Does a Marginal Grant at LTFF Look Like? Funding Priorities and Grantmaking Thresholds at the Long-Term Future Fund

The Long-Term Future Fund (LTFF) makes small, targeted grants with the aim of improving the long-term trajectory of humanity. We are currently fundraising to cover our grantmaking budget for the next 6 months. We would like to give donors more insight into how we prioritize different projects, so they have a better sense of how we plan to spend their marginal dollar. Below, we’ve compiled **fictional** **but representative **grants to illustrate what sort of projects we might fund depending on how much we raise for the next 6 months, assuming we receive grant applications at a similar rate and quality to the recent past. 

Our motivations for presenting this information are a) to provide transparency about how the LTFF works, and b) to move the EA and longtermist donor communities towards a more accurate understanding of what their donations are used for. Sometimes, when people donate to charities (EA or otherwise), they may wrongly assume that their donations go towards funding the *average*, or even more optimistically, the *best* work of those charities. However, it is usually more useful to consider the marginal impact for the world that additional dollars would buy. By offering illustrative examples of the sort of projects we might fund at different levels of funding, we hope to give potential donors a better sense of what their donations might buy, depending on how much funding has already been committed. We hope that this post will help improve the quality of thinking and discussions about charities in the EA and longtermist communities.

\[...\]

see the full post [here](https://forum.effectivealtruism.org/posts/7RrjXQhGgAJiDLWYR/what-does-a-marginal-grant-at-ltff-look-like-funding)
# How to decide under low-stakes uncertainty

Say you're stuck in uncertainty between two actions you're considering. They seem about equally good, but you suspect one is better, and it's not obvious which. You already have all the information to obviously collect about the problem.

For situations where getting it right really matters, try harder to get more information, and use methods more reliable than those presented here.

For lower-stakes problems:

1.  Assign one option to 0/tails and the other to 1/heads.
2.  Flip a coin, i.e. use a one-bit random number generator.
3.  Start accepting the decision from the coinflip, and observe your revealed unconscious intuition, for you will probably have some.
4.  If that intuition opposes the coin's decision, go with the intuition. Otherwise, you either actively came to agree with the random decision, or it still seems terribly uncertain, so you go with the random choice.

This is classic advice. I'm just sharing it. The "innovation" here is a way to handle cases where it's inconvenient to flip a coin or do the equivalent.

Mental coinflip
===============

1.  Assign one option to 0/tails and the other to 1/heads.
2.  Think about something unrelated.
3.  Notice the first word that pops into your mind.
4.  Count letters in that word. If it's even, you get 0; if it's odd, you get 1.
5.  Use that 0 or 1 to continue at step 3 of the general procedure.

This is very crude randomness, but it doesn't really have to be random, just uncorrelated with the topic of decision. You should not use a mental coinflip to try to run a fair game of chance.

There may be a bias in the length-parity of words you think of. I, so far, haven't noticed one. But that wouldn't necessarily be a problem. Once you notice such a bias, you can exploit it by assigning actions to 0/1 so that the mental coinflip bias opposes a bias you naturally have.

E.g. if you pick odd-length words more often, when the choice is between "doing nothing" and "doing something", assign "doing something" to 1. Laziness may lead you to want to "do nothing" more often than you should.

More than two options
=====================

You can use the same methods here for decisions more complicated than a dichotomy.

1.  List the options you're considering.
2.  Split that list into two subsets.
3.  Apply the coinflip-assisted decision procedure to pick one subset.
4.  If the selected subset has just one item, that's your decision. Otherwise, repeat this procedure on the selected subset.
# [Linkpost] Personal and Psychological Dimensions of AI Researchers Confronting AI Catastrophic Risks

*This is a linkpost for* [*https://yoshuabengio.org/2023/08/12/personal-and-psychological-dimensions-of-ai-researchers-confronting-ai-catastrophic-risks*](https://yoshuabengio.org/2023/08/12/personal-and-psychological-dimensions-of-ai-researchers-confronting-ai-catastrophic-risks)[*. *](https://yoshuabengio.org/2023/08/12/personal-and-psychological-dimensions-of-ai-researchers-confronting-ai-catastrophic-risks.)

Yoshua Bengio: 

> For most of these years, I did not think about the dual-use nature of science because our research results seemed so far from human capabilities and the work was only academic. It was a pure pursuit of knowledge, beautiful, but mostly detached from society until about a decade ago. I now believe that I was wrong and short-sighted to ignore that dual-use nature. I also think I was not paying enough attention to the possibility of losing control to superhuman AIs.

> \[...\] it started to dawn on me that my previous estimates of when human-level AI would be reached needed to be radically changed. Instead of decades to centuries, I now see it as 5 to 20 years with 90% confidence.  
> And what if it was, indeed, just a few years?

> I started reading more about AI safety and came to a critically important conclusion: we do not yet know how to make an AI agent controllable and thus guarantee the safety of humanity! And yet we are – myself included until now – racing ahead towards building such systems.

> It is painful to face the idea that we may have been contributing to something that could be greatly destructive. Human nature will lead us towards brushing aside these thoughts or finding comfort in reassuring arguments rather than face the full horror of such possibilities. Bringing the benefits of AI to the table is not sufficient to compensate if the possible negative outcomes include catastrophic misuses of AI on par with nuclear war and pandemics, or even existential risk.

> As scientists, we should avoid making claims we can’t support; but as decision-makers we also ought to act under uncertainty to take precautions. In spite of our differences in points of view, it’s time for our field of AI to seriously discuss the questions: what if we succeed? What if potentially dangerous superhuman AI capabilities are developed sooner than expected? Let’s embrace these challenges and our differences, while being mindful of each other’s humanity and our unique emotional and psychological journeys in this new era of AI.
# Stepping down as moderator on LW

Just a brief announcement that I'm stepping down as a moderator on LW. The reason is that I've been doing [emotional coaching](https://tinyurl.com/KajCoaching) for some time now, and several of my clients are a part of the rationalist / effective altruism communities. This often involves them sharing intimate personal details about what's happening with them. While this hasn't created any conflict of interest so far, I feel that I should not hold an official position of power within the community (like being an LW mod) while also having a therapist-type role for people within that same community.

In practice, I haven't used my mod powers much aside from occasionally deleting spam or answering questions about the site that people have messaged me with, so this doesn't change very much.

I do intend to continue posting normally and feel good about my time with the mod team. :)
# What is the most effective anti-tyranny charity?

Specifically: Big people hurting little people. Where can I get my largest bang per buck? I could donate to generic libertarian organizations, but I don't think they fight the generic brand of tyranny I want to fight, and my guess is that most of them are ineffective.
# Any research in "probe-tuning" of LLMs?

Is there any research in "probe-tuning" of LLMs, i.e., tuning LLM's parameter weights such that a specific [probe (classifier)](https://arxiv.org/abs/1610.01644) is more reliably detecting certain markers throughout the context, such as grammatical errors, aggression, manipulation, certain political bias, etc.?

This is different from classical fine-tuning and RLHF. As well as classical fine-tuning, probe-tuning is a supervised ML method: it is based on human-annotated texts (contexts). However, probe-tuning should be more effective than classical fine-tuning for detecting many occurrences of a certain marker throughout the context. Probe-tuning doesn't train on LLM's own "original rollouts" at all, only on LLM's activations during the context pass through the LLM.

I imagine than before doing actual probe-tuning, first we should determine which probe in the LLM is most aligned to the training data (annotations) already, so that probe-tuning likely just attenuates some vaguely existing concept within the LLM.
# Private Biostasis & Cryonics Social

**Note**: I'm posting on the behalf of the organizers; I'm not planning to attend myself.

> Join us for a private meeting with small plates and drinks on the last day of Longevity Summit Dublin on August 20 . With short impulse talks about cryonics and its impact on longevity.
> The evening will provide an excellent opportunity to learn more about this upcoming mega trend, share insights, ideas, and build new relationships with like-minded people. Register at the link below:
> 
> https://lu.ma/7921a2ar
# Jason Crawford / The Roots of Progress in Bangalore, August 21 to September 8

I’ll be in Bangalore for three weeks starting next week (August 21–September 8). I’d love to meet anyone interested in the progress movement, and I’m looking into organizing a progress meetup.

If you’d like to meet me and/or join a meetup—or especially if you could help *host* the meetup—please [share your info with me via this brief form](https://forms.gle/PbWvdW4UH5KoDTUB7). Thanks a lot!
# What does it mean to “trust science”?

> *And this, my children, is why we do not say things like “I believe in science”. I mean, don’t get me wrong, science definitely exists—I’ve seen it. But not everything that calls itself science is science, and even good science sometimes gets wrong results. –*[*Megan McArdle*](https://twitter.com/asymmetricinfo/status/1684701303463714821)

Should we “trust science” or “believe in science”?

I think this is a fuzzy idea that we would do well to make clear and precise. What does it mean to “trust science?”

Does it mean “trust scientists”? Which scientists? They disagree, often vehemently. Which statements of theirs? Surely not all of them; scientists do not speak *ex cathedra* for “Science.”

Does it mean “trust scientific institutions”? Again, which ones?

Does it mean “trust scientific papers”? Any one paper can be wrong in its conclusions or even its methods. The study itself could have been mistaken, or the writeup might not reflect the study.

And it certainly can’t mean “trust science news,” which is [notoriously inaccurate](https://twitter.com/jasoncrawford/status/1662142462524112908).

More charitably, it could mean “trust the scientific process,” if that is properly understood to mean not some rigid Scientific Method but a rational process of observation, measurement, evidence, logic, debate, and iterative revision of concepts and theories. Even in that case, though, what we should *trust* is not the particular output of the scientific process at any given time. It can make wrong turns. Instead, we should trust that it will find the truth eventually, and that it is our best and only method for doing so.

The motto of science is not “trust us.” (!) The true motto of science is the opposite. It is that of the Royal Society: *nullius in verba*, or roughly: “take no one’s word.”

There is no capital-S Science—a new authority to substitute for God or King. There is only science, which is nothing more or less than the human faculty of reason exercised deliberately, systematically, methodically, meticulously to discover general knowledge about the world.

So when someone laments a lack of “trust” in science today, what do they mean? Do they mean placing religion over science, faith over reason? Do they mean the growing distrust of elites and institutions, a sort of folksy populism that dismisses education and expertise in general? Or do they mean “you have to follow my favored politician / political program, because Science”? (That’s the one to watch out for. Physics, chemistry and biology can point out problems, but we need history, economics and philosophy to solve them.)

Anyway, here’s to science—the system that asks you not to trust, but to *think*.

*Adapted from* [*a 2019 Twitter thread*](https://twitter.com/jasoncrawford/status/1192851511023325184)*.*
# Stampy's AI Safety Info - New Distillations #4 [July 2023]

Hey! This is another update from the distillers at [AI Safety Info.](https://aisafety.info/)

![](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/EELddDmBknLyjwgbu/pvqp2geuw04y2yhsbwmn)

Here are a couple of the answers that we wrote up over the last month (July 2023). As always let us know in the comments if there are any questions that you guys have that you would like to see answered. 

The list below redirects to individual links, and the collective URL above renders all of the answers in the list on one page at once.

These are the new articles. There has also been significant work in overhauling already live on site articles to improve their quality based on the feedback we have been receiving from readers.

*   [Isn't the real concern misuse?](https://aisafety.info?state=9B85_)
*   [What is Vingean uncertainty?](https://aisafety.info?state=9USW_)
*   [What is a "polytope" in a neural network?](https://aisafety.info?state=9NRR_)
*   [What are the power-seeking theorems?](https://aisafety.info?state=92JB_)
*   [How does "chain-of-thought" prompting work?](https://aisafety.info?state=8EL7_)
*   [What is "Constitutional AI"?](https://aisafety.info?state=904J_)
*   [How can LLMs be understood as “simulators”?](https://aisafety.info?state=9FQK_)
*   [What evidence do experts usually base their timeline predictions on?](https://aisafety.info?state=6478_)
*   [Wouldn't AIs need to have a power-seeking drive to pose a serious risk?](https://aisafety.info?state=A3MU_)
*   [What is a “treacherous turn”?](https://aisafety.info?state=9AKZ_)
*   [What is reinforcement learning from human feedback (RLHF)?](https://aisafety.info?state=88FN_)
*   [What is an agent?](https://aisafety.info?state=5632_)
*   [What is the Alignment Research Center's research strategy?](https://aisafety.info?state=85EK_)
*   [Wouldn't a superintelligence be smart enough not to make silly mistakes in its comprehension of our instructions?](https://aisafety.info?state=6984_)
*   [What are the differences between subagents and mesa-optimizers?](https://aisafety.info?state=8AF5_)
*   [What is the difference between verifiability, interpretability, transparency, and explainability?](https://aisafety.info?state=97FU_)

Cross posted to EA Forum: [https://forum.effectivealtruism.org/posts/ttBKSopeo59AedWZA/stampy-s-ai-safety-info-new-distillations-4-july-2023](https://forum.effectivealtruism.org/posts/ttBKSopeo59AedWZA/stampy-s-ai-safety-info-new-distillations-4-july-2023)
# Announcing Foresight Institute's AI Safety Grants Program

Foresight Institute has received funding to support projects in three AI safety areas we think are potentially under-explored:  
  
1\. Neurotechnology, brain computer interface, whole brain emulation, and "lo-fi" uploading approaches to produce human-aligned software intelligence  
  
2\. Computer security, cryptography, and related techniques to help secure AI systems  
  
3\. Multi agent simulations, game theory and related techniques to create safe multipolar AI scenarios that avoid collusion and foster positive sum dynamics  
  
The grant application process is now open and accepts rolling applications. We expect to grant between $1 - 1.2 million per year across projects and look forward to receiving your project proposals that could make a significant difference for AI safety within short timelines.   
  
Please visit [https://foresight.org/ai-safety/](https://foresight.org/ai-safety/) for full details and application instructions. 

I would appreciate it if you would consider sharing the opportunity with others who may benefit from applying.

Feel free to comment here or email me at [allison@foresight.org](mailto:allison@foresight.org) with any questions, feedback, or ideas for collaborations.
# AI labs' requests for input

AI labs sometimes publicly ask for input on their actions, their products, and the future of AI. See OpenAI's [Democratic inputs to AI](https://openai.com/blog/democratic-inputs-to-ai) and [ChatGPT Feedback Contest](https://cdn.openai.com/chatgpt/ChatGPT_Feedback_Contest_Rules.pdf), and maybe bug bounty programs for security vulnerabilities ([OpenAI](https://openai.com/blog/bug-bounty-program), [Google](https://bughunters.google.com/about/rules/6625378258649088/google-and-alphabet-vulnerability-reward-program-vrp-rules), [Meta](https://www.facebook.com/whitehat)). I'd like to collect these; please reply with other examples you're aware of. I'm also interested in ideas/recommendations for labs on what they should request input on or how they should do so (e.g. [bug bounty for model outputs](https://www.lesswrong.com/posts/5dKDLv4knhXLvNHT5/recommendation-bug-bounties-and-responsible-disclosure-for)).

* * *

Labs also seek input non-publicly. For example, labs have used external red-teaming and [model evals](https://evals.alignment.org/blog/2023-08-01-new-report/), [worked with biosecurity experts to understand how near-future AI systems can contribute to biological weapons](https://www.anthropic.com/index/frontier-threats-red-teaming-for-ai-safety), and [consulted external forecasters](https://cdn.openai.com/papers/gpt-4.pdf#page=59). Various kinds of external audits have been proposed. I'm also interested in collecting examples and ideas for this non-public input, but less so.

*Crossposted to the* [*EA Forum*](https://forum.effectivealtruism.org/posts/DjAQMc9rAEyjkmwYb/ai-labs-requests-for-input)*.*
# Call for Papers on Global AI Governance from the UN

Copied from their LinkedIn page:

📢 𝐂𝐚𝐥𝐥 𝐟𝐨𝐫 𝐏𝐚𝐩𝐞𝐫𝐬 𝐨𝐧 𝐆𝐥𝐨𝐛𝐚𝐥 𝐀𝐈 𝐆𝐨𝐯𝐞𝐫𝐧𝐚𝐧𝐜𝐞🌍

Exciting news! As we gear up for the High-level Advisory Body on AI, we're inviting thought leaders, researchers, and enthusiasts to contribute short papers (~2000 words) on pivotal themes:

1️⃣ Key Issues on Global AI Governance: Dive into studies and recommendations that the High-Level Advisory Body on AI should prioritize, especially those needing global governance attention.

2️⃣ Current Efforts in Global AI Governance: Share analyses on bilateral, multilateral, and inter-regional initiatives. We're keen on understanding varying philosophical approaches, critiques, and suggestions.

3️⃣ Models in Global AI Governance: Whether you're analyzing existing models or proposing fresh perspectives, we're all ears. Surveys and analyses of other proposals are also encouraged.

We champion diversity! 🌈 We're eager to hear from a myriad of groups, regions, and methodologies. Your insights will serve as foundational material (with due credit) for the High-Level Advisory Body on AI.

📅 Deadline: 30 September

📧 Submission: Send your paper (hyperlink or PDF) to [techenvoy@un.org](mailto:techenvoy@un.org). Ensure your title page has the author(s) details, affiliation, and contact info. If it's an Executive Summary of a more extensive piece, kindly attach the main paper's link.
# Do agents with (mutually known) identical utility functions but irreconcilable knowledge sometimes fight?

Been pondering; will conflict always exist? A major subquestion: Suppose we all merge utility functions and form an interstellar community devoted to optimizing the merger. It'll probably make sense for us to specialize in different parts of the work, which means accumulating specialist domain knowledge and becoming mutually illegible.

When people have very different domain knowledge, they also fall out of agreement about what the borders of their domains are. (*EG: A decision theorist is insisting that they know things about the trajectory of AI that ML researchers don't. ML researchers don't believe them and don't heed their advice.*) In these situations, even when all parties are acting in good faith, they know that they wont be able to reconcile about certain disagreements, and it may seem to make sense, from some perspectives, to try to just impose their own way, in those disputed regions.

Would there be any difference between the dispute resolution methods that would be used here, and the dispute resolution methods that would be used between agents with different core values? (war, peace deals, and most saliently,)

Would the parties in the conflict use war proxies that take physical advantages in different domains into account? (EG: Would the decision theorist block ML research in disputed domains where their knowledge of decision theory would give them a *force* advantage?)
# Diet Experiment Preregistration: Long-term water fasting + seed oil removal 

I am intrigued by exfatloss's [recent theory](https://exfatloss.substack.com/p/seed-oils-explain-the-8-mysteries) that the primary cause of the obesity epidemic is slow, subtle damage from linoleic acid over many years. Specifically, I want to test the broader possibility that the composition of adipose tissues is what's causing the problem. Since we do not have a decade to test his hypothesis naturally, I'll retcon my current eating habits for a modified experiment.

I am just now transitioning from a very low calorie diet to a water fast, having lost around fifteen pounds in the last few weeks. I am going to attempt to continue my water fast for the next thirty days, or until I reach a BMI of <25. After the end of my water fast, I am going to rigorously exclude linoleic acid and other PUFAs from my diet; my meals will consist mostly of vegetables, rice, specially ordered low-PUFA pork, and fish. Then I'm going to eat to satiety for another month.

Normally what is expected to happen after a water fast is that you regain all of the weight you just lost, because you haven't modified your body's dietary "set point" and you wasted all of your mental strength trying to keep yourself from eating. However, if the linoleic acid hypothesis is true, all of the PUFA I've consumed over my many years as a good American are what's causing my set point to remain so high in the first place. So for this self-experiment I'm going to deliberately purge most of my remaining fat stores, fill them up with a traditional Japanese diet, and see if my "natural weight" changes. 

It won't invalidate the lineolic acid hypothesis if the damage is just irreversible, but I can at least test out the hypothesis that the *running* composition of my fat is what causes my metabolism to freak out.
# China's position on autonomous weapons

> While continuing to endorse the necessity to develop a legally binding instrument with respect to LAWS, in 2018 China also proposed a somewhat narrow definition of LAWS comprising five essential features:

>>  The first is lethality, which means sufficient payload (charge) for it to be lethal. The second is autonomy, which means absence of human intervention and control during the entire process of executing a task. Third is impossibility of termination, meaning that once activated there is no way of terminating the device. Fourth is indiscriminate effects, meaning that the device will execute the task of killing and maiming regardless of conditions, scenarios, or targets. Fifth is evolution, meaning that through interaction with the environment the device can learn autonomously and expand its functions and capabilities in a way that exceeds human expectations.

> [...] China’s restrictive definition sets an extremely high threshold in regard to the kinds of technologies that may be eligible for legal regulation. 

> Chinese delegates to the GGE that we interviewed, however, do not view this definition as “narrow.” For them, rapid technological advancements could soon make weapons with zero human oversight a reality, especially in technologically advanced countries like the USA. 

Military UAVs with autonomous targeting already exist. Turkey [probably used one](https://thebulletin.org/2021/05/was-a-flying-killer-robot-used-in-libya-quite-possibly/). [Israel has also been](https://www.youtube.com/watch?v=G7yIzY1BxuI) developing autonomous weapons. The [Switchblade 600](https://en.wikipedia.org/wiki/AeroVironment_Switchblade) isn't autonomous, but US defense contractors are interested in adding autonomous targeting to such products and seem to be lobbying against US regulations on that. The Chinese government seems to be strongly in favor of autonomous weapons, but also strongly against literal Terminators a la the movies.

Thus far, the focus has been on autonomous targeting for small short-range electric UAVs, but autonomy is easier for larger UAVs that can carry more sensors and computing, and datalinks are harder over long distances - especially if satellites were targeted in eg a war over Taiwan between the US and a China-Russia alliance.
# Regreasing a KitchenAid Mixer

I recently got an old KitchenAid mixer from a friend. It's a k45ss, from maybe the early 80s, and the general advice I see is that with typical home use they should be regreased every ~10y. I ordered 4oz of food-safe synthetic [grease](https://www.amazon.com/dp/B08ZYCQVRD) and followed Mr. Mixer's YouTube guide (parts [1](https://www.youtube.com/watch?v=EW9BQ-oPFkk), [2](https://www.youtube.com/watch?v=L-qMOyhb2xc), [3](https://www.youtube.com/watch?v=UmnD6Y5jNMg)). With my phone playing the videos I followed along: opening the mixer up, removing the old grease, adding the new grease, and putting it back together. There were slight differences (my strain relief was annoying, and I didn't have a [pin](https://youtu.be/L-qMOyhb2xc?si=dYgKmFXzWSKtX7o1&t=133) inside the gear assembly and needed to remove the worm assembly first instead) but it was pretty close.

The planetary gear assembly was pretty low on grease, and the grease there was a bit dry:

[![](https://www.jefftk.com/kitchen-aid-planetary-low-grease.jpg)](https://www.jefftk.com/kitchen-aid-planetary-low-grease-big.jpg)

The upper area had more grease, though the grease was a harder than I expect it was supposed to be:

[![](https://www.jefftk.com/kitchen-aid-head-internal-old-grease.jpg)](https://www.jefftk.com/kitchen-aid-head-internal-old-grease-big.jpg)

The gears were in good shape, and were nice and solid feeling.

Degreasing, regreasing, and assembly went fine. Probably 1.5hr all together? When I was done, however, I realized I had a part left over: one of three washers.

[![](https://www.jefftk.com/kitchen-aid-extra-washer.jpg)](https://www.jefftk.com/kitchen-aid-extra-washer-big.jpg)

I looked back over the video, and it talked about installing two of these washers, which I remembered doing, but I was stumped by how I ended up with a third. There wasn't any logical place which should have a washer but didn't.

I talked about this at dinner, and one of my housemates said they also had an old KitchenAid that had probably never been regreased (k45, cord isn't grounded, maybe late 60s). We decided to do theirs together, and maybe we'd see where the washer went.

The combination of having a second pair of hands and having just done one the previous evening made it go a lot faster: less than an hour. Very similar to mine, except the strain relief was even more annoying. We did figure out the washer mystery, though: there are two washers in the planetary mechanism, stacked on top of each other.

I'm not sure whether it's worth opening the bottom of mine back up to install the extra washer. I'm leaning towards yes (they wouldn't put in two washers for no reason, doesn't require taking the whole thing apart) but possibly I don't really need to?
# What wiki-editing features would make you use the LessWrong wiki more?

The LessWrong wiki does not seem as well used as it should be. I guess this is a lack of editors.

[@Vladimir_Nesov](https://www.lesswrong.com/users/vladimir_nesov?mention=user) made a good point that many standard wiki-editing features are missing which makes the prospect unappealing.

> The point is that absence of the feature makes engagement with the wiki less promising, as it becomes inconvenient and hence infeasible in practice to protect it in detail, and so less appealing to invest effort in it. I mentioned that as a hypothesis for explaining currently near-absent editor engagement

So in the spirit of this, what features would cause you personally to edit a lot more than you currently do? 

I may try and pay some devs to write the pull requests if they seem feasible.
# What AI Posts Do You Want Distilled?

I'd like to distill AI Safety posts and papers, and I'd like to see more distillations generally. Ideally, posts and papers would meet the following criteria:

*   Potentially high-impact for more people to understand
*   Uses a lot of jargon or is generally complex and difficult to understand
*   Not as well-known as you think they should be (in the AI X-risk space)

What posts meet these criteria?
# Lemonade Stand Costs?

My kids were thinking of having a lemonade stand, and in the past I've said that if they're doing it as a fundraiser we'll cover the cost of ingredients, but if they're doing it for their own money then they need to reimburse us. But how much should that be?

The recipe they're using is approximately:

*   2C lemon juice
*   2C sugar
*   Water
*   Ice

Plus cups to serve it in. Makes about one gallon, or sixteen 8oz servings.

The water is [basically free](https://www.jefftk.com/p/rough-utility-costs) and the ice is below the rate that our icemaker replenishes: also effectively free. At our local grocery store 32oz of lemon juice is $3.50, sugar is $0.85/lb, and cups are ~$0.10/each. So each 8oz serving of juice is:

*   1oz lemon juice: $0.11.
*   1oz sugar: $0.05.
*   cup: $0.10
*   total: $0.26

If they sell until they run out of lemonade, call it a quarter a cup. They usually sell each cup for a dollar, so that's a gross margin of 75%. Not bad!
# Chariots of Philosophical Fire

Sharing because I feel that the contribution of analytical philosophy is sometimes under appreciated here, particularly compared to other attempts at philosophy:

“In grappling with these mysteries, Oxford philosophers developed and refined old and new techniques. Reasoning, deduction, explanation, more care and precision in language, crisper concepts, deeper distinctions, elaborate models, thought experiments, devastating counterexamples, intuitive principles that press to surprising conclusions—on all of these things, Oxford led the way, and the rest of the Anglosphere followed. Its legacy is less a set of ideas or even a series of sacred tenets and Delphic sayings than it is a devotion to rigor, clarity, truth, and a very practical British revulsion to nonsense.

… Creative genius is evenly distributed in neither time nor place. A survey of the past shows that genius is not randomly scattered about, like the seeds of a dandelion, but concentrates: ancient Athens, Renaissance Florence, Silicon Valley, among other examples. Why these fertile eras and places appear, peak, and then decline is understudied as a historical phenomenon. Oxford philosophy in the twentieth century, though not as astonishing as Florence or as productive as Silicon Valley, is nevertheless an example of the clustering of philosophical genius.

… Third—and strangely left unexplored by the authors—nearly all the philosophers in these books did not have Ph.D.s in philosophy. Gilbert Ryle, J. L. Austin, Isaiah Berlin, Iris Murdoch, Philippa Foot, Elizabeth Anscombe, A. J. Ayer, R. M. Hare, Bernard Williams, and Derek Parfit—not a single doctoral degree in philosophy among them. A first in their undergraduate exams—meaning a grade of “A+”—was enough to send them on their way. Yes, each won prizes and fellowships, but none wrote anything like a dissertation under the supervision of an advisor. One might conclude, as the analytic tradition fades into its senility, that requiring a graduate degree in philosophy made academic philosophers worse at philosophy.

… Perhaps most importantly, Oxford was rich in social and intellectual influences outside the tutorials. Again, all three books bring this vividly to life. A new club seems to have formed every decade to discuss philosophy. The old discussion groups stuck around, too. The Jowett Society, the Wee Teas, the Metaphysicals, J. L. Austin’s Saturday Morning Meetings, the Tuesday Group—out of these clubs, friendships formed in breakaway circles. Those circles became crucibles for refining thought, shaping and sharing a vision about what counted as good work and what big debates were worth addressing. More than mentorship, collaborative and somewhat rivalrous friendships set the pace, escalated the level of play, and expanded the edge of the possible. Only your friends can rev you up to desecrate the monuments of the last generation.“