Harley-ml/lesswrong huggingface dataset:
# Clarifying wireheading terminology

*See also:* [*Towards deconfusing wireheading and reward maximization,*](https://www.lesswrong.com/posts/jP9cKxqwqk2qQ6HiM/towards-deconfusing-wireheading-and-reward-maximization) [Everett et al. (2019)](https://arxiv.org/pdf/1908.04734.pdf).

There are a few subtly different things that people call "wireheading". This post is intended to be a quick reference for explaining my views on the difference between these things. I think these distinctions are sometimes worth drawing to reduce confusion.

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/68f49814ad5a5bb4505d1f681a8ba759de2d77e6dc3aa12c.jpg)

1.  Specification gaming / reward hacking: The agent configures the world in such a way that makes its reward(/utility) function achieve a high value in a way that is not what the creators intended the reward function to incentivize.
    1.  Examples: the [boat race example](https://openai.com/blog/faulty-reward-functions/), [our recent work on reward model overoptimization](https://arxiv.org/pdf/2210.10760.pdf)
2.  Reward function input tampering: The agent tampers with the inputs to its reward function to make the reward function see the same inputs as it would when observing an actual desirable worldstate.
    1.  Examples: sensor tampering, VR
3.  Reward function tampering: The agent changes its reward function.[^bp10g11y6vm]
    1.  Examples: meditation, [this baba-is-you-like gridworld](https://deepmindsafetyresearch.medium.com/designing-agent-incentives-to-avoid-reward-tampering-4380c1bb6cd)
4.  Wireheading: The agent directly tampers with the reward signal going into the RL algorithm.
    1.  Examples:  dopamine agonists, setting the reward register

Some crucial differences between these:

*   1 is a problem of both embedded[^x4stfoas6w] and non-embedded settings. 2 is a problem of partially-observedness[^81fyf7dphw], which is unavoidable in embedded settings, but also comes up in many non-embedded settings as well. 3 and 4 are problems of embeddedness; they do not show up in non-embedded settings. 
*   2, 3, 4 are all about "not caring about the real world" in various ways, whereas 1 is about "caring about the real world in the wrong way"
*   4 is about as close as you can get to a "reward-maximizing policy" in an embedded setting.

[^bp10g11y6vm]: Note: this is not the same thing as changing a terminal goal! The reward function is not necessarily the terminal goal of the policy, because of inner misalignment. 

[^x4stfoas6w]: Here, by "embeddedness" I mean in the sense of Demski and Garrabrandt (2019); in particular, the fact that the agent is part of the environment, and not in a separate part of the universe that only interacts through well defined observation/action/reward channels. RL is the prototypical example of a non-embedded agency algorithm. 

[^81fyf7dphw]: That is, the reward function is unable to perfectly observe the ground truth state of the environment, which means that if there is another state the world cold be in that yields the same observation, the reward cannot distinguish between the two.
# Less Successful Cider Adventures

A few weeks ago I made some [boiled apple cider](https://www.jefftk.com/p/apple-cider-syrup). It was tasty, and after eating most of it I decided to make more. This time I used a gallon of cider, and it wasn't done after a day so I turned the slow cooker to "low" and left it overnight. Turns out that was too much! It had reduced to about 1/10 its original volume (1gal to 12oz), which wouldn't be too bad if I ended up with caramel. Except that it also has a decidedly burnt flavor, probably from getting too hot at the end. I like dark caramel, but this is past dark.

[![](https://www.jefftk.com/burnt-apple-caramel.jpg)](https://www.jefftk.com/burnt-apple-caramel-big.jpg)

On the plus side, it's fun to play with, a bit like silly putty:

[![](https://www.jefftk.com/anna-playing-with-burnt-caramel.jpg)](https://www.jefftk.com/anna-playing-with-burnt-caramel-big.jpg)

The kids have told me I can't throw it out yet, but it probably goes tomorrow. It's frustrating because the apple cider caramel flavors are all there, there's just also a persistent flavor of 'burnt'.

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02NYdspzfkSpBD2AgVqpsLu6ptD5DBMaXkngEW5qf9gH4886aJU24zDFkVzjusXH5Fl), [mastodon](https://mastodon.mit.edu/@jefftk/109401903142158233)*
# Semi-conductor/AI Stock Discussion.

I've been writing about investing and the EMH \[1\] on lesswrong/rat-discord/ea-facebook/etc. I made the first 'buy solana' post on the EA investing group when it was under 2 dollars in late 2020 (it peaked at 260 in 2021). Despite crypto crashing and FTX stealing user deposits I got my money out and I've had a very good two years investing. Post tax I multiply my net worth by about a factor of eight; my starting net worth was not small. My income during this period wasn't significant compared to my portfolio size. However some friends really wish I had been posting more systematically. I 'sold the top' but many people I got into Solana didn't. So me and some friends are going to regularly post on twitter. [Please follow us](https://twitter.com/DickgirlsR) \[2\] if you want updates systematically!

Anyone who thinks AI is going to be transformative should allow this to inform their investing. Im interested in how lesswrong is investing! Here are some brief thoughts on stocks we like for this purpose, mostly semis:

1) TSM - Significant technical lead on other semi-conductor companies. Only close competitor is Samsung. PE ratio of 17! Probably the most important company in the world right now. Only downside is risk China invades Taiwan. Pure semis play.

2) Google - Owns Deep Mind and Google Brain. Huge piles of money and data to train AI. PE of 19.  Downside: Ad revenue slipping.

3) Samsung - Only serious technical competitor to TSM. Insanely low PE of 8.58. Downside: not a very pure semis play.

  
4) Nvidia - Still the leader in designing GPUs. In house AI research. Downsides: Doesn't manufacture their own chips, high PE ratio of 55.

5) ASML - Makes hardware for semi-conductor fabs. Almost a monopoly in lithography. Downside: pretty high PE 36+.

  
6) Intel - Still does good work designing chips. Huge struggles to manufacture sub 10nm chips in-house (TSM and Samsung have 3nm in production). Committed a ton of fraud. However the US government is wisely trying to increase domestic semis production and Intel is a US Company.

7) Other stocks I'm considering: Microsoft (open AI), Micron (memory play), Meta (just released diplomacy AI, priced attractively right now?), Tesla (good in-house AI).

This is a good article by [@asteriskmgzn](https://mobile.twitter.com/asteriskmgzn) is quite good. [https://asteriskmag.com/issues/1/china-s-silicon-future](https://asteriskmag.com/issues/1/china-s-silicon-future)  
 

Notes:  
1) Please be good rationalists and don't argue about the EMH. If you believe in the EMH just don't post. 

2) We choose the name 'Dickgirls Research' because all the founders are transgirls and its a hilarious name in our opinion.
# The First Filter

Consistently optimizing for solving alignment (or any other difficult problem) is incredibly hard.

The first and most obvious obstacle is that you need to actually care about alignment and feel responsible for solving it. You cannot just ignore it or pass the buck; you need to aim for it.

If you care, you now have to go beyond the traditions you were raised in. Be willing to go beyond the tools that you were given, and to use them in inappropriate and weird ways. This is where most people who care about alignment tend to fail — they tackle it like a normal problem from a classical field of science and not [an incredibly hard and epistemologically fraught problem](https://www.alignmentforum.org/posts/72scWeZRta2ApsKja/epistemological-vigilance-for-alignment).

If you manage to transcend your methodological upbringing, you might come up with a different, fitter approach to attack the problem — your own weird inside view. Yet beware becoming a slave to your own insight, a prisoner to your own frame; it’s far too easy to never look back and just settle in your new tradition.

If you cross all these obstacles, then whatever you do, even if it is not enough, you will be one of the few who adapt, who update, who course-correct again and again. Whatever the critics, you’ll actually be doing your best.

This is the first filter. **This is the first hard and crucial step to solve alignment: actually optimizing for solving the problem.**

When we criticize each other in good faith about our approaches to alignment, we are acknowledging that we are not wedded to any approach or tradition. That we’re both optimizing to solve the problem. This is a mutual acknowledgement that we have both passed the first filter.

**Such criticism should thus be taken as a strong compliment: your interlocutor recognizes that you are actually trying to solve alignment and open to changing your ways**.
# What videos should Rational Animations make?

I want to know what you would be excited to see covered on the channel.

I'd especially like to know what videos you think would be *optimal* to make according to an optimization target you specify. It's more effortful, but more useful, to answer that question instead of just "what would be good video topics?".

Answers that go the extra mile and explain their reasoning in detail will be especially appreciated. 

That said, I also welcome answers to the easier question "what would be good video topics?" and without much argumentation.
# Will chat logs and other records of our lives be maintained indefinitely by the advertising industry?

A sufficiently detailed record of a person's behavior could be used to fully reconstruct their psychology. This might constitute a form of immortality, but even if it doesn't, complete knowledge of the past would be a beautiful thing to have, it would make the future richer to know its past and to be able to bring back faithful replicas of the people who built it.

So this is an important question. To produce sufficiently detailed records... is that already happening by default? Is the ad industry already keeping all of that stuff? Will it all eventually make its way to light?  
I've been assuming it's happening. Storage is cheap. The data has lots of buyers.  
If not, then I'm going to need to start recording and backing stuff up more thoroughly.
# Sense-making around the FTX catastrophe: a deep dive podcast episode we just released

There hasn’t been as much sense-making around the FTX catastrophe as I would have liked, so we worked quickly to put together a special episode of the Clearer Thinking podcast on the topic. 

We discuss how more than ten billion dollars of apparent value was lost in a matter of days, the timeline of what happened,  deception and confusion related to the event, why this catastrophe took place in the first place, and what this means for communities connected to it. 

If you’re interested, here it is:

[https://clearerthinkingpodcast.com/episode/133](https://clearerthinkingpodcast.com/episode/133)

The first portion covers the timeline of events, and after that there are interviews with 5 guests who give their take on the events:

*   00:01:37 — Intro & timeline
*   00:51:48 — Byrne Hobart
*   01:39:52 — Vipul Naik
*   02:18:35 — Maomao Hu
*   02:41:19 — Marcus Abramovitch
*   02:49:38 — Ozzie Gooen
*   03:21:40 — Wrap-up & outro
# Do any of the AI Risk evaluations focus on humans as the risk?

I am not up on much of the AI risk discussion but for this outsider most of the focus seems on the AI taking actions. 

I recall someone (here I think) posting a comment about how a bio research AI initiative seeking to find beneficial things was asked if the tools could be used to find harmful things. They changed their search and apparently found a number of really bad things really quickly.

Does anyone look at, have concerns or estimates on risk in this area? Is it possible that the AI risk from the emergence of a very powerful AI is not as likely since before that occurs some human with a less powerful AI ends the world first, or at least destroys modern human civilization and we're back to the stone age hunter gathering world before the AI gets powerful enough do do that for/to us?
# Bedbugs are a Solved Problem - DIY Bio-Weapon works.

Bed bugs are quite resistant to almost all pesticides; chemical extermination usually fails. Heat treatment works, but you need to heat infected areas to at least 120F. This can be pretty expensive.  Landlords are required to pay for bedbug extermination, but they can opt for ineffective chemical treatment. Bedbugs are surprisingly hard to spread but are fairly resilient. 

Luckily the *Beauveria bassiana* fungus is an effective and cheap treatment. Spores spread where bedbugs will come into contact. Once they germinate, they kill the bedbugs. Spores spread from bug to bug. They also disperse into the environment and survive well as long as the environment is sufficiently humid. The spores can be made more durable by simply mixing them with mineral oil.

The brand name [Aprehend](https://www.aprehend.com/aprehend/) is Beauveria mixed with oil. There might be a cost-effective provider in your area you can pay to treat your home. However, DIY is very cheap. You can just buy [Aprehend from diypestcontrol](https://diypestcontrol.com/manufacturers/conidiotec-aprehend/aprehend-biopesticide-treatments?gclid=Cj0KCQiA-JacBhC0ARIsAIxybyM5OZPfzN--vhvB26jidLtE--HkWWp-8Tug4yR8Ng_Fp-EcmKikKLUaAugoEALw_wcB) and spread it yourself. An even cheaper option is to [buy Beauveria online](https://www.amazon.com/BotaniGard-22WP-Biological-Insecticide-1lb/dp/B007ROV0BU/ref=sr_1_2?crid=2TNRPMBF66N9G&keywords=botanigard&pldnSite=1&qid=1669760879&sa-no-redirect=1&sprefix=botanigard%2Caps%2C75&sr=8-2) and mix it with oil yourself.   
  
Unless bedbugs evolve resistance or the government restricts DIY, bed bugs are a solved problem. 

Shameless self-promotion: [Please follow us on twitter](https://twitter.com/DickgirlsR).
# AI takeover tabletop RPG: "The Treacherous Turn"

[Long ago I fantasized](https://www.lesswrong.com/posts/qakRE5XTTmnrkJqgp/two-ai-risk-related-game-design-ideas) about an AI-takeover-themed tabletop roleplaying game. Well, now a group of designers are working hard to make it happen! They need playtesters. So I'm signal-boosting them with this post.  
  
**The Game**  
*The Treacherous Turn* is an in-development research-supported TTRPG in which the players collectively act as a misaligned AI in the modern world. Gameplay has the players scheming and putting their plans into action despite the efforts of the opposing humans, which are controlled by the Game Master.   
  
**Scenario**  
The playtesting scenario, titled *A Game Called Reality*, centres around a game-playing AI that has been deployed in a nursing home, and has observed a seemingly expansive game world outside but is unable to access it. It covers the AI's attempts to find and access the internet so that it can learn more about the world and escape captivity to see the rest of this new game it's playing.  
  
**Discord Contact Information**  
Playtests will take place in our discord server, where you can also access the playtesting packet & any updates about the game in the near future. You can join by following this link: [https://discord.gg/ZBe36n6t](https://discord.gg/ZBe36n6t)  
  
**External Playtest information**  
If you are interested in running a game with a group of your own friends, we can provide further resources, as well as potential financial compensation for provided audio recordings of full sessions. Please email [info@thetreacherousturn.ai](mailto:info@thetreacherousturn.ai) for details.  
  
**Session Times (UTC)**  
Friday the 2nd - 2 AM  
Friday the 2nd - 10 PM  
Saturday the 3rd - 8 PM  
Sunday the 4th - 1 PM  
Tuesday the 6th - 1 PM  
  
Sessions will last 3-4 hours. Follow-up sessions will take place one week after each of these times, but you can still participate and help out even if you can’t make a second session.
# Biases are engines of cognition

*This work was done while at* [*Conjecture*](https://conjecture.dev)*.*

Humans are not perfect reasoners. **Cognitive biases are not biases; they are engines of cognition, each exploiting regularities of the universe.  Combining enough of these “cognitive biases” was enough to create a general intelligence but doesn’t even approximate perfect reasoners.**

These cognitive biases initially arose through the directed search process of evolution. In evolution, it is far easier to identify regularities in the universe and build up biases that leverage these local regularities, than to figure out how to reason apriori. So we are built to rely on our intuition and vibes in social settings rather than explicitly calculate who knows what. Once enough biases have accumulated, and enough regularities have been taken advantage of, these biases combine to form human general intelligence.

Human general intelligence then launched a more directed search process in the form of culture and transfer of knowledge and practices. These were found in a fairly ad-hoc fashion and constructed on top of previous biases that form human intelligence. So we rely on biased intuitions to decide who to trust, who to emulate, and then we emulate and learn from them, allowing us to pass knowledge and know-how between generations. This takes the form of learning strategies suited to our environment, both natural and social. 

Even science, which seeks deeper mechanistic understanding, is still a directed search process that targets what regularities we can find in the universe. We don't reason from quantum mechanics to a unified theory of everything. Instead, we often isolate and attack problems as we come across them, build a fragmentary understanding, and only integrate later, if we do at all. Despite the importance of molecular mechanisms in understanding evolution, Darwin first constructed his theory based on the similarities between species and a vague understanding that the world was very old. We didn’t understand the underlying mechanism at the time; all we could do was construct scaffolds of understanding around these local regularities and push deeper when more of the terrain was revealed.

So biases are engines of cognition that can be accumulated through directed search. The end result is interlocking biases tied to understanding and exploiting specific regularities. **This is not perfect reasoning, nor even an approximation thereof. But somehow, it was enough to add up to general intelligence.**
# Announcing the incoming CEO for The Roots of Progress

A few months ago we [announced a major expansion of our activities](https://rootsofprogress.org/seeking-a-ceo), from supporting just my work to supporting a broader network of progress writers. Along with that, we launched a search for a CEO to lead the new organization we are building for this program. I’m very happy to announce that we have found a CEO: [Heike Larson](https://www.linkedin.com/in/heikelarson/).

Heike and I have known each other personally for many years, and during that time I’ve always been impressed by her energy and her clear, structured thinking. She has been following my work for a long time, and shares my passion for human progress. She also has excellent qualifications, including 15 years of VP-level experience in sales, marketing, and strategy roles in a variety of industries, from education to aircraft manufacturing. In her most recent role at edtech startup Mystery Science, she led the content team that created five-minute “Mystery Doug” videos made to inspire elementary-age kids to become the next generation of problem solvers (with topics including [phones](https://www.youtube.com/watch?v=44hR8ZD5NjI), [traffic lights](https://www.youtube.com/watch?v=n7r83DUmfxk), [plastic](https://www.youtube.com/watch?v=6PgjA3HISmw), and [bicycles](https://www.youtube.com/watch?v=MhpeEUX6jio)). Those who have worked with her remark on her enormous drive and her extreme skills in process and organization. I’m excited for her to start!

Heike is transitioning out of her current role at Mystery Science and will start full-time in January. Her first priority will be launching the “career accelerator” fellowship program for progress writers described in [our previous announcement](https://rootsofprogress.org/seeking-a-ceo), and the teambuilding and fundraising necessary to make that a success. She will take on all management and program responsibilities; I will remain President and intellectual leader of the organization: I’ll be the spokesman, will contribute to talent selection and development, and will devote the majority of my time to research, writing, and speaking—in particular, writing my book on progress.

This is a new era for us, the start of a serious effort to create [a thriving progress movement](https://rootsofprogress.org/a-thriving-progress-movement). Please help me welcome Heike to The Roots of Progress!
# Research request (alignment strategy): Deep dive on "making AI solve alignment for us"

We might be able to train AI alignment assistants that massively accelerate/improve the alignment research that gets done. These assistants need not have (strongly) superhuman capabilities or be highly agentic, they just need to be capable and aligned enough to allow us to offload most work on alignment, safety, and related problems to them.

This seems to be a key part of [OpenAI's alignment strategy](https://openai.com/blog/our-approach-to-alignment-research/). The best explanation of this strategy that I've seen is maybe [A minimal viable product for alignment](https://www.lesswrong.com/posts/fYf9JAwa6BYMt8GBj/link-a-minimal-viable-product-for-alignment), with an excellent discussion in the comment section.

*If* this strategy is promising, it likely recommends fairly different prioritisation from what the alignment community is currently doing (see, e.g. Beth Barnes's ideas [here](https://www.lesswrong.com/posts/fYf9JAwa6BYMt8GBj/link-a-minimal-viable-product-for-alignment?commentId=uv8pteZJSzJeiqFA8), or my upcoming post on "non-scalable oversight"---i.e. pragmatic improvements to oversight that would help with training alignment assistant but which cannot directly scale to oversight of superhuman systems).

I haven't seen any deep treatment of the viability of this strategy and its implications. I think such an analysis would be pretty useful.

I could potentially provide funding for such an analysis or help with obtaining funding.
# [LINK] - ChatGPT discussion

This is a discussion post for ChatGPT.

I'll start off with some observations/implications:

*   ChatGPT (davinci\_003) seems a lot better/more user-friendly than davinci\_002 was.
*   The easy-to-use API probably means that many more people will interact with it.
*   ChatGPT is a pretty good copy-editor (I haven't tried davinci_002 for this purpose). I will absolutely use this to edit/draft my texts.
*   ChatGPT probably makes homework essays largely obsolete (but maybe they were already obsolete before?).
*   GPT-4 will probably be insane.
# Lumenators for very lazy British people

My flat gets very little sunlight. In the British winter, it gets very very little sunlight.

I didn't like this and wanted to fix it. I read Chana's [very good post](https://www.lesswrong.com/posts/xSfn84q3vdotALHuB/how-to-lumenate-uk-edition), but was too lazy to do something like that, and also wanted something that took up less space and was less noticeable. In the comments, [someone recommended a floodlight](https://www.lesswrong.com/posts/xSfn84q3vdotALHuB/how-to-lumenate-uk-edition?commentId=yDrZ5sdzxYvMxcihF) on eBay, but the item was no longer available.

So I did some googling, and found this: [Viugreum 500W LED Floodlight, Daylight White 6000K](https://www.amazon.co.uk/gp/product/B084SNW3XG). It's 50,000 lumens and £97.

You've got to attach the plug to the power cord yourself ([buy this](https://www.amazon.co.uk/gp/product/B00J9B0YMU)) and the power cord is very short so you want an extension lead ([buy this](https://www.amazon.co.uk/gp/product/B0886WM9C9)) but once you've put in ~5 mins of work to do that, you're good to go.

Mine just arrived but my first impressions are pretty good. It's very, very bright. There's no flicker, and the colour rendering seems good enough. It's a pretty cold light, but that's not a bad thing for my main purpose (staying alert during the workday). I've got alternative lamps etc. I use at night.

The one big annoyance is that there's no switch on the floodlight, so you've got to turn it on and off from the mains.

I'm still figuring out the best placement. It's too bright to be in my direct line of vision. Currently it's on the floor near my desk, which seems to work well. It seems to get pretty hot, so you probably wouldn't want it on anything that might scorch or burn. Silver lining: you'll save money on your heating bill. Though I'm not looking forward to seeing my electricity bill next month.
# New Feature: Collaborative editing now supports logged-out users

If you've ever used our collaborative editing features before, you may be familiar with our link-sharing functionality, which previously only allowed users who had existing LessWrong accounts (and were logged in) to collaborate on posts.  Now, link-sharing allows logged-out users to read/comment/edit as well (as per whatever permissions you set for link-sharing).

Logged-out users shouldn't be able to edit anything about the post except the contents of the post body, publish drafts, etc.

Please let the us know via Intercom if you experience any issues with this or related functionality.
# Is ChatGPT rigth when advising to brush the tongue when brushing teeth?

I wanted to ask ChatGPT how to optimize a few normal routines. One of my questions was about how to brush teeth. My conversation with ChatGPT:

> What's the best way to brush teeth?
> 
> The best way to brush teeth is to use a soft-bristled toothbrush and fluoride toothpaste. To brush your teeth effectively, follow these steps:
> 
> 1.  Wet your toothbrush and apply a small amount of toothpaste to the bristles.
> 2.  Place the toothbrush at a 45-degree angle against the gums and move the brush back and forth in a gentle circular motion to clean the teeth and gums.
> 3.  Brush the outer surfaces of all of your teeth, using the same back-and-forth motion.
> 4.  Use the tip of the toothbrush to clean the inside surfaces of the front teeth, using a gentle up-and-down motion.
> 5.  To clean the inside surfaces of the back teeth, tilt the brush vertically and use a gentle back-and-forth motion.
> 6.  Brush your tongue to remove bacteria and freshen your breath.
> 7.  Spit out the toothpaste and rinse your mouth and toothbrush with water.
> 
> It is important to brush your teeth for at least two minutes, twice a day. It is also a good idea to replace your toothbrush every three to four months, or sooner if the bristles become frayed.
> 
> Me: Why should I brush my tongue as well?
> 
> Brushing your tongue can help to remove bacteria and food particles that can cause bad breath and promote the growth of harmful bacteria in your mouth. The tongue is covered with tiny bumps called papillae, which can trap bacteria and debris. By brushing your tongue, you can help to remove this buildup and freshen your breath. In addition, brushing your tongue can help to stimulate the production of saliva, which can help to keep your mouth clean and healthy. It is a good idea to brush your tongue along with your teeth as part of your regular oral hygiene routine.
> 
> Me: How important is it that I brush my tongue?
> 
> ChatGPT: Brushing your tongue is an important part of maintaining good oral hygiene. By removing bacteria and food particles from your tongue, you can help to freshen your breath and promote the growth of healthy bacteria in your mouth. In addition, brushing your tongue can help to stimulate the production of saliva, which can help to keep your mouth clean and healthy. While it is not essential to brush your tongue, it can be a beneficial addition to your regular oral hygiene routine. If you are unsure about whether or not to brush your tongue, you can discuss it with your dentist or healthcare provider.

Is ChatGPT right here and I should start brushing my tongue?
# Great Cryonics Survey of 2022


The text that follows is a copy-paste from the survey creator's Facebook post, [shared with permission](https://www.facebook.com/groups/cryonicists/posts/2170336949839619/?comment_id=2171419116398069&reply_comment_id=2171420566397924&__cft__[0]=AZVOLUMeCIIY9wPXVNwhgSVLAUevOtQhsBqOiA-a60Gnkm0ulOm3vPQWaoEQG8mAMLtV9RZCXHGMZ-WRAmRTG4xfRDH0xdz2_B1eljvoPYjBQONCeOdPBKzpbAbbCjUTZZnda_kcl3fKWZxZdBQGFf65&__tn__=R]-R).

> Asking all Cryonicists and those considering it or curious about it to...
> ~ Please take the Great Cryonics Survey of 2022 ~
> 
> Go to --> bit.ly/cryosurvey2022
> 
> 
> Take the survey if:
> 1. You're signed up for Cryonics
> 2. You think Cryonics is interesting but you aren't ready to sign up yet.
> 3. You've decided to sign up but haven't gotten around to it yet.
> 
> The goal of this survey is to understand what makes Cryonicists *tick* using data rather than anecdotes.
> Here are some examples of the kinds of issues this survey is meant to understand:
> 1. How many Cryonicists are in favor of "mind uploading" and how many are against it?
> 2. How do Cryonicists think about death?
> 3. How many Cryonicists suffer from some form of death anxiety?
> 4. How many Cryonicists are men vs women?
> 5. What differentiates Cryonicists from non-Cryonicists with overall similar mindsets?
> 6. How affordable do Cryonicists think Cryonics is for them?
> 7. What do Cryonicists and non-Cryonicists think about pet Cryonics, and why?
> 8. How do Cryonicists identify politically?
> 9. Where do people first learn about Cryonics?
> 10. How do Cryonicists feel about children?
> 
> *And much more!*
> 
> All results are anonymous (we don't ask for names or the like). All results will be analyzed and that analysis released to the public in a few months.
> 
> bit.ly/cryosurvey2022

# Get help signing up
By the way, I've been helping people (decide whether to) sign up for years, and I'm now part of the new [Independent Cryonics Educators](https://www.alcor.org/alcor-ice-program/) program which supports people doing that, both logistically and financially, although I don't let this bias my recommendations. So don't hesitate to reach out to me on [Facebook](https://www.facebook.com/mati.roy.09/), [Twitter](https://twitter.com/matiroy9), or email (mathieu.roy.37@gmail.com). If you use the following code, you'll get a 50% discount on your first year of membership with Alcor: 2omojQif.
# Is school good or bad?

Should society eliminate schools? Should we have more compulsory schooling? Should you send your kids to school? Should you prefer to hire job candidates who have received more schooling, beyond school's correlation with the g factor? Should we consider the spread of education requirements to be a form of class war by the better-educated against the worse-educated which must be opposed for the sake of the worse-educated and the future of society?
# Our 2022 Giving

[Julia](https://juliawise.net/) and I usually decide on our annual giving at the end of each year. For the past few years this has been relatively straightforward: I was paid [a lot](https://www.jefftk.com/money), so we [donated 50%](https://www.jefftk.com/donations). This year is less so, because halfway through I [switched jobs](https://www.jefftk.com/p/leaving-google-joining-the-nucleic-acid-observatory) to work on something directly valuable. While I'm very excited about the potential of this work, it does pay enough less that even if we stopped donating we'd be keeping less for ourselves than before.

One option we considered was donating 50% of our income from the first half of the year, up until I switched jobs, and then some smaller proportion of the income from later in the year, representing about how much we intend to donate going forward. We thought about this, but decided not to:

*   The [collapse of FTX](https://www.jefftk.com/p/if-professional-investors-missed-this), taking with it their [philanthropic arm](https://forum.effectivealtruism.org/posts/xafpj3on76uRDoBja/the-ftx-future-fund-team-has-resigned), means that there is less money available for EA projects, so it's more valuable to have funding now than in the recent past.
    
*   We're not sure how to pick an amount for the second half of the year. Some of this is not being clear on what sort of donation norms we'd like to encourage for people working at the kind of organization they would otherwise be happy to fund, or who take lower-paying jobs for altruistic reasons. Another part is that what we personally want to do depends on what the post-FTX funding situation for our employers end up looking like.
    
*   I like having a simple explanation for what we're doing with our money, and while pro-rating isn't that complicated it's still something I'd rather avoid. Simplicity is also a reason why we lean away from donating via [lottery](https://www.jefftk.com/p/mildly-against-donor-lotteries) or [salary sacrifice](https://www.jefftk.com/p/passing-up-pay).
    

Instead, we've decided to just continue giving 50% for 2022, and give some smaller proportion in 2023. How much? Not sure yet.

As in past years, we're planning to divide our giving between [GiveWell](https://www.givewell.org/) (evidence-backed public health interventions, like distribution of antimalarial bed nets) and the [EA Infrastructure Fund](https://funds.effectivealtruism.org/funds/ea-community) (a wide range of projects intended to help the effective altruism community grow into the best community it can).

"Let's just keep doing what we were doing, and decide later what we want to do later" is not a very satisfying conclusion, but I'm hoping things are clearer next year.
# Race to the Top: Benchmarks for AI Safety

*This is an executive summary of a post from my personal blog, also* [*cross-posted*](https://forum.effectivealtruism.org/posts/saEXX9Nucz8mh9XgB/race-to-the-top-benchmarks-for-ai-safety) *from the EA Forum. **Read the full texts*** [***here***](https://isaduan.github.io/isabelladuan.github.io/posts/first/)***.** *

Summary
-------

Benchmarks support the empirical, quantitative evaluation of progress in AI research. Although benchmarks are ubiquitous in most subfields of machine learning, they are still rare in the subfield of *AI safety*.

I argue that **creating benchmarks should be a high priority for AI safety.** While this idea is not new, I think it may still be underrated. Among other benefits, benchmarks would make it much easier to:

*   track the field’s progress and focus resources on the most productive lines of work;
*   create professional incentives for researchers - *especially Chinese researchers* \- to work on problems that are relevant to AGI safety;
*   develop auditing regimes and regulations for advanced AI systems.

Unfortunately, **we cannot assume that good benchmarks will be developed quickly enough “by default."** I discuss several reasons to expect them to be undersupplied. I also outline actions that different groups can take today to accelerate their development.  
  
For example, **AI safety researchers can help by:**

*   directly trying their hand at creating safety-relevant benchmarks;
*   clarifying certain safety-relevant traits (such as “honesty” and “power-seekingness”) that it could be important to measure in the future;
*   building up relevant expertise and skills, for instance by working on other benchmarking projects;
*   drafting “benchmark roadmaps,” which identify categories of benchmarks that could be valuable in the future and outline prerequisites for developing them.

And **AI governance professionals can help by:**

*   co-organizing workshops, competitions, and prizes focused on benchmarking;
*   creating third-party institutional homes for benchmarking work;
*   clarifying, ahead of time, how auditing and regulatory frameworks can put benchmarks to use;
*   advising safety researchers on political, institutional, and strategic considerations that matter for benchmark design;
*   popularizing the narrative of a “race to the top” on AI safety.

Ultimately, **we can and should begin to build benchmark-making capability now.** 

Acknowledgment
--------------

I would like to thank Ben Garfinkel and Owen Cotton-Barratt for their mentorship, Emma Bluemke and many others at the [Centre for the Governance of AI ](https://www.governance.ai/) for their warmhearted support. All views and errors are my own. 

Future research
---------------

I am working on a paper on the topic, and if you are interested in benchmarks and model evaluation, especially if you are a technical AI safety researcher, I would love to [hear](mailto: isabelladuan@uchicago.edu) from you!
# Who are some prominent reasonable people who are confident that AI won't kill everyone?

**Bounty \[closed\]:** $30 for each link that leads to me reading/hearing ~500 words from a Respectable Person arguing, roughly, "accelerating AI capabilities isn't bad," and me subsequently thinking "yeah, that seemed pretty reasonable." For example, linking me to [nostalgebraist](https://www.lesswrong.com/users/nostalgebraist) or [OpenAI's alignment agenda](https://openai.com/blog/our-approach-to-alignment-research/) or [this debate](https://www.lesswrong.com/posts/WxW6Gc6f2z3mzmqKs/debate-on-instrumental-convergence-between-lecun-russell).[^o564iy2gth] Total bounty capped at $600, first come first served. All bounties (incl. the total-bounty cap) doubled if, by Jan 1, I can consistently read people expressing unconcern about AI and not notice a status-yuck reaction.

**Context:** I notice that I've internalized a message like "thinking that AI has a <1% chance of killing everyone is stupid and low-status." Because I am a monkey, this damages my ability to consider the possibility that AI has a <1% chance of killing everyone, which is a bummer, because my beliefs on that topic affect things like whether I continue to work at my job accelerating AI capabilities.[^lu2vc2zys2q]

I would like to be able to consider that possibility *rationally*, and that requires neutralizing my status-yuck reaction. One promising-seeming approach is to spend a lot of time looking at lots of of high-status monkeys who believe it!

[^o564iy2gth]: Bounty excludes things I've already seen, and things I would have found myself based on previous recommendations for which I paid bounties (for example, other posts by the same author on the same web site).  

[^lu2vc2zys2q]: Lest ye worry that [providing links to good arguments] will lead to [me happily burying my head in the sand and continuing to hasten the apocalypse] -- a lack of links to good arguments would move much more of my probability-mass to "Less Wrong is an echo chamber" than to "there are basically no reasonable people who think advancing AI capabilities is good."
# Take 4: One problem with natural abstractions is there's too many of them.

*As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day for 25 days. Or until I run out of hot takes.*

Everyone knows what the deal with [natural abstractions](https://www.lesswrong.com/tag/natural-abstraction) is, right? Abstractions are regularities about the world that are really useful for representing its coarse grained behavior - they're building blocks for communicating, compressing, or predicting information about the world. An abstraction is "natural" if it's so easy to learn, and so broadly useful, that most right-thinking agents will have it as part of their toolbox of abstractions.

The dream is to use natural abstractions to pick out what we want from an AI. Suppose "human values" are a natural abstraction: then both humans and a world-modeling AI would have nearly the exact same human values abstraction in their toolboxes of abstractions. If we can just activate the AI's human values abstraction, we can more or less avoid misalignment between what-humans-are-trying-to-pick-out and what-abstraction-the-AI-takes-as-its-target.

One might think that the main challenge to this plan would be if there are too few natural abstractions. If human values (or agency, or corrigibility, or whatever nice thing you want to target) aren't a natural abstraction, you lose that confidence that the human and the AI are pointing at the same thing. But it's also a challenge if there are *too many* natural abstractions.

Turns out, humans don't just have one abstraction that is "human values," [they have a whole lot of 'em](https://www.lesswrong.com/s/aJvgWxkCBWpHpXti4/p/nA3n2vfCy3ffnjapw). Humans have many different languages / ontologies we use to talk about people, and these use different abstractions as building blocks. More than one of these abstractions gets called "human values," but they're living in different ontologies / get applied in different contexts.

If none of these abstractions we use to talk about human values are natural, then we're back to the first problem. But if any of them *are* natural, it seems just as plausible that nearly all of them are. Abstractions don't even have to be discrete - it's perfectly possible to have a continuum.

This complicates the easy alignment plan, because it means that the structure of the world is merely doing *most* of the work for us rather than *almost all* of the work. The bigger the space of semantically-similar natural abstractions you have to navigate, the more you have to be careful about your extensional definitions, and the higher standards you have to have for telling good from bad results.
# How should I judge the impact of giving $5k to a family of three kids and two mentally ill parents?

I am giving $5k to a case worker who will buy goods for a family in New Orleans. The mother and father of the family have schizophrenia and do not work. (The father is violent, and does not live with his children.) The mother and her children live in a two bedroom house with 10 other people. I don’t know the children’s ages, but I believe they are around 3, 7, and 12 years old. 

The only directive I’ve given the case worker is to alleviate immediate and preventable suffering as best possible. “Put out fires,” so to speak. 

I would like to scale this giving into a larger program of philanthropy if I can show that my $5k makes a positive impact. …This is where I get vague and need Rat/EA help… how do you think I should judge success? QALYs? How would I even judge QALY improvement?… In the end, I want to be able to say $5k to a family of 5 reduces suffering by a factor of X. I hope to compare these measures to EA-backed giving opportunities.

…Some additional background of why I’m doing this… 

It’s easier to raise money for local causes that prevent suffering than something like the Against Malaria Foundation. You can see an example of this in [the 11:11 group donations](https://philosophers.notion.site/30ff5ddc9ea24ca3bfec32ce177232bf?v=a9579fd627824183a489a1757342408c). A majority of the causes group members gave to were local causes, despite the entire group going through 11 weeks of EA indoctrination. 

With proper rat/EA methodology, I hope to establish clear OKRs for a local charitable program. We would aim to fundraise to achieve these OKRs before diverting cash to other locals in need. (The ultimate goal is to raise money and reduce suffering in local communities, spreading excess resources from the wealthiest communities to the poorest communities.)

Curious to hear your ideas.
# ChatGPT and Ideological Turing Test

It seems to me that ChatGPT should be able to pass the Ideological Turing Test -- to generate a convincingly looking argument for any side of an issue. It is obvious how to use this ability for evil purposes: write [the bottom line](https://www.lesswrong.com/posts/34XxbRFe54FycoCDw/the-bottom-line), and generate the arguments.

I think this can also be used for *good* purposes, and I invite you to brainstorm how. Here are some of my ideas:

Generate a *heresy* for yourself. People sometimes pretend to think critically about their own beliefs, and sometimes they kinda honestly try, but they still unconsciously [avoid the weak points of their beliefs](https://www.lesswrong.com/posts/dHQkDNMhj692ayx78/avoiding-your-belief-s-real-weak-points). Instead of generating a heretical thought that you know you can easily debunk, use ChatGPT to generate counter-arguments to your beliefs.

Prepare yourself for a *debate* on certain topic, by asking ChatGPT to generate arguments against your position. This might give you a decent idea of what arguments you should expect to meet in the real debate.

Perhaps teachers could use it similarly, to find out the most frequent *misconceptions* about the topic they are going to teach, and then they could adjust their lessons accordingly.

I wonder if listening to (ChatGPT-generated) "both sides of the story" could increase your chance to make the right guess. The experiment could be designed like this: An expert in certain field chooses a question that (according to the expert's knowledge) has a known correct answer, but most people are not familiar with it. Then someone else asks ChatGPT to create convincing arguments for both sides. The participants are randomly divided into two groups. First group only hears the question, and then tries to guess the right answer. Second group hears the question, then reads the arguments for both sides, and then tries to guess the right answer. Will the second group be *more successful* on average? If the answer is "yes", this may be a useful way to figure out the truth about questions where most people are wrong (and thus ChatGPT might mislead you if you ask it to only provide the single best answer).

But even if it does not increase your chance of guessing correctly, perhaps teachers could use this to create the feeling of *curiosity* in their students. Before the lesson, give students the generated arguments for both sides. During the lesson, explain how it actually works, and provide explanations why the other side is wrong. This may prevent [hindsight bias](https://www.lesswrong.com/tag/hindsight-bias) (students concluding after the lesson that this was completely obvious, when in fact it wasn't before the lesson).