Harley-ml/lesswrong huggingface dataset:
# Double-Crux Workshop (Pilot program)

The Norfolk Virginia Rationalists will be hosting a workshop this Sunday on the art of the [Double-Crux](https://www.lesswrong.com/tag/double-crux): a technique for uncovering the root causes of (and potentially resolving) complex disagreements. Anyone and everyone is welcome; no prior knowledge is required. The workshop will be presented by Yitzi Litt (that's me!), and as this is the first time Yitzi has ever done this (😅), expect it to be a smaller-scale event which may not be perfectly polished.

The workshop will take place at the local Kung Foo Tea from noon to  roughly 1:30pm. Feel free to [join us on Discord](https://discord.gg/2hHvDtRNaw), or DM me or [Willa](https://www.lesswrong.com/users/willa) for more info.

Looking forward to seeing you!

  
—Yitzi Litt
# What would you expect a massive multimodal online federated learner to be capable of?

One intuitive picture I have for what very rapid ML capability gains might look like is a massive multimodal deep learning model that uses some form of online federated learning to continually learn from many devices simultaneously, and which is deployed to hundreds of millions or billions of users. For example, imagine a multimodal Google or Facebook chatbot with say 10 trillion parameters that could interact with billions of users simultaneously and which improved its weights from every interaction. My impression is that we basically have the tech for this today, or we will in the very near future. Now add in some RL to actively optimize some company-relevant goals (ad revenue, reported user satisfaction, etc.) and, intuitively at least, that seems to me like it's very close to the kind of scary AGI we've been talking about. But that seems like it could easily be 2-5 years in the future rather than 10-50.

Did I misunderstand something? How close to scary-type AGI would you expect this kind of model to become within a few months after deployment?
# Artificial Moral Advisors: A New Perspective from Moral Psychology

Tired: can humans solve artificial intelligence alignment?

Wired: can artificial intelligence solve human alignment?

“Philosophers have recently put forward the possibility of achieving moral enhancement through artificial intelligence… proposing various forms of ‘artificial moral advisor’ (AMA) to help people make moral decisions without the drawbacks of human cognitive limitations. In this paper, we provide a new perspective on the AMA, drawing on empirical evidence from moral psychology to point out several challenges to these proposals that have been largely neglected by AI ethicists.”
# [Linkpost] Can lab-grown brains become conscious?

*Linkpost for:* [*https://media.nature.com/original/magazine-assets/d41586-020-02986-y/d41586-020-02986-y.pdf*](https://media.nature.com/original/magazine-assets/d41586-020-02986-y/d41586-020-02986-y.pdf)

I figured this was probably of interest to people here, e.g. for seeing how scientists feel about this type of ethical concern and for thinking about the potential consequences of lab-grown brains. The beginning of the article:

> In Alysson Muotri’s laboratory, hundreds of miniature human brains, the size of sesame seeds, float in Petri dishes, sparking with electrical activity. 
> 
> These tiny structures, known as brain organoids, are grown from human stem cells and have become a familiar fixture in many labs that study the properties of the brain. Muotri, a neuroscientist at the University of California, San Diego (UCSD), has found some unusual ways to deploy his. He has connected organoids to walking robots, modified their genomes with Neanderthal genes, launched them into orbit aboard the International Space Station, and used them as models to develop more human-like artificial-intelligence systems. Like many scientists, Muotri has temporarily pivoted to studying COVID-19, using brain organoids to test how drugs perform against the SARS-CoV-2 coronavirus. But one experiment has drawn more scrutiny than the others. 
> 
> In August 2019, Muotri’s group published a paper in Cell Stem Cell reporting the creation of human brain organoids that produced coordinated waves of activity, resembling those seen in premature babies1 . The waves continued for months before the team shut the experiment down.
# Have you considered getting rid of death?

No person who wishes to continue living should have to die, ever. Check out the post on substack! 

I believe that human immortality (increasing healthspan, eliminating causes of death, etc.) is something we as a species ought to urgently work towards. I am not a scientist, so I will primarily talk about the philosophical, political, and governance related implications of longevity. 

I intend to increase my knowledge of biology & other important to immortality disciplines because that's enjoyable but also so that I may more accurately evaluate people, policies, and projects in the longevity / immortality space. Any suggestions for increasing my biology knowledge? I'm at about a "took biology for science majors in undergrad plus has read biology popular science books for fun" knowledge level right now.

EDIT 2022/09/13: I've created a Google Group called [Immortality Studies](https://groups.google.com/g/immortality-studies/about) to serve as a public discussion & mailing list plus coordination point for all of us interested in getting rid of death. I prefer to communicate about this subject area publicly, but you are welcome to message me here on LessWrong or email me at walambert@pm.me for private conversations (it may take me some time to get back to you since I have little slack in my life right now and am also prioritizing public conversations for coordination & transparency reasons).
# An Audio Introduction to Nick Bostrom

![Introduction to Nick Bostrom cover image](https://radiobostrom.com/images/twitter-card-introduction-series.jpg)

Team Radio Bostrom has just released [An Audio Introduction to Nick Bostrom](https://radiobostrom.com/introduction).

It offers a deep dive into Nick Bostrom's ideas, including: existential risk, wise philanthropy, the ethics of AI, transhumanism, and the case for speeding up some investments in technology while slowing down others.

We'd love to hear your thoughts!

## How you can help

If you'd like to help spread the world, please like and retweet this:
[https://twitter.com/peterhartree/status/1564137587496947712](https://twitter.com/peterhartree/status/1564137587496947712)

And please, share your thoughts on how we could improve this. Or tell us some other people who might be good for a similar series...

See also: [Radio Bostrom launch announcement](https://forum.effectivealtruism.org/posts/zXdvCQHtyZqGu4XwD/radio-bostrom-audio-narrations-of-papers-by-nick-bostrom).
# *New* Canada AI Safety & Governance community

Hello everyone,

Wyatt here, the new AIS/G field-builder for Canada. In June I received an EA LTFF fund grant to "connect, expand and enable the AGI Safety & Governance community in Canada", with the goal of making Canada a net positive force on the world stage for transformational AI.

We now have a slack server, events calendar, some proposed projects, and a monthly newsletter (coming out this Wednesday), so **if you are based in Canada and want to be in the loop,** [**sign up here**](https://airtable.com/shrjJqdF2YdXnuLxM)**.**

Looking forward to the year ahead.
# How to plan for a radically uncertain future?

While playing around with Stable Diffusion last week I had an epiphany. I realized that while I believe with high probability that the world will be radically different in 15 years, all my investments, living arrangements, and other long term plans implied a belief that the world would be roughly similar to now. This shook me to my core.

More concretely, I have investments in suburban American property, a 401k plan that I maximize and invest in a broadly diversified low cost fund, many children whom I educate in roughly conventional subjects (from college age down to toddler age), and a job that I intend to keep for the long term. But I believe there is a less than 1% probability that my 401k will mean anything by the time I can withdraw it. I don't believe that my property (or any suburban American property) will be valuable in 15 years under most scenarios I consider likely. I don't think my children's education will serve them particularly well during their adulthood. 

What are my beliefs about the future? I believe that AI and biology at least (two subjects I know well and follow closely) are accelerating quickly, with logarithmic increases of capability and similar log decreases of cost. I believe that it is early days for both fields, and these log curves will be extended further for years to come.

Given those beliefs, the thing I expect most is radical change, in a direction that I can't predict well. I'm not biased towards a doom scenario (AI takeover, engineered unstoppable plague), nor towards a utopian scenario (beneficent AGI, biological immortality). But I am biased strongly away from the continuation of the status quo, away from any regime where I can sell my house, live off my 401k, watch my children take jobs that exist today and have their own families. 

I suspect others here hold similar beliefs about the future. How are you preparing now? How do you invest, how do you rear your young, what actions do you take that let you sleep well, knowing that whatever tomorrow brings your rationality has armed you in advance?
# ML Model Attribution Challenge [Linkpost]

From the [ML Model Attribute Challenge website](https://mlmac.io): 

> Large language models (LLMs) may enable the generation of misinformation at scale, especially when fine-tuned on malicious text. However, it is unclear how widely adversaries use LLMs to generate misinformation today, partly because no general technical method has been widely deployed to attribute a fine-tuned LLM to a base LLM.
> 
> The ML Model Attribution Challenge (MLMAC) challenges contestants to develop creative technical solutions for LLM attribution. Contestants will attribute synthetic text written by fine-tuned language models back to the base LLM, establishing new methods that may provide strong evidence of model provenance. Model attribution would allow regulatory bodies to trace intellectual property theft or influence campaigns back to the base model. This competition works toward AI assurance by developing forensic capabilities and establishing the difficulty of model attribution in natural language processing and could contribute to the maturation of AI forensics as a subfield of AI security.
> 
> This competition incentivizes open research in this area. Each winning contestant must publish their method, and after the competition concludes, we will release query logs for each fine-tuned model that can enable post-hoc analysis of blind attribution methods. We have planned subsequent activity in this area to institutionalize practical model attribution research.

The contest concludes next Friday, September 9th. $10,000 in prizes will be awarded amongst the winning submissions. MLMAC is a joint effort of organizers including Schmidt Futures, Hugging Face, Microsoft, and MITRE. 

For more analysis of risks from automated misinformation, see "[Risks from Automated Persuasion](https://www.lesswrong.com/posts/5cWtwATHL6KyzChck/risks-from-ai-persuasion)" (Barnes, 2021), "[Persuasion Tools](https://www.lesswrong.com/posts/qKvn7rxP2mzJbKfcA/persuasion-tools-ai-takeover-without-agi-or-agency#Six_examples_of_persuasion_tools)" (Kokotajlo, 2020), and [my summary](https://www.lesswrong.com/posts/H4jH7J2BbeokH3Chu/key-papers-in-language-model-safety#Persuasion). For a broader discussion of language model misuse, see "[Ethical and social risks of harm from Language Models](https://arxiv.org/pdf/2112.04359.pdf)" (Weidinger et al., 2021).
# Survey of NLP Researchers: NLP is contributing to AGI progress; major catastrophe plausible

I was part of a group that ran a [PhilPapers-style](https://philpapers.org/surveys/) survey and metasurvey targeting NLP researchers who publish at venues like ACL. Results are [here](https://nlpsurvey.net/) ([Tweet-thread version](https://twitter.com/_julianmichael_/status/1563203792006684673)). It didn't target AGI timelines, but had some other questions that could be of interest to people here:

*   **NLP is on a path to AGI:** 58% agreed that *Understanding the potential development of artificial general intelligence (AGI) and the benefits/risks associated with it should be a significant priority for NLP researchers.*
    *   Related: 57% agreed that *Recent developments in large-scale ML modeling (such as in language modeling and reinforcement learning) are significant steps toward the development of AGI.*
*   **AGI could be revolutionary:** 73% agreed that *In this century, labor automation caused by advances in AI/ML could plausibly lead to economic restructuring and societal changes on at least the scale of the Industrial Revolution.*
*   **AGI could be catastrophic:** 36% agreed that *It is plausible that decisions made by AI or machine learning systems could cause a catastrophe this century that is at least as bad as an all-out nuclear war.*
    *   46% of women and 53% of URM respondents agreed.
    *   The comments suggested that people took a pretty wide range of interpretations to this, including things like OOD robustness failures leading to weapons launches.
*   **Few scaling maximalists:** 17% agreed that *Given resources (i.e., compute and data) that could come to exist this century, scaled-up implementations of established existing techniques will be sufficient to practically solve any important real-world problem or application in NLP.*
    *   The metasurvey responses predicted that 47% would agree to this, so there are fewer scaling maximalists than people expected there to be.
*   **Optimism about ideas from cognitive science:** 61% agreed that *It is likely that at least one of the five most-cited systems in 2030 will take clear inspiration from specific, non-trivial results from the last 50 years of research into linguistics or cognitive science. *
    *   This strikes me as very optimistic, since it's pretty clearly false about the most cited systems today.
*   **Optimism about the field:** 87% agreed that *On net, NLP research continuing into the future will have a positive impact on the world.*
    *   32% of respondents who agreed that NLP will have a positive future impact on society also agreed that there is a plausible risk of global catastrophe.
*   **Most NLP research is crap:** 67% agreed that *A majority of the research being published in NLP is of dubious scientific value.*
# AI Box Experiment: Are people still interested?

I have a novel plan for the AI player that I believe will work against most gatekeeper players. Even knowing how it works, it would work against me! It isn't quite ready, but I'd like to know if there is still interest around the topic before finishing work on the preparation. 

There's enough incentive for both players to play well that I don't think that we need to involve money. (I'm also very reluctant to send money to strangers on the internet!)

We would be playing under Tuxedage's rules: [https://tuxedage.wordpress.com/2013/09/04/the-tuxedage-ai-box-experiment-ruleset/](https://tuxedage.wordpress.com/2013/09/04/the-tuxedage-ai-box-experiment-ruleset/). 

Edit: I have finished preparation, and I'm now scheduling the time with the Gatekeeper.

Edit 2: The game was a Gatekeeper victory (i.e. I, the AI Player, lost). Read about it here: [Gatekeeper Victory: AI Box Reflection](https://www.lesswrong.com/posts/h5yfjFpdtqE3vATQ6/gatekeeper-victory-ai-box-reflection).
# How might we make better use of AI capabilities research for alignment purposes?

When I check ArXiv for new AI alignment research papers, I see mostly capabilities research papers, presumably because most researchers are working on capabilities. I wonder if there’s alignment-related value to be extracted from all that capabilities research, and how we might get at it. Is anyone working on this, or does anyone have any good ideas?
# Progress links and tweets, 2022-08-31

**Links**
---------

*   [The Breakthrough Institute is hiring directors to run its new DC office](https://thebreakthrough.org/articles/washington-co-directors) (via [@atrembath](https://twitter.com/atrembath/status/1561078837772840960))
*   [Stable Diffusion: an AI image generator you can download and run](https://stability.ai/blog/stable-diffusion-public-release) (via [@elidourado](https://twitter.com/elidourado/status/1561818563517861891))

**Queries**
-----------

*   [Why didn’t clockwork technology get applied to other practical purposes for hundreds of years?](https://twitter.com/jasoncrawford/status/1563222565891362828)
*   [Who are the best economic historians that focus a lot on technology?](https://twitter.com/_brianpotter/status/1561698612001882125) (@_brianpotter)
*   [What questions in economics have been “settled” since 1945?](https://twitter.com/dkedrosky/status/1561725273032142848) (@dkedrosky)

**Quotes**
----------

*   [We should all have a little more awe and gratitude for the amazing world we live in](https://twitter.com/jasoncrawford/status/1562909025507102720)
*   [In 1809, books or carpets in the home were considered luxuries](https://twitter.com/jasoncrawford/status/1564099755264942081)
*   [Voltaire on the religious tolerance engendered by capitalism](https://twitter.com/jasoncrawford/status/1564100399191232512)

**Tweets**
----------

*   [Let’s retire the phrase “lifted out of poverty”](https://twitter.com/jasoncrawford/status/1562537534453714944)
*   [Negativity should be productive](https://twitter.com/jasoncrawford/status/1563223033090740226)
# Safety Committee Resources

A friend recently asked me for some advice about starting a safety committee for their contra dance. This got me pulling together some resources, so I thought I'd share them here as well.

Overall, I think safety committees can be positive, but there are also a lot of ways you can accidentally make things worse. For example, in [BIDA](https://bidadance.org)'s case, when we first started we did not do a very good job laying out what people could expect from us. A group of amateur volunteers is, for example, not generally going to be able to get to the bottom of disputes, and by leading people to expect service beyond what we could provide we let them down. Which is quite bad: sharing traumatic details is generally painful, and you really don't want to leave someone wishing they had never come to you.

I think the current version of our "[How We Can Help](https://www.bidadance.org/safety/approach)" page does pretty well here, but earlier versions didn't, and we also didn't do a good job in conversations with people reporting issues. I [wrote some](https://www.jefftk.com/p/new-bida-safety-docs) about this re-evaluation at the time.

Other resources, for people who are thinking of starting doing this kind of work:

*   [Julia Wise](https://juliawise.net/)'s [The community health team's work on interpersonal harm in the community](https://forum.effectivealtruism.org/posts/NbkxLDECvdGuB95gW/the-community-health-team-s-work-on-interpersonal-harm-in). She does this work professionally [in the EA community](https://forum.effectivealtruism.org/posts/hYh6jKBsKXH8mWwtc/a-contact-person-for-the-ea-community), and this post both lays out some of how her team thinks about this work and gives a long list of situations they've handled. While the EA community is not a dance community, there's still a lot that's relevant.
    
*   My [Taking Someone Aside](https://www.jefftk.com/p/taking-someone-aside) and [Taking a Safety Report](https://www.jefftk.com/p/taking-a-safety-report), getting into some detail about two specific components of the work.
    
*   [Dennis Merritt](https://www.denniscmerritt.com/)'s [Tales of a Dance Monarch](https://www.denniscmerritt.com/wordpress_org/tales-of-a-dance-monarch/). He walks through eleven different situations he was involved in over the years, and I think the post is very helpful for getting a sense of what sorts of situations come up in a dance context.
    
*   [Tales From the Front Line of Ombuds Work: Handling Sexual Harassment Cases](https://www.ombudsassociation.org/assets/docs/JIOA_Articles/Special_Issue_Full_Narrative.pdf). Twelve anonymized cases in a university context. Somewhat less applicable, since it's such a different setting and framework, but it's still helpful to read example cases.
    

If anyone else has resources to recommend, please share!

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid02RCgcoPKNWGEZjHYXK85MGPf3uRcLHiGJ1vdzVBhyTqURwFd54nSDGvXeyDQDauR5l)*
# Infra-Exercises, Part 1

There have been a few remarks about how my and Vanessa’s posts on Infra-Bayesianism seem interesting, but that it's quite a formidable body of work to chew through. 

So anyone interested in learning Inframeasure theory to the level of proficiency where they are actually able to develop their own proofs about it and casually deploy it as a tool on other problems is probably not best served by the existing posts. You can only go so far by reading about something without ever applying it.

And so, to help people get a taste for the underlying math, I have constructed a sheet of exercises analogous to Scott's exercise sheets about fixpoints, to guide the reader through inventing most of the basic theory for themselves.

Said problem sheet was dramatically improved by Jack Parker, Connall Garrod, and Thomas Larsen as part of a SERI project, by including proof sketches, relevant definitions, and commentary, and generally making the whole exercise sheet dramatically more polished than it would otherwise be. Several others also contributed to testing out the exercises and providing solutions and feedback, most prominently, Viktoria Malyasova.  
  
This first sheet focuses on introducing Inframeasures, and working through the Fundamental Theorem of Inframeasures. Admittedly, this is a bit distant from much of the later work (further problem sheets are upcoming), but it *is* a very important result to have a solid understanding of.

There is more information on how to approach the problems in the introduction of the document. We may potentially release more problem sheets in the future, depending on what the feedback to this one looks like. And if the polished versions are not created, I will publish the unpolished ones.

The link to the problems is [right here.](https://drive.google.com/file/d/1aKvftxhG_NL2kfG3tNmtM8a2y9Q1wFHb/view?usp=sharing)
# A Starter-kit for Rationality Space

LessWrong has gotten big over the years: 31,260 posts, 299 sequences, and more than 120,000 users.[^wolelcybp4c] It has budded offshoots like the [alignment](https://www.alignmentforum.org/) and [EA](https://forum.effectivealtruism.org/) forums and earned itself [recognition as "cult"](https://www.reddit.com/r/OutOfTheLoop/comments/3ttw2e/what_is_lesswrong_and_why_do_people_say_it_is_a/). Wonderful!

There is a dark side to this success: as the canon grows, it becomes harder to absorb newcomers (like myself).[^66gf2ici9i3] I imagine this was the motivation for the recently launched "[highlights from the sequences](https://www.lesswrong.com/highlights)".

To make it easier on newcomers (veterans, you're also welcome to join in), [I've created an Obsidian starter-kit for taking notes](https://github.com/jqhoogland/rationalia-starter) on the LessWrong core curriculum ([the Sequences](https://www.lesswrong.com/rationality), [Codex](https://www.lesswrong.com/codex), [HPMOR](https://www.lesswrong.com/hpmor), [best of](https://www.lesswrong.com/bestoflesswrong), [concepts](https://www.lesswrong.com/tags/all), various [jargon](https://www.lesswrong.com/tag/lesswrong-canon-on-rationality), and other odds and ends).

There's built-in support to export notes & definitions to [Anki](https://ankisrs.net/), goodies for tracking your progress through the content, useful metadata/linking, and pretty visualizations of rationality space...

![Vault graph](https://github.com/jqhoogland/rationalia-starter/raw/main/Misc/Media/vault-graph.png)

In the South, we see the HPMOR peninsula; in the West, the "Posts" nebulae; and in the North/East, "Concepts" space.

It's not perfect — I'll be doing a lot of fine-tuning of broken links & missing tags as I work my way through all the content — but there should be enough already in-place that you can get some value out of it. I'd love to hear your feedback, and if you're interested in contributing, please reach out. I'll also soon be adding support for the [AF](https://www.alignmentforum.org/) and the [EAF](https://forum.effectivealtruism.org/) .

More generally, I'd love to hear your suggestions for new aspiring rationalists. For example, a round of users proposed alternative reading orders about a decade ago (by [Academian](https://www.lesswrong.com/posts/JKnLHkKtfpJxPXArd/draft-of-a-suggested-reading-order-for-less-wrong), [jimrandomh](https://www.lesswrong.com/posts/JKnLHkKtfpJxPXArd/a-suggested-reading-order-for-less-wrong-2011), and [XiXiDu](https://www.lesswrong.com/posts/zEm5B4t3Sonj8PySN/rewriting-the-sequences?commentId=dJKmpMWhg7ny3ZFS4)). Is it time for a revisit in 2022?

[**Check out the repo**](https://github.com/jqhoogland/rationalia-starter)
==========================================================================

[^wolelcybp4c]: From what I can scrape off the graphql endpoint. 

[^66gf2ici9i3]: Already a decade ago, jimrandomh was worrying about LW's intimidation factor — we're now about an order of magnitude further along.
# A Richly Interactive AGI Alignment Chart

Hi, most of you have probably seen the AGI Alignment charts generated by Rob Bensinger and Michaël Trazzi. I thought those charts could be useful to help people connect with others on Twitter that share similar (or very different) views on the two seminal questions on this topic ("AGI when?" and "AGI safe?") However, it would need to be possible for people to add themselves to the chart, and have richer tools for filtering and browsing avatars on the chart.

Here is my own chart that adds all these features: [https://agialignment.com](https://agialignment.com)

Please let me know if you have any difficulty adding yourself to the chart, or updating/deleting your info from the chart if I already put you on there (based on your existing public predictions, or my own crude guestimates, if I thought you had said enough on the subject for this to be worth attempting)

I know the security-minded folks among you are going to be suspicious of the "log in with twitter" feature of my site, but I think upon reflection you would have to agree this is a rare, appropriate use case for this feature. Note that I only ask for minimal rights and then do my best to guide users to immediately fully disconnect my app from twitter again. All the code is available on twitter and is open source.

FYI if you are in an AI alignment org and want to collaborate to improve this chart and/or take stewardship of it, get in touch. 

Here is the announcement thread for this interactive chart on twitter, in case there is useful info in the future in this discussion: https://twitter.com/lisperati/status/1565496833677201409
# Exporting Hangouts History

Yesterday I received a worrying email:

> From: The Google Apps Team  
> Subject: Important information about the upgrade from Google Hangouts to Google Chat
> 
> Earlier this year, we announced that all Hangouts users would be upgraded to Google Chat. As the last step of the upgrade, starting November 1, 2022, Hangouts on the web will be upgraded to Chat on the web. We are emailing you because **some of your conversations or portions of conversations won't automatically migrate** from Hangouts to Chat.
> 
> If you wish to keep your Hangouts conversations, we strongly encourage you to use Google Takeout and download your data before January 1, 2023 when Hangouts data will be deleted.
> 
> ...

(Emphasis mine)

I followed the instructions and exported the data:

$ wc -l Takeout/Hangouts/Hangouts.json
1041592 Takeout/Hangouts/Hangouts.json

A million lines of data? What does this look like?

$ head -n 25 Takeout/Hangouts/Hangouts.json
{
  "conversations": \[
    {
      "conversation": {
        "conversation_id": {
          "id": "Ugx..."
        },
        "conversation": {
          "id": {
            "id": "Ugx..."
          },
          "type": "STICKY\_ONE\_TO_ONE",
          "self\_conversation\_state": {
            "self\_read\_state": {
              "participant_id": {
                "gaia_id": "103...",
                "chat_id": "103..."
              },
              "latest\_read\_timestamp": "136..."
            },
            "status": "ACTIVE",
            "notification_level": "QUIET",
            "view": \[
              "INBOX_VIEW"
            \],

Oh dear. Very far from human-readable.

I've now written a script to convert it to something readable ([code](https://github.com/jeffkaufman/hangouts-takeout/blob/main/process-hangouts-json.py)):

$ python3 process-hangouts-json.py Takeout/Hangouts/Hangouts.json
...
Jonah Sidman (2013-05-31 11:48 AM):
    I feel like I had a public transit question for you but now I don't remember what it was

Jeff Kaufman (2013-05-31 11:48 AM):
    well, let me know
...

Spot checking, I don't see anything in my script that's missing in the [online Chat interface](https://chat.google.com/), but it's hard to be sure.

I hope this script can be useful to others trying to read what Hangouts exports!

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/pfbid023kEe3h2oFUMLfUne4wNUBpve92kFo2obDr7uSxPHTYBGZQhKgTYMNQnFt3d5wpw9l)*
# Laziness in AI

I did a quick google search and couldn't find much in the way of "lazy ai", but it's starting to seem more and more obvious as a strategy for alignment.  Here's the thought process: 

\-\- Given almost any goal, an AI will want to obtain more power to be able to accomplish the goal, or to be more sure that it did accomplish the goal, or to prevent anyone from getting in its way of accomplishing the goal.

\-\- So even an AI with "human-level" intelligence will want to get more computing power, have safeguards against being shut down, and obtain resources like money and contacts.

\-\- Regular humans don't usually act like this[^hldorpyqf4l]. This has nothing to do with their intelligence (after all, we specified "human-level" intelligence), but instead with their willpower/personality/motivation. A normal human is more concerned with comfort, more afraid of hard work, and more complacent to do what everyone else is doing: not much. These traits together can be (somewhat harshly) described as Laziness.

So maybe one further safeguard to put on an AI is to make it lazy? No lazy person has ever attempted to take over the world, after all. But lazy people are often nice, and willing to, say, answer questions.

What would this look like in actuality? Give the AI harsh utility penalties for doing hard work, especially hard work *right now*. For example, at any given time, half of the AI's utility function depends on not doing work above a certain threshold *in the next hour* (So if the AI does well, 50% of its utility function would be permanently completed in its first hour, and after its second hour, 75%, and after its third hour, 87.5%). 

Is this an insanely bad idea? Would this cripple the AI of its most valuable tool? Are there other human characteristics that would possibly make sense to implant in an AI, like boredom or self-consciousness? Has anyone else written about this?

[^hldorpyqf4l]: Most people don't look at the world and say "what do I most want?" and "how can I best achieve it?" and then spend the next sixty years working at 100% capacity to gather resources, grow in power, and bend all of humanity to their iron will in hopes that this will give them a better chance at accomplishing their task.
# Can someone explain to me why most researchers think alignment is probably something that is humanly tractable?

From the view of someone who mostly has no clue what they are talking about (that person being me), I don’t understand why people working in AI safety seem to think that a successful alignment solution (as in, one that stops everyone from being killed or tortured) is something that is humanly achievable. 

To be more clear, if someone is worried about AI x-risk, but is also not simultaneously a doomer, then I do not know what they are hoping for.

I think there’s a fairly high chance that I’m largely making false assumptions here, and I understand that individual alignment schemes generally differ from each other significantly. 

My worry is that different approaches, underneath the layers of technical jargon which mostly go over my head, all are ultimately working towards something that looks like “accurately reverse engineer human values and also accurately encode them”.

If the above characterization is correct, then I don’t understand why the situation isn’t largely considered hopeless. Do people think that would be much more doable/less complex than I do? Why? 

Or am I just completely off-base here? (EDIT: by this I mean to ask if I’m incorrect in my assumption of what alignment approaches are ultimately trying to do)
# Request for Alignment Research Project Recommendations

Recent lists of alignment research projects such as [Richard Ngo's list of conceptual research projects](https://www.alignmentforum.org/posts/27AWRKbKyXuzQoaSk/some-conceptual-alignment-research-projects) and [Neel Nanda's list of future research directions](https://www.alignmentforum.org/posts/N6WM6hs7RQMKDhYjB/a-mechanistic-interpretability-analysis-of-grokking#Future_Directions) for understanding grokking and phase changes have been tremendously useful for me as a beginner in the field. They've given me a much clearer idea about the sorts of projects that would be useful for testing my fit for research while also having some potential to be at least somewhat valuable contributions. I believe many others feel the same.

Reading these lists has made me wonder whether there are other people as well who have ideas they think deserve further exploration but which they don't have time to work on themselves. If you have any such ideas, I'd love to see them, even if they're not that thoroughly thought out! Links to lists from other people would also be much appreciated. I think it would be valuable for many people who wish to start contributing to the field but feel like they haven't had enough opportunities to develop their [research taste](https://colah.github.io/notes/taste/) yet to have a resource containing various tractable project ideas related to multiple different research agendas.
# Bay Solstice 2022 Call For Volunteers

It's September, which means going to beaches, holding barbecues, and . . . preparing for this year's Bay Area Winter Solstice. This year's team will include me, Shauna, Nova, and Mingyuan. In the meantime, we're looking for volunteers.

First, we have logistics volunteers. Some of these, especially AV and lighting, will require a commitment to be kind of agenty in figuring out what you need to do, attending dress rehearsals to help test things, etc. The simple tasks will just require you to show up day of.  If you want to help with any of the following logistics tasks, please email [smkravec@celest.ai](mailto:smkravec@celest.ai):

1.  Someone to monitor the Twitch stream/simulcast and give updates to AV on that.
2.  Someone to work closely with Nova on the video side of AV; filming the event and running the live stream (possibly that's worth a separate role, they'd be responsible for the audio submix to the stream)
3.  Someone to help setup and run lighting
4.  5-10 people who can help with simple tasks on the day of, like ushering, taking tickets, and moving large objects

Second, we have creative tasks. We seem pretty good for people willing to do creative tasks this year, so you don't need to volunteer unless you're really enthusiastic. If you do volunteer, expect to have to audition. If you're interested in any of the following creative tasks, please email [scott@slatestarcodex.com](mailto:scott@slatestarcodex.com):

1.  Giving a speech (either writing your own, or giving a reading selected by the team)
2.  Singing a song (either your own, or performing one selected by the team)

We'll give volunteers more information by email, and we'll try to give everyone more information as we get closer to the event.

Thanks!
# Russian Food for Petrov Day

For the past few years I've considered Russian food to be the traditional Petrov Day food for my own celebrations, but I've never managed to write a post about it at a reasonable time. Today, a few weeks before Petrov Day, I managed to think about it at a reasonable time and figured I'd rather do an 80/20 post about it, and hope people contribute more in the comments. 

Why Russian Food?
-----------------

*   It's good for celebrations to have specific food, such as turkey for Thanksgiving. They feel more special that way. 
    *   Most people don't eat Russian food on a regular basis, so it can be a "special" thing, in a way that e.g. hamburgers wouldn't be.
*   The biggest Petrov Day stories (Petrov and Arkhipov) are about Soviets during the Cold War, so it's thematic.
*   Many rationalists, such as myself, are of Russian / Eastern European descent so already have holiday memories around Russian food.

Easy Mostly Veg*an Russian Foods
--------------------------------

*   Kvas, a beverage with tastes in between soda and beer, very lightly alcoholic, and available at your local Eastern European store.
*   Kasha aka buckwheat. [Very easy to make](https://www.inhabitedkitchen.com/grandmas-grain-buckwheat-kasha/). If you are a veg*n there are good fake chik'n bouillons available, and I strongly recommend cooking this with chicken / chik'n broth instead of plain water. Mushrooms and onions only if you're feeling fancy.
*   Veg\*an Sausage. It's just a good easy veg\*n protein option, and sausage is often paired with kasha.
*   Borscht, a vegetable soup with beets and cabbage, topped with (optional) sour cream. For an easy version, search for a recipe that uses canned beets. I'll put my dad's easy recipe in a comment below.
*   Black bread with butter and (optional) salmon roe.
*   Russian chocolates, individually wrapped. Available at your local Eastern European store. The Mishas (little bear) and chocolate covered jellies are generally well-liked.
# EA, Veganism and Negative Animal Utilitarianism

Veganism is on the face of it an extremely simple moral philosophy. If Animals have feelings, then eating animals is bad. Duh.

Effective Altruists tend to be people who actually take moral arguments at face value rather than ignoring them, and unsurprisingly many effective altruists are vegan.

But effective altruists also tend to be utilitarians, and as such are likely to want to minimise total animal suffering in aggregate, rather than taking the more deontologist view that you personally shouldn't cause suffering by eating animals.

So presumably vegan EAs assume that most farmed animals lives aren't worth living and so better not to eat farmed animals, even though this will cause them never to be raised in the first place.

It seems to me that most wild animals lives are just as terrible as those of farmed animals, especially for the vast majority of animals who give birth to thousands of young of which on average only 2 will ever reach adulthood. Their lives are presumably also not worth living.

As a utilitarian then, it should be far more important to wipe out as many animal habitats as possible rather than avoiding eating a relatively small number of animals by being a vegan.

I'm interested if anyone espouses such negative animal utilitarianism, or if anyone has any responses to it?
# (Link) I'm Missing a Chunk of My Brain

I'm blown away by this story. This perfectly normal woman is missing a big chunk of the left hemisphere, where the functions associated with language are typically placed. I guess that the main takeaway could be:

> "It seems that networks in the brain arrange in a particular way, but if you lose crucial brain regions as a baby - when the brain is still very plastic - these networks can reroute"
# A Game About AI Alignment (& Meta-Ethics): What Are the Must Haves?

I'm working on a pc & mobile game about metaethics, ethics & AI alignment. (Our steam page + announcement teaser will be up in 1-2 weeks.) It's important to me that we nail the AI alignment part and give people a good idea about why AI alignment is hard and optimism shouldn't be our default position.

Some constraints that come with the medium:

1.  We can't go into arguments that are too technical
2.  We need to keep the exposition of these ideas short

What do you think of the following framing of why AI alignment is hard? Are we missing any crucial considerations? (This is merely our internal script and will later be turned into more digestible dialogues, mini games, etc.)

There are 2 ways alignment can pan out: Either the values are programmed into the system, or they are learned by the system.

**1) Values are programmed into the system. But:**

*   There is no agreement on which values.
*   All ethical theories we have use ontologies (valence, pain, phenomenology, rights, justice) that don’t easily translate into code.
*   All the ethical theories we have (including those on moral uncertainty) have \*some\* cases (often in the area of population ethics) where most people disagree with their results.
*   In practice we usually use acceptable proxy values when designing AI rewards (maximize link clicks instead of maximizing user value). But using proxy values is extremely risky for sufficiently powerful optimizers. They might find optimized world states which massively neglect our actual values.  
     

**2) Values are learned by the system. But:**

*   There is much disagreement about the right dataset.
*   Even if we agree on a data set, it is unclear if the AI will generalize from the data set in an acceptable way.
*   Probably(?) the AI will find the most value in exploits. Exploits are cases where 1) the AI achieves its goals to a high degree, but 2) the AI has generalized poorly (by our standards). Examples: wireheading, goal modification, etc.  
     

**Other reasons for concern:**

*   Ethical truths are probably different from empirical truths. An advanced AI may learn empirical truths on its own from enough data, but it seems unlikely that it will automatically converge on the ethical truth. Instead, it seems that any degree of intelligence can be combined with any kind of goal. (Orthogonality Thesis)
*   There are some instrumental goals that probably many sufficiently advanced intelligences will converge on, for example accumulating resources, preventing others from interfering with its values, trying not to be shut down (e.g. by deceiving others into believing it has the same goals as them), etc. These instrumental goals make an iterative approach (run, test, fix, repeat) problematic.
# Beta Readers are Great

Back in January, I posted a [call for "beta readers"](https://www.cold-takes.com/seeking-beta-readers/): people who read early drafts of my posts and give honest feedback.

**The beta readers I picked up that way are one of my favorite things about having started Cold Takes.**

Basically, one of my goals with Cold Takes has been to explain my weirdest views clearly, but it's hard to write clearly without detailed feedback on where I'm making sense and where I'm not. I have lots of preconceptions and assumptions that I don't naturally notice. And writing a blog alone doesn't get me that feedback, because:

*   Most people don't want to _explain how they experienced a piece_ \- if they aren't enjoying it, they just want to click away.
*   And the people who _do_ want to help me out (e.g., friends and colleagues) aren't necessarily going to be honest enough, or representative enough of my target audience (which is basically "People who are interested in my topics but don't already have a ton of background on them").

I've tried a bunch of things to find good beta readers, from recruiting friends of friends (worked well for a bit, but I've written a lot of posts and it was hard to get sustained participation) to paying [Mechanical Turk workers](https://www.mturk.com/) to give feedback (some was good, but in general they were uninterested in my weird topics and rushed through the readings and the feedback as fast they could).

The people who came in through the recruiting call in January have been just what I wanted: they're interested in the topics of Cold Takes, but they don't already know me and my thoughts on them, and they give impressively detailed, thoughtful feedback on their reactions to pieces - often a wonderful combination of "intelligent" and "honest that a lot of the stuff I was saying confused the hell out of them." **Getting that kind of feedback has been a privilege.**

So: THANK YOU to the following beta readers, each of whom has submitted at least 3 thoughtful reviews (and gave permission to be listed here):

Lars Axelsson

Jeremy Campbell

Kanad Chakrabarti

Craig Chatterton

Justin Dickerson

Ethan Edwards

Edward Gathuru

Stian Grønlund

Bridget Hanna

Tyler Heishman

Adam Jermyn

Elliot Jones

Ed William

Scott Leibrand

Evan R. Murphy

John O’Neill

Jaime Sevilla

Josh Simpson

Joshua Templeton

George Thoma

Martin Trouilloud

Morgan Wack

Kevin Whitaker

Arjun Yadav

Patrick Young

If you want to sign up as a beta reader, you can use [this form](https://forms.gle/9GeYVsSW5gXZf21X8). I have a bunch of drafts coming on AI, as I'm working on a sequel to the [most important century series](https://www.cold-takes.com/most-important-century/) (working title is "The Most Important Century II: So What Do We Do?")
# What are you for?

My favorite movie reviewer is a YouTuber named [moviebob](https://www.youtube.com/c/moviebobcentral). Moviebob frequently talks about the idiotic things other reviewers are saying. I would not know these other reviewers even existed were it not for moviebob. Moviebob is a memetic amplifier for the reviewers he disagrees with.

The solution is simple: Ignore people who are wrong.

It can be worthwhile paying attention to [people who seem slightly wrong](https://www.lesswrong.com/posts/GpKrHPGcwtqiBxKJA/the-jordan-peterson-vs-sam-harris-debate) because they are wrong in interesting ways. Sometimes they are even right. But you should neither take seriously nor argue with those people who value winning an argument more than the truth. "Us" vs "Them" arguments are an endless source of such people.

"Us" vs "Them" definitions make it easy to draw battlelines. Which is a problem. The first casualty of war is the truth. Ignoring people who are wrong require operating counter to our tribal impulses.

Another problem with "Us" vs "Them" is the definitions are circular. How are "They" defined? "Not Us". Be wary of any idea which defines itself by what it's not. Such an idea is usually half of a [symbiotic conflict](https://www.lesswrong.com/posts/Myi7eyEGTPiPa3pbm/symbiotic-conflicts). Attach yourself to ideas that define themselves in absolute terms instead. Otherwise you end up mired in circular logic.

Circular ideas are a waste of attention. Including—sometimes—Atheism. Atheism is at its best when it is pro science. Atheism is at its worst when it is anti religion.

The easiest—and worst—way to demarcate "Us" is to define "Us" as "not Them". [Speak in the positive.](https://www.lsusr.com/blog/speak-in-the-positive.html) What are you for?