Harley-ml/lesswrong huggingface dataset:
# Jobs: Help scale up LM alignment research at NYU

[NYU is hiring alignment-interested researchers!](https://docs.google.com/document/u/0/d/1r-ZsBQK07OHohLa9c9qCoLHZVV80EvnIOfpsdfVk0eg/edit?fromCopy=true)

*   I'll be working on setting up a CHAI-inspired research center at NYU that'll focus on empirical alignment work, primarily on large language models. I'm looking for researchers to join and help set it up.
*   The alignment group at NYU is still small, but should be growing quickly over the next year. I'll also be less of a hands-on mentor next year than in future years, because I'll simultaneously be doing visiting position at Anthropic. So, for the first few hires, I'm looking for people who are relatively independent, and have some track record of doing alignment-relevant work.
*   That said, I'm not necessarily looking for a lot of experience, as long as you think you're in a position to work productively on some relevant topic with a few peer collaborators. For the pre-PhD position, a few thoughtful forum posts or research write-ups can be a sufficient qualification. We're looking for ML experimental skills and/or conceptual alignment skills/knowledge that could be relevant to empirical work, not necessarily both.
*   Our initial funding is coming from Open Philanthropy, for a starting project inspired by AI Safety Via Debate. Very early results from us (and a generally-encouraging note from Paul C.) [here](https://www.alignmentforum.org/posts/CL8RFFCdsBwWWfKYS/a-small-negative-result-on-debate).
*   Pay and benefits really are negotiable, and we're willing to match industry offers if there's a great fit. Don't let it stop you from applying.
# AI Alignment YouTube Playlists

I created two AI Alignment playlists on Youtube. One that is slide-heavy and the other is not. I separated them into two playlists for two reasons.

1.  It’s useful to separate for a dataset I am working on.
2.  Media is easier to consume when you don’t have to pay attention to the slides and pictures someone is describing.

Not slide-heavy (currently 216 videos): [https://youtube.com/playlist?list=PLTYHZYmxohXp0xvVJmMmpT_eFJovlzn0l](https://youtube.com/playlist?list=PLTYHZYmxohXp0xvVJmMmpT_eFJovlzn0l) 

Slide-heavy (currently 366 videos): [https://youtube.com/playlist?list=PLTYHZYmxohXpn5uf8JZ2OouB1PsDJAk-x](https://youtube.com/playlist?list=PLTYHZYmxohXpn5uf8JZ2OouB1PsDJAk-x)

If you would like to contribute and add more videos to the playlists or create new Alignment-relevant playlists, let me know!

If you like access to the audio and youtube auto-generated subs in .txt format, I have stored them here: [https://drive.google.com/drive/folders/1qVo4TyHKrsJvbJ3UrIOLW45j\_7\_wwnbZ?usp=sharing](https://drive.google.com/drive/folders/1qVo4TyHKrsJvbJ3UrIOLW45j_7_wwnbZ?usp=sharing) 

I've batched up the files into buckets of 90-ish hours (except for the final bucket which is less) since I plan on loading them into otter.ai and that website only accepts 100 hours per user (per month). Additionally, if you would like to help load some of the audio files in your own otter.ai account, please let me know! I want to create transcripts of the audio files and add them to a dataset very soon.
# LessWrong Now Has Dark Mode

One of the more commonly requested features on LessWrong is a dark mode. Well, now we have it! To activate, hover over the user-menu in the top right, open the Themes submenu, and click on Dark Mode there.

Notes:

*   This feature is beta, by which I mean there may be aesthetic problems, especially in less-used areas of the site. Please give feedback in the comments here. *Hopefully* there is no black-on-black text anywhere, but this is the sort of thing that could happen, so if you do come across any then please report it.
*   You need to be logged in to pick a theme, for UI reasons (the theme-select option is in the user menu because there wasn't a good place to put it). If you log in, switch to dark mode, and then log back out, it will remember your choice, though, so you don't have to stay logged in if you don't want.
*   Posts that use custom colors will mostly have their colors overridden to defaults, to prevent black-on-black text. We do have a way to provide dark-mode overrides for specific custom colors which we've used on a few specific posts. If there are important historical posts with black-on-black problems, or which use custom colors in an important way that didn't come through, please let me know in the comments here and I'll try to fix it.
*   For technical reasons, we don't inherit the operating system dark-mode preference. Let us know in the comments if this is important to you and we might prioritize it.
# Ceiling Fan Air Filter

[Filter cubes](https://en.wikipedia.org/wiki/Corsi%E2%80%93Rosenthal_Box) are a great way to cheaply filter a lot of air, but they're bulky and noisy. [Elevating](https://www.jefftk.com/p/elevated-air-purifier-cubes) them can get them out of the way if you have a high enough ceiling, but it's still not ideal. What if we built something around a device that is intended to be up there: a ceiling fan?

Let's say your blades are a foot from the ceiling, and sweep a diameter of 52". Shroud the fan with a regular octagon of 12"x24" furnace filters, and the air will flow in through the filters and down:

[![](https://www.jefftk.com/ceiling-fan-air-filter.jpg)](https://www.jefftk.com/ceiling-fan-air-filter-big.jpg)

Relative to box fans, ceiling fans move a lot more air for a given level of noise, because they are so much bigger. Since noise often makes people turn air filters down or off, a quiet high-volume filter that doesn't get in the way could be very useful.

You could make a simple one with duct tape, but it should be possible to make one that is reasonably attractive if you used a metal grille for the outside. This could also make changing the filters much easier, if the framework held the filters and allowed you to slide them into place. Putting lights on the inside and using the filter as a diffuser could also have a nice effect, like a [paper lantern](https://en.wikipedia.org/wiki/Paper_lantern).

*Comment via: [facebook](https://www.facebook.com/jefftk/posts/10100250386864552)*
# Stuff I might do if I had covid

In case anyone wants a rough and likely inaccurate guide to what I might do if I had covid to mitigate it, I looked into this a bit recently and wrote notes. It’s probably better than if one’s plan was to do less than a few hours of research, but is likely flawed all over the place and wasn’t written with public sharing in mind, and um, isn’t medical advice:

[![Screen_Shot_2022-05-07_at_1.35.17_PM](https://hosting.photobucket.com/images/i/katjasgrace/Screen_Shot_2022-05-07_at_1.35.17_PM.png?width=960&height=720&fit=bounds)](https://photobucket.com/u/katjasgrace/p/0dc9afd2-c217-4caa-8921-429331332c31)

[Here’s](https://docs.google.com/document/d/11cF4k9NUltfiEYv8yhsw2Ku8qRITQY5IqxJ71jFpVO4/edit?usp=sharing) a Google doc version, where any comments you leave might be seen by the next person looking (and you might see comments added by others).

[Here’s](https://docs.google.com/document/d/10awWMz3DyJi2OibnVlMg-2sipWfZ3jm8P_MyLUbK_ZE/edit?usp=sharing) a much longer doc with the reasoning, citations and more comments.

(I continue to guess that long covid is worth avoiding.)
# ProjectLawful.com: Eliezer's latest story, past 1M words

So if you read Harry Potter and the Methods of Rationality, and thought...

"You know, HPMOR is pretty good so far as it goes; but Harry is much too cautious and doesn't have nearly enough manic momentum, his rationality lectures aren't long enough, and all of his personal relationships are way way *way* too healthy."

...then have I got the story for you! [Planecrash aka Project Lawful aka Mad Investor Chaos and the Woman of Asmodeus](https://projectlawful.com), is a story in roleplay-format that I as "Iarwain" am cowriting with Lintamande, now past 1,000,000 words.

It's the story of Keltham, from the world of dath ilan; a place of high scientific achievement but rather innocent in some ways.  For mysterious reasons they've screened off their own past, and very few now know what their prescientific history was like.

Keltham dies in a plane crash and ends up in the country of Cheliax, whose god is "Asmodeus", whose alignment is "Lawful Evil" and whose people usually go to the afterlife of "Hell".

And so, like most dath ilani would, in that position, Keltham sets out to bring the industrial and scientific revolutions to his new planet!  Starting with Cheliax!

(Keltham's new friends may not have been entirely frank with him about exactly what Asmodeus wants, what Evil really is, or what sort of place Hell is.)

This is not a story for kids, even less so than HPMOR. There is romance, there is sex, there are deliberately bad kink practices whose explicit purpose is to get people to actually hurt somebody else so that they'll end up damned to Hell, and also there's math.

The starting point is Book 1, [Mad Investor Chaos and the Woman of Asmodeus](https://www.projectlawful.com/posts/4582). I suggest logging into ProjectLawful.com with Google, or creating an email login, in order to track where you are inside the story.

Please avoid story spoilers in the comments, especially ones without [spoiler protection](https://www.lesswrong.com/posts/2rWKkWuPrgTMpLRbp/lesswrong-faq#How_do_I_insert_spoiler_protections_); this is not meant as an "ask Eliezer things about MICWOA" thread.
# LessWrong Berlin Meetup - Street Epistemology

After a long pause of LessWrong meetups with a workshop part, we are back to doing exercises as part of our LessWrong meetups. For this meetup, I invited George Lanetz to present Street Epistomology. In his words:

"Your rationality is my business". We need to have conversations with people around us about important matters. Everyone knows debates. Lesswrongers also know Double Crux. But the most important topics are often clouded by ugh fields, and it's hard to think about them.

With SE we can help each other to practice noticing and exploring epistemology of our own and people around us, while maintaining the connection necessary to continue challenging discussions.  
This Sunday I'll read a brief into, offer a demonstration of the method and facilitate practice in groups of three.  
You can read more at the site  
[https://streetepistemology.com/](https://streetepistemology.com/)  
or watch some SE interviews  
[https://youtube.com/playlist?list=PLh10RgQgGuM-tnT7fKwgF4Dt57oh_yL5r](https://youtube.com/playlist?list=PLh10RgQgGuM-tnT7fKwgF4Dt57oh_yL5r)

Schedule:  
16:00 Arrive  
16:15 Start of the Street Epistemology workshop.  
17:45 Open socializing
# What's keeping concerned capabilities gain researchers from leaving the field?

My guess is that there are at least a few capabilities gain researchers who're concerned about the impact their work will have. My guess is that at least a few of these would like to leave, but haven't. 

My question is: where are these people? What's stopping them from leaving? And how can I help?

*   How much of it is finance? Capabilities gain pays well. How many researchers are trapped by their ~million dollar a year salary?
*   How much of it is just inertia? Many people think that if someone wanted to leave, they already would have. [But trivial costs are not trivial.](https://www.lesswrong.com/posts/reitXJgJXFzKpdKyd/beware-trivial-inconveniences) People delay leaving a job all the time. Some of them are quite likely working in capabilities gain research.
*   How much of it is just. Uncertainty about what else to do. Concerns about whether to it's better to leave, or to try and steward the ship from the inside. Many, many other things. How many of these other myriad concerns can be resolved? 

My suspicion is that a lot of work can be done to simply help people leave the field. In principle: a low friction scholarship can be presented on a silver platter to those wanting to transition away. Potentially useful information, and personalised support can be provided to help people interested in leaving. ([AI safety support](https://www.aisafetysupport.org/) is trying to work with those interested in entering AI safety; something like that, but for leaving capabilities gain.) A simple conversation can be started, which may prompt people to address that voice of concern in the corner of their mind more head on. 

There is a lot of uncertainty about what low hanging fruit exist in simply helping researchers leave. Whether these fruit exist at all. How many. Are they poisonous. 

The first step in resolving this uncertainty is to just look for some concerned researchers, and talk to them. 

So, dear concerned capabilities gain researcher. 

What's keeping you from leaving? 

(DM me if you wish for privacy)
# How would public media outlets need to be governed to cover all political views?

As one of their questions Reporters without Borders [asks](https://rsf.org/sites/default/files/medias/file/2022/04/RSF_Index_Questionnaire_2022_EN.pdf) for their World Press Freedom Index among others:

> Do public media outlets cover all political views?*
> 
> Does the law provide mechanisms to guarantee pluralism and editorial independence?*
> 
> Do public media outlets ever ignore sensitive information regarding the government or  
> administration that is covered by private media?*
> 
> Is the pluralism of opinions of people in the country reflected in the media?*

Part of the case of the EU against Hungary is that its press is largely government-controlled or controlled by supporters of the government. Voices critical of the government have a lower share of the public attention. Philanthropically funded journalism that intends to provide critical media gets attacked as being funded by Soros and intended to manipulate the Hungarian people. 

COVID-19 showed that there are similar dynamics in the United States and other European states where voices that are critical of the regime have a hard time being published . The fighting critical content for being Russian disinformation and Hungarian strategy of fighting critical voices for being influenced by Soros follows similar dynamics where outside influence is overblown and the narrative allows for acting against critical voices.

While some national governments have state media, the EU currently doesn't have its own media outlet. Given the EU perspective of the problems in Hungary, funding critical journalism would be a good intervention. If the EU would start its own media, there's the question of media governance. How could EU-funded public media be governed so that it will represent voices from the full pluralism of opinions of people?
# "A Generalist Agent": New DeepMind Publication

Linkpost for ["A Generalist Agent"](https://www.deepmind.com/publications/a-generalist-agent)

Abstract:

"Inspired by progress in large-scale language modeling, we apply a similar approach towards building a
single generalist agent beyond the realm of text outputs. The agent, which we refer to as Gato, works as a
multi-modal, multi-task, multi-embodiment generalist policy. The same network with the same weights
can play Atari, caption images, chat, stack blocks with a real robot arm and much more, deciding based
on its context whether to output text, joint torques, button presses, or other tokens. In this report we
describe the model and the data, and document the current capabilities of Gato"
# Deepmind's Gato: Generalist Agent

[From the abstract](https://www.deepmind.com/publications/a-generalist-agent), emphasis mine:

> The agent, which we refer to as Gato, works as a multi-modal, multi-task, multi-embodiment generalist policy. **The same network with the same weights can play Atari, caption images, chat, stackblocks with a real robot arm and much more, deciding based on its context whether to output text, joint torques, button presses, or other tokens.**

(Will edit to add more as I read. ETA: [**1a3orn**](https://www.lesswrong.com/users/1a3orn) [posted first](https://www.lesswrong.com/posts/5onEtjNEhqcfX3LXG/a-generalist-agent-new-deepmind-publication).)

1.  It's only 1.2 billion parameters. (!!!) They say this was to avoid latency in the robot control task.
2.  It was trained offline, purely supervised, but could in principle be trained online, with RL, etc
3.  Performance results:

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/3f01e18b66b42a0963a09d48474a4f62abe40aae04287d8c.png)

The section on broader implications is interesting. Selected quote:

> In addition, generalist agents can take actions in the the physical world; posing new challenges that may require novel mitigation strategies. For example, physical embodiment could lead to users anthropomorphizing the agent, leading to misplaced trust in the case of a malfunctioning system, or be exploitable by bad actors. Additionally, while cross-domain knowledge transfer is often a goal in ML research, it could create unexpected and undesired outcomes if certain behaviors (e.g. arcade game fighting) are transferred to the wrong context. The ethics and safety considerations of knowledge transfer may require substantial new research as generalist systems advance. Technical AGI safety (Bostrom, 2017) may also become more challenging when considering generalist agents that operate in many embodiments. For this reason, preference learning, uncertainty modeling and value alignment (Russell, 2019) are especially important for the design of human-compatible generalist agents. It may be possible to extend some of the value alignment approaches for language (Kenton et al., 2021; Ouyang et al., 2022) to generalist agents. However, even as technical solutions are developed for value alignment, generalist systems could still have negative societal impacts even with the intervention of well-intentioned designers, due to unforeseen circumstances or limited oversight (Amodei et al., 2016). This limitation underscores the need for a careful design and a deployment process that incorporates multiple disciplines and viewpoints. 

They also do some scaling analysis and yup, you can make it smarter by making it bigger.

What do I think about all this?

Eh, I guess it was already priced in. I think me + most people in the AI safety community would have predicted this. I'm a bit surprised that it works as well as it does for only 1.2B parameters though.
# What to do when starting a business in an imminent-AGI world?

As reported by [la3lorn](https://www.lesswrong.com/posts/5onEtjNEhqcfX3LXG/a-generalist-agent-new-deepmind-publication) and [Daniel Kokotajlo](https://www.lesswrong.com/posts/xxvKhjpcTAJwvtbWM/deepmind-s-gato-generalist-agent), Gato is here and appears to me to represent a sub-human AGI, or near enough as makes no difference in a timeline sense. I think this probably means a general thickening of deep learning applications everywhere, and the introduction of a kind of "stack" AI that can do things we used to need whole organizations to do - as an example, I mean things like do patent research, label patent diagrams, and file patent lawsuits.

I also have an idea about a business I would like to start. This is already a notoriously trying task with low probability of success, and I wonder how much more so it will be in a world that will populate with AGI patent trolls, along with whatever else, well before hitting any kind of clear success mark.

**So my question is**: what do we do to account for powerful AI, showing up soon, when we are starting a business?

Note that what I am interested in here is *non-AI* businesses in particular and non-software businesses in general, because this looks like the threshold for it spilling across a bunch of new domains.
# Can moderators fix old sequences posts?

I'm re-reading sequences now and I'm noticing eye-opening things like so many have SEQ RERUN, useless copies that clutter up the link space while having very few comments that they seem to be intended for.
Can the moderators do something about this? Remove links to them, maybe even delete them by moving the comments to the original posts.
(I don't know if there is functionality to do this, and also how morally acceptable it would be, and if anyone but me is worried about these problems, maybe no one else needs to "fix" this?)
And also I notice,  that in the old entries, the answers are not child comments, which creates a terrible confusion when reading by karma, when it is not clear what the question was and where to find it.
(Same requests and questions here)
# [Link post] Promising Paths to Alignment - Connor Leahy | Talk

**Date of recording: **Tuesday May 10, 2022

**Description**  
[AI Safety Support](https://www.aisafetysupport.org/home) recently hosted a closed, introductory talk by Connor Leahy on Promising Paths to Alignment. 

The talk covers: (i) why Alignment is such a difficult problem, (ii) current approaches to solving it, and (iii) some info on Connor’s new Alignment Research startup, [Conjecture](https://www.lesswrong.com/posts/jfq2BH5kfQqu2vYv3/we-are-conjecture-a-new-alignment-research-startup).

I highly recommend this talk for developing a better understanding of the technical Alignment research landscape - particularly for those considering or pursuing a related career.
# The AI Countdown Clock

I made [this clock](https://aicountdown.com), counting down the time left until we build AGI:

![](https://substackcdn.com/image/fetch/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fbucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com%2Fpublic%2Fimages%2Faf4ca51c-0afa-41c8-998a-4678f8ab7754_2880x1592.png)

It uses the most famous [Metaculus prediction](https://www.metaculus.com/questions/3479/date-weakly-general-ai-system-is-devised/) on the topic, inspired by several recent dives in the expected date. Updates are automatic, so it reflects the constant fluctuations in collective opinion.

Currently, it’s sitting in 2028, i.e. the end of the next presidential term. The year of the LA Olympics. Not so far away.

There were a few motivations behind this project:

1.  **Civilizational preparedness.** Many people are working on making sure this transition is a good one. Many more probably should be. I don’t want to be alarmist, but the less abstract we can make the question, the better. In this regard, it’s similar to the [Doomsday Clock](https://en.wikipedia.org/wiki/Doomsday_Clock).
2.  **Personal logistics.** I frequently find myself making decisions about long-term projects that would be deeply affected by the advent of AGI. Having kids, for example.   
      
    The prediction is obviously far from absolute, and I’m not about to stop saving more than 5 years and 11 months of living expenses. But it’s good to be reminded that the status quo is no longer the best model for the future.
3.  **Savoring the remainder.** Most likely, AGI will be the beginning of the end for humanity. Not to say that we will necessarily go extinct, but we will almost definitely stop being “human,” in the recognizable/traditional sense.  
      
    For many years, I’ve used the [Last Sunday](https://chrome.google.com/webstore/detail/the-last-sunday-reminder/aiojhapcgfgmiacbbjfgedhlcchmpelh?hl=en) as my new tab page. It shows you how many Sundays you have left in your life, if you live to an average age. I’ve gotten some strange looks, when it accidentally pops up during a presentation. I know it seems morbid, like a fixation on the end. But I don’t see it that way; it’s not about the end, but the finitude of the middle. That precious scarcity.  
      
    I’ve spent a lot of time thinking about the meaning of being human, but this mostly dissolves that angst. It’s like: if I live in San Francisco, my daily decisions about what to do here are impossibly broad. But if I’m a tourist, visiting for a few days, then my priority is to do all of the most “San Francisco” things I can: see the Golden Gate, eat a burrito in the Mission, sit among purple-red succulents on a foggy beach cliff.  
      
    So, if I only have X years left of being human, I should focus on doing the most human things I can. Whatever that means to me. Conveniently, this applies to the worlds with and without AGI, since in the latter, I’ll still die. But the shorter timeline makes it more real.

Let me know if you have any feedback or suggestions.
# Definition Practice: Applied Rationality

In the exercise of defining what we mean by a term, we explore our assumptions about the nature of the term and what it’s pointing at. Sometimes this leads us to get a better handle on a concept. Other times, it shows us our confusion.

As an example of the value of definitions, [YCombinator considers](https://www.ycombinator.com/library/4b-how-to-pitch-your-company) it important for founders to be able to clearly and concisely say a company's purpose, and that the ability to do so is indicative of plausible success..

Here are two exercises:

1.  Write a paragraph to define the term *applied rationality.*
2.  Write a sentence to define the term *applied rationality.*

I encourage you to write your definitions before reading what others wrote. While I suggest that you first attempt to define the term in a paragraph and then move to the sentence variation, you can also change the order if that seems easier.

Please use the answer feature for your answer to these exercises, and the comment function for other needs.
# Why I'm Optimistic About Near-Term AI Risk

I'm not worried about AI posing an existential risk in the next 10-20 years. Recent developments in AI capabilities actually make me feel *more* optimistic about this. The fact that relatively simple models can perform a wide array of tasks suggests that we can build [satisfactory AI](https://www.lesswrong.com/posts/BEWdwySAgKgsyBzbC/satisf-ai-a-route-to-reducing-risks-from-ai) without the need to use sophisticated, potentially dangerous agents in the near-term.

My expectation for how AI will develop over the next decade is that companies will continue to focus on transformer-based foundation models. The general capability of these models will increase for a while simply by using more data, improving training procedures, and leveraging specialized hardware. Eventually, companies will start hitting bottlenecks in the amount of data required for optimal training at a given capability level. But before that, deployment of these systems will favor smaller, faster, and more auditable models leading companies to focus on distilled models specializing in specific tasks. 

These specialized models will be oriented towards augmenting human productivity, producing entertainment, or automating specific tasks. The slow pace at which industries change their practices and utilize the benefits of a new technology will moderate the adoption of AI. As adoption increases, these [AI services](https://www.lesswrong.com/tag/ai-services-cais) will gain autonomy, producing more value at lower cost. Continued specialization will result in mostly autonomous AI's derived from generally capable foundation models that are distilled down for variety of tasks.

I'm not claiming that these [Tool AI's](https://www.gwern.net/Tool-AI) won't eventually be dangerous, but I can't see this path leading to high existential risk in the next decade or so.
 
I think most people in the AI safety field would agree with me on this, so why write it up?

I want to make this point explicit and foster a discussion about near-term AI safety. If AI will become dangerous soon, the field needs to act very quickly. Researchers would have to consider eschewing movement building, trading goodwill for influence, and gambling on near-term approaches. People who would take more than a decade to have an impact would have less reason to join in the first place while investments in infrastructure in the field would become less valuable. 

It's important for those concerned to be on the same page about near-term risks in order to avoid the unilaterialist's curse. Recent, pessimistic takes about AI risk make it seem superficially as if the consensus has shifted, but I don't think this is representative of the field as a whole. I remain optimistic that innovations in foundation models will produce a lot of value without a large increase in risk, providing more time to build.
# To what extent is your AGI timeline bimodal or otherwise "bumpy"?

For example, you might think:

> It's likely that AGI will be invented before 2050; however, if it isn't, then that must mean either that AGI is impossible, or that it requires much more advanced technology than I currently think it does, or else that there was some kind of large-scale civilizational collapse in the meantime.

For that matter, any non-[exponential](https://en.wikipedia.org/wiki/Exponential_distribution) distribution has this property, where the non-occurrence of the event by a certain time will change your expectation of it going forward. I'm curious if people think this is the case for AGI, and if so, why. (Also curious if this question has been asked before.)
# [Link] New Stanford Encyclopedia of Philosophy article, "Normative Theories of Rational Choice: Rivals to Expected Utility"

[https://plato.stanford.edu/entries/rationality-normative-nonutility/](https://plato.stanford.edu/entries/rationality-normative-nonutility/)

I thought this looked like it may be of interest to people here.
# Proposal: Twitter dislike button

The popular story of Twitter’s role in the ruin of civilization is that it is a runaway trash fire of reciprocal anger and offense, where otherwise nice people are possessed by overwhelming outrages, and drawn into throwing their own energy behind creating the vilest and most vindictive responses to what they see, turning away from reason and hurting others in turn, and so the place continues.

I’m not sure how much of Twitter activity this accounts for (apparently [Michael Nielsen enjoys an entirely different place](https://twitter.com/michael_nielsen/status/975884635535101952), and my experience seems pretty nice too). But I think there’s a real pattern of this kind, which makes game theoretic sense, and goes something like this:

1.  People say things
2.  People read these things
3.  If anything seems objectionable to any of these people, then they repost those things with commentary, and everyone else reads them extra.
4.  In the next round, people (or the ones who who get attention) say objectionable things (that they expect will get attention), about objectionable things (that they have their attention on from the last round)
5.  etc.

To lay out the effects of all this more clearly:

1.  People disproportionately read things they don’t like, which is presumably bad for them
2.  People get the visceral sense that others are disproportionately writing things they don’t like, which is misleading, and not in a helpful-for-public-friendship way
3.  Things people don’t like get extra space in the public conversation
4.  People who tend to write things that others don’t like get extra power and attention instead of less
5.  Writing things other people don’t like is incentivized (if you want attention, writing things other people don’t like is probably somewhat better than writing things people do like, and way better than writing things they don’t feel strongly about).

Supposing something like this model is true, and bad, it seems to me that there is a really simple solution: **add a dislike button.**

That is, what if when a person sees a thing they don’t like, instead of broadcasting it to others, they register their disapproval by quietly clicking a different button next to the heart, and then Twitter shows it to other people less instead of more? You can still retweet it if you especially want other people to see it more, but adding attention wouldn’t be the default disapproval vote.

This is not an original idea, and the other major websites that do it have not, to my knowledge, been run out of business by a dearth of disagreement. I think they are also not so much known for the above dynamic.

I posit that a Twitter downvote button would be great. What am I missing?
# On saving one's world

If the world is likeliest to be saved by sober scholarship, then let us be sober scholars in the face of danger.

If the world is likeliest to be saved by playful intellectual exploration, then let us be playful in the face of danger.

Strategic, certainly; aware of our situation, of course; but let us not throw away the one mental mode that can actually save us, if that's in fact our situation.

If the world is likeliest to be saved by honest, trustworthy, and high-integrity groups, who by virtue of their trustworthiness can much more effectively collaborate and much more quickly share updates; then let us be trustworthy. What is the path to good outcomes otherwise?

CFAR has a notion of "flailing". Alone on a desert island, if you injure yourself, you're likelier to think fast about how to solve the problem. Whereas injuring yourself around friends, you're more likely to "flail": lean into things that demonstrate your pain/trouble to others.

To my eye, a lot of proposals that we set aside sober scholarship, or playful intellectual exploration, or ethical integrity, look like flailing. I don't see an argument that this setting-aside actually chains forward into good outcomes; it seems performative to me, like hoping that if our reaction "feels extreme" enough, some authority somewhere will take notice and come to the rescue.

Who is that authority?

If you have a coherent model of this, we can talk about it and figure out if that's really the best strategy for eliciting their aid.

But if no one comes to mind, consider the possibility that you're executing a social instinct that's adaptive to threats like tigers and broken legs, but maladaptive to threats like Unfriendly AI.

If you feel scared about something, I generally think it's good to be honest about that fact and discuss it soberly, rather than hiding it. I don't think this is incompatible with rigorous scholarship or intellectual play.

But I would clearly distinguish "being honest about your world-models and feelings, because honesty is legitimately a good idea" from "making it your main strategy to do whatever action sequence feels emotionally resonant with the problem".

An "extreme" key doesn't necessarily open an "extreme" lock. A dire-sounding key doesn't necessarily open a dire-feeling lock. A fearful or angry key doesn't necessarily open a lock that makes you want to express fear or anger.

Rather, the lock's exact physical properties determine which exact key (or set of keys) opens it, and we need to investigate the physical world in order to find the right key.
# Workday Air Quality Measurements

I continue to be very excited about my new [M2000 meter](https://temtopus.com/collections/temtop-co2-monitor/products/temtop-m2000-2nd-generation-air-quality-monitor-for-pm2-5-pm10-particles-co2-hcho-temperature-humidity-settable-audio-alarm-data-export-recording-curve-easy-calibration), and I brought it with me today to measure air quality during my commute and workday. Here's what I found:

[![](https://www.jefftk.com/2022-05-17--commute-and-workday.png)](https://www.jefftk.com/2022-05-17--commute-and-workday-big.png)  
*[full-size image](https://www.jefftk.com/2022-05-17--commute-and-workday-big.png)*

| Time | Activity | CO2 (ppm) | pm2.5 ug/m3 |
| --- | --- | --- | --- |
| 8:00 | Left house | 426 | 2.9 |
| 8:15 | Davis station | 468 | 69.9 |
| 8:24 | Red line, new car | 802 | 29.3 |
| 8:36 | Kendall station, outside | 477 | 11  |
| 8:40 | Hallways, elevators, breakfast in cafe | 563 | 2.8 |
| 9:18 | Office, door open | 475 | 1   |
| 10:00 | Closed door | 514 | 1   |
| 10:31 | Open door | 553 | 1   |
| 11:02 | Team meeting, 14 people, ~15x30 | 1052 | 1   |
| 12:04 | Hallways, elevator | 686 | 1.6 |
| 12:10 | Lunch in cafe | 943 | 2.6 |
| 12:50 | Hallways, elevator | 763 | 2.1 |
| 12:57 | Office, door open | 537 | 1   |
| 2:02 | Closed door | 539 | 1   |
| 2:31 | Two people | 599 | 1   |
| 2:54 | One person, open door | 587 | 1   |
| 3:01 | Closed door, two people | 598 | 1   |
| 3:31 | Open door, zero people | 538 | 1   |
| 4:04 | Closed door, two people | 591 | 1   |
| 4:38 | Kendall station | 487 | 36.6 |
| 4:44 | Red line, old car | 953 | 21.6 |
| 4:56 | Davis station | 537 | 134.5 |
| 4:58 | Outside | 445 | 2.25 |

I spent most of my workday in my office, with the door open. I closed the door for meetings, either virtual or in person. The "zero people" at 3:31 is me going on a walking 1:1: with a report and leaving my meter behind at my desk.

Currently I wear a mask on the subway (and stations) and at work when I'm not in my office or a conference room with a small number of people. Based on these readings, it would be safe from a covid perspective to remove my mask in the subway station, but given the high level of particulate pollution I might as well leave it on.

(At this point I'm not wearing a mask because I think it would be directly harmful to myself or my family to get covid, but instead because the steps we would need to take to reduce the risk of infecting others once we knew we were infected would be very inconvenient.)
# How to get into AI safety research

Recently, I had a conversation with someone from a math background, asking how they could get into AI safety research. Based on my own path from mathematics to AI alignment, I recommended the following sources. It may prove useful to others contemplating a similar change in career:

* [Superintelligence](https://www.amazon.com/Superintelligence-Dangers-Strategies-Nick-Bostrom/dp/1501227742) by Nick Bostrom. It condenses all the main arguments for the power and the risk of AI, and gives a framework in which to think of the challenges and possibilities.
* Sutton and Barto's Book: [Reinforcement Learning: An Introduction](https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf). This gives the very basics of what ML researchers actually do all day, and is important for understanding more advanced concepts. It gives (most of) the vocabulary to understand what ML and AI papers are talking about.
* [Gödel without too many tears](https://www.logicmatters.net/resources/pdfs/GWT3_LM.pdf). This is how I managed to really grok logic and the completeness/incompleteness theorems. Important for understanding many of MIRI's and LessWrong's approaches to AI and decision theory.
* [Safely Interruptible agents](https://www.deepmind.com/publications/safely-interruptible-agents). It feels bad to recommend one of my own papers, but I think this is an excellent example of bouncing between ML concepts and alignment concepts to make some traditional systems interruptible (so that we can shut them down without them resisting the shutdown).
* [Alignment for Advanced Machine Learning Systems](https://intelligence.org/files/AlignmentMachineLearning.pdf). Helps give an overall perspective on different alignment methods, and some understanding of MIRI's view on the subject (for a deeper understanding, I recommend diving into MIRI's or Eliezer's publications/writings).

You mileage may vary, but these are the sources that I would recommend. And I encourage you to post any sources you'd recommend, in the comments.
# What's up with the recent monkeypox cases?

Is it a thing I should be allocating attention to?

Wikipedia tells me that human-to-human transmission of monkeypox is pretty rare/difficult. There has been some community spread recently at least in the UK, but that's been speculated to have been from sex. On the other hand, there's been a bunch of cases reported in Portugal and Spain recently as well. Is that just normal background rate being given more attention than usual by the news cycle? Or is this actually an unusually high number of cases? Most importantly, is the number of cases significant evidence of increased human-to-human transmission?
# How has the total amount of gain-of-function research worldwide grown/shrunk over time?

Given the advance in underlying biological tools/technology, I would not find it surprising if gain-of-function research had increased 10x over the last decade. (This is just based on priors; getting an actual estimate is the point of this question.)

Why is this interesting? Assuming that both original covid and at least one strain were lab leaks, the rate of growth of gain-of-function research has a huge impact on how often we should expect future pandemics from lab leaks. If the amount of gain-of-function research has been roughly constant over the last 20 years, then the two covid leaks suggest a base rate of something like one or two pandemic-level leaks every 20 years, and covid was probably unusually hard to contain in a lab for some reason on top of that. On the other hand, if the field has grown 10x in the past 5 years, that suggests a base rate more like a leak every 2-3 years (and probably getting faster over time as the field continues to grow). If that's the case, then the base rate for pandemics will be way higher going forward than in the past.
# Podcast: Spencer Greenberg talks to me about dealing with our groupstruckness and boundedness

I [talked to Spencer Greenberg](https://clearerthinkingpodcast.com/episode/105) a little while ago for his podcast [Clearer Thinking](https://clearerthinkingpodcast.com/). The recording is out today. According to his website we discussed:

> What does it mean to be “groupstruck”? How does groupstruck-ness differ from the bystander effect, normalcy bias, and other related cognitive biases? How do we break people out of being groupstruck? What does it mean to be a “bounded” person? How can we build up better decision-making heuristics? What sorts of decisions do people usually not quantify but should (and vice versa)? How can we make rational relationship decisions without coming across as “calculating” or cold? How does anthropic reasoning affect our hypotheses about the nature of the universe and life within it (i.e., the Fermi paradox, the simulation hypothesis, etc.)?

You can listen or read the transcript [here](https://clearerthinkingpodcast.com/episode/105).
# Why does gradient descent always work on neural networks?

My amateur understanding of neural networks is that they almost always train using stochastic gradient descent. The quality of a neural network comes from its size, shape, and training data, but not from the training function, which is always simple gradient descent.

This is a bit unintuitive to me because gradient descent can only find the minimum of a function if that function is convex, and I wouldn't expect typical ML problems (e.g., "find the dog in this picture" or "continue this writing prompt") to have convex cost functions. So why does gradient descent always work?

One explanation I can think of: it *doesn't* work if your goal is to find the optimal answer, but we hardly ever want to know the optimal answer, we just want to know a good-enough answer. For example, if a NN is trained to play Go, it doesn't have to find the best move, it just has to find a winning move. Not sure if this explanation makes sense though.
# Who is available for contract work? (A la the Hacker News "Who Wants To Be Hired" threads)

The intention of this thread is similar to the "Hacker News Freelancer/Seeking Freelancer" threads.

Please lead with either AVAILABLE FOR PROJECTS or SEEKING FREELANCER, your location, and whether remote work is a possibility.

I was inspired to post this when a friend asked me about which rationalist adjacent people were open to contract work, though other than one researcher that I'd seen mention their rate on their website, I didn't know who to mention. Though I'm sure there are quite a few who are! So, perhaps this will solve a coordination problem.
# [Short version] Information Loss --> Basin flatness

This is an overview for advanced readers.  Main post: [Information Loss --> Basin flatness](https://www.lesswrong.com/posts/wPudaEemohdYPmsye/information-loss-greater-than-basin-flatness)

**Summary:**

Inductive bias is related to, among other things:

*   Basin flatness
*   Which solution manifolds (manifolds of zero loss) are higher dimensional than others.  This is closely related to "basin flatness", since each dimension of the manifold is a direction of zero curvature.

In relation to basin flatness and manifold dimension:

1.  It is useful to consider the **"behavioral gradients"** $\nabla_\theta f(\theta,x_i)$ for each input. 
2.  Let $G$ be the matrix of behavioral gradients.  (The $i^{th}$ column of $G$ is $g_i=\nabla_\theta f(\theta,x_i)$).[^kwxi3hytsx]  We can show that $dim(manifold)\leq N-Rank(G)$.[^6pvsd1m9v8x]
3.  $Rank(Hessian)=Rank(G)$.[^he171k2fwv8][^awnozwzw1ed]
4.  $$Flat basin  $\approx$  Low-rank Hessian  $=$  Low-rank $G$  $\approx$  High manifold dimension
5.  High manifold dimension  $\approx$  Low-rank $G$  $=$  Linear dependence of behavioral gradients 
6.  A case study in a very small neural network shows that "information loss" is a good qualitative interpretation of this linear dependence.
7.  Models that throw away enough information about the input in early layers are guaranteed to live on particularly high-dimensional manifolds.  Precise bounds seem easily derivable and might be given in a future post.

See the [main post](https://www.lesswrong.com/posts/wPudaEemohdYPmsye/information-loss-greater-than-basin-flatness) for details.

[^kwxi3hytsx]: In standard terminology, \(G\) is the Jacobian of the concatenation of all outputs, w.r.t. the parameters. 

[^6pvsd1m9v8x]: \(N\) is the number of parameters in the model.  See claims 1 and 2 here for a proof sketch. 

[^he171k2fwv8]: Proof sketch for \(Rank(Hessian)=Rank(G)\): \(span(g_1,..,g_k)^\perp\) is the set of directions in which the output is not first-order sensitive to parameter change.  Its dimensionality is \(N-rank(G)\).At a local minimum, first-order sensitivity of behavior translates to second-order sensitivity of loss.So \(span(g_1,..,g_k)^\perp\) is the null space of the Hessian.So \(rank(Hessian)=N-(N-rank(G))=rank(G)\) 

[^awnozwzw1ed]: There is an alternate proof going through the result \(Hessian = 2GG^T\).  (The constant 2 depends on MSE loss.)