# An epistemic advantage of working as a moderate

\[epistemic status: the points I make are IMO real and important, but there are also various counterpoints; I'm not settled on an overall opinion here, and the categories I draw are probably kind of dumb/misleading\]

Many people who are concerned about existential risk from AI spend their time advocating for radical changes to how AI is handled. Most notably, they advocate for costly restrictions on how AI is developed now and in the future, e.g. the Pause AI people or the MIRI people. In contrast, I spend most of my time thinking about relatively cheap interventions that AI companies could implement to reduce risk assuming a low budget, and about how to cause AI companies to marginally increase that budget. I'll use the words "radicals" and "moderates" to refer to these two clusters of people/strategies. In this post, I’ll discuss the effect of being a radical or a moderate on your epistemics.

I don’t necessarily disagree with radicals, and most of the disagreement is unrelated to the topic of this post; see footnote for more on this.[^4ibsdj7juce]

I often hear people claim that being a radical is better for your epistemics than being a moderate: in particular, I often hear people say that moderates end up too friendly to AI companies due to working with people from AI companies. I agree, but I think that working as a moderate has a huge advantage for your epistemics.

The main advantage for epistemics of working as a moderate is that almost all of your work has an informed, intelligent, thoughtful audience. I spend lots of time talking to and aiming to persuade AI company staff who are generally very intelligent, knowledgeable about AI, and intimately familiar with the goings-on at AI companies. In contrast, as a radical, almost all of your audience—policymakers, elites, the general public—is poorly informed, only able or willing to engage shallowly, and needs to have their attention grabbed intentionally. The former situation is obviously way more conducive to maintaining good epistemics.

I think working as a moderate has a bunch of good effects on me:

*   I'm extremely strongly incentivized to know what's up. If I try to bullshit about AI, the people I'm talking to will notice that I'm bullshitting and judge me harshly.
*   I'm strongly incentivized to make arguments that I can justify. My interlocutors know enough about what's going on that they can poke at particular parts of my argument and judge me harshly if the argument is flimsy. And if they're not persuaded by an argument, I can go off and try to find arguments or evidence that will persuade them.
*   I don't need to spend as much time optimizing for virality: my audience is mostly already willing to hear from me.
*   I don't need to engage in coalitional politics where I make common cause with activists who are allied with me for some contingent reason.
*   Most of the spicy things I say can be said privately to just the people who need to hear them, freeing me from thinking about the implications of random third parties reading my writing.
*   I genuinely expect to change my mind as a result of conversations I have about my work. The people I talk to often have something to teach me.
*   I am not incentivized to exude confidence or other emotional affect. I don't have to worry that if I caveat my arguments appropriately they won't be as persuasive.
*   Because I'm trying to make changes on the margin, details of the current situation are much more interesting to me. In contrast, radicals don't really care about e.g. the different ways that corporate politics affects AI safety interventions at different AI companies.
*   I have specific asks and can see how people respond to them. Radicals don't really get to see whether people take specific actions based on their advocacy. I think this leads them to have greater risk of getting bullshitted by people who claim to be aligned with them but actually aren't. (Though the moderates have also had substantial issues with this in the past.)
    *   Radicals often seem to think of AI companies as faceless bogeymen thoughtlessly lumbering towards the destruction of the world. In contrast, I think of AI companies as complicated machines full of intelligent people, many of whom are well-meaning, that are thoughtlessly lumbering towards the destruction of the world due to some combination of ignorance, greed, and contemptible personal ambition. I think that the frustration and anger that I feel as a result of my work is more thoroughly textured than the frustration and anger that radicals feel.

Many people I know who work on radical AI advocacy spend almost all their time thinking about what is persuasive and attention-grabbing for an uninformed audience. They don't experience nearly as much pressure on a day-to-day basis to be well informed about AI, to understand the fine points of their arguments, or to be calibrated and careful in their statements. They update way less on the situation from their day-to-day work than I do. They spend their time as big fish in a small pond.

I think this effect is pretty big. People who work on radical policy change often seem to me to be disconnected from reality and sloppy with their thinking; to engage as soldiers for their side of an argument, enthusiastically repeating their slogans. I think it's pretty bizarre that despite the fact that LessWrongers are usually acutely aware of the epistemic downsides of being an activist, they seem to have paid relatively little attention to this in their recent transition to activism. Given that radical activism both seems very promising and is popular among LessWrongers regardless of what I think about it, I hope we try to understand the risks (perhaps by thinking about historical analogies) of activism and think proactively and with humility about how to mitigate them. 

I'll note again that the epistemic advantages of working as a moderate aren’t in themselves strong reasons to believe that moderates are right about their overall strategy.

I will also note that I work as a moderate from outside AI companies; I believe that working inside AI companies carries substantial risks for your epistemics. But IMO the risks from working at a company are worth conceptually distinguishing from the risks resulting from working towards companies adopting marginal changes.

In this post, I mostly conflated "being a moderate" with "working with people at AI companies". You could in principle be a moderate and work to impose extremely moderate regulations, or push for minor changes to the behavior of governments. I did this conflation mostly because I think that for small and inexpensive actions, you're usually better off trying to make them happen by talking to companies or other actors directly (e.g. starting a non-profit to do the project) rather than trying to persuade uninformed people to make them happen. And cases where you push for minor changes to the behavior of governments have many of the advantages I described here: you're doing work that substantially involves understanding a topic (e.g. the inner workings of the USG) that your interlocutors also understand well, and you spend a lot of your time responding to well-informed objections about the costs and benefits of some intervention.

Thanks to Daniel Filan for helpful comments.

[^4ibsdj7juce]:  Some of our difference in strategy is just specialization: I’m excited for many radical projects, some of my best friends work on them, and I can imagine myself working on them in the future. I work as a moderate mostly because I (and Redwood) have comparative advantage for it: I like thinking in detail about countermeasures, and engaging in detailed arguments with well-informed but skeptical audiences about threat models.And most of the rest of the difference in strategy between me and radicals is downstream of genuine object-level disagreement about AI risks and how promising different interventions are. If you think that all the interventions I'm excited for are useless, then obviously you shouldn't spend your time advocating for them.