# Demon Threads

_tldr: a Demon Thread is a discussion where everything is subtly warping towards aggression and confusion (i.e. as if people are under demonic influence), even if people are well intentioned and on the same 'side.' You can see a demon thread coming in advance, but it's still hard to do anything about._

_("Flame Wars" are similar but I felt the connotation was more like "everything has_ already _gone to hell, and people aren't even pretending to be on the same side")_

* * *

I kept wanting to reference this post when discussing internet discussion policy, and I kept forgetting that nobody has written it yet. So here it is.

Suggested Background Reading:

*   [Politics is Hard Mode](https://www.lesserwrong.com/posts/jxfu7CTc3NidinuXD/politics-is-hard-mode) (Rob Bensinger)
*   [Civility is Never Neutral](https://www.lesserwrong.com/posts/RhEEYwKMbK7rSAu69/civility-is-never-neutral) (Ozy)
*   [Writing that Provokes Comments](https://www.lesserwrong.com/posts/GHBLFPDhzeSQHx2eM/writing-that-provokes-comments) (me)
*   [Musings on Double Crux and Productive Disagreement](https://www.lesserwrong.com/posts/T3nb24aZf7d5S2Lnm/musings-on-double-crux-and-productive-disagreement) (me)

If someone in the future linked you to this post, it's probably because a giant sprawling mess of angry, confused comments is happening - or is about to happen - and it's going to waste a lot of time, make people upset, and probably _less_ likely to listen to each other about whatever the conversation ostensibly is about.

I have some ideas on what to do _instead_, which I discuss in [this followup post.](https://www.lesserwrong.com/posts/LxrpCKQPbdpSsitBy/short-circuiting-demon-threads-working-example)

But for now, this post is meant to open a discussion, explore the mechanics of how demon threads work, and then in the comments brainstorm solutions about how to handle them.

Wrong On the Internet
=====================

I find “Someone Is Wrong On the Internet” to be a weird, specific feeling.

It's distinct from someone being _factually wrong_ \- people can be wrong, point it out, and hash out their disagreements without a problem. But a common pattern I’ve witnessed (and experienced) is to notice someone being wrong in a way that _feels distinctly bad_, like if you don’t correct them, something precious will get trampled over.

This is when people seem most prone to jump into the comments, and it’s when I think people _should be most careful_.

Sometimes there actually _is_ an important thing at stake.

There usually isn’t.

It often _feels_ like there is, because our social intuitions were honed for tribes of a hundred or two, instead of a world of 7 billion. We live in a different world now. If you actually want to have an impact on society, yelling at each other on the internet is almost certainly not the best way to do so.

When there _actually is_ something important at stake, I think there are usually better plans than “get into a giant internet argument.” Think about what your goals are. Devise a plan that actually seems like it might help.

Different situations call for different plans. For now, I want to talk about the common anti-pattern that often happens instead.

Demon Threads  are explosive, frustrating, many-tentacled conversations that draw people in regardless of how important they are. They come in two forms:

*   **Benign Demon Threads** are mostly _time wasting._ Nobody gets _that_ angry, it's just a frustrated mess of "you're wrong" "no y_ou're_ wrong" and then people spend loads of digital ink arguing about something that doesn't matter much.
*   **Malignant Demon Threads** feed upon emotions of defensiveness, anger, tribal affiliation and righteousness - and inflame those emotions, drawing more people into the fire.

(A malignant demon thread is cousin to the _flame war_ \- people hurling pure insults at each other. What makes a malignant demon thread insideous is the way it can warp discussion even among people who are earnestly trying to communicate, seek truth and solve problems)

If you find yourself in a malignant demon thread, I think it's likely you are not only _not helping_, but are actually hurting your cause.

The Demon Seed
==============

_How to write so that people will comment \[disclaimer: not necessarily_ **_good_** _advice\]_

> _1\. Be wrong_

> _2\. Be controversial_

> _3\. Write things people feel qualified to have opinions on._

> _4\. Invoke social reality._

> _\- [Writing That Provokes Comments](https://www.lesserwrong.com/posts/GHBLFPDhzeSQHx2eM/writing-that-provokes-comments)_

In the comments on YouTube, or the worst parts of Facebook or tumblr, demon threads are not surprising. People write comments that inflame ideological warfare all the time. Internets be internets. People be people. What can you do?

The surprising thing is how this works in places where everyone should really know better. The powers of demons are devious and subtle.

There's an experiment — insert obligatory replication crisis disclaimer — where one participant is told to gently poke another participant. The second participant is told to poke the first participant the same amount the first person poked them.

It turns out people tend to poke back slightly harder than they were first poked.

Repeat.

A few iterations later, they are striking each other really hard.

I think something like this is at work in the mechanics of demon threads.

The **Demon Seed** is the first comment in what will soon become a demon thread. It might look pretty innocuous. Maybe it feels _slightly_ rude, or slightly oblivious, or pushing a conversation that should be about concrete empirical facts slightly towards being about social consensus (or, vice versa?).

It feels 1% outside the bound of a reasonable comment.

And then someone **waters the demon seed**. They don't want to let the point stand, so they respond with what seems like a fair rebuke.

Maybe they're self-aware that they're feeling annoyed, so they intentionally dial back the aggression of their response. "Ah, actually this probably comes across as too hostile, so I'll tweak my wording to reduce hostility by 4%." But, actually, the words were 6% more hostile than they thought, and now they've escalated 2%.

Repeat 2-3 times. The demon seed is watered. Latent underlying disagreements about how to think properly_..._ or ideal social norms... or which coalitions should be highest status... or pure, simple _you're insulting me and I'm angry_…

They have festered and now they are ready to explode.

Then someone makes a comment that pushes things over the edge, and a demon thread is born.

(It is, of course, possible to skip steps 1-4 and just write a blatantly rude, incendiary comment. I'm trying to describe how this happens _even when everyone is well intentioned and mostly trusts each other)_

From there, if you're _lucky_ it's contained to two people. But often, well meaning bystanders will wander by and think "Ah! People are being wrong on the internet! Wrong about things I am qualified to have opinions on! I can help!"

And it grows.

Then people start linking it from elsewhere, or FB algorithms start sharing it because people are _commenting_ so the thread must be _important_.

It grows further.

And it consumes days of people's attention and emotional energy. More importantly, it often entrenches people's current opinions, and it burns people's _good will_ that they might have been willing to spend on honest, cooperative discourse.

Why Demon Threads are Bad
=========================

I think demon threads are not just a bad plan - I think they are often _net negative_ plan.

The reason is best expressed in Conor Moreton's Idea Innoculation and Inferential Distance. \[Edit: the full article is no longer available\]

> **Inferential distance** is the gap between \[your hypotheses and world model\], and \[my hypotheses and world model\]. It's _just how far out_ we have to reach to one another in order to understand each other.

> If you share political and intellectual and cultural foundations, it's (relatively) easy. If you have completely different values and assumptions, (say you get dropped off in the 15th century and need to argue with Christopher Columbus) it may be nigh impossible.

> It's right in the name—_inferential_ distance. It's not about the "what" so much as it is about the "how"—how you infer new conclusions from a given set of information. When there's a large inferential distance between you and someone else, you don't just disagree on the object level, you also often disagree about **_what counts as evidence,_ _what counts as logic,_** and **_what counts as self-evident truth._**

> What makes this really bad is **idea inoculation.**

> When a person is exposed to a weak, badly-argued, or uncanny-valley version of an idea, they afterwards are _inoculated_ against stronger, better versions of that idea. The analogy to vaccines is extremely apt—your brain is attempting to conserve energy and distill patterns of inference, and once it gets the shape of an idea and attaches the flag "bullshit" to it, it's _ever after_ going to lean toward attaching that same flag to any idea with a similar shape.

> When you combine idea inoculation with inferential distance, you get a recipe for disaster—if your first attempt to bridge the gap fails, your second attempt will _also_ have to overcome the person's rapidly developing resistance.

> You might think that each successive attempt will bring you closer to the day that you finally establish common ground and start communicating, but alas—often, each attempt is just increasing their resistance to the core concept, as they build up a library of all the times they saw something like this defeated, proven wrong, made to look silly and naive.

A demon thread is a _recipe for bad attempts at communicating_. Lots of people are yelling at once. Their defenses are raised. There's a sense that if you give in, you or your people look like losers or villains.

This'll make people worse at listening _and_ communicating.

Why the Internet Worse
======================

"Demon threads" can happen in person, but they're worse online.

One obvious reason is that **the internet is more anonymous**. This reduces consequences to the person writing a comment, and makes the target of the comment easier to round off to a bad stereotype or an abstract representation of The Enemy.

Other things people do:

**A. People end up writing long winded monologues** without anyone interrupting them to correct basic, wrong assumptions.

i.e. "you're just wrong because you think X, therefore... \[complicated argument\]", without providing opportunity for someone to respond "no I don't actually think X at all". And then, having written out \[complicated argument\] you're already _invested_ in it, despite it being built on faulty premises.

**B. Lots of people are writing.** Especially as the demon thread grows. After 24 hours of its existence, the thread will have so much content it's a huge investment to actually read everything that's been said.

**C. The comments aren't necessarily displayed in order.** Or, if they are, people aren't reading them in order, they're reading whatever it's largest or most interesting.

**D. The internet is full of lots of other content competing for attention.**

This all means that:

**E. People are _skimming_.** This is most true when lots of people are writing lengthy monologues, but even when the thread first begins, people's eyes may be bouncing around to different tabs or different threads within a page so they _aren't even reading what's being said_, not with the intentionality and empathy they would when confronted with a real person in front of them.

And they might _first_ be reading the most explosive, recent parts of a thread rather than piecing together the actual order of escalation, which may make people look less reasonable than they were.

This all adds up to giant threads being a _uniquely bad way to resolve nuanced, emotionally fraught issues._

Containment?
============

Demon threads are like wildfires. _Maybe_ you can put them out, with coordinated effort. You can also try to ignore them and hope they burn themselves out.

But if you wanted to actually stop it, the best bet is to do so is before they're erupted in the first place.

I've developed a sense of what seeds look like. I'll see a comment, think "god, this is going to become a demon thread in like two hours", and then sure enough, two hours later people are yelling at each other and everything is awful and everyone involved seems _really sure_ that they are helping somehow.

Some flags that a demon thread might be about to happen:

**Flags Regarding: Tension and Latent Hostility**

*   When you look a comment and want to respond, you feel a visceral sense of "you're _wrong_", or "ugh, those people \[from group that annoy me\]" or "important principle must be defended!" or “I am literally under attack.”
*   You feel physiological defensiveness or anger - you notice the hairs on the back of your neck or arms standing on end, or a tightness in your chest, or however those emotions manifest in your body.
*   People in the thread seem to be talking past each other.
*   For whatever reason, tensions seem to be escalating.

**Flags Regarding: Social Stakes**

*   The argument seems like it's about who should be high or low status, which people or groups are virtuous and which are not, etc.
*   The argument is about social norms (in particular if at stake is whether some people will end up feeling unwelcome or uncomfortable in a given community/space that is important to them - this is _extremely_ threatening)
*   More generally - the argument touches _in some way_ on social reality, in ways that might have ramifications beyond the immediate conversation (or that people are _afraid_ might have such ramifications).

If some of the above seem true (in particular, at least one of the first group and at least one of the second), then I think it's worth stepping back and being _very careful_ about how you engage, even if no comment seems especially bad yet.

Potential Solutions
===================

The first line of defense is to notice what’s happening - recognize if you’re feeling defensive or angry or talking past each other. Brienne's [Noticing Sequence](http://agentyduck.blogspot.com/p/noticing.html) is pretty good for this (as well as her particular posts on training the skills of [Empathy](http://agentyduck.blogspot.com/2015/05/tortoise-report-3-empathy_16.html) and handling [Defensiveness](http://agentyduck.blogspot.com/2015/08/tortoise-report-5-defensiveness.html) \- these may not work for everyone but I found the underlying thought process useful).

But while noticing is necessary, it’s not _sufficient._

Rather than list my first guesses here, I’ll be discussing them in the comments and following this up with a “best-seeming of the potential solutions” post.

Meanwhile, some factors to consider as you decide what to do:

*   How are you involved?

*   Are you one of the people initially arguing, or a bystander?
*   How much do you normally trust the people involved?
*   Is it possible to take the conversation private?

*   Are we on the demon _seed_ or demon _thread_ stage? Is there common knowledge about either?
*   What are the actual stakes?
*   What are the moderation tools available to you?
*   Are you in a venue where you have the ability to shape conversational norms?

*   Do you directly control them (i.e. personal blog or feed?)
*   Does _anyone_ have direct ownership of the venue? (either technically, or culturally)
*   Is there anything you can do _unilaterally_ to make the conversation better, or will it require help from others?

*   Are you building a site where you get to develop entire new tools to deal with this class of problem?

With that in mind…

In whatever venues you most find yourself demon-thread-prone, what sort of plans can you actually think of that _might actually help?_

* * *

_Note: I have since written a followup post with a [working example of what I think people should usually do instead of demon threads.](https://www.lesserwrong.com/posts/LxrpCKQPbdpSsitBy/short-circuiting-demon-threads-working-example)_