The Invisible Labor of Content Moderation
Tags: Politics, Twitter, Social Media, Facebook, Content Moderation

We’ve learned that moderation is an incredibly difficult (and often thankless) task. On top of this, the safe harbor provision tells us that platforms don’t have to moderate sites if they don’t want to (perhaps this is why Mark Zuckerberg held steadfast in 2016 that Facebook is not a media company, so as not to be burdened with the legal obligations that apply to media companies vs. social platforms). So… why do it?

There are myriad reasons to aim for a well-moderated site: a platform’s environment and mood can influence a user’s propensity to engage in a certain kind of behavior. But beyond this, there can be economic consequences when platforms lose users who are driven away by abuse or uncomfortable content. In many cases, this is more formally known as an Eternal September: the permanent destruction of a community’s norms (and, thus, an inability to effectively moderate) due to an influx of newcomers.

Content moderation is necessary, but it’s incredibly difficult to execute effectively. There seem to be three “imperfect solutions:” paid moderation, community moderation, and algorithmic moderation. In each of these, what works? What doesn’t, and what can we learn?

Paid Moderation

With paid moderation, a third party reviews any claims, which helps avoid brigading and supports a more calibrated and neutral evaluation of the content. Facebook is just one example of a platform that utilizes this method, employing about 15,000 content moderators directly or indirectly. If there are three million posts to review each day, that equates to 200 per person: 25 every hour in an eight-hour shift, or under 150 seconds per post.

A demanding job like this can result in major emotional trauma for moderators who spend days on end reviewing troubling content (related: Facebook paid $52 million to content moderators suffering from PTSD). Additionally, evaluators may only have seconds to make a snap judgement about a specific piece of content, so there’s still room for error.

Community Moderation

We see community moderation in sites like Reddit, Twitch, and Steam. On Reddit, for example, users moderate specific subreddits, removing content that breaks the rules. One user, who moderates over 60 subreddits, considers moderating “a glorified janitor’s job, and there is a unique pride that janitors have… When I’m finished for the day I can stand back and admire the clean and functioning subreddit, something a lot of people take for granted.”

Stronger actions have worked on Reddit, too: Reddit’s 2015 ban of two subreddits due to violations of anti-harassment policy resulted in accounts leaving the platform entirely (or migrating to other subreddits, drastically reducing their hate speech). On Twitch, community moderators are responsible for removing content and banning users in real time. One study about the pro and anti-social behavior on Twitch showed that moderating content or banning users substantially decreases negative behaviors in the short term.

Community moderation leverages its users’ intrinsic motivation, and local experts of a given corner of the Internet are more likely to have the necessary context to make difficult calls. That said, however, community moderators can often feel bitter that they don’t get the recognition they deserve, and can resent that the platform seems to profit off their free labor. On top of all this, community moderation can vary in quality: it’s not necessarily consistent, fair, or just.

These issues are well articulated in this New York Times article, which writes that moderators are “forces for stability and civility in the raucous digital realm. Or that is, they’re supposed to be.” Community moderators can have extreme power on their respective platforms, and they can wield it with often profound unpredictability. During the infamous 2015 Reddit Revolt, moderators upset with a decision made at Reddit’s corporate level chose to shut down their forums, which collectively garnered millions of visits each day. These moderators wanted to voice their feeling that the company’s leadership “[did] not respect the work put in by thousands of unpaid volunteers.” A week after the Reddit Revolt, Ellen Pao — the company’s interim chief executive — resigned.

Algorithmic Moderation

When Facebook, YouTube, Twitter, and other tech companies sent workers home to protect them from the coronavirus, they ran into a host of new challenges regarding content moderation: as these platforms began relying more heavily on automated systems to flag content, they began seeing more and more posts erroneously marked as spam because of algorithm weaknesses. Some content moderation can’t be done outside the office due to privacy and security reasons — consequently, these tech companies were suddenly dealing with a lot more information to moderate, and a lot less staff.

Every large social computing system utilizes algorithmic moderation in some form. Perhaps algorithmic moderation’s biggest strength is that the system can act quickly to remove content, before people are hurt or otherwise negatively affected. However, these systems clearly aren’t perfect — they make embarrassing errors, often unintended by the creators of the algorithm. These errors are also often interpreted by users as intentional platform policy. Additionally, even if a perfectly fair, accountable algorithm were possible, culture would evolve and training data would inevitably become out of date.

These solutions — paid, community, and algorithmic moderation — each have their own strengths and weaknesses. To account for the gaps, many social computing systems use multiple tiers: Tier 1 is typically algorithmic moderation for the most common and easy-to-catch problems, while uncertain judgements move to human moderators in Tier 2. These human moderators can be paid (like Facebook) or community (like Reddit), depending on the platform — they can monitor flagged content (or, sometimes, all new content that comes in), or review an algorithmically curated queue.