What Pornhub and Peloton Have in Common With Facebook
By: Will Oremus
Tags: Pornhub, Content Moderation, Pattern Matching, Peloton, Facebook

What Pornhub and Peloton Have in Common With Facebook

An online speech expert explains why no online platform will be spared from content-moderation controversy

Photo: Brett Carlsen/Getty Images

For years a battle of ideas has raged over the limits of online speech, focused largely on Facebook, Twitter, and to a lesser extent YouTube. Innately resistant to the messy and expensive work of policing users’ speech, those vast platforms have grudgingly enlarged their moderation workforces, expanded their content policies, and toughened their enforcement in response to media backlashes, congressional hearings, regulatory threats, advertiser boycotts, and revolts from their own employees. Professional racists such as Milo Yiannopoulos, conspiracy theorists such as Alex Jones, and even grassroots movements such as QAnon have all been booted from major platforms for violating policies after significant backlash. Public officials such as Donald Trump now find themselves fact-checked or their posts hidden. Misinformation about Covid-19 and voting is being throttled or taken down.

These battles are far from over, and may never be. But for all the attention paid to a few high-profile platforms, countless other online forums have largely avoided scrutiny of their content policies — so far.

That will change, says Evelyn Douek, a law lecturer at Harvard who researches online speech. And it will change in a hurry, she believes. She has developed a working theory of online speech that explains why, and offers some clues for what platforms should be doing about harmful expression — ideally before they find themselves in the headlines for the wrong reasons.

The Pattern

Content moderation controversies aren’t just for social networks anymore.

On Friday, New York Times columnist Nick Kristof published an exposé about Pornhub, the massive pornography site that has sought to position itself as a more or less respectable internet giant with a mischievous streak. His reporting makes the case that Pornhub systematically profits from heinous sexual crimes by hosting and monetizing user-posted videos without sufficiently careful oversight. From the story:

That supposedly “wholesome Pornhub” attracts 3.5 billion visits a month, more than Netflix, Yahoo or Amazon. Pornhub rakes in money from almost three billion ad impressions a day. One ranking lists Pornhub as the 10th-most-visited website in the world. Yet there’s another side of the company: Its site is infested with rape videos. It monetizes child rapes, revenge pornography, spy cam videos of women showering, racist and misogynist content, and footage of women being asphyxiated in plastic bags. A search for “girls under18” (no space) or “14yo” leads in each case to more than 100,000 videos. Most aren’t of children being assaulted, but too many are.

Pornhub’s response mirrors that of other platforms unaccustomed to being confronted publicly with their ugliest underbellies. The company’s comments were terse and defensive, insisting that it has the problem under control while declining to discuss how, or with what resources. One anonymous moderator estimated to Kristof that Pornhub’s parent company, Mindgeek, has just 80 moderators worldwide. In short, it seems to be doing the bare minimum and hoping to get away with it. A clearly indignant Kristof — arguably going a bit overboard — calls on everyone from payment processors to Canadian Prime Minister Justin Trudeau to crack down on the firm. At the same time, he acknowledges that a rival porn site, Xvideos, may have “even fewer scruples” and “attract more videos,” even as it mostly skates by in his piece. Whether or not Pornhub is quite as culpable as Kristof contends, it seems likely that the online porn industry is in for some nasty news cycles.

The company’s comments were terse and defensive, insisting that it has the problem under control while declining to discuss how, or with what resources. One anonymous moderator estimated to Kristof that Pornhub’s parent company, Mindgeek, has just 80 moderators worldwide. In short, it seems to be doing the bare minimum and hoping to get away with it. A clearly indignant Kristof — arguably going a bit overboard — calls on everyone from payment processors to Canadian Prime Minister Justin Trudeau to crack down on the firm. At the same time, he acknowledges that a rival porn site, Xvideos, may have “even fewer scruples” and “attract more videos,” even as it mostly skates by in his piece. Whether or not Pornhub is quite as culpable as Kristof contends, it seems likely that the online porn industry is in for some nasty news cycles. The story fits a pattern that Douek has been tracking for a while now. “It’s the inevitable life cycle of a user-generated content platform,” she says. It goes like this:

1. Someone, often an activist or journalist, finds abhorrent content on Platform X.

2. It becomes a scandal, and pressure builds as more examples are uncovered.

3. The platform eventually takes action, but does so in a haphazard and reactive way, geared toward solving the PR problem rather than the underlying issues.

4. Sooner or later, its inconsistent approach to moderation sparks a backlash of its own.

Perhaps it should have been obvious that this day would come for a platform such as Pornhub. But Douek’s theory holds that it will also come for platforms that do not seem like obvious candidates for content moderation controversies. In short, she says, “ If you’re going to have users generating content, you’re going to have users generating harmful content ,” Douek said in a phone interview. “And you need to be thinking in advance about what you’re going to do about it.” It has the ring of a law of the internet, akin to Poe’s Law or Godwin’s Law. Perhaps we could call it Douek’s Law, though she rejected the moniker when I half-jokingly suggested it.

,” Douek said in a phone interview. “And you need to be thinking in advance about what you’re going to do about it.” It has the ring of a law of the internet, akin to Poe’s Law or Godwin’s Law. Perhaps we could call it Douek’s Law, though she rejected the moniker when I half-jokingly suggested it. Obvious as the thesis might seem, every week now seems to bring fresh examples of platforms that have failed to take its implications to heart. We’ve seen Spotify struggle to explain why it was okay for its podcast star Joe Rogan to host Alex Jones, whom Spotify had banned from its platform two years earlier. The fast-growing newsletter platform Substack so far has resisted calls to take down “Stop the Steal” posts even as Facebook has taken aggressive action against groups pushing debunked election-fraud claims. Even Peloton, of all companies, faced a minor content moderation dustup after a Washington Post editor spotted QAnon hashtags among the options for connecting with like-minded cyclists. (Peloton later removed them.)

We’ve seen Spotify struggle to explain why it was okay for its podcast star Joe Rogan to host Alex Jones, whom Spotify had banned from its platform two years earlier. The fast-growing newsletter platform Substack so far has resisted calls to take down “Stop the Steal” posts even as Facebook has taken aggressive action against groups pushing debunked election-fraud claims. Even Peloton, of all companies, faced a minor content moderation dustup after a Washington Post editor spotted QAnon hashtags among the options for connecting with like-minded cyclists. (Peloton later removed them.) For platforms squarely in the social media business, and especially those that algorithmically amplify content, there’s no excuse at this point for treating content moderation as anything short of central to the operation . That’s why Douek often picks on YouTube as a company that has too often tried to duck the scandals facing its rivals, rather than tackling them head-on. She argued in Wired last month that its CEO Susan Wojcicki should have been alongside Facebook’s Mark Zuckerberg and Twitter’s Jack Dorsey when a Senate committee grilled them about their role in the 2020 election.

. That’s why Douek often picks on YouTube as a company that has too often tried to duck the scandals facing its rivals, rather than tackling them head-on. She argued in Wired last month that its CEO Susan Wojcicki should have been alongside Facebook’s Mark Zuckerberg and Twitter’s Jack Dorsey when a Senate committee grilled them about their role in the 2020 election. As Douek sees it, the progress that Facebook and Twitter have made on content moderation will eventually ratchet up the pressure on other platforms , from YouTube on down. The more restrictive the social media giants become, and the more problems they address (however imperfectly), the more those problems will migrate to sites with less-developed rules and moderation systems. And the more pressure those smaller platforms will face to address them. For example, Joe Rogan hosting Alex Jones on Spotify probably wouldn’t have sparked such an outcry if he hadn’t already been deplatformed elsewhere. My own employer, Medium, drew scrutiny for its moderation of Covid-related posts after Facebook announced tougher policies against pandemic misinformation.

, from YouTube on down. The more restrictive the social media giants become, and the more problems they address (however imperfectly), the more those problems will migrate to sites with less-developed rules and moderation systems. And the more pressure those smaller platforms will face to address them. For example, Joe Rogan hosting Alex Jones on Spotify probably wouldn’t have sparked such an outcry if he hadn’t already been deplatformed elsewhere. My own employer, Medium, drew scrutiny for its moderation of Covid-related posts after Facebook announced tougher policies against pandemic misinformation. As the Peloton example illustrates, however, not every online platform that hosts user-generated content is in a position to delve headfirst into the thickets of online speech policy. Or as Douek put it, “Not every bike startup is going to want to hire a whole bunch of experts on internet culture or signaling.” That’s an issue I discussed with another prominent online speech researcher, UCLA’s Sarah Roberts, when I wrote about online gaming platforms’ problems with child exploitation. For some of the larger and more social platforms, such as Roblox or Minecraft, investing in moderation is almost certainly necessary and worthwhile. But for others, Roberts suggested it might be simpler just to shut down messaging features that they lack the will or resources to adequately moderate.

Or as Douek put it, “Not every bike startup is going to want to hire a whole bunch of experts on internet culture or signaling.” That’s an issue I discussed with another prominent online speech researcher, UCLA’s Sarah Roberts, when I wrote about online gaming platforms’ problems with child exploitation. For some of the larger and more social platforms, such as Roblox or Minecraft, investing in moderation is almost certainly necessary and worthwhile. But for others, Roberts suggested it might be simpler just to shut down messaging features that they lack the will or resources to adequately moderate. Douek offered another prediction: The rise of content moderation as a service . There are already startups offering A.I. tools for detecting patterns that might indicate hate speech or child grooming. It isn’t a stretch to imagine a future that includes firms offering human platform moderation, for those unwilling or unable to take it on themselves. “I think we’re going to move to a place where we start to view content moderation as part of the infrastructure of these kinds of platforms,” Douek said. “But there are lots of concerns with that. It’s not infrastructure. We don’t want to depoliticize these issues and think of them as just technical issues, or non-value-based judgments because they clearly are.”

. There are already startups offering A.I. tools for detecting patterns that might indicate hate speech or child grooming. It isn’t a stretch to imagine a future that includes firms offering human platform moderation, for those unwilling or unable to take it on themselves. “I think we’re going to move to a place where we start to view content moderation as part of the infrastructure of these kinds of platforms,” Douek said. “But there are lots of concerns with that. It’s not infrastructure. We don’t want to depoliticize these issues and think of them as just technical issues, or non-value-based judgments because they clearly are.” There is a sense in which online speech is an unsolvable problem, Douek said. “There’s always going to be harmful content, there are always going to be debates over what harmful content is, and where to draw the line, and there are always going to be mistakes.” But that doesn’t mean we should throw up our hands, she added. She suggested at least two places to start making progress. One is more transparency from platforms, without which it’s easy for critics to misdiagnose their problems and arrive at misguided solutions. The other is for platforms and their critics alike to focus less on quick, concrete outcomes — such as banning a loathed individual when the backlash gets loud enough — and more on rules and processes that will be consistently enforceable in the future. “We will never come to a stable agreement on perfect content moderation rules,” Douek said. “So we need to construct better ways to disagree about them. Forever.”

Undercurrents

Under-the-radar trends, stories, and random anecdotes worth your time.

Speaking of Facebook and content moderation, Facebook’s Oversight Board has selected its first six cases. Depending on your perspective, the board is either a laudable experiment in accountable self-governance on Facebook’s part, or a PR exercise meant to paper over the fact that Facebook has built a fundamentally ungovernable platform. (Can it be both?) As Rest of World’s Louise Matsakis noted, the initial slate of content moderation decisions appealed to the board “concern many of the same topics that have plagued Facebook for years: hate speech, nipples, Nazis.” Most of the cases came from outside the United States, which makes sense, given that most Facebook users do, too.

Depending on your perspective, the board is either a laudable experiment in accountable self-governance on Facebook’s part, or a PR exercise meant to paper over the fact that Facebook has built a fundamentally ungovernable platform. (Can it be both?) As Rest of World’s Louise Matsakis noted, the initial slate of content moderation decisions appealed to the board “concern many of the same topics that have plagued Facebook for years: hate speech, nipples, Nazis.” Most of the cases came from outside the United States, which makes sense, given that most Facebook users do, too. NYC Media Lab’s Justin Hendrix pointed out that five of the six “deal with images posted with an intent other than what the image itself contains.” To me that highlights Facebook’s struggles to discern context and subtext when applying its rules — a common problem with moderation at scale.

The cases notably skirt the platform’s biggest recent domestic controversies, which revolved around Trump and election misinformation. They also drew relatively little media attention stateside, probably for the same reason. The selection suggests the board is taking a sober, global approach to its role — but maybe also that it’s erring on the conservative side with respect to taking controversial stands or potentially irking the company that funded it. Taking up a Trump case would have been polarizing, sure. It would have also been a show of independence, and a bid for relevance, from a board that faces justifiable skepticism on both counts.

Is techno-optimism making a comeback? Techno-optimist Caleb Watney of the Progressive Policy Institute thinks so, and he has the Substack links to back it up. He tweeted a quartet of recent posts making the case for a return of genuine innovation as a force in the economy. The cycle appears to have been kickstarted by economist Tyler Cowen, who in 2011 wrote a book called The Great Stagnation, publishing a brief post raising the possibility that said stagnation may be nearing an end. The most interesting of the responses may be the one from Noah Smith, who suggests that innovation in the 2020s may be driven more by advances in science than software.

Techno-optimist Caleb Watney of the Progressive Policy Institute thinks so, and he has the Substack links to back it up. He tweeted a quartet of recent posts making the case for a return of genuine innovation as a force in the economy. The cycle appears to have been kickstarted by economist Tyler Cowen, who in 2011 wrote a book called The Great Stagnation, publishing a brief post raising the possibility that said stagnation may be nearing an end. The most interesting of the responses may be the one from Noah Smith, who suggests that innovation in the 2020s may be driven more by advances in science than software. A lot of this feels like understandable enthusiasm about the forthcoming Covid-19 vaccines, more than the defense of Big Tech one might expect. And anyone who was already inclined to be skeptical of “techno-optimism” will probably notice that the authors here share some demographic traits that make them, well, more likely to be techno-optimists. Still, the takes are useful as a reminder that technology means more than just software, and retains the capacity to change society for the better, even as that has become a tired cliche in Silicon Valley. As Smith puts it, “technological progress in the 2020s will not necessarily be concentrated in ‘tech’ as we know it.”

Chart of the Week

Complaints about scented candles having no scent, mapped against Covid case counts.

— Kate Petrova