Content Moderation for End-to-End Encrypted Messaging
Jonathan Mayer
Princeton University
October 6, 2019
Thursday evening, the Attorney General, the Acting Homeland Security Secretary, and top law
enforcement officials from the U.K. and Australia sent an ​open letter​ to Mark Zuckerberg. The
letter emphasizes the scourge of child abuse content online, and the officials call on Facebook topress pause on end-to-end encryption for its messaging platforms.
The letter arrived the same week as a widely shared New York Times a​ rticle​, describing howreports of child abuse content are multiplying. (As the article notes, it is unclear how much of
the increase is due to improved reporting and detection and how much is due to growing
criminal activity.) The article provides a heartbreaking account of how the National Center for
Missing and Exploited Children (NCMEC) and law enforcement agencies are overburdened
and under-resourced in addressing horrible crimes against children.
The law enforcement letter, remarks at a Department of Justice e​ vent​ on Friday, the New YorkTimes article, and new NCMEC ​materials​ on encryption appear to reflect a common technicalassumption:
1. Content moderation is fundamentally incompatible with end-to-end encrypted
messaging.1
The law enforcement remarks appear to reflect an additional technical assumption:

1

L​ aw Enforcement Letter​: “Our understanding is that much of [Facebook’s reporting on harmful content]. . . will no longer be possible if Facebook implements its proposals as planned. NCMEC estimates that
70% of Facebook’s reporting – 12 million reports globally – would be lost.”
Deputy Attorney General Rosen Remarks​: “The [hash-based] monitoring practices I’ve described areinconceivable with end-to-end encryption.”
FBI Director Wray Remarks​: “With the spread of user-controlled default encryption, providers frequently
can’t identify horrific images within encrypted data. . . . Most of the tips Facebook currently provides are
based on content. With end-to-end encryption, those would dry up. Facebook itself would no longer beable to see the content of its users’ accounts.”
NCMEC Materials​: “The use of end-to-end encryption would prevent the companies or any third-party
from detecting illegal activity occurring on their platforms . . . .”New York Times Article​: “Data obtained through a public records request suggests Facebook’s plans toencrypt Messenger in the coming years will lead to vast numbers of images of child abuse going
undetected.”2. Enabling content moderation for end-to-end encrypted messaging fundamentally poses
the same challenges as enabling law enforcement access to message content.2
My goal in this discussion paper is to provide a technical clarification for each of these points.
1. Forms of content moderation may be compatible with end-to-end encrypted messaging,
without compromising important security principles or undermining policy values.
2. Enabling content moderation for end-to-end encrypted messaging is a different problem
from enabling law enforcement access to message content. The problems involve
different technical properties, different spaces of possible designs, and different
information security and public policy implications.
I aim to demonstrate these clarifications by formalizing specific content moderation properties
for end-to-end encrypted messaging, then offering at least one possible protocol design for each
property.
Context and ScopeChild exploitation is an extraordinarily sensitive topic, and encryption policy has been a policy
firestorm for years. Before turning to technical analysis, I would like to briefly emphasize thecontext for and narrow scope of this discussion paper.
First, my technical clarification in response to law enforcement and NCMEC is in no way a
criticism. A couple years ago, I had the opportunity to visit NCMEC’s headquarters and engage
with law enforcement officers in child exploitation units. I have the utmost respect for the
professionals who work around the clock to combat child exploitation.
Second, this discussion paper is inherently preliminary and an agenda for further
interdisciplinary research (including my own). I have written it to organize my current thinking
and to reflect constructive conversations with colleagues in computer science, law, and public
policy. I am not yet prepared to normatively advocate for or against the protocol designs that I
describe below. I am not claiming that these concepts provide sufficient content moderation
capabilities, the same content moderation capabilities as current systems, or sufficientrobustness against evasion.3 I am also not claiming that these designs adequately address
2

The framing for the Department of Justice event invoked “lawless spaces” and “warrant-proof
encryption,” linking the challenges of content moderation to the challenges of law enforcement access todata. Remarks at the event from A​ ttorney General Barr​, D​ eputy Attorney General Rosen​, and ​FBI
Director Wray​ all raised content moderation in connection with law enforcement access. Similarly, Alan
Rozenshtein at the University of Minnesota Law School has written a b​ log post​ linking the New YorkTimes article to encryption and law enforcement access.
3
Note that robustness against evasion is also a challenge for messaging services that do not implement
end-to-end encryption. A user might encrypt content before sending it, for example, or might switch to adifferent service.

2

information security risks or public policy values, such as free speech, international humanrights, or economic competitiveness.
I do not know if there is a viable path forward for content moderation and end-to-end
encrypted messaging that will be acceptable to technology platforms, law enforcement,
NCMEC, civil society groups, information security experts, and other stakeholders. I do haveconfidence that, if such a path exists, we will only find it through open research and dialogue.
Technical Properties for End-to-End Encrypted Messaging
A foundational principle of applied cryptography is that, when designing or evaluating asystem, we should be precise about the technical properties that we aim to accomplish. So, what
are the properties that we might want for addressing harmful content in an end-to-end
encrypted messaging app? Here is my attempt at a nonexhaustive taxonomy.
●●

Content Moderation○

User Reporting: I​ f a user receives a message that he or she believes contains
harmful content, can the user report that message to the service provider?

○

Known Content Detection: ​Can the service provider automatically detect whena user shares content that has previously been labeled as harmful?

○Classifier-based Content Detection: C​ an the service provider detect when a user
shares new content that has not been previously identified as harmful, but that
an automated classifier predicts may be harmful?

○Content Tracing:​ If the service provider identifies a message that contains
harmful content, and the message has been forwarded by a sequence of users,
can the service provider trace which users forwarded the message?

○

Popular Content Collection: C​ an the service provider curate a set of content thathas been shared by a large number of users, without knowing which users
shared the content?

Lawful Access○Message History: ​With appropriate legal process, can the service provider
disclose messages that a user previously sent or received to law enforcement?

○

Message Interception: W​ ith appropriate legal process, can the service provider
disclose messages that a user sends or receives going forward to law
enforcement?

3Note that each of these technical properties is distinct. Each poses a different set of design
challenges. And, importantly, there may be room for stakeholder consensus on content
moderation that has proven elusive on lawful access.
In the balance of this paper, I will discuss how to possibly implement each of the content
moderation properties.
User Reporting
In an end-to-end encrypted messaging system, a service provider cannot read the content of
messages—but intended recipients can. Enabling user reports is, therefore, not a technicalchallenge. If a user receives a message that he or she believes contains harmful content, the user
can simply submit that message to the service provider (e.g., with a “Report Message” option).
The technical challenge, rather, is a​ uthenticating​ user reports. How can a service provider haveconfidence that a specific sender actually sent the flagged message, and at a specific time?
Authentication is especially important for law enforcement, since harmful content in a
messaging app may become forensic evidence in a criminal proceeding.
There is, thankfully, a straightforward means of establishing whether a message reported by auser is authentic. The service provider can associate each message with a cryptographic
statement of authenticity that it was sent by a particular sender at a particular time, without
learning the content of the message. If a user reports a message to the service provider, theprovider can then check the statement of authenticity and verify the flagged message.
Facebook dubbed this design “message franking,” and it is a​ lready deployed in Facebook
Messenger​. More recent academic work has formalized and expanded on Facebook’s approach(​Grubbs et al. 2017​, ​Dodis et al. 2018​, ​Leontiadis & Vaudenay 2018​, ​Tyagi et al. 2019​,
Huguenin-Dumittan & Leontiadis 2019​).
Known Content Detection
In a traditional messaging platform, when a user sends a message, the service provider cancheck the content against a dataset of known harmful content. How can a service provider
accomplish that sort of known content matching in an end-to-end encrypted messaging app?
The technical foundation for known content detection is a concept called “perceptual hashing.”
A perceptual hash is simply a number derived from a piece of content, with several important
properties: If pieces of content are identical, they will have the same hash. If pieces of content
are perceptually similar (e.g., scaled or rotated images), they will likely have similar hashes.4
4

Hamming distance is a common metric for evaluating whether two perceptual hashes are similar.

4And if two pieces of content are not perceptually similar, they will likely not have similar
hashes.
Perceptual hashing is widely used today for detecting known harmful content. Service
providers can share perceptual hashes without sharing harmful content, and perceptual
hash-based content matching requires both relatively little storage and relatively little
computation. Perceptual hashing algorithms like F​ acebook PDQ and TMK+PDQF​, ​Microsoft
PhotoDNA​, w​ Hash​, d​ Hash​, ​pHash​, and a​ Hash​ are free and relatively easy to implement.
A possible direction for detecting known harmful content in an end-to-end encrypted
messaging app is to implement perceptual hashing within the app. Consider the following
design for checking the content of a message that a user sends or receives.
1. Begin with a set of perceptual hashes, derived from known harmful content.
2. Insert the hashes into a data structure that is small enough to fit in device storage andthat can quickly check whether a new hash is similar to a hash in the data structure.5
Note that this data structure would be probabilistic; depending on the implementation,
it could generate false positives or false negatives.
3. Distribute the data structure with the messaging app, and update the data structure as
the service provider learns about new perceptual hashes of harmful content.
4. When a user sends or receives a message, the app generates a perceptual hash for themessage content and checks it against the data structure.
5. If there is a possible match to known harmful content, the app automatically reports the
unencrypted message to the service provider for further evaluation (e.g., manual
review).6
This design involves a tradeoff between storage space on the user’s device and the performanceof similarity detection. That could have significant implications: a false positive means themessaging app has disclosed content that is not known to be harmful, and a false negativemeans the messaging app has failed to disclose content that is likely harmful.
It may be possible to improve this tradeoff between device storage and detection performance.
For example, it may be possible to construct a protocol (either in addition to or separate fromthe design above) that checks a perceptual hash against a service provider’s database, without
5

A combination of locality-sensitive hashing and a Bloom filter could provide these properties, as could amore sophisticated distance-sensitive Bloom filter (see ​Kirsch & Mitzenmacher 2006​, H​ ua et al. 2011​, andGoswami et al. 2017​).
6
If there is not a match, it may also be possible for the app to generate a zero-knowledge proof that themessage content is not known to be harmful.

5revealing the hash that the app is checking.7 C​ loudflare​ and ​Google​, for example, already use asimilar approach for checking whether a user’s hashed password is in a large dataset ofcompromised hashed passwords (see ​Thomas et al. 2019​ and L​ i et al. 2019​).
Classifier-based Content Detection
Facebook​ and ​Google​ (among other technology platforms) have publicly confirmed that theyuse machine learning classifiers to detect new instances of harmful content. How could they
continue to apply classifiers in an end-to-end encrypted messaging environment?
A straightforward approach would be to generate machine learning models for harmful
content, then deploy the models in the messaging app. That type of decentralized, in-app
machine learning application is now quite common, and it is supported by popular machinelearning libraries.
Consider the following design, which a service provider could use for each distinct category ofharmful content:
1. Generate a classifier for the category of harmful content.8 This step could require a large
volume of data and significant computational capacity; a service provider would
implement it on its own infrastructure.
2. If necessary, reduce the size of the classifier so that is small enough to fit in device
storage.9
3. Distribute the classifier with the messaging app, and update the classifier as the service
provider implements improvements or incorporates new data.
4. When a user sends or receives a message, the app runs the classifier on the message
content. This step can be fast—evaluating with a machine learning classifier is mucheasier than training the classifier.
5. If the classifier predicts that the message content is harmful, the app automaticallyreports the unencrypted message to the service provider for further evaluation (e.g.,manual review).

7This task could possibly be formulated as an oblivious transfer, private information retrieval, or private
set intersection problem, for instance. Or it could be structured as a database query with k-anonymity,
trading some rigor for easier implementation.
8
This could be a straightforward application of supervised learning to labeled harmful and non-harmful
content.
9
There is an active research subfield examining how to generate compact or pruned machine learningmodels, trading accuracy and other performance characteristics against storage and computational
requirements.

6

Much like the approach to known content detection, this approach would have tradeoffs. Moresophisticated models would require greater on-device storage and computation, but they could
also increase classification accuracy (or other performance characteristics). False positive andfalse negatives would pose the same problems as for known content detection.
One possible improvement to the protocol would be to tune the in-app models for efficiency
and a low false negative rate. If a model predicts that message content may be harmful, the app
could submit the unencrypted content (or features derived from the content) to a moresophisticated server-side model.10
Another possible, longer-term improvement would be to use a protocol that allows a messaging
app to submit message content to the service provider for classification, without revealing thecontent to the service provider. This type of privacy-preserving “outsourcing” for machinelearning is an active area of research, and recent progress is promising (e.g., D​ owlin et al. 2016​,
Liu et al. 2017​, J​ uvekar et al. 2018​, and J​ iang et al. 2019​).
Content Tracing
When a user receives a message that contains harmful content, sometimes the user will forward
the message to additional users.11 If a service provider cannot read a piece of content, how can it
determine the sequence of forwards that preceded the content?
One approach would be to retain metadata for all messages. When a piece of content is flagged
as harmful, the service provider could try to use metadata attributes (e.g., message size andtimestamp) to reconstruct the content’s path. That approach is probabilistic and may beinaccurate, though, and retaining message metadata has a significant privacy impact.12
A better approach would be to build on the message franking concept, which enables a service
provider to authenticate a reported message.13 Each message could come with not just a​The app could attempt to protect a user’s identity unless the server-side model classifies content as
harmful. For example, the app could submit the content for classification using an anonymizing network,a mixnet, or a metadata-private messaging protocol. Since content often contains identifying information,these approaches would not necessarily be effective.
11
This issue has been especially problematic for WhatsApp, which responded by ​limiting​ message
forwarding.
12
Facebook has ​indicated​ that it plans to use message metadata to identify and trace harmful content.
13
Consider the following possible design: At each stage of forwarding, the sender would add the priormessage franking material to the bundle, generate a random symmetric key, and encrypt the bundle with
the new key. It would then encrypt the key with an asymmetric key for the service provider, append theencrypted key to the message content, and send the message with franking. If the message recipient or
the app flags the content as harmful, the app can share the latest message, its franking, the franking
bundle, and the encrypted franking bundle key with the service provider. The service provider can thenauthenticate the message, decrypt the franking bundle key, and iteratively decrypt each franking in thebundle. At the end of this process, the service provider has sufficient information to authenticate theentire series of forwards that preceded a message.
10

7

franking for the last sender, but also a “franking bundle” for previous forwards. If a user
forwards a message, the app could add the latest franking to the bundle. The service provider
would generate a cryptographic statement of the bundle’s authenticity, without either theprovider or the recipient learning the bundle’s contents. If a user or the app flags a message, theservice could then unpack the bundle, read the sequence of message frankings, and reconstruct
the sequence of message forwards.
Tyagi et al. 2019​ have proposed a similar concept, using server-side storage rather than afranking bundle.14 Their paper also proposes an approach for forward content tracing, enablinga service provider to trace an entire tree of forwarding activity if a piece of content is reported.
Popular Content Collection
A service provider might want to know about pieces of content that have been shared by a large
number of users. One reason is for manual labeling—a provider might want to have humanmoderators review popular content. Another reason is for automated detection of harmful
content—building a machine learning model for harmful content requires training data thatconsists of ​non-harmful​ content. How could an end-to-end secure messaging platform compile acollection of popular content, when it is unable to access message content?
One possibility would be to rely on the subset of users who have not enabled end-to-end
encryption. So long as end-to-end encryption is optional for a messaging app, that might besufficient.
Another option would be to rely on platforms that don’t use end-to-end encryption. For
example, it might be a reasonable assumption that popular content on Facebook is roughly
similar to popular end-to-end encrypted content on WhatsApp.
Yet another option would be to enable a service provider to collect widely sharedcontent—content that is likely to be non-harmful—without revealing which users shared thecontent. For example, a service provider could maintain a count of how often content with a
specific hash is shared, without knowing which users have shared content with that hash.15
When the shares of a specific hash exceed a threshold, the service provider could flag it, and the14

One possible advantage of the server-side approach is that it does not disclose the number of preceding
forwards, which may be a desirable property. A franking bundle, by contrast, reveals the number offorwards with its size. That issue could be mitigated by using a fixed size for franking bundles, at theexpense of additional message overhead.
15
This design could possibly be implemented as a differentially private count for each hash. The hashes
should be cryptographic hashes (rather than perceptual hashes), since the design depends on theproperty that the service provider cannot learn about content from its hash. It may be possible to improvethe design by using a secret sharing approach for each hash, such that a service provider would only beable to request a copy of the content associated with a hash if it can cryptographically demonstrate that alarge number of users have shared content with that hash.

8next time a user shares matching content, their app could submit the content to the service
provider in a way that does not disclose the user’s identity.16
A possible longer-term solution to this challenge would be enabling the service provider torefine its classifiers for harmful content, without sending non-harmful content back to theservice provider. Each instance of the messaging app could make small improvements to theclassifiers using the user’s message content, and the app would periodically submit those
improvements to the service provider—without disclosing any message content. This type ofprivacy-preserving “federated learning” is an area of ongoing, promising research (e.g., ​Geyer
et al. 2018​).
Parting Thoughts
In closing, I would like to reemphasize the narrow goal of this paper: demonstrating that formsof content moderation may be technically possible for end-to-end secure messaging apps, andthat enabling content moderation is a different problem from enabling law enforcement accessto content. I am not yet advocating for or against the protocols that I have described. But I do
see enough of a possible path forward to merit further research and discussion.
Acknowledgments
Thanks to J​ oe Bonneau​, ​Ed Felten​, N​ ick Feamster​, B​ en Kaiser​, ​Anunay Kulshrestha​, ​Arvind
Narayanan​, and ​Laura Roberts​ for comments on an earlier draft. All views, errors, andomissions are solely my own.

16

The app could, for example, use an anonymizing network, a mixnet, or a metadata-private messaging
protocol.

9