October 2021
1st edition

The
Limits to
DigitalConsent:
Understanding the risks of ethical consent and data collection
for underrepresented communities
Cade Diehm (The New Design Congress), Kelsey Smith (Simply Secure),
Ame Elliott (Simply Secure), Georgia Bullen (Simply Secure)Within ethical design practices, informed user consent is a key requirement in the effort to invert thepower imbalances fostered by the digital platforms that enable platform owners (organizations,
governments, etc.) to build data profiles and make decisions about people. Much work has been
completed by designers, activists, and aligned policymakers to translate data governance policies into
plain-language and comprehensible consent. But the question remains: How well do these efforts
account for the broader socio-technical power structures inherent in all personal data collection?
Through a series of interviews with advocates for individuals and communities whose lives are often
dramatically affected by data surveillance, this study finds that ongoing attempts to cultivate informed
consent into data-driven systems likely fall short of their stated goals. The current implementations of
ethical digital consent indicate that platform designers and policymakers have an insufficientunderstanding of systems complexity at scale, political accountability, the power dynamics inherent
to organizational politics, and the second- or third-order effects of local-first data strategies. This
study documents the inherent threats and risks for consent-driven digital technologies that current
approaches do not address.

INTRODUCTION
In response to the increased awareness ofmassive and invasive data collection on the
lives of individuals and communities, platform
designers and policymakers alike have sought
to introduce new methods to help people better
understand where their data is going.

THE LIMITS TO DIGITAL CONSENT

One approach to rebalancing power in datapractices is a digital consent system – a new
type of user experience paradigm that
communicates a platform designer’s intent andobligation to the user with a focus ontransparency, ethical governance, and trust.
When someone enters the digital system, they
are provided with a decision tree that allows
them to signal their intent to grant or withhold
consent to data collection. Despite a presumed
common purpose, however, digital consent
systems vary wildly in their execution. A
common implementation is a system-level
permission dialogue that is presented in theinterface when a software application attempts
to access a sensor, personal data, or othersensitive system1. These consent models aretask and context driven – they interrupt a
process the person using the application istrying to perform. Often, declining orwithdrawing consent in these circumstances
prevents the person from completing an action
they are already invested in – and the user
experience has commonly been exploited toincrease the likelihood of a person granting
consent2. Other consent systems, such as theEuropean Union’s General Data Protection
Regulations3, are more monolithic in nature.
These implementations often legally require theplatform designer to ask people to make achoice as soon as they enter an application or
platform. This leaves little room for negotiation1

https://www.apple.com/privacy/control/; https://developer.android.com/
guide/topics/permissions/overview
2
https://uxplanet.org/getting-to-yes-best-practices-for-ios-permissionsdialogs-9d62892142cc
3
https://gdpr-info.eu/

2or the cultivation of a deeper understanding by
the public of the implications of consent.
Current thinking around digital consent does
not sufficiently address the risks inherent inindividual participation in data-heavy digitalplatforms. Even for projects with explicit
missions to empower people and theircommunities – such as Open Source orFederated collectively owned platforms – thepotential for harm remains significant and thecurrent paradigm does little to truly mitigate
threats. Though this study provides only apartial snapshot of a complex and fast-evolving
world, it reveals numerous shortcomings in thedesign of consent-driven digital technologies –highlighting the related threats and risks thatmany current approaches fail to address.

Current thinking arounddigital consent does not
sufficiently address therisks inherent inindividual participationin data-heavy digitalplatforms.

METHODOLOGYThis qualitative research study was conducted
between June and August 2020. The needsfinding interviews were remote and informal,
lasting 45-60 minutes and conducted via video
chat. We spoke with five participants who wereselected based on their experience working
with underrepresented communities within the
United States and United Kingdom. Participants
were selected due to their professional work
intersecting with underrepresented
demographics and digital technology systems.
Of the participants, four identified as female,one identified as male, and the selected group
was racially diverse.
The composition of the participant pool was
informed by the study’s initial focus on
disinformation and the 2020 US election. To
be selected, participants were required to haveon-the-ground perspective adjacent tounderrepresented communities and sensitive
situations, such as people and communities
experiencing systemic injustice, living under
oppressive regimes, or experiencing
residential insecurity.
Participants were approached through a written
invitation and subsequently expressed theirunderstanding and consent to the interview
process as per Simply Secure’s researchpolicies4. All participants received a stipend fortheir time.

1. What are the benefits for both theindividual/community and a data-collecting
organization of data donation/accumulation?
2. Who is most at risk, and how visible arethey within common consent design
paradigms?
3. How does data accumulation complicate thedynamics of these communities?
4. What does consent mean for thesecommunities and situations? Including:a. The socio-technical or political effects ofgiving/granting consent to a digital platformor organization on individuals and theircommunity, andb. The socio-technical or political effects ofan individual withdrawing consent from a
digital platform.
5. How can risks be lessened or de-escalated?
During interviews, the following general
structure was observed:
a. Participant introduction: What is your role
relative to your positioning as an advocate for acommunity?
b. How has the accumulation of data led todangerous examples for communities? Who ismost at risk? Why?
c. What harm minimization strategies have youwitnessed in response to systems of data
accumulation? From the community? Fromadvocates? From platform designers,
organizations, and technology companies?
d. For you, what is the worst case scenario
when it comes to data accumulation in thebrowser?
e. What would make you more optimistic
towards data accumulation?

THE LIMITS TO DIGITAL CONSENT

The study was based on Socio-technical
Security principles. Socio-technical Security isan emerging security research framework that“acknowledges how the interplay between
actors produces emergent threats to participant
communities,” focusing on human relationships
as the basis for a security model, rather thancomputing devices, software applications, or
network connections5.

Interviews explored five key questions:4

https://simplysecure.org/blog/participant-rights
https://www.usenix.org/conference/foci19/presentation/goerzen
5

3

THE LIMITS TO DIGITAL CONSENT

Notably, given that the focus of the research
was solely on issues of consent, this study
did not address any potential information
security or technical implementation issues. In
addition, the statements, impressions, andunderstandings provided by interviewees were
not independently fact-checked – but instead
taken at face value in order to best represent
their perceptions of the risks of ethicalconsent and data collection forunderrepresented communities.

4PARTICIPANT
PROFILESAt the beginning of each interview, participants
were asked to briefly describe themselves
within the context of the study’s focus.
Participant quotes throughout this report areparaphrased and are acknowledged to be an
accurate representation of an individual
participant’s contribution to the report.
Participant #1“I am a criminal defense lawyer, so I review
evidence collected by federal agencies. I also
work with political organizations that areconcerned about surveillance, infiltration, andother forms of digital security. I see both sides:
people who are trying to keep data private, andpeople whose data is working against them.
Some years ago, I started doing activist work.
This intersected with my work as a publicdefender when people became concerned withdigital security. A lot of services and products
went online. There was a proliferation ofdigital tools.”

Participant #3“I am an activist and advocate. I work withinorganizational power structures to instigate forchange. I often ask myself, how do I make surethe people I’m advocating for are safe andgetting the resources that they need? How do I
make safe space for others to come in / takepower / shape governance? I consider our job,collectively, is to decolonize the US
government. Within my work I ask, ‘What does
shifting that power look like?’”
Participant #4“I am a person. I use the Internet. I am acommunity researcher, especially for black
people using the Internet and understanding
how tech affects the marginalized from a
research perspective. I focus oncommunication online and in general – howto communicate safely.”
Participant #5“I work at an anti-harassment organization. We
work to make online space safer for allcommunities. I experienced online abuse andharassment and saw gaps in addressing theissues. I also saw gaps when hearing
marginalized communities. Some years ago, we
started as a campaign, later becoming a
movement, and now we are an establishedcharity. We focus on women and people whoare non-binary – they are most likely to betargeted. We aren’t just concerned with womenas a gender, we consider intersectionality as
well. We do advocacy work with government
and technology companies. We fight for better
platforms and long-term solutions.”

THE LIMITS TO DIGITAL CONSENT

Participant #2“I work in a leadership role at a nonprofit
advocacy organization and have worked inlarge tech organizations before. I work inengagement and communities. I’ve beeninterested in digital privacy issues for a decade.
It’s important among marginalized and diverse
communities, especially pertaining to black andbrown people, women, children, seniors,
ethnicities, people who are Muslim, and more.
Digital privacy training needs to be moreavailable and concrete. I’m interested in theintersection of technology and human rights. I
want to know how to amplify voices, and build
for, not against communities. I’m involved inunderstanding AI, surveillance, and facialrecognition. Those have the potential tosuppress communities – people like me. I also
speak in advocate roles. I live in the US on avisa – I have been for a number of years. I
worry that my work will affect my residency.”

5

KEY FINDINGSFROM INTERVIEWS

THE LIMITS TO DIGITAL CONSENT

Each of the six findings below are supported byresponses from one or more of the participants.
The quotes included at the end of each finding
are paraphrased summaries of key discussion
points and are edited for privacy and clarity.
Alongside the right to withdraw at any time
during the study, each participant had theopportunity to confirm that the quote attributed
to them accurately represents their
participation in the study.

6

Finding #1:
The consent model for techis outdated.
Today’s digital consent frameworks areconceptually derived from institutional or
academic ethics frameworks, yet digitalplatforms pose unique challenges that theseframeworks are not equipped nor currently
designed to address. This study finds that thelack of broader societal protections – such as
the frequency of data
breaches6 andcorresponding lack of
corporate accountability7,or the rise of algorithmic
discrimination8 andinstitutional misuse of data9– has led to an imbalanceof power that significantly
impairs the ability of an individual to givemeaningful consent. As a result, people areforced to consent into a system that does not
foster individual autonomy. Although ethicalconsent systems attempt to fully inform people
about their data relationship with a platform,
the complexity of modern digital systems,
coupled with the cascading consequences ofthe broader network design of each ethicalplatform, ensure that the goal of ethicallysourced and informed consent data collection
is extraordinarily unlikely.

account the self-assessment of individual safety
and self-preservation, which are complex andcontextually regionalized. Further complicating
matters, consent is often deemed to have been
made on behalf of others who appear in aplatform indirectly or as a result of interactingwith a consenting user – yet those individuals
have not been given the opportunity to consent
themselves. Taken together, these issues reveal
a consent model out of step with the goals of
informed participation, power rebalancing, andindividual agency.

#2:
People are forced Participant“People are well aware of
to consent into a what’s happening. They
already consented so theysystem that does
go further down the hole,
not foster
thinking, 'I’m never going
individual autonomy. to be president, with the“What are the ripple effects? People think theirdata is theirs. They own it. We know it’s not.
How is it going to be used? To improve your life
or not? And what about those around you?”
Participant #3:“You can never fix the data problem unless you
fix the political problem.”“We need data sets and we need to allow forcommunities to opt in. Right now it’s justGoogle and Facebook because they have big
enough data sets. But the consent model isdesigned for the individual. If I opt in, I am
consenting on behalf of others – this is
irreconcilable with the current model. This is

THE LIMITS TO DIGITAL CONSENT

The complexities inherent in digital systems, therequirements for ethical informed consent, andthe design practices deployed as a part ofhuman or human-rights centred design
approach are at odds with each other. Current
consent models are almost always designed asa one-size-fits-all approach for the mostcommon use cases to consider, within whichtrust and the lack of trust are framed only fromthe perspective of the platform or researcher.
In practice, these concepts fail to take intothings in my search history,
it’s going to be all in the public.’
As people are storing and creating, holding,selling data, consent is not prioritized. The
companies are asking, how do we do this under
the radar? Privacy policies are long andconfusing on purpose. It’s fake consent: wedon’t realize the implications.”

6.

https://haveibeenpwned.com/PwnedWebsites
https://www.ftc.gov/enforcement/cases-proceedings/refunds/equifaxdata-breach-settlement8
https://ainowinstitute.org/AI_Now_2016_Report.pdf
9https://ainowinstitute.org/litigatingalgorithms.pdf
7.

7

smaller than the social contract but bigger than
the individual.”“I have friends who don’t give me theiraddresses because of their personal threat
models. No consent framework exists todaythat takes this into account.”
Participant #4:“It’s the amount of times you have to ask. The
lack of change logs. ‘We made this better,’ but
no one tells us why the previous one was bad.
Trust is a thing that sounds good. There’s not asensible framework that they don’t have. You
want to be stewards – creating the space forstewardship to be possible.”“There is a lack of oppositional research inthese programs. [Oppositional research shouldbe] based on multiple identities and genders.
They might introduce the best tool to protect
data, but it’s not for someone who doesn’t havea social security number for instance, so it’s not
usable. Lack of trust does not have to bemalicious, it could be self-preserving. When it’s
not built for me, trust is lost.”“Gaining a better understanding requires aninterdisciplinary approach at the start, notthe middle.”

THE LIMITS TO DIGITAL CONSENT

Participant #5:“I had to self-teach around data accumulationand privacy because of [my job]. It’s not easy tounderstand. The whole topic needs to be
demystified. Even the term ‘data accumulation,’
I get it, but having to explain it to someone
else… not so easy. People don’t see it asa priority.”8

Finding #2:
Local-first data storage is
not inherently safer for people
or communities.
The collection and ownership of personal databy an organization creates a significant power
imbalance10. In response, platform designers
may turn to a “local-first” approach to data
storage, in which an individual owns andcontrols their data in an ethical digital system.
This “local is better” assumption is helpful for
mitigating issues related todata abuse or company
control, and is frequently
proposed as a usefultechnique for rebalancing
the power dynamic
between people andorganizations. Yet such a
proposition does notconsider its implications
beyond these immediate
relationships, nor how assigning data
ownership to an individual can amplify power
discrepancies between the individual and othernon-associated parties. In particular, the “localfirst” data ownership paradigm introduces
vulnerability to people who might be the targetof harassment from technically sophisticatedantagonists11 or prejudice from law
enforcement12.

individuals14, generate narratives for statefacilitated oppression15, and overwhelm
judicial institutions.
Participant #1:“With a warrant they can look at your local
device. But a search warrant is just one methodof collecting data for the investigation. They
can get a warrant based on speculation.”“They use services like Cellebrite for extraction.
The report is a map of everything: contacts,browsing history, erotic sites, deleted files, etc.
These types of requests
are routine, so thecompanies that do it are nolonger a specialty. It’s thepush of a button. And they
have so much data. My
suspicion is that they areso overwhelmed with datathat they just ignore thestuff that’s hard. For highprofile cases, there are lotsof resources and they’ll spend more time trying
to crack a phone or digging through messy
data. It’s useful for us to understand how theactual process is being used by an adversary,such as the government. It’s an index key word
search of every file on your machine.”

In a local-firstdata ownershipsystem, the risksof data accumulation
are placed on
the individual.

10.

http://en.collaboratory.de/w/Power_in_the_Age_of_the_Feudal_Internet
https://www.amnesty.org/en/latest/research/2021/07/forensicmethodology-report-how-to-catch-nso-groups-pegasus/
12
https://ainowinstitute.org/AI_Now_2016_Report.pdf
11.

Participant #4:“Risks are multiple, intersectional. What makes
it risky is mitigated by other factors. What are
the things that are risky for people? And designfor that? There are differences between
persons who are undocumented, persons in thesystems but waiting for documentation, andthose with undocumented relatives. But when
we research we study just ‘undocumented.’”THE LIMITS TO DIGITAL CONSENT

In a centralized system, an organization canemploy digital security or legal13 expertise tofight attempts by third parties to access data
beyond their interests or the interests of theconsenting individual. But in a local-first data
ownership system, the risks of data
accumulation are placed on the individual.
Accessible by targeted technical attack, search
warrants, extrajudicial investigation, or theft,
the local-first data storage approach (including
cached data) adds vulnerability and complexity
into people’s lives, and has been successfully
used for years to prosecute targeted“The remote is harder to access. In some ways
it’s harder to get things off a remote server. But
fundamentally, it’s not that different. It’s justharder for them to know what’s there, and it’sharder to control. Unless the provider is helping
them out, which many do.”13

https://www.techdirt.com/articles/20130618/14341223521/googlewithout-ad14
https://www.eff.org/wp/riaa-v-people-five-years-later
15
https://simplysecure.org/resources/techreports/NYC15-MobMsg.pdf

9Finding #3:
Data creation, including thepotential for data creation,is silencing.
The accelerating rate and scope of datacollection, combined with increasing
community awareness and savvy surrounding
the dangers posed by such datasets, producesa chilling effect on those who wish to speak up
and self-advocate but hesitate to do so due tothe associated added risk. In a broader social
context, recent examples of the chilling effectof data collection include the rise in publicawareness of how behavior in social media has
direct consequences for health insurance16 or
financial stability17, as wellas the rising awareness of
personal data purchased
by US armed forces fromapps used by religious
minorities18. Beyond the
public sphere, data
accumulation reduces therange of voices throughthe process of datadeanonymization19 – a factthat the public is becoming
increasingly savvy to.

system will not be subject to unintended social
graphing, deanonymization, or shadow
profiling. Once the data is created andcollected, people are unable to monitor thebroader analysis process their data will be
subjected to. As such, an ethical consent
system remains unsuitable for those who
cannot – or are unwilling to – bear the risk ofbeing subjected to these activities.
Participant #1:“A lot of people are more and more cognizantof their data trail. They have strategies liketurning off the phone, having two phones(‘protest phone’), or leaving their phone in theother room.”
Participant #2:“The new laws in the US
like Muslim bans (and othercommunities) havetargeted me. ICE[Immigration and Customs
Enforcement Agency, afederal US Government
agency] can have accessto my social media
accounts at any time. Are
those programs active? It’s
unclear to me. But I thinkthey have access to snoop
on me and collect my data.
And it’s dangerous – I
might push the line andthey can revoke my status.
If I were undocumented, that would make it soscary. [People who are undocumented] can’t be
vocal or stand up for themselves. That’s thepower that data can have.”

The chilling effectof data accumulation
is cascading. Data
accumulation
perpetuates
inequalities throughthe ripple effect ofinterpersonal
relationships thatcan be inferred or
measured via data.

THE LIMITS TO DIGITAL CONSENT

The chilling effect of data
accumulation is cascading.
Data accumulation
perpetuates inequalities
through the ripple effect of interpersonal
relationships that can be inferred or measured
via data. Well-known examples of this include
Facebook’s efforts to build shadow-profiles on
individuals20 beyond the company’s ecosystem
by using data accumulated from invasive
collection activities of the company’s user base.
Ethical data collection and processing has notsatisfactorily provided meaningful assurance or
technical solutions to ensure that such a16

http://www.idpublications.org/wp-content/uploads/2014/09/ANALYSISOF-INSURANCE-UNDERWRITING-USING-SOCIAL-MEDIA-NETWORKINGDATA.pdf
17
https://www.researchgate.net/publication/
272300274_Credit_Scoring_with_Social_Network_Data18
https://www.vice.com/en/article/jgqm5x/us-military-location-dataxmode-locate-x

10“The advancement of trolls, in the early days of
misinformation and harassment led to distrustin the platform (Twitter), around 2012. Whenthat happened, the trust in the platform wasjaded. You couldn’t open yourself up to the19

https://zviewcontent.cgi?article=1016&context=hightechevents
https://www.europarl.europa.eu/resources/library/media/
20180524RES04208/20180524RES04208.pdf
20

place. Those problems have gone on to be big
problems elsewhere too.““The vulnerabilities are key issues. We knowsome of them and we don’t know some ofthem. We are most vulnerable when we don’tknow them. Like for Clearview AI: we are sovulnerable. I am in the small part of the
population that knows about them. Their
practices are hidden, below the radar.”THE LIMITS TO DIGITAL CONSENT

11

Finding #4:
Everyone – not just members ofunderrepresented communities –is at risk.
The current industry framing of privacy andnetwork threats is heavily influenced by colonial
bias, drawing the majority of its guidance andframing from “empirical evidence on Westernbased, white, and middle-classdemographics.”21 These shortcomings create
blindspots that allow threats to thrive, affecting
entire communities who fall outside of thisnarrow definition. Interviewees described theoften unexplored, or downplayed, risks of data
creation and collection, noting that data
footprints generate vulnerability for almost
everyone. Simplistic mechanisms of opt in/out
consent, the challenge ofunderstanding datacollection within thecontext of complex
systems (as described in
Finding #1), and the loss ofcontrol over the collected
data directly threaten
marginalized communities,
but also pose risks to thegeneral public (in part
because one can neverfully anticipate a transitionof an individual orcommunity into adestabilized or
marginalized situation). In order to mitigate thebroader, society-wide risks associated withdata collection and consent, participants
advocated both for a deeper understanding of
the institutional and colonial biases inherent inprivacy studies, as well as for platform
designers to develop sensitivity to this transitionof users or communities into precarious ordestabilized situations.22

problematic. Current data collection can – andalmost assuredly will – be used to feed futuretechnologies whose risks are impossible atpresent to understand, predict, or mitigate.
Time and again, even well-intentioned or
purportedly neutral datasets have produced
disastrous impacts. Early facial recognition
datasets, for example, were later revealed asrefined tools for perpetuating racial inequities,
enabling authoritarianism, and training machinelearning or algorithmic governance systems23.
Participant #1:“Keystroke logging [is the worst case scenario
for data accumulation in a web browser]. In
some sense, it’s the increase in surface area.
Activist/political perspective: crime isideological. Site history to prove terrorism.
52,000 antifa search hits or on the harddrive.
Maybe it’s not even usedas evidence, it’s just said toa judge. Maybe it doesn’tmean anything, but it adds
to the narrative. They are
just playing on prejudicesand stereotypes andexpanding on the bad guy
persona.”

THE LIMITS TO DIGITAL CONSENT

Current datacollection can – andalmost assuredly
will – be used tofeed futuretechnologies whose
risks are impossible “Social media posts could
be used for any purpose. If
at present toposts a meme
understand, predict, someonethat says “focus” and you
or mitigate.
like it, they can say, ‘you

Future use of collected data is particularly

21

https://journals.sagepub.com/doi/abs/10.1177/1527476418806092
https://cdn.ttc.io/s/tacticaltech.org/smartphone-as-lifeline.pdf
23
https://news.bloomberglaw.com/privacy-and-data-security/dating-siteprofiles-capture-prompts-privacy-violation-lawsuit
22

12

were focusing on thecrime!’ It’s building narratives from anything inthe social graph. There’s nothing stoppingthem. They’ll use anything.
It could be something that’s not at all
incriminating. Like a picture of food or friends:‘You took a selfie? So then you can’t be injuredor whatever.’”“It could also be something related toimmigration: a person wants to reunite with
their deported spouse. Yet, the agent sees

pictures of that person going out, having fun.
Then they say, ‘Oh, you don’t want to reunitewith your spouse? You’re going out and having
drinks?’ That can be used against you.
It’s difficult to say you have no associations –everyone is connected. Those connections are
enforced by the data out here.”
Participant #2:“It’s obvious, we are going to find out that a lot
more people are at risk than we thought. With
facial recognition, AI, etc., and the inequities inthe black and brown communities. Data
collected on them is used against them.
For immigrants, undocumented immigrants,
and visa holders, so much of their data can becalled up.”“For women, it’s doxxing and women are
suffering. Communities, genders, ethnicities …
Ethnicities being torn apart online.”
Participant #4:“Once data collection became a businesspeople didn’t know how to stop it. Data is more
important than the product. At the same time,
when people advise you to get off Facebook, I
think ‘where would you like to migrate themillions of people who use Facebook as theinternet? And you insult them at the sametime?’ I don’t actively use Facebook, but I am
not going to delete my account. I have family
members around the world. That is theplatform that they can use. My great aunt is notgoing to figure out another platform.”

THE LIMITS TO DIGITAL CONSENT

Participant #5:“Everyone is at risk. There is a knowledge gap
online. There is a lack of understanding how
platforms work, digital citizenship. It’s acrossthe board. Men don’t realize the way they use
the platform could perpetuate harm. It’s making
the situation worse with violence against
women. This is completely under-researched
and the social biases are apparent. Allyship is
important for tech justice. Not just making theplatform safer, but for other people to know
what’s going on.”13

Finding #5:
Ethical platform designers must
consider themselves as thepotential bad actor.
While those who demand design justice andethical platform development are wellintentioned in their pursuit of empowering
people and communities, the reality is many
recent dangerous technological advances
started out just as well-intentioned before
resulting in systemic abuse or worse.
Institutional change beyond
the control of a platform
designer, and sometimes
occurring far after a userbase has given consent,
can easily lead to theintroduction of situations
ripe for exploitation. It
simply is not possible foreven the best-intentioned
designer to accurately
foresee changes inpartnerships or leadership
through whichcarelessness or ulterior
motives may poison aplatform’s original values.

When building a digital consent system, thepotential for weaponized design26 should be
examined by the team responsible for
implementing the consent system. This could
take the form of a conceptual and governanceintervention through both participatory design
with community members and criticalengagement with practitioners familiar with
Socio-technical Security and threat modelling
practices (e.g., the Trike security framework27)
to ensure that a digital consent system is not
implicated in a designed system that harms
users while performing exactly as designed.

Participant #2:
Participants“Talented tech-savvy
criticized thepeople are getting all data
mistaken optimistic about a woman, and using
it in ways that can beassumption that
deadly.”institutional
are eager to build
cultures that govern “Wetools without thinking ofcollected data
the implications. Once weeither share or will are done with Covid, what
happens to the data andmaintain thetechnology, will people stillpolitical and social use it? Once data andtechnologies are out there,
values held by
it’s out there for good.
a team of
There’s no going back.
platform designers. People create things and

THE LIMITS TO DIGITAL CONSENT

Both directly and indirectly,
participants criticized themistaken optimistic
assumption that institutional cultures thatgovern collected data either share or will
maintain the political and social values held bya team of platform designers. For digitalconsent, the belief in the institution is often
paired with design ethics, preventing platform
designers and policymakers from deeplyinterrogating the impact of their work24.
Institutional alignment is difficult to measure,and training in digital ethics can often fallshort of expected or desired organizational
cultural change25.

24

https://tv.undersco.re/videos/watch/f769b6e4-d992-4a73-bac1b05ee6116368
25
https://people.engr.ncsu.edu/ermurph3/papers/fse18nier.pdf
26
https://newdesigncongress.org/en/pub/on-weaponised-design27
https://www.octotrike.org/

14

say, ‘I never imagined it
would be used for this.’
The trickle effects are scalable. Scary things
like the Facebook scroll that destroyed theattention span of the population. It was neverintended that way. We are using thesetechnologies in ways we never imagined. There
aren’t enough people and guards to set up for
limitations and understanding how to protect
those people.”
Participant #1:“There is a conflict with the business model ofcompanies. It’s antithetical to how they make

money. Google’s business is the collecting,forming, and selling model. Google is just an
example, there are a million others. That’s thegame. It’s irreconcilable. We have toacknowledge that exists. It’s hard to deal withand grapple with. Being aware of that is going
to be useful for a minimization analysis. It’s notjust what you think you’re doing that’s being
collected. It’s not just you doing bad things thatcan be used against you. It’s everything. Whereyou were, where you weren’t. Your phone wasturned off, turned on. Anything.”
Participant #3:“Bad stewardship of data is a huge problem. It
doesn’t matter how you address this in the
moment, without solving for data protection,
the political context or motivations will change.
You can be sure of that. You can encrypt orotherwise try to protect it, but it is too tempting
to use datasets beyond their intended purpose.
Power shapes the usability of data, not just thedata itself.”

differential representation – yet everyone needs
a PhD to enter. The thought is that research is
not worthy unless you have a PhD. How will
your methods be for people who don’t havehigh school diplomas?”
Participant #5:“The mainstream tech companies are like lifeand death for some people. They use tech fortheir sexual expression so they have to becareful depending on their situation. They
cannot accept dangerous data accumulation –there are massive repercussions. That’s where
the alternative tech comes in. My worry is, big
tech will buy them all out. What is going tohappen in five years’ time? Like Fitbit? There isno incentive for tech companies to do the rightthing. We need a regulator.”“What companies do internally matters when
they are trying to engage with
underrepresented communities. Why should
these communities trust these governancemodels when they can’t govern themselveswith community responsibility?”

THE LIMITS TO DIGITAL CONSENT

Participant #4:“Researchers do not respect people enough.
I’ve learned that over the past decade.
Research does not look like the actual spaces.
The research community should be apologizing.
[The labels] ‘Alt-left’ and ‘alt-right’ areoffensive. It leads to ‘cancel culture’ and antivaccine beliefs. You do not have black
researchers, researchers of color, multiple
generations of the Asian diaspora. Has any
researcher talked to actual users? What about
popular research methods for research withyounger people? How are you going to do thiswork if the generation has not passed the ageof majority? How are you doing it so that it isgood and kind? I don’t think it’s responsible.
Every study should be opt-in, regulated. At thispoint, there may be 16 different centers, alldoing the same research.”“I often hear about efforts to incorporate
15

Finding #6
Participants are overwhelmed byboth the potential for harm and theindifference of decision-makers.
Underlying all participant interviews was arecurring theme. They – along with thecommunities they represent – are intimidated
by past and current issues, and paralyzed by
the scale and potential of harm, as well as thereal or perceived institutional resistance bytechnology companies and legislators to tackle
these problems. Participants repeatedly
expressed skepticism over the successful
creation of a new platform capable of true
ethical data collection and custodianship – evenone operated by a recognizable stakeholder
with a reputation of advocating on behalf ofindividuals and communities.
Participant #1:“Accumulation of data is a nightmare. It makes
everything so difficult. For me, it makes it
impossible to know the full scope of theevidence. The massive
amounts of data makes it
difficult to form a defense.
I don’t know what they’reusing against you. Before,
you could assume they
didn’t have evidence, now
it’s flipped, you have toassume that they haveevidence. Most lawyers
don’t have tools or time.
There’s a huge processing
power imbalance,
combined with ignorance
and fear. The prospect of
putting that in front of ajury, it’s a rough game. They are over-collecting
data. The government can’t deal with it all. And
it helps them more – all presumptions are in
their favor. It completely overwhelms thedefense; it’s like looking for needles in ahaystack. The more they collect, the less I can
do my job. They are not required to tell you
what they want to use. I have a duty to lookthrough all the data. Hardly anyone has thecapacity to do this work. They’ll use anything toconvict.”
Participant #2:“I’m timid to suggest benefits of data donation.
Overall it depends on regulations and policies. I
know not all companies are awful. If there areregulations and if organizations are stipulated, it
has the potential for greatness. But I see toomuch harm.”
Participant #3:“When I participate in health research, myimmediate fear is legal protections andtechniques for anonymizing data. I worry about
insurance companies – how anonymized isanonymization? My health profile is very
unique. I want to believe in the power of data
research. I am a short-term pessimist, longterm optimist.”
Participant #4:“There is a difference between the need forsystemic change and constantly reinventing
and making things that don’t work. That iswhere intersectional
intervention from the startand clear communication
comes in. Safetymoderation, for example, is
now a gig [unpaid
community moderation].
Email is another example,
where you are
overwhelmed byadministering your inbox.
You have to spend yourtime making sure you are
unsubscribed.”

THE LIMITS TO DIGITAL CONSENT

Participants
repeatedly expressed
skepticism over the
successful creationof a new platform
capable of true
ethical datacollection andcustodianship.

16“People are gearing up –trying to figure out what will and what will notwork. So much is on the fly, in the moment.
The problem is not that it’s reactive. Advocatesoften yell for help, but don’t get listened to,
don’t get what they need to help mitigate. Helpfrom technology vendors is often paternalistic.”
Participant #5:“Because violence against women is acontinuum. It started offline, continued online.
Facebook started as a way to rate women. With
the murder of George Floyd, people just started
thinking ‘maybe we should be stop being sowhite and so male.’ Twitter Voice wasintroduced recently, and women could havetold you how it could be used against them.
Although the team who managed it was black,when you ask if the team was diverse? You getcrickets. That is the problem.”

THE LIMITS TO DIGITAL CONSENT

17

Conclusion

This study engaged five participants as
community representatives from the UnitedStates and the United Kingdom to produce
qualitative research examining the efforts to
produce digital consent systems that seek torebalance power between people andcommunities, and data-collecting organizations.
These interviews were combined with a reviewof the historical and current issues related todata collection in an effort to critically examine
whether digitally-facilitated consent actually
represents an individual’s informed
understanding of the implication of thatconsent – both for themselves, their own
personal relationships, and their wider
communities. This study concludes that efforts
to refine digital consent with these objectives inmind have not succeeded.

THE LIMITS TO DIGITAL CONSENT

Data accumulation has great power over a
person’s agency, their relationships, and thecommunities within which they operate. The
associated harms are therefore pervasive andreach almost every human being. Current andproposed implementations of platformfacilitated data collection employ a trustpromotion or informed consent model, where
people make a decision about thetrustworthiness of a platform and consent tothat platform accessing or generating data
about them. Digital consent is derived fromdecades-old evolutions in participant advocacy
in social and health sciences,28 and it is often
combined with dominant human-centered
design paradigms – such as designing for
obviousness via simple language andfrictionless interfaces – to provide coherent
interfaces when asking for consent.29 Ethicaldigital consent further builds upon these28

https://kantarainitiative.org/confluence/download/attachments/
101811330/MEF-whitepaper-understanding-digital-consent.pdf?api=v2,
https://openscholarship.wustl.edu/cgi/
viewcontent.cgi?article=6460&context=law_lawreview

18This study concludes
that efforts to refine
digital consent have notsucceeded.
While consent models
serve the humanities
well, they are
unsuitable fortechnology design.

Practitioners must
examine the systemic
shortcomings of digitalconsent and commit to an
ongoing iteration ofconsent and data
governance within
platforms. Platform
designers andpolicymakers must notassume that collection
is safe, and in turn
must design data storage
systems accordingly.

practices, employing frameworks such as the
Design Justice Network’s Network Principles30
or Society-centered design31 in an attempt tonegotiate and rebalance power within digitalsystems. Yet while these consent models serve
the humanities well, they are unsuitable fortechnology design.
In general, today’s popular digital consent
paradigms fail to address the issues identified
by the participants in this study. The gaps
between the goals of digital consent
implementation and the reality of outcomes forcommunities and individuals were described byboth the interviewees as well as the study’s
literature review. Each participant expressed
resentment or frustration towards institutions,driven by specific regionalized examples ofharm caused by digital systems. Common toeach participant’s experience was that material
risks or poor outcomes experienced byparticipants or the communities they represent
are amplified by digital platforms, and that
digital consent failed to account for these
realities. In some cases, participant resentment
was driven by what they considered to be
willful ignorance of the ramifications of data
accumulation by those who design thesesystems and the platforms and pathways forbringing people into these systems. In others,
resentment was drawn from direct experienceof institutional marginalization in either aprofessional or personal context. Many
communities that our participants either
identify with or have advocated for have been
subject to intergenerational institutional
violence and rightly continue to combine these
historical impacts with contemporary
experiences in formulating their own models for
self-preservation and safety.
THE LIMITS TO DIGITAL CONSENT

Going forward, practitioners must examine thesystemic shortcomings of digital consent andcommit to an ongoing iteration of consent anddata governance within platforms. Platform
designers and policymakers must not assume
that collection is safe, and in turn must design

29

Krug, Steven. Don’t Make Me Think, Revisited: A Common Sense
Approach to Web Usability, New Riders, 2014.
30
https://designjustice.org/read-the-principles
31
https://societycentered.design/

19

data storage systems accordingly. This includes
on-device local storage. A common assumption
inherent to designed systems of consent anddata accumulation/ownership is that theorganization the designer represents maintains
the same values around privacy and trust, andthat these values will be maintained indefinitely(or at least as long as the data remains useful).
Yet in practice the political framework for thosein custodial control of datasets can change
drastically and quickly – posing a risk to all
whose data was collected.
Regarding organizational change, data
governance often fails to anticipate secondand third-order effects, such as abuse or harm
facilitated by a partner organization andassisted by seemingly unrelated collected data,
or institutional racism drawn from conclusions
within a collected data set. These detrimental
effects can also stem from a data breach orother action taken by someone outside thecollecting institution. They all must be
accounted for.

THE LIMITS TO DIGITAL CONSENT

Yet regardless of designer intent, the public is
gaining a better understanding of theconsequences of the often false assumption ofaligned values between designer andinstitution, as well as the unintended effects ofdataset abuse during organizational or societal
political shifts. This expansion of knowledge
brings an increasing expectation of
accountability on behalf of data custodians. For
platform designers, meeting these heightened
expectations requires careful assessment oflocal-first and centralized data governancemodels. Considerations should also be taken to
research and develop mitigation strategies forproblems of data accumulation within the
person’s device and beyond, including custodial
ownership of the larger dataset over time.
Whether seeking to deploy private analytics ordevelop partnerships for ethnographic
research, platform-based personal datacollection and analysis comes with a highdegree of fast-moving risk. Platform designers
must therefore develop a set of sensitive anddirect plans for navigating unexpectedscenarios in which they or an associated20

organization with platform access suffer apublic collapse of social trust. Notably, thesescenarios will likely unfold outside of theplatform designer’s control, restricting theirability at that time to deploy crisis management
strategies or modify their system for informed
consent. But that does not mean that theycannot be designed for. And they should. That’s
why a platform designer who positions themself
as an advocate for internet privacy and selfdetermination must utilize an expanded systems
approach to consent, take on a broader
understanding of the risks of data
accumulation, and be unafraid to employ a
combined design/communication strategy that
not only acknowledges but embraces thelikelihood of digital threats in order to mitigate
them and obtain consent.

ABOUT
This study was supported by Reset, an initiative engaged in programmatic work on technology anddemocracy. Reset seeks to change the way the internet enables the spread of news and information
so that it serves the public good over corporate and political interests – ensuring tech companies
once again work for democracy rather than against it.
The research behind this study was conducted, in part, to help support and inform consent work
related to Mozilla Rally.

The New Design Congress is a Berlin-based research organizationthat recognizes all infrastructure as expressions of power, andsees interfaces and technologies as social, economic, political andecological accelerants. The organization is a fiscally sponsored
project of Simply Secure.
Simply Secure is a women-led nonprofit working to transform thecurrent design paradigm in tech by centering the needs of people
as the guiding principle. Through educational programs, resource
building, and open research partnerships, we are decentralizing
the power of digital design by increasing the number of people
who can perform the essential tasks that shape our collective
online future. We envision a world where everyone – particularly
the most systematically underserved – has the knowledge,
network, and digital tools needed to enrich their lives.

THE LIMITS TO DIGITAL CONSENT

Copyright © 2021 Simply Secure
1st Edition, September 2021
newdesigncongress.org
simplysecure.org

21