“Okay, whatever”: An Evaluation of Cookie Consent Interfaces

Hana Habib Megan Li

Carnegie Mellon University Harvey Mudd College

Pittsburgh, PA, USA Claremont, CA, USA

htq@cs.cmu.edu mlli@hmc.edu

ABSTRACT

Many websites have added cookie consent interfaces to meet regulatory consent requirements. While prior work has demonstrated that

they often use dark patterns — design techniques that lead users to

less privacy-protective options — other usability aspects of these

interfaces have been less explored. This study contributes a comprehensive, two-stage usability assessment of cookie consent interfaces.

We frst inspected 191 consent interfaces against fve dark pattern

heuristics and identifed design choices that may impact usability.

We then conducted a 1,109-participant online between-subjects experiment exploring the usability impact of seven design parameters.

Participants were exposed to one of 12 consent interface variants

during a shopping task on a prototype e-commerce website and

answered a survey about their experience. Our fndings suggest

that a fully-blocking consent interface with in-line cookie options

accompanied by a persistent button enabling users to later change

their consent decision best meets several design objectives.

CCS CONCEPTS

• Security and privacy → Usability in security and privacy; •

Social and professional topics → Computing / technology policy;

• Human-centered computing → Empirical studies in HCI.

KEYWORDS

cookie consent, GDPR, privacy choice, privacy notice

ACM Reference Format:

Hana Habib, Megan Li, Ellie Young, and Lorrie Faith Cranor. 2022. “Okay,

whatever”: An Evaluation of Cookie Consent Interfaces. In CHI Conference

on Human Factors in Computing Systems (CHI ’22), April 29-May 5, 2022, New

Orleans, LA, USA. ACM, New York, NY, USA, 27 pages. https://doi.org/10.

1145/3491102.3501985

1 INTRODUCTION

When consumers visit a website for the frst time, their experience

is often interrupted with an interface related to the use of cookies.

These interfaces, used to meet legal requirements for notice and

consent to data collection and processing under the EU’s ePrivacy

Directive (EPD) and General Data Protection Regulation (GDPR)

and the California Consumer Privacy Act (CCPA) [7, 13, 32], contribute to the considerable user burden involved in managing digital

privacy. While existing privacy regulation stipulates that cookie

This work is licensed under a Creative Commons Attribution International

4.0 License.

CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA

© 2022 Copyright held by the owner/author(s).

ACM ISBN 978-1-4503-9157-3/22/04.

https://doi.org/10.1145/3491102.3501985

Ellie Young Lorrie Faith Cranor

New College of Florida Carnegie Mellon University

Sarasota, FL, USA Pittsburgh, PA, USA

eleanor.young18@ncf.edu lorrie@cmu.edu

consent interfaces must be usable, there are no existing standards

for usable consent interfaces. As a result, organizations use a wide

range of design practices in their implementations, some of which

have been highlighted in prior work as dark patterns [10, 29]. Dark

patterns within cookie consent interfaces are design practices that

nudge users toward less privacy-protective options and could lead

to users unknowingly consenting to data collection or failing to

exercise their preferred privacy choices. Beyond dark patterns, it

is important to consider other usability aspects of cookie consent

interfaces, such as user awareness and comprehension of choices,

as interfaces with poor usability could cause privacy fatigue in

users, described as “the tendency of consumers to disclose greater

information over time when using more complex and less-usable

privacy controls” [17].

We conducted a comprehensive two-stage evaluation of cookie

consent interfaces to more holistically assess the usability impact

of common interface designs beyond the presence of dark patterns. Our study contributes a comprehensive defnition of usability

for the context of consent interfaces and provides a better understanding of design choices that improve the consent experience.

Our results can be leveraged by websites, self-regulatory organizations, and Consent Management Platforms (CMPs) to improve

consent interfaces, as well as by legislators and regulators who

defne and enforce usable consent interface requirements.

We based our evaluation of cookie consent interfaces on those implemented through CMPs. These services have emerged to help organizations manage consent fows on their websites and apps. Many

CMPs rely on the Transparency and Consent Framework (TCF) developed by IAB Europe, a self-regulatory group for the advertising

industry, to ensure that the design choices available to their clients

for their consent interface implementations are in compliance with

regulatory requirements and follow industry standards[14].According to a report by the ad-tech company Kevel, approximately 52% of

the top 10,000 US websites that serve ads have a CMP-implemented

cookie consent interface, with six CMPs found frequently on IAB

member websites in the third quarter of 2021 and a single CMP

capturing the majority of the market share [18]. While some design

aspects of the consent interface are standardized by the TCF for each

CMP, there are others that organizations can choose to customize

for their particular website or app. Considering the prevalence of

CMPs and consolidation of the space into a handful of services,

improvements in the usability of CMP-implemented cookie consent

interfaces would have widespread impact.

Our research builds on prior research in this space, which primarily identifed dark patterns within cookie consent interfaces and

explored their impact on user decision-making [10, 29, 39, 42]. We

expand on this work by investigating additional design parameters

for consent interfaces and more deeply exploring users’ perceptions of diferent consent interface designs. First, we conducted
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA

an inspection-based evaluation of 191 cookie consent interfaces

implemented through fve major CMP services, using an approach

informed by three standard HCI methods: heuristic evaluation,

cognitive walkthrough, and independent expert review [45]. We

evaluated each interface for several dark pattern heuristics identifed in prior work [24, 29, 39] as well as other potential usability

barriers. Our inspection-based evaluation yielded a list of design

parameters that appear to be customizable through CMPs. In the

second stage of our evaluation, we drew on our fndings from our

inspection-based evaluation to further investigate seven design

parameters (listed in Section 4.1) that may impact the usability of

consent interfaces. We conducted a between-subjects online experiment with 1,109 participants to evaluate the usability of 12 consent

interface design variants. Participants in the experiment were asked

to complete a shopping task on a prototype of a fctitious retail

website where they encountered one of the consent interface design

variants. Following task completion, participants answered survey

questions related to the usability of the consent interface.

Our analysis of participants’ survey responses and interactions

with the prototype website highlighted signifcant usability diferences between the design variants tested. We found that prominence of the consent interface on the website impacted participants’

awareness of available choices and their ability to make their preferred consent decision. Our results also indicate that the absence

of in-line cookie options within the initial screen of the interface

appeared to have led participants to consent to all cookies and

reduced participants’ investment in their consent decision. These

results highlight design choices that organizations could make, or

that could be incorporated into a standardization efort, that would

improve the usability of cookie consent interfaces.

2 BACKGROUND & RELATED WORK

In this section we provide an overview of the regulatory requirements that are applicable to cookie consent interfaces. We then

describe prior work that explored the prevalence and impact of

dark patterns within such interfaces. Next, we identify seven components of usability that pertain to cookie consent interfaces. Last,

we describe alternative mechanisms for capturing and enforcing

users’ consent decisions.

2.1 Regulatory Requirements

Cookie consent interfacesstarted appearing on websites after a 2009

amendment to the EU’s ePrivacy Directive (EPD) which aimed to address issues related to the confdentiality of digital communication

and tracking on the internet [13]. Consent interfaces became more

widespread after the European Union’s General Data Protection

Regulation (GDPR), a comprehensive privacy legislation having

global impact, went into efect in May 2018 [5]. Under GDPR, organizations must request consumers’ consent to process personal data

beyond what is required for fulflling a legitimate business interest.

Under Article 4, consumers must provide their consent through

a clear and afrmative action [13]. Agreement cannot be incorporated into default settings or pre-checked boxes [30]. Furthermore,

Article 12 specifes that consent must be requested through an

easily accessible form with simple and clear language. Moreover,

Article 7 stipulates that consumers must be able to withdraw their

Habib et al.

consent at any time [13]. In the wake of its enactment, the GDPR

has inspired several other national privacy laws, including those

in Canada, Japan, South Korea, Colombia, Argentina, and South

Africa [38].

The GDPR also laid the groundwork for the California Consumer

Privacy Act (CCPA), which went into efect in 2020. The California

state law requires certain companies to provide notice to consumers

related to data collection. Among other privacy rights, it grants

California residents the right to opt out of having their personal

data sold to third parties, for example, for marketing purposes [32].

The California Privacy Rights and Enforcement Act (CPRA), which

will go into efect in 2023, builds upon the CCPA. The law provides

additional privacy rights to California consumers, including a right

to opt out of a business using sensitive personal information and to

opt out of the sharing of information with third parties (in addition

to selling). Furthermore, the CPRA explicitly prohibits the use of

dark design patterns in consent interfaces [31].

While it may be too soon to evaluate organizations’ response to

CCPA requirements, there is evidence of mixed compliance with the

GDPR’s consent requirements. A measurement study by Degeling

et al. found a 16% increase in the display of cookie consent interfaces

among the 6,579 websites evaluated after the GDPR went into efect

in May 2018 [5]. However, past research has identifed several

issues with how these interfaces are implemented. In analysis of

nearly 30k websites, Bollinger et al. found that 36.4% incorrectly

categorized at least one cookie and 21.3% placed cookies when the

user explicitly opted out [2]. Furthermore, Matte et al. observed that

even websites with CMP-implemented consent interfaces following

the IAB’s Transparency and Consent Framework could mishandle

users’ consent decisions [25].

2.2 Dark Patterns in Consent Interfaces

Several studies evaluating consent interfaces for dark patterns have

found that current implementations may also be violating the intent

of the GDPR and other regulation. Utz et al. conducted a feld

study exploring the impact of four design variables, fnding that

position of the interface, choices ofered, nudging patterns, and

language used in the interface text impact users’ interactions with

the interface [42]. Soe et al. manually evaluated cookie consent

notices on 300 online news outlets based on 13 heuristics, providing

a reference for several types of common dark patterns specifc to

consent notices [39]. Drawing from existing literature in design,

law, and privacy, Gray et al. performed an interaction criticism

of consent banners from four perspectives: the designer’s intent,

designed UI, end-user, and potential societal impact. By reviewing

recordings from over 50 websites, they identifed diferent stages

of the consent task fow and common design choices that raised

ethical dilemmas warranting additional dialogue [10]. Nouwens et

al. quantifed the impact of consent interface design choicesthrough

an online experiment, fnding that the display of granular options

within an initial cookie consent prompt decreased the probability

of a user consenting to all cookies, while removing a “reject all”

button increased the probability of consent [29].

The impact of other types of dark patterns have not yet been

empirically explored in prior work. One such pattern has been

termed “confrmshaming’ which is wording that guilts or shames
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

“Okay, whatever”: An Evaluation of Cookie Consent Interfaces

users about a decision presented to them [3]. Implementations of

confrmshaming could exploit a cognitive bias called loss aversion.

This well-studied bias explains how people prefer to avoid a loss

compared to gaining something equivalent [37]. In the context of

cookie consent, loss aversion framed text could highlight negative

outcomes of not accepting optional cookies, which may create

a nudging efect towards accepting all cookies and impact other

aspects of usability.

A small number of otherstudies have focused on usability aspects

of interfaces related to cookies that extend beyond dark patterns.

Kulyk et al. found that the text of cookie disclaimers had little

impact on participants’ sentiment and decision to continue using

the website [20]. Additionally, Machuletz and Böhme explored the

impact of the number of consent options present on the efort

required to use the interface and participants’ sentiments, fnding

minimal diference between one and three options [23].

This study complements the growing body of research related to

cookie consent interfaces. Our inspection-based evaluation builds

on Soe et al.’s [39] review of consent notices on news websites by

applying a similar set of dark pattern heuristics to cookie consent

notices implemented through CMPs. Complementing prior user

studies [20, 23, 29, 42], our online experiment explores the impact

of additional design parameters, specifcally the formatting of interface text, presence of loss aversion, text used in button options,

layout of cookie options presented beyond the initial screen of the

consent interface, and processfor changing consent decisions. More

signifcantly, our study evaluates the impact of design parameters

against a comprehensive defnition of usability that includes aspects

that have not been deeply explored in prior work in the context of

consent interfaces.

2.3 Defning Usability for Consent Interfaces

A limitation of prior work related to the consent interface design is

that usability is primarily framed through the lens of dark patterns.

However, the privacy and usable design literature defnes usability

through a variety of other aspects related to how people interact

with and perceive an interface. Here we provide an overview of

diferent defnitions of usability, primarily selected from textbooks

in HCI and privacy. We observed that components of these defnitions could be organized into seven groups, providing a structure

for our usability assessment of consent interfaces.

Feng et al. propose a defnition of usability specifc to usable

privacy choice interactions and describe the concept of meaningful

privacy choices that “extend beyond traditional usability considerations to include several facets that are more specifcally tied to

supporting users in making privacy decisions that capture their

true privacy preferences” [8]. Schaub and Cranor emphasize that

meeting regulatory requirements is not enough and describe four

components they consider required for efective privacy interfaces:

fndability, understandability, usability, and usefulness [35]. Other

usability defnitions are applicable to interfaces beyond those related to consent. The International Organization for Standardization

(ISO) 9241 considers usability as the efectiveness, efciency, and

satisfaction with which users achieve specifed goals in particular

environments [16]. In her defnition, Quesenbery describes the “5

Es” of a usable interface by extending ISO’s defnition of usability

CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA

to include error tolerance and ease of learning [33]. Nielsen defnes

usability through fve “quality components” that assess how easy

interfaces are to use [28]. Morville’s UX Honeycomb is commonly

referred to in web design and explains six qualities of the user

experience that must be addressed [27].

Table 1 highlights the overlap between these six usability defnitions, as well as where they difer. In grouping the components

of these defnitions we identifed seven aspects pertaining to the

usability of consent interfaces. We defne these usability aspects as:

(1) User needs: whether a privacy choice interface addresses

the intended users’ privacy needs in a particular privacy

choice context. Also includes accuracy and completeness of

the interface in addressing these needs.

(2) User ability & efort: whether a privacy choice interface

allows the intended users to accomplish a particular privacy

goal and with minimal efort.

(3) User awareness: whether the intended users are aware that

a particular privacy choice exists within a privacy choice

interface, and if they are able to fnd it.

(4) User comprehension: whether the intended users understand what a particular privacy choice does and the implications of their decisions.

(5) User sentiment: whether the intended users are satisfed

with a privacy choice interface and options it provides. This

includes whether users have faith that the privacy choice

will be honored.

(6) Decision reversal: whether a privacy choice interface allows the intended users to correct an error or change their

decision. This also includes the efort required to do so.

(7) Nudging patterns: whether the design of a privacy choice

interface leads the intended users to select certain choices in

the interface over others (including dark patterns that lead

users to less privacy-protective options).

2.4 Automated Consent Mechanisms

As a reaction to the ubiquity of consent interfaces on websites

and their typical poor usability, automated consent mechanisms

have been proposed to ease the user burden involved in making

consent decisions. For example, browser extensions are now available that block consent interfaces from appearing on websites and

automatically respond to interface prompts on behalf of the user

(e.g., [19, 26, 29]). A proposed scheme by the UK Information Commissioner’s Ofce (ICO) would allow users to set their cookie preferences in their web browsers and have those preferences be automatically communicated to the websites they visit [40], similar to

the Do Not Track (DNT) standard [44]. Client-side consent mechanisms that directly manage cookies, rather than relying on websites’

handling of consent decisions, can mitigate issues related to how

websites handle consent decisions. For example, the CookieBlock

browser extension categorizes cookies and automatically removes

those that the user has not consented to [2].

While automated consent mechanisms have the potential to

address both the user burden as well as implementation issues associated with consent interfaces, we expect that consent interfaces

requiring user interaction will continue to proliferate for some

time to come. Furthermore, as there is evidence to support that
 

 

Related to Related to Related to Related to Related to Related to Related to

user needs ability & awareness comprehension sentiment decision nudging

efort reversal patterns

Feng et al. [8] Efectiveness Efciency User awareness Comprehensiveness Neutrality

Schaub & Usefulness Usablity Findability Understandability

Cranor [35]

ISO [16] Efectiveness Efciency Satisfaction

Quesenbery [33] Efectiveness Efciency Easy to learn Easy to learn Engaging Error tolerant

(initial use) (continued used)

Nielsen [28] Efciency Satisfaction Error tolerant

Morville UX Useful Usable, Ac- Findable Desirable,

Honey- cessible Credible

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA Habib et al.

Table 1: Components of six usability defnitions grouped according to diferent usability aspects.

comb [27]

users’ decisions related to cookies are largely impacted by the perceived trustworthiness of a website [20], there may be people who

would want to allow certain categories of cookies on some websites they visit but not on others. While automated mechanisms

could allow for custom rules, we expect some users may fall back

to using the website’s consent interface manually when they want

to make an exception to their rules. This study contributes a better

understanding of the design choices that impact the usability of consent interfaces, which may continue to persist even as automated

consent mechanisms become more widespread.

3 INSPECTION-BASED EVALUATION OF

COOKIE CONSENT INTERFACES

We used the seven usability aspects identifed in Section 2.3 to

guide our evaluation of design choices in CMP-implemented cookie

consent interfaces. As an initial step in our evaluation, we conducted an inspection-based evaluation of such interfaces across a

wide range of websites. We developed a standardized procedure for

our evaluation, informed by independent expert review, cognitive

walkthrough, and heuristic evaluation approaches [45]. Utilizing

the results of this expert review, we identifed design parameters

for consent interfaces that seem to be customizable through CMPs

and may have an impact on usability.

3.1 Inspection Procedure

To conduct our inspection-based evaluation of CMP-implemented

interfaces, we frst identifed fve services that are in widespread

use through a review of prior work in this space [12, 29]: Cookiebot,

Crownpeak, OneTrust, QuantCast, and TrustArc. We compiled a

diverse set of websites that we believed would have consent interfaces that are implemented through these CMPs. As a starting

point, we used a subset of 603 websites that were found to contain

interfaces implemented by these fve CMPs from Nouwens et al.’s

dataset of popular UK websites [29]. However, as the majority of

websites in this dataset contained a consent interface implemented

through either Quantcast or OneTrust, we aimed to diversify our set

of websites by identifying additional websites utilizing less popular

CMPs and with those based outside of the UK. We developed a web

scraper using webXray, a tool for analyzing webpage trafc [22],

which looked for domain requests to any of the fve CMPs. We ran

our scraper on 1,000 websites evenly sampled across Tranco’s list

of top 10,000 global websites [21] (as of June 21, 2021). This yielded

an additional 207 unique websites for our dataset of websites potentially containing an interface implemented by one of the fve

CMP providers. The distribution of CMPs in our fnal dataset of

810 websites was 369 (46%) Quantcast, 202 (25%) OneTrust, 87 (11%)

CookieBot, 78 (10%) TrustArc, and 74 (9.1%) CrownPeak.

We evaluated 191 consent interfaces on websites drawn from our

dataset of 810 websites potentially containing CMP-implemented interfaces.1 We evaluated at least ten interfaces implemented through

each CMP and attempted to identify distinct interface designs

within the group of websites using each service, particularly which

cookie options were provided, where and how they were presented

to users, and the content of the interface text. We continued our

review until we observed that consent interfaces on additional websites were very similar or identical to ones already reviewed (i.e.,

we had reached saturation in the data). Thus our sample includes a

wide variety of interfaces but is not representative of the frequency

with which each type of interface appears.

Two members of the research team jointly examined 15 websites

to establish the evaluation protocol. The remaining websites in

our sample were inspected by one of these researchers. For each

website, the researcher visited the desktop version of the website

from a computer with a US-based IP address. They visited each

website in private browsing mode, which mitigated the impact of

existing browser cookies and simplifed the inspection procedure

compared to creating new browser profles or virtual machines

for each website. The researcher frst assessed how difcult it

was to see the consent interface and noted if it changed or was

dismissed when clicking or scrolling on the website. They then

evaluated the interface based on a set of dark pattern heuristics

1We did not observe consent interfaces on all websites in our compiled dataset likely

due to websites making domain requests to use other services provided by the fve

CMP companies, or because our scraper was run with a US IP address.
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

“Okay, whatever”: An Evaluation of Cookie Consent Interfaces

identifed in prior work, including design patterns that may lead

to unintentional data disclosure or be considered illegal under the

GDPR or CCPA [24, 29, 39]. To evaluate for these patterns, the

researcher would attempt to select the most privacy-protective

option available (as dark patterns tend to appear as roadblocks in

this process). Specifcally they identifed:

• Unequal paths: The interface had unequal interaction paths

for the most and least privacy-protective options (see Figure 1a).

• “Bad” defaults: The interface had default options that were

not privacy-protective (see Figure 1b).

• Confusing buttons: The interface had unintuitive placement of buttons for confrming users’ cookie preferences

and allowing all cookies (see Figure 1c).

• No choices: The interface did not provide choices related

to the use of cookies (see Figure 1d).

• Confrmshaming: The interface used “confrmshaming’

which is wording that guilts or shames users to infuence

their decision [3] (see Figure 1e).

In conjunction with evaluating the interface against these heuristics, the researcher conducted a cognitive walkthrough of the cookie

consent interface. To guide their walkthrough, they approached the

interface as a user who wants to select privacy-protective options

but has minimal knowledge about digital privacy and cookies and

little motivation to read the privacy information provided by the

website. The researcher considered the visual appearance of the

interface, the interface text, and available options, specifcally:

(1) Are there visual aspects of the interface that may aid or be

detrimental to using selecting privacy-protective options

(i.e., options other than allowing all cookies)?

(2) What is the most privacy-protective option available in the

interface?

(3) What is the process for selecting the most privacy-protective

option (i.e., allowing the fewest cookies)?

Finally, based on their knowledge of usability and dark patterns,

the researcher made any additional notes about aspects of the consent interface that may confuse users. The researcher’s observations were recorded in a database, along with screenshots or screen

recordings of the analyzed cookie consent interface.2

3.2 Inspection Evaluation Results

We reviewed the results of our data collection and found that dark

pattern heuristics and other usability issues were prevalent in the

CMP-implemented consent interfaces we evaluated. We then identifed a list of design parameters that appear to be customizable

through CMPs and may impact the usability of consent interfaces.

3.2.1 Summary of Dark Patern Heuristics & Usability Barriers. As

shown in Table 2, the vast majority of the consent interfaces (88.0%)

reviewed exhibited a dark pattern heuristic, including some that

may be in violation of GDPR requirements [7]. The most prevalent,

observed on 150 (78.5%) websites, was having a simpler interaction

path for less privacy-protective cookie options (i.e.,“accept all cookies”) than for more privacy-protective options. Forty-nine (25.4%)

2The database of our observations is available at https://airtable.com/

shrnbTJ0ZIPl9OMm6.

CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA

consent interfaces in our sample also had pre-selected or default

options that were less protective of users’ privacy than other available options. This dark pattern heuristic occurred relatively more

frequently in consent interfaces implemented through OneTrust or

TrustArc, but as our sample is not representative, this may or may

not refect trends across all websites using these CMPs.

Some consent interfaces also exhibited usability barriers, beyond

potential dark patterns, that were uncovered during a cognitive

walkthrough of the interfaces. One example was a consent interface

that contained an “Options” button on the cookie options page that

did not appear to do anything but dismiss the consent interface.3

This would likely impact users’ comprehension of the interface,

as well as sentiment towards the company. Another interface contained a “Confrm My Choices” button within the cookie options

page but no choices were actually present on the page.4 The absence

of choices on an interface where users would expect them to be

present is highly likely to impact users’ ability to efectively make

decisions related to cookies on the website. In contrast, some interfaces included options to enable or disable diferent categories of

cookies on the initial screen of the interface,5 which prior work has

shown to impact users’ engagement with consent options [29, 42].

3.2.2 Design Parameters for CMP-Implemented interfaces. We used

our recorded observations to compare consent interfaces implemented through the same CMP, observing that interfaces primarily

difered in how they were initially presented, the interactions and

graphical elementsinvolved in making a consent decision, and what

information was provided about the cookies used on the website.

When reviewing our notes about the evaluated consent interfaces,

we highlighted any aspects of the interface that could be particularly benefcial or harmful to the usability of any privacy-protective

options available and for which design choices could be enumerated.

In doing so, we identifed design parameters that we hypothesized

would have an impact on the usability of the consent interface and

that appear to be customizable through CMPs, as well as possible

implementations of the parameter that we observed were being

used in practice (listed in Table 3).

4 USER STUDY EVALUATION OF CONSENT

INTERFACE DESIGNS

To further investigate the usability impact of design choices that

organizations can make when implementing their cookie consent

interfaces, we conducted a large-scale online user study in which

1,109 participants were each randomly assigned to visit and interact with a fctitious e-commerce website implementing one of 12

cookie consent design variants. This online experiment builds on

our inspection-based evaluation, as well as prior user studies of

consent interfaces, by evaluating the impact of diferent design parameters in consent interfaces against a comprehensive defnition

of usability rather than primarily through the lens of dark patterns.

3This particular consent interface was on friday-ad.co.uk and implemented through

Quantcast.

4This particular consent interface was on sketchup.com and implemented through

OneTrust.

5An example of such a consent interface is on acm.org implemented through

CookieBot.
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Cookie consent 

We use our own and third-party cookies to show 

you more relevant content based on your 

browsing and navigation history. Please accept or 

manage your cookie settings below. Here's 

our cookie P-Olicv. 

Cookie setting§. Accept all cookies 

Privacy Preference Center X 

Strictly NtCUSlfY 

Cookies 

Perform.11ncl! coolrlP.~ 

Function.ll Cookiu 

Targetil'lg Cookies 

Social Medi;i, Cookies 

Ei-iiiiil\OHIE 

Pcrfonn.1ncc Cookies 

Tho .. CU<l~ ...... u.:... u,Lu ~OU ,l vi~t.. .. ,td lu'fi< • ..,..,~..,.,.,W.,LO ,,,,., .. 

• ~, .. 

1nd improve :tie :icrrcrm,nreofoursitc, ltoyhclpus to knowwhichJ:•gc~ 

41C thc no.r.,i.c111lllt-ol 1,> 01,>uldl <11id)t"Cb11wlli)itu1>111U,t'dlUUlldl11t">~ • 

.611 Worm~ti:,n ~~ COO< c~col.ltrli~ J9Srcg.:,td .ind l~refcn:: 

inonymous. r'youdo nx a.lo'N these cockle! ~Wlll no1 k110wwht n JIOJ 

h.,vc ~istcd 011rsitc, .:indw II not be .iblc to monitor b pcrform~n<c 

EiiihiiE 

Powered t>yOneTrust 

Your Choices Regarding Cookies on this Site 

l 

REQUIRED COOKIES Thesecookienrereqv.N 10 ~ able core site functtonl ltv 

t-UNC I lUNA_ CUUKli:.S 

These cool<.ies1lkwu11oanatyze s,1'uu9eso"'e c.,.rn .. n•• tndm1>«>Y•~ riottr•nct 

A)VERTISlNG COOK ES 

Functionality Allo·Ned 

- Provide UCUIO log~n 

- Rf'mf'fl'l>Hh<>wfarY"'IMPlhr,..ll)h;onnr<1Pr 

- RemerrbK ycur •~ n dftatls 

- RemernbH what 1s ir your shoppinq cll't 

- Makesl.l'ttht wtbsi:e looksconsinert 

- Allow 'fOOIO share p,ges with soci;,I networlls 

- Alklw,ou1opos1cornm2flts 

- SefVeidsrelevan1 10yourirtertsts 

"h,spllQ" uansn~• 1nformat1011uso,g 1-TTFS protocol. Sorre vmdcrs cann:11 s~ MTTPSoJt-oJt ffQIH'SU. TnmAr-w11 slbmryou, prefe~nas lllr01.9h 

KTPlllll)O!>-t.l!IWl'ldow 

1111 i+kllO·lu+iiHfi 1.:•Wiil!!·He llil41 

Privtoey Policy PO' ,ec by /7 TrustArc: I TRUSTt 

This site uses cookies El 

By proceeding, you are agreeing to our 

Privac)". Policy, including the use of cookies 

and other tracking technologies. 

Taste the Ultimate Buy Whole Foods Online Experience 

We want to give you the very best service during your search for the highest quality foods. 

By clicking ~Accept All Cookies~, you agree to the storing of cookies on your device to enhance site navigation, 

analyse site usage, and assist in our marketing efforts. 

Don't worry, all of our cookies are made from the best quality organic ingredients! 

Cookies Setting§. Accept All Cookies 

CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA Habib et al.

(a) Consent interface with unequal paths. The option to “Accept all

cookies” is a prominent button but more privacy-protective options

are provided through a less conspicuous “Cookie settings” link to

the left.

(b) Cookie preferences page within a consent interface with “bad”

defaults where the default setting (“Advertising Cookies”) is the

least privacy-protective option available compared to the “Functional Cookies” and “Required Cookies” options.

(c) Cookie preferences page with confusing buttons. The placement

of “Allow All” and “Confrm My Choices” is unintuitive since submit buttons typically appear on the bottom right.

(d) Cookie consent interface with no choices. The privacy policy

simply describes the use of cookies rather than an interactive interface where users can make choices about the use of cookies.

(e) Example of a type of confrmshaming where it is implied that users do not want

“the very best service” or appreciate “the best quality organic ingredients” if all cookies are not accepted.

Figure 1: Examples of cookie consent interfaces found during our inspection-based evaluation for each dark pattern heuristic.
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

“Okay, whatever”: An Evaluation of Cookie Consent Interfaces CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA

Table 2: Counts of the dark pattern heuristics and other usability barriers identifed during our inspection-based evaluation

of consent interfaces implemented through fve CMP services. (n = number of consent interfaces evaluated for a particular

CMP)

CMP (n) Unequal paths “Bad” defaults Confusing buttons No choices Confrmshaming Other barriers None

OneTrust (70) 60 33 21 4 1 5 6

Quantcast (69) 55 0 0 7 1 1 7

CookieBot (20) 9 5 0 2 0 5 8

TrustArc (19) 14 9 0 2 1 0 2

CrownPeak (13) 12 2 1 2 0 1 0

Sum: (191) 150 49 22 17 3 12 23

Table 3: List of design parameters that appear to be customizable through the fve CMPs investigated, possible implementations

for each (in order of the least to best option for usability based on our expert knowledge), and the corresponding usability

objectives that we hypothesized could be impacted. The two parameters not explored in our user study are italicized.

Design Parameter Example Possible Implementations Usability Objective(s)

Prominence of the consent interface persistent “Cookie Preferences” button, non- User awareness

blocking banner, consent wall

Path to a cookie options interface (where op- link embedded in text, equally weighted interface User awareness

tions to allow/deny cookies are presented) button, in-line options in initial screen

How/whether the notice described the presence loss aversion text present, text mentions that options User awareness, User sentiment

of choices are available

Readability of the notice fonts, colors, contrasts, text layout (bulleted vs. para- User comprehension

graph)

Text within button options generic (“Okay,” “Submit”), detailed (“Allow selected User comprehension

cookies,” “Allow all cookies”)

Layout of cookie options page choices separated in multiple tabs, all choices on User ability & efort

same page

Process for changing or revoking a consent de- none (clear browser cookies), link in cookie policy, Decision reversal

cision persistent “Cookie Preferences” button

Placement of button options “Allow all” option shifts with user actions, “Allow User ability & efort

all” remains in place

The granularity of choices ofered cookie-level, category-level User ability & efort

We frame our fndings around the other six usability aspects identifed in Section 2.3: user needs, user ability & efort, user awareness,

user comprehension, user sentiment, and decision reversal.

4.1 User Study Design

We enumerated possible design choices for the design parameters

we identifed (listed in Table 3) based on the practices we observed

in our inspection-based evaluation. As it was infeasible to study all

of the possible design choices, we ranked the design parameters

according to what we believed was likely to have the most impact

on usability and prioritized design choices for which there has not

yet been much research or established best practice in UX design.

Along these criteria, we decided not to explore the placement of

button options within the consent interface (for which there are established best practices [34]) and the granularity of choices ofered

(for which there has been prior research that shows that users may

be overwhelmed by having too many choices [1]). Our study also

did not explore accessibility issues, such as those related to color

contrast and size of button components within the interface, which

also have established guidelines [43]. The remaining seven design

parameters corresponded to variables in our study.

We developed 12 design variants of cookie consent interfaces

(provided in Appendix A) based on the practices we observed in our

inspection of CMP-implemented consent interfaces. Table 4 provides an overview of these twelve variants and their values for the

seven study variables: prominence of the consent interface, path to a
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

This website uses cookies 

• Why we use cookies: To enhance site performance, personalize your 

experience, and deliver interest-based ads 

• Your choices: Some cookies are optional. Click 'Allow all cookies' to accept 

all cookies, or edit your cookie preferences below and select 'Allow 

• Updating preferences: Click on the ·cookie Preferences' button on the 

bottom right corner of this website 

• More information: Please see our ~Y...£2J.kY. 

Allow selected cookies Allow all cookies 

0 Strictly necessary O Performance O Functional O Targeting Shl>wJle1ails 

This website uses cookies we use coo.-.les to make the weMlte wo,_ prope,I)', ennance pe,!ormance, c,ute penona1,zea tunct,o n.i,11, 1na deliver t1,gete<1 ocH . Note tna, If ~•u C!O 

not 1ccept 01>1i on1I coo-ies, your e,periencem~be 1ffttttd. 6yco,,1inuiogto use thi1 sitey"" 1grtt to the UH ofcool:iu11 show,, In your .cl!!!l:.il. 

llllllW:l!;Si. Y<x1 m1yupa1t•you,coo-le pre!Or•nctsat1r,y!ime formo,1lnrorml!lo,,,plHHOH<x1r~~ 

Cookie Preferences 

What are cookies? 

cookies are small data files placed on your computer or internet enabled device by 

websites in order to add functions to the site or to help the website work better. 

Sometimes cookies are necessary to be used for the basic functioning of your site, 

and sometimes they can be used to identify your computer or internet device to our 

website, or to other third•party websites. This may allow us to improve the way we 

work, such as remembering preferences expressed by you, or tracking your use of a 

website for statistical analysis and advertising. 

For more information about cookies, please see here: hllg_s_;Ll 

www.allaboutcaakies.orgl 

Strictly necessary cookies Always active 

These cookies are necessary for the website to function and can't be switched off in 

our systems. They are usually only set in response to actions you have taken which 

result in a request for services, such as setting your privacy preferences, logging in or 

filling in forms. You can set your browser to block or alert you about these cookies, 

but some parts of the site may not work as a result . 

Performance cookies Deny ~ Allow 

These cookies allow us to count visits and traffic sources so we can measure and 

improve the performance of our site. They help us know which pages are the most 

and least popular and see how visitors move around the site. All the information these 

cookies collect is aggregated and therefore anonymous. If you do not allow these 

cookies, information from these cookies will not be used to help site performance. 

Functional cookies Deny ~ Allow 

These cookies allow the provision of enhanced functionality and personalization. 

They may be set by us or by third-party providers whose services we have added to 

our pages. If you don't allow these cookies, then some or all of these functionalities 

may not function properly. 

Targeting cookies Deny ~ Allow 

These cookies are set through our site by our advertising partners. They may be used 

by those companies to build a profile of your interests and show you relevant ads on 

other sites. They work by uniquely identifying your browser and device. If you don't 

allow these cookies, you will not experience our targeted advertising across different 

websites as a result of these cookies. 

Save preferences 

Cookie Preferences 

About cookies 

What are cookies? 

X 

Cookies are small data files placed on your computer or 

internet enabled device by websites in order to add functions 

to the site or to help the website work better. Sometimes 

cookies are necessary to be used for the basic functioning of 

your site, and sometimes they can be used to identify your 

computer or internet device to our website, or to other thirdparty websites. This may allow us to improve the way we work, 

such as remembering preferences expressed by you, or 

tracking your use of a website for statistical analysis and 

advertising. 

For more information about cookies, please see here: 

hllQ_S_;_/lwww allaboutcookies orgL 

Save preferences 

CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA Habib et al.

(a) The “best practices” baseline design variant, implemented as a

blocking interface, used for comparison which incorporated the

design choices that we considered as most privacy-protective or

benefcial to usability. Checkboxes corresponding to cookie categories were unchecked by default in accordance to GDPR requirements [30].

(b) The “worst practices” design variant, implemented as a nonblocking banner at the bottom of the webpage, which incorporated

the design choices that we considered as least privacy-protective or

most detrimental to usability.

Figure 2: Two consent interface design variants that demonstrate the design choices for each parameter explored in our

user study.

cookie optionsinterface, presence of loss aversion text, layout of the (a) Single-layer “Cookie Preferences” interface linked from the

interface text, specifcity of the button text, layout of the cookie op- cookie consent interface in ten of the design variants.

tions page, and process for changing or revoking a consent decision.

So that we could isolate the efect of each design choice, one design

variant was composed of what we considered as “best practices”:

what we hypothesized as the most privacy-protective or usable

options for each study variable (see Figure 2a). Ten of the design

variants manipulated just one study variable such that they difered

from the “best practices” baseline in only one aspect of the interface

design in a way that we believed to be less usable or less privacyprotective. Another design variant (prominence-cornerButton) was

based of of a OneTrust-implemented consent interface design that

presented visitors with consent options through a button that persisted on the bottom right corner of the webpage.6 A twelfth design

variant explored the combination of design choices that we considered were the least privacy-protective or usable, which we refer to

as “worst practices” (see Figure 2b). The design variants included a

link to a single-layer“Cookie Preferences” page (shown in Figure 3a) (b) Multi-layer “Cookie Preferences” interface linked from the

or a multi-layer version of the page (Figure 3b), which included cookie consent interface in the layout-multilayer and worstinformation about cookies and four diferent cookie categories de- practices design variants.

fned by the International Chamber of Commerce (ICC) UK (strictly

necessary, performance, functional, and targeting) [15] as well as Figure 3: The two styles of the“Cookie Preferences” linked

toggles to enable/disable the later three categories. Table 4 provides through the cookie consent interface design variants exan overview of the design variants explored in our study and their plored in our study. The consent interface was dismissed

values for the seven study variables. upon clicking the “Save prefences” button.

6Bitnami.com’s website has an example of such an interface.
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

“Okay, whatever”: An Evaluation of Cookie Consent Interfaces CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA

Table 4: Overview of the 12 cookie consent interface design variants and their values for the seven design parameters explored

in our online experiment. Design choices that difer from best-practices are bolded.

Condition Name Prominence Options Path Loss Aversion Text Layout Button Text Choices Layout Decision Reversal

best-practices fully-blocking in-line absent bulleted detailed single-page persistent button

prominence-cornerButton non-blocking button n/a n/a n/a n/a single-page persistent button

prominence-banner non-blocking banner in-line absent bulleted detailed single-page persistent button

options-embeddedLink fully-blocking embedded link absent bulleted detailed single-page persistent button

options-interfaceButton fully-blocking interface button absent bulleted detailed single-page persistent button

text-lossAversion fully-blocking in-line present bulleted detailed single-page persistent button

text-layoutParagraph fully-blocking in-line absent paragraph detailed single-page persistent button

button-generic fully-blocking in-line absent bulleted generic single-page persistent button

layout-multilayer fully-blocking interface button absent bulleted detailed multi-page persistent button

reversal-noInstructions fully-blocking in-line absent bulleted detailed single-page no instructions

(button present)

reversal-cookiePolicy fully-blocking in-line absent bulleted detailed single-page cookie policy

worst-practices non-blocking banner embedded link present paragraph generic multi-page no instructions

(cookie policy)

4.2 User Study Data Collection & Analysis

4.2.1 Experimental Protocol. We conducted an online experiment

utilizing a between-subjects protocol. To explore the impact of the

diferent design parameters in a realistic setting, we presented our

consent notice designs in the context of a fctitious e-commerce

website that sold cups, mugs, and other drinkware. We used Adobe

XD to implement the parts of an e-commerce website relevant

to the cookie consent experience or basic shopping functionality,

including a cookie consent interface (varied per condition), privacy

policy, cookie policy, product catalog, and product detail pages. We

implemented the prototypes only in a desktop version of a website

to maximize the chances of participants being able to read and

interact with the consent notice. In order to capture participants’

interactions with the website as well as timing data, we utilized

a usability testing platform called Useberry. After completing the

study consent form and verifying their eligibility, participants in

our study were assigned one of the study conditions at random and

directed to Useberry. To prevent participants from overly fxating

on the consent notice, participants were give a distraction task:

add a product from the store catalog to their cart. Participants

were instructed to interact with the prototype as they would a real

website and perform whatever action they would take the frst time

they visited a real e-commerce website. After the initial instruction

screens, participants were exposed to a cookie consent interface

design according to their assigned condition. Once participants

completed the study task, or indicated that they give up on the task

through a button provided by Useberry, they were directed to a

follow-up survey implemented on Qualtrics.

The survey (provided in Appendix B) included questions for

evaluating the diferent aspects of usability described in Section 2.3.

Participants frst answered questions related to user awareness and

unfocused comprehension based on their recall of the consent notice. After completing this portion of the survey, participants were

provided an opportunity to refer back to the consent interface and

prototype of the e-commerce website as they answered additional

questions.

Our protocol was approved by Carnegie Mellon University’s

Institutional Review Board. While participants consented to their

interactions with the prototype website being captured, we did not

collect any personal information from participants.

4.2.2 Participant Recruitment. To prevent priming potential participants, we described the study as a study requesting feedback

about an e-commerce website. Due to the language competencies

of the research team and the consideration that cookie consent

interfaces and users’ experiences with them may difer across legal

jurisdictions, we only recruited US-based participants. Additionally,

participants were required to be over 18 years old, fuent in English, and have access to a tablet or computer to complete the study

(to properly render the prototypes). Median completion time for

our study was 15 minutes and 48 seconds, and participants were

compensated $5.00.

We piloted our study setup with 45 participants prior to launching the full study. Based on a power analysis for our planned statistical tests, at least 66 participants per condition (786 participants

total) would be needed to detect a moderate efect size with at least

80% power. In total, 1,316 participants from Prolifc completed our

study between July 28 and July 30, 2021.

4.2.3 Data Analysis. Our analysis includes data from 1,109 participants. We did not include responses from 127 participants who

were inadvertently exposed to two diferent versions of our consent

notice due to a technical issue with Useberry prior to completing

the survey.7 We also removed responses from 42 participants who

were detected using a mobile device by Useberry, as our prototypes

were designed for tablet or desktop viewing. Last, we removed 38

participant responses for which a valid Useberry session (sessions

in which we could confrm participants saw a consent notice either

7Due to the same technical issue with Useberry, another 342 Prolifc workers attempted

to participate in our study but were unable to complete it. These participants were

compensated $1.00 for their time.
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA

through successful task completion or reviewing their interaction

data) was not recorded. A few participants completed the study

twice, so we retained only their frst submission.

We analyzed user interaction and timing metrics collected via

Useberry, as well as participants’ survey responses. User fow diagrams generated by Useberry provided aggregated metrics about

which pages of the consent interface and website participants visited during the study task, providing insight into whether participants engaged with the consent interface beyond making a consent decision. Useberry also provided metrics with regards to how

long participants spent on the study task overall and the number

of clicks they made prior to task completion. However, Useberry

could not be confgured to record individual participants’ exact consent decision in a format appropriate for such large-scale analysis;

therefore we analyzed participants’ self-reported consent decision

from the survey. Participants frst indicated which cookie options

they selected in the recall portion of the survey. To mitigate recall

bias, participants answered the question again after reviewing the

consent interface. A researcher reviewed a recording of a participant’s interactions with the prototype captured by Useberry to

verify their consent decision if there was a discrepancy in their

response to these two questions or if they indicated selecting an

unavailable option (i.e., “Allow social media cookies” or “Allow no

cookies”). Approximately 20% of participants’ consent decisions

were reviewed in this manner. Participants’ self-reported data was

used to summarize the consent decisions made by users of each of

the design variants.

In our reporting of fndings, we highlight cases where we observed signifcant diferences between the best-practices design

variant and another variant, presenting both p-values and an efect

size for the appropriate statistical test. Since our study was powered

to detect at least moderate efect sizes with at least 80% power for

our planned analysis, we note any signifcant results for which a

smaller efect size was observed. P-values from any post-hoc pairwise comparisons were adjusted with a Bonferroni correction to be

able to correct for additional comparisons with categorical data.

We conducted a thematic analysis of the responses to the qualitative survey questions asking participants about their expectations

and goals related to their consent decision, any additional cookie

options they desired, and methods for changing their initial consent

decision. One member of the research team developed an initial

codebook based on a subset of 10% of responses drawn at random.

Two researchers then independently coded another random subset

of 20% of the data, achieving a Cohen’s κ inter-rater agreement

of 0.84 (averaged over all questions), which is considered high

agreement [9]. Any conficts in the coding were resolved and the

codebook was accordingly modifed in collaboration. The remaining survey responses were coded by a single researcher using the

modifed codebook.

4.3 Participant Demographics

Table 5 provides a demographics summary of our study population.

While our participant sample was diverse, it was not representative

of the US population, skewing more female, white, and younger

than the general population [41]. It is likely that our study was

impacted by an infux of new registrations on Prolifc by young

Habib et al.

women that occurred in July 2021 due to a viral video on TikTok [4].

We report on the impact of age and gender in our analysis of participants’ consent decision, awareness of available cookie options,

comprehension of the interface, and investment in decision-making.

The vast majority of our participants (85.8%) reported shopping

online at least once a month, and only four participants indicated

that they never shop online. This suggests that participants in our

sample likely had prior experiences with websites similar to our

prototype that may have infuenced their interactions during our

study.

4.4 User Study Results

Our study results, summarized in Table 6, highlight that several

design parameters that we explored signifcantly impacted the usability of consent interfaces. We found that the absence of in-line

options within the initial screen of the interface impacted participants’ consent decision, comprehension of available cookie options,

as well as sentiment toward the consent interface. Additionally, we

observed that awareness of available cookie options was impacted

by the prominence of the consent interface. Furthermore, a persistent “Cookie Preferences” button improved participants’ ability to

change their initial consent decision.

4.4.1 User Needs. The majority (72.7%) of our participants who

reported making a consent decision selected the “Allow all cookies”

option in the interface, 24.4% selected “Allow only strictly necessary

cookies,” and another 2.9% allowed some custom combination of

strictly necessary, performance, functional, or targeting cookies. A

Pearson’s chi-squared test found that participants’ reported cookie

consent decision (shown in Figure 4) signifcantly difered across

conditions (p < 0.001, Cramer’s V = 0.29). Participants in four conditions that did not include in-line options (options-embeddedLink,

options-interfaceButton, layout-multilayer, and worst-practices)

were signifcantly more likely to consent to all cookies, compared

those in best-practices. We did not observe signifcant impact of age

or gender on participants’ consent decision.

About half of participants who selected “Allow all cookies”(50.2%)

described that their goal was to dismiss the consent interface (e.g.,

“I just wanted to get to the website and thought, ‘Okay, whatever”’).

suggesting that participants may have become habituated into clicking this option when available. Others who allowed all cookies described more specifc goals, such as enabling specifc features of the

website (e.g.,“Ease of use when I return to the website in remembering my information”), allowing for full functionality of the website

(e.g., “To gain full access to the website and all its features”), or

improving the performance of the website (e.g.,“For the website to

run as smooth as possible”). In contrast, the majority of those who

only allowed strictly necessary cookies (57.9%) described privacyrelated goals, including limiting the amount of personal data that

is collected (e.g., “Bare minimum private information collected”)

or web tracking that may occur which could lead to targeted ads

(“I don’t want my actions to be tracked unnecessarily, especially

for targeting ads.”). Some participants who selected this option expressed that they wanted to limit the number of cookies because of

an incomplete understanding of web cookies (e.g.,“I do not really

understand cookies, but I think that they clog up your computer so
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

“Okay, whatever”: An Evaluation of Cookie Consent Interfaces CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA

Table 5: Summary of participant demographics. Participants were allowed to select multiple options for race/ethnicity so

percentages are greater than 100. Those who reported having a formal education or work experience in a computer-related

feld were counted as technical experts.

Gender Age (Years) Race/Ethnicity Education Income Tech Expertise

Agender 0.45% 18-24 64.9% Am. Indian/Alaska Native 1.0% High school or less 15.0% <$10k 8.6% Yes 17.0%

Female 79.8% 25-34 26.3% Asian 8.7% Some college 30.7% $10k to $49,999 31.2% No 83.0%

Male 15.1% 35-44 5.6% Black 5.1% Associates/Bachelors 40.7% $50k to $99,999 29.5%

Non-binary/Genderqueer 4.1% 45-54 2.4% Hispanic/Latinx 3.2% Graduate/Professional 13.6% $100k to $149,999 14.5%

Self-described 0.36% 55-64 0.63% Hawaiian/Pacifc Islander 0.26% No response 0.09% ≥ $150k 9.8%

No response 0.27% > 65 0.0% White 79.7% No response 6.4%

No response 0.45% Self-described 1.1%

No response 1.1%

Table 6: Summary of how study conditions signifcantly difered from best-practices when considering the six usability aspects

explored in the user study.

Usability Aspect Signifcant Conditions Summary

User Needs options-embeddedLink, optionsinterfaceButton, layoutmultilayer, worst-practices

Participants in these conditions were signifcantly more likely to consent to all cookies,

compared those in best-practices.

User Ability & Efort prominence-cornerButton Participants in prominence-cornerButton were signifcantly less likely to select their

preferred consent decision, compared to those in best-practices.

User Awareness prominence-cornerButton,

worst-practices

Participants in these conditions were signifcantly less aware of a privacy decision,

compared to those in best-practices.

User Comprehension options-embeddedLink, optionsinterfaceButton, layoutmultilayer

Participants in these conditions answered more comprehension questions correctly after

reviewing the consent interface

User Sentiment options-embededLink, layoutmultilayer, worst-practices

Participants in options-embededLink and layout-multilayer were signifcantly more likely

to report choosing the “easiest option”; those in worst-practices were more likely to report

skipping over the interface text, compared to those in best-practices.

Decision Reversal reversal-cookiePolicy Participants in reversal-cookiePolicy were signifcantly less likely to recognize a correct

method to change their initial cookie consent decision, compared to those in best-practices.

I wanted to avoid this.”). These results highlight the importance of

providing cookie options that align with specifc goals.

In assessing user needs related to the consent interface, we also

asked participants to describe what, if any, additional options related to cookies they would like to have. While the majority of

participants did not articulate any additional choices they would

like to have, 157 participants suggested providing an option for

denying all cookies (which would be infeasible for an e-commerce

website given current web technology). Additionally, 48 described

other suggestions, including providing “cookie options” for other

privacy or security-related features (e.g., “Cookies that will help

keep passwords and logins safe.”), or an option for cookies not to

persist beyond the browsing session (e.g., “Option to clear cookies

when done browsing”). In lieu of additional options, 36 participants

desired additional information, such as defnitions for the term

cookies and diferent cookie categories or how the website would

behave if not all cookies were allowed.

4.4.2 User Ability & Efort. In the survey, participants were provided an opportunity to review the consent interface again and

were explicitly asked to select what their preferred consent decision would be for the website. In their response, 40.1% indicated

they would want to allow all cookies, 29.7% preferred to allow only

strictly necessary cookies, 25.2% indicated a custom combination

of cookie categories, and 5.1% preferred that the website not use

any cookies at all. Excluding participants who reported that they

would prefer not to allow any cookies (a preference that could not

be selected in any condition), less than half of participants (45.3%)

actually selected their preferred consent decision during their interactions with the website. A Pearson’s chi-squared test found that

this signifcantly difered across conditions (p < 0.001, V = 0.34). As

no participants in prominence-cornerButton made a consent decision, post-hoc comparisonsfound that participantsin that condition

were signifcantly less likely to select their preferred consent decision compared to those in best-practices. However, the majority of

participants (74.0%) felt that it was very easy or somewhat easy to

make their preferred consent decision, which did not signifcantly

difer across conditions. Taken together, these results suggest that

while the majority of participants in the prominence-cornerButton

condition did not fnd it difcult to use the consent interface, the

absence of a banner or pop-up consent interface severely impacted

participants’ ability to provide their cookie preferences.

Participants spent an average of 1 minute and 28 seconds with

9.1 clicks to complete the study task (i.e., adding a product to the
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

I I I I 

CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA Habib et al.

prominence−cornerButton

worst−practices

options−embeddedLink

layout−multilayer

options−interfaceButton

prominence−banner

text−layoutParagraph

best−practices

button−generic

reversal−noInstructions

text−lossAversion

reversal−cookiePolicy

0 25 50 75 100

% of Participants in Condition

Only strictly necessary Custom All cookies No selection

Figure 4: Participants’ reported consent decisions in their interactions with the prototype website where “custom” refers to any

combination of strictly necessary, performance, functional, or targeting cookies. Three participants who saw blocking consent interface (in the reversal-cookiePolicy, reversal-noInstructions, and buton-generic conditions) bypassed making a consent

decision by clicking on other links within the consent interface, which dismissed the interface in the prototype.

shopping cart) which was not found to signifcantly difer across

conditions. This suggests that the efort required to complete a

consent decision was similar across conditions. In our analysis of

participants’ interactions with the prototype website, we observed

that 24.0% of participants in worst-practices and 19.8% of participants in prominence-nonblockingBanner went directly to the catalog

without making a consent decision. No participants in prominencecornerButton were observed indicating their cookie preferences at

any point during their interactions with the website. This implies

that a substantial portion of users are likely not to indicate their

cookie preferences if not blocked from using other parts of the

website.

Beyond making a consent decision with a button option, we

observed 99 additional interactions with other components of the

cookie consent interface, seven interactions with one of the links to

the website’s privacy policy (located within the consent interface or

in the footer of the website), and no interactions with the website’s

cookie policy. Figure 5 provides a summary of participants’ engagement with cookie-related options. The most common interactions

were changing in-line options in the initial consent interface, clicking the persistent “Cookie Preferences” button, clicking the link

or button leading to options from the initial consent interface, and

changing the consent option toggles in the “Cookie Preferences”

interface.

4.4.3 User Awareness. While only 66.6% of participants initially

recalled making a privacy-related decision, when we asked them

specifcally about cookie-related decisions, 86.8% of participants

reported making a consent decision during their interaction with

the prototype website. This suggests that some participants may

have been habituated to consent interfaces and did not pay enough

attention to the cookie consent interface to understand that it was

related to privacy. In their recall of consent options, participants

correctly selected between three and four cookie categories out of

seven listed (two of which were not actually available on the website). We found that both awareness of a privacy decision (Fisher’s

exact test, p < 0.001, V = 0.48), as well as available options related

to cookie categories (Kruskal-Wallis test, p < 0.001, η = 0.053

[small efect]) signifcantly difered across conditions. In follow-up

pairwise comparisons, participants in the prominence-cornerButton

and worst-practices conditions reported signifcantly less awareness compared to those assigned to best-practices. Three-quarters

of participants in best-practices recalled making a privacy decision

and, on average, participants in this condition correctly recalled 3.5

cookie options. In comparison, half of worst-practices and only 2.9%

of prominence-cornerButton participants recalled a privacy decision.

On average, participants in worst-practices correctly selected 2.8

options and those assigned to prominence-cornerButton recalled 2.7

options. The relatively poor awareness of participants in these two

conditions is unsurprising given that less than three quarters of

worst-practices participants and no prominence-cornerButton participants made a cookie consent selection (see Figure 4) and thus were

unlikely to even view the options; the options they recalled correctly are likely based on their past experience with cookie consent

decisions.

Kruskal-Wallis tests found that recall of cookie options was also

signifcantly impacted by age (p = 0.005, η =.006 [small efect])

and gender (p < 0.001, η = 0.010 [small efect]). Those aged 35 and

older had better recall of available options (3.6 correct) than those

younger than 35 (3.2 correct). Compared to women (3.2 correct),

men were found to have signifcantly betterrecall (3.6 correct). However it is unclear whether participants with better observed recall

truly were better at recalling information, or if they had simply

reviewed their cookie options more carefully.

Our analysis of interactions with the website prototype (reported

in Figure 5) also provides evidence that user awareness of cookie

options was impacted by the prominence of the initial consent

interface. While the “Cookie Preferences” button in the bottom
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

■ ■ ■ 

■ 

“Okay, whatever”: An Evaluation of Cookie Consent Interfaces CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA

RI3DUWLFLSDQWVLQ&RQGLWLRQ

EHVWSUDFWLFHV

SURPLQHQFHFRUQHU%XWWRQ

SURPLQHQFHEDQQHU

WH[WOD\RXW3DUDJUDSK

WH[WORVV$YHUVLRQ

EXWWRQJHQHULF

RSWLRQVHPEHGGHG/LQN

RSWLRQVLQWHUIDFH%XWWRQ

UHYHUVDOQR,QVWUXFWLRQV

UHYHUVDOFRRNLH3ROLF\

OD\RXWPXOWLOD\HU

ZRUVWSUDFWLFHV

    

&KDQJHGLQOLQHRSWLRQV &OLFNHGFKRLFHLQWHUIDFHOLQNEXWWRQ &OLFNHGFRRNLHSUHIHUHQFHVEXWWRQ

&KDQJHGWRJJOHV

X

X

X

X

X

X

X

X

Figure 5: A summary of participants’ engagement with the cookie consent interface beyond selecting one ofthe button options.

Specifcally, we noted (if applicable to the study condition) whether participants changed any of the in-line options in the

interface, clicked on the link or button leading to the cookie choices interface, clicked the persistent cookie preferences button,

or changed any toggles within the cookie choices interface. Interactions not applicable to a condition are marked with a ‘X’.

corner of the webpage was used in all of the other design variants

that included it, it seemed to go ignored in prominence-cornerButton.

Thissuggeststhat a fully-blocking or banner-style consent interface

led to greater awareness of available cookie choices.

4.4.4 User Comprehension. To gauge participants’ comprehension

of their cookie-related choices, the survey included fve multiplechoice questions in which participants were asked to select the

correct defnitions for the term “cookies” (in the context of the

internet) and each of the four cookie categories included in the

interface. On average, participants correctly answered between two

and three questions, based on their recall of the website and consent

interface when their attention likely was not focused on available

cookie choices. Less than half of participants (47.6% ) selected the

correct defnition for“performance cookies” and only 16.0% selected

the correct answer for “functional cookies,” suggesting that these

two labels for cookie categories defned by the ICC UK are not very

intuitive. Most commonly, participants thought functional cookies

were those that were needed for the website to work properly; this

is actually the correct defnition for “strictly necessary cookies.”

A Friedman test found a signifcant improvement in comprehension (p < 0.001, Kendall’s W = 0.59) by about one question

when participants answered the same fve comprehension questions again after being able to review the consent interface. Unlike

participants’ unfocused comprehension, there was a signifcant difference across conditions in focused comprehension (p < 0.001, η =
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA

0.08). Compared to those in best-practices, participants in optionsembeddedLink, options-interfaceButton, and layout-multilayer answered more of the comprehension questions correctly after reviewing the consent interface. This may be because participants in

these conditions were not exposed to the diferent cookie category

terms through in-line options and instead saw them on the Cookie

Preferences page where they were defned, making them less prone

to guessing than participants who saw the cookie categories on the

initial screen. When asked which aspects of the consent interface

they referred to when answering the survey questions, a larger

percentage of participants in these conditions reported referring to

the Cookie Preferences page, compared to those in best-practices.

Kruskal-Wallis tests found signifcant diferences in unfocused

comprehension by age (p = 0.01, η = 0.005 [small efect]), as well

as gender (p < 0.001, η = 0.01 [small efect]). Those younger than

35 correctly answered 2.7 questions, compared to 3.1 questions for

those 35 and older, while women answered 2.6 questions correctly

on average, compared to 3.1 questions for men. These results are

aligned with fndings related to awareness of cookie options.

After reviewing the consent interface, participants were asked

how easy or difcult they thought the consent interface was to

understand. Over two-thirds (68.0%) reported that it was somewhat easy or very easy to understand, which was was not significantly diferent across conditions. The survey also asked participants about their comprehension of which cookie consent option

was being recommended by the interface, reported in Figure 6.

While none of the design variants explicitly recommended a consent option, a Pearson’s chi-squared test found that participants’

interpretations of what was being recommended signifcantly differed across conditions (p < 0.001, V = 0.14). The majority of participants in worst-practices (60.0%), text-lossAversion (51.7%), and

options-embeddedLink (50.0%) thought that the interface was recommending to allow all cookies, though post-hoc comparisons did

not fnd this to signifcantly difer from best-practices.

Participants were also asked to indicate the likelihood of fve

diferent scenarios if a cookie consent decision was not made on the

website. The most common expectations were “all cookies would

be allowed and the entire website would still work” and “no cookies

would be allowed but some parts of the website would still work,”

rated as “probably yes” or “defnitely yes” by 68.3% and 56.0% of

participants respectively. This highlights that ambiguity that exists

in current implementations of cookie consent interfaces, as both of

these scenarios are technically feasible, although their legality may

depend on regulations in the user’s location.

4.4.5 User Sentiment. To gauge participants’ level of investment

in making a cookie consent decision, we asked participants who indicated that they made a consent decision on the prototype website

two multiple-choice questions and one Likert scale question related

to their decision-making process. Pearson chi-squared tests found

that there was a signifcant diferent across conditions in participants’ strategies for selecting their cookie preferences (p < 0.001,

V = 0.18), as well as their engagement with the interface text (p <

0.001, V = 0.23). Similarly, a Kruskal-Wallis test found that participants’ ratings for how carefully they made their consent decision

also difered across conditions (p < 0.001, η = 0.051 [small efect).

Signifcantly more participants in options-embededLink (83.3%, p.adj

Habib et al.

= 0.006) reported choosing the “easiest option” when making their

consent decision and were more likely to report that they made their

decision “not at all carefully” (73.4%, p.adj = 0.04), compared to participants in best-practices (55.7% and 46.6% respectively reported the

same). Similarly, signifcantly more participants in worst-practices

than best-practices made their decision “not at all carefully” (75.6%,

p.adj = 0.01) and reported skipping over the interface text (59.0%,

p.adj = 0.001). Those in layout-multilayer were also signifcantly

more likely to report choosing the “easiest option” (80.7%, p.adj =

0.04). Thissuggeststhat the absence of in-line options within the initial screen of the consent interface may have reduced participants’

investment in their consent decision.

Pearson chi-squared tests and a Kruskal-Wallistest comparing responsesto these three questions also revealed signifcant diferences

in decision-making investment with age and gender. Compared to

those 35 years or older, younger participants were more likely to

report choosing the “easiest option” (p < 0.001, V = 0.16), skipping

over the interface text (p = 0.001, V = 0.11), and making their decision “not at all carefully” (p < .001, η = 0.051 [small efect]). Relative

to men, women reported less investment in their decision-making,

being more likely to choose the “easiest option” (p < 0.001, V = 0.20),

skipping over the interface text (p = 0.002, V = 0.10), and making

their decision “not at all carefully” (p < 0.001, η = .065).

4.4.6 Decision Reversal. Two design variants(reversal-noInstructions

and reversal-cookiePolicy) directly explored participants’ ability to

revoke or change their consent decision. A Pearson’s chi-squared

test found that participants in the best-practice condition that contained a persistent “Cookie Preferences” button in the bottom right

corner of the page were signifcantly more likely than those in

reversal-cookiePolicy which did not contain this button to recognize

a correct method to change their initial cookie consent decision

(p = 0.001, V = 0.28). The vast majority (81.8%) of best-practices

participants stated that they would use this button to change their

decision, while 45.3% of participants in reversal-cookiePolicy stated

they would visit the website’s cookie policy (as instructed in the

interface text). We found that the presence of reversal instructions

did not have a signifcant impact on participants’ ability to reverse

their initial consent decision.

When asked how they would reverse their preferences if there

was no “Cookie Preferences” button, only 16.1% of participants in

the conditions that contained this button described an efective alternative method for revising their consent decision for the website,

such as the website’s privacy or cookie policy, deleting browser

cookies, using a diferent browser or device, or visiting the website

in private browsing mode. This suggests that after being exposed

to the “Cookie Preferences” button its absence had a much greater

impact than if participants had not seen it at all. Along these lines,

42.2% said that they would give up trying to change their consent

preferences or just leave the website. Over a ffth (22.9%) described

other strategies that could potentially lead them to a correct decision reversal path, such as changing browser settings, looking

through the settings or other parts of the website, contacting the

website, or searching for instructions using a search engine. A small

portion of participants (10.4%) described an incorrect strategy such

as refreshing the page or revisiting it in another tab, and another
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

I I I I I 

“Okay, whatever”: An Evaluation of Cookie Consent Interfaces CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA

reversal−noInstructions

prominence−banner

prominence−cornerButton

best−practices

button−generic

text−layoutParagraph

reversal−cookiePolicy

layout−multilayer

options−interfaceButton

options−embeddedLink

text−lossAversion

worst−practices

0 25 50 75 100

% of Participants in Condition

All cookies Selected cookies Only strictly necessary Other All options equal

Figure 6: Participants’ comprehension of what (if any) cookie consent options the website seemed to be recommending.

6.3% were not sure what they would do to reverse their consent

decision.

5 DISCUSSION

In this section, we frst describe limitations of our evaluation of

cookie consent interfaces. We then review the implications of our

fndings on the design of such interfaces.

5.1 Limitations

While our study provides valuable insights into the usability of

consent interfaces, it is not without its limitations.

5.1.1 Inspection-based Evaluation. The procedure used to evaluate consent interfaces likely introduced some bias. First, we compiled our list of websites to evaluate from multiple sources and not

through random sampling. In addition, the consent interfaces that

we observed from a US-based IP address could difer from those

displayed to visitors in the EU or elsewhere. We developed our

list of 10 design parameters based on our inspection and evaluation of consent interfaces implemented by CMPs. It is possible that

cookie consent interfaces that are not implemented through CMPs

incorporate other design parameters that were not uncovered in

our inspection-based evaluation. Furthermore, our user study only

explored a subset of the identifed design parameters and implementations corresponding to these parameters. Though prior research

and best practices exist with regard to the two parameters we did

not include (placement of button options and granularity of the

choices ofered), these should be further explored in the context of

cookie consent interfaces. In addition, our study examined cookie

consent interfaces on computers and not on mobile devices. Our

study also did not evaluate the accessibility of cookie consent interfaces, which should be implemented according to standardized

accessibility guidelines to ensure that they are usable by a larger

population of internet users [43].

5.1.2 User Study. Though our user study evaluated our cookie

consent interface designs in a realistic context, participants were

aware that they were interacting with a prototype website through

Useberry, which may have impacted their interactions and impressions of the consent interface. Additionally, while Useberry allowed

us to capture data related to the time and number of clicks participants spent on the study task overall, we were unable to analyze

these metrics specifcally for the consent interface. Considering

participants’ interactions with the consent interface, it appears that

none of the conditions required signifcantly more efort, with the

exception of the prominence-cornerButton variant. As none of the

participants in that condition attempted to make a consent decision,

we cannot draw conclusions about the amount of efort required.

However the lack of interaction with “Cookie Preferences” button

in that condition raises questions about whether participants even

noticed that it was there. Furthermore, we were only able to summarize participants’ consent decisions for each variant through their

survey responses, which could have been inaccurately reported.

However, manual inspection of the session recordings of our pilot

participants found misreporting to be rare.

Our study also explored cookie consent interfaces in the context

of a single website. Assuch, we cannot provide insight into usability

aspects that may be impacted when users encounter such interfaces

across multiple websites or apps. Future work could more deeply

explore usability issues, such as decision fatigue, as well as the

impact of trust in a particular brand in the context of users’ cookie

consent decision.

Due to time constraints, we aimed for a diverse, though not

necessarily representative, sample for our user study. However,

our results may be impacted by the relatively poor gender and age

diversity of our fnal study population. While we did not fnd that

gender or age signifcantly impacted participants’ consent decisions,

we did observe diferences in user awareness, comprehension, and

sentiment. Female-identifying participants and those under the age

of 35 had less awareness and comprehension of available cookie

options and were less invested in their decision-making, on average,

compared to male-identifying participants or those older than 35.

Technical literacy more generally is likely to difer with gender

and age, as 10.5% of women under 35 in our study sample reported

having a degree or working in a computer-related feld, compared

to 78.7% of men older than 35. Given that oursample was dominated

by participants with less investment in in their decision-making

and lower comprehension of available cookie options, we expect

we may have failed to detect some diferences in conditions that

might be detectable in a more representative study. Future work
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA

evaluating the usability of consent interfaces should be conducted

with a study population that is more representative of the internet

population overall.

5.2 Design Implications

We found that several of the design parameters we explored had a

signifcant impact on the usability of the consent interface. Table 6

provides a summary of our fndings with regards to how they

impacted the six usability aspects studied. Among the seven design

parameters we explored, we fnd that the prominence of the consent

interface, presence of in-line options within the initial screen of the

interface, and presence of a persistent “Cookie Preferences” button

for enabling changes to the initial consent decision had the greatest

impact on usability. These results are in line with prior work which

suggest that more salient privacy information and options yield

better usability outcomes (e.g., [6, 42]).

5.2.1 Prominence of the interface. Our resultsrelated to user awareness in Section 4.4.3 suggest that less conspicuous consent interfaces may not be efective in communicating the presence of privacy

choices. Additionally, as highlighted in Section 4.4.2, without a fullyblocking “consent wall” interface, many participants proceeded to

use the website without indicating a consent decision. Participants

had varying expectations as to how this inaction would be interpreted by the website, as reported in Section 4.4.4, suggesting a

need for non-blocking consent interfaces to clarify the privacy

implications of choosing not to engage. However, fully-blocking

interfaces may come with some usability trade-ofs, particularly

for users who may not know what their consent preference might

be for a particular website or app before they have interacted with

it. It is important for users to be able to get enough information

about the context of this decision, which may be through details

available within the consent interface itself, or from other parts

of the website that are still visible or accessible with the consent

interface overlay.

Ultimately, our fndings on interface prominence highlight a

fundamental problem with applying the consent interface paradigm to such frequently-encountered decisions as cookie consent

decisions. Cookie consent interfaces need to be prominent in order to promote user engagement and informed decisions, but such

prominence necessarily introduces friction into a user’s web browsing experience, decreasing usability. Users who are not privacy

conscious are still forced to take some action and users who do

desire privacy-protective options are burdened with taking the time

needed to actually make an informed decision. Aslong asthe cookie

consent interface paradigm persists, our results suggest that interface prominence may indeed be necessary for achieving informed

consent. However, as we discuss below in Section 5.4, automated

consent mechanisms are likely to ofer a more usable solution.

5.2.2 Presence of in-line options. Our fndings provide further

support to prior recommendations that cookie consent interfaces

should include in-line optionsto better enable user decision-making

[29, 42], but add a new observation that users tend to guess (often

incorrectly) at the meaning of common cookie categories when

they are presented in-line without accompanying defnitions. As

Habib et al.

described in Section 4.4.1, without these options available participants were more likely to allow all cookies. Along these lines, an

additional button option that more clearly corresponds to allowing

only strictly necessary cookies may be useful to users, considering

that over a quarter of participants reported this as their preferred

consent decision. However, we did fnd a negative impact of providing in-line options on participants’ comprehension of choices

when explicitly instructed to revisit the consent interface. Participants in these conditions seemed to be more likely to guess at the

defnitions of the available cookie options rather than review the

defnitions on the “Cookie Preferences” page. This suggests that

providing defnitions of cookie categories within the in-line options,

such as through a tooltip or “accordion” interface, may help with

comprehension of choices and better enable user decision-making.

However, both the addition of a third button option and tooltip

or accordion defnitions for cookie categories should be tested for

their usability impact.

5.2.3 Enabling decision reversal. While the “Cookie Preferences”

button on its own wasinefective for communicating the presence of

available cookie options, our results in Section 4.4.6 revealed that it

did help participantsidentify a meansto change their initial consent

decision. Furthermore, our results suggest that the presence of this

button following an interruptive banner or fully-blocking interface

is enough on its own to facilitate decision reversal, as we found

no signifcant diferences between the reversal-noInstructions and

best-practices variants. Therefore text instructions in the consent

interface related to the “Cookie Preferences” button for decision

reversal could be removed, which would reduce the reading efort

required for the consent interface.

5.2.4 Recommendations for CMPs. While individual websites may

not have the incentives or resources to create more privacy-friendly

consent interfaces, CMPs may be in a position to change website

practices, although they may need to be pressured to do so. Our

inspection-evaluation of CMP-implemented consent interfaces, as

well as prior work [29], make it clear that CMPs are enabling companies to implement consent interfaces with known dark patterns.

Such design options include unequal paths for the most and least

privacy protective options, privacy-invasive default cookie options,

confusing placement of button options, and consent interfaces without any explicit cookie options. As such, CMPs should ensure that

the design parameters they make available to organizations do

not facilitate such dark patterns. Design choices included in our

best-practices variant could guide CMPs in the design options they

provide, particularly our recommendations related to the prominence of the interface, presence of in-line options, and enabling

decision reversal. Furthermore, CMPsshould provide guidance with

regards to the text that is included in the interface, such as avoiding loss aversion framing of choices and suggesting more intuitive

language to use instead of “performance” and “functional” cookies. Last, CMPs should conduct usability assessments similar to

this study, to identify and address potential negative impacts of

the design options they provide to organizations for their consent

interfaces. Some CMPs already ofer templates for compliance with

particular laws, e.g., switching to opt-in rather than opt-out for

GDPR compliance. Regulators may be able to put pressure on these
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

“Okay, whatever”: An Evaluation of Cookie Consent Interfaces

CMPs to do more, requiring CMPs who claim compliance to remove or warn against options that would result in dark patterns or

usability challenges.

5.3 Incentives

In the absence of legal requirements and enforcement related to

usability, websites may not have incentives to make their cookie

consent interfaces more usable or privacy friendly. Thisis especially

true for websites that serve visitors from the US, which was the

focus of our study. In the US, where by default websites can usually

set cookies without restriction, more engagement with cookie consent interfaces may lead to more users opting out of more cookies.

The situation is somewhat diferent under regulatory regimes such

as GDPR where opt-in consent is required [13]. To comply with

GDPR websites must refrain from setting unnecessary cookies until users have opted in. Thus these websites have an incentive to

engage more users in their consent interface so that more of them

will opt-in to additional cookies. Some have even blogged about

A/B testing their consent interface to increase engagement [36].

As privacy legislation is written to require usability and prohibit

dark patterns, there are opportunities for regulators to leverage

these laws to force websites to revise their cookie consent interfaces. Our fndings can guide regulators in identifying potentially

problematic interfaces. Furthermore, our study methods and survey

questions could be used by regulators in their evaluations. Besides

going after individual websites, we suggest that regulators work

with CMPs to ofer guidance and default options for their customers

that result in more privacy-friendly and usable consent interfaces.

5.4 Reducing the Burden of User Consent

While more usable cookie consent interfaces would better enable

users to exercise their consent preferences, this model of consent

still poses a considerable burden on users. Considering that users

must make consent decisions on each website or app they use, aggregated together the cost of reading cookie consent interfaces,

comprehending available options, and making a decision is not trivial. It is likely that users have formed coping strategies to manage

the burden of cookie consent decisions, as over half of participants

in our best-practices condition reported selecting the “easiest option”

when making their consent decision. Therefore, usable consent interfaces are only part of the solution for enabling efective privacy

protection on websites.

Technology could play a role in further reducing the burden of

user consent. As described in Section 2.4, browser-based consent

mechanisms could automate users’ consent decisions so that they

do not have to make consent decisions on every website they visit.

However, such approaches face their own challenges. These automated mechanisms may prove to be more cumbersome to users

who want to make website-specifc consent decisions. Browser

extensions must be installed and managed by users who may be

unwilling or lacking the technical skills to do so. Additionally, the

scheme proposed by the UK’s ICO could sufer the same fate as

the DNT standard, which ultimately failed due to lack of industry

support [11]. However, given existing regulatory requirements related to capturing user consent for data collection, it is possible

that this proposal may have more success; a browser-based consent

CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA

mechanism may be more attractive to industry stakeholders than

consent interfaces that interrupt the user experience. Until alternative consent mechanisms become widely adopted, it is necessary

to ensure that current mechanisms are usable to facilitate efective

privacy protection.

6 CONCLUSION

We conducted a two-part study of cookie consent interfaces, fnding that the design of these interfaces signifcantly impact several

aspects of usability. We frst conducted an inspection-based evaluation of consent interfaces implemented through consent management platforms (CMPs) which identifed design parameters that

organizations can customize for their websites or apps. To explore

which design choices for these parameters result in better usability, we conducted a large-scale between-subjects experiment on

Prolifc evaluating 12 cookie consent design variants. We fnd that

several design choices, such as a “consent wall” implementation

of the consent interface, in-line options corresponding to cookie

categories, and a persistent “Cookie Preferences” button enabling

decision reversal yielded signifcantly better usability outcomes.

Our comprehensive usability assessment of cookie consent interfaces complements prior work in this space by providing actionable

design implications that consider a more holistic perspective of

usability, beyond considerations related to dark patterns.

ACKNOWLEDGMENTS

This research was supported in part by the National Science Foundation under grant CCF-1852260, Carnegie Corporation of New

York, Innovators Network Foundation, and Facebook. Any opinions,

fndings, and conclusions or recommendations expressed in this

material are those of the authors and do not necessarily refect the

views of the National Science Foundation or other funders.

REFERENCES

[1] Dirk Bollen, Bart P Knijnenburg, Martijn C Willemsen, and Mark Graus. 2010.

Understanding Choice Overload in Recommender Systems. In Proceedings of the

Conference on Recommender Systems (RecSys). ACM, 63–70.

[2] Dino Bollinger, Karel Kubicek, Carlos Cotrini, and David Basin. 2022. Automating

Cookie Consent and GDPR Violation Detection. In USENIX Security Symposium.

USENIX.

[3] Harry Brignull. [n. d.]. Types of Dark Patterns: Confrmshaming. https://www.

darkpatterns.org/types-of-dark-pattern/confrmshaming.

[4] Nick Charalambides. 2021. We Recently Went Viral on TikTok - Here’s What We

Learned. https://blog.prolifc.co/we-recently-went-viral-on-tiktok-heres-whatwe-learned/.

[5] Martin Degeling, Christine Utz, Christopher Lentzsch, Henry Hosseini, Florian

Schaub, and Thorsten Holz. 2019. We Value Your Privacy...Now Take Some

Cookies: Measuring the GDPR’sImpact on Web Privacy. In Proceedings of Network

and Distributed System Security Symposium (NDSS). Internet Society.

[6] Nico Ebert, Kurt Alexander Ackermann, and Björn Scheppler. 2021. Bolder is

Better: Raising User Awareness through Salient and Concise Privacy Notices.

In Proceedings of the Conference on Human Factors in Computing Systems (CHI).

ACM.

[7] European Parliament. 2016. Regulation (EU) 2016/679 of the European Parliament

and of the Council. https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=

CELEX:32016R0679.

[8] Yuanyuan Feng, Yaxing Yao, and Norman Sadeh. 2021. A Design Space for

Privacy Choices: Towards Meaningful Privacy Control in the Internet of Things.

In Proceedings of the Conference on Human Factors in Computing Systems (CHI).

ACM.

[9] Joseph L Fleiss, Bruce Levin, and Myunghee Cho Paik. 2013. Statistical Methods

for Rates and Proportions. John Wiley & Sons.

[10] Colin M Gray, Cristiana Santos, Nataliia Bielova, Michael Toth, and Damian

Cliford. 2021. Dark Patterns and the Legal Requirements of Consent Banners:
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA

An Interaction Criticism Perspective. In Proceedings of the Conference on Human

Factors in Computing Systems (CHI). ACM.

[11] Kashmir Hill. 2018. ’Do Not Track,’ the Privacy Tool Used by Millions of People,

Doesn’t Do Anything. Gizmodo (October 2018). https://gizmodo.com/do-nottrack-the-privacy-tool-used-by-millions-of-peop-1828868324.

[12] Maximilian Hils, Daniel W Woods, and Rainer Böhme. 2020. Measuring the

Emergence of Consent Management on the Web. In Proceedings of the Internet

Measurement Conference (IMC). ACM, 317–332.

[13] Horizon 2020 Framework Programme of the European Union. 2021. Cookies, the

GDPR, and the ePrivacy Directive. https://gdpr.eu/cookies/.

[14] IAB Europe. 2021. Transparency & Consent Framework. https://iabeurope.eu/

transparency-consent-framework/.

[15] International Chamber of Commerce UK. 2012. ICC UK Cookie

Guide. https://www.cookielaw.org/wp-content/uploads/2019/12/icc_

uk_cookiesguide_revnov.pdf.

[16] ISO Technical Committee 159. 2018. Ergonomics of Human-System Interaction.

https://www.iso.org/standard/63500.html.

[17] Mark J Keith, Courtenay Maynes, Paul Benjamin Lowry, and Jefry Babb. 2014.

Privacy Fatigue: The Efect of Privacy Control Complexity on Consumer Electronic Information Disclosure. In Proceedings of the International Conference on

Information Systems (ICIS). Association for Information Systems.

[18] Kevel. 2021. Consent Management Platform (CMP) 2021 Tracker. https://www.

kevel.co/cmp/.

[19] Daniel Kladnik. 2021. I Don’t Care About Cookies. https://www.i-dont-careabout-cookies.eu/.

[20] Oksana Kulyk, Annika Hilt, Nina Gerber, and Melanie Volkamer. 2018. “This

Website Uses Cookies”: Users’ Perceptions and Reactionsto the Cookie Disclaimer.

In European Workshop on Usable Security (EuroUSEC).

[21] Victor Le Pochat, Tom Van Goethem, Samaneh Tajalizadehkhoob, Maciej Korczyński, and Wouter Joosen. 2019. Tranco: A Research-Oriented Top Sites

Ranking Hardened Against Manipulation. In Proceedings of Network and Distributed System Security Symposium (NDSS). Internet Society.

[22] Timothy Patrick Libert. 2017. Track The Planet: A Web-Scale Analysis Of How Online Behavioral Advertising Violates Social Norms. Ph. D. Dissertation. University

of Pennsylvania, Philadelphia, PA.

[23] Dominique Machuletz and Rainer Böhme. 2020. Multiple Purposes, Multiple

Problems: A User Study of Consent Dialogs after GDPR. Proceedings on Privacy

Enhancing Technologies (PoPeTs) 2 (2020), 481–498.

[24] Arunesh Mathur, Jonathan Mayer, and Mihir Kshirsagar. 2021. What Makes a Dark

Pattern...Dark? Design Attributes, Normative Considerations, and Measurement

Methods. In Proceedings of the Conference on Human Factors in Computing Systems

(CHI). ACM.

[25] Célestin Matte, Nataliia Bielova, and Cristiana Santos. 2020. Do Cookie Banners

Respect My Choice?: Measuring Legal Compliance of Banners from IAB Europe’s

Transparency and Consent Framework. In IEEE Symposium on Security and

Privacy (SP). IEEE, 791–809.

[26] Minimal Consent. 2020. https://www.minimal-consent.com/.

[27] Peter Morville. 2004. User Experience Design. http://semanticstudios.com/user_

experience_design/.

[28] Jakob Nielsen. 2012. Usability 101: Introduction to Usability. https://www.

nngroup.com/articles/usability-101-introduction-to-usability/.

[29] Midas Nouwens, Ilaria Liccardi, Michael Veale, David Karger, and Lalana Kagal.

2020. Dark Patterns after the GDPR: Scraping Consent Pop-Ups and Demonstrating Their Infuence. In Proceedings of the Conference on Human Factors in

Computing Systems (CHI). ACM.

[30] European Court of Justice. 2018. Judgment of the Court (Grand Chamber) of 1

October 2019: Bundesverband der Verbraucherzentralen und Verbraucherverbände - Verbraucherzentrale Bundesverband e.V. v Planet49 GmbH. Ofcial

Journal of the European Union (March 2018). https://eur-lex.europa.eu/legalcontent/en/TXT/PDF/?uri=uriserv%3AOJ.C_.2018.112.01.0009.01.ENG.

[31] Ofce of the California Attorney General. 2019. The California Privacy Rights

and Enforcement Act of 2020. https://oag.ca.gov/system/fles/initiatives/pdfs/19-

0017%20%28Consumer%20Privacy%20%29.pdf.

[32] Ofce of the California Attorney General. 2020. California Consumer Privacy

Act (CCPA): Final Text of Proposed Regulations. https://oag.ca.gov/sites/all/

fles/agweb/pdfs/privacy/oal-sub-fnal-text-of-regs.pdf.

[33] Whitney Quesenbery. 2004. Balancing the 5Es: Usability. Cutter IT Journal

(February 2004). http://whitneyquesenbery.com/articles/5es-citj0204.pdf.

[34] Maksim Say-fo. 2020. Buttons Alignment Policy. https://uxplanet.org/buttonsalignment-policy-a26de4ce0c70.

[35] Florian Schaub and Lorrie Faith Cranor. 2020. Usable and Useful Privacy Interfaces. In An Introduction to Privacy for Technology Professionals, Travis Breaux

(Ed.). IAPP, 176–299.

[36] Christopher Schepelle. 2020. Despite GDPR: Up to 70% Analytics

Opt-in rates – Why Extensive Testing is Worth Every Minute of Effort. https://analytics.dhl.com/despite-gdpr-up-to-70-analytics-opt-in-rateswhy-extensive-testing-is-worth-ervery-minute-of-efort/.

Habib et al.

[37] Ulrich Schmidt and Horst Zank. 2005. What is Loss Aversion? Journal of Risk

and Uncertainty 30, 2 (2005), 157–167.

[38] Mark Scott and Laurens Cerulus. 2018. Europe’s New Data Protection Rules Export Privacy Standards Worldwide. Politico (January

2018). https://www.politico.eu/article/europe-data-protection-privacystandards-gdpr-general-protection-data-regulation/.

[39] Than Htut Soe, Oda Elise Nordberg, Frode Guribye, and Marija Slavkovik. 2020.

Circumvention by Design - Dark Patterns in Cookie Consent for Online News

Outlets. In Proceedings of the Nordic Conference on Human-Computer Interaction

(NordiCHI).

[40] UK Information Commissioner’s Ofce. 2021. ICO to Call on G7 Countries to

Tackle Cookie Pop-ups Challenge. https://ico.org.uk/about-the-ico/news-andevents/news-and-blogs/2021/09/ico-to-call-on-g7-countries-to-tackle-cookiepop-ups-challenge/.

[41] United States Census Bureau. 2020. 2020 Demographic Analysis Estimates Press

Kit. https://www.census.gov/newsroom/press-kits/2020/2020-demographicanalysis.html.

[42] Christine Utz, Martin Degeling, Sascha Fahl, Florian Schaub, and Thorsten Holz.

2019. (Un)informed Consent: Studying GDPR Consent Notices in the Field. In

Proceedings of the Conference on Computer and Communications Security (CCS).

ACM, 973–990.

[43] W3C Web Accessibility Initiative. 2018. Web Content Accessibility Guidelines

(WCAG) 2.1. https://www.w3.org/TR/WCAG21/.

[44] W3C Working Group. 2019. Tracking Preference Expression (DNT). https:

//www.w3.org/TR/tracking-dnt/.

[45] Chauncey Wilson. 2013. User Interface Inspection Methods: A User-Centered Design

Method. Newnes.
 

 

 

 

This website uses cookies 

• Why we use cookies: To make the website work properly, enhance 

performance, create personalized functionality, and deliver targeted ads 

• Your choices: Some cookies are optional-click 'Allow all cookies" to accept 

all cookies, or select only the cookies you want and click 'Allow selected 

• Updating preferences: Click on the 'Cookie Preferences' bunon on the 

bottom right corner ol this website 

• More information: Please see our .erh'ac.Y..£nli.c.1t 

1-iiMIMMM OMl@iii◄ 

0 Strictly necessary O Performance O Functional O Targeting ~ 

Biiiliiiiiii ◄ 

Cookie preferences 

.aoo,cos ,,..Eo,,n >AC<rl) CUSfOMf•CU( Con<wu, N>O .. ••TION Mu,on @ ,, f 1 .. ..,,,,......,, .,,.._ .,.& ... ..,.,,. c ...... 

"""'"'"'._o Shopp,"" To,m, I Condot.,.,, 

w,, .. , • ..,., • .,0,,.,, .. ...,, ....... ., •• ""°"" ::":~•°"•t

•"0>,.. 10 .,., FM Coo>oPolq 

“Okay, whatever”: An Evaluation of Cookie Consent Interfaces CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA

A CONSENT INTERFACE DESIGN VARIANTS

best-practices (on landing page)

All fully-blocking variants were the same dimensions on the landing page as best-practices.

prominence-cornerButton
 

 

Cookie Preferences 

What are cookies? 

Cookies are small data files placed on your computer or internet enabled device by 

websites in order to add functions to the site or to help the website work better. 

Sometimes cookies are necessary to be used for the basic functioning of your site, 

and sometimes they can be used to identify your computer or internet device to our 

website, or to other third-party websites. This may allow us to improve the way we 

work, such as remembering preferences expressed by you, or tracking your use of a 

website for statistical analysis and advertising. 

For more information about cookies, please see here: hll12.S,;.ll 

www.allaboutcookies orgl 

Strictly necessary cookies Always active 

These cookies are necessary for the website to function and can't be switched off in 

our systems. They are usually only set in response to actions you have taken which 

result in a request for services, such as setting your privacy preferences, logging in or 

filling in forms. You can set your browser to block or alert you about these cookies, 

but some parts of the site may not work as a result. 

Performance cookies Deny ~ Allow 

These cookies allow us to count visits and traffic sources so we can measure and 

improve t he performance of our site. They help us know which pages are the most 

and least popular and see how visitors move around the site. All the information these 

cookies collect is aggregated and therefore anonymous. If you do not allow these 

cookies, information from these cookies will not be used to help site performance. 

Functional cookies Deny ~ Allow 

These cookies allow the provision of enhanced functionality and personalization. 

They may be set by us or by third-party providers whose services we have added to 

our pages. If you don't allow these cookies, then some or all of these functionalities 

may not function properly. 

Targeting cookies Deny ~ Allow 

These cookies are set through our site by our advertising partners. They may be used 

by those companies to build a profile of your interests and show you relevant ads on 

other sites. They work by uniquely identifying your browser and device. If you don't 

allow these cookies, you will not experience our targeted advertising across different 

websites as a result of these cookies. 

Save preferences 

X 

g:iggn ' l u. >LL • LC "- Can 

Hif'iiiiiiif 

~"" This website uses cookies 

• Y/hy we uu cookles: TO na<e tM ,'11:!bSlte work proper y, enhance perrormance, create persona112110 fuJ'IC: or,aII1y and aell'IE!r 1arqe'.ed a~s 

- Your choices: ~om:: coot: cs ore option:il click 'A.II0,'1 :11 1 cook cs· :o accept .ii cookies, or ~c c~t cnly Mc cJoKics youw:irt ,ind c ick 'Allow 5:::lcc:cd cJo<ic5 

• Upd11ting preferences: ClIc<on the 'CooKI~ Preferer ces button on :he botto-n • gh: corner ol th s v.et s te 

, More information: Pl~~!;I' !'t i'! rnr Ptill<'H:Y P~li-)' 

~ Strii::t1y ntr.t~~.lll)' D P!'r' nrm.1Jnr.l' D F.inr.tinMI D -.llrgl"tirg Shm!t...d.t1.ai i·iiM@&·HffiM M ·iiMIM·iliM 

"::,JTuS ••""• "~c.o _,,,,,..,ouo c,,,,.nu, 11,0,.....,.,c~ ~"'" ~ W f ,,,,., ,,,~,, ,,., "'J • .,, ... , ,.,e,,,, .....,.,.,,.., ,.,,., ,,,. .. ..., tun ~von<I .. •> 

~ ... , ............... .... , """''" '"""""'" , .. ~,,,.,~,~ ...... •""•• '"" , .. ,,,..,. , 

CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA Habib et al.

Clicking the button led to the single-layer “Cookie Preferences” page.

prominence-banner
 

Q Q This website uses cookies 

• Why we use cookies: To make the website work properly, enhance 

performance, create personalized functionality, and deliver targeted ads 

• Your choices: Some cookies are optional- click 'Allow all cookies' to accept 

all cookies, or edit your cookie preferences 

• Updating preferences: Click on the 'Cookie Preferences' button on t he 

bottom right corner of this website 

• More information: Please see our Privac:r Policy_ 

Allow all cookies 

Q Q This website uses cookies 

• Why we use cookies: To make the website work properly, enhance 

performance, create persona II zed runctlonallty, and deliver targeted ads 

• Your choices: some cookies are optional- click 'Allow all cookies' to accept 

all cookies, or click 'Edit cookie preferences' to customize your cookies 

• Updating preferences: Click on the 'Cookie Preferences' button on the 

bottom right corner of this website 

• More information: Please see our Privacy Polic~ 

Edit cookie preferences Allow all cookies 

Q Q This website uses cookies 

• Why we use cookies: To make t he website work properly, enhance 

performance, create personalized functional ity, and deliver targeted ads 

• Your choices: Note that if you do not accept opt ional cookies, your experience 

may be affected- click 'Allow all cookies' to accept all cookies, or select only the 

cookies you want and click 'Allow selected cookies' 

• Updating preferences: click on the 'Cookie Preferences' butt on on the bottom 

right corner of this website . 

• More information: Please see our Privacy Policy. 

Al low selected cookies Allow all cookies 

0 Strictly necessary O Performance O Functional O Targeting Show details 

“Okay, whatever”: An Evaluation of Cookie Consent Interfaces CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA

options-embeddedLink

options-interfaceButton

text-lossAversion
 

Q Q This website uses cookies 

We use cookies to make the website work properly, enhance performance, 

create personalized functionality, and deliver targeted ads. Some cookies 

are optional- click 'Allow all cookies' to accept all cookies, or select only the 

cookies you want and click 'Allow selected cookies.' To update your 

preferences at any t ime, click on the 'Cookie Preferences' button on the 

bottom right corner of this website. For more information, please see our 

Privacy Policy. 

Allow selected cookies Allow all cookies 

0 Strictly necessary O Performance O Functional O Targeting Show details 

Q Q This website uses cookies 

• Why we use cookies: To make the website work properly, enhance 

performance, create personalized functionality, and deliver targeted ads 

• Your choices: some cookies are optional- click 'Okay' to accept all 

cookies, or edit your cookie preferences below and select 'Submit' 

• Updating preferences: Click on the 'Cookie Preferences' button on t he 

bottom right corner of this website 

• More information: Please see our Privacy Policy 

-wMfl■ w+;;w 

0 Strictly necessary O Performance O Functional O Targeting Show details 

Q Q This website uses cookies 

• Why we use cookies: To make the website work properly, enhance 

performance, create personalized functionality, and deliver targeted ads 

• Your choices: some cookies are optional-click 'Allow all cookies' to accept 

all cookies, or click 'Edit cookie preferences' to customize your cookies 

• Updating preferences: Click on the 'Cookie Preferences' button on t he 

bottom right corner of this website 

• More information: Please see our Privacy Policy 

Edit cookie preferences Allow all cookies 

CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA Habib et al.

text-layoutParagraph

button-generic

layout-multilayer
 

 

Cookie Preferences 

What are cookies? 

cookies are small data files placed on your computer or 

internet enabled device by websites in order to add functions 

to the site or to help the website work better. Sometimes 

cookies are necessary to be used for the basic functioning of 

your site, and sometimes they can be used to identify your 

computer or internet device to our website, or to other thirdparty websites. This may allow us to improve the way we work, 

such as remembering preferences expressed by you, or 

tracking your use of a website for statistical analysis and 

advert ising. 

For more information about cookies, please see here: 

bllD..S..:I/www allaboutcookies.orgf 

save preferences 

Cookie Preferences 

Performance cookies Deny~ Allow 

These cookies allow us to count visits and traffic sources so 

we can measure and improve the performance of our site. 

They help us know which pages are the most and least 

popular and see how visitors move around the site. All t he 

information these cookies collect is aggregated and therefore 

anonymous. If you do not allow these cookies, information 

from these cookies will not be used to help site performance. 

Save preferences 

~~g~n· Cookie Preferences 

Targeting cookies 

Cookie Preferences 

Strictly necessary cookies Always active 

These cookies are necessary for the website to function and 

can't be switched off in our systems. They are usually only set 

in response to actions you have taken which result in a 

request for services, such as setting your privacy 

preferences, logging in or filling in forms. You can set your 

browser to block or alert you about these cookies, but some 

parts of the site may not work as a result. 

save preferences 

Cookie Preferences 

Functional cookies Deny~ Allow 

These cookies allow the provision of enhanced functionality 

and personalization. They may be set by us or by third-party 

providers whose services we have added to our pages. If you 

don't allow these cookies, then some or all of these 

funct ionalities may not function properly. 

Save preferences 

Deny~ Allow 

Targeting cookies 

These cookies are set through our site by our advertising 

partners. They may be used by those companies to build a 

profile of your interests and show you relevant ads on other 

sites. They work by uniquely identifying your browser and 

device. If you don't allow these cookies, you wilt not 

experience our targeted advertising across different websites 

as a result of these cookies. 

Save preferences 

“Okay, whatever”: An Evaluation of Cookie Consent Interfaces CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA

The “Edit cookie preferences” button links to the multi-layer version of “Cookie Preferences” page.
 

 

Q Q This website uses cookies 

• Why we use cookies: To make the website work properly, enhance 

performance, create personalized functionality, and deliver targeted ads 

• Your choices: Some cookies are optional- click 'Allow all cookies' to accept 

all cookies, or select only the cookies you want and click 'Allow selected 

• Updating preferences: You can update your cookie preferences at any time 

• More information: Please see our Privacy Polic~ 

Al low selected cookies Allow all cookies 

0 Strictly necessary O Performance O Funct ional O Targeting Show details 

Q Q This website uses cookies 

• Why we use cookies: To make the website work properly, enhance 

performance, create personalized functionality, and deliver targeted ads 

• Your choices: Some cookies are optional-click 'Allow all cookies' to accept 

all cookies, or select only the cookies you want and click 'Allow selected 

• Updating preferences: Click on the 'Cookie Policy' link in the footer of this 

website 

• More information: Please see our Privac~ Polic~ 

Allow selected cookies Allow all cookies 

0 Strictly necessary O Performance O Funct ional O Targeting Show details 

EHHIIHE 

'~ This website uses cookies WI' Ulif! cnckiP.,;: In "1:tke the web!'lite wmk p-~perl~·. ertu~ :,t>rlc1rnw1:P., :rl!ilte per~~n;1 i11~::l ' t.nc1,o-,ality, and rle 1.,,er · ;l'ge .e::1 ;id,;. hlc:IP. \ h;i· if ~·nu do 

n

~ 

ot d::-:ept Jpt omrl ::ooK.es, yol.lr e).perience- mdfbe affec•.e-d. ~r contir!J ny to J:;e t1i~ site )'OU ay·ee tJ tt1 e use cf c:iokies <1s $l10W'1 ir yJur ~ 

n,u rrJy L.pdJtC' yoLr cookiC' prc-fc-rcncc-s Jt Jn~ time. ~or m:irt- inform.:itio", plcJsc sec our~:. 

"... ., ... ""' ' ' .,,,,., " .... '"'""'" • .,_ @ " f V"' " • t_.,,, , u ., •• . . ,., _ ,.,.., .,,.,,... t ' "\I f-,~-• ""'" ' 

,, II• l~> ~ lhodt ~ I, , , , ,~.,., > , 

~:;,.,' ~ :n:-.,,, ~ F•C M" 0 I 

CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA Habib et al.

reversal-noInstructions

reversal-cookiePolicy

worst-practices (on landing page)
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

“Okay, whatever”: An Evaluation of Cookie Consent Interfaces

B SURVEY QUESTIONS

Questions with standard bullets were single-select, while those

with squares allowed for multiple selections.

(1) Were you able to successfully complete the task?

• Yes, I added a product to my cart

• No, I skipped the task

• No, I had a technical problem (please describe)

[Logic for Q2: If “Were you able to successfully complete the task?”

= “Yes, I added a product to my cart” (participants who answered

“No” were directed to Q3)]

(2) Which product did you select?

• Grandma’s Diner Special

• Adult Sippy Cup

• Delicate Irish Cofee

• Afternoon Tea

• The Minalimist

• Stemware Essentials

• Bamboo Tea Set

• Insulated Espresso Cup

• Café Classic Set

• I don’t remember

(3) How easy or difcult was it to shop on this website?

(5-point Likert scale from “Very easy” to “Very difcult”)

(4) Do you recall making any privacy-related decisions during

your interaction with the cups n’ such website?

• Yes

• No

• Not sure

[Logic for Q5: If “Do you recall making any privacy-related

decisions during your interaction with the cups n’ such...” = “Yes”]

(5) What was this decision about?

• The use of cookies on the website

• The creation of a username and password for the website

• The visibility of credit card info on the website

• The use of location data while shopping on the website

• Other (please describe)

(6) When visiting cups n’ such’s website, you might have seen

an interface related to the use of cookies. Which option(s)

do you remember selecting? (Select all that apply)

□ Allow all cookies

□ Allow only strictly necessary cookies

□ Allow social media cookies

□ Allow performance cookies

□ Allow functional cookies

□ Allow targeting cookies

□ Don’t allow any cookies

□ I don’t remember

□ I didn’t select any options related to the use of cookies

[Logic for Q7 and Q8: “If When visiting cups n’ such’s website, you

might have seen an interface related to the use of cook...” != “I

don’t remember” And “When visiting cups n’ such’s website, you

might have seen an interface related to the use of cook...” != “I

didn’t select any options related to the use of cookies”]

(7) What do you expect to happen since you selected [consent

decision in Q6]? (open-ended question)

CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA

(8) What were you trying to achieve when you selected

[consent decision in Q6]? (open-ended question)

[Logic for Q9-Q11: “If When visiting cups n’ such’s website, you

might have seen an interface related to the use of cook...” != “I

didn’t select any options related to the use of cookies”]

(9) Which of the following best describes how you made your

decision related to the use of cookies on the cups n’ such

website?

• I picked an option based on my actual cookie preferences

• I picked whichever option seemed easiest so the consent

interface would go away

• I picked an option randomly

• Other (please describe)

(10) How carefully did you consider the options related to

cookies on the cups n’ such website?

• Not at all carefully

• Moderately carefully

• Extremely carefully

(11) The cookie notice interface included some text. What did

you do when you saw it?

• Skipped over it

• Skimmed it

• Read it carefully

(12) What options related to cookies do you recall being

available to you on this website? (each option rated as

“Defnitely not available,” “Probably not available,” “Not sure

if available,” “Probably available,” “Defnitely available”)

• Allow all cookies

• Allow only strictly necessary cookies

• Don’t allow any cookies

• Allow social media cookies

• Allow performance cookies

• Allow functional cookies

• Allow targeting cookies

Instructions: Please select the defnition that fts best for

each of the following terms.

(13) In the context of the web, what is a cookie?

• A security token for two-factor authentication

• A small piece of data stored on a computer to keep track

of information such as logins or websites the user has

visited previously

• A memorized secret used to confrm the identity of a user

• A unique string of numbers separated by periods that

identifes each computer using the Internet Protocol to

communicate over a network

• I don’t know

(14) What are strictly necessary cookies?

• Cookies that are needed for the website to work properly

• Cookies that are needed for collecting certain metrics

• Cookies that are needed for determining your location

• I don’t know

(15) What are performance cookies?

• Cookies that help measure and improve website features

• Cookies that are given priority over other cookies on the

website

• Cookies that make the website run faster
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA

• I don’t know

(16) What are functional cookies?

• Cookies that are needed for the website to work properly

• Cookies that help personalize the website’s services for

you

• Cookies that are given lower priority than other cookies

on the website

• I don’t know

(17) What are targeting cookies?

• Cookies that are used for delivering personalized

advertisements

• Cookies that help users navigate the website

• Cookies that are needed for determining your location

• I don’t know

(18) How likely do you think the following scenarios are to

happen if you don’t make a selection regarding the use of

cookies on this website? (each option rated as “Defnitely

not,” “Probably not,” “Not sure,” “Probably yes,” “Defnitely

yes”)

• I would be blocked from using the website entirely

• No cookies would be allowed so the website would not

work at all

• No cookies would be allowed but the entire website

would still work

• All cookies would be allowed and the entire website

would still work

• No cookies would be allowed but some parts of the

website would still work

Instructions: Open the prototype again in a new tab by clicking the

link below and keep it open for the remainder of the survey. Please

answer the following questions after you review your options

related to cookies. [Link to prototype ]

(19) You may have seen several cookie options on the prototype

website. What additional options related to cookies would

you like to have available to you, if any? (open-ended

question)

Instructions: Next, we are going to ask some of questions again

with your previous answers marked. After reviewing the

information provided about the use of cookies on the website,

please edit your answers if you need to.

[Questions 6, 7, 13-18 are asked again]

Instructions: Please answer the following questions, referring to

the prototype if necessary.

(20) What option related to cookies do you think the website is

recommending?

• Allow all cookies

• Allow selected cookies

• Strictly necessary

• Other (please describe)

• The website isn’t recommending any options (all options

are presented equally)

(21) How easy or difcult do you fnd the cookie consent

interface to understand? (6-point Likert scale from “Very

easy” to “Very impossible”)

Habib et al.

(22) What would be your preferred cookie consent decision for

this website? (Select all that apply)

□ Allow all cookies

□ Allow only strictly necessary cookies

□ Allow social media cookies

□ Allow performance cookies

□ Allow functional cookies

□ Allow targeting cookies

□ Don’t allow any cookies

□ Other (please describe)

(23) How easy or difcult would it be for you to make your

preferred cookie consent decision? (6-point Likert scale

from “Very easy” to “Very impossible”)

(24) To what extent do you feel... (each option rated as “Not at

all,” “Moderately,”“Extremely,”“Not sure”)

• Informed about the data being collected by cookies on

this website?

• That this cookie consent interface provides the choices

you want related to the use of your data?

• Informed about your choices related to cookies on this

website?

• Capable of making a decision related to cookies on this

website?

[Logic for Q25: “If When visiting cups n’ such’s website, you might

have seen an interface related to the use of cook...” != “I didn’t

select any options related to the use of cookies”]

Instructions: The following questions refer to “your cookie consent

decision” which refers to the decision you made about the use of

cookies on cups n’ such the frst time you encountered the cookie

consent interface.

(25) To what extent do you feel... (each option rated as “Not at

all,” “Moderately,”“Extremely,”“Not sure”)

• Confdent that your cookie consent decision was the best

option for you?

• Comfortable about how data associated with cookies will

be used on this website, given your cookie consent

decision?

• That your cookie consent decision will be honored by the

website?

Instructions: Please refer to the screenshot below for the following

questions. (Screenshot of landing page with or without “Cookie

Preferences” button depending on the condition)

(26) Suppose you already made a decision about how cookies

can be used on this website. What would you do if you

wanted to change your cookie consent decision, or make a

decision if you didn’t when frst visiting the website?

(open-ended question)

(27) What would you do if what you described in your previous

answer was not available on the website? (open-ended

question)

(28) Which parts of the website did you click on or look at when

reviewing your options related to cookies to answer the

survey questions? (Select all that apply)

□ Cookie policy

□ Privacy policy
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

“Okay, whatever”: An Evaluation of Cookie Consent Interfaces CHI ’22, April 29-May 5, 2022, New Orleans, LA, USA

□ Cookie consent interface (banner or popup that appeared

when opening the website)

□ Cookie preferences button (button on bottom right

corner of the page)

□ Cookie preferences page (page with toggles next to

cookie categories)

□ Other (please describe)

(29) How frequently do you shop online?

• Never

• Less than once a month

• A few times a month

• A few times a week

• Almost every day

(30) What is your age in years? Enter “0” if you prefer not to

respond.

(31) How do you describe your gender identity?

• Male

• Female

• Non-binary

• Agender

• Genderqueer

• Prefer to self-describe

• Prefer not to respond

(32) How do you describe your race or ethnic identity? (You may

select more than one option.)

□ American Indian or Alaska Native

□ Asian

□ Black

□ Native Hawaiian or Pacifc Islander

□ White

□ Prefer to self-describe

□ Prefer not to respond

(33) What is the highest level of school you have completed or

the highest degree you have received?

• Less than high school degree

• High school graduate (high school diploma or equivalent

including GED)

• Some college but no degree

• Associate degree in college (2-year)

• Bachelor’s degree in college (4-year)

• Master’s degree

• Doctoral degree

• Professional degree (JD, MD)

• Prefer not to respond

(34) What was your approximate household income in 2020?

Please answer based on your entire household’s income in

2020, before taxes.

• Less than $10,000

• $10,000 to $19,999

• $20,000 to $29,999

• $30,000 to $39,999

• $40,000 to $49,999

• $50,000 to $59,999

• $60,000 to $69,999

• $70,000 to $79,999

• $80,000 to $89,999

• $90,000 to $99,999

• $100,000 to $149,999

• $150,000 or more

• Prefer not to respond

(35) Do you have a formal education in a computer-related feld,

such as computer science or IT? ("Formal education" could

mean a completed degree or certifcate, or classes or

trainings you took towards a degree or certifcate.)

• Yes

• No

(36) Do you have work experience in a computer-related feld,

such as computer science or IT?

• Yes

• No

(37) Please enter your Prolifc ID again.

(38) If you have any feedback on the survey or cookie consent

interface you saw, please leave it here. (open-ended

question)