# The Effect of Design Patterns on (Present and Future) Cookie Consent Decisions: Supplemental Materials

Nataliia Bielova, Laura Litvine, Anysia Nguyen, Mariam Chammat, Vincent

Toubiana, Estelle Hary

► To cite this version:

Nataliia Bielova, Laura Litvine, Anysia Nguyen, Mariam Chammat, Vincent Toubiana, et al.. The Effect of Design Patterns on (Present and Future) Cookie Consent Decisions: Supplemental Materials. USENIX 2024 - 33rd USENIX Security Symposium, Aug 2024, Philadelphia, United States. pp.1-9. hal-04235032

HAL Id: hal-04235032

https://inria.hal.science/hal-04235032

Submitted on 10 Oct 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>
# The Effect of Design Patterns on (Present and Future) Cookie Consent Decisions:
## Supplemental Materials

Nataliia Bielova*

Inria research centre at Université Côte d'Azur

Anysia Nguyen

Behavioural Insights Team (BIT)

Vincent Toubiana†

Commission Nationale de l'Informatique et des Libertés (CNIL)

Laura Litvine

Behavioural Insights Team (BIT)

Mariam Chammat

Interministerial Directorate for Public Transformation (DITP)

Estelle Hary‡

RMIT University

This document provides supplemental materials directly cited in the proceedings of the USENIX Security Symposium 2024 entitled “The Effect of Design Patterns on (Present and Future) Cookie Consent Decisions” by Nataliia Bielova, Laura Litvine, Anysia Nguyen, Mariam Chammat, Vincent Toubiana and Estelle Hary.

provider before participants enter the survey itself. This allowed us to better qualify the population participating in the survey and ensure that it is as similar as possible to adult users residing in France (see Section 3 of the main paper). Table 2 provides an overview of demographics of our participants.

## A Consent banners design variants

Figure 1 shows designs of all banners in our experiment. While banners used in the study were in French language, here we provide English translation for all banners.

## B Screenshots of the task

When participants land on the fictitious e-commerce website, they first see a cookie banner they’d have to interact with before being able to continue to the website. Figure 2 presents screenshots of examples of what the participants would see for one of the three fictitious websites offering to buy Bluetooth speakers.

## C Identifying behavioral levers

Table 1 shows 18 potential designs to be tested, together with the results of the voting described in Section 3.2 of the main paper.

## D User demographics

Socio-demographic data such as age, gender and professional status was automatically collected through the Cint panel

## E Results of all survey questions

We provide all the results of the survey questions presented in Appendix A of the main paper. Below is the mapping of the survey questions to the resulting tables:

* Q1: Table 3

* Q2: Table 4

* Q3: Table 5, first line

* Q4: Table 5, second line

* Q5: Table 6

* Q6: Table 7

* Q7: because of small sample size (only 66 participants have chosen to personalise), the collected data are not exploitable and interpretable and we therefore omit to present the results here.

* Q8: Table 8

* Q9: Table 9

* Q10: Table 10

* Q11: Table 11

* Q12: Table 12

* Q13: Table 13

* Q14: Table 14

<footnote>* The work was primary carried out while Nataliia Bielova was a Senior Privacy Fellow at the LINC lab of the CNIL in 2021-2022.</footnote>

<footnote>† The views and opinions expressed in this paper do not necessarily reflect the views of the CNIL or any individual Commissioner.</footnote>

<footnote>‡ The work was carried out while Estelle Harry was a designer at the LINC lab of the CNIL before May 2023.</footnote>
<figure><img src="image_2.png" /><figcaption>Figure 1: Designs of all six consent banners evaluated in this study, translated into English.</figcaption></figure>
<figure><img src="image_3.png" /><figcaption>(a) Before interacting with the cookies banner.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>(b) After interacting with the cookie banner.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>(c) Example of information participants can find on the website.</figcaption></figure>
<table><caption>Table 1: 18 potential designs to be tested, together with the results of the voting.</caption><thead><tr><th>Objective</th><th>Intervention</th><th>Ranking following the vote</th></tr></thead><tbody><tr><td colspan="3"><strong>Improve the knowledge of Web users</strong></td></tr><tr><td rowspan="4">Simplify</td><td>Shorten, summarise, clarify text / include an FAQ</td><td>3</td></tr><tr><td>Use rule of thumbs to simplify decision</td><td>13</td></tr><tr><td>Use images, emojis, infographics</td><td>1</td></tr><tr><td>Use visual aids that can be used as heuristics (e.g. traffic lights)</td><td>7</td></tr><tr><td rowspan="4">Make refusing / personalising attractive</td><td>Renew banners regularly</td><td>17</td></tr><tr><td>Increase feeling of importance and urgency to interact with the banner</td><td>14</td></tr><tr><td>Use personalised feedback to explain to users the implications of their choice</td><td>12</td></tr><tr><td>Highlight consequences of their choice</td><td>2</td></tr><tr><td>Encourage web designers to display disadvantages and not only advantages of accepting</td><td>11</td></tr><tr><td>Make it social</td><td>Use social norms</td><td>18</td></tr><tr><td rowspan="2">Make it timely</td><td>Prevent all interactions with the website until users make their choice</td><td>16</td></tr><tr><td>Introduce a pop-up that checks whether users really want to accept cookies</td><td>8</td></tr><tr><td colspan="3"><strong>Improve access to refusal option</strong></td></tr><tr><td rowspan="4">Simplify refusal</td><td>Display both refuse and accept option similarly</td><td>9</td></tr><tr><td>Make refusing more visible and attractive to choose</td><td>5</td></tr><tr><td>Recommend that consent be asked every 6 months, independently of their past choices</td><td>10</td></tr><tr><td>Promote plug-ins that automatically deal with consent banners</td><td>15</td></tr><tr><td rowspan="2">Make decline an option by default</td><td>Pre-select "Refuse all"</td><td>4</td></tr><tr><td>Put "Refuse all" on the first layer, and "Accept all" on the second</td><td>6</td></tr></tbody></table>

## F Regression tables

We provide results for all regression variables in Tables 15 for consent decisions on the first website, Table 16 for average decision time on the first website, and Table 17 for consent decisions on the third website.

## G Consent decisions for desktop vs. mobile

Table 18 presents the decline/personalise rate for desktop and mobile participants.

<table><caption>Table 2: Summary of participant demographics.</caption><thead><tr><th>Characteristics</th><th>Percent (%)</th></tr></thead><tbody><tr><td><strong>Gender</strong></td><td></td></tr><tr><td>Women</td><td>52%</td></tr><tr><td><strong>Age</strong></td><td></td></tr><tr><td>18-30</td><td>24%</td></tr><tr><td>31-49</td><td>34%</td></tr><tr><td>50+</td><td>42%</td></tr><tr><td><strong>Education</strong></td><td></td></tr><tr><td>&gt;Bac + 3</td><td>33%</td></tr><tr><td><strong>Annual household revenue</strong></td><td></td></tr><tr><td>&gt;40 000€</td><td>65%</td></tr><tr><td><strong>Employment status</strong></td><td></td></tr><tr><td>Employed</td><td>71%</td></tr><tr><td>Retired</td><td>15%</td></tr><tr><td>Student</td><td>6%</td></tr><tr><td>Unemployed</td><td>5%</td></tr><tr><td>Other</td><td>2%</td></tr><tr><td><strong>Urban-Rural</strong></td><td></td></tr><tr><td>Rural</td><td>35%</td></tr><tr><td>Sub-urban</td><td>22%</td></tr><tr><td>Urban</td><td>43%</td></tr></tbody></table>
<table><caption>Table 3: Participants’ level of comfort with sharing data (N = 4,026).</caption><thead><tr><td>Level of comfort</td><td>Percentage</td></tr></thead><tbody><tr><td>Comfortable even without knowing how it is used</td><td>52%</td></tr><tr><td>Not comfortable in any given case</td><td>31%</td></tr><tr><td>Comfortable only if knowing how it is used</td><td>17%</td></tr></tbody></table>

<table><caption>Table 4: Participants’ high level of comfort (comfortable/very comfortable) regarding sharing various types of data (N = 4,026).</caption><thead><tr><td>Type of data</td><td>Cumulative percentage for "comfortable" and "very comfortable" level</td></tr></thead><tbody><tr><td>Past product purchases</td><td>74%</td></tr><tr><td>Email address</td><td>47%</td></tr><tr><td>Browsing history</td><td>46%</td></tr><tr><td>Postal address</td><td>45%</td></tr><tr><td>Social media</td><td>43%</td></tr><tr><td>Geolocation</td><td>42%</td></tr><tr><td>Phone number</td><td>36%</td></tr><tr><td>Contacts</td><td>36%</td></tr><tr><td>Bank data</td><td>14%</td></tr></tbody></table>
<table><caption>Table 5: Participants' recall of choice and banners. One star indicates that this value is significantly different from the highest value at least to the 5% level.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Overall</th><th rowspan="2">Control</th><th colspan="2">Dark Pattern</th><th colspan="3">Behavioral Lever</th></tr><tr><th>No reject</th><th>Highlighted accept</th><th>Consequences</th><th>Tricolor</th><th>Highlighted decline</th></tr></thead><tbody><tr><td>Remembers their choice</td><td>85%</td><td>86%*</td><td>87%</td><td>87%*</td><td>84%*</td><td>90%</td><td>74%*</td></tr><tr><td>Remembers seeing different/ similar banners</td><td>36%</td><td>47%*</td><td>27%*</td><td>20%*</td><td>41%*</td><td>59%</td><td>24%*</td></tr><tr><td>N</td><td>4,026</td><td>690</td><td>693</td><td>672</td><td>660</td><td>641</td><td>670</td></tr></tbody></table>

<table><caption>Table 6: Participants' reasons for refusing consent on the first website. One star indicates that this value is significantly different from the highest value at least to the 5% level.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Overall</th><th rowspan="2">Control</th><th colspan="2">Dark Pattern</th><th colspan="3">Behavioral Lever</th></tr><tr><th>No reject</th><th>Highlighted accept</th><th>Consequences</th><th>Tricolor</th><th>Highlighted decline</th></tr></thead><tbody><tr><td>Avoid being tracked</td><td>57%</td><td>62%</td><td>N/A</td><td>57%*</td><td>71%</td><td>48%*</td><td>43%*</td></tr><tr><td>Avoid targeted advertising</td><td>46%</td><td>57%</td><td>N/A</td><td>52%</td><td>44%*</td><td>41%*</td><td>44%*</td></tr><tr><td>Out of habit</td><td>37%</td><td>43%</td><td>N/A</td><td>51%</td><td>27%*</td><td>38%*</td><td>38%*</td></tr><tr><td>The fastest option</td><td>7%</td><td>7%*</td><td>N/A</td><td>4%*</td><td>4%*</td><td>8%*</td><td>13%</td></tr><tr><td>No accept option available</td><td>3%</td><td>1%</td><td>N/A</td><td>0%*</td><td>3%</td><td>4%</td><td>3%</td></tr><tr><td> Clicked randomly</td><td>4%</td><td>1%*</td><td>N/A</td><td>0%*</td><td>3%*</td><td>3%*</td><td>8%</td></tr><tr><td>N</td><td>978</td><td>110</td><td>N/A</td><td>114</td><td>306</td><td>230</td><td>218</td></tr></tbody></table>

<table><caption>Table 7: Participants' reasons for accepting consent on the first website. One star indicates that this value is significantly different from the highest value at least to the 5% level.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Overall</th><th rowspan="2">Control</th><th colspan="2">Dark Pattern</th><th colspan="3">Behavioral Lever</th></tr><tr><th>No reject</th><th>Highlighted accept</th><th>Consequences</th><th>Tricolor</th><th>Highlighted decline</th></tr></thead><tbody><tr><td>Out of habit</td><td>50%</td><td>50%</td><td>47%</td><td>51%</td><td>50%</td><td>53%</td><td>50%</td></tr><tr><td>Fear for not being able to access the website</td><td>35%</td><td>35%</td><td>32%*</td><td>35%</td><td>40%</td><td>33%*</td><td>39%</td></tr><tr><td>The fastest option</td><td>33%</td><td>34%</td><td>32%*</td><td>37%</td><td>30%*</td><td>31%*</td><td>33%</td></tr><tr><td>Option to refuse not available</td><td>9%</td><td>5%*</td><td>20%</td><td>7%*</td><td>6%*</td><td>3%*</td><td>5%*</td></tr><tr><td>To receive personalised advertisement</td><td>10%</td><td>11%</td><td>9%</td><td>10%</td><td>12%</td><td>9%</td><td>11%</td></tr><tr><td> Clicked randomly</td><td>5%</td><td>5%</td><td>5%</td><td>5%</td><td>5%</td><td>3%*</td><td>7%</td></tr><tr><td>N</td><td>2,982</td><td>574</td><td>666</td><td>551</td><td>351</td><td>395</td><td>445</td></tr></tbody></table>

<table><caption>Table 8: Participants' reported satisfaction with the choice they made. One star indicates that this value is significantly different from the highest value at least to the 5% level.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Overall</th><th rowspan="2">Control</th><th colspan="2">Dark Pattern</th><th colspan="3">Behavioral Lever</th></tr><tr><th>No reject</th><th>Highlighted accept</th><th>Consequences</th><th>Tricolor</th><th>Highlighted decline</th></tr></thead><tbody><tr><td>Satisfied by 1st banner choice</td><td>56%</td><td>54%*</td><td>44%*</td><td>55%*</td><td>64%</td><td>64%</td><td>58%*</td></tr><tr><td>Dissatisfied</td><td>44%</td><td>46%*</td><td>56%</td><td>45%*</td><td>36%*</td><td>36%*</td><td>42%*</td></tr><tr><td>Equally satisfied</td><td>16%</td><td>18%</td><td>20%</td><td>17%</td><td>12%*</td><td>13%*</td><td>14%</td></tr><tr><td>N</td><td>4,026</td><td>690</td><td>693</td><td>672</td><td>660</td><td>641</td><td>670</td></tr></tbody></table>
<table><caption>Table 9: Reported opinion on the banner seen on the first and second website. One star indicates that this value is significantly different from the highest value at least to the 5% level.</caption><thead><tr><th rowspan="2"></th><th rowspan="2">Overall</th><th rowspan="2">Control</th><th colspan="2">Dark Pattern</th><th colspan="3">Behavioral Lever</th></tr><tr><th>No reject</th><th>Highlighted accept</th><th>Consequences</th><th>Tricolor</th><th>Highlighted decline</th></tr></thead><tbody><tr><td>Navigating the banner is simple</td><td>85%</td><td>84%*</td><td>81%*</td><td>86%</td><td>87%</td><td>88%</td><td>87%</td></tr><tr><td>Information easy to understand</td><td>82%</td><td>79%*</td><td>77%*</td><td>82%</td><td>84%</td><td>83%</td><td>85%</td></tr><tr><td>Choices presented with my best interest in mind</td><td>68%</td><td>68%*</td><td>64%*</td><td>70%</td><td>67%*</td><td>73%</td><td>69%*</td></tr><tr><td>Easy control the level of data protection</td><td>72%</td><td>71%*</td><td>64%*</td><td>74%</td><td>76%</td><td>77%</td><td>73%*</td></tr><tr><td>More websites should use this banner format</td><td>80%</td><td>78%*</td><td>72%*</td><td>82%</td><td>82%</td><td>84%</td><td>81%</td></tr><tr><td>N</td><td>4,026</td><td>690</td><td>693</td><td>672</td><td>660</td><td>641</td><td>670</td></tr></tbody></table>

<table><caption>Table 10: Participants' responses in percentage regarding data collected by cookies (N = 4,026).</caption><thead><tr><th>Type of data collected</th><th>Percentage</th></tr></thead><tbody><tr><td>Browsing history</td><td>62%</td></tr><tr><td>Past product purchases</td><td>58%</td></tr><tr><td>Geolocation</td><td>57%</td></tr><tr><td>Information shared on my social network</td><td>36%</td></tr><tr><td>Postal address</td><td>30%</td></tr><tr><td>Phone number</td><td>29%</td></tr><tr><td>Email sent and received</td><td>28%</td></tr><tr><td>Contacts</td><td>23%</td></tr><tr><td>Bank data</td><td>13%</td></tr><tr><td>None</td><td>6%</td></tr><tr><td>All</td><td>4%</td></tr></tbody></table>

<table><caption>Table 12: Participants' responses in percentage on the means with which cookies collect data (N = 4,026).</caption><thead><tr><th>Cookies collect data by ...</th><th>Percentage</th></tr></thead><tbody><tr><td>Tracking what I do online (browsing history, services and products purchased)</td><td>74%</td></tr><tr><td>Tracking information I actively share (e.g. on social media)</td><td>44%</td></tr><tr><td>Tracking information that others have shared about me (e.g. on social media)</td><td>35%</td></tr><tr><td>Listening to what I say through the micro-phones on my computer and phone</td><td>15%</td></tr></tbody></table>

<table><caption>Table 11: Participants' responses in percentage regarding reasons for which data is collected via cookies (N = 4,026).</caption><thead><tr><th>Reasons</th><th>Percentage</th></tr></thead><tbody><tr><td>Decide which ads to show me</td><td>61%</td></tr><tr><td>Resell my data to third parties</td><td>48%</td></tr><tr><td>Subscribe to newsletter and send me marketing emails</td><td>42%</td></tr><tr><td>Improve my user experience on the website</td><td>36%</td></tr><tr><td>Adjust prices or sales to researched products</td><td>29%</td></tr><tr><td>Adapt language and display to my geolocation</td><td>24%</td></tr><tr><td>Allow to view media content (e.g. YouTube video) on the website</td><td>22%</td></tr><tr><td>Ensure security on the website</td><td>17%</td></tr><tr><td>Report user data to the government</td><td>12%</td></tr></tbody></table>

<table><caption>Table 13: Participants' willingness to use browser extension to aromatically accept or refuse cookies (N = 4,026).</caption><thead><tr><th>Willingness</th><th>Percentage</th></tr></thead><tbody><tr><td>Yes, definitely</td><td>24%</td></tr><tr><td>Yes, probably</td><td>38%</td></tr><tr><td>No, probably not</td><td>6%</td></tr><tr><td>No, definitely not</td><td>3%</td></tr><tr><td>Not sure</td><td>29%</td></tr><tr><td>Banking</td><td>33%</td></tr></tbody></table>
<table><caption>Table 14: Types of websites on which participants are somewhat comfortable/very comfortable to accept cookies (N = 4,026).</caption><thead><tr><td>Type of website</td><td>Cumulative percentage for "comfortable" and "very comfortable" level to accept cookies</td></tr></thead><tbody><tr><td>E-commerce</td><td>55%</td></tr><tr><td>Information</td><td>51%</td></tr><tr><td>Government</td><td>51%</td></tr><tr><td>Health</td><td>51%</td></tr><tr><td>Social media</td><td>41%</td></tr><tr><td>Banking</td><td>33%</td></tr></tbody></table>
<table><caption>Table 15: Participants' reported consent decisions on the consent banners of the first website (N = 3,947). Three stars (***), indicate a significant difference with the decision made when seeing the control (p < 0.001). Estimates are calculated using odds ratio from a logistic regression controlling for age, gender, education, and employment status of participants.</caption><thead><tr><th></th><th>Treatment</th><th>% non-acceptance (personalize or reject)</th><th>95% CI</th></tr></thead><tbody><tr><td rowspan="2">Dark pattern</td><td>No-decline</td><td>3.90***</td><td>2.54; 5.87</td></tr><tr><td>Highlighted-accept</td><td>18.00</td><td>14.19; 22.57</td></tr><tr><td rowspan="3">Behavioral Lever</td><td>Highlighted-decline</td><td>33.58***</td><td>27.71; 39.18</td></tr><tr><td>Consequences</td><td>46.73***</td><td>40.20; 52.85</td></tr><tr><td>Tricolor</td><td>38.38***</td><td>32.08; 44.23</td></tr><tr><td>Control</td><td>Control</td><td>16.81</td><td></td></tr></tbody></table>

<table><caption>Table 16: Participants' average decision time (N = 3,947) in their interactions with the consent banners on the first website. Three stars (***), indicate a significant difference with the time spent when seeing the control (p < 0.001). Estimates are calculated using coefficients from a linear regression controlling for age, gender, education, and employment status of participants. The variable for "time spent" was winsorized at the top (1%) to deal with outliers, which most likely corresponds to participants leaving their computer and coming back to the experiment later.</caption><thead><tr><th></th><th>Treatment</th><th>Average decision time (seconds)</th><th>95% CI</th></tr></thead><tbody><tr><td rowspan="2">Dark pattern</td><td>No-decline</td><td>4.03</td><td>3.33; 4.52</td></tr><tr><td>Highlighted-accept</td><td>3.78</td><td>2.94; 4.13</td></tr><tr><td rowspan="3">Behavioral Lever</td><td>Highlighted-decline</td><td>4.24</td><td>3.80; 4.99</td></tr><tr><td>Consequences</td><td>5.70***</td><td>4.72; 6.68</td></tr><tr><td>Tricolor</td><td>5.72***</td><td>4.74; 6.70</td></tr><tr><td>Control</td><td>Control</td><td>4.10</td><td></td></tr></tbody></table>

<table><caption>Table 17: Participants' reported consent decisions (N = 3,947) when facing the Control banner upon visiting the third website. Three stars (***), indicate a significant difference (p < 0.001) with participants having seen the Control banner on the first and second website (control group). Estimates are calculated using odd ratio from a logistic regression controlling for age, gender, education, and employment status of participants.</caption><thead><tr><th></th><th>Treatment</th><th>% who personalise or reject</th><th>95% CI personalise or reject</th></tr></thead><tbody><tr><td rowspan="2">Dark pattern</td><td>No-decline</td><td>9.66***</td><td>7.17; 12.79</td></tr><tr><td>Highlighted-accept</td><td>20.09</td><td>16.31; 25.15</td></tr><tr><td rowspan="3">Behavioral Lever</td><td>Highlighted-decline</td><td>29.7***</td><td>24.33; 34.99</td></tr><tr><td>Consequences</td><td>35.81***</td><td>30.13; 41.75</td></tr><tr><td>Tricolor</td><td>27.31***</td><td>22.49; 32.91</td></tr><tr><td>Control</td><td>Control</td><td>18.84</td><td></td></tr></tbody></table>

<table><caption>Table 18: Non-acceptance (decline/personalise) rate depending on device used. One star indicates that this value is significantly different from the highest value at least to the 5% level.</caption><thead><tr><th></th><th>Control</th><th colspan="2">Dark Pattern</th><th colspan="2">Behavioral Lever</th></tr><tr><th></th><th></th><th>No reject</th><th>Highlighted accept</th><th>Consequences</th><th>Tricolor</th><th>Highlighted decline</th></tr></thead><tbody><tr><th>Desktop</th><td>27%</td><td>4%</td><td>29%</td><td>54%</td><td>36%</td><td>35%</td></tr><tr><th>Mobile</th><td>14%</td><td>4%</td><td>15%</td><td>45%</td><td>39%</td><td>33%</td></tr><tr><th>N</th><td>690</td><td>691</td><td>672</td><td>658</td><td>640</td><td>669</td></tr></tbody></table>