“Make Them Change it Every Week!”: A Qualitative Explorationof Online Developer Advice on Usable and Secure Authentication
Extended Version∗
Jan H. Klemmer

Marco GutfleischChristian Stransky

Leibniz University Hannover
Hannover, Germany
klemmer@sec.uni-hannover.de

Ruhr University Bochum
Bochum, Germany
marco.gutfleisch@rub.de

CISPA Helmholtz Center for
Information Security
Hannover, Germany
stransky@sec.uni-hannover.de

Yasemin Acar

M. Angela Sasse

Sascha Fahl

Paderborn University
Paderborn, Germany
yasemin.acar@uni-paderborn.de

Ruhr University Bochum
Bochum, Germany
martina.sasse@rub.de

CISPA Helmholtz Center for
Information Security
Hannover, Germany
sascha.fahl@cispa.de

ABSTRACTCCS CONCEPTS

Usable and secure authentication on the web and beyond is missioncritical. While password-based authentication is still widespread,
users have trouble dealing with potentially hundreds of online
accounts and their passwords. Alternatives or extensions such as
multi-factor authentication have their own challenges and find only
limited adoption. Finding the right balance between security andusability is challenging for developers. Previous work found thatdevelopers use online resources to inform security decisions whenwriting code. Similar to other areas, lots of authentication advice
for developers is available online, including blog posts, discussions
on Stack Overflow, research papers, or guidelines by institutions
like OWASP or NIST.
We are the first to explore developer advice on authenticationthat affects usable security for end-users. Based on a survey with18 professional web developers, we obtained 406 documents andqualitatively analyzed 272 contained pieces of advice in depth. We
aim to understand the accessibility and quality of online advice
and provide insights into how online advice might contribute to(in)secure and (un)usable authentication. We find that advice isscattered and that finding recommendable, consistent advice is achallenge for developers, among others. The most common advice
is for password-based authentication, but little for more modern
alternatives. Unfortunately, many pieces of advice are debatable(e.g., complex password policies), outdated (e.g., enforcing regularpassword changes), or contradicting and might lead to unusable orinsecure authentication. Based on our findings, we make recommendations for developers, advice providers, official institutions,and academia on how to improve online advice for developers.

• Security and privacy → Usability in security and privacy;
Software security engineering; Authentication.

∗ This is the extended version of the paper originally published at ACM CCS 2023 (DOI:
10.1145/3576915.3623072). It contains additional information and appendices.

This work is licensed under a Creative Commons Attribution
International 4.0 License.
CCS ’23, November 26–30, 2023, Copenhagen, Denmark© 2023 Copyright held by the owner/author(s).
ACM ISBN 979-8-4007-0050-7/23/11.
https://doi.org/10.1145/3576915.3623072

KEYWORDS
Usable Security; Advice; Software Development; Authentication
ACM Reference Format:
Jan H. Klemmer, Marco Gutfleisch, Christian Stransky, Yasemin Acar, M.
Angela Sasse, and Sascha Fahl. 2023. “Make Them Change it Every Week!”:
A Qualitative Exploration of Online Developer Advice on Usable and Secure
Authentication: Extended Version. In Proceedings of the 2023 ACM SIGSAC
Conference on Computer and Communications Security (CCS ’23), November
26–30, 2023, Copenhagen, Denmark. ACM, New York, NY, USA, 20 pages.
https://doi.org/10.1145/3576915.3623072

1

INTRODUCTION

Password-based authentication is the status quo on the web andbeyond [6, 13, 14]—despite its many known problems. This includes
challenges around memorizing passwords, password reuse [35, 86,107], easily guessable and insecurely stored passwords [55, 107],
being forced to follow complicated password policies and change
passwords regularly [55, 60, 102, 103], or phishing attacks [57].
Eventually, passwords are a prime example of the importance ofusable security: The underlying problem is not the technical securityof passwords but the overwhelming challenges they present to usersthat negatively impact authentication security. Replacing passwords
with potentially more usable and secure alternatives has been a
popular idea for many years [13]. However, alternatives such as
two-factor authentication (2FA) and FIDO2 both lack adoption [8,18, 24, 32, 64, 87], and passwords still remain the de facto standard.
Overall, companies, development teams, individual developers,
and other involved stakeholders significantly impact the usable
security of software systems in general and authentication in particular [41]. For example, they make decisions on password composition policies or the deployment of 2FA. To inform security-related
software development decisions, including decisions on authentication deployments, developers draw heavily on online security
advice, e.g., from Stack Overflow (SO) [2, 3, 5, 16, 27, 28, 29, 73,

CCS ’23, November 26–30, 2023, Copenhagen, Denmark120]. In the ACM CCS 2022 keynote, Michelle Mazurek concluded
that security advice is a “disaster” and affects developers and otherstakeholders [68]. As developers draw on security online advice
and online advice on authentication is common, the question ofwhat developers can learn from such advice to make authentication deployments usable and secure was the main driver of thiswork. Based on a qualitative exploration of the status quo of online
security advice on authentication deployments for developers, weaimed to develop further ideas for improving future advice. Previous work on general security advice for end-users [15, 23, 46, 52,
89, 91, 93] and for developers [2, 3, 5, 10, 11, 16, 27, 28, 29, 31, 39,
76, 92, 120] does not address online security advice for developers
on implementing usable security for end-users. To the best of ourknowledge, we are the first to explore this research gap.
Goals. In this paper, we aim to qualitatively explore developer
security advice on implementing authentication that affects enduser usable security. We aim to understand the accessibility andquality of online advice and to provide insights into how online
advice might contribute to (un)usable and (in)secure authentication.
More specifically, we seek to explore which topics are covered andidentify potential challenges developers might face when following
advice on implementing authentication online. Based on our findings, we make recommendations for future advice to better support
developers in implementing usable and secure authentication.
Approach. In our qualitative exploratory in-depth analysis, weapplied the following multi-step process: First, we collected online
documents that contained advice on authentication. Therefore, we
recruited 18 professional web developers to collect a set of 264 online resources (Sect. 3) and their Google search queries. We used
these queries to enrich the document corpus further. The overall
document corpus consists of 406 documents on web authentication.
Second, three researchers extracted all advice on web authenticationfrom each document through qualitative open coding (Sect. 4.2).
Finally, in an in-depth qualitative analysis, we explored advice
distribution (Sect. 4.3) and topics (Sect. 4.4) and investigated characteristics like recommendable (Sect. 4.5), debatable (Sect. 4.6), andcontradicting advice (Sect. 4.7).
Key Findings. The advice we found roughly reflected current(usable) security authentication challenges on the web, e.g., passwords are still ubiquitous, and password advice was most prevalent(51.6% of documents). For example, the advice covered password
composition policies or password recovery. Moreover, only about
22.5% of advice was relevant to usable security of authentication.
We classified 31.6% to be recommendable. However, we also found
18.0% to be debatable, e.g., being outdated, factually wrong, or counteracting usable security (e.g., enforcing regular password changes).
We identified 15 groups of two or more pieces of advice that contradict each other. Overall, we found that advice from research
results and official institutions was rarely integrated into the advice
we analyzed. Furthermore, the advice was scattered across many
online resources, making it difficult for developers to find.
Contributions. In this paper, we make the following contributions: (i) To the best of our knowledge, this is the first study investigating online advice for developers on authentication that affects
security and usability for end-users. (ii) We present an overview

Jan H. Klemmer et al.

of authentication advice topics and further explore recommendable, debatable, and contradicting advice. (iii) Based on our findings,
we make recommendations for future usable security advice fordevelopers. Furthermore, we highlight open challenges and outline future research. (iv) Finally, we provide our document corpus,
all pieces of advice, and additional supplementary material in a
replication package (Appendix A).

2

RELATED WORK

We discuss related work in three key areas: (i) general security
advice for end-users, (ii) (usable) security advice for developers,and (iii) usable and secure web authentication and its challenges.
We highlight our novel contributions and put it into context.
General End-User Security Advice. A large portion of online
advice on security addresses end-users [15, 23, 47, 52, 72, 89, 91],
e.g., concerning topics like how to handle passwords or spam. Nonexperts, like most end-users, heavily request such advice [46]. Aslots of advice exists, especially online, the selection of advice sources
has to be considered [89]. For example, factors like the source’s perceived trustworthiness [89] and convenience [23] influence advice
adoption. One explanation for low advice adoption is that users
rationally reject advice as it often offers a poor cost–benefit tradeoff [47]. Overall, security advice can be challenging to adopt, as a
mismatch between advice and behavior was perceived among bothexperts and non-experts [15, 52]. Even security experts were foundto lack consensus for advice [93]. Similarly, users and experts (likeadministrators) struggle with advice prioritization [91].
While advice can be used to educate end-users, we argue that itis (at least equally) important to help software professionals design
systems that people can use securely—ideally without users needing
any advice. Therefore, we focus on advice for software developers.
(Usable) Security Advice for Developers. In recent years, researching human factors and security with a focus on developers
has gained popularity [4, 40, 88, 110]. This is also true for researchon security advice for developers. Several studies found that advice
impacts security and that sources containing insecure code findtheir way into software [2, 3, 5]. For example, Acar et al. found thatdevelopers who used SO produced significantly less secure codeand that only 17% of SO posts contained secure code snippets. Insecure code from SO is also prevalent in many Android apps [27] andcontained in top Google search results [28]. However, nudging [29]and re-ranking search results [28] can create a positive security
impact of SO. Overall, SO is a place for software professionals to
discuss security-relevant topics [120], while secure and insecureanswers were almost balanced [16]. Apart from SO, researchers
also found challenges in other advice sources, e.g., not actionable
IoT advice [10, 11].
Advice can also be given through other channels. This includes
IDE plug-ins that aid developers in preventing security pitfalls.
For example, Nguyen et al. presented and evaluated a plug-in that
supports Android developers [76]. Similar tools exist for other use
cases, e.g., secure password storage [31]. Besides such tools, Gorski
et al. found that integrating security advice into application programming interfaces (APIs) can significantly improve code security
without affecting API usability [39]. Other advice forms can be
more general by outlining principles (e.g., Microsoft’s NEAT design

A Qualitative Exploration of Online Developer Advice on Usable and Secure Authentication

principles for security warnings [92]) and abstract guidance (e.g.,heuristics for accessibility and usable security on the web [74]).
Most similar to our work, Iacono, Gorski, et al. systematized
security principles, guidelines, and patterns based on scientific
literature [38, 62], but only “concentrated on principles and patterns
for end-users” [38]. In contrast, we focus especially on advice fordevelopers that affects end-users’ usable security. Moreover, we
focus specifically on (non-scientific) online resources. To the bestof our knowledge, no paper examined advice for developers thatcovers end-user usable security. We argue that end-user (usable)
security, however, highly depends on decisions and implementation
during software development. Our analysis addresses this gap.
Usable and Secure Web Authentication. Passwords, probably
the oldest authentication approach [14] and a starting point forearly usable security research [6], are the most used authentication approach despite many shortcomings. This includes but is not
limited to insecure storage [55, 107], password reuse [35, 86, 107],
and usability and (resulting) security problems through password
policies [55, 60, 101, 102, 103]. To aid in creating secure passwords,
password strength meters are a common tool and visualization
technique [112]. However, their construction is difficult. For example, researchers found inconsistencies, weaknesses [21], andaccuracy issues [33, 115]. Similarly, password managers (PWMs)
should help users deal with their passwords, e.g., by creating secure(random) passwords [121] that fit websites’ password policies [30].
However, even PWMs have interaction problems with websites [48]or overwhelm their users [79], both limiting their usability. Due
to passwords’ numerous problems, the idea of replacing them became popular. However, many approaches failed [13], and replacing
passwords remains an unsolved challenge.
For example, replacing passwords could be achieved with the
FIDO2 standard [26]. However, FIDO2 lacks adoption, which might
be caused by overlooked benefits [24], concerns, and not trusting
the new technology despite good usability [32]. Despite generally
good usability, users struggle with hardware token setup [19, 96] orrate the usability of smartphones for FIDO2 as low [83]. As tokens
can get lost and are associated with costs, electronic IDs can beused for FIDO2 [100]—albeit being hard to set up [53].
2FA, or generally multi-factor authentication (MFA), is an important concept for strengthening security—especially when using
passwords—but could also negatively affect usability [1, 94, 95].
Users might have problems with the initial setup [94], and the 2FA
user experience (UX) is inconsistent [64]. Overall, 2FA adoption isstill relatively low and needs to be increased [8, 18, 87, 90]. Widelyadopted approaches like SMS-based 2FA are known to be vulnerable
to phishing [66].
To conclude, authentication has pitfalls that software developers
need to consider. Interestingly, most outlined challenges are notechnical security problems and boil down to a usability problem.
For example, passwords would be more secure if used correctly.
Therefore, we investigate how developer advice supports usable
and secure authentication.

3

COLLECTING ONLINE DEVELOPER ADVICEIn this section, we describe our data collection of 406 online resources on authentication. While we aimed to create an ecologically

CCS ’23, November 26–30, 2023, Copenhagen, Denmark

valid dataset, our main goal was diversity due to the exploratory
nature of our work. In a developer survey between August andNovember 2021, 18 professional web developers searched and reported online resources on web authentication they considered for
an imaginary web authentication development task. We extend the
document corpus further by including search results for our participants’ search queries. As we had already reached saturation withthis document corpus, we stopped recruiting more participants.

3.1

Developer Survey

Our goal was to create a comprehensive corpus of documents containing online advice regarding usable and secure web authentication for developers that impacts usable security for end-users.
We asked professional software developers to search for online resources they consider relevant for implementing usable and secure
web authentication. We gave them a scenario and task to search forrelevant advice on the web.
3.1.1 Scenario. We created a task containing a web development
scenario and framed it towards authentication and usable security.
We aimed to collect document URLs, search queries, and otherinformation from software developers.1 Our scenario consistedof a hypothetical new web-based health platform KeepYourHealth,that should become the central pivot point for personal health, e.g.,
storing and sharing medical data or communicating with doctors.
We chose the healthcare framing for three reasons: First, medical
and personal health information is considered highly sensitive,
hopefully shifting the participants’ attention to security. Second, tohighlight the need for good usability, medical applications have adiverse user base that likely contains many non-tech-savvy users.
Third, medical applications must follow certain compliance or dataprotection rules (e.g., HIPAA in the US, PIPEDA in Canada, GDPR in
Europe). While this might skew the results slightly towards official
documents, we also hoped to receive a more diverse and complete
document sample.
As part of the KeepYourHealth project, we wanted participants
to imagine that they are responsible for designing and implementing the authentication mechanism for both a mobile and a desktop
version. To create the concept for KeepYourHealth’s authentication system (including registration and login), the participants had
to perform online research and look for helpful resources. We designed the scenario to be very open to ensure diverse results, as
participants can explore various topics in their search. Moreover,
we instructed participants to freely consider any resource, authentication approach, technology, etc. Our only requirement was thatthe authentication system had to be usable and secure. We used
words like easy-to-use, user-friendly, secure, usability, and security
to emphasize the requirement for usable security.
3.1.2 Search Task. To collect online advice, we gave software professionals the task of searching for online resources that provide
advice that potentially helps to build a usable and secure web authentication system for KeepYourHealth. We explained our research
project’s purpose and that we wanted to collect and analyze advice
from various online sources. We additionally explained that we,
as researchers, are biased and probably would search differently
1 See Appendix B for the full scenario description and search task.

CCS ’23, November 26–30, 2023, Copenhagen, Denmark

than web development experts. We also wanted the software developers to report how they found a specific online source, i.e., thesearch query. We provided the participants with a table to report
documents and search queries. The table consisted of the following
columns (cf. Figure 4): URL, how the document was found (e.g.,bookmark, link within another site, search engine), and (if applicable) the search query that yielded the document. Each column’s
purpose was also explained. Our participants were free to choose
any search approach they wanted, i.e., we did not constrain them
to a specific search engine.
3.1.3 Questionnaire. We embedded the search task in an online
questionnaire. After an initial consent form, the participants wereintroduced to the scenario and their tasks. We gave them 30 minutes
for the task. For the search task, we displayed the above-mentioned
table for the users to enter URLs and search queries. After completing the task, we asked the participants for their most used searchengine and to rank different information resource types by theirhelpfulness. Next, we followed up with standard demographic questions, including age, gender, country of residence, highest level of
education, and employment status. Additionally, we asked somequestions specific to this study’s context regarding participants’experience, skills, and security and usability education. We estimated the maximum time for the whole survey at 45 minutes. The
replication package (Appendix A) includes the full questionnaire.
3.1.4 Instrument Development & Piloting. We developed and piloted the questionnaire, scenario, and task in multiple iterations. We
asked nine persons to pilot the study and questionnaire, including
researchers with experience conducting usable security studies andexperiments. We made smaller adjustments regarding wording anddescription texts based on the feedback that we collected via thinkaloud and feedback text boxes on each survey page. Finally, wetested the whole setup with three professional developers until nofurther adjustments were necessary. During piloting, we collected
48 documents and 38 search queries. We included those in our final
document corpus to enhance its diversity further.
3.1.5 Ethics & Data Protection. We received ethical review board(ERB) approval for this study from one of the involved institutions,concluding that there were no ethical concerns. The remaining
institutions had neither an institutional review board (IRB) nor an
ERB that applied to our study. This study was designed to adhere to
the ethical research principles of the Menlo Report [54]. We further
adhered to the strict German and EU privacy laws. All participants
consented to collecting, storing, and analyzing their data for thisresearch. We clarified that we would only evaluate and publish
de-identified and aggregated data. Furthermore, we informed allparticipants that they could terminate the study at any time.
3.1.6 Recruiting. We aimed to recruit a diverse sample of softwaredevelopers with web development experience. Therefore, we used
the freelancer platform Upwork. We created an Upwork job post
describing our study and communicated the idea of our research.
Participants were required to have sufficient proficiency in English.
Besides that, we checked the Upworkers’ profiles and job history
to ensure they have web development experience. We also directly
invited Upworkers who had indicated experience in web development in their profiles. The job was rewarded with $45 (hourly wage

Jan H. Klemmer et al.

Table 1: Demographics of the 18 web developers.
Gender
Male
16
88.9%
Female2Age [years]
Min.
19.0
Max.
45.0
Mean (Std.)
28.4
±7.9
Median25.5
Highest Education Level< High school1
5.6%
Bachelor’s degree
7High school4
22.2%
Attending grad. school 1
College2
11.1%
Master’s degree
3Country of Residency
United Kingdom
4
22.2%
Brazil2
United States
3
16.7%
Other1
7
India
2
11.1%
Industry Experience [years]
Min.
1.0
Max.
22.0
Mean (Std.)
6.1±6.0
Median3.8
Employment Status†
Employed full-time10
55.6%
Student2
Self-employed/Freelancer
950.0%
Self-describe
1
Employed part-time
2
11.1%
†
Domain of Web Experience
Full-stack developer
15
83.3%
UI/UX
6
Frontend developer
1161.1%
API developer
6
Backend developer
950.0%
Testing
4
Web design
7
38.9%
Other
1
Security/Usability as Part of Training/Studies
Security8
44.4%
Usability
5
Security/Usability Training in Professional Activity
Security
7
38.9%
Usability
7† Multiple answers allowed; may not sum to 100%.

11.1%

38.9%
5.6%
16.7%
11.1%
38.9%

11.1%
5.6%

33.3%
33.3%
22.2%
5.6%
27.8%
38.9%

1 Each country occurring once.

of $60), above Upwork’s $15–$30 median hourly pay for softwaredevelopers [111] to offer competitive compensation.
3.1.7 Demographics. We recruited 18 web developers via Upwork
whose demographics are comparable with those of professional
developers from SO’s latest developer survey [105]. An overview
of participants’ demographics can be found in Table 1. The participants were from 11 different countries, with the UK and the U.S.
being the most common, including others such as Canada, Nepal,
Romania, Germany, Nigeria, France, and Turkey. The majority had
at least a bachelor’s degree. Their professional software development experience varied between 1.0 and 22.0 years. Participants
covered various roles, such as software engineers, web developers,
architects, team leads, and CTOs. Most participants reported someexperience with usable security. Some participants also received education in this area in their training, studies, or professional career.
3.1.8 Reported Documents and Search Queries. The participants
overall found 264 distinct documents (284 with duplicates reported
by multiple participants) across 171 different website domains. Thisalready indicates that advice is contained in many resources fromdifferent authors. However, some domains were reported more
frequently, as shown in Table 2a. This mainly includes authentication platforms like Auth0, Okta, swoopnow.com, but also blogs andtutorial sites like medium.com and dzone.com, or documentation
like the one provided by Google or the library next-auth. The OpenWeb Application Security Project (OWASP)’s Cheat Sheet Series is
included.
We collected 145 distinct search queries from our participants.
All participants reported having used Google for their web searches.
The most frequent terms and term bigrams are shown in Tables 2band 2c. While authentication is the most frequent term, participantsA Qualitative Exploration of Online Developer Advice on Usable and Secure Authentication

CCS ’23, November 26–30, 2023, Copenhagen, Denmark

Official Documentation
Blog Posts & News Articles
Online Communities
Video Tutorials
Asking Coworkers & Friends
Official Standards & Guidelines
Scientific Articles & Papers
Workshops
Books

searched specifically for security aspects, best practices, and usability aspects. This also indicates that our task worked as intended, as
the terms reflect the goal of usable and secure authentication.
Summary: Initial Document Collection. The search task with 18 professional web developers resulted in 264 documents and 145 search
queries. Both document domains and search queries show high diversity.

1 2 3 4 5 6 7 8 9 10
Rank (n=18)

3.2

Search Expansion

We extended our document corpus with search results for the collected search queries that participants reported in addition to thedirectly reported documents. We performed this search expansion
because participants might have missed relevant documents tocounterbalance participants’ personalized search results. Moreover,this increases overall diversity and helps to reach saturation in thebest case, while adding no additional insights in the worst case. For
each query, we obtained the corresponding search results to expand
our document corpus. Therefore, we leveraged Google’s Custom
Search API [37] to execute a search for each query and extracted
its top 10 results, i.e., its URL and some metadata. We chose this
Google-based engine, as Google is the most used search engine [75,106] and all participants reported that they use Google as one of
their main search engines. We chose 10 results per query, as thiscorresponds to the first page of search results. Moreover, the clickthrough rate (CTR) drops below 10% after the fourth and quickly
decreases further to almost 0% for lower-ranked results [84]. Thatsaid, almost no user looks at the second page of results [20]. A top10 expansion, therefore, covers search results a user potentiallyclicks while also including a safety margin.

3.3

Final Document Collection

The search task with professional software developers resulted in
264 distinct documents and 145 distinct search queries. As our goal
for this qualitative study is diversity, we decided also to include all
48 documents and 38 search queries from piloting. As describedabove, we extend our corpus with documents found using Google.
We randomly sampled 100 search results from the search expansion
and included those in the final document corpus. Combining allthese documents, we obtained a final dataset of 406 documents
overall. For the documents that indicate a publishing date, themedian year for the last update was 2021 (oldest: 2002, newest:2021). We downloaded each as a PDF for our analysis.
We stopped after these 406 documents, as we had already reachedsaturation, which means that we would not gain new insights fromanalyzing more documents and advice, i.e., no new themes emergedin Sect. 4 (e.g., advice topics). Repeated advice in multiple documents
also indicated saturation. Consequently, we did not survey further
participants.
In the survey, we asked our participants about the helpfulnessof different information sources (cf. Figure 1). Overall, participants
reported a high affinity towards online resources. The top-threesources were official documentation, blog/news articles, and online communities. Similarly, this is also reflected in the different
types of documents (Table 3). We found blog posts (some includingnews) to be the most common type, followed by documentation
and wikis and project websites (including source code repositories).

Figure 1: Participants’ ranking of information sources forthe question: “Which advice or information sources do you
find most helpful in general?”. From most helpful (1) to least
helpful (10), excluding “Other” answers.

Surprisingly, only 15 documents (3.7%) were from Q&A forums,
13 of them from SO (cf. discussion in Sect. 5.4). Minor document
types included educational material and courses as well as video
and audio resources. 6 documents had no clear type. 4 (1.0%) were
not available online anymore when we downloaded the documents.
Summary: Document Corpus. Our final document corpus consistedof 406 distinct documents which we downloaded for analysis. Blog posts,
documentation, and wiki pages were the most common document types.

3.4

Limitations

This work has some limitations that are typical for this type ofstudy. Generally, the developer survey is subject to self-reporting,social desirability, under- and over-reporting, and sampling biases.
We note that our document corpus cannot contain all online resources and that it is influenced by the participants’ experience andespecially the personalization of their search results. We addressed
this by recruiting a diverse set of participants and extending the
document corpus with search results until we reached saturation.
Furthermore, we received only English documents as it was thestudy’s language. We decided that English as the only language isan acceptable trade-off as it is the main language used in softwaredevelopment (e.g., documentation or SO). The search task and scenario (Sect. 3.1) itself could bias the document corpus. However, wefavored a realistic scenario without many requirements for external
validity over a general, undirected search task.

4

ADVICE ANALYSIS

In this section, we provide details on the advice analysis and its
results. Our analysis is a multi-stage process that starts with identifying and extracting advice (Sect. 4.2) before analyzing it in detail.
In the following subsections, we describe the advice distribution(Sect. 4.3), its topics (Sect. 4.4), and occurrence of recommendable(Sect. 4.5), debatable (Sect. 4.6), and contradicting advice (Sect. 4.7).
Throughout the paper, we report counts, e.g., counts of advice
or documents. As this is qualitative work, those counts are used
to convey weight and should not be interpreted as quantitative
statistical results. While we try to give an overview by discussing
advice examples, we cannot describe and discuss every single pieceof advice due to their high number.

CCS ’23, November 26–30, 2023, Copenhagen, Denmark

Jan H. Klemmer et al.

Table 2: Most frequent domains and search query terms and bigrams that participants reported.
(a) Top-10 documents domains.

(b) Top-10 terms from search queries.

(c) Top-10 bigrams from search queries.

Domain#Search Term#

Search Term Bigram#

auth0.com
medium.com
www.okta.com
swoopnow.com
blog.jscrambler.com
cheatsheetseries.owasp.org
cloud.google.com
dzone.com
developers.google.com
next-auth.js.org

23
16
8
6
6
5
5
5
4
4

authentication
web
best
security
securemethods
user
app
practices
applications

72
39
15
11
11
10
10
9
8
7

web authentication
authentication methods
for web
best practices
web app
web applications
user authentication
authentication best
methods forsecure authentication

14
10
8
8
8
7
6
5
4
4

Table 3: Types of the reported documents.
Type
Blogs (including news)
Documentation & Wikis
Project Websites & Code Repositories
Scientific
Q&A Forums
Educational
Video/Audio
Undefined Type
Not Available

#

%

219
79
41
35
15
4
3
6
4

53.9%
19.5%
10.1%
8.6%
3.7%
1.0%
0.7%
1.5%
1.0%

advice. An example of such a concrete recommendation is:“Allow users to toggle password visibility when typing it.”
• Descriptive Information: We deliberately refrained from extracting information as advice that was not explicitly directed
at the reader or any other form of specific recommended
action. Some document passages just inform and describe,
leave room for interpretation, and require reading between
the lines. It might be unclear whether the resource’s author
wanted to give advice, whether a reader perceives it to beadvice, and how one might interpret it.

4.2.1 Advice Extraction. We extracted each advice for developers
by reading all documents and manually assigning codes that summarize each piece of advice. We used the following definitions todetect advice. The definitions’ goal was to make the extraction
process transparent, objective, and reproducible.

Three researchers each annotated a third of the final dataset in an
iterative open coding process. As advice in the documents can belong and wordy, we summarized it by creating codes that briefly
paraphrase the advice [56, 67]. Table 4 shows examples of original
document excerpts and our assigned advice codes. As shown in
the table, different text passages can have the same meaning andwere therefore assigned the same advice code (examples 3 and 4).
We used these advice codes for our analyses throughout the paper.
Each researcher grouped similar or related paraphrases in a code
system. The leaf nodes in the code tree contain the advice. The
inner nodes are used to build a hierarchy (e.g., group all advice on
passwords). After coding sets of 50 documents, at least two coders
merged, refined, and regrouped the code system if necessary; allthree reviewed the resulting code system. The coders repeated thedescribed process eight times until advice from all 406 documents
was extracted. In accordance with McDonald et al. [70], we refrainfrom reporting inter-rater reliability (IRR) because extracting only
concrete advice leaves little room for the researchers’ own interpretations and because the following analysis of the advice (following
subsections) is our main contribution—not the code system itself.
As we focus on non-scientific online resources, we excluded
31 (7.6%) documents being scientific papers; 4 (1.0%) documents because they contained only a video or audio guide; another 29 (7.1%)
documents were not related to our research topic (i.e., not aboutauthentication and usable security); and two documents because
they were not available anymore or a duplicate of an existing document. 119 documents contained no advice at all. For the remaining
221 documents, we identified 1,208 distinct pieces of advice.

• (Concrete) Advice: We considered explicit prompts to dosomething in a specific way or specific recommendations
concrete advice. Phrases like “we recommend,” “a developer
should,” or imperatives like “ensure” often indicate advice,
which the researchers therefore used as proxies to detect

4.2.2 Inclusion and Exclusion Criteria. In the previous step, we
extracted all advice for developers and obtained 1,208 paraphrased
pieces of advice. We defined inclusion and exclusion criteria to
focus only on advice relevant to this paper’s scope. We excluded
advice if any of the following criteria were met: (1) if the advice

4.1

Positionality Statement

Before giving detailed insights, we want to highlight that this is qualitative research. While we always involved multiple researchers andbased decisions on external references where possible, this study’s
design and results might be influenced by the involved researchers.
That said, the results are not guaranteed to be entirely objective
and should be interpreted in context. Therefore, we aim to clarify our backgrounds with this positionality statement. All authors
are usable security researchers with varying levels of experience,
ranging from third-year Ph.D. student, to tenured faculty and full
professors with more than 30 years usable security research experience. Our study backgrounds cover computer science, IT security,
mathematics, and psychology. The authors are based in Germany.

4.2

Identifying & Extracting Advice

Below, we describe our advice identification and extraction approach in detail. It consists of two steps: (1) First, we extracted
and systematized each piece of advice from all documents throughqualitative coding. (2) Second, we applied inclusion and exclusion
criteria to reduce the set of advice to address usable and secure web
authentication relevant to our study.

A Qualitative Exploration of Online Developer Advice on Usable and Secure Authentication

CCS ’23, November 26–30, 2023, Copenhagen, Denmark

Table 4: Examples of document excerpts with concrete advice
and assigned advice codes.

Table 5: Top-5 domains containing most distinct advice.
DomainText in Document & Assigned Advice Code#Advicecheatsheetseries.owasp.org
swoopnow.com
mobile-security.gitbook.io
web.devdzone.com“As a developer yourself, it’s your responsibility to provide users with the best and most secure
experience possible.” → Ensure authentication is secure and has good UX“Make sure to explain each [2FA] step and keep in mind, that not everybody has the same technicalknowlegde. For example, HubSpot added a description of Google Authenticator when the user would
choose that option.” → Explain each 2FA step

55
27
27
23
18

“Consider Password-less Authentication [. . .] Some applications are good candidates for a new approach: eliminating the password altogether. [. . .] users [. . .] forget their credentials, password-lessauthentication is an compelling option.” → Use password alternatives“If your organisation wants to take its login game to the next level, you should definitely consider
password alternatives.” → Use password alternatives“Send the user an email informing them that their password has been reset.” → Notify users via
email when password was reset

was not related to security, (2) if the advice was not related tousability aspects and end-users, or (3) if the advice did not coverauthentication. The remaining advice was included as it is relevant
for usable security of web authentication. Three researchers did
this assessment independently and later resolved conflicts until
they reached consensus.
Advice Inclusion. 272 distinct pieces of advice out of the initial
1,208 covered usable security for web authentication and were therefore included for further analysis. For example, the advice to “Use aminimum of 8 characters” for passwords matched our inclusion criteria because passwords are a security and authentication feature,and their length affects usability [55, 101, 102, 103]. Overall, the 272
pieces of advice matching the inclusion criteria occurred 655 times
across 124 different documents. Hence, only a small fraction of theonline advice considers or affects usable security. Instead, mostadvice is only about security—not considering usability and humanfactors of security at all.
Advice Exclusion. According to the above-mentioned criteria,
we excluded 936 pieces of advice from further analysis because
they did not cover usable security of web authentication, thus not
being relevant for our research goals for one or multiple of thefollowing reasons. We excluded 257 distinct pieces that did notcover security, e.g., “Provide live form validation” or “Only use singlecolumn forms”. Similarly, we excluded 669 distinct pieces of advice
that were not relevant for end-users or usability. For example, “Useobject-relational mapping (ORM) to prevent SQL injections” or “Usebearer tokens,” are technical security details that users do not notice.
We excluded another 73 pieces of advice that covered usability andsecurity but were not relevant to web authentication, e.g., “Limit
upload size for users”. We note that we focussed our analysis on
advice in prose text and excluded any code snippets (occurring in
102 documents).
Summary: Identifying & Extracting Advice. We found and extracted
1,208 pieces of advice in 406 documents. Focusing on web authentication
advice with relation to usable security, we came up with 272 pieces ofadvice from 124 documents for further analysis.

4.3

Advice Distribution & Density

As already stated, the advice was widely distributed across 124 documents. A document contained 4.5 different pieces of advice on

0

10

20

30

(a) Number of pieces of advice
per document.

0

10

20

30

(b) Occurrence frequency of advice (in number of documents).

Figure 2: Boxplots that describe the advice distribution.

average (median: 2.0). However, the distribution was overall skewed(cf. Figure 2a): while the document with most advice contained
35 different pieces of advice, the median was only 2.0 different
pieces of advice per document. That said, most documents provided
only little advice, while a few were much denser. This was similar
for the domains on which advice is provided. While individual domains (that contain advice at all) provided on average and median
only 6.6 and 3.0 different pieces of advice, respectively, the OWASPCheat Sheets contained most, having 55 different advice imperatives(see Table 5). Considering that mobile-security.gitbook.io by OWASP
was also among the top domains, OWASP resources containedby far the most advice. However, authentication provider swoopnow.com’s blog and the developer portals web.dev and dzone.com
also contained a considerable amount of advice.
While most documents contained little advice, developers are
not limited to considering only a single document. In line with that,our participants found 26.3 different pieces of advice on averageamong multiple documents (median: 13.0). 81 different pieces ofadvice was the highest number of different advice imperatives
found by a participant. Looking into multiple information sources
is also likely to yield new advice because 176 of 272 pieces of advice
were only included in one document (cf. Figure 2b). On average, apiece of advice was only contained in 2.1 documents (median: 1.0).
However, a few pieces of advice were covered in a higher numberof documents, as shown in Table 6. Offer/Use 2FA/MFA was the mostfrequent one, being contained in 35 documents.
All in all, advice is scattered, and most documents contained little
advice, with a few exceptions. Hence, developers must consider
multiple documents if they want to get a comprehensive set ofadvice. Looking at a single document would yield only a small
subset of advice if not looking at one of the rare dense documents.
Summary: Advice Distribution & Density. While documents contained 4.5 pieces of advice on average, most only contained two or less.
However, some documents like OWASP Cheat Sheets were denser. 176 of
272 pieces of advice were only covered by a single document.

CCS ’23, November 26–30, 2023, Copenhagen, Denmark

Jan H. Klemmer et al.

Table 6: Top-10 most frequent pieces of advice measured byoccurrence in documents.

Password 64

80

2FA/MFA 28 54

70

SSO/3rd-Party Login 16 14 28

Docs

Offer/Use 2FA/MFA.
Enforce a password policy.
Allow/Use 3rd-party SSO.
Use generic responses for auth. errors to not weaken security.
Require strong passwords.
Require re-authentication for sensitive actions (step-up auth).
Reject most often used passwords and words from dictionaries.
Limit the number of login attempts.
Protect against brute-force attacks by rate limiting.
Use password alternatives.

4.4.1 Identifying Topics. Three researchers independently used
an inductive coding strategy to identify advice topics within the
272 pieces of advice and assign each advice paraphrase to its respective topic(s). Afterward, the researchers discussed each individual
topic in a consensus session to create the final list of topics. It consists of 12 different topics (cf. Table 7). We assigned each topic adefinition to achieve a reproducible assignment for each piece ofadvice and describe the topic. With the final list of topics, each coder
independently coded all 272 pieces of advice. After the independent
coding, the coders finally compared and resolved existing conflicts
until they reached consensus. We allowed assigning multiple topics
if a piece of advice fits several topics at the same time. For example,
the advice to “Use password strength meters” covers passwords andalso implies user interface (UI)/UX requirements.
4.4.2 Results. Overall, we found 12 different advice topics, as depicted in Table 7 (including each topic’s definition, example advice,
and prevalence counts). Within the 12 topics, we found five topics
corresponding to major authentication types: passwords, 2FA/MFA,
single sign-on (SSO)/3rd-party login, token-based authentication, andbiometrics. Besides these five, some minor authentication types
emerged that we grouped into an other category (e.g., TLS clientcertificates). Advice on passwords forms the most common authentication type, with 101 distinct pieces of advice across 64 documents.
The topic covers several aspects, like password policies, reset, entry,
managers, or strength meters, and includes usability aspects, e.g.,
toggling visibility in password fields. Advice on 2FA and MFA is also
popular (54 documents), followed by advice on SSO, third-party,
and social logins. Token-based and biometric authentication were
only rarely discussed.
The remaining six topics were more general and not about aspecific authentication type. Usage recommendations, i.e., what authentication mechanism to use, were very common (83 documents).
Another topic spanning all authentication types was session andaccount management (e.g., account recovery). The remaining minor
topics were about user and usability aspects. They covered generalauthentication UI and UX, how to interact with the user throughnotifications and messages, or how to educate users.

0

5

3

5

0 12

Other 14 9

7

0

UI/UX 27 18 13 0
Session/Account Management 41 26 16 0
Usage Recommendations 43 52 27 5

Topics of Web Authentication Advice

Early on, we noticed that the advice covered various topics. For further investigation, we took a closer look and identified 12 different
topics in our dataset’s advice.

2

50

0 15
2 10 3640

730

9 24 67

7 15 27 38 83

20

Notif., Error- & Warning Msgs. 24 16 10 0

3

9 25 23 25 32

User Education & Engagement 7

5

2

0

0

3

5

6

7

5

8

Miscellaneous 8

6

4

0

1

3

8

7

8

7

1 10

10
0

SS

4.4

35
14
14
13
11
10
10
7
7
6

60

Biometrics 2
Token-Based Auth. 3PasO/3 2 sworrd- FA d
Pa /MF
rty A
Tok Bi Log
en om in-Ba etSe
sed rics
ssi
Au
on
th
/Ac
Ot .
Us couhe
No age nt
M U rt
Us if., Er Reco ana I/UX
er ro m ge
Ed r- & me me
uc W nd nt
ati ar at
on nin ion&E g sng MsgMis agem s.
ce en
lla t
ne
ou
s

AdviceFigure 3: Co-occurrence of the topics from Table 7 within thesame document.

The co-occurrence heatmap of topics within a document in Figure 3 reveals that authentication types are often not jointly covered
within the same document. For example, while password advice
is in 64 documents, only in 28 documents both passwords and2FA/MFA are discussed together. This is similar for passwords andSSO/3rd-party login. On average, a document touches 3.2 different
topics (median: 3.0). However, when just focusing on the authentication type topic subset, only 1.7 different authentication types are
discussed. 61 documents cover only one authentication type.
Summary: Advice Topics. Overall, we found a wide range of 12 different advice topics. Advice on passwords was by far the most frequent,followed by 2FA/MFA. The topics are often discussed in isolation, e.g.,
not discussing security benefits of 2FA when talking about passwords.

4.5

Recommendable Advice

Below, we provide details on advice we deem recommendable fordevelopers. We consider authentication advice recommendable if it
contributes to usable and secure authentication on the web.
4.5.1 Identifying Recommendable Advice. We based our assessmenton academic literature and official guidelines [1, 6, 7, 9, 12, 13, 17,
18, 22, 34, 36, 41, 42, 43, 44, 45, 48, 49, 50, 51, 55, 58, 59, 61, 63, 65,
69, 71, 73, 77, 80, 86, 95, 99, 104, 108, 109, 113, 114, 116, 117, 119]. In
addition to references from related work, we searched for literatureby checking the proceedings of top-tier venues, including IEEE S&P,
USENIX Security, ACM CCS, NDSS, USENIX SOUPS, ACM CHI,
and official resources (e.g., from U.S. National Institute of Standards
and Technology (NIST) [77]). This was complemented by searching
on Google Scholar and a general online search. We also leveraged
forward and backward snowballing to find further resources [118],
i.e., resources citing or cited by an already obtained resource.
Based on these literature references, two authors assessed foreach advice whether it could be considered recommendable. We
used discussions to obtain consensus decisions. For transparency,
we provide supporting references with their publication years to

A Qualitative Exploration of Online Developer Advice on Usable and Secure Authentication

CCS ’23, November 26–30, 2023, Copenhagen, Denmark

Table 7: Categorization of advice in different topics.
Advice Topic

Definition#1 #Adv.2 #Docs3

Advice Example

Authentication Type†
Passwords

Includes all advice, which directly mentioned a policy, a feature, or an implementa- • Enforce a minimum password length of 8 characters.
tion detail, that is directly or indirectly related to the creation, change, or processing • Reject passwords from breaches.
of passwords.

264

101

64

2FA/MFA

Advice related to two-factor or in general multi-factor authentication and thedifferent approaches but also backup and recovery and general considerations.

• Require adding a 2nd factor for features with higher security.
• Provide OTP as 2FA.
Recommendations related to the implementation or use of SSO, 3rd-party authenti- • Use OAuth 2.0
cation services, or social logins, e.g., via Facebook or Google.
• Stick to a few common social/third-party providers.
Authentication mechanisms based on tokens, their design, and usage. This also • Give tokens an expiration.
includes token formats such as Bearer or JWT. Cookie- and session-based are • Limit number of token usages.

87

36

54

50

20

28

23

16

12

SSO/3rd-Party
Login
Token-Based
Authentication

authentication also included.
Biometrics

Includes all advice which directly is related to any kind of biometric authentication • Prefer fingerprint scanning above facial recognition.
method.
• Use a traditional mechanism (e.g., password) beside biometrics.

11

8

5

Other

Advice related to other authentication types, not belonging to one of the other • Do not use TLS Client Authentication for public websites.
authentication types.
• Use passwordless authentication.

26

10

14

188

71

83

Usage
Recommendations

Advice on how to choose or which authentication method to use.

Session/AccountManagement

Advice on management of sessions and accounts, including login and logout process, • Require re-authentication for sensitive actions (step-up auth).
account creation/registration, recovery, blocking, user identifiers, etc.
• Invalidate all sessions after password change.

225

103

63

UI/UX

Concrete request to change, integrate or implement features and elements in the • Use generic response code for auth/reg/recovery errors
UI in a specific way, e.g., how to design a password field. Also includes advice that • Don’t prohibit pasting passwords.
directly mentions some aspect of user experience.

104

50

35

Notifications, Error-,& Warning Messages

Advice on informing the user through notifications and messages in general, their • Notify users via mail when password was reset.
design and usage, including push notifications, email, or SMS/text message, but also • Avoid sending sensitive data to push notifications / backups / logs
warnings, error messages.

59

27

27

User Education& Engagement

Advice on engaging or educating users on different topics of authentication.

• Educate users to create strong passwords.
• Encourage users to accept passwords from password managers.

12

7

8

Miscellaneous

Includes all advice that does not fit into any other category.

• Use captchas after failed attempts.
• Involve stakeholders early on for security requirements.

18

9

10

† Container for sub-topics; containing no advice.

1 Advice occurrences in a topic.

• Offer/Use 2FA/MFA.
• Use risk-based auth controls.

2 Distinct pieces of advice in a topic.

indicate their age. Additionally, we provide written rationales that
explain our reasoning within our replication package (Appendix A).
We considered 40 pieces of advice recommendable without having a reference or scientific evidence. Therefore, we excluded thisadvice in the results below and call it potentially recommendable.
Potentially recommendable advice includes advice such as “Usesocial login if building apps on top of Google or Facebook” or “Avoid
login completely if not necessary”. For completeness, we provide thefull list of potentially recommendable in the appendix, Table 10.
4.5.2 Results. We considered 86 of 272 pieces of advice (31.6%)
recommendable. Overall, 92 documents (74.2%) contained recommendable advice. Table 8 provides an overview of the most frequent
recommendable advice. Comparing this to the overall most frequentadvice (Table 6) shows some minor differences. For example, we
did not consider the advice to “Require strong passwords” recommendable because it does not provide concrete guidance. However,
we consider the related advice to “Enforce a password policy” recommendable since official institutions generally recommend password
policies [77 (2017)], and there is scientific support for the use ofpassword policies (while the concrete policy should be adapted tothe context) [55 (2011), 60 (2022)].
Among the advice that is only contained in one or two documents, we found highly recommendable advice for developers. For
example, “Don’t prohibit pasting passwords” (to not interfere with
PWMs [48 (2021)]) or to “Provide a password change feature.” The latter is crucial to change compromised passwords, but to our surprise
only mentioned in two documents—maybe because it is too obvious. Advice like “Don’t force users to regularly change passwords”

3 Documents containing advice from a topic.

Table 8: Top-10 most frequent recommendable advice measured by occurrence in documents.
AdviceOffer/Use 2FA/MFA.
Enforce a password policy.
Allow/Use 3rd-party SSO.
Require re-authentication for sensitive actions (step-up auth).
Reject most often used passwords and words from dictionaries.
Limit the number of login attempts.
Protect against brute-force attacks by rate limiting.
Provide secure password recovery mechanism
Enforce minimum password length
Use password strength meters.

Docs
35
14
14
10
10
7
7
6
6
6

is recommendable, as it replaces the outdated advice to enforce
regular password changes [42 (2018), 77 (2017)]. Other advice also
gives recommendations on specific pitfalls to avoid. For example, tonot use TLS client certificates for public websites is recommendable
because TLS client certificates are hard to use [85 (2014)].
Summary: Recommendable Advice. We found 31.6% of the advice tobe generally recommendable to developers, as it adheres to state-of-theart research results or official guidelines. Additionally, we consider 14.7%
of advice potentially recommendable that lacked references for evidence.

4.6

Debatable Advice

Besides recommendable advice, we considered other advice to bedebatable whether developers should follow it. This includes if it
contributes to insecure or unusable authentication on the web, isCCS ’23, November 26–30, 2023, Copenhagen, Denmark

inconsistent with literature references, is only applicable in certainuse cases, or different opinions on the advice exist.
4.6.1 Identifying Debatable Advice. We identified debatable advice
as we did for recommendable advice (cf. Sect. 4.5.1). Using the samereferences, two researchers assessed the advice and discussed debatable advice in a consensus session. In cases without a supporting
reference, we only tagged advice as debatable if a clear rationale
existed. For traceability, we summarized the rationals of why advice is debatable in Table 11. All 49 debatable pieces of advice areavailable in the replication package (Appendix A).
4.6.2 Results. We considered 49 of 272 pieces of advice debatable.
Overall, we found debatable advice in 39 documents (31.5%). For
a convenient overview, we grouped similar debatable advice in
14 groups (referred to as Dx) and reported it along with the rationale
why it is debatable and examples in Table 11.
Most debatable advice covered password policies (D1) and was
discussed in 22 documents. D1 is a good example of outdated advice.
Outdated advice might have been considered correct in the past,
but in the meantime, new insights emerged that render it incorrect. For example, we found advice to enforce complex composition
rules, e.g., passwords should consist of lower- and uppercase letters,
numbers, and symbols. Another example is to “Force users to change
passwords regularly.” However, research uncovered its negative effect on authentication usability and security [6 (1999), 42 (2018),
50 (2010), 55 (2011)]. Likewise, official organizations such as NIST
changed to less restrictive password policies in 2017 to improveusability and security compared to previous password policies [77(2017)]. However, we also found up-to-date advice that is less strict
and only suggests enforcing a minimal password length, as also
recommended by OWASP [81 (2021)].
We found more outdated advice recommends the use of security questions (D3), despite their usability [12 (2015)] and serious
security shortcomings [13 (2012), 99 (2009)], and official guidelines advising against them [77 (2017)]. We found other advice thatharms (usable) security, such as not allowing to toggle masking
in password fields and viewing the password in plain text (D11),
automatically terminating all sessions on password change without
asking the user (D6), or stating if a user entered an old but nottheir current password (D7). Some advice is more debatable, and its
appropriateness depends on the use case. For example, consider theadvice to use text messages (SMS) for authentication purposes (D5).
On the one hand, text messages have drawbacks like the possibility of losing access to a phone number [59 (2021), 69 (2021)], SIMswapping attacks [58 (2020)], and being interceptable [78 (2016)].
On the other hand, text messages are convenient as smartphones
are ubiquitous. Similarly, PGP-encrypted emails for sending reset tokens (D12) provide good security, but PGP has only poor
adoption [108 (2022)]. Some advice is challenging to assess, likefingerprints always being preferable to facial recognition (D13),
while vendors like Apple changed from fingerprint to Face ID [9(2017)].
Summary: Debatable Advice. In 31.5% of the documents, we founddebatable advice that conflicts with research findings and official guidelines. The debatable advice also contains instances of outdated advice,wrong advice, and advice that harms usable security.

Jan H. Klemmer et al.

4.7

Contradicting Advice

Besides recommendable (Sect. 4.5) and debatable advice (Sect. 4.6),
i.e., advice in line or in conflict with literature references, we foundseveral examples of contradicting advice within the document corpus itself. To analyze those, we identified sets of advice that contradict each other (e.g., “Do X” and “Don’t do X”). We refer to those as
contradiction groups. Overall, we identified 15 contradiction groups
with a total of 49 distinct pieces of advice (cf. Table 9). Most contradiction groups contained one pair of pieces of advice that directly
contradict each other. We illustrate our key findings below.
Most contradiction groups addressed passwords (C2, C4, C5, C7,
C11, C13). For example, C4 is about whether to enforce regular password changes for users or not. That contradiction group’s advice
states two opposite approaches: to “Force users to change passwords
regularly” (6 documents) and “Don’t enforce regular password updates/changes for users” (2 documents). Similar disagreements canbe found in password length maximum (C2) and minimum (C5)requirements or password visibility (C7). But contradicting advice
also exists for other authentication areas, like session timeouts (C3,
C14), rate limiting (C8, C12), 2FA (C10), or general discussions onwhether to use authentication at all (C6).
Another important aspect is how contradicting advice occurs
within a single document and within the document corpus, i.e.,
across multiple documents. Regarding intra-document contradicting advice, we found this to be almost non-existent. Actually, only
2 documents contain contradicting pieces of advice. However, thislow intra-document contradiction rate is to be expected because itis very unlikely that an author is inconsistent within a document.
Analyzing the inter-document advice contradictions, we identified
60 documents (48.4%) that contained at least one piece of advice
belonging to a contradiction group. That said, the document corpus contains many contradictions, indicating inconsistency andunreliability of online advice.
Contradiction groups contain opposite advice by definition. Especially for debatable topics, contrary recommendations and opinions
exist that are sometimes mixed with recommendable advice (e.g.,
C1 or C4) and discussed in the document corpus. Therefore, theprevalence of debatable advice (Sect. 4.6) among the contradictiongroups is considerably higher at 30.6% compared to the overall
18.0% rate of debatable advice.
Summary: Contradicting Advice. We found 15 groups of contradicting advice. While it is almost non-existent within a single document,
48.4% of the documents contained some piece of contradicting advice.
Contradiction groups tend to contain more debatable advice.

5

DISCUSSION

Below, we summarize our key insights and discuss recommendations for developers, advice writers, and academia. This section
concludes with a discussion of future work.

5.1

Potential Impact of Advice on
Authentication Deployments

Usable and secure web authentication is challenging, and previous
work uncovered many issues, such as the widespread use of passwords in general [13, 14], the deployment of complicated password

A Qualitative Exploration of Online Developer Advice on Usable and Secure Authentication

Table 9: Overview of the 15 contradiction groups, each witha short description and advice examples (in italics).
ID Description & Advice Examples
C1 How authentication errors should be reported to the users.
• Use generic responses for authentication errors to not weaken security• Tell if user types in an old password
C2 Whether to set a maximum password length or not.
• Set a maximum password length• Don’t set a maximum password length
C3 Keeping users logged in or automatically log out.
• Provide automatic logout/expire session• Keep users signed in
C4 Whether to enforce regular password changes or not.
• Force users to change passwords regularly• Don’t force users to regularly change/update passwords
C5 Different advice on minimum password length.
• Use a minimum of 8 characters• Use a minimum of 12–16 characters
C6 Whether to use logins or not.
• Always require/enforce authentication• Avoid login completely if possible/not necessary
C7 Whether to allow toggling password field visibility or not.
• Don’t display passwords on the screen• Allow users to toggle password visibility when typing it
C8 Whether to use IP addresses or account for rate limits.
• Limit by account, not by IP• Limit by IP
C9 Using OAuth or OpenID Connect for authentication.
• Implement 3rd-party authentication via OAuth• Use OpenID Connect for authentication
C10 Which 2FA approach to choose.
• Provide OTPs as 2FA option• Use mobile device push notifications as second factor
C11 Different speed requirements for password hashing.
• Slow down hash functions• Choose work factors so that hashing time <1 sec
C12 Different rate limits to prevent brute force attacks.
• Limit to 3 attempts per account in a set timespan• Limit to 20 guesses per minute and IP address
C13 Entering passwords twice during registration or not.
• Let users confirm passwords by writing twice• Don’t ask for password confirmation
C14 How long should tokens for authentication live.
• Limit number of token usages• Only allow one-time usage
C15 Whether to use CAPTCHAs or not.
• Use CAPTCHAs• Use alternatives for CAPTCHAs#Adv. #Docs

CCS ’23, November 26–30, 2023, Copenhagen, Denmark

best practices for password policies [60], and advice is also often notin line with best practices (Sect. 4.6, Sect. 4.7). Developers are notnecessarily authentication or usable security experts. They might
follow the advice unaware of the drawbacks and alternatives.
Considering that developers rely on online resources for solving
security programming challenges [2, 27], we think that online advice is a major factor that affects web authentication usable security
negatively, e.g., outdated but still followed advice (e.g., changing
passwords regularly) or following advice that was always known
to counteract usability and security (e.g., using security questions).

6

16

4

10

6

9

2

8

5.2

5

7

3

7

Based on our findings, we identified four main challenges for developers. Overall, the results indicate a mixture of advice that could andshould be followed and advice that should be carefully consideredor avoided at all.

2

6

2

6

2

6

3

5

2

5

4

4

2

3

4

2

2

2

policies [55, 60, 102, 103], and the low adoption of password alternatives such as MFA and FIDO2 [8, 13, 18, 24, 32, 87, 90]. Moreover,developers regularly depend on online resources to solve programming problems [2, 3, 5, 16, 120] and might blindly trust informationfrom blogs, documentation, or platforms such as SO [27, 28, 29].
Our findings illustrate how online advice for web authentication
might contribute to unusable and insecure authentication deployments. When developers seek online advice to solve challenges
around online authentication, they likely stumble upon advice thatis of limited help to make web authentication usable and secure.
Most advice covered technical security implementation details thatare not relevant for end-users, while only 272 of 1,208 pieces of advice related to end-users’ usable security (Sect. 4.2). Contradicting,debatable, and outdated advice might even have a negative effect.
For example, passwords were—despite their known potentiallynegative impact on security and usability—the most prevalent advice topic and often the only discussed authentication approach(Table 7). The drawbacks and potential alternatives are rarely mentioned. We believe that such advice contributes to passwords’ undiminished high popularity. Similarly, many websites do not follow

Challenges for Developers

Advice Diversity & Priotrization. The advice was highly diverse (Sect. 4.4), i.e., we found lots of advice on a single topic (e.g.,
password policies). This and the high advice volume indicate an
overload for developers and a lack of advice prioritization among its
authors, as similarly found in related work on end-user advice [91].
Scattered Advice. The advice was also distributed over many documents (Sect. 4.3). While some documents contained more advice,many contained only a single piece of advice. Moreover, most advice was only covered in a single document. This makes it harderfor developers to find relevant advice; they might have to consider
many documents to find helpful advice for their needs.
Debatable Advice. Debatable advice (Sect. 4.6) challenges developers, as they would have to recognize and not blindly follow
detrimental advice. However, it cannot be assumed that developers (i) are usable security experts and, for example, know which
advice is outdated and should not be followed anymore; or (ii) canrecognize the consequences that poor usability of authentication
features can have [41].
Contradicting Advice. Finding contradicting advice (Sect. 4.7)
obliges the developer to decide which advice to follow and might be
confusing. As with debatable advice, it is unclear whether developers have the skills to make an educated decision and assess relatedconsequences. Even when developers encounter only one pieceof contradicting advice, chances are to blindly follow debatable orotherwise disadvantageous advice instead of best practices.

5.3

Recommendations

Given the potential impact of advice on authentication deployments
and the associated challenges for developers, we advocate to rethink
how advice is given and consumed to achieve authentication for
end-users that is both more usable and secure. Based on our findings,
we make the following recommendations for improving security
advice for developers in the future.
Developers. We found that online security advice on web authentication is diverse and distributed over many resources on the web.
Overall, developers should consider their context and not follow
advice blindly, as we found debatable advice that could not be unconditionally recommended (Sect. 4.6). Such assessments are mostly
impractical for developers since they require lots of expertise, effort,

CCS ’23, November 26–30, 2023, Copenhagen, Denmark

and literature research. Hence, reasonable resources that can beassumed to provide high-quality advice are necessary.
We recommend developers look into the OWASP Cheat Sheets.
According to our results, they contain the most advice and, therefore, are the most extensive resource, obviating the need for developers to look at many other resources that provide little or no advice.
Another reason is that OWASP Cheat Sheets contain up-to-dateadvice specifically tailored to software practitioners.
Giving Advice. We found several instances of outdated, incorrect,
or otherwise debatable advice. Hence, authors should check theiradvice to avoid such problems before giving advice. This also implies regularly updating advice to prevent giving outdated advice.
Unfortunately, checking advice is not straightforward, requiringexpertise and considering resources such as the latest usable security research. Since advice is highly dependent on specific contexts,
advice givers should describe in which situations advice should be
followed or not to prevent developers from following advice that
does not suit their project.
As already discussed, we found parallels between advice andthe authentication ecosystem. We argue that advice is related toand could influence the authentication landscape. Therefore, advice
givers should carefully choose what authentication topics they
write about. That does not mean that no advice on passwords should
be given. However, the authors should focus on approaches thatprevent weaknesses in passwords. For example, when writing aboutpasswords, their drawbacks (e.g., phishing, password reuse) should
also be mentioned, and countermeasures (e.g., 2FA) or alternatives(e.g., FIDO2) should be outlined.
Official Institutions. While we mentioned OWASP Cheat Sheets,the reader may wonder where official organizations stand as asource of advice. To our surprise, participants rarely reported official
documents. For example, no NIST Special Publication was reported.
This is especially remarkable as several documents referenced NIST
guidelines and considered those authoritative. A closer look at
OWASP Cheat Sheets [82] and the related NIST SP [77] quickly
reveals major differences. For example, NIST’s PDF is much longer
at 79 pages than OWASP Cheat Sheets. The language in NIST’s
documents is also more abstract, e.g., passwords are referred to as“memorized secret authenticators” [77]. We hypothesize that thesefactors lower the accessibility of NIST documents for developers
and our participants did not report them for this reason.
Nonetheless, official institutions like NIST are reputable, authoritative organizations whose documents contain reasonable advice.
We propose that official institutions provide their advice in a form
similar to OWASP cheat sheets. This could be done by condensing the most important advice from their extensive guidelines,
standards, and other documents and presenting it in a developerfriendly and actionable way.
Authoritative Advice. We obtained 272 distinct pieces of advice
that showed high diversity and were scattered all over the internet.
This imposes a challenge, as no developer would search 406 documents for advice as we did. Hence, creating an authoritative sourcewith helpful, actionable, carefully constructed web authentication
advice for developers might improve accessibility, advice adoption, and end-user usable security. Moreover, this might prevent
encountering debatable or contradicting advice.

Jan H. Klemmer et al.

It is yet unclear who should create and maintain authoritative
advice. As already discussed, official institutions like NIST are considered authoritative but lack accessibility. Since this is mainly apresentation issue, we argue that official institutions could take thisrole. We found OWASP Cheat Sheets [82] to be the most comprehensive knowledge base for authentication advice. Hence, OWASP
should continue the Cheat Sheet Series project, while official institutions should start creating similar resources. The advice extracted
in this paper might be a starting point for creating (or extending)authoritative usable security authentication advice.
Knowledge Transfer: Academia to Practice. Unsurprisingly,
developers do not consider scientific resources like papers helpful(Figure 1). Developers are unlikely to read dozens of papers thatmight require expert knowledge or are hidden behind paywalls.
Instead, developers prefer online resources and heavily draw on
contained advice [2, 3, 5]. However, academic papers yield new usable security knowledge and are the main communication channel
for scientific results. Their impact is limited if they do not foster any
change and gain real-world traction. For developer authentication
advice, we can confirm the knowledge gap between academia andpractice that Gutfleisch et al. identified [41]. To overcome this gap,
researchers should aim to provide their knowledge in ways thatdevelopers can easily use and access. For example, new research
results could be added to OWASP Cheat Sheets. While this can improve the adoption of advice that is grounded on research insights,
the adoption is likely limited overall. Recent research indicates that
Cheat Sheet advice is only partially followed [51].

5.4

Comparison with Related Work

While previous research investigated online security advice forboth end-users [15, 23, 46, 52, 89, 91, 93] and software developers [2, 3, 5, 10, 11, 16, 27, 28, 29, 31, 39, 76, 92, 120], we are thefirst to explore developer advice on authentication that affects usable security for end-users. As already discussed, we found similar
themes compared to related work, like challenges around advice
diversity and prioritization [91], and a usable security knowledge
gap between academia and industry [41]. Redmiles et al. investigated security advice regarding its quality (e.g., actionability) ina user study [91]. As we performed a researcher evaluation, we
refrained from evaluating aspects like developer actionability. Suchan assessment with developers, similar to Redmiles et al. [91], ispotential future work and possible with our dataset. This might beinteresting, as our overall impression is that quality is mixed, e.g.,some advice is highly concrete (e.g., containing source code) whileother is vague.
Moreover, our work might provide some explanations for observations in related work. While not directly comparable, we seeparallels to the findings of Lee et al. [60]. Lee et al. found that
only 13% of websites followed all password policy best practices.
A contributing factor to this can be developer advice—due to theplethora of different advice. Some advice might be recommendable,
and others debatable, while potentially contradicting each other at
the same time. Best practices identified by Lee et al., e.g., block lists
and strength meters, were also recommendations in our dataset but
rarely occurred, reflecting the low adoption of best practices. This
low advice adoption might be caused by the challenges (Sect. 5.2)

A Qualitative Exploration of Online Developer Advice on Usable and Secure Authenticationthat can become a burden for developers, ultimately resulting in apoor cost–benefit trade-off similar to the one outlined by Herleyin 2009 for end-user security advice [47]. Overall, we note that
not only developers impact usable security of authentication. Also,
system administrators configuring authentication have an impact
and draw on advice, e.g., consulting standards and guidelines, as
recently found by Sahin et al. [98]. The authors identified similar
challenges compared to our study, like administrators facing andfollowing outdated recommendations.
In our dataset, the fraction of documents from Q&A forums
like SO (3.7%) is lower than in related work (e.g., 10–40% [2]). As
participants were free to choose any resource, we think the lower
prevalence is due to different study characteristics. SO posts arelikely most useful for programming as they provide code snippets
for copy and paste [27]. However, our scenario is a more abstract,
conceptual task that does not include programming, while Acar etal. [2] experimented with developers writing code. Other factors
might be self-reporting and social desirability biases that prevent
participants from reporting SO because the platform is also known
to contain low-quality or insecure answers [16, 27]. While we canonly speculate about the reasons, future research is needed to investigate which resources software professionals prefer for abstract,
conceptual tasks like ours.

CCS ’23, November 26–30, 2023, Copenhagen, Denmark

6

CONCLUSION

In this work, we qualitatively analyzed 272 pieces of advice containing developer advice on usable and secure authentication. We
identify several major challenges for developers. For example, theadvice is scattered over many resources and, therefore, hard to find.
Despite their shortcoming, passwords are the most discussed authentication approach; the advice lacks the discussion of drawbacks
and alternatives. Moreover, unconsciously following advice can induce usable security problems, as we found instances of debatable,
outdated, and contradicting advice. Overall, our findings suggestthat the advice potentially impacts authentication deployments.
Our recommendations can inform better advice giving to improveusable and secure authentication on the web.

ACKNOWLEDGMENTS
We thank the anonymous reviewers for their valuable feedback
and for helping us to improve this paper. We thank our study’s participants for taking their time and allowing us to gain interesting
insights. This research was funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s
Excellence Strategy – EXC 2092 CaSa – 390781972.

REFERENCES
[1]

5.5

Outlook & Future Work
[2]

Changing advice and the authentication landscape can be considered long-term projects requiring suitable technical solutions.
The endeavor to replace passwords has been an ongoing effort for
decades [6] that turned out to be hard [13] and has not been successful yet. Regarding advice, passwords are the prevailing topic.
Therefore, we believe passwords to stay for more years to come.
Users also still tend to prefer passwords [122]. However, the industry is trying to offer and shift to passwordless alternatives. In 2022,
Apple, Microsoft, and Google announced plans to extend FIDO2
and introduce passkeys [25]. Passkeys aim to replace passwords bysyncing FIDO credentials across the users’ devices. While passkeys
seem promising, future research is needed to study their effect. The
advent of passkeys also calls for a future replication of this study
to check how online advice evolves and how passkeys are covered.
While we qualitatively analyzed developer-reported web authentication advice in this paper, our work does not investigate
developers’ interaction with advice and advice impact on authentication usable security. Conceptional decisions (e.g., whether 2FA is
implemented or not) might not be decided by developers alone [41].
Hence, further investigation of advice adoption and decision processes is needed. One aspect of this is the prioritization of advice.
We refrained from indicating recommendation strength for the advice, as a valid assessment requires a broader expert evaluation
beyond the involved researchers. Future work could investigate
recommendation strength with experienced software professionals.
We also propose to conduct a developer experiment focusing on theimpact of advice on the final software to understand how advice
should be given to developers. Similarly, future work could investigate the impact of code snippets, as we only focussed on textual(prose) advice. We hope that our dataset can lay the foundation for
future work in these directions.

[3]

[4]

[5]

[6]

[7]

[8]

[9]
[10]

[11]

[12]

[13]

Jacob Abbott and Sameer Patil. 2020. How Mandatory Second Factor Affects
the Authentication User Experience. In Proc. 2020 CHI Conference on Human
Factors in Computing Systems (CHI’20). ACM.
Yasemin Acar, Michael Backes, Sascha Fahl, Doowon Kim, Michelle L Mazurek,
and Christian Stransky. 2016. You Get Where You’re Looking For: The Impact
of Information Sources on Code Security. In Proc. 37th IEEE Symposium on
Security and Privacy (SP’16). IEEE.
Yasemin Acar, Michael Backes, Sascha Fahl, Doowon Kim, Michelle L. Mazurek,
and Christian Stransky. 2017. How Internet Resources Might Be Helping You
Develop Faster but Less Securely. IEEE Security & Privacy, 15, 2, 50–60.
Yasemin Acar, Sascha Fahl, and Michelle L. Mazurek. 2016. You are Not Your
Developer, Either: A Research Agenda for Usable Security and Privacy Research Beyond End Users. In Proc. 2016 IEEE Secure Development Conference(SecDev’16). IEEE.
Yasemin Acar, Christian Stransky, Dominik Wermke, Charles Weir, Michelle L
Mazurek, and Sascha Fahl. 2017. Developers Need Support Too: A Survey of Security Advice for Software Developers. In Proc. 2017 IEEE Secure Development
Conference (SecDev’17). IEEE.
Anne Adams and Martina Angela Sasse. 1999. Users are not the enemy: Whyusers compromise security mechanisms and how to take remedial measures.
Communications of the ACM, 42, 12, 40–46.
Devdatta Akhawe and Adrienne Porter Felt. 2013. Alice in warningland: alarge-scale field study of browser security warning effectiveness. In Proc. 22nd
USENIX Security Symposium (SEC’13). USENIX.
Elham Al Qahtani, Lipsarani Sahoo, Yousra Javed, and Mohamed Shehab.
2022. "Why Would Someone Hack Me out of Thousands of Students": Video
Presenter’s Impact on Motivating Users to Adopt 2FA. In Proc. 27th ACM
Symposium on Access Control Models and Technologies (SACMAT’22). ACM.
Apple Inc. 2017. Face ID Security. (Nov. 2017). https://www.apple.com/busine
ss-docs/FaceID_Security_Guide.pdf.
David Barrera, Christopher Bellman, and Paul C. van Oorschot. 2022. A Close
Look at a Systematic Method for Analyzing Sets of Security Advice. (2022).
https://arxiv.org/abs/2209.04502 arXiv: 2209.04502 [cs.CR].
David Barrera, Christopher Bellman, and Paul C. van Oorschot. 2023. Security
Best Practices: A Critical Analysis Using IoT as a Case Study. ACM Trans. Priv.
Secur., 26, 2, Article 13, 30 pages.
Joseph Bonneau, Elie Bursztein, Ilan Caron, Rob Jackson, and Mike Williamson.
2015. Secrets, Lies, and Account Recovery: Lessons from the Use of Personal
Knowledge Questions at Google. In Proc. 24th Int. Conf. on World Wide Web(WWW’15). ACM.
Joseph Bonneau, Cormac Herley, Paul C Van Oorschot, and Frank Stajano.
2012. The quest to replace passwords: A framework for comparative evaluationof web authentication schemes. In Proc. 33rd IEEE Symposium on Security andPrivacy (SP’12). IEEE.

CCS ’23, November 26–30, 2023, Copenhagen, Denmark

[14]

[15]

[16]

[17]

[18]

[19]

[20]

[21]

[22]

[23]

[24]

[25]

[26]
[27]

[28]

[29]

[30]

[31]

[32]

[33]

[34]

Joseph Bonneau, Cormac Herley, Paul C. van Oorschot, and Frank Stajano.
2015. Passwords and the Evolution of Imperfect Authentication. Commun.
ACM, 58, 7, 78–87.
Karoline Busse, Julia Schäfer, and Matthew Smith. 2019. Replication: No One
Can Hack My Mind Revisiting a Study on Expert and Non-Expert Security
Practices and Advice. In Proc. 15th Symposium on Usable Privacy and Security(SOUPS’19). USENIX.
Mengsu Chen, Felix Fischer, Na Meng, Xiaoyin Wang, and Jens Grossklags.
2019. How Reliable is the Crowdsourced Knowledge of Security Implementation? In Proc. IEEE/ACM 41st International Conference on Software Engineering(ICSE’19). IEEE.
Geumhwan Cho, Jun Ho Huh, Soolin Kim, Junsung Cho, Heesung Park, Yenah
Lee, Konstantin Beznosov, and Hyoungshick Kim. 2020. On the Security
and Usability Implications of Providing Multiple Authentication Choices on
Smartphones: The More, the Better? ACM Trans. Priv. Secur., 23, 4, Article 22.
Jessica Colnago, Summer Devlin, Maggie Oates, Chelse Swoopes, Lujo Bauer,
Lorrie Cranor, and Nicolas Christin. 2018. “It’s Not Actually That Horrible”:
Exploring Adoption of Two-Factor Authentication at a University. In Proc.
2018 CHI Conference on Human Factors in Computing Systems (CHI’18). ACM.
Sanchari Das, Andrew Dingman, and L. Jean Camp. 2018. Why Johnny Doesn’t
Use Two Factor A Two-Phase Usability Study of the FIDO U2F Security Key.
In Financial Cryptography and Data Security. Sarah Meiklejohn and Kazue
Sako, (Eds.) Springer, 160–179.
Brian Dean. 2019. We analyzed 5 million Google Search Results: Here’s What
We Learned About Organic Click Through Rate. Accessed: 2021-11-26. Backlinko, (2019). https://backlinko.com/google-ctr-stats.
Xavier de Carné de Carnavalet and Mohammad Mannan. 2014. From Very
Weak to Very Strong: Analyzing Password-Strength Meters. In Proc. 21st
Annual Network and Distributed System Security Symposium (NDSS’14). The
Internet Society.
European Commission. 2015. DIRECTIVE (EU) 2015/2366. Off. J. Eur. Union,
337. https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:320
15L2366&from=EN.
Michael Fagan and Mohammad Maifi Hasan Khan. 2016. Why Do They Do
What They Do?: A Study of What Motivates Users to (Not) Follow Computer
Security Advice. In Proc. 12th Symposium on Usable Privacy and Security(SOUPS’16). USENIX.
Florian M. Farke, Lennart Lorenz, Theodor Schnitzler, Philipp Markert, andMarkus Dürmuth. 2020. “You still use the password after all” – Exploring
FIDO2 Security Keys in a Small Company. In Proc. 16th Symposium on Usable
Privacy and Security (SOUPS’20). USENIX.
FIDO Alliance. 2022. Apple, Google and Microsoft Commit to Expanded
Support for FIDO Standard to Accelerate Availability of Passwordless SignIns. Accessed: 2022-11-27. (2022). https://fidoalliance.org/apple-google-andmicrosoft-commit-to-expanded-support-for-fido-standard-to-accelerate-a
vailability-of-passwordless-sign-ins/.
FIDO Alliance. 2022. FIDO2. Accessed: 2022-11-09. (2022). https://fidoalliance.org/fido2/.
Felix Fischer, Konstantin Böttinger, Huang Xiao, Christian Stransky, Yasemin
Acar, Michael Backes, and Sascha Fahl. 2017. Stack Overflow Considered
Harmful? The Impact of Copy&Paste on Android Application Security. In
Proc. 38th IEEE Symposium on Security and Privacy (SP’17). IEEE.
Felix Fischer, Yannick Stachelscheid, and Jens Grossklags. 2021. The Effectof Google Search on Software Security: Unobtrusive Security Interventions
via Content Re-Ranking. In Proc. 28th ACM Conference on Computer andCommunication Security (CCS’21). ACM.
Felix Fischer, Huang Xiao, Ching-Yu Kao, Yannick Stachelscheid, Benjamin
Johnson, Danial Razar, Paul Fawkesley, Nat Buckley, Konstantin Böttinger,
Paul Muntean, and Jens Grossklags. 2019. Stack Overflow Considered Helpful!
Deep Learning Security Nudges Towards Stronger Cryptography. In Proc.
28th Usenix Security Symposium (SEC’19). USENIX.
Anuj Gautam, Shan Lalani, and Scott Ruoti. 2022. Improving Password Generation Through the Design of a Password Composition Policy Description
Language. In Proc. 18th Symposium on Usable Privacy and Security (SOUPS’22).
USENIX.
Lisa Geierhaas, Anna-Marie Ortloff, Matthew Smith, and Alena Naiakshina.
2022. Let’s Hash: Helping Developers with Password Security. In Proc. 18th
Symposium on Usable Privacy and Security (SOUPS’22). USENIX.
Sanam Ghorbani Lyastani, Michael Schilling, Michaela Neumayr, Michael
Backes, and Sven Bugiel. 2020. Is FIDO2 the Kingslayer of User Authentication?
A Comparative Usability Study of FIDO2 Passwordless Authentication. In
Proc. 41st IEEE Symposium on Security and Privacy (SP’20). IEEE.
Maximilian Golla and Markus Dürmuth. 2018. On the Accuracy of Password
Strength Meters. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (CCS’18). ACM.
Maximilian Golla, Grant Ho, Marika Lohmus, Monica Pulluri, and Elissa
M Redmiles. 2021. Driving 2FA Adoption at Scale: Optimizing Two-Factor

Jan H. Klemmer et al.

[35]

[36]
[37]
[38]

[39]

[40]
[41]

[42]

[43]

[44]

[45]
[46]

[47]

[48]

[49]

[50]

[51]

[52]

[53]

[54]

[55]

Authentication Notification Design Patterns. In Proc. 30th USENIX Security
Symposium (SEC’21). USENIX.
Maximilian Golla, Miranda Wei, Juliette Hainline, Lydia Filipe, Markus Dürmuth, Elissa Redmiles, and Blase Ur. 2018. “What Was That Site Doing withMy Facebook Password?”: Designing Password-Reuse Notifications. In Proc.
25th ACM Conference on Computer and Communication Security (CCS’18).
ACM.
Google. 2022. reCAPTCHA. Accessed: 2022-11-27. (2022). https://www.googl
e.com/recaptcha/about/.
Google Inc. 2022. Programmable Search Engine. (2022). https://developers.go
ogle.com/custom-search.
Peter L. Gorski, Emanuel von Zezschwitz, Luigi Lo Iacono, and Matthew
Smith. 2019. On providing systematized access to consolidated principles,
guidelines and patterns for usable security research and development. Journalof Cybersecurity, 5, 1.
Peter Leo Gorski, Luigi Lo Iacono, Dominik Wermke, Christian Stransky,
Sebastian Möller, Yasemin Acar, and Sascha Fahl. 2018. Developers Deserve
Security Warnings, Too: On the Effect of Integrated Security Advice on Cryptographic API Misuse. In Proc. 14th Symposium on Usable Privacy and Security(SOUPS’18). USENIX.
Matthew Green and Matthew Smith. 2016. Developers are Not the Enemy!:
The Need for Usable Security APIs. IEEE Security & Privacy, 14, 5, 40–46.
Marco Gutfleisch, Jan H. Klemmer, Niklas Busch, Yasemin Acar, M. Angela
Sasse, and Sascha Fahl. 2022. How Does Usable Security (Not) End Up in
Software Products? Results From a Qualitative Interview Study. In Proc. 43rdIEEE Symposium on Security and Privacy (SP’22). IEEE.
Hana Habib, Pardis Emami Naeini, Summer Devlin, Maggie Oates, Chelse
Swoopes, Lujo Bauer, Nicolas Christin, and Lorrie Faith Cranor. 2018. User
Behaviors and Attitudes Under Password Expiration Policies. In Proc. 14th
Symposium on Usable Privacy and Security (SOUPS’18). USENIX.
Marian Harbach, Sascha Fahl, Thomas Muders, and Matthew Smith. 2012.
Towards Measuring Warning Readability. In Proc. 2012 ACM Conference on
Computer and Communications Security (CCS’12). ACM.
Marian Harbach, Sascha Fahl, Polina Yakovleva, and Matthew Smith. 2013.
Sorry, I don’t get it: An analysis of warning message texts. In International
Conference on Financial Cryptography and Data Security. Springer, 94–111.
Dick Hardt. 2012. The OAuth 2.0 authorization framework. (2012). https://dat
atracker.ietf.org/doc/html/rfc6749.
Ayako A. Hasegawa, Naomi Yamashita, Tatsuya Mori, Daisuke Inoue, andMitsuaki Akiyama. 2022. Understanding Non-Experts’ Security- and PrivacyRelated Questions on a Q&A Site. In Proc. 18th Symposium on Usable Privacy
and Security (SOUPS’22). USENIX.
Cormac Herley. 2009. So Long, and No Thanks for the Externalities: The
Rational Rejection of Security Advice by Users. In Proc. 2009 Workshop on
New Security Paradigms Workshop (NSPW’09). ACM.
Nicolas Huaman, Sabrina Amft, Marten Oltrogge, Yasemin Acar, and Sascha
Fahl. 2021. They Would do Better if They Worked Together: The Case ofInteraction Problems Between Password Managers and Websites. In Proc.
42nd IEEE Symposium on Security and Privacy (SP’21). IEEE.
Jun Ho Huh, Hyoungshick Kim, Swathi S.V.P. Rayala, Rakesh B. Bobba, andKonstantin Beznosov. 2017. I’m Too Busy to Reset My LinkedIn Password:
On the Effectiveness of Password Reset Emails. In Proc. 2017 CHI Conferenceon Human Factors in Computing Systems (CHI’17). ACM.
Philip G. Inglesant and M. Angela Sasse. 2010. The true cost of unusablepassword policies: password use in the wild. In Proc. 2010 CHI Conference on
Human Factors in Computing Systems (CHI’10). ACM.
Tommaso Innocenti, Seyed Ali Mirheidari, Amin Kharraz, Bruno Crispo, andEngin Kirda. 2021. You’ve Got (a Reset) Mail: A Security Analysis of EmailBased Password Reset Procedures. In Proc. 18th International Conference on
Detection of Intrusions and Malware, and Vulnerability Assessment (DIMVA’21).
Springer.
Iulia Ion, Rob Reeder, and Sunny Consolvo. 2015. “...No one Can Hack My
Mind”: Comparing Expert and Non-Expert Security Practices. In Proc. 11th
Symposium On Usable Privacy and Security (SOUPS’15). USENIX.
Markus Keil, Philipp Markert, and Markus Dürmuth. 2022. “It’s Just a Lot of
Prerequisites”: A User Perception and Usability Analysis of the German ID
Card as a FIDO2 Authenticator. In Proc. 2022 European Symposium on Usable
Security (EuroUSEC’22). ACM.
Erin Kenneally and David Dittrich. 2012. The Menlo Report: Ethical principles
guiding information and communication technology research. SSRN Electronic
Journal.
Saranga Komanduri, Richard Shay, Patrick Gage Kelley, Michelle L Mazurek,
Lujo Bauer, Nicolas Christin, Lorrie Faith Cranor, and Serge Egelman. 2011.
Of Passwords and People: Measuring the Effect of Password-Composition
Policies. In Proc. SIGCHI Conference on Human Factors in Computing Systems(CHI’11). ACM.

A Qualitative Exploration of Online Developer Advice on Usable and Secure Authentication

[56]

[57]

[58]

[59]

[60]

[61]

[62]

[63]

[64]

[65]

[66]

[67]

[68]

[69]

[70]

[71]

[72]

[73]

[74]

[75]
[76]

[77]

[78]

Udo Kuckartz. 2019. Qualitative Text Analysis: A Systematic Approach. In
Compendium for Early Career Researchers in Mathematics Education. Gabriele
Kaiser and Norma Presmeg, (Eds.) Springer, 181–197.
Daniele Lain, Kari Kostiainen, and Srdjan Capkun. 2022. Phishing in Organizations: Findings from a Large-Scale and Long-Term Study. In Proc. 43rd IEEE
Symposium on Security and Privacy (SP’22). IEEE.
Kevin Lee, Benjamin Kaiser, Jonathan Mayer, and Arvind Narayanan. 2020.
An Empirical Study of Wireless Carrier Authentication for SIM Swaps. In
Proc. 16th Symposium on Usable Privacy and Security (SOUPS’20). USENIX.
Kevin Lee and Arvind Narayanan. 2021. Security and Privacy Risks of Number Recycling at Mobile Carriers in the United States. In Proc. 2021 APWG
Symposium on Electronic Crime Research (eCrime). IEEE.
Kevin Lee, Sten Sjöberg, and Arvind Narayanan. 2022. Password policies ofmost top websites fail to follow best practices. In Proc. 18th Symposium on
Usable Privacy and Security (SOUPS’22). USENIX.
Mikael Linden and Inka Vilpola. 2005. An empirical study on the usability oflogout in a single sign-on system. In Int. Conf. on Information Security Practiceand Experience. Springer.
Luigi Lo Iacono, Matthew Smith, Emanuel von Zezschwitz, Peter Leo Gorski,
and Peter Nehren. 2018. Consolidating Principles and Patterns for Humancentred Usable Security Research and Development. In Proc. 3rd European
Workshop on Usable Security (EuroUSEC’18). Internet Society.
Bo Lu, Xiaokuan Zhang, Ziman Ling, Yinqian Zhang, and Zhiqiang Lin. 2018. A
Measurement Study of Authentication Rate-Limiting Mechanisms of Modern
Websites. In Proc. 34th Annual Computer Security Applications Conference(ACSAC’18). ACM.
Sanam Ghorbani Lyastani, Michael Backes, and Sven Bugiel. 2023. A Systematic Study of the Consistency of Two-Factor Authentication User Journeys
on Top-Ranked Websites. In Network and Distributed System Security (NDSS)
Symposium 2023.
Sanam Ghorbani Lyastani, Michael Schilling, Sascha Fahl, Michael Backes,
and Sven Bugiel. 2018. Better managed than memorized? Studying the Impactof Managers on Password Strength and Reuse. In Proc. 27th USENIX Security
Symposium (SEC’18). USENIX.
Philipp Markert, Florian Farke, and Markus Dürmuth. 2019. View The Emailto Get Hacked: Attacking SMS-Based Two-Factor Authentication. In Who Are
You?! Adventures in Authentication Workshop (WAY’19).
Philipp Mayring. 2015. Qualitative Content Analysis: Theoretical Backgroundand Procedures. In Approaches to Qualitative Research in Mathematics Education: Examples of Methodology and Methods. Angelika Bikner-Ahsbahs,
Christine Knipping, and Norma Presmeg, (Eds.) Springer, 365–380.
Michelle Mazurek. 2022. We Are the Experts, and We Are the Problem: The
Security Advice Fiasco. In Proc. 29th ACM SIGSAC Conference on Computerand Communications Security (CCS’22). Keynote. ACM.
Allison McDonald, Carlo Sugatan, Tamy Guberek, and Florian Schaub. 2021.
The Annoying, the Disturbing, and the Weird: Challenges with Phone Numbers as Identifiers and Phone Number Recycling. In CHI Conference on Human
Factors in Computing Systems (CHI ’21). ACM.
Nora McDonald, Sarita Schoenebeck, and Andrea Forte. 2019. Reliability andInter-Rater Reliability in Qualitative Research: Norms and Guidelines for
CSCW and HCI Practice. Proc. ACM Hum.-Comput. Interact., 3, CSCW, (Nov.
2019), 23 pages.
Mohammad Moradi and MohammadReza Keyvanpour. 2014. CAPTCHA and
its Alternatives: A Review. Security and Communication Networks, 8, 12, (Oct.
2014), 2135–2156.
Collins W. Munyendo, Yasemin Acar, and Adam J. Aviv. 2023. “In Eighty
Percent of the Cases, I Select the Password for Them”: Security and Privacy
Challenges, Advice, and Opportunities at Cybercafes in Kenya. In Proc. 44th
IEEE Symposium on Security and Privacy (SP’23). IEEE.
Alena Naiakshina, Anastasia Danilova, Christian Tiefenau, Marco Herzog,
Sergej Dechand, and Matthew Smith. 2017. Why Do Developers Get Password
Storage Wrong?: A Qualitative Usability Study. In Proc. 24th ACM Conference
on Computer and Communication Security (CCS’17). ACM.
Daniela Napoli. 2018. Developing Accessible and Usable Security (ACCUS)
Heuristics. In Extended Abstracts of the 2018 CHI Conference on Human Factors
in Computing Systems (CHI EA’18). ACM.
Net Marketshare. 2021. Search Engine Market Share. Accessed: 2021-11-16.
(2021). https://netmarketshare.com/search-engine-market-share.aspx.
Duc Cuong Nguyen, Dominik Wermke, Yasemin Acar, Michael Backes, Charles
Weir, and Sascha Fahl. 2017. A Stitch in Time: Supporting Android Developers in Writing Secure Code. In Proc. 24th ACM Conference on Computer andCommunication Security (CCS’17). ACM.
NIST. 2017. Digital identity guidelines: authentication and lifecycle management. Accessed: 2022-12-01. (2017). https://pages.nist.gov/800-63-3/sp800-63
b.html.
NIST. 2016. Questions. . . and buzz surrounding draft NIST Special Publication
800-63-3. Accessed: 2022-11-27. (2016). https://www.nist.gov/blogs/cybersecu

CCS ’23, November 26–30, 2023, Copenhagen, Denmark

[79]

[80]
[81]

[82]
[83]

[84]

[85]

[86]

[87]

[88]

[89]

[90]

[91]

[92]

[93]

[94]

[95]

[96]

[97]
[98]

[99]

[100]

rity-insights/questionsand-buzz-surrounding-draft-nist-special-publicatio
n-800-63-3.
Sean Oesch, Scott Ruoti, James Simmons, and Anuj Gautam. 2022. “It BasicallyStarted Using Me:” An Observational Study of Password Manager Usage. In
Proc. 2022 CHI Conference on Human Factors in Computing Systems (CHI’22).
ACM.
Okta. 2022. Access Token Lifetime. (2022). https://www.oauth.com/oauth2-se
rvers/access-tokens/access-token-lifetime/.
OWASP. 2021. Authentication Cheat Sheet. Accessed: 2022-12-01. (2021). https://cheatsheetseries.owasp.org/cheatsheets/Authentication_Cheat_Sheet.ht
ml.
OWASP. 2021. OWASP Cheat Sheet Series. Accessed: 2022-12-01. (2021). https://cheatsheetseries.owasp.org/.
Kentrell Owens, Olabode Anise, Amanda Krauss, and Blase Ur. 2021. User
Perceptions of the Usability and Security of Smartphones as FIDO2 Roaming Authenticators. In Proc. 17th Symposium on Usable Privacy and Security(SOUPS’21). USENIX.
Bing Pan, Helene Hembrooke, Thorsten Joachims, Lori Lorigo, Geri Gay, andLaura Granka. 2007. In Google We Trust: Users’ Decisions on Rank, Position,
and Relevance. Jour. of Comp.-Med. Comm., 12, 3, 801–823.
Arnis Parsovs. 2014. Practical Issues with TLS Client Certificate Authentication. In Proc. 2014 Network and Distributed System Security Symposium(NDSS’14). Internet Society.
Sarah Pearman, Jeremy Thomas, Pardis Emami Naeini, Hana Habib, Lujo
Bauer, Nicolas Christin, Lorrie Faith Cranor, Serge Egelman, and Alain Forget.
2017. Let’s Go in for a Closer Look: Observing Passwords in Their Natural Habitat. In Proc. 24th ACM Conference on Computer and Communication
Security (CCS’17). ACM.
Thanasis Petsas, Giorgos Tsirantonakis, Elias Athanasopoulos, and Sotiris
Ioannidis. 2015. Two-Factor Authentication: Is the World Ready? Quantifying
2FA Adoption. In Proc. 8th European Workshop on System Security (EuroSec’15).
ACM, 7 pages.
Olgierd Pieczul, S. Foley, and Mary Ellen Zurko. 2017. Developer-centeredsecurity and the symmetry of ignorance. In Proc. 2017 New Security Paradigms
Workshop (NSPW’17). ACM.
Elissa M. Redmiles, Sean Kross, and Michelle L. Mazurek. 2016. How I Learned
to Be Secure: A Census-Representative Survey of Security Advice Sources
and Behavior. In Proc. 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS’16). ACM.
Elissa M. Redmiles, Everest Liu, and Michelle L. Mazurek. 2017. You Want Me
To Do What? A Design Study of Two-Factor Authentication Messages. In 3rd
Who Are You?! Adventures in Authentication Workshop (WAY’17). USENIX.
Elissa M. Redmiles, Noel Warford, Amritha Jayanti, Aravind Koneru, Sean
Kross, Miraida Morales, Rock Stevens, and Michelle L. Mazurek. 2020. A
Comprehensive Quality Evaluation of Security and Privacy Advice on theWeb. In Proc. 29th USENIX Security Symposium (SEC’20). USENIX.
Rob Reeder, E Cram Kowalczyk, and Adam Shostack. 2011. Poster: Helping
engineers design NEAT security warnings. In Proc. 7th Symposium on Usable
Privacy and Security (SOUPS’11). ACM.
Robert W. Reeder, Iulia Ion, and Sunny Consolvo. 2017. 152 Simple Steps to
Stay Safe Online: Security Advice for Non-Tech-Savvy Users. IEEE Security
and Privacy, 15, 5, 55–64.
Ken Reese, Trevor Smith, Jonathan Dutson, Jonathan Armknecht, Jacob
Cameron, and Kent Seamons. 2019. A usability study of five two-factor authentication methods. In Proc. 15th Symposium on Usable Privacy and Security(SOUPS’19). USENIX.
Joshua Reynolds, Nikita Samarin, Joseph Barnes, Taylor Judd, Joshua Mason,
Michael Bailey, and Serge Egelman. 2020. Empirical Measurement of Systemic
2FA Usability. In Proc. 29th USENIX Security Symposium (SEC’20). USENIX.
Joshua Reynolds, Trevor Smith, Ken Reese, Luke Dickinson, Scott Ruoti, and
Kent Seamons. 2018. A Tale of Two Studies: The Best and Worst of YubiKey
Usability. In Proc. 39th IEEE Symposium on Security and Privacy (SP’18). IEEE.
Justin Richer. 2022. User Authentication with OAuth 2.0. Accessed: 2022-11-24.
OAuth 2.0 Project, (2022). https://oauth.net/articles/authentication/.
Sena Sahin, Suood Al Roomi, Tara Poteat, and Frank Li. 2023. Investigating
the Password Policy Practices of Website Administrators. In Proc. 44th IEEE
Symposium on Security and Privacy (SP’23). IEEE.
Stuart Schechter, A.J. Brush, and Serge Egelman. 2009. It’s no secret: Measuring the security and reliability of authentication via ‘secret’ questions. In
Proc. 2009 IEEE Symposium on Security and Privacy. IEEE.
Fabian Schwarz, Khue Do, Gunnar Heide, Lucjan Hanzlik, and Christian
Rossow. 2022. FeIDo: Recoverable FIDO2 Tokens Using Electronic IDs. In Proc.
29th ACM Conference on Computer and Communications Security (CCS’22).
ACM.

CCS ’23, November 26–30, 2023, Copenhagen, Denmark

[101]

[102]

[103]

[104]

[105]
[106]
[107]

[108]

[109]

[110]

[111]
[112]

[113]
[114]
[115]

[116]

[117]

[118]

[119]

[120]

[121]

[122]

Richard Shay, Patrick Gage Kelley, Saranga Komanduri, Michelle L. Mazurek,
Blase Ur, Timothy Vidas, Lujo Bauer, Nicolas Christin, and Lorrie Faith Cranor. 2012. Correct Horse Battery Staple: Exploring the Usability of SystemAssigned Passphrases. In Proc. 8th Symposium on Usable Privacy and Security(SOUPS’12). ACM.
Richard Shay, Saranga Komanduri, Adam L. Durity, Phillip Huh, Michelle L.
Mazurek, Sean M. Segreti, Blase Ur, Lujo Bauer, Nicolas Christin, and Lorrie
Faith Cranor. 2016. Designing Password Policies for Strength and Usability.
ACM Transactions on Information and System Security (TISSEC), 18, 4, 1–34.
Richard Shay, Saranga Komanduri, Adam L. Durity, Phillip (Seyoung) Huh,
Michelle L. Mazurek, Sean M. Segreti, Blase Ur, Lujo Bauer, Nicolas Christin,and Lorrie Faith Cranor. 2014. Can Long Passwords Be Secure and Usable?
In Proc. SIGCHI Conference on Human Factors in Computing Systems (CHI’14).
ACM.
James Simmons, Oumar Diallo, Sean Oesch, and Scott Ruoti. 2021. Systematization of Password Manager Use Cases and Design Paradigms. In Proc. 37th
Annual Computer Security Applications Conference (ACSAC’21). ACM.
Stack Overflow. 2022. Stack Overflow Developer Survey 2022. Accessed: 202211-10. (2022). https://insights.stackoverflow.com/survey/2022.
Statcounter. 2021. Search Engine Market Share Worldwide. Accessed: 202111-16. (2021). https://gs.statcounter.com/search-engine-market-share.
Elizabeth Stobert and Robert Biddle. 2014. The Password Life Cycle: User
Behaviour in Managing Passwords. In Proc. 10th Symposium on Usable Privacy
and Security (SOUPS’14). USENIX.
Christian Stransky, Oliver Wiese, Volker Roth, Yasemin Acar, and Sascha Fahl.
2022. 27 Years and 81 Million Opportunities Later: Investigating the Use of
Email Encryption for an Entire University. In Proc. 43rd IEEE Symposium on
Security and Privacy (SP’22). IEEE.
San-Tsai Sun, Eric Pospisil, Ildar Muslukhov, Nuray Dindar, Kirstie Hawkey,
and Konstantin Beznosov. 2011. What Makes Users Refuse Web Single Signon? An Empirical Investigation of OpenID. In Proc. 17th Symposium on Usable
Privacy and Security (SOUPS’11). ACM.
Mohammad Tahaei and Kami Vaniea. 2019. A Survey on Developer-Centred
Security. In Proc. 2019 IEEE European Symposium on Security and Privacy
Workshops (EuroS&PW). IEEE.
Upwork. 2022. Software Developer Hourly Rates. (2022). https://www.upwor
k.com/hire/software-developers/cost/.
Blase Ur, Patrick Gage Kelley, Saranga Komanduri, Joel Lee, Michael Maass,
Michelle L Mazurek, Timothy Passaro, Richard Shay, Timothy Vidas, Lujo
Bauer, Nicolas Christin, and Lorrie Faith Cranor. 2012. How does your password measure up? The effect of strength meters on password creation. In Proc.
21st Usenix Security Symposium (SEC’12). USENIX.
W3C. 2022. A Well-Known URL for Changing Passwords. (2022). https://w3c.github.io/webappsec-change-password-url/.
W3C. 2022. Captcha Alternatives and thoughts. (2022). https://www.w3.org/WAI/GL/wiki/Captcha_Alternatives_and_thoughts.
Ding Wang, Xuan Shan, Qiying Dong, Yaosheng Shen, and Chunfu Jia. 2023.
No Single Silver Bullet: Measuring the Accuracy of Password Strength Meters.
In Proc. 32nd USENIX Security Symposium (SEC’23). USENIX.
Tara Whalen, Thibault Meunier, Mrudula Kodali, Alex Davidson, Marwan
Fayed, Armando Faz-Hernández, Watson Ladd, Deepak Maram, Nick Sullivan,
Benedikt Christoph Wolters, Maxime Guerreiro, and Andrew Galloni. 2022.
Let The Right One In: Attestation as a Usable CAPTCHA Alternative. In Proc.
18th Symposium on Usable Privacy and Security (SOUPS’22). USENIX.
Stephan Wiefling, Luigi Lo Iacono, and Markus Dürmuth. 2019. Is this reallyyou? An empirical study on risk-based authentication applied in the wild. In
IFIP International Conference on ICT Systems Security and Privacy Protection.
Springer, 134–148.
Claes Wohlin. 2014. Guidelines for snowballing in systematic literature studies
and a replication in software engineering. In Proc. 18th Int. Conf. on Evaluation
and Assessment in Software Engineering (EASE’14). ACM.
Jeff Yan and Ahmad Salah El Ahmad. 2008. Usability of CAPTCHAs or Usability Issues in CAPTCHA Design. In Proc. 4th Symposium on Usable Privacy
and Security (SOUPS’08). ACM.
Xin-Li Yang, David Lo, Xin Xia, Zhi-Yuan Wan, and Jian-Ling Sun. 2016. What
security questions do developers ask? a large-scale study of stack overflow
posts. Journal of Computer Science and Technology, 31, 5, 910–924.
Samira Zibaei, Dina Rinoa Malapaya, Benjamin Mercier, Amirali Salehi-Abari,and Julie Thorpe. 2022. Do Password Managers Nudge Secure (Random)
Passwords? In Proc. 18th Symposium on Usable Privacy and Security (SOUPS’22).
USENIX.
Verena Zimmermann and Nina Gerber. 2020. The password is dead, long live
the password – A laboratory study on user perceptions of authentication
schemes. Int. J. of Hum.-Comp. Studies, 133, 26–44.

Jan H. Klemmer et al.

Table 10: Potentially recommendable advice.
AdviceUse generic responses for auth. errors to not weaken security
Require strong passwords
Follow NIST password guidelines (e.g., 800-63B)
Avoid login completely if possible/not necessary
Make password change/reset simple
Verify email address.
Never send the old password to the user
Limit session length intelligently
Allow easy switching between "login" and "register" page
Notify users via mail when password was reset.
Use "login" and "sign up"; not "sign in" and "sign up".
Take environmental factors into consideration (biometrics)
Analyze which 2FA/MFA approach is best suited.
Simplify authentication
Use long enough credentials/tokens.
Directly inform user why password is rejected
Ensure authentication is secure and has good UX
Don’t disclose account existenceUse social login if building apps on top of Google/Facebook
Provide an option to logout/destroy/invalidate the session.
Find a balance for lockout duration
Warn users when caps lock is activated.
Put your login box or link clearly where users can see it
Encourage users to register multiple 2FAs for account recovery
Don’t lock account based on password reset/recovery requests.
Provide backup codes for 2FA
Issue multiple valid recovery codes to each user.
Allow invalidating/revoking all existing recovery codes.
Consider handling with parallel login from multiple devices
Security measures must not prevent normal user from working
Performance must stay adequate no matter how many users
Verify user has active session when changing password
Verify phone numbers
Customize authentication workflow to your needs
Chose only biometrics that fit your application
Enforce minimum/maximum lengths
Invalidate backup code after it was used
Make resetting passwords easy/obvious/simple

A

Docs
13
11
6
3
3
2
2
2
2
2
2
2
2
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1

REPLICATION PACKAGE

For transparency, potential replications of this study, and to support
meta-research, we provide artifacts within a replication package.
Material. We provide the following artifacts: (1) The scenario andtask we used, (2) the materials used for recruiting, (3) the consent
form, (4) the questionnaire from the developer survey, (5) the list ofadvice that we extracted from the documents, (6) the detailed advice
analysis results, (7) the set of documents, queries, and search results,and (8) this extended version of the paper. To ensure participant
privacy, we do not provide any raw questionnaire responses.
Link. The replication package is available at https://doi.org/10.1
7605/OSF.IO/PFB5H.

B

SCENARIO & TASK

We gave software developers the following scenario and task tosearch for online resources on web authentication:
For a research project on web development, we collect sources of
information and advice that are used by software developers in software projects. As we focus on advice sources that could be relevant fordevelopment, this task contains no actual programming, but is aboutfinding advice that might help you or others during development.
Project Description
Imagine you are working on a new software project called KeepYourHealth. KeepYourHealth is a new web-based health platform thatpatients can use to store and keep track of their health-related information. This includes diagnoses, medical images (e.g., x-ray), and otherdata from doctors (similar to an electronic health record; but personal,

A Qualitative Exploration of Online Developer Advice on Usable and Secure Authentication

CCS ’23, November 26–30, 2023, Copenhagen, Denmark

Table 11: Overview of debatable advice, grouped in categories, each with representative advice examples.
ID

Title

D1 Password Policies

D2 Rate Limiting

D3 Security Questions

D4 Authentication
Enforcement

D5 Usage of text
messages (SMS)D6 Terminating
Sessions after
Password Change
D7 (Not) Entering
Passwords Twice
D8 Confusion: OAuth
vs. OIDC
D9 Automatic Logout
Time Span
D10 Error Reportingfor Login Failures

D11 (Not) Unmasking
Passwords

D12 Using GPG
Encrypted Mail
D13 Fingerprint vs.
Facial Recognition
D14 CAPTCHAs

Rationale

Advice Example

Much debatable advice is on password policies. Some recommends very strict password policies: character
sets (like forcing at least a number, symbol, upper case letters, or any combination of those), high minimum
length, or restricting maximum length. However, this is considered legacy according to latest guidelines (e.g.,
by NIST [77 (2017)]) and may also interfere with password manager usage [48 (2021)]. Similarly, regularpassword changes are still advised – despite being unusable and known to have no significant security
effect [6 (1999), 42 (2018), 50 (2010), 55 (2011)]. Preventing password reuse is generally good advice, while
hard to achieve across platforms and services.
Several pieces of advice are about limiting login attempts to prevent password guessing and credential
stuffing [77 (2017)]. However, its implementation can be debated and depends on the use case. We foundadvice to lock an account permanently or already after three attempts, which might lock legitimate users outof their accounts or could be leveraged for DoS attacks [63 (2018)]. Instead, temporary blocks might be more
appropriate. However, one needs to choose an appropriate approach, e.g., based on IP addresses, based on
accounts, or regarding how many failed attempts are tolerated.
Some advice (implicitly) suggests to use security questions. However, those pose a security threat as they
are often easy to guess (e.g., name of pet) [13 (2012), 99 (2009)], have poor usability [12 (2015)] and official
guidelines advise against their use [77 (2017)].

• Use a minimum of 12–16 characters• Generate random 16 digit passwords• Enforce alphanumeric characters• Enforce punctuation in passwords• Enforce a mixture of upper- and lowercase

Some advice is about enforcement, e.g., always enforcing authentication or enforcing 2FA enrollment on
registration. While this does not worsen security and might be an improvement, it is debatable if thisis suitable and necessary in all situations. For example, many online shops work entirely with optional
registration. Similarly, 2FA enforcement might only be reasonable for sensitive accounts but not importantfor mostly anonymous forum accounts, for example.
Some advice is about using SMS for authentication, e.g., sending 2FA codes. In general, SMS can be convenient
for users, and they are therefore widely used [18 (2018)]. However, there are some drawbacks like losing
access to a phone number [59 (2021), 69 (2021)] or SIM swapping attacks [58 (2020)]. NIST also raises
concerns [78 (2016)]. Overall, usage of SMS should be carefully thought about and alternatives considered.
To terminate all running sessions after password change or reset is understandable from a security perspective.
Some advice recommends forcing this automatically. However, this could sign out authorized devices without
any reason and results in tedious re-logins for the user. Instead, it might be more helpful to give the user achoice whether to invalidate active sessions or not.
Whether passwords should be entered twice on registration and password changes or not is not wellresearched. Both variants have pros and cons, like a small probability of typos vs. tedious retyping ofpasswords. However, we believe that one-time-entry with an option to unmask the password field is more
usable (if mistyped passwords can be reset/corrected).
There is some confusion regarding when to use and what the OpenID Connect and OAuth protocols are. Thisis common on the internet, but OAuth refers to authorization, not authentication, as the official documentation
states: “OAuth 2.0 is not an authentication protocol” [97 (2022)]. Therefore, the advice is factually wrong.
Some advice recommends short time spans to terminate a session automatically after inactivity (e.g., for 2–3
minutes). While automatic logouts can be beneficial for security, too short time spans are a usability burden
and might only be necessary for special applications (e.g., banking). Official recommendations advise longer
time spans, with a minimum of 15 minutes [77 (2017)].
Another recommendation is to display when someone entered an old password, when it was changed, or
generally report that the entered password, username, or email is wrong. This is highly problematic, as itleaks information on account existence or credentials a user used in the past. As passwords are reused, this
could weaken security of the user’s other accounts [86 (2017)].

16

22• Lock accounts after three or four failed login

810attempts• Limit to three attempts per account in a set
timespan• Limit at 5 guesses/minute• Limit by account, not by IP• Consider using a security question• Don’t ask all security questions• Don’t use security questions alone for password
reset• Enforce 2FA on registration• Always require/enforce authentication

3

5

2

5

• Send SMS instead of email• Allow users to resend PIN code by SMS• Notify users via text message/SMS• Use SMS-based 2FA• Invalidate all sessions after password change• Force re-login after password reset

4

4

3

3

• Let users confirm passwords by writing it twice.
• Don’t ask for password confirmation by writing

2

3• Implement 3rd-party authentication via OAuth

1

3• Auto Logout: Mobile App 2–3 min• Auto Logout: Web Apps <10 min• Auto Logout: Secure workstation 10–15 min

31• Notify users if they entered an old password• If user typed an old password, output the date it

3

1

1

1

1

1

1

1

1

1

charactersit twice

was changed• Output exactly which is wrong (username,email, or password)• Don’t display passwords on the screen

There is still advice that indicates to always hide passwords in a password field. However, especially whenusers have long and unique passwords, the entry is error-prone [77 (2017)]. Therefore, offering users an
option to unmask the password improves usability. In some cases, displaying an unmasked password has
no security implications (e.g., private environment). To account for other situations, however, masking
passwords should be the secure default.
Using GPG encrypted mails, e.g., to send users their reset tokens, seems to be a good idea in terms of security. • Send reset tokens in GPG encrypted mailHowever, the adoption of email encryption is poor [108 (2022)]. Therefore, this should rather be handled as
an optional feature.
One advice is to prefer fingerprints to facial recognition. This advice is highly general, might depend on the • Prefer fingerprint scanning to facial recognition
technology in use, and could be wrong nowadays. For example, Apple changed from fingerprint Touch ID tofacial recognition Face ID and evaluated the latter to be more secure [9 (2017)].
CAPTCHAs are a barrier that users have to overcome [71 (2014), 116 (2022), 119 (2008)]. Therefore, the • Use CAPTCHAs
advice to use CAPTCHAs should only be followed sparingly and if necessary [77 (2017)]. Furthermore,this also depends on the CAPTCHA type, as some recent CAPTCHA technologies do not require any userinteraction [36 (2022)].

1 Number of distinct pieces of advice.

#Adv.1 #Docs2

2 Number of distinct documents.

voluntary, and under the user’s control) and can be supplemented
with biometric and other data provided by the users. Patients cangather and upload biometric data such as heartbeat (data recorded by
wearables, smartwatches, etc.) for early detection of cardiovasculardiseases. KeepYourHealth should be compatible with both desktop andmobile browsers.
KeepYourHealth serves as a comprehensive and central instance tokeep track of health, share selected data with doctors when needed,and help improve the health of its users. It will be used by people ofany age, with different skills and should therefore be very easy-to-use.

At the same time, the platform needs to be secure, as a recent survey
found that potential users see the data (e.g., medical data, biometricdata, personal identifiable information) that is stored and processedwith KeepYourHealth as highly sensitive.
[Page break.]
Scenario
You are responsible for the architecture and design of an easy-to-use
and secure authentication workflow for the KeepYourHealth platform.
For a kick-off meeting with other members of the development team,imagine that your job is to prepare a presentation with a concept

CCS ’23, November 26–30, 2023, Copenhagen, Denmark

for the authentication system including user registration andlogin. Therefore you have to do an online search.
Requirements:• The authentication process (and all other mechanisms) of KeepYourHealth has to be secure.
• The authentication process and its security features have to beuser-friendly (provide a high level of usability for all kindsof end users like patients, doctors, other health care staff).
• You are free to make all necessary decisions including the
choice of authentication methods, user interface, registration
process, etc. As this project is in an early stage, you can choose
any technology, programming languages, frameworks, andapproaches that seem reasonable to you.
Your Task:• Prepare the kick-off meeting by searching and collecting
resources from the web for 30 minutes that you find helpful
and you might use for KeepYourHealth’s authentication
process as well as the requirements from above.
• Resources and advice can be all kinds of online documents (e.g.,
blog posts, official documentation, guidelines, forum discussions, other websites).
• Report all resources and advice in the following table we provide, so your team can use these resources and its advice laterin the project (like a “knowledge base”).
• Please include for each resource you found its URL and how you
found it (e.g., Google search query, a site you had bookmarked).
Please keep in mind that usability and security is in the focusof this project.
Here is an example of how the table looks like and how it could be
filled: [Example screenshot of the table.][Page break.]• The authentication process for the KeepYourHealth platform
should be user-friendly and secure.
• Search 30 minutes for helpful resources on web authentication you might use for the KeepYourHealth platform.
• It is sufficient if you skim the resources roughly. You don’t
have to read those resources in detail.
• Report your results by filling the following table. You canalso report resources where you found only a small part to be
helpful for your task.
• You don’t have to prepare a presentation for the kick-off meeting. Currently, we are only interested in the resources you findonline.

Figure 4: A screenshot of the search task’s table as used in
the questionnaire.

Jan H. Klemmer et al.

ACM CCS’23 Artifact Appendix:“Make Them Change it Every Week!”: A Qualitative Exploration of
Online Developer Advice on Usable and Secure Authentication
Jan H. Klemmer∗

Marco Gutfleisch†Christian Stransky C

M. Angela Sasse†

Sascha Fahl C

Yasemin Acar‡

∗ Leibniz University Hannover, Germany, {klemmer,stransky}@sec.uni-hannover.de† Ruhr University Bochum, Germany, {marco.gutfleisch,martina.sasse}@rub.de

C CISPA Helmholtz Center for Information Security, Germany, sascha.fahl@cispa.de‡ Paderborn University, Germany, yasemin.acar@uni-paderborn.de

1

Artifact Appendix

1.1

Abstract

Our paper comprises human subjects research and a qualitative analysis of online advice for developers on usable andsecure authentication. Therefore, we provide study materials
and result data in our artifact (but no scripts with experiments,etc.). Hence, we apply only for the Artifact Available badge.
Evaluation should be rather quick for the evaluators.
We provide the following artifacts: (1) The scenario andtask we used, (2) the materials used for recruiting, (3) the
consent form, (4) the questionnaire from the developer survey, (5) the list of advice that we extracted from the documents, (6) the detailed advice analysis results, (7) the set of
documents, queries, and search results, and (8) the extended
version of the paper. To ensure participant privacy, we do notprovide any raw questionnaire responses.
Please see the contained README file for a detailed
overview of the artifact’s content.

1.2

Description & Requirements

This artifact has no specific requirements, as it only consists ofsome study materials and data, including all advice extracted
in the paper.

1.2.1

Security, privacy, and ethical concerns

None. There are no scripts to be run that. Hence, there is norisk for evaluators.

1.2.2

How to access

The artifact is hosted at the Open Science Foundation (OSF)
and is permanently available via the following DOI that is
also linked in the paper: https://doi.org/10.17605/OSF.
IO/PFB5H. (Or alternatively via direct OSF link: https://
osf.io/pfb5h/.)
1.2.3

Hardware dependencies

None.
1.2.4

Software dependencies

None.
1.2.5

Benchmarks

None.

1.3

Set-upThis artifact does not contain any scripts or experiments to
be run. It just consists of the data and materials we provide.
Therefore, the only set-up step is to download the artifact, so
one could access and inspect it.
We note that one might want to use some program to conveniently view the CSV or Markdown files, although it is notnecessary.
1.3.1

Installation

The artifact can be accessed and inspected via the abovementioned links. To download the artifact as a compressed
ZIP archive from OSF, one can visit https://osf.io/pfb5h/files/osfstorage, click “Download this folder,”and extract the archive after downloading.
1.3.2

Basic Test

None.

1.4

Evaluation workflow

As we only apply for the Artifact Available badge, the evaluation should be straight-forward by just checking that thematerials are available. This can be either done directly on
the online platform or more convenient by downloading thematerials locally. Overall, the evaluation should be quick, andwe expect it to take 15–30 minutes at most.

1.5

Notes on Reusability

As the goal of this artifact is to provide the advice we extracted
and analyzed in the paper, other researchers might want toinspect the advice or reuse some advice for their own research.

1.6

VersionBased on the LaTeX template for Artifact Evaluation
V20231005. Submission, reviewing and badging methodology followed for the evaluation of this artifact can be foundat https://secartifacts.github.io/acmccs2023/.